AWS Lambda and python numpy module

Question

I am trying to import a python deployment package in aws lambda. The python code uses numpy. I followed the deployment package instructions for virtual env but it still gave

Answer 1

Using Numpy is a real pain.

Numpy needs to be properly compiled on the same OS as it runs. This means that you need to install/compile Numpy on an AMI image in order for it erun properly in Lambda.

The easiest way to do this is to start a small EC2 instance and install it there. Then copy the compiled files (from /usr/lib/python/site-packages/numpy). These are the files you need to include in your Lambda package.

I believe you can also use the serverless tool to achieve this.

Answer 2

NumPy must be compiled on the platform that it will be run on. Easiest way to do this is to use Docker. Docker has a lambda container. Compile NumPy locally in Docker with the lambda container, then push to AWS lambda.

The serverless framework handles all this for you if you want an easy solution. See https://stackoverflow.com/a/50027031/1085343

Answer 3

I was having this same issue and pip install wasn't working for me. Eventually, I found some obscure posts about this issue here and here. They suggested going to pypi, downloading the .whl file (select py version and manylinux1_x86_64), uploading, and unzipping. This is what ultimately worked for me. If nothing else is working for you, I would suggest trying this.

Answer 4

For Python 3.6, you can use a Docker image as someone already suggested. Full instructions for that here: https://medium.com/i-like-big-data-and-i-cannot-lie/how-to-create-an-aws-lambda-python-3-6-deployment-package-using-docker-d0e847207dd6

The key piece is:

docker run -it dacut/amazon-linux-python-3.6

Answer 5

The easiest way is to use AWS Cloud9, there is no need to start EC2 instances and prepare deployment packages.

Step 1: start up Cloud9 IDE

• Go to the AWS console and select the Cloud9 service.
• Create environment
• enter Name
• Environment settings (consider using t2.small instance type, with the default I had sometimes problems to restart the environment)
• Review
• click Create environment

Step 2: create Lambda function

• Create Lambda Function (bottom of screen)
• enter Function name and Application name
• Select runtime (Python 3.6) and blueprint (empty-python)
• Function trigger (none)
• Create serverless application (keep defaults)
• Finish
• wait couple of seconds for the IDE to open

Step 3: install Numpy

• at the bottom of the screen there is a command prompt

• go to the Application folder, for me this is

  cd App
  

• install the package (we have to use pip from the virtual environment, because the default pip points to /usr/bin/pip and this is python 2.7)

  venv/bin/pip install numpy -t .
  

Step4: test installation

• you can test the installation by editing the lambda_function.py file:

  import numpy as np
  def lambda_handler(event, context):
      return np.sin(1.0)
  

• save the changes and click the green Run button on the top of the screen

• a new tab should appear, now click the green Run button inside the tab

• after the Lambda executes I get:

  Response
  0.8414709848078965
  

Step 5: deploy Lambda function

• on the right hand side of the screen select your Lambda function
• click the upwards pointing arrow to deploy
• go to the AWS Lambda service tab

• the Lambda function should be visible, the name has the format

  cloud9-ApplicationName-FunctionName-RandomString