Pandas & AWS Lambda

Question

Does anyone have a fully compiled version of pandas that is compatible with AWS Lambda?

After searching around for a few hours, I cannot seem to find what I\'m looki

Answer 1

I managed to deploy a pandas code in aws lambda using python3.6 runtime . this is the step that i follow :

1. Add required libraries into requirements.txt
2. Build project in a docker container (using aws sam cli : sam build --use-container)
3. Run code (sam local invoke --event test.json)

this is a helper : https://github.com/ysfmag/aws-lambda-py-pandas-template

Answer 2

After some tinkering around and lot's of googling I was able to make everything work and setup a repo that can just be cloned in the future.

Key takeaways:

1. All static packages have to be compiled on an ec2 amazon Linux instance
2. The python code needs to load the libraries in the lib/ folder before executing.

Github repo: https://github.com/moesy/AWS-Lambda-ML-Microservice-Skeleton

Answer 3

My solution has been to maintain 2 requirements.txt style files of packages that go in my layer, one named provided_packages.txt and one named provided_linux_installs.txt

Before deployment (if the packages are not already installed) I run:

pip install -r provided_packages.txt -t layer_name/python/lib/python3.8/site-packages/.
pip download -r provided_linux_installs.txt --platform manylinux1_x86_64 --no-deps -d layer_name/python/lib/python3.8/site-packages

cd layer_name/python/lib/python3.8/site-packages 
unzip \*.whl
rm *.whl

Then deploy normally (I am using cdk synth & cdk deploy \* --profile profile_name)

In case helpful, my provided_linux_installs.txt looks like this:

pandas==1.1.0
numpy==1.19.1
pytz==2020.1
python-dateutil==2.8.1

Answer 4

The repo mthenw/awesome-layers lists several publicly available aws lambda layers.

In particular, keithrozario/Klayers has pandas+numpy and is up-to-date as of today with pandas 0.25.

Its ARN is arn:aws:lambda:us-east-1:113088814899:layer:Klayers-python37-pandas:1

Answer 5

There are some precompiled packages on github by ryfeus.

Answer 6

Another option is to download the pre-compiled wheel files as discussed on this post: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-package-compatible/

Essentially, you need to go to the project page on https://pypi.org and download the files named like the following:

• For Python 2.7: module-name-version-cp27-cp27mu-manylinux1_x86_64.whl
• For Python 3.6: module-name-version-cp36-cp36m-manylinux1_x86_64.whl

Then unzip the .whl files to your project directory and re-zip the contents together with your lambda code.

NOTE: The main Python function file(s) must be in the root folder of the resulting deployment package .zip file. Other Python modules and dependencies can be in sub-folders. Something like:

my_lambda_deployment_package.zip
├───lambda_function.py
├───numpy
│   ├───[subfolders...]
├───pandas
│   ├───[subfolders...]
└───[additional package folders...]