Using psycopg2 with Lambda to Update Redshift (Python)

Question

I am attempting to update Redshift from a Lambda function using python. To do this, I am attempting to combine 2 code fragments. Both fragments are functional when I run the

Answer 1

oh boy! while some of the answers may be really great and working! Just stumbled upon this https://pypi.org/project/aws-psycopg2/ and it worked like a charm for me. steps :

mkdir aws-psycopg2

cd aws-psycopg2

vi get_layer_packages.sh

export PKG_DIR="python"

rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}

docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
    pip install -r requirements.txt --no-deps -t ${PKG_DIR}

vi requirements.txt

aws-psycopg2

then do : chmod +x get_layer_packages.sh

./get_layer_packages.sh

zip -r aws-psycopg2.zip .

upload this zip to the AWS Lambda Layer!

Answer 2

So many answers but didn't work for me! Please note AWS-Lambda doesn't have most of DB-related libraries preinstalled and so you need to add a zip file with code and libraries to get it working. Please follow the steps here: Link

Answer 3

In order for this to work you need to build psycopg2 with statically linked libpq.so library. Check out this repo https://github.com/jkehler/awslambda-psycopg2. It has already build psycopg2 package and instructions how to build it yourself.

Back to your questions:

What is causing this problem?

psycopg2 needs to be build an compiled with statically linked libraries for Linux.

Does it matter that Lambda uses Python 2.7 when I use 3.4?

Yes it does, lambda only supports 2.7 version. Just create virtual environment and install all necessary packages in there.

Does it matter that I zipped the contents of my file on a Windows machine?

As long as all the libraries you zipped could ran on Linux it doesn't

Has anyone been able to successfully connect to Redshift from lambda?

yes.

Answer 4

To use psycopg2 with aws lambda, use import aws-psycopg2

As aws supports psycopg2 but the way to import psycopg2 is a bit different as aws itself has a compiled library for psycopg2, so we need to import is using aws-psycopg2

Answer 5

Assuming your packaging is correct, the no module named psycopg2 error typically indicates the binary file(s) of your psycopg2 deployment is incorrect for your target OS or Python version.

For Lambdas, we have found that the psycopg2 binary works (using manylinux_x86_64). There is a reported risk of segfault due to the presence of competing libssl binaries though we haven't had that yet. (this is basically a +1 for jshammon's answer above)

The "proper solution" is probably jkehlers recompile specifically for Lambda missing only lib_pq.so, but it doesn't currently support ssl+py3.7 and we are too Windows to recompile it ourselves.

Answer 6

I just came across this same problem. I stumbled across the same github project that was noted in the other answer which explained the problem as follows:

Due to AWS Lambda missing the required PostgreSQL libraries in the AMI image, we needed to compile psycopg2 with the PostgreSQL libpq.so library statically linked libpq library instead of the default dynamic link.

This has been noted in the previous answer, and I started to follow the instructions to build myself a version of psycopg2 with a statically linked PostgreSQL library. I found a much easier option though. I noticed on the psycopg2 github page the following:

You can also obtain a stand-alone package, not requiring a compiler or external libraries, by installing the psycopg2-binary package from PyPI:

$ pip install psycopg2-binary

The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources.

When I pip installed the psycopg2-binary package and included it in my requirements.txt file I was able to connect to my postgresql database from a lambda function flawlessly. I am using chalice which I highly recommend. I realize that psycopg2 recommends not using the binary version for production, but I don't see a huge difference between using the binary version or compiling and statically linking it yourself. Someone please correct me if I'm wrong on that.