AWS Lambda not importing LXML

Question

I am trying to use the LXML module within AWS Lambda and having no luck. I downloaded LXML using the following command:

pip install lxml -t folder


        

Answer 1

The lxml library is os dependent thus we need to have precompiled copy. Below are the steps.

1. Create a docker container.
   docker run -it lambci/lambda:build-python3.8 bash

2. Create a dir named 'lib'(anything you want) and Install lxml into it.
   mkdir lib
pip install lxml -t ./lib --no-deps

3. Open another cmd and run
   docker ps

4. copy the containerid

5. Copy the files from container to host.
   mkdir /home/libraries/opt/python/lib/python3.8/site-packages/
docker cp <containerid>:/var/task/lib /home/libraries/opt/python/lib/python3.8/site-packages/

6. Now you have lxml copy of files compiled from amazonlinux box, If you like to have lxml as Lambda layer. Navigate to /home/libraries/opt and zip the folder named python. Now you can attach the zip in your lambda as layer.

7. If you want lxml library inside lambda. Navigate to /home/libraries/opt/python/lib/python3.8/site-packages/ and copy the lxml folder in your lambda.

Answer 2

Expanding a bit on Mask's answer. In the case of installing lxml in particular, the libxslt and libxml2 libraries are already installed on the AMI that executes the AWS lambda. Therefore it is no need to start a subprocess with a different LD_LIBRARY_PATH as in that answer, it is however necessary to run pip install lxml on an AMI image (it might be possible to cross-compile as well but I don't know how).

Launch an ec2 machine with Amazon Linux ami
Run the following script to accumulate dependencies:
set -e -o pipefail
sudo yum -y upgrade
sudo yum -y install gcc python-devel libxml2-devel libxslt-devel

virtualenv ~/env && cd ~/env && source bin/activate
pip install lxml
for dir in lib64/python2.7/site-packages \
    lib/python2.7/site-packages
do
    if [ -d $dir ] ; then
        pushd $dir; zip -r ~/deps.zip .; popd
    fi
done 

Note that the last steps from Marks answer is left out. You can use lxml straight from the python file that contains the handler method.