AWS Lambda not importing LXML

Question

I am trying to use the LXML module within AWS Lambda and having no luck. I downloaded LXML using the following command:

pip install lxml -t folder
         


        

Answer 1

I have solved this using the serverless framework and its built-in Docker feature.

Requirement: You have an AWS profile in your .aws folder that can be accessed.

First, install the serverless framework as described here. You can then create a configuration file using the command serverless create --template aws-python3 --name my-lambda. It will create a serverless.yml file and a handler.py with a simple "hello" function. You can check if that works with a sls deploy. If that works, serverless is ready to be worked with.

Next, we'll need an additional plugin named "serverless-python-requirements" for bundling Python requirements. You can install it via sls plugin install --name serverless-python-requirements.

This plugin is where all the magic happens that we need to solve the missing lxml package. In the custom->pythonRequirements section you simply have to add the dockerizePip: non-linux property. Your serverless.yml file could look something like this:

service: producthunt-crawler

provider:
  name: aws
  runtime: python3.8

functions:
  hello:
    # some handler that imports lxml
    handler: handler.hello

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    fileName: requirements.txt
    dockerizePip: non-linux

    # Omits tests, __pycache__, *.pyc etc from dependencies
    slim: true

This will run the bundling of python requirements inside a pre-configured docker container. After this, you can run sls deploy to see the magic happen and then sls invoke -f my_function to check that it works.

When you've used serverless to deploy and add the dockerizePip: non-linux option later, make sure to clean up your already built requirements with sls requirements clean. Otherwise, it just uses the already built stuff.