Configuring Amazon Elastic Beanstalk with PostGIS

Question

Does anyone have any experience setting up Amazon Elastic Beanstalk with PostGIS (so that I can take advantage of Geodjango)?

There are a number of features that th

Answer 1

Open ssh shell of your aws beanstalk. Execute below command step by step. You will get working postgis.

cd /home/ec2-user

sudo yum -y install gcc gcc-c++ make cmake libtool libcurl-devel libxml2-devel rubygems swig fcgi-devel libtiff-devel freetype-devel curl-devel libpng-devel giflib-devel libjpeg-devel cairo-devel freetype-devel readline-devel openssl-devel python27 python27-devel

# PROJ
wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
tar -zxvf proj-4.8.0.tar.gz
cd proj-4.8.0
./configure
make
sudo make install
cd ..

# GEOS
wget http://download.osgeo.org/geos/geos-3.4.2.tar.bz2
tar -xvf geos-3.4.2.tar.bz2
cd geos-3.4.2
./configure
make
sudo make install
cd ..

# GDAL
wget http://download.osgeo.org/gdal/1.10.1/gdal-1.10.1.tar.gz
tar -zxvf gdal-1.10.1.tar.gz
cd gdal-1.10.1
./configure
make
sudo make install
cd ..

# PostGIS
wget http://download.osgeo.org/postgis/source/postgis-2.1.0.tar.gz
tar -xvf postgis-2.1.0.tar.gz
cd postgis-2.1.0
./configure
make
sudo make install
cd ..

Then create a symbolic link:

ln -s /usr/local/lib/libgdal.so /usr/lib/libgdal.so.1
/sbin/ldconfig

Answer 2

As i mentioned here my solution for a 2017.03 image was:

commands:
  01_yum_update:
    command: sudo yum -y update
  02_epel_repo:
    command: sudo yum-config-manager -y --enable epel
  03_install_gdal_packages:
    command: sudo yum -y install gdal gdal-devel

files:
  "/etc/httpd/conf.d/wsgihacks.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      WSGIPassAuthorization On

packages:
  yum:
    git: []
    postgresql95-devel: []
    gettext: []
    libjpeg-turbo-devel: []
    libffi-devel: []

Answer 3

If you want to use radtek's solution and want to use the latest Amazon AMI (2014.9), you might face dependency problems. This solved it for me.

Answer 4

You can also do it without a custom AMI, just use ebextensions. I tested this with Amazon Instance (2013.09) ami-35792c5c so use that one instead of the newer ones. If you have your Django in Elastic Beanstalk 101 completed, you know about ebextensions. The ebextensions below will quickly get going you can use the following ebextensions. Just place the following in your .ebextensions folder at the base of your repository. I also include postgres 9.3 and memcached in these config files:

00_repo_ostgis.config:

files:
  "/etc/yum.repos.d/pgdg-93-redhat.repo":
    mode: "000644"
    owner: root
    group: root
    content: |
      [pgdg93]
      name=PostgreSQL 9.3 $releasever - $basearch
      baseurl=http://yum.postgresql.org/9.3/redhat/rhel-6-$basearch
      enabled=1
      gpgcheck=1
      gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93

      [pgdg93-source]
      name=PostgreSQL 9.3 $releasever - $basearch - Source
      failovermethod=priority
      baseurl=http://yum.postgresql.org/srpms/9.3/redhat/rhel-6-$basearch
      enabled=0
      gpgcheck=1
      gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93

commands:
  epel_repo:
    command: yum-config-manager -y --enable epel
  remi_repo:
    command: yum-config-manager -y --enable remi

packages:
  rpm:
    pgdg-redhat93-9.3-1: 'http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-redhat93-9.3-1.noarch.rpm'
    remi: 'http://rpms.famillecollet.com/enterprise/remi-release-6.rpm'
    qt4-devel: 'http://mirror.centos.org/centos/6/os/x86_64/Packages/qt-4.6.2-28.el6_5.x86_64.rpm'

01_app_postgis.config:

packages:
  yum:
    libtiff-devel: ''
    libjpeg-devel: ''
    libzip-devel: ''
    freetype-devel: ''
    postgresql-devel: ''
    gdal: ''
    gdal-python: ''
    geos: ''
    proj: ''
    libmemcached: ''
    libmemcached-devel: ''
    cyrus-sasl-devel: ''
    zlib-devel: ''

container_commands:
  01_collectstatic:
    command: 'PYTHONPATH=.:..:../lib cd site/<your_project> && ./manage.py collectstatic -c --noinput && cd ../..'
    leader_only: true
  02_migrate:
    command: 'PYTHONPATH=.:..:../lib cd site/<your_project> && ./manage.py migrate --noinput && cd ../..'
    leader_only: true

option_settings:
  - namespace: aws:elasticbeanstalk:container:python
    option_name: WSGIPath
    value: site/<your_project>/wsgi.py
  - namespace: aws:elasticbeanstalk:container:python:staticfiles
    option_name: /static/
    value: site/<your_project>/static/
  - option_name: DJANGO_SETTINGS_MODULE
    value: settings_prod

The structure of my project is a bit different. My settings file and urls.py I moved to the root of my project directory so i had to change the path to settings in wsgi.py. So adjust this accordingly. Just use the container_commands and option_settings you were using before.

Your requirements.txt file should contain at minimum:

Django==1.7.1
Pillow
psycopg2

I store most other python dependencies in ../lib which I include in my PYTHONPATH so my repo structure is like this:

<your_project>/
├── requirements.txt
├── .ebextensions/
│   ├── 00_repos_postgis.config
│   └── 01_app_postgis.config
└── site/
    ├── <your_project>
    │   ├── wsgi.py
    │   ├── settings_prod.py  # used for EB, like settings_local.py but uses env vars
    │   └── settings.py
    └── lib/
        └── <all pip dependencies>

Checkout the deployment tool I built, it uses fabric. I took what I liked from EB CLI tools and adjusted till it was tailored for django: https://github.com/radlws/django-awseb-tasks

NOTE: It is extremely important that you use AMI ami-35792c5c when you launch your environment. Its the only one that worked for me for this setup. If other instances work please feel free to edit them into this answer. Also see my original question.

Answer 5

Here in alternative answer with which the python3 bindings for gdal will work in the AWS EB instance. pygdal is a nice package for “virtualenv and setuptools friendly version of standard GDAL python bindings”, but does not support 1.7.3, which is the gdal version installed using sudo yum -y install gdal gdal-devel, e.g this answer.

Thus, I had to install an updated gdal and also setup linkages to go with it. This method draws on information in yellowcap’s, radek’s and various other Stackoverflow posts.

As yellowcap said, you must set ENVIRONMENT VARIABLES that make the library linkages you need. In the AWS EB console, Configuration --> Software configuration, edit the Environment Properties adding: LD_LIBRARY_PATH  = “/usr/local/lib/:$LD_LIBRARY_PATH”

LD_PRELOAD = ”/usr/lib/libgdal.so.1

Put the following in your .ebextensions folder at the base of your repository:

django.config:

option_settings:
  "aws:elasticbeanstalk:application:environment":
    DJANGO_SETTINGS_MODULE: “<your project>.settings.server_settings"
    PYTHONPATH: "$PYTHONPATH"
  "aws:elasticbeanstalk:container:python":
    WSGIPath: “<your project>/wsgi.py"
  "aws:elasticbeanstalk:container:python:staticfiles":
    "/static/": "www/static/"

commands:
  01_yum_update:
    command: sudo yum -y update
  02_pip_upgrade:
    command: /opt/python/run/venv/bin/pip install --upgrade pip
    ignoreErrors: false

packages:
  yum:
    postgresql95-devel: []
    git: []
    libffi-devel: []

container_commands:
  01_migrate:
    command: "python manage.py migrate"
    leader_only: true
  02_collectstatic:
    command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput”

gdal.config

files:
 "/opt/elasticbeanstalk/hooks/appdeploy/pre/01_install_gdal_prerequisites.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
        #!/usr/bin/env bash

        sudo yum install -y gcc-c++ gcc libpng libtiff postgresql95-devel git libffi-devel libxml2-devel libxslt-devel htop

        wget http://download.osgeo.org/proj/proj-4.9.3.tar.gz
        tar -zvxf proj-4.9.3.tar.gz
        cd proj-4.9.3
        ./configure
        make -j 2
        sudo make install
        cd ..
        rm proj-4.9.3.tar.gz

        wget http://download.osgeo.org/gdal/2.3.2/gdal-2.3.2.tar.gz
        tar -zvxf gdal-2.3.2.tar.gz
        cd gdal-2.3.2
        ./configure --with-static-proj4=/usr/local/lib --with-python --with-threads --with-pg=/usr/bin/pg_config
        make -j 2
        sudo make install
        cd ..
        rm gdal-2.3.2.tar.gz

        ln -s /usr/local/lib/libgdal.so /usr/lib/libgdal.so.1
        sudo ldconfig

in requirements.py include the pygdal package

pygdal==2.3.2.4

You must be sure that the gdal version number matches the pygdal version number, see pygdal documentation.

Result

The following works the EC2 instance virtual env, where your django project is located:

$ python manage.py shell
In [1]: from osgeo import gdal

In [2]: 

Answer 6

If you want to use geodjango with Amazon Elastic Beanstalk you need to create a custom AMI where you can install PostGIS and then point your Elastic Beanstalk Application to that AMI when spinning up.

Here is a good tutorial on how to customize an EBS AMI. There is also an AWS tutorial for that but I found the first one easier to understand. On my custom AMI I installed geos, gdal, proj4 and postgis from source, and postgres using yum install postgres. Below are the commands i used to install all libraries into the AMI.

For the django app to find the libraries, I also set an additional environmental variable in the AWS EBS Console. In the menubar of my environment, I went to configuration --> software configuration and edited the Environment Properties by adding the property LD_LIBRARY_PATH set as /usr/local/lib/:$LD_LIBRARY_PATH.

Since the beanstalk app instances are not forseen to run the database themselves, I also set up a Amazon RDS Postgres hosted database which is a relatively new service, it supports PostGIS.

If you put that all together, you should get a very scaleable GeoDjango app!

sudo yum install postgresql postgresql-devel postgresql-server postgresql9-contrib gcc gcc-c++ make libtool curl libxml2 libxml2-devel python-devel

wget http://download.osgeo.org/proj/proj-4.8.0.zip
unzip proj-4.8.0.zip
cd proj-4.8.0
./configure
make
sudo make install
cd ..

wget http://download.osgeo.org/geos/geos-3.4.2.tar.bz2
tar -xvf geos-3.4.2.tar.bz2
cd geos-3.4.2
./configure
make
sudo make install
cd ..

wget http://download.osgeo.org/gdal/1.10.1/gdal1101.zip
unzip gdal1101.zip
cd gdal-1.10.1
./configure --with-python=yes
make
sudo make install
cd ..

wget http://download.osgeo.org/postgis/source/postgis-2.1.1.tar.gz
tar -xvf postgis-2.1.1.tar.gz
cd postgis-2.1.1
./configure
make
sudo make install