Run a python script via AWS Data Pipelines

Question

I use AWS Data Pipelines to run nightly SQL queries that populate tables for summary statistics. The UI\'s a bit funky, but eventually I got it up and working.

Now I\'d

Answer 1

1. You need to store your python script on S3 bucket
2. Create Shell script that installs python and all your dependencies, copies your python script from S3 to local storage and runs it. Shell script example.
3. Store this shell script on S3
4. Use ShellCommandActivity to launch your shell script.

You can use this template as an example: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-template-redshiftrdsfull.html It uses python script stored on s3 to convert MySQL schema to RedShift schema.

Example of python shell script that runs python program:

#!/bin/bash
curl -O https://s3.amazonaws.com/datapipeline-us-east-1/sample-scripts/mysql_to_redshift.py
python mysql_to_redshift.py