I use AWS Data Pipelines to run nightly SQL queries that populate tables for summary statistics. The UI\'s a bit funky, but eventually I got it up and working.
Now I\'d
You can use this template as an example: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-template-redshiftrdsfull.html It uses python script stored on s3 to convert MySQL schema to RedShift schema.
Example of python shell script that runs python program:
#!/bin/bash
curl -O https://s3.amazonaws.com/datapipeline-us-east-1/sample-scripts/mysql_to_redshift.py
python mysql_to_redshift.py
I faced similar situation, here how i over come it.
I am going to describe how i did it with Ec2Resource. if you are looking for solution in EMRCluster refer @franklinsijo answer.
Steps
1. Store your python script in s3.
2. create a shell script(hello.sh)(given bellow) and store it to s3
3. Create a Ec2Resource Node and ShellCommandActivity Node and provide these information.
ShellCommandActivity
. And it should runs on your DefaultResource
Here is the shell script(hello.sh) which download your python program from s3 and stores locally, install python and required 3rd party library and finally execute your python file.
hello.sh
echo 'Download python file to local temp'
aws s3 cp s3://path/to/python_file/hello_world.py /tmp/hello.py
# Install python(on CentOs )
sudo yum -y install python-pip
pip install <dependencies>
python /tmp/hello.py
I had hard time while trying with bang line so do not included them here.
if aws cp
command doesn't works(awscli is older), here is a quick solution for this case.
s3DataNode
.ShellCommandActivity
ShellCommandActivity
Command
echo 'Install Python2'
sudo yum -y install python-pip
pip install <dependencies>
python ${INPUT1_STAGING_DIR}/hello_world.py