Can I test AWS Glue code locally?

Question

After reading Amazon docs, my understanding is that the only way to run/test a Glue script is to deploy it to a dev endpoint and debug remotely if necessary. At the same time, i

Answer 1

You can do this as follows:

1. Install PySpark using

   >> pip install pyspark==2.4.3
   

2. Prebuild AWS Glue-1.0 Jar with Python dependencies: Download_Prebuild_Glue_Jar

3. Copy the awsglue folder and Jar file into your pycharm project from github

4. Copy the Python code from my git repository

5. Run the following on your console; make sure to enter your own path:

   >> python com/mypackage/pack/glue-spark-pycharm-example.py
   

From my own blog