Can I test AWS Glue code locally?

Question

After reading Amazon docs, my understanding is that the only way to run/test a Glue script is to deploy it to a dev endpoint and debug remotely if necessary. At the same time, i

Answer 1

I think the key here is to define what kind of testing do you want to do locally. If you are doing unit testing (i.e. testing just one pyspark script independent of the AWS services supporting that script) then sure you can do that locally. Use a mocking module like pytest-mock, monkeypatch or unittest to mock the AWS and Spark services external to your script while you test the logic that you have written in your pyspark script. For module testing, you could you a workbook environment like AWS EMR Notebooks, Zeppelin or Jupyter. Here you would be able to run your Spark code against test datasources, but you can mock the AWS Services.
For integration testing (i.e. testing your code integrated with the services it depends on, but not a production system) you could launch a test instance of your system from your CI/CD pipeline and then have compute resources (like pytest scripts or AWS Lambda) automate the workflow implemented by your script.