Airflow ModuleNotFoundError: No module named 'pyspark'

问题

I installed Airflow on my machine which works well and I have a local spark also (which is operational too). I want to use airflow to orchestrate two sparks tasks: task_spark_datatransform >> task_spark_model_reco. The two pyspark modules associated to these two tasks are tested and work well under spark.

I also create a very simple Airflow Dag using bashOperator* to run each spark task. For example, for the task task_spark_datatransform I have:

task_spark_datatransform = BashOperator (task_id = 'task_spark_datatransform', bash_command = spark_home + 'spark-submit --master local [*]' + srcDir + 'dataprep.py'),
where, in my case, spark_home = '/usr/bin/spark/bin/'

*As indicated in several serious tutorials on this same subject.

Question: Why doesn't Airflow recognize pyspark?

Log:

[2019-09-20 10:21:21 +0200] [5945] [INFO] Worker exiting (pid: 5945)
[2019-09-20 10:21:51 +0200] [5554] [INFO] Handling signal: ttin
[2019-09-20 10:21:51 +0200] [6128] [INFO] Booting worker with pid: 6128
[2019-09-20 10:21:51,609] {__init__.py:51} INFO - Using executor SequentialExecutor
[2019-09-20 10:21:52,021] {__init__.py:305} INFO - Filling up the DagBag from /home/ach/airflow/dags
[2019-09-20 10:21:52,026] {__init__.py:416} ERROR - Failed to import: /home/ach/airflow/dags/spark_af.py
Traceback (most recent call last):
  File "/home/ach/airflow/lib/python3.7/site-packages/airflow/models/__init__.py", line 413, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/home/ach/airflow/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ach/airflow/dags/spark_af.py", line 3, in <module>
    import dataprep
  File "/home/ach/airflow/dags/dataprep.py", line 2, in <module>
    from pyspark.sql import SparkSession
ModuleNotFoundError: No module named 'pyspark'

回答1:

Looks like you are missing pyspark:

Run the following:

pip install pyspark

来源：https://stackoverflow.com/questions/58024889/airflow-modulenotfounderror-no-module-named-pyspark

标签

apache-spark

pyspark

airflow