Airflow 1.10 Installation Failing

I have a working Airflow environment using Airflow version 1.9 that is running on an Amazon EC2-Instance. I need to upgrade to the latest version of Airflow which is 1.10. I have the option of either upgrading from version 1.9 or installing 1.10 freshly on a new server. Airflow version 1.10 is not listed on Pip so I'm installing it from Git via this command,

pip-3.6 install git+git://github.com/apache/incubator-airflow.git@v1-10-stable

This command successfully installs Airflow version 1.10. You can see that by running the command airflow version and viewing the output,

  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
   v1.10.0

When I tried starting up the Airflow scheduler with airflow scheduler I get the following exception,

ModuleNotFoundError: No module named 'MySQLdb'
[2018-08-14 14:03:16,195] {celery_executor.py:112} ERROR - Error syncing the celery executor, ignoring it:
[2018-08-14 14:03:16,195] {celery_executor.py:113} ERROR - No module named 'MySQLdb'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 94, in sync
    state = task.state
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 471, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 410, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 365, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 122, in _get_task_meta_for
    session = self.ResultSession()
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
    **self.engine_options)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 45, in create_session
    engine = self.get_engine(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 42, in get_engine
    return create_engine(dburi, poolclass=NullPool)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 391, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 110, in dbapi
    return __import__('MySQLdb')
ModuleNotFoundError: No module named 'MySQLdb'
[2018-08-14 14:03:16,196] {celery_executor.py:112} ERROR - Error syncing the celery executor, ignoring it:
[2018-08-14 14:03:16,196] {celery_executor.py:113} ERROR - No module named 'MySQLdb'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 94, in sync
    state = task.state
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 471, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 410, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 365, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 122, in _get_task_meta_for
    session = self.ResultSession()
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
    **self.engine_options)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 45, in create_session
    engine = self.get_engine(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 42, in get_engine
    return create_engine(dburi, poolclass=NullPool)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 391, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 110, in dbapi
    return __import__('MySQLdb')
ModuleNotFoundError: No module named 'MySQLdb'
[2018-08-14 14:03:16,197] {celery_executor.py:112} ERROR - Error syncing the celery executor, ignoring it:
[2018-08-14 14:03:16,197] {celery_executor.py:113} ERROR - No module named 'MySQLdb'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 94, in sync
    state = task.state
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 471, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 410, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 365, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 122, in _get_task_meta_for
    session = self.ResultSession()
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
    **self.engine_options)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
    engine, session = self.create_session(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 45, in create_session
    engine = self.get_engine(dburi, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 42, in get_engine
    return create_engine(dburi, poolclass=NullPool)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 391, in create_engine
    return strategy.create(*args^C[2018-08-14 14:03:16,424] {jobs.py:1585} INFO - Exited execute loop
[2018-08-14 14:03:16,433] {jobs.py:1599} INFO - Terminating child PID: 13615

Here's what my lib folder has,

[/usr/local/lib/python3.6/site-packages]# cd /usr/local/lib64/python3.6/site-packages/sqlalchemy/
root@ip-1-2-3-4
[/usr/local/lib64/python3.6/site-packages/sqlalchemy]# ll
total 320
drwxr-xr-x  3 root root  4096 Aug 13 17:17 connectors
-rwxr-xr-x  1 root root 40456 Aug 13 17:17 cprocessors.cpython-36m-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root 51408 Aug 13 17:17 cresultproxy.cpython-36m-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root 21944 Aug 13 17:17 cutils.cpython-36m-x86_64-linux-gnu.so
drwxr-xr-x  3 root root  4096 Aug 13 17:17 databases
drwxr-xr-x 10 root root  4096 Aug 13 17:17 dialects
drwxr-xr-x  3 root root  4096 Aug 13 17:17 engine
drwxr-xr-x  3 root root  4096 Aug 13 17:17 event
-rwxr-xr-x  1 root root 49746 Mar  6 14:01 events.py
-rwxr-xr-x  1 root root 12030 Mar  6 14:01 exc.py
drwxr-xr-x  4 root root  4096 Aug 13 17:17 ext
-rwxr-xr-x  1 root root  2249 Mar  6 14:01 __init__.py
-rwxr-xr-x  1 root root  3093 Mar  6 14:01 inspection.py
-rwxr-xr-x  1 root root 10967 Mar  6 14:01 interfaces.py
-rwxr-xr-x  1 root root  6712 Mar  6 14:01 log.py
drwxr-xr-x  3 root root  4096 Aug 13 17:17 orm
-rwxr-xr-x  1 root root 49883 Mar  6 14:01 pool.py
-rwxr-xr-x  1 root root  5217 Mar  6 14:01 processors.py
drwxr-xr-x  2 root root  4096 Aug 13 17:17 __pycache__
-rwxr-xr-x  1 root root  1200 Mar  6 14:01 schema.py
drwxr-xr-x  3 root root  4096 Aug 13 17:17 sql
drwxr-xr-x  5 root root  4096 Aug 13 17:17 testing
-rwxr-xr-x  1 root root  1713 Mar  6 14:01 types.py
drwxr-xr-x  3 root root  4096 Aug 13 17:17 util
root@ip-1-2-3-4
[/usr/local/lib64/python3.6/site-packages/sqlalchemy]# pwd
/usr/local/lib64/python3.6/site-packages/sqlalchemy
root@ip-1-2-3-4
[/usr/local/lib64/python3.6/site-packages/sqlalchemy]# cd /usr/local/lib/python3.6/site-packages/sqlalchemy/
bash: cd: /usr/local/lib/python3.6/site-packages/sqlalchemy/: No such file or directory

I'm just confused why Airflow's installation didn't take care of all it's needed dependencies. Am I installing Airflow incorrectly? I really need to be on version 1.10 because version 1.9 has a major bug in it as discovered here and here.

There are a number of install extras ("optional dependencies") one can provide when doing a fresh install. Airflow doesn't install them all by default because there are dozens and some require special dependencies like Mesos or Kubernetes.

https://airflow.readthedocs.io/en/stable/installation.html#extra-packages

Note that for 1.10.0-1.10.2 you now need to preface install commands or export this env var:

export SLUGIFY_USES_TEXT_UNIDECODE=yes

This is no longer required for 1.10.3 and up.

Once 1.10 is released you'll be able to install extras like this:

pip install apache-airflow[celery,devel,postgres]

When installing from git, the pip syntax for installing extras is a little more complicated:

pip install git+git://github.com/apache/incubator-airflow.git@v1-10-stable#egg=apache-airflow[celery,devel,postgres]

If you're trying to install Airflow with MySQL support, you can include the mysql extra:

pip install git+git://github.com/apache/incubator-airflow.git@v1-10-stable#egg=apache-airflow[mysql]

If you really do want to install all extras, you can use the all extra:

pip install git+git://github.com/apache/incubator-airflow.git@v1-10-stable#egg=apache-airflow[all]

Note: If you previously installed any extras for apache-airflow 1.9 on PyPI, you'd need to provide them again here when installing 1.10 from GitHub since pip doesn't associate the GitHub repo with the PyPI package.

Questions

Are you running Python 3.6.5?
Do you still get the same error if you include mysql extra on install?

来源：https://stackoverflow.com/questions/51861453/airflow-1-10-installation-failing

标签

pip

airflow

airflow-scheduler