Airflow task running tweepy exits with return code -6

Question

I have a simple Airflow DAG which has only one task - stream_from_twitter_to_kafka

Here is the code for the DAG:

default_args = {
    \"owner\": \"me         


        

Answer 1

This is unrelated to airflow or tweepy.

This issue is specific to Mac OS High Sierra and above. https://stackoverflow.com/a/52230415/4434664 solved my issue.

Basically, airflow test merely runs the task in-process, but the scheduler would start a worker process which would call fork(), and apparently, High Sierra introduced some new security changes that's breaking fork() usages in python.

This also caused problems in ansible. Refer https://github.com/ansible/ansible/issues/32499#issuecomment-341578864

Answer 2

I have exatcly the same issue with different operator, sftp_to_s3_operator, Where airflow test works but with environment variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in my .zshrc or .bash_profile file doesn't work.