schedule_interval and other gotchas with SubDagOperator

后端 未结 1 1791
情歌与酒
情歌与酒 2021-01-21 23:32

Airflow documentation clearly states

SubDAGs must have a schedule and be enabled. If the SubDAG’s schedule is set to None or @once, the Sub

1条回答
  •  悲&欢浪女
    2021-01-22 00:01

    If you are just running your DAG once, then you probably won't have any issues with SubDags (as in your example) - especially if you have a bunch of worker slots available. Try letting a few DagRuns of your example accumulate and see if everything runs smoothly if you try to delete and re-run some.

    The community has advised moving away from SubDags because unexpected behavior starts happening when you need to re-run old DagRuns or run bigger backfills.

    It is not so much that the DAG won't work, but more that unexpected can happen that may affect your workflows that isn't worth the risk when all you are getting in return is a nicer looking DAG.

    Even though known solutions exist, implementing them may not be worth the effort.

    0 讨论(0)
提交回复
热议问题