airflow pool used slots is greater than slots limit

≯℡__Kan透↙ 提交于 2021-02-10 05:34:49

问题


There are three Sensor task and use same pool, the pool 'limit_sensor' is set to 1, but the pool limit not work, three pool is running together

sensor_wait = SqlSensor(
    task_id='sensor_wait',
    dag=dag,
    conn_id='dest_data',
    sql="select count(*) from test",
    poke_interval=10,
    timeout=60,
    pool='limit_sensor',
    priority_weight=100
)

same_pool1 = SqlSensor(
    task_id='same_pool1',
    dag=dag,
    conn_id='dest_data',
    sql="select count(*) from test",
    poke_interval=10,
    timeout=60,
    pool='limit_sensor',
    priority_weight=10
)

same_pool2 = SqlSensor(
    task_id='same_pool2',
    dag=dag,
    conn_id='dest_data',
    sql="select count(*) from test",
    poke_interval=10,
    timeout=60,
    pool='limit_sensor',
    priority_weight=10
)

there is backfill log, airflow 1.10.0

[2018-10-12 11:20:35,036] {jobs.py:2198} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 0 | succeeded: 0 | running: 3 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0

in the web ui Admin->Pools you can see:

Pool    Slots    Used Slots    Queued Slots
limit_sensor    1    3    0

how should i do to make pool limit work? thanks.

来源:https://stackoverflow.com/questions/52771847/airflow-pool-used-slots-is-greater-than-slots-limit

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!