Tasks targeted at dynamic backend fail frequently, silently

两盒软妹~` 提交于 2019-12-22 09:23:55

问题


I had converted some tasks to run on a dynamic backend.

The tasks are failing silently [no logged error, no retry, nothing] ~20% of the time (min:10%, max:60%, sample:large, long term). Switching the task away from the backend restores retries and gets the failure rate back to ~0%.

Any ideas?


回答1:


Converting it to a backend exacerbated the problem but wasn't the problem.

I had specified a task_retry_limit and the queue was a push queue. With a backend the number of instances is specified. (I believe you can replicate this issue on the frontend by ramping up requests rapidly, to a big number).

Tasks were failing 503: Instance Unavailable until they hit the task_retry_limit. This is visible temporarily in Task Queues, but will not show up in Logs.

I should be using pull queues. Even if my use case was stupid I'd probably +1 a task dying due to multiple 503: Instance Unavailable logging something so it doesn't appear like a phantom task.




回答2:


Which runtime are you using on the backend? Try running the backend for a bit without dynamic set to true and exercise the failing component.

On my project, I have seen tasks that target a static backend disappear on occasion, but no where near the rate you are seeing.



来源:https://stackoverflow.com/questions/17067361/tasks-targeted-at-dynamic-backend-fail-frequently-silently

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!