apache beam 2.19.0 not running on cloud dataflow anymore due to Could not find a version that satisfies the requirement setuptools>=40.8

问题

Since a few days our python dataflow jobs result in an error on worker startup:

"ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 
(from versions: none)"

ERROR: Command errored out with exit status 1: /usr/local/bin/python3 
/usr/local/lib/python3.5/site-packages/pip install --ignore-installed --no-user --prefix 
/tmp/pip-build-env-qz0ogm1p/overlay --no-warn-script-location --no-binary :none: --only-binary
 :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the 
logs for full command output.

This seems to not be an issue with the requirements.txt, which are:

ua-parser
numpy
pandas
Deprecated

Did anyone come across this and maybe a solution? I did try to set a specific setuptools version (46.1.3) which resulted in another error:

ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)

"ERROR: Command errored out with exit status 1: /usr/local/bin/python3 
/usr/local/lib/python3.5/site-packages/pip install --ignore-installed --no-user --prefix 
/tmp/pip-build-env-2awl4fw3/overlay --no-warn-script-location --no-binary :none: --only-binary 
:none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the
 logs for full command output." 

Thanks

回答1:

I had the same issue than yours. To fix it, I added setuptools in the requirements.txt and make sure it is ordered before apache-beam.

回答2:

The Deprecated package changed from Deprecated-1.2.7 to Deprecated-1.2.8, which caused the error. Dataflow workers run with setuptools 45.1.0 for python 3.5.9, so installing another version through the updated Deprecated package resulted in an error.

来源：https://stackoverflow.com/questions/61078477/apache-beam-2-19-0-not-running-on-cloud-dataflow-anymore-due-to-could-not-find-a