How do you successfully invoke gsutil rsync from a python script?

问题

I am trying to execute the following line

gsutil -m rsync s3://input gs://output

in python. When running this line in the shell terminal it works fine. However, I am trying to run this in a python script by using the following line.

subprocess.Popen(["gsutil", "-m", "rsync", "s3://input", "gs://output"])

However it just hangs forever. It outputs the following:

Building synchronization state...
Starting synchronization...

The bash command successfully prints:

Building synchronization state...
Starting synchronization...
Copying s3://input/0000
[0/1 files][  1.0 MiB/ 5.1 MiB]   (number here)% Done

and the file shows in my gs bucket

回答1:

I'm guessing this is because the last two lines are probably written to stderr instead of stdout. Can you try using the call to Popen as a context manager and then calling communicate() to read from the output streams?

proc = subprocess.Popen(["gsutil", "-m", "rsync", "s3://input", "gs://output"])
try:
    outs, errs = proc.communicate(timeout=15)
    # now you can do something with the text in outs and errs
except TimeoutExpired:
    proc.kill()
    outs, errs = proc.communicate()

来源：https://stackoverflow.com/questions/52828000/how-do-you-successfully-invoke-gsutil-rsync-from-a-python-script