Concatenate gzipped files with Python, on Windows

问题

Is there a memory-efficient way to concatenate gzipped files, using Python, on Windows, without decompressing them?

According to a comment on this answer, it should be as simple as:

cat file1.gz file2.gz file3.gz > allfiles.gz

but how do I do this with Python, on Windows?

回答1:

Just keep writing to the same file.

with open(..., 'wb') as wfp:
  for fn in filenames:
    with open(fn, 'rb') as rfp:
      shutil.copyfileobj(rfp, wfp)

回答2:

You don't need python to copy many files to one. You can use standard Windows "Copy" for this:

copy file1.gz /b + file2.gz /b + file3.gz /b allfiles.gz

Or, simply:

copy *.gz /b allfiles.gz

But, if you wish to use Python, Ignacio's answer is a better option.

回答3:

cat file1.gz file2.gz file3.gz > allfiles.gz

works, then this should work too:

fileList = ['file1.gz', 'file2.gz', 'file3.gz']
destFilename = 'allfiles.gz'

bufferSize = 8  # Adjust this according to how "memory efficient" you need the program to be.

with open(destFilename, 'wb') as destFile:
    for fileName in fileList:
        with open(fileName, 'rb') as sourceFile:
            chunk = True
            while chunk:
                chunk = sourceFile.read(bufferSize)
                destFile.write(chunk)

回答4:

Fortunately, gzipped files can be directly concatenated via the cat CL command, but unfortunately there doesn't seem to be an obvious python command to do this (in the standard library gzip anyways). However, I only looked briefly. There are probably libraries out there to accomplish this.

Nonetheless, a way to accomplish this using the standard library is to call cat using subprocess:

from subprocess import check_call
command = "cat {} {} > {}".format(file1_path, file2_path, output_name)
check_call(command.split())  # Check call takes a list

To generalize this to arbitrary numbers of inputs, you can do:

inputs = ['input1', 'input2', ... 'input9001']
output_name = 'output.gz'

command = "".join(['cat ', '{} ' * len(inputs), '> {out}'])
_call_ = command.format(*inputs, out=output_name).split()

check_call(_call_)

I hope that is helpful to someone.

来源：https://stackoverflow.com/questions/18208898/concatenate-gzipped-files-with-python-on-windows

标签

python

gzip

concatenation