Unzipping directory structure with python

Question

I have a zip file which contains the following directory structure:

dir1\\dir2\\dir3a
dir1\\dir2\\dir3b

I\'m trying to unzip it and maintai

Answer 1

It sounds like you are trying to run unzip to extract the zip.

It would be better to use the python zipfile module, and therefore do the extraction in python.

import zipfile

def extract(zipfilepath, extractiondir):
    zip = zipfile.ZipFile(zipfilepath)
    zip.extractall(path=extractiondir)

Answer 2

Don't trust extract() or extractall().

These methods blindly extract files to the paths given in their filenames. But ZIP filenames can be anything at all, including dangerous strings like “x/../../../etc/passwd”. Extract such files and you could have just compromised your entire server.

Maybe this should be considered a reportable security hole in Python's zipfile module, but any number of zip-dearchivers have exhibited the exact same behaviour in the past. To unarchive a ZIP file with folder structure safely you need in-depth checking of each file path.

Answer 3

I tried this out, and can reproduce it. The extractall method, as suggested by other answers, does not solve the problem. This seems like a bug in the zipfile module to me (perhaps Windows-only?), unless I'm misunderstanding how zipfiles are structured.

testa\
testa\testb\
testa\testb\test.log
> test.zip

>>> from zipfile import ZipFile
>>> zipTest = ZipFile("C:\\...\\test.zip")
>>> zipTest.extractall("C:\\...\\")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\zipfile.py", line 940, in extractall
  File "...\zipfile.py", line 928, in extract
  File "...\zipfile.py", line 965, in _extract_member
IOError: [Errno 2] No such file or directory: 'C:\\...\\testa\\testb\\test.log'

If I do a printdir(), I get this (first column):

>>> zipTest.printdir()
File Name
testa/testb/
testa/testb/test.log

If I try to extract just the first entry, like this:

>>> zipTest.extract("testa/testb/")
'C:\\...\\testa\\testb'

On disk, this results in the creation of a folder testa, with a file testb inside. This is apparently the reason why the subsequent attempt to extract test.log fails; testa\testb is a file, not a folder.

Edit #1: If you extract just the file, then it works:

>>> zipTest.extract("testa/testb/test.log")
'C:\\...\\testa\\testb\\test.log'

Edit #2: Jeff's code is the way to go; iterate through namelist; if it's a directory, create the directory. Otherwise, extract the file.

Answer 4

There's a very easy way if you're using Python 2.6: the extractall method.

However, since the zipfile module is implemented completely in Python without any C extensions, you can probably copy it out of a 2.6 installation and use it with an older version of Python; you may find this easier than having to reimplement the functionality yourself. However, the function itself is quite short:

def extractall(self, path=None, members=None, pwd=None):
    """Extract all members from the archive to the current working
       directory. `path' specifies a different directory to extract to.
       `members' is optional and must be a subset of the list returned
       by namelist().
    """
    if members is None:
        members = self.namelist()

    for zipinfo in members:
        self.extract(zipinfo, path, pwd)

Answer 5

Note that zip files can have entries for directories as well as files. When creating archives with the zip command, pass the -D option to disable adding directory entries explicitly to the archive. When Python 2.6's ZipFile.extractall method runs across a directory entry, it seems to create a file in its place. Since archive entries aren't necessarily in order, this causes ZipFile.extractall to fail quite often, as it tries to create a file in a subdirectory of a file. If you've got an archive that you want to use with the Python module, simply extract it and re-zip it with the -D option. Here's a little snippet I've been using for a while to do exactly that:

P=`pwd` && 
Z=`mktemp -d -t zip` && 
pushd $Z && 
unzip $P/<busted>.zip && 
zip -r -D $P/<new>.zip . && 
popd && 
rm -rf $Z

Replace <busted>.zip and <new>.zip with real filenames relative to the current directory. Then just copy the whole thing and paste it into a command shell, and it will create a new archive that's ready to rock with Python 2.6. There is a zip command that will remove these directory entries without unzipping but IIRC it behaved oddly in different shell environments or zip configurations.

Answer 6

Filter namelist to exclude the folders

All you have to do is filter out the namelist() entries ending with / and the problem is resolved:

  z.extractall(dest, filter(lambda f: not f.endswith('/'), z.namelist()))

nJoy!