Traversing FTP listing

Question

I am trying to to get all directories\' name from an FTP server and store them in hierarchical order in a multidimensional list or dict

So for example, a server that

Answer 1

If we are using Python look at:

http://docs.python.org/library/os.path.html (os.path.walk)

If there already is a good module for this, don't reinvent the wheel. Can't believe the post two spots above got two ups, anyway, enjoy.

Answer 2

Here's a first draft of a Python 3 script that worked for me. It's much faster than calling cwd(). Pass in server, port, directory, username, and password as arguments. I left output as a list as an exercise for the reader.

import ftplib
import sys

def ftp_walk(ftp, dir):
    dirs = []
    nondirs = []
    for item in ftp.mlsd(dir):
        if item[1]['type'] == 'dir':
            dirs.append(item[0])
        else:
            nondirs.append(item[0])
    if nondirs:
        print()
        print('{}:'.format(dir))
        print('\n'.join(sorted(nondirs)))
    else:
        # print(dir, 'is empty')
        pass
    for subdir in sorted(dirs):
        ftp_walk(ftp, '{}/{}'.format(dir, subdir))

ftp = ftplib.FTP()
ftp.connect(sys.argv[1], int(sys.argv[2]))
ftp.login(sys.argv[4], sys.argv[5])
ftp_walk(ftp, sys.argv[3])

Answer 3

You're not going to like this, but "it depends on the server" or, more accurately, "it depends on the output format of the server".

Different servers can be set to display different output, so your initial proposal is bound to failure in the general case.

The "naive and slow implementation" above will cause enough errors that some FTP servers will cut you off (which is probably what happened after about 7 of them...).

Answer 4

If the server supports the MLSD command, then use the “a directory and its descendants” code from that answer.

Answer 5

Here is a naive and slow implementation. It is slow because it tries to CWD to each directory entry to determine if it is a directory or a file, but this works. One could optimize it by parsing LIST command output, but this is strongly server-implementation dependent.

import ftplib

def traverse(ftp, depth=0):
    """
    return a recursive listing of an ftp server contents (starting
    from the current directory)

    listing is returned as a recursive dictionary, where each key
    contains a contents of the subdirectory or None if it corresponds
    to a file.

    @param ftp: ftplib.FTP object
    """
    if depth > 10:
        return ['depth > 10']
    level = {}
    for entry in (path for path in ftp.nlst() if path not in ('.', '..')):
        try:
            ftp.cwd(entry)
            level[entry] = traverse(ftp, depth+1)
            ftp.cwd('..')
        except ftplib.error_perm:
            level[entry] = None
    return level

def main():
    ftp = ftplib.FTP("localhost")
    ftp.connect()
    ftp.login()
    ftp.set_pasv(True)

    print traverse(ftp)

if __name__ == '__main__':
    main()