Using Python's ftplib to get a directory listing, portably

Question

You can use ftplib for full FTP support in Python. However the preferred way of getting a directory listing is:

# File: ftplib-example-1.py

import ftplib

f         


        

Answer 1

This is from Python docs

>>> from ftplib import FTP_TLS
>>> ftps = FTP_TLS('ftp.python.org')
>>> ftps.login()           # login anonymously before securing control 
channel
>>> ftps.prot_p()          # switch to secure data connection
>>> ftps.retrlines('LIST') # list directory content securely
total 9
drwxr-xr-x   8 root     wheel        1024 Jan  3  1994 .
drwxr-xr-x   8 root     wheel        1024 Jan  3  1994 ..
drwxr-xr-x   2 root     wheel        1024 Jan  3  1994 bin
drwxr-xr-x   2 root     wheel        1024 Jan  3  1994 etc
d-wxrwxr-x   2 ftp      wheel        1024 Sep  5 13:43 incoming
drwxr-xr-x   2 root     wheel        1024 Nov 17  1993 lib
drwxr-xr-x   6 1094     wheel        1024 Sep 13 19:07 pub
drwxr-xr-x   3 root     wheel        1024 Jan  3  1994 usr
-rw-r--r--   1 root     root          312 Aug  1  1994 welcome.msg

Answer 2

There's no standard for the layout of the LIST response. You'd have to write code to handle the most popular layouts. I'd start with Linux ls and Windows Server DIR formats. There's a lot of variety out there, though.

Fall back to the nlst method (returning the result of the NLST command) if you can't parse the longer list. For bonus points, cheat: perhaps the longest number in the line containing a known file name is its length.

Answer 3

I found my way here while trying to get filenames, last modified stamps, file sizes etc and wanted to add my code. It only took a few minutes to write a loop to parse the ftp.dir(dir_list.append) making use of python std lib stuff like strip() (to clean up the line of text) and split() to create an array.

ftp = FTP('sick.domain.bro')
ftp.login()
ftp.cwd('path/to/data')

dir_list = []
ftp.dir(dir_list.append)

# main thing is identifing which char marks start of good stuff
# '-rw-r--r--   1 ppsrt    ppsrt      545498 Jul 23 12:07 FILENAME.FOO
#                               ^  (that is line[29])

for line in dir_list:
   print line[29:].strip().split(' ') # got yerself an array there bud!
   # EX ['545498', 'Jul', '23', '12:07', 'FILENAME.FOO']

Answer 4

That helped me with my code.

When I tried feltering only a type of files and show them on screen by adding a condition that tests on each line.

Like this

elif command == 'ls':
    print("directory of ", ftp.pwd())
    data = []
    ftp.dir(data.append)

    for line in data:
        x = line.split(".")
        formats=["gz", "zip", "rar", "tar", "bz2", "xz"]
        if x[-1] in formats:
            print ("-", line)

Answer 5

The reliable/standardized way to parse FTP directory listing is by using MLSD command, which by now should be supported by all recent/decent FTP servers.

import ftplib
f = ftplib.FTP()
f.connect("localhost")
f.login()
ls = []
f.retrlines('MLSD', ls.append)
for entry in ls:
    print entry

The code above will print:

modify=20110723201710;perm=el;size=4096;type=dir;unique=807g4e5a5; tests
modify=20111206092323;perm=el;size=4096;type=dir;unique=807g1008e0; .xchat2
modify=20111022125631;perm=el;size=4096;type=dir;unique=807g10001a; .gconfd
modify=20110808185618;perm=el;size=4096;type=dir;unique=807g160f9a; .skychart
...

Starting from python 3.3, ftplib will provide a specific method to do this:

• http://bugs.python.org/issue11072
• http://hg.python.org/cpython/file/67053b135ed9/Lib/ftplib.py#l535

Answer 6

I happen to be stuck with an FTP server (Rackspace Cloud Sites virtual server) that doesn't seem to support MLSD. Yet I need several fields of file information, such as size and timestamp, not just the filename, so I have to use the DIR command. On this server, the output of DIR looks very much like the OP's. In case it helps anyone, here's a little Python class that parses a line of such output to obtain the filename, size and timestamp.

import datetime

class FtpDir:
    def parse_dir_line(self, line):
        words = line.split()
        self.filename = words[8]
        self.size = int(words[4])
        t = words[7].split(':')
        ts = words[5] + '-' + words[6] + '-' + datetime.datetime.now().strftime('%Y') + ' ' + t[0] + ':' + t[1]
        self.timestamp = datetime.datetime.strptime(ts, '%b-%d-%Y %H:%M')

Not very portable, I know, but easy to extend or modify to deal with various different FTP servers.