How can I make os.walk
traverse the directory tree of an FTP database (located on a remote server)? The way the code is structured now is (comments provided):
I wrote a library pip install walk-sftp. Event though it is named walk-sftp I included a WalkFTP class that lets you filter by start_date of files & end_date of files. You can even pass in a processing_function that returns True or False to see whether your process to clean & store data works. It also has a log parameter (pass filename) that uses pickle & keeps track of any progress so you don't overwrite or have to keep track of dates making backfilling easier.
https://pypi.org/project/walk-sftp/