SSH.NET SFTP Get a list of directories and files recursively

Question

I am using Renci.SshNet library to get a list of files and directories recursively by using SFTP. I can able to connect SFTP site but I am not sure how to get a list of director

Answer 1

Try this:

var filePaths = client.ListDirectory(client.WorkingDirectory);

Answer 2

Here is a full class. It's .NET Core 2.1 Http trigger function app (v2)

I wanted to get rid of any directories that start with '.', cause my sftp server has .cache folders and .ssh folders with keys. Also didn't want to have to deal with folder names like '.' or '..'

What I will end up doing is projecting the SftpFile into a type that I work with and return that to the caller (in this case it will be a logic app). I'll then pass that object into a stored procedure and use OPENJSON to build up my monitoring table. This is basically the first step in creating my SFTP processing queue that will move files off my SFTP folder and into my Data Lake (blob for now until I come up with something better I guess).

The reason I used .WorkingDirectory is because I created a user with home directory as '/home'. This lets me traverse all of my user folders. My app doesn't need to have a specific folder as a starting point, just the user 'root' so to speak.

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Renci.SshNet;
using Renci.SshNet.Sftp;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace SFTPFileMonitor
{
    public class GetListOfFiles
    {
        [FunctionName("GetListOfFiles")]
        public async Task<IActionResult> RunAsync([HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequest req, ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            List<SftpFile> zFiles;
            int fileCount;
            decimal totalSizeGB;
            long totalSizeBytes;

            using (SftpClient sftpClient = new SftpClient("hostname", "username", "password"))
            {
                sftpClient.Connect();
                zFiles = await GetFiles(sftpClient, sftpClient.WorkingDirectory, new List<SftpFile>());
                fileCount = zFiles.Count;
                totalSizeBytes = zFiles.Sum(l => l.Length);
                totalSizeGB = BytesToGB(totalSizeBytes);
            }

            return new OkObjectResult(new { fileCount, totalSizeBytes, totalSizeGB, zFiles });
        }
        private async Task<List<SftpFile>> GetFiles(SftpClient sftpClient, string directory, List<SftpFile> files)
        {
            foreach (SftpFile sftpFile in sftpClient.ListDirectory(directory))
            {
                if (sftpFile.Name.StartsWith('.')) { continue; }

                if (sftpFile.IsDirectory)
                {
                    await GetFiles(sftpClient, sftpFile.FullName, files);
                }
                else
                {
                    files.Add(sftpFile);
                }
            }
            return files;
        }
        private decimal BytesToGB(long bytes)
        {
            return Convert.ToDecimal(bytes) / 1024 / 1024 / 1024;
        }
    }
}

Answer 3

This library has some quirks that make this recursive listing tricky because the interaction between the ChangeDirectory and ListDirectory do not work as you may expect.

The following does not list the files in the /home directory instead it lists the files in the / (root) directory:

sftp.ChangeDirectory("home");
sftp.ListDirectory("").Select (s => s.FullName);

The following does not work and returns a SftpPathNotFoundException:

sftp.ChangeDirectory("home");
sftp.ListDirectory("home").Select (s => s.FullName);

The following is the correct way to list the files in the /home directory

sftp.ChangeDirectory("/");
sftp.ListDirectory("home").Select (s => s.FullName);

This is pretty crazy if you ask me. Setting the default directory with the ChangeDirectory method has no effect on the ListDirectory method unless you specify a folder in the parameter of this method. Seems like a bug should be written for this.

So when you write your recursive function you'll have to set the default directory once and then change the directory in the ListDirectory call as you iterate over the folders. The listing returns an enumerable of SftpFiles. These can then be checked individually for IsDirectory == true. Just be aware that the listing also returns the . and .. entries (which are directories). You'll want to skip these if you want to avoid an infinite loop. :-)

EDIT 2/23/2018

I was reviewing some of my old answers and would like to apologize for the answer above and supply the following working code. Note that this example does not require ChangeDirectory, since it's using the Fullname for the ListDirectory:

void Main()
{
    using (var client = new Renci.SshNet.SftpClient("sftp.host.com", "user", "password"))
    {
        var files = new List<String>();
        client.Connect();
        ListDirectory(client, ".", ref files);
        client.Disconnect();
        files.Dump();
    }
}

void ListDirectory(SftpClient client, String dirName, ref List<String> files)
{
    foreach (var entry in client.ListDirectory(dirName))
    {

        if (entry.IsDirectory)
        {
            ListDirectory(client, entry.FullName, ref files);
        }
        else
        {
            files.Add(entry.FullName);
        }
    }
}

Answer 4

@Carlos Bos

This library has some quirks that make this recursive listing tricky because the interaction between the ChangeDirectory and ListDirectory do not work as you may expect.

correct

It works well when the ChangeDirectory() parameter is "."

but if you do

SftpClient sftp ...;
sftp.ChangeDirectory("some_folder");
//get file list
List<SftpFile> fileList = sftp.ListDirectory("some_folder").ToList();

then there is an assertion because the ListDirectory() call expects "some_folder/some_folder"

The workaround I use is to save and restore the current directory before a remote upload/rename to "some_folder", and you need to list that folder before the operation (e.g to see the file already exists)

string working_directory = sftp.WorkingDirectory;
sftp.ChangeDirectory("some_folder");
sftp.RenameFile("name", "new_name");
sftp.ChangeDirectory(working_directory);

to check if the file exists, this call is sufficient

sftp.Exists(path)

or if you want to add some other criteria, like case sensitive or not

 public FileExistence checkFileExists(string folder, string fileName)
    {
      //get file list
      List<SftpFile> fileList = sftp.ListDirectory(folder).ToList();

      if (fileList == null)
      {
        return FileExistence.UNCONFIRMED;
      }

      foreach (SftpFile f in fileList)
      {
        Console.WriteLine(f.ToString());
        //a not case sensitive comparison is made
        if (f.IsRegularFile && f.Name.ToLower() == fileName.ToLower())
        {
          return FileExistence.EXISTS;
        }
      }

      //if not found in traversal , it does not exist
      return FileExistence.DOES_NOT_EXIST;
    }

where FileExistence is

public enum FileExistence
    {
      EXISTS,
      DOES_NOT_EXIST,
      UNCONFIRMED
    };

Answer 5

I have achieved this using recursion. Created a class TransportResponse like this

 public class TransportResponse
{
    public string directoryName { get; set; }
    public string fileName { get; set; }
    public DateTime fileTimeStamp { get; set; }
    public MemoryStream fileStream { get; set; }
    public List<TransportResponse> lstTransportResponse { get; set; }
}

I create a list of TransportResponse class. If the directoryName is not null, it will contain a list of the same class which will have the the files inside that directory as a MemoryStream ( this can be changed as per your use case)

List<TransportResponse> lstResponse = new List<TransportResponse>();
using (var client = new SftpClient(connectionInfo))
  {
          try
          {
                    Console.WriteLine("Connecting to " + connectionInfo.Host + " ...");
                    client.Connect();
                    Console.WriteLine("Connected to " + connectionInfo.Host + " ...");
           }
           catch (Exception ex)
           {
                    Console.WriteLine("Could not connect to "+ connectionInfo.Host +" server. Exception Details: " + ex.Message);
           }
           if (client.IsConnected)
           {
                    var files = client.ListDirectory(transport.SourceFolder);
                    lstResponse = downloadFilesInDirectory(files, client);
                    client.Disconnect();
            }
            else
            {
                    Console.WriteLine("Could not download files from "+ transport.TransportIdentifier +" because client was not connected.");
             }
   }



private static List<TransportResponse> downloadFilesInDirectory(IEnumerable<SftpFile> files, SftpClient client)
    {
        List<TransportResponse> lstResponse = new List<TransportResponse>();
        foreach (var file in files)
        {
            if (!file.IsDirectory)
            {
                if (file.Name != "." && file.Name != "..")
                {
                    if (!TransportDAL.checkFileExists(file.Name, file.LastWriteTime))
                    {
                        using (MemoryStream fs = new MemoryStream())
                        {
                            try
                            {
                                Console.WriteLine("Reading " + file.Name + "...");
                                client.DownloadFile(file.FullName, fs);
                                fs.Seek(0, SeekOrigin.Begin);
                                lstResponse.Add(new TransportResponse { fileName = file.Name, fileTimeStamp = file.LastWriteTime, fileStream = new MemoryStream(fs.GetBuffer()) });
                            }
                            catch(Exception ex)
                            {
                                Console.WriteLine("Error reading File. Exception Details: " + ex.Message);
                            }
                        }
                    }
                    else
                    {
                        Console.WriteLine("File was downloaded previously");
                    }
                }
            }
            else
            {
                if (file.Name != "." && file.Name != "..")
                {
                    lstResponse.Add(new TransportResponse { directoryName = file.Name,lstTransportResponse = downloadFilesInDirectory(client.ListDirectory(file.Name), client) });
                }                
            }
        }

        return lstResponse;
    }

Hope this helps. Thanks