Download multiple files concurrently from FTP using FluentFTP with a maximum value

问题

I would like to download multiple download files recursively from a FTP Directory, to do this I'm using FluentFTP library and my code is this one:

private async Task downloadRecursively(string src, string dest, FtpClient ftp)
{

    foreach(var item in ftp.GetListing(src))
    {
        if (item.Type == FtpFileSystemObjectType.Directory)
        {
            if (item.Size != 0)
            {
                System.IO.Directory.CreateDirectory(Path.Combine(dest, item.Name));
                downloadRecursively(Path.Combine(src, item.Name), Path.Combine(dest, item.Name), ftp);
            }
        }
        else if (item.Type == FtpFileSystemObjectType.File)
        {
            await ftp.DownloadFileAsync(Path.Combine(dest, item.Name), Path.Combine(src, item.Name));
        }
    }
}

I know you need one FtpClient per download you want, but how can I make to use a certain number of connections as maximum, I guess that the idea is to create, connect, download and close per every file I find but just having a X number of downloading files at the same time. Also I'm not sure if I should create Task with async, Threads and my biggest problem, how to implement all of this.

Answer from @Bradley here seems pretty good, but the question does read every file thas has to download from an external file and it doesn't have a maximum concurrent download value so I'm not sure how to apply these both requirements.

回答1:

Use:

ConcurrentBag class to implement a connection pool;
Parallel class to parallelize the operation;
ParallelOptions.MaxDegreeOfParallelism to limit number of the concurrent threads.

var clients = new ConcurrentBag<FtpClient>();

var opts = new ParallelOptions { MaxDegreeOfParallelism = maxConnections };
Parallel.ForEach(files, opts, file =>
{
    file = Path.GetFileName(file);

    string thread = $"Thread {Thread.CurrentThread.ManagedThreadId}";
    if (!clients.TryTake(out var client))
    {
        Console.WriteLine($"{thread} Opening connection...");
        client = new FtpClient(host, user, pass);
        client.Connect();
        Console.WriteLine($"{thread} Opened connection {client.GetHashCode()}.");
    }

    string remotePath = sourcePath + "/" + file;
    string localPath = Path.Combine(destPath, file);
    string desc =
        $"{thread}, Connection {client.GetHashCode()}, " +
        $"File {remotePath} => {localPath}";
    Console.WriteLine($"{desc} - Starting...");
    client.DownloadFile(localPath, remotePath);
    Console.WriteLine($"{desc} - Done.");

    clients.Add(client);
});

Console.WriteLine($"Closing {clients.Count} connections");
foreach (var client in clients)
{
    Console.WriteLine($"Closing connection {client.GetHashCode()}");
    client.Dispose();
}

Another approach is to start a fixed number of threads with one connection for each and have them pick files from a queue.

For an example of an implementation, see my article for WinSCP .NET assembly:
Automating transfers in parallel connections over SFTP/FTP protocol

回答2:

I'd split this into three parts.

Recursively build a list of source and destination pairs.
Create the directories required.
Concurrently download the files.

It's the last part that is slow and should be done in parallel.

Here's the code:

private async Task DownloadRecursively(string src, string dest, FtpClient ftp)
{
    /* 1 */
    IEnumerable<(string source, string destination)> Recurse(string s, string d)
    {
        foreach (var item in ftp.GetListing(s))
        {
            if (item.Type == FtpFileSystemObjectType.Directory)
            {
                if (item.Size != 0)
                {
                    foreach(var pair in Recurse(Path.Combine(s, item.Name), Path.Combine(d, item.Name)))
                    {
                        yield return pair;
                    }
                }
            }
            else if (item.Type == FtpFileSystemObjectType.File)
            {
                yield return (Path.Combine(s, item.Name), Path.Combine(d, item.Name));
            }
        }
    }

    var pairs = Recurse(src, dest).ToArray();
    
    /* 2 */
    foreach (var d in pairs.Select(x => x.destination).Distinct())
    {
        System.IO.Directory.CreateDirectory(d);
    }

    /* 3 */
    var downloads =
        pairs
            .AsParallel()
            .Select(x => ftp.DownloadFileAsync(x.source, x.destination))
            .ToArray();
    
    await Task.WhenAll(downloads);
}

It should be clean, neat, and easy to reason about code.

来源：https://stackoverflow.com/questions/66241158/download-multiple-files-concurrently-from-ftp-using-fluentftp-with-a-maximum-val

标签

multithreading

concurrency

ftp

fluentftp