How to improve the Performance of FtpWebRequest?

Question

I have an application written in .NET 3.5 that uses FTP to upload/download files from a server. The app works fine but there are performance issues:

1. It take

Answer 1

Look at this page - http://www.ietf.org/rfc/rfc959.txt

It says of using different port when connecting to be able to reuse the connection.
Does that work?

Answer 2

Personally I have migrated all of our apps away from using FTP for file upload/download, and instead rolled a solution based on XML Web Services in ASP.NET.

Performance is much improved, security is as much or as little as you want to code (and you can use the stuff built in to .NET) and it can all go over SSL with no issues.

Our success rate getting our clients' connections out through their own firewalls is FAR better than running FTP.

Answer 3

I have done some experimentation (uploading about 20 files on various sizes) on FtpWebRequest with the following factors

KeepAlive = true/false

ftpRequest.KeepAlive = isKeepAlive;

Connnection Group Name = UserDefined or null

ftpRequest.ConnectionGroupName = "MyGroupName";

Connection Limit = 2 (default) or 4 or 8

ftpRequest.ServicePoint.ConnectionLimit = ConnectionLimit;

Mode = Synchronous or Async

see this example

My results:

1. Use ConnectionGroupName,KeepAlive=true took (21188.62 msec)

2. Use ConnectionGroupName,KeepAlive=false took (53449.00 msec)

3. No ConnectionGroupName,KeepAlive=false took (40335.17 msec)

4. Use ConnectionGroupName,KeepAlive=true;async=true,connections=2 took (11576.84 msec)

5. Use ConnectionGroupName,KeepAlive=true;async=true,connections=4 took (10572.56 msec)

6. Use ConnectionGroupName,KeepAlive=true;async=true,connections=8 took (10598.76 msec)

Conclusions

1. FtpWebRequest has been designed to support an internal connection pool. To ensure, the connection pool is used, we must make sure the ConnectionGroupName is being set.

2. Setting up a connection is expensive. If we are connecting to the same ftp server using the same credentials, having the keep alive flag set to true will minimize the number of connections setup.

3. Asynchronous is the recommended way if you have a lot of files to ftp.

4. The default number of connections is 2. In my environment, a connection limit of 4 will give to me the most overall performance gain. Increasing the number of connections may or may not improve performance. I would recommend that you have the connection limit as a configuration parameter so that you can tune this parameter in your environment.

Hope you would find this useful.

Answer 4

I know it's an old thread, but I recently had to go through a similar situation.

We needed to download 70+ XML files from an ftp server in under 25 minutes without opening more than 5 connections during that time-frame.

These were all the alternatives we tried:

• wget - It was fast, but every GET meant a new connection. We had to stop due to the amount of connections created. We were having some issues with GETM that's well-documented in the web so that wasn't an option.
• FtpWebRequest - Every Create call would log a new connection, even though we used the KeepAlive and ConnectionGroupName properties. Plus it was very slow.
• Webclient - We didn't check if it created a new connection for every file (although I assume it does), but it copied 1 file/minute. So it wasn't an option.

We ended up using old-fashioned ftp batch script. It's fast and I only use one connection to download all the files. It isn't flexible, but it's much faster than everything else I've tried (75 files in under 20 minutes).

Answer 5

You should definitely check out BITS which is a big improvement over FTP. The clear-text passwords aren't the only weakness in FTP. There's also the issue of predicting the port it will open for a passive upload or download and just overall difficulty when clients are using NAT or firewalls.

BITS works over HTTP/HTTPS using IIS extensions and supports queued uploads and downloads that can be scheduled at low priority. It's overall just a lot more flexible than FTP if you are using Windows on the client and server.

BITS for PowerShell

BITS for .NET

Answer 6

Single Point of advice:

LOWER BUFFER/CHUNK-SIZES SIGNIFICANTLY REDUCE PERFORMANCE

Reason: Many more disk i/o, memory i/o, ftp stream init and many many more factors