File.Copy vs. Manual FileStream.Write For Copying File

Question

My problem is in regards file copying performance. We have a media management system that requires a lot of moving files around on the file system to different locations inc

Answer 1

Try to remove the Flush call, and move it to be outside the loop.

Sometimes the OS knows best when to flush the IO.. It allows it to better use its internal buffers.

Answer 2

Dusting off reflector we can see that File.Copy actually calls the Win32 API:

if (!Win32Native.CopyFile(fullPathInternal, dst, !overwrite))

Which resolves to

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
internal static extern bool CopyFile(string src, string dst, bool failIfExists);

And here is the documentation for CopyFile

Answer 3

File.Copy was build around CopyFile Win32 function and this function takes lot of attention from MS crew (remember this Vista-related threads about slow copy performance).

Several clues to improve performance of your method:

1. Like many said earlier remove Flush method from your cycle. You do not need it at all.
2. Increasing buffer may help, but only on file-to-file operations, for network shares, or ftp servers this will slow down instead. 60 * 1024 is ideal for network shares, at least before vista. for ftp 32k will be enough in most cases.
3. Help os by providing your caching strategy (in your case sequential reading and writing), use FileStream constructor override with FileOptions parameter (SequentalScan).
4. You can speed up copying by using asynchronous pattern (especially useful for network-to-file cases), but do not use threads for this, instead use overlapped io (BeginRead, EndRead, BeginWrite, EndWrite in .net), and do not forget set Asynchronous option in FileStream constructor (see FileOptions)

Example of asynchronous copy pattern:

int Readed = 0;
IAsyncResult ReadResult;
IAsyncResult WriteResult;

ReadResult = sourceStream.BeginRead(ActiveBuffer, 0, ActiveBuffer.Length, null, null);
do
{
    Readed = sourceStream.EndRead(ReadResult);

    WriteResult = destStream.BeginWrite(ActiveBuffer, 0, Readed, null, null);
    WriteBuffer = ActiveBuffer;

    if (Readed > 0)
    {
      ReadResult = sourceStream.BeginRead(BackBuffer, 0, BackBuffer.Length, null, null);
      BackBuffer = Interlocked.Exchange(ref ActiveBuffer, BackBuffer);
    }

    destStream.EndWrite(WriteResult);
  }
  while (Readed > 0);

Answer 4

You'll never going to able to beat the operating system at doing something so fundemental with your own code, not even if you crafted it carefully in assembler.

If you need make sure that your operations occur with the best performance AND you want to mix and match various sources then you will need to create a type that describes the resource locations. You then create an API that has functions such as Copy that takes two such types and having examined the descriptions of both chooses the best performing copy mechanism. E.g., having determined that both locations are windows file locations you it would choose File.Copy OR if the source is windows file but the destination is to be HTTP POST it uses a WebRequest.

Answer 5

One thing that stands out is that you are reading a chunk, writing that chunk, reading another chunk and so on.

Streaming operations are great candidates for multithreading. My guess is that File.Copy implements multithreading.

Try reading in one thread and writing in another thread. You will need to coordinate the threads so that the write thread doesn't start writing away a buffer until the read thread is done filling it up. You can solve this by having two buffers, one that is being read while the other is being written, and a flag that says which buffer is currently being used for which purpose.

Answer 6

Here's a similar answer

How do I copy the contents of one stream to another?

Your main problem is the call to Flush(), that will bind your performance to the speed of the I/O.