Is there an optimal byte size for sending data over a network?

Question

I assume 100 bytes is too small and can slow down larger file transfers with all of the writes, but something like 1MB seems like it may be too much. Does anyone have any sugge

Answer 1

The one thing I will add is that for a given ethernet connection it takes about as long to send a small packet as a large one. As other's have said: if you're just sending a stream of data let the system handle it. But if you're worried about individual short messages back and forth, a typical ethernet packet is about 1500 bytes- as long as you keep it under that you should be good.

Answer 2

Here is the formula you need:

int optimalChunkSize = totalDataSize / progressBar1.Width;

Using this, each chunk you send will increment the progress bar by 1 pixel. A smaller chunk size than this is pointless, in terms of user feedback.

Answer 3

Make a function named CalcChunkSize Add some private variables to your class:

Private PreferredTransferDuration As Integer = 1800 ' milliseconds, the timespan the class will attempt to achieve for each chunk, to give responsive feedback on the progress bar.
Private ChunkSizeSampleInterval As Integer = 15    ' interval to update the chunk size, used in conjunction with AutoSetChunkSize. 
Private ChunkSize As Integer = 16 * 1024           ' 16k by default  
Private StartTime As DateTime
Private MaxRequestLength As Long = 4096            ' default, this is updated so that the transfer class knows how much the server will accept      

Before every download of a chunk, check if its time to calculate new chunksize using the ChunkSizeSampleInterval

            Dim currentIntervalMod As Integer = numIterations Mod Me.ChunkSizeSampleInterval
            If currentIntervalMod = 0 Then
                Me.StartTime = DateTime.Now
            ElseIf currentIntervalMod = 1 Then
                Me.CalcChunkSize()
            End If

numIterations is set to 0 outside the download-loop and after every downloaded chunk set to numIterations += 1

Have the CalcChunkSize doing this:

Protected Sub CalcAndSetChunkSize()
    ' chunk size calculation is defined as follows 
    ' * in the examples below, the preferred transfer time is 1500ms, taking one sample. 
    ' * 
    ' * Example 1 Example 2 
    ' * Initial size = 16384 bytes (16k) 16384 
    ' * Transfer time for 1 chunk = 800ms 2000 ms 
    ' * Average throughput / ms = 16384b / 800ms = 20.48 b/ms 16384 / 2000 = 8.192 b/ms 
    ' * How many bytes in 1500ms? = 20.48 * 1500 = 30720 bytes 8.192 * 1500 = 12228 bytes 
    ' * New chunksize = 30720 bytes (speed up) 12228 bytes (slow down from original chunk size) 
    ' 

    Dim transferTime As Double = DateTime.Now.Subtract(Me.StartTime).TotalMilliseconds
    Dim averageBytesPerMilliSec As Double = Me.ChunkSize / transferTime
    Dim preferredChunkSize As Double = averageBytesPerMilliSec * Me.PreferredTransferDuration
    Me.ChunkSize = CInt(Math.Min(Me.MaxRequestLength, Math.Max(4 * 1024, preferredChunkSize)))
    ' set the chunk size so that it takes 1500ms per chunk (estimate), not less than 4Kb and not greater than 4mb // (note 4096Kb sometimes causes problems, probably due to the IIS max request size limit, choosing a slightly smaller max size of 4 million bytes seems to work nicely) 
End Sub

Then just use the ChunkSize when requesting next chunk.

I found this in the "Sending files in chunks with MTOM web services and .Net 2.0" by Tim_mackey and have found it very useful myself to dynamically calculate most effective chunksize.

The source code in whole are here: http://www.codeproject.com/KB/XML/MTOMWebServices.aspx

And author here: http://www.codeproject.com/script/Membership/Profiles.aspx?mid=321767

Answer 4

One empirical test you can do, if you haven't already, is of course to use a sniffer (tcpdump, Wireshark etc.) and look at what packet sizes are achieved when using other software for up/downloading. That might give you a hint.

Answer 5

I believe your problem is that you use blocking sockets and not non-blocking ones.

When you use blocking sockets and you send 1M of data the network stack can wait for all of the data to be placed in a buffer, if the buffers are full you'll be blocked and your progress bar will wait for the whole 1M to be accepted into the buffers, this may take a while and your progress bar will be jumpy.

If however you use non-blocking sockets, whatever buffer size you use will not block, and you will need to do the waiting yourself with select/poll/epoll/whatever-works-on-your-platform (select is the most portable though). In this way your progress bar will be updated fast and reflect the most accurate information.

Do note that at the sender the progress bar is partially broken any way since the kernel will buffer some data and you will reach 100% before the other side really received the data. The only way around this is if your protocol includes a reply on the amount of data received by the receiver.

As others said, second guessing the OS and the network is mostly futile, if you keep using blocking sockets pick a size that is large enough to include more data than a single packet so that you don't send too little data in a packet as this will reduce your throughput needlessly. I'd go with something like 4K to include at least two packets at a time.

Answer 6

You'd need to use Path MTU Discovery, or use a good default value (ie less than 1500 bytes).