Why Http request with Fiddler is blazing fast

Question

I wrote a multithread program in c# that crawls a web site, but when I started Fiddler in the background request completed 12x faster, it is really strange for me and when I

Answer 1

The reason is the limit on the number of simultaneous http connections which is ignored when using Fiddler.

I've run into the same behavior while using System.Net.Http.HttpClient to perform multiple (~80) concurrent requests. With Fiddler running, all the requests were completing much faster. Keep-alive was certainly enabled.

I used Wireshark to see what's happening and first thing I noticed that the manner of the http traffic was different. With Fiddler requests were thrown all at once and responses followed nicely grouped as well. Without Fiddler the requests were interleaving with responses.

Secondly, tcpview showed that my code without Fiddler opened only 2 tcp connections to the server. With Fiddler started, the number of connections significanly increased. There were tens of them from my app to Fiddler and then from Fiddler to the server.

It is well known that the http standard recommends the number of http connections should be no more than 2 and it seems the limit is implemented as a default setting in http clients.

In .NET applications we can control the limit with ServicePointManager.DefaultConnectionLimit static property. As an experiment, having it set to 100 made the requests completing with the same time with and without Fiddler.

The setting can also be controlled through app.config:

    <system.net>
    <connectionManagement>
        <add address="*" maxconnection="100" />
    </connectionManagement>
</system.net>

Now, why is the default connection limit not respected while using Fiddler? Looks like the limit is different when an http client uses a proxy and Fiddler does act as a proxy. I did not find much information about the proxy connection limit besides this old article.

Answer 2

Can you show some sample code so people can confirm this? Otherwise it'll become wild guessing.

My best guess: Fiddler uses keepalive which will save the trouble of opening the connection over and over again. You can confirm this by disabling both Reuse client connections and Reuse connections to servers: if it's then as slow as usual (or slower), the benefit is gained from keeping the connection alive.