How can I perform a GET request without downloading the content?

Question

I am working on a link checker, in general I can perform HEAD requests, however some sites seem to disable this verb, so on failure I need to also perform a G

Answer 1

If you are using a GET request, you will receive the message-body whether you want to or not. The data will still be transmitted to your endpoint regardless of whether or not you read it from the socket or not. The data will just stay queued in the RecvQ waiting to be selected out.

For this, you really should be using a "HEAD" request if possible, which will spare you the message body.

Answer 2

Couldn't you use a WebClient to open a stream and read just the few bytes you require?

using (var client = new WebClient())
        {
            using (var stream = client.OpenRead(uri))
            {
                const int chunkSize = 100;
                var buffer = new byte[chunkSize];
                int bytesRead;
                while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
                {
                    //check response here
                }
            }
        }

I am not sure how WebClient opens the stream internally. But it seems to allow partial reading of data.

Answer 3

When you do a GET, the server will start sending data from the start of the file to the end. Unless you interrupt it. Granted, at 10 Mb/sec, that's going to be a megabyte per second so if the file is small you'll get the whole thing. You can minimize the amount you actually download in a couple of ways.

First, you can call request.Abort after getting the response and before calling response.close. That will ensure that the underlying code doesn't try to download the whole thing before closing the response. Whether this helps on small files, I don't know. I do know that it will prevent your application from hanging when it's trying to download a multi-gigabyte file.

The other thing you can do is request a range, rather than the entire file. See the AddRange method and its overloads. You could, for example, write request.AddRange(512), which would download only the first 512 bytes of the file. This depends, of course, on the server supporting range queries. Most do. But then, most support HEAD requests, too.

You'll probably end up having to write a method that tries things in sequence:

• try to do a HEAD request. If that works (i.e. doesn't return a 500), then you're done
• try GET with a range query. If that doesn't return a 500, then you're done.
• do a regular GET, with a request.Abort after GetResponse returns.