Reading data from an open HTTP stream

Question

I am trying to use the .NET WebRequest/WebResponse classes to access the Twitter streaming API here "http://stream.twitter.com/spritzer.json".

I need to be able to open the connection and read data incrementally from the open connection.

Currently, when I call WebRequest.GetResponse method, it blocks until the entire response is downloaded. I know there is a BeginGetResponse method, but this will just do the same thing on a background thread. I need to get access to the response stream while the download is still happening. This just does not seem possible to me with these classes.

There is a specific comment about this in the Twitter documentation:

"Please note that some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example."

They point to the Appache HttpClient, but that doesn't help much because I need to use .NET.

Any ideas whether this is possible with WebRequest/WebResponse, or do I have to go for lower level networking classes? Maybe there are other libraries that will allow me to do this?

Thx Allen

Answer 1

I ended up using a TcpClient, which works fine. Would still be interested to know if this is possible with WebRequest/WebResponse though. Here is my code in case anybody is interested:

using (TcpClient client = new TcpClient())
{

string requestString = "GET /spritzer.json HTTP/1.1\r\n";
requestString += "Authorization: " + token + "\r\n";
requestString += "Host: stream.twitter.com\r\n";
requestString += "Connection: keep-alive\r\n";
requestString += "\r\n";

client.Connect("stream.twitter.com", 80);

using (NetworkStream stream = client.GetStream())
{
    // Send the request.
    StreamWriter writer = new StreamWriter(stream);
    writer.Write(requestString);
    writer.Flush();

    // Process the response.
    StreamReader rdr = new StreamReader(stream);

    while (!rdr.EndOfStream)
    {
        Console.WriteLine(rdr.ReadLine());
    }
}
}

Answer 2

BeginGetResponse is the method you need. It allows you to read the response stream incrementally:

class Program
{
    static void Main(string[] args)
    {
        WebRequest request = WebRequest.Create("http://stream.twitter.com/spritzer.json");
        request.Credentials = new NetworkCredential("username", "password");
        request.BeginGetResponse(ar => 
        {
            var req = (WebRequest)ar.AsyncState;
            // TODO: Add exception handling: EndGetResponse could throw
            using (var response = req.EndGetResponse(ar))
            using (var reader = new StreamReader(response.GetResponseStream()))
            {
                // This loop goes as long as twitter is streaming
                while (!reader.EndOfStream)
                {
                    Console.WriteLine(reader.ReadLine());
                }
            }
        }, request);

        // Press Enter to stop program
        Console.ReadLine();
    }
}

Or if you feel more comfortable with WebClient (I personnally prefer it over WebRequest):

using (var client = new WebClient())
{
    client.Credentials = new NetworkCredential("username", "password");
    client.OpenReadCompleted += (sender, e) =>
    {
        using (var reader = new StreamReader(e.Result))
        {
            while (!reader.EndOfStream)
            {
                Console.WriteLine(reader.ReadLine());
            }
        }
    };
    client.OpenReadAsync(new Uri("http://stream.twitter.com/spritzer.json"));
}
Console.ReadLine();

Answer 3

Have you tried WebRequest.BeginGetRequestStream() ?

Or something like this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create (http://www.twitter.com );
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream()); 

string str = reader.ReadLine();
while(str != null)
{
   Console.WriteLine(str);
   str = reader.ReadLine();
}

Answer 4

Just use WebClient. It is designed for simple cases like this where you don't need the full power of WebRequest.

System.Net.WebClient wc = new System.Net.WebClient();
Console.WriteLine(wc.DownloadString("http://stream.twitter.com/spritzer.json"));