Reading from a URL Connection Java

Question

I\'m trying to read html code from a URL Connection. In one case the html file I\'m trying to read includes 5 line breaks before the actual doc type declaration. In this cas

Answer 1

You're using DataInputStream to read data that wasn't encoded using DataOutputStream. Examine the documented behavior for your call to DataInputStream#readUtf(); it first reads two bytes to form a 16-bit integer, indicating the number of bytes that follow comprising the UTF-encoded string. The data you're reading from the HTTP server is not encoded in this format.

Instead, the HTTP server is sending headers encoded in ASCII, per RFC 2616 sections 6.1 and 2.2. You need to read the headers as text, and then determine how the message body (the "entity") is encoded.

Answer 2

This works fine:

package url;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;

/**
 * UrlReader
 * @author Michael
 * @since 3/20/11
 */
public class UrlReader
{

    public static void main(String[] args)
    {
        UrlReader urlReader = new UrlReader();

        for (String url : args)
        {
            try
            {
                String contents = urlReader.readContents(url);
                System.out.printf("url: %s contents: %s\n", url, contents);
            }
            catch (Exception e)
            {
                e.printStackTrace();
            }
        }
    }


    public String readContents(String address) throws IOException
    {
        StringBuilder contents = new StringBuilder(2048);
        BufferedReader br = null;

        try
        {
            URL url = new URL(address);
            br = new BufferedReader(new InputStreamReader(url.openStream()));
            String line = "";
            while (line != null)
            {
                line = br.readLine();
                contents.append(line);
            }
        }
        finally
        {
            close(br);
        }

        return contents.toString();
    }

    private static void close(Reader br)
    {
        try
        {
            if (br != null)
            {
                br.close();
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }
}

Answer 3

This:

public class Main {
    public static void main(String[] args) 
        throws MalformedURLException, IOException 
    {
        URL pageUrl = new URL("http://www.google.com");
        URLConnection getConn = pageUrl.openConnection();
        getConn.connect();
        BufferedReader dis = new BufferedReader( 
                                 new InputStreamReader(
                                     getConn.getInputStream()));
        String myString;
        while ((myString = dis.readLine()) != null)
        {
            System.out.println(myString);
        }
    }
}

Works perfectly. The URL you are supplying, however, returns nothing.