Java - Quickest way to check if URL exists

Question

Hi I am writing a program that goes through many different URLs and just checks if they exist or not. I am basically checking if the error code returned is 404 or not. Howev

Answer 1

Seems you can set the timeout property, make sure it is acceptable. And if you have many urls to test, do them parallelly, it will be much faster. Hope this will be helpful.

Answer 2

Try to ask the next DNS Server

class DNSLookup
{
    public static void main(String args[])
    {
        String host = "stackoverflow.com";
        try
        {
            InetAddress inetAddress = InetAddress.getByName(host);
            // show the Internet Address as name/address
            System.out.println(inetAddress.getHostName() + " " + inetAddress.getHostAddress());
        }
        catch (UnknownHostException exception)
        {
            System.err.println("ERROR: Cannot access '" + host + "'");
        }
        catch (NamingException exception)
        {
            System.err.println("ERROR: No DNS record for '" + host + "'");
            exception.printStackTrace();
        }
    }
}

Answer 3

Try sending a "HEAD" request instead of get request. That should be faster since the response body is not downloaded.

huc.setRequestMethod("HEAD");

Again instead of checking if response status is not 400, check if it is 200. That is check for positive instead of negative. 404,403,402.. all 40x statuses are nearly equivalent to invalid non-existant url.

You may make use of multi-threading to make it even faster.