403 error while getting the google result using jsoup [duplicate]

Answer 1

Google doesn't allow robots, you couldn't use jsoup to connect google. You can use the Google Web Search API (Deprecated) but the number of requests you may make per day will be limited.

Answer 2

try this:

Document doc =con.connect("http://www.google.com/search?q=lakshman").ignoreHttpErrors(true).timeout(5000).get();

in case userAgent did not work Just like it didn't for me.

Answer 3

Replace statement

Document doc =con.connect("http://www.google.com/search?q=lakshman").timeout(5000).get();

with statement

Document doc=Jsoup.connect("http://www.google.com/search?q=lakshman").userAgent("Chrome").get();

Answer 4

Actually, you can evade 403 error by just adding a user-agent

doc = Jsoup.connect(url).timeout(timeout)
                    .userAgent("Mozilla")

But that is against the google policy I think.

EDIT: Google catches robots quicker than you think. You can however, use this as a temporary solution.

Answer 5

You just need to add the UserAgent property to HTTP header as follows:

Jsoup.connect(itemUrl)
     .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36")
     .get()

Answer 6

In some cases you need to set a referrer. It helped in my case.

The full source here

    try{

        String strText = 
                Jsoup
                .connect("http://www.whatismyreferer.com")
                .referrer("http://www.google.com")
                .get()
                .text();

        System.out.println(strText);

    }catch(IOException ioe){
        System.out.println("Exception: " + ioe);
    }