“Exception in thread ”main“ java.lang.NullPointerException” error when running web scraper program

问题

I'm fairly new to web scraping and have limited knowledge on Java.

Every time I run this code, I get the error:

Exception in thread "main" java.lang.NullPointerException

    at sws.SWS.scrapeTopic(SWS.java:38)
    at sws.SWS.main(SWS.java:26)
Java Result: 1
BUILD SUCCESSFUL (total time: 0 seconds)

My code is:

import java.io.*;

import java.net.*;

import org.jsoup.Jsoup;

import org.jsoup.nodes.Document;

public class SWS
{

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args)
    {
        scrapeTopic("wiki/Python");

    }

    public static void scrapeTopic(String url)
    {
        String html = getUrl("http://www.wikipedia.org/" + url);



        Document doc = Jsoup.parse(html);

        String contentText = doc.select("#mw-content-text > p").first().text();

        System.out.println(contentText);


    }


    public static String getUrl(String Url)
    {
        URL urlObj = null;

        try 
        {
            urlObj = new URL(Url);

        }

        catch(MalformedURLException e)
        {
            System.out.println("The url was malformed");

            return "";
        }


        URLConnection urlCon = null;

        BufferedReader in = null;

        String outputText = "";

        try
        {
            urlCon = urlObj.openConnection();

            in = new BufferedReader(new InputStreamReader(urlCon.getInputStream()));
             String line = "";

             while ((line = in.readLine()) != null)
             {
                 outputText += line;

             }

             in.close();
        }

         catch(IOException e)
         {
             System.out.println("There was a problem connecting to the url");

             return "";

         }

        return outputText;



    }

}

I've been staring at my screen for sometime now and in need of help!

Thanks in advance.

回答1:

In the following code:

 String contentText = doc.select("#mw-content-text > p").first().text()

If doc.select("#mw-content-text > p") doesn't find any element that match the query and returns an empty element calling first() on such element should give a NullPointerException.

check the jsoup document page of Element.select and Elements.first()

回答2:

In this line

doc.select("#mw-content-text > p").first().text();

Your doc.select obviously dont find anything, so it returns null. You then call first() method on null and thats why it ends with error.

回答3:

Your code works fine for me exactly as it is.

For the purpose of debugging and diagnosing whether the possible errors noted by other answers, you'd likely do well to use some temporary variables and step through the code in a debugger.

public static void scrapeTopic(String url)
{
    String html = getUrl("http://www.wikipedia.org/" + url);
    Document doc = Jsoup.parse(html);

    Elements select = doc.select("#mw-content-text > p");
    Element first = select.first();
    String contentText = first.text();

    System.out.println(contentText);
}

来源：https://stackoverflow.com/questions/19337516/exception-in-thread-main-java-lang-nullpointerexception-error-when-running-w

标签

java

web

screen-scraping

jsoup