Jsoup returns Status 400

问题

I want to crawl data from this URL : http://www.expedia.co.jp/Osaka-Hotels-Hotel-Consort.h5522663.Hotel-Information?chkin=2017/12/13&chkout=2017/12/14&rm1=a2

So I have written following code

import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import static org.jsoup.Connection.*;

/**
 * Created by avi on 11/24/17.
 */
public class ExpediaCurl {

    public static void main(String[] args) {
        final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";


        Response res = null;
        try {
            res = Jsoup.connect("http://www.expedia.co.jp/Osaka-Hotels-Hotel-Consort.h5522663.Hotel-Information?")
                    .data("adults", "2", "children", "0",  "chkin", "2017/12/13", "chkout", "2017/12/14")
                    .method(Method.POST)
                    .execute();

            Document doc = res.parse();
            Map<String,String> cookies = res.cookies();
            System.out.println(cookies.toString());
            Document doc2 = Jsoup.connect("https://www.expedia.co.jp/infosite-api/5522663/getOffers?clientid=KLOUD-HIWPROXY&token=bc9e0d07b9dcc14506a3ce5587703329a02e9f9e&brandId=0&countryId=0&isVip=false&chid=&partnerName=&partnerPrice=0&partnerCurrency=&partnerTimestamp=0&adults=2&children=0&chkin=2017%2F12%2F13&chkout=2017%2F12%2F14&swpToggleOn=false&daysInFuture=&stayLength=&ts=1513141494203&evalMODExp=true&tla=OSA")
                    .cookies(cookies) // <-- I believe This cookie is not working
                    .userAgent(USER_AGENT)
                    .get();
            System.out.printf(doc2.html());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

I am getting following error

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=400, URL=https://www.expedia.co.jp/infosite-api/5522663/getOffers?clientid=KLOUD-HIWPROXY&token=bc9e0d07b9dcc14506a3ce5587703329a02e9f9e&brandId=0&countryId=0&isVip=false&chid=&partnerName=&partnerPrice=0&partnerCurrency=&partnerTimestamp=0&adults=2&children=0&chkin=2017%252F12%252F13&chkout=2017%252F12%252F14&swpToggleOn=false&daysInFuture=&stayLength=&ts=1513141494203&evalMODExp=true&tla=OSA
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:679)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:628)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:260)
    at org.jsoup.helper.HttpConnection.get(HttpConnection.java:249)
    at ExpediaCurl.main(ExpediaCurl.java:31)

Can someone Help me out that what is wrong I am doing.

回答1:

Try to use

String url = "http://www.expedia.co.jp/" + URLEncoder.encode(Osaka-Hotels-Hotel-Consort.h5522663.Hotel-Information?", "UTF-8")

来源：https://stackoverflow.com/questions/47785985/jsoup-returns-status-400

标签

java

web-crawler

jsoup