Set user-agent property in https connection header

Question

I am unable to properly set the user-agent property for an https connection. From what I\'ve gathered, http-header properties can be set through either the

Answer 1

I've found/verified the problem by inspecting http communictions using WireShark. Is there any way around this?

There is no problem here. The User-Agent header is set whether the request is transported via HTTP / HTTPS. Even setting it to something unreasonable like blah blah works on HTTPS. The headers shown below were captured when the underlying protocol used was HTTPS.

Request headers sent via HTTPS

User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

---

User-Agent: blah blah
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

---

Here's the code that triggers the request.

        // localhost:52999 is a reverse proxy to xxx:443
        java.net.URL url = new java.net.URL( "https://localhost:52999/" );
        java.net.URLConnection conn = url.openConnection();
        conn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0");
        conn.connect();
        java.io.BufferedReader serverResponse = new java.io.BufferedReader(new java.io.InputStreamReader(conn.getInputStream()));
        System.out.println(serverResponse.readLine());
        serverResponse.close();

Normally, HTTPS requests cannot be sniffed (like @Perception mentioned). Piping the request through a proxy that replaces the root CA with its own fake CA will allow you to see the traffic. A simpler method is to just look at the access log of the target server. But as you can see from the HTTPS request snippet above, the User-Agent header that is sent is correct.

Answer 2

I've found/verified the problem by inspecting http communictions using WireShark. Is there any way around this

This is not possible. Communication over an SSL socket is completely obscured from casual observation by the encryption protocol. Using packet capture software you will be able to view the initiation of the SSL connection and the exchange of encrypted packets, but the content of those packets can only be extracted at the other end of the connection (the server). If this were not the case then the HTTPS protocol as a whole would be broken, as the whole point of it is to secure HTTP communications from man-in-the-middle type attacks (where in this case the MITM is the packet sniffer).

Example Capture of an HTTPS request (partial):

.n....E... .........../..5..3..9..2..8.. ..............@........................Ql.{...b....OsR..!.4.$.T...-.-.T....Q...M..Ql.{...LM..L...um.M...........s. ...n...p^0}..I..G4.HK.n......8Y...............E...A..>...0...0......... ).s.......0 ..*.H.. .....0F1.0...U....US1.0...U. . Google Inc1"0 ..U....Google Internet Authority0.. 130327132822Z. 131231155850Z0h1.0...U....US1.0...U... California1.0...U... Mountain View1.0...U. . Google Inc1.0...U....www.google.com0..0

Theoretically, the only way to know if your User-Agent header is actually being excluded is if you have access to the Google servers, but in actuality there is nothing in either the HTTPS specification or Java's implementation of it that excludes headers that would normally have been sent over HTTP.

Example Capture of HTTP request:

GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0
Host: www.google.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, /; q=.2
Connection: keep-alive

Both example captures were generated with the exact same code:

URL url = new URL(target);
URLConnection conn = url.openConnection();
conn.setRequestProperty("User-Agent",
        "Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0");
conn.connect();
BufferedReader serverResponse = new BufferedReader(
        new InputStreamReader(conn.getInputStream()));
System.out.println(serverResponse.readLine());
serverResponse.close();

Except that for HTTPS the target was "https://www.google.com", and for HTTP it was "http://www.google.com".

---

Edit 1:

Based off your updated question, using the -Dhttp.agent property does indeed append 'Java/version' to the user agent header, as described by the following documentation:

http.agent (default: “Java/<version>”)
Defines the string sent in the User-Agent request header in http requests. Note that the string “Java/<version>” will be appended to the one provided in the property (e.g. if -Dhttp.agent=”foobar” is used, the User-Agent header will contain “foobar Java/1.5.0” if the version of the VM is 1.5.0). This property is checked only once at startup.

The 'offending' code is in a static block initializer of sun.net.www.protocol.http.HttpURLConnection:

static {
    // ...
    String agent = java.security.AccessController
            .doPrivileged(new sun.security.action.GetPropertyAction(
                    "http.agent"));
    if (agent == null) {
        agent = "Java/" + version;
    } else {
        agent = agent + " Java/" + version;
    }
    userAgent = agent;

    // ...
}

An obscene way around this 'problem' is this snippet of code, which I 1000% recommend you not use:

protected void forceAgentHeader(final String header) throws Exception {
    final Class<?> clazz = Class
            .forName("sun.net.www.protocol.http.HttpURLConnection");

    final Field field = clazz.getField("userAgent");
    field.setAccessible(true);
    Field modifiersField = Field.class.getDeclaredField("modifiers");
    modifiersField.setAccessible(true);
    modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);
    field.set(null, header);
}

Using this override with https.proxyHost, https.proxyPort and http.agent set gives the desired result:

CONNECT www.google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0
Host: www.google.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, /; q=.2
Proxy-Connection: keep-alive

But yea, don't do that. Its much safer to just use Apache HttpComponents:

final DefaultHttpClient client = new DefaultHttpClient();
HttpHost proxy = new HttpHost("127.0.0.1", 8888, "http");
HttpHost target = new HttpHost("www.google.com", 443, "https");
client.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpProtocolParams
        .setUserAgent(client.getParams(),
                "Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0");
final HttpGet get = new HttpGet("/");

HttpResponse response = client.execute(target, get);