HTTP connection pooling using HttpClient

Question

• How can I create a pool of connections using HttpClient?
• I have to make frequent connections to the same server. Is it worth creating such a pool?
• Is it possible to keep live connections and use it for various requests, and if yes how can I do so?

I am developing in Java, using Apache HTTP Client.

Answer 1

[assuming Java, and Apache's HttpClient]

Use a ThreadSafeClientConnManager. Pass a single global instance to the constructor of every HttpClient instance. I don't think there's any point in pooling the HttpClients themselves.

Answer 2

PoolingClientConnectionManager is Deprecated now . from (4.3 version) use PoolingHttpClientConnectionManager.

Answer 3

I have spent recent days working on this so just want to share some "everyone-known" knowledges with you.

First, as you are dealing with the same server, it is recommended to use a single HTTP client to execute your requests. With the help of PoolingHttpClientConnectionManager, your client can be used to execute multiple requests concurrently. The official example of multithreaded request execution can be found here.

Secondly, HTTP/1.1 (and enhanced versions of HTTP/1.0) allows HTTP clients to keep the connections open after transactions complete so that it can be reused for future requests. This is often refered as Persistent Connection.

Also for the purpose of reusing client for multiple requests, the response header from a server often include an attribute call Keep-Alive that contain the time current connection will be kept alive. Besides that, Apache Http Client also provides you an interface ConnectionKeepAliveStrategyto customize your own policy for reusing connection.

Answer 4

ThreadSafeClientConnManager is deprecated now, use PoolingClientConnectionManager instead.

Answer 5

For HttpClient 4x:

ThreadSafeClientConnManager ... manages a pool of client connections and is able to service connection requests from multiple execution threads.

Connections are pooled on a per route basis. A request for a route for which the manager already has a persistent connection available in the pool will be serviced by leasing a connection from the pool rather than creating a brand new connection.

http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html

Answer 6

This is an example of an Apache HttpClient 4.3 pool of connections which do not require authentication:

public class PoolOfHttpConnections{
   static String[] urisToGet = {"http://www.site1.com", "http://www.site2.com"};

    public static void main(String[] args) throws Exception {
           CloseableHttpClient httpclient = HttpClients.createDefault();
           // create a thread for each link
           GetThread[] threads = new GetThread[urisToGet.length];
           for (int i = 0; i < threads.length; i++) {
               HttpGet httpget = new HttpGet(urisToGet[i]);
               threads[i] = new GetThread(httpClient, httpget);
           }

           // start the threads
           for (int j = 0; j < threads.length; j++) {
               threads[j].start();
           }
           // join the threads
           for (int j = 0; j < threads.length; j++) {
               threads[j].join();
           }
    } //end main

    private static class GetThread extends Thread {

            private final CloseableHttpClient httpClient;
            private final HttpContext context;
            private final HttpGet httpget;

            public GetThread(CloseableHttpClient httpClient, HttpGet httpget) {
                   this.httpClient = httpClient;
                   this.context = HttpClientContext.create();
                   this.httpget = httpget;
            }

            @Override
            public void run() {
                   try {
                       CloseableHttpResponse response = httpClient.execute(httpget, context);
                       try {
                           HttpEntity entity = response.getEntity();
                           System.out.println("----------------------------------------");
                           Date date = new Date();
                           System.out.println("Beginning*******************");
                           System.out.println(date.toString());
                           System.out.println("There are "+urisToGet.length+" threads running in parallel!");
                           System.out.println(response.getStatusLine());
                           if (entity != null) {
                              System.out.println("Response content length: " + entity.getContentLength());
                           }
                           System.out.println(EntityUtils.toString(entity));
                           EntityUtils.consume(entity);
                       } finally {
                         response.close();
                         System.out.println("End*******************");
                       }
                   } catch (ClientProtocolException ex) {
                          // Handle protocol errors
                   } catch (IOException ex) {
                          // Handle I/O errors
                   }
            }
    } /*end private class*/  }//end public class PoolOfHttpConnections

Answer 7

HttpClient has already have a connection pool.So you do not need to create it. Just use it.