In requests library, how can I avoid “HttpConnectionPool is full, discarding connection” warning?

后端 未结 2 447
长发绾君心
长发绾君心 2021-02-07 06:47

I\'m using python requests library with sessions:

def _get_session(self):
    if not self.session:
        self.session = requests.Session()
    return self.sess         


        
相关标签:
2条回答
  • 2021-02-07 07:13

    I'd like to clarify some stuff here.

    What max_poolsize argument does is limit the number of TCP connections that can be stored in the connection pool simultaneously. Normally, when you want to execute a HTTP requests, requests will try to take a TCP connection from its connection pool. If there are no available connections, requests will create a new TCP connection, and when it is done making a HTTP request, it will try to put it back in the pool (it will not remember whether the connection was taken from the connection pool or not).

    The Full exception being raised in requests code is just an example of a common Python pattern usually paraphrased as it is easier to ask for forgiveness than for permission. It has nothing with dropping TCP connections.

    0 讨论(0)
  • 2021-02-07 07:17

    From Requests docs in http://docs.python-requests.org/en/latest/api/

     class requests.adapters.HTTPAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False)
    

    The built-in HTTP Adapter for urllib3.

    Provides a general-case interface for Requests sessions to contact HTTP and HTTPS urls by implementing the Transport Adapter interface. This class will usually be created by the Session class under the covers.

    Parameters:

    • pool_connections – The number of urllib3 connection pools to cache.
    • pool_maxsize – The maximum number of connections to save in the pool.
    • max_retries (int) – The maximum number of retries each connection should attempt. Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
    • pool_block – Whether the connection pool should block for connections.

    and a little below, comes an example

    import requests
    s = requests.Session()
    a = requests.adapters.HTTPAdapter(max_retries=3)
    s.mount('http://', a)
    

    Try this

    a = requests.adapters.HTTPAdapter(pool_connections = N, pool_maxsize = M)
    

    Where N and M are suitable for your program.

    0 讨论(0)
提交回复
热议问题