问题
Is there a way to tell Python urllib3 to not reuse idle connections after some period of time, and instead to close them?
Looking in https://urllib3.readthedocs.io/en/latest/reference/index.html#module-urllib3.connectionpool doesn't seem to show anything relevant.
回答1:
Remember:
A connection pool is a cache of database connections maintained so that the connections can be "reused" when future requests to the database are required.
You can do this is many ways (I guess):
- Set retries to one.
This breaks your connection if it failed one time. To set it:
import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=1) # is zero for default
s.mount('http://', a)
- Change pool connections.
The "pool_connections" is the number of host-pools to keep around. For example, if you're connecting to 100 different hosts, and pool_connections=10, then only the latest 10 hosts' connections will be re-used. To set that:
s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1))
s.get('https://www.example.com')
This will stop the reuse of pools.
- Pool Maxsize
This is cared only if you use Session in a multithreaded environment. To set it:
s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1, pool_maxsize=1))
- Configure Maxsize
he :class:~connectionpool.ConnectionPool class keeps a pool of individual :class:~connection.HTTPConnection instances. These connections are used during an individual request and returned to the pool when the request is complete. By default only one connection will be saved for re-use. To set it (it is, by default):
from urllib3 import HTTPConnectionPool
pool = HTTPConnectionPool('www.example.com', maxsize=0) #likely to slow you down cuz it never stores the pools
maxsize – Number of connections to save that can be reused. More than 1 is useful in multithreaded situations.
- Let the Pool Manager do that!
The PoolManager uses a Least Recently Used (LRU) policy for discarding old pools. That is, if you set the PoolManager num_pools to 10, then after making requests to 11 or more different hosts, the least recently used pools will be cleaned up eventually. So to do that:
from urllib3 import PoolManager
manager = PoolManager(1) # not the manager cleans up pools used for one time
r = manager.request('GET', 'http://www.example.com/')
Also, the docs says that:
Cleanup of stale pools does not happen immediately.
So for that Use RecentlyUsedContainer (Docs contains only one line).
Note:
Setting arguments if PoolManager affects all the pools connected thereby.
Hopes this helps you. Get the advanced usage docs HERE .
回答2:
Firstly, you could define the num_pools variable so that if it crosses the value of the pools already created, then it would discard the least recently used. You could also set timeout on the connection so that the connections could automatically close after the timeout. I was also not able to find anything to determine the idle connections, but I guess this might help
https://laike9m.com/blog/requests-secret-pool_connections-and-pool_maxsize,89/
来源：https://stackoverflow.com/questions/58345555/python-urllib3-close-idle-connection-after-some-time