解决Python下urllib3报错requests.packages.urllib3.connectionpool:Connection pool is full
Python问题复现
本地新建request.Session, 多线程(或线程池)打开url(或高并发不断访问同个站点时)并发较高时, 出现以下报错:
requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection
测试代码
def test_pool_max_size():
# 测试 pool_maxsize 对多线程访问的影响
def thread_get(url):
s.get(url)
s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1, pool_maxsize=1))
ts = []
for _ in range(2):
t = Thread(target=thread_get, args=('https://www.ask.com',))
ts.append(t)
t.start()
for t in ts:
t.join()
代码分析
HTTPAdapter用于对指定网址的连接管理, 参数包含pool_connections=1, pool_maxsize=1.
在HTTPAdapter中的init中会新建poolmanager
def __init__(self, pool_connections=DEFAULT_POOLSIZE,pool_maxsize=DEFAULT_POOLSIZE ...):
...
self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)
def init_poolmanager(self, connections, maxsize ...):
...
self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
block=block, strict=True, **pool_kwargs)
在PoolManager中num_pools也就是pool_connections用于构造RecentlyUsedContainer容器, 该容器用于保存最近使用的HTTPConnectionPool/HTTPSConnectionPool. HTTPConnectionPool/HTTPSConnectionPool管理指定(url, port)的所有连接.
def __init__(self, num_pools=10 ...):
...
self.pools = RecentlyUsedContainer(num_pools,
dispose_func=lambda p: p.close())
HTTPConnectionPool/HTTPSConnectionPool也有一个pool, 是LifoQueue容器, 由pool_maxsize参数构造, 保存的是HTTPConnection. HTTPConnection保存与服务器的socket连接.
pool_connections, pool_maxsize解析
pool_connections在poolmanager中限制缓存中不同url对应的HTTPConnectionPool/HTTPSConnectionPool数目.
报错解析:
pool_maxsize在HTTPConnectionPool/HTTPSConnectionPool中限制缓存中同一个url连接的数目. 单线程时运行时, 只会同时存在一个连接, 不会出现连接数过多的问题. 多线程时, 当同一网址的url请求数量大于pool_maxsize时, 发起url调用时不会报错, 在请求返回时, 会将连接放入HTTPConnectionPool/HTTPSConnectionPool中的LifoQueue, 当并发请求数量大于pool_maxsize时, LifoQueue不够放入所有的请求, 就会报错Connection pool is full, discarding connection.
参考:
urllib3连接池参数pool_connections, pool_maxsize相关源码解析 - 时时除草，时时耕耘 - CSDN博客
https://blog.csdn.net/bolun365/article/details/82955886
python - urllib3 connectionpool - 连接池已满，丢弃连接 - Stack Overflow
https://stackoverflow.com/questions/53765366/urllib3-connectionpool-connection-pool-is-full-discarding-connection
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////解决办法////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
参考:
multithreading - example urllib3 and threading in python - Stack Overflow
https://stackoverflow.com/questions/3731379/example-urllib3-and-threading-in-python
