what's the DBUtils
DBUtils 是一套用于管理数据库连接池的Python包,为高频度高并发的数据库访问提供更好的性能,可以自动管理连接对象的创建和释放。并允许对非线程安全的数据库接口进行线程安全包装。
DBUtils提供两种外部接口:
- PersistentDB :提供线程专用的数据库连接,并自动管理连接。
- PooledDB :提供线程间可共享的数据库连接,并自动管理连接。
实测证明 PersistentDB 的速度是最高的,但是在某些特殊情况下,数据库的连接过程可能异常缓慢,而此时的PooledDB则可以提供相对来说平均连接时间比较短的管理方式。
另外,实际使用的数据库驱动也有所依赖,比如SQLite数据库只能使用PersistentDB作连接池。 下载地址:http://www.webwareforpython.org/downloads/DBUtils/
DBUtils使用方法
连接池对象只初始化一次,一般可以作为模块级代码来确保。 PersistentDB的连接例子:
import DBUtils.PersistentDB persist=DBUtils.PersistentDB.PersistentDB(dbpai=MySQLdb,maxusage=1000,**kwargs)
这里的参数dbpai指使用的底层数据库模块,兼容DB-API的。maxusage则为一个连接最大使用次数,参考了官方例子。后面的**kwargs则为实际传递给MySQLdb的参数。
获取连接: conn=persist.connection() 实际编程中用过的连接直接关闭 conn.close() 即可将连接交还给连接池。
PooledDB使用方法同PersistentDB,只是参数有所不同。
- dbapi :数据库接口
- mincached :启动时开启的空连接数量
- maxcached :连接池最大可用连接数量
- maxshared :连接池最大可共享连接数量
- maxconnections :最大允许连接数量
- blocking :达到最大数量时是否阻塞
- maxusage :单个连接最大复用次数
- setsession :用于传递到数据库的准备会话,如 [”set name UTF-8″] 。
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
db=pooled.connection() cur=db.cursor() cur.execute(sql) res=cur.fetchone() cur.close() # or del cur db.close() # or del db
![](/qrcode.jpg)
python不用连接池的MySQL连接方法
import MySQLdb conn= MySQLdb.connect(host='localhost',user='root',passwd='pwd',db='myDB',port=3306) #import pymysql #conn = pymysql.connect(host='localhost', port='3306', db='game', user='root', password='123456', charset='utf8') cur=conn.cursor() SQL="select * from table1" r=cur.execute(SQL) r=cur.fetchall() cur.close() conn.close()
用连接池后的连接方法
import MySQLdb from DBUtils.PooledDB import PooledDB pool = PooledDB(MySQLdb,5,host='localhost',user='root',passwd='pwd',db='myDB',port=3306) #5为连接池里的最少连接数 conn = pool.connection() #以后每次需要数据库连接就是用connection()函数获取连接就好了 cur=conn.cursor() SQL="select * from table1" r=cur.execute(SQL) r=cur.fetchall() cur.close() conn.close()
DBUtils下载地址:https://pypi.python.org/pypi/DBUtils/
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import sys import threading import MySQLdb import DBUtils.PooledDB connargs = { "host":"localhost", "user":"user1", "passwd":"123456", "db":"test" } def test(conn): try: cursor = conn.cursor() count = cursor.execute("select * from users") rows = cursor.fetchall() for r in rows: pass finally: conn.close() def testloop(): print ("testloop") for i in range(1000): conn = MySQLdb.connect(**connargs) test(conn) def testpool(): print ("testpool") pooled = DBUtils.PooledDB.PooledDB(MySQLdb, **connargs) for i in range(1000): conn = pooled.connection() test(conn) def main(): t = testloop if len(sys.argv) == 1 else testpool for i in range(10): threading.Thread(target = t).start() if __name__ == "__main__": main()
看看 10 线程的测试结果。
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
$ time ./main.py testloop testloop testloop testloop testloop testloop testloop testloop testloop testloop real 0m4.471s user 0m0.570s sys 0m4.670s $ time ./main.py -l testpool testpool testpool testpool testpool testpool testpool testpool testpool testpool real 0m2.637s user 0m0.320s sys 0m2.750s
虽然测试方式不是很严谨,但从测试结果还是能感受到 DBUtils 带来的性能提升。当然,我们我们也可以在 testloop() 中一直重复使用一个不关闭的 Connection,但这却不适合实际开发时的情形。
DBUtils 提供了几个参数,便于我们更好地调整资源利用。
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
DBUtils.PooledDB.PooledDB(self, creator, mincached=0, maxcached=0, maxshared=0, maxconnections=0, blocking=False, maxusage=None, setsession=None, failures=None, *args, **kwargs) Docstring: Set up the DB-API 2 connection pool. creator: either an arbitrary function returning new DB-API 2 connection objects or a DB-API 2 compliant database module mincached: initial number of idle connections in the pool (0 means no connections are made at startup) maxcached: maximum number of idle connections in the pool (0 or None means unlimited pool size) maxshared: maximum number of shared connections (0 or None means all connections are dedicated) When this maximum number is reached, connections are shared if they have been requested as shareable. maxconnections: maximum number of connections generally allowed (0 or None means an arbitrary number of connections) blocking: determines behavior when exceeding the maximum (if this is set to true, block and wait until the number of connections decreases, otherwise an error will be reported) maxusage: maximum number of reuses of a single connection (0 or None means unlimited reuse) When this maximum usage number of the connection is reached, the connection is automatically reset (closed and reopened). setsession: optional list of SQL commands that may serve to prepare the session, e.g. ["set datestyle to ...", "set time zone ..."] failures: an optional exception class or a tuple of exception classes for which the connection failover mechanism shall be applied, if the default (OperationalError, InternalError) is not adequate args, kwargs: the parameters that shall be passed to the creator function or the connection constructor of the DB-API 2 module
DBUtils 仅提供给了连接池管理,实际的数据库操作依然是由符合 DB-API 2 标准的目标数据库模块完成的。
一个面向对象使用DBUtils的栗子
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
# coding=utf-8 """ 使用DBUtils数据库连接池中的连接,操作数据库 OperationalError: (2006, ‘MySQL server has gone away’) """ import json import pymysql import datetime from DBUtils.PooledDB import PooledDB import pymysql class MysqlClient(object): __pool = None; def __init__(self, mincached=10, maxcached=20, maxshared=10, maxconnections=200, blocking=True, maxusage=100, setsession=None, reset=True, host='127.0.0.1', port=3306, db='test', user='root', passwd='123456', charset='utf8mb4'): """ :param mincached:连接池中空闲连接的初始数量 :param maxcached:连接池中空闲连接的最大数量 :param maxshared:共享连接的最大数量 :param maxconnections:创建连接池的最大数量 :param blocking:超过最大连接数量时候的表现,为True等待连接数量下降,为false直接报错处理 :param maxusage:单个连接的最大重复使用次数 :param setsession:optional list of SQL commands that may serve to prepare the session, e.g. ["set datestyle to ...", "set time zone ..."] :param reset:how connections should be reset when returned to the pool (False or None to rollback transcations started with begin(), True to always issue a rollback for safety's sake) :param host:数据库ip地址 :param port:数据库端口 :param db:库名 :param user:用户名 :param passwd:密码 :param charset:字符编码 """ if not self.__pool: self.__class__.__pool = PooledDB(pymysql, mincached, maxcached, maxshared, maxconnections, blocking, maxusage, setsession, reset, host=host, port=port, db=db, user=user, passwd=passwd, charset=charset, cursorclass=pymysql.cursors.DictCursor ) self._conn = None self._cursor = None self.__get_conn() def __get_conn(self): self._conn = self.__pool.connection(); self._cursor = self._conn.cursor(); def close(self): try: self._cursor.close() self._conn.close() except Exception as e: print e def __execute(self, sql, param=()): count = self._cursor.execute(sql, param) print count return count @staticmethod def __dict_datetime_obj_to_str(result_dict): """把字典里面的datatime对象转成字符串,使json转换不出错""" if result_dict: result_replace = {k: v.__str__() for k, v in result_dict.items() if isinstance(v, datetime.datetime)} result_dict.update(result_replace) return result_dict def select_one(self, sql, param=()): """查询单个结果""" count = self.__execute(sql, param) result = self._cursor.fetchone() """:type result:dict""" result = self.__dict_datetime_obj_to_str(result) return count, result def select_many(self, sql, param=()): """ 查询多个结果 :param sql: qsl语句 :param param: sql参数 :return: 结果数量和查询结果集 """ count = self.__execute(sql, param) result = self._cursor.fetchall() """:type result:list""" [self.__dict_datetime_obj_to_str(row_dict) for row_dict in result] return count, result def execute(self, sql, param=()): count = self.__execute(sql, param) return count def begin(self): """开启事务""" self._conn.autocommit(0) def end(self, option='commit'): """结束事务""" if option == 'commit': self._conn.autocommit() else: self._conn.rollback() if __name__ == "__main__": mc = MysqlClient() sql1 = 'SELECT * FROM shiji WHERE id = 1' result1 = mc.select_one(sql1) print json.dumps(result1[1], ensure_ascii=False) sql2 = 'SELECT * FROM shiji WHERE id IN (%s,%s,%s)' param = (2, 3, 4) print json.dumps(mc.select_many(sql2, param)[1], ensure_ascii=False)