一、协程
1、协程
阻塞:代码执行到会产生阻塞操作的地方(如IO,socket.recv()等)会等待操作完成才继续往下执行。
非阻塞:即执行到阻塞操作时不会等待,而是将IO类操作交给其他程序或系统内核进程,然后会继续往下执行,等待系统返回完成信号,直接处理结果。
协程是阻塞模式
协程可以在单线程下实现高并发效果,在遇到IO时可以进行切换,然后执行其他任务,IO操作完成后再切换回来。协程本质上还是单线程,需要在进程的配合下才能理用多CPU。
协程在IO操作的时候时阻塞模式,即在某个方法内出现IO操作时就被阻塞,不能继续往下执行,IO完成后再继续往下执行,效果和生成器一样,但对于整个线程来说可以实现在多个方法或代码之前来回切换。
协程定义:
- 必须在只有一个单线程里实现并发
- 修改共享数据不需加锁
- 用户程序里自己保存多个控制流的上下文栈
- 一个协程遇到IO操作自动切换到其它协程
import time def consumer(name): print("{name} starting eating baozi".format(name=name)) while True: new_baozi = yield # 等待producer制作包子 time.sleep(1) print("{name} eating baozi {noew_baozi}".format(name=name, noew_baozi=new_baozi)) def producer(*args): for con in args: next(con) # 先next一次,让consumer执行到new_baozi = yield处 n = 0 while n < 3: n += 1 print("the {n} time making a baozi".format(n=n)) for con in args: con.send(n) # 手动切换到consumer函数,并返回new_baozi的值 if __name__ == '__main__': con1 = consumer("c1") con2 = consumer("c2") con3 = consumer("c3") producer(con1, con2, con3) """ 输出: c1 starting eating baozi c2 starting eating baozi c3 starting eating baozi the 1 time making a baozi c1 eating baozi 1 c2 eating baozi 1 c3 eating baozi 1 the 2 time making a baozi c1 eating baozi 2 c2 eating baozi 2 c3 eating baozi 2 the 3 time making a baozi c1 eating baozi 3 c2 eating baozi 3 c3 eating baozi 3 """
2、Gevent
Gevent 是一个第三方库,可以轻松通过gevent实现并发同步或异步编程,在gevent中用到的主要模式是Greenlet, 它是以C扩展模块形式接入Python的轻量级协程。 Greenlet全部运行在主程序操作系统进程的内部,但它们被协作式地调度。
from gevent import monkey # 自动识别那些模块会产生IO操作,然后自动进行切换 from urllib.request import urlopen import gevent monkey.patch_all() # 加载所有默认的补丁 def func(url): print("get :{url} ".format(url=url)) response = urlopen(url) data = response.read() print("{data_len} bytes received from {url}".format(data_len=len(data), url=url)) url_list = ["http://baidu.com", "http://taobao.com", "http://jd.com", "http://douyu.com"] print("同步模式") for i in url_list: func(i) print("异步模式") gevent.joinall([gevent.spawn(func, url) for url in url_list]) # [gevent.spawn(func, url) for url in url_list] 列表生成式 # gevent.joinall([]) # 这个函数可以使list里面的函数遇到io操作会主动进行切换,否则就需要等待io操作完成 """ 输出: 同步模式 get :http://baidu.com 118442 bytes received from http://baidu.com get :http://taobao.com 144945 bytes received from http://taobao.com get :http://jd.com 108724 bytes received from http://jd.com get :http://douyu.com 94158 bytes received from http://douyu.com 异步模式 get :http://baidu.com get :http://taobao.com get :http://jd.com get :http://douyu.com 118442 bytes received from http://baidu.com 94158 bytes received from http://douyu.com 108724 bytes received from http://jd.com 144945 bytes received from http://taobao.com """
上面程序的重要部分是将task函数封装到Greenlet内部线程的gevent.spawn
。 初始化的greenlet列表存放在list中,此list被传给gevent.joinall
函数,后者阻塞当前流程,并执行所有给定的greenlet。执行流程只会在 所有greenlet执行完后才会继续向下走
扫描二维码关注公众号,回复:
3552669 查看本文章
3、单线程socket并发
import socket import gevent from gevent import monkey monkey.patch_all() # 实现自动切换 def server(port): s = socket.socket() s.bind(("0.0.0.0", port)) s.listen(500) while True: conn, addr = s.accept() # 在这里阻塞,有新消息的时候会返回conn.recv print(addr) gevent.spawn(handle_request, conn) # 生gevent,执行hand_request def handle_request(conn): try: while True: data = conn.recv(1024) # 在这里阻塞,有新连接来的时候会回到s.accept send_msg = "recv:" + data.decode("utf8") conn.send(send_msg.encode("utf8")) if not data: conn.shutdown(socket.SHUT_WR) except Exception as e: print(e) finally: conn.close() if __name__ == '__main__': server(8888)
import socket import threading import time from multiprocessing import Process def client(pid): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(("localhost", 8888)) while True: time.sleep(1) # msg = input(">>").encode("utf8") msg = str(pid).encode("utf8") s.sendall(msg) data = s.recv(1024) print(data.decode("utf8")) if __name__ == '__main__': client_list = [] for i in range(1000): t = threading.Thread(target=client, args=(i,)) # 内存29% CPU24% # t = Process(target=client, args=(i,)) # 内存100% CPU100% t.start() client_list.append(t) for client in client_list: client.join()