协程(协作式,非抢占式,进程线程都是抢占式)
协程:又称微线程,纤程。英文名Coroutine。
优点:
- 协程极高的执行效率。因为子程序切换不是线程切换,而是程序自身控制,因此,没有线程切换的开销,和多线程比,线程数量越多,协程的性能优势就越明显。
- 不需要多线程的锁机制,因为只有一个线程,也不存在同时写变量冲突,在协程中控制共享资源不加锁,只需要判断状态就好了,所以执行效率比多线程高很多。
因为协程是一个线程执行,那怎么利用多核cpu呢?最简单的方法就是多进程+协程,既充分利用多核,又充分发挥协程的高效率,可获得极高的性能。
这儿还得说到yield,yield是协程的关键字,因为协程是用户态的切换
yield:
yield,一个函数加上yield后就是一个生成器。
def f():
print('ok')
yield 6
print('ok2')
yield
gen = f()
print(gen)
res =next(gen)
print(res)
运行结果:
< generator object f at 0x7fb2bd7286d0>
ok
6
def f():
print('ok')
yield 6
print('ok2')
yield
gen = f()
print(gen)
res =next(gen)
print(res)
next(gen)
运行结果:
< generator object f at 0x7f875df3b6d0>
ok
6
ok2
还有一种执行方式通过send()
def f():
print('ok')
s = yield 6
print(s)
print('ok2')
yield
gen = f()
print(gen)
res =next(gen)
print(res)
gen.send(5)
运行结果:
< generator object f at 0x7ff22a83a6d0>
ok
6
5
ok2
yield简单实现协程:
import time
import queue
def consumer(name):
print('。。。。。>ready to start')
while True:
new_test = yield
print('[%s] is starting test %s '%(name,new_test))
def producer():
r = con.__next__()
r = con2.__next__()
n = 0
while 1:
time.sleep(1)
print('\033[32;1m[producer]\033[0m is making test %s and %s'%(n,n+1))
con.send(n)
con2.send(n+1)
n +=2
if __name__ == '__main__':
con = consumer('c1')
con2 = consumer('c2')
producer()
运行结果是:
。。。。。>ready to start
。。。。。>ready to start
[producer] is making test 0 and 1
[c1] is starting test 0
[c2] is starting test 1
[producer] is making test 2 and 3
[c1] is starting test 2
[c2] is starting test 3
[producer] is making test 4 and 5
[c1] is starting test 4
[c2] is starting test 5
……….
……….
整个过程都是通过yield保存状态以及切换,这是协程的底层基于yield实现的。
关键点是:什么时候切换!这都是我们自己设置切换条件,就不需要依赖操作系统了。
greenlet(协程模块,python对底层的封装):
greenlet是一个用C实现的协程模块,相比与python自带的yield,它可以使你在任意函数之间随意切换,而不需把这个函数先声明为generator。
from greenlet import greenlet
def test1():
print(12)
gr2.switch()
print(34)
gr2.switch()
def test2():
print(56)
gr1.switch()
print(78)
gr1 = greenlet(test1)
gr2 = greenlet(test2)
gr1.switch()
运行结果是:
12
56
34
78
greenlet封装函数,通过switch切换。
协程主要解决的也是IO密集型的操作
当然在真正使用的时候,我们不可能每次切换都要我们自己写个switch来切换,所以我们用到了gevent。
gevent
gevent会自动帮你IO切换
gevent代码:
import gevent
import requests,time
start = time.time()
def f(url):
print('GET: %s' % url)
resp = requests.get(url)
data = resp.text
print('%d bytes received from %s.'%(len(data),url))
gevent.joinall([
gevent.spawn(f,'https://www.python.org/'),
gevent.spawn(f,'https://www.baidu.com/'),
gevent.spawn(f,'https://www.sina.com.cn/'),
])
print('cost time',time.time()-start)
运行结果:
GET: https://www.python.org/
48808 bytes received from https://www.python.org/.
GET: https://www.baidu.com/
2443 bytes received from https://www.baidu.com/.
GET: https://www.sina.com.cn/
569081 bytes received from https://www.sina.com.cn/.
cost time 8.537341833114624
import gevent
import requests,time
start = time.time()
def f(url):
print('GET: %s' % url)
resp = requests.get(url)
data = resp.text
print('%d bytes received from %s.'%(len(data),url))
# gevent.joinall([
# gevent.spawn(f,'https://www.python.org/'),
# gevent.spawn(f,'https://www.baidu.com/'),
# gevent.spawn(f,'https://www.sina.com.cn/'),
# ])
f('https://www.python.org/')
f('https://www.baidu.com/')
f('https://www.sina.com.cn/')
print('cost time',time.time()-start)
运行结果:
GET: https://www.python.org/
48808 bytes received from https://www.python.org/.
GET: https://www.baidu.com/
2443 bytes received from https://www.baidu.com/.
GET: https://www.sina.com.cn/
569089 bytes received from https://www.sina.com.cn/.
cost time 12.51875305175781
可以明显看出用协程的运行效率高于串行的,而且越多越明显。
协程的本质上就是一个线程,所以切换是没有开销的,多线程切换也是要开销的,而且协程也没有锁的概念了。
但协程也有一个问题:它能用多核吗?
它只有一个线程,所以用不了多核,所以我们要用多进程来实现多核,在通过协程来进行切换,这就是多进程+协程实现并发的一个很好的方案。