"Hey, I've been under the control of Party A before I graduated. What can I do when I join the company later." Xiaobai complained so much, but he still didn't dare to neglect it psychologically, so he could only reluctantly close the game in front of him and open it up. Think about the Python code.
"The current program is single-threaded, so let's optimize it with a multi-threaded model. Hey, I'm too witty!" Xiaobai sincerely thanked the knowledge of threads he learned some time ago. "It seems that threadingmodules 's up to you threading!"
After two or three hours, Xiaobai finally changed the program. He breathed a sigh of relief, clicked Run, and started to test the running time.
"Woc! Why is the running time longer?" Looking at the test results displayed on the screen, Xiaobai was dumbfounded. Why isn't this multi-threading useful? I obviously came from the official document!
Debugging failed, Xiaobai had to seek help from his good friend Xiaoming.
"Hahaha, you actually use python's multithreading? Don't you know that python's multithreading is called "chicken ribs" by many people? "
"Ah, no? There is such a saying? If I knew I would definitely not need to change it with multi-threading... But why is python's multi-threading useless? What's going on with the multi-threaded crawler ?"
"Let me first explain to you what multithreading in python is all about. Python is an interpreted language , and its execution is controlled by an interpreter . We generally use the default Cpythoninterpreter , which I think You should know."
"Of course, this was covered by the teacher when I first started learning Python."
"Then do you know what GIL is ?" Xiao Ming asked.
Xiaobai scratched his head and replied awkwardly, "I've never heard of it."
"Also, if you know the GIL, you can understand Python's multithreading. GIL, the full name is Global Interpreter Lock , the global interpretation lock, which is specially used by the interpreter ."
"Clam? Does the interpreter need a lock?"
"This lock has no magical effect. Let me test you first. Can C language achieve thread-level time slice rotation in user mode ?"
"No! It was mentioned in the operating system, I still remember what you taught me last time~" Xiaobai replied proudly.
About the thread, the details are here: https://my.oschina.net/u/4198159/blog/3115533
"But python can do it! In python, the interpreter can keep track of how long each thread has been executing - and when the time is up, it can switch to another thread."
"It's kind of interesting, it sounds like the interpreter acts as the operating system and then provides the ability to rotate time slices for the python threads ."
tobe Note: In theory, the C language can also do it, after all, the python interpreter is written in the C language.
"The explanation is in place. Let me go back to the GIL. Before multi-core appeared, threads already existed. The GIL was used to lock threads. When a thread is about to execute, the interpreter will put the GIL. If the lock is given to this thread, other threads cannot run because there is no lock. When the thread holding the lock blocks or runs 100 bytecodes, the interpreter will give the lock to other threads ."
"But this GIL lock is global , which means that even in a multi-core case, only one thread can run at a time. On the whole, the entire program is serial ."
Xiaobai suddenly realized: "No wonder my program is still slow. It turns out that python's multi-threading can not only use multi-core, but also slows down the execution speed of my program because of thread switching ! I think many people must have encountered this problem. , why doesn't the Python community modify this feature to allow multithreading to be parallelized?"
Xiao Ming sighed: "How can it be so simple, it is difficult to modify the lock design. I heard that it took many years for MYSQL to split buffer pool mutexthis global lock. However, the Python community has done a lot of rescue work for this, such as **When the thread sleeps (sleep), it will actively release the GIL while waiting for the connection, and let other threads continue to execute.**Take the crawler as an example, a single crawler will always spend time downloading web pages, and a lot of CPU time is wasted , After providing the sleep mechanism, these crawlers can release the GIL lock while waiting for the download, giving the opportunity to other crawlers, so that the overall running speed can be greatly improved."
"I seem to understand," Xiaobai felt that he had been connected to the second line of Ren and Du: "That is to say , Python's multi-threading is suitable for I/O-intensive programs, but it is not so friendly for computationally-intensive programs ~ Eh, etc. Wait, what do I do? I'm also going to optimize my program with multithreading!"
"There are still ways to let python take advantage of multi-core, for example, let python call code in C language and implement multi-threading in C language , because there is no GIL lock in C language, these threads will not be constrained by GIL, so they can parallel."
Xiaobai shook his head vigorously: "No no no no no, you asked me to change the Python code I finally wrote to C? Isn't this killing me! I refuse!"
"Isn't there a second method - using multi-process , there is a multiprocessingmodule can create multiple processes, because different processes use different interpreters, so they have their own GIL, do not interfere with each other, naturally Concurrency is done. ”
"This method sounds normal, I'll go back and try it right away, thank you!"
I hope you gain something after reading my article~ (please like it!)
If this article is helpful to you, welcome to pay attention to the ravings of my public account tobe , and take you deep into the world of computers~ There are surprises in the background of the public account to reply to the keyword [computer]~
