Python multithreaded programming-basic articles

1. Pre-school information

1. Need to master python basic grammar, python file operation, python module application

2. Development tools: pycharm, python3.6 and above

3. Learning content: process, thread, multitasking application

2. Introduction to Multitasking

2.1 Concept:

Multitasking refers to the execution of multiple tasks at the same time

2.2 Two manifestations of multitasking

Concurrency (Alternating to perform multiple tasks in a period of time)

Parallel (really execute multiple tasks together in a period of time)

Third, the process

3.1 Concept of process:

The process is the smallest unit of resource allocation, and it is the basic unit of the operating system for resource allocation and scheduling operation. Popular understanding: a running program is a process. For example: running qq, wechat, etc., they are all a process.

3.2 The role of multi-process

3.3 Process creation steps

Import process package

import multiprocessing

Create a process object through the process class

Process object = multiprocessing.Process()

Start the process to perform tasks

Process object.start()

3.4 Create a process object through the process class

3.5 Simple examples of single-task and multi-task

Single task

import time

def sing():
    for i in range(3):
        print("唱歌。。。")
        time.sleep(0.5)
def dance():
    for i in range(3):
        print("跳舞。。。")
        time.sleep(0.5)
if __name__ == '__main__':
    time1 = time.time()
    sing()
    dance()
    print(time.time()-time1)

Output the result
Sing. . .
Sing. . .
Sing. . .
dancing. . .
dancing. . .
dancing. . .
3.0027265548706055

Use multiple processes to achieve multitasking

import multiprocessing
import time

def sing():
    for i in range(3):
        print("唱歌。。。")
        time.sleep(0.5)
def dance():
    for i in range(3):
        print("跳舞。。。")
        time.sleep(0.5)
if __name__ == '__main__':
    time1 = time.time()
    s1 = multiprocessing.Process(target=sing)
    d1 = multiprocessing.Process(target=dance)
    s1.start()
    d1.start()
    s1.join() #这个方法可以等待子进程结束后再继续往下运行，通常用于进程间的同步
    d1.join()
    print(time.time()-time1)

Output the result
Sing. . .
dancing. . .
Sing. . .
dancing. . .
Sing. . .
dancing. . .
1.7515902519226074

3.6 The process executes tasks with parameters

import multiprocessing
import time

def sing(name, num):
    for i in range(num):
        print("%s在唱歌。。。"%name)
        time.sleep(0.5)
def dance(num):
    for i in range(num):
        print("跳舞。。。")
        time.sleep(0.5)
if __name__ == '__main__':
    # 以元组形式传参
    s1 = multiprocessing.Process(target=sing, args=('小明', 3))
    # 以字典形式传参
    d1 = multiprocessing.Process(target=dance, kwargs={"num": 5, })
    s1.start()
    d1.start()

Note
1. Tuple parameters must be passed in the same order as the parameters
2. When passing parameters in dictionary mode, the key of the dictionary must be the same as the parameter name

3.7 Get the process number

The role of the process number:

When the number of processes in the program is increasing, if there is no way to distinguish between the main process and the child process as well as different child processes, then effective process management cannot be carried out. In order to facilitate management, each process actually has its own number. .

There are two ways to get the process number:

1. Get the current process number

os.getpid()

2. Get the current parent process number

os.getppid()

import multiprocessing
import time
import os
def sing(name, num):
    print("唱歌进程的编号： ", os.getpid())
    print("唱歌进程的父进程的编号： ", os.getppid())
    for i in range(num):
        print("%s在唱歌。。。"%name)
        time.sleep(0.5)
def dance(num):
    print("跳舞进程的编号： ", os.getpid())
    print("跳舞进程的父进程的编号： ", os.getppid())
    for i in range(num):
        print("跳舞。。。")
        time.sleep(0.5)
if __name__ == '__main__':
    print("主进程的编号： ", os.getpid())
    # 以元组形式传参   顺序和参数顺序一致
    s1 = multiprocessing.Process(target=sing, args=('小明', 3))
    # 以字典形式传参   key和参数名保持一致
    d1 = multiprocessing.Process(target=dance, kwargs={"num": 5, })
    s1.start()
    d1.start()

Output result:
The number of the main process: 10620 The number of the
singing process: 18412 The number
of the parent process of the singing process: 10620
Xiao Ming is singing. . .
The number of the dancing process: 9924 The number
of the parent process of the dancing process: 10620
dancing. . .
Xiao Ming is singing. . .
dancing. . .
Xiao Ming is singing. . .
dancing. . .
dancing. . .
dancing. . .

3.8 Points to note in the process

The main process will wait for the execution of all child processes to end at the end

import time
import multiprocessing

def work():
    for i in range(10):
        print("工作中。。。。")
        time.sleep(0.2)

if __name__ == '__main__':
    work_process = multiprocessing.Process(target=work)
    work_process.start()

    time.sleep(1)
    print("主进程执行结束")

Output result:
working. . . .
At work. . . .
At work. . . .
At work. . . .
The execution of the main process is finished
. . . .
At work. . . .
At work. . . .
At work. . . .
At work. . . .
At work. . . .

Set the main daemon process

After the main process ends, the remaining work in the child process will not continue

import time
import multiprocessing

def work():
    for i in range(10):
        print("工作中。。。。")
        time.sleep(0.2)

if __name__ == '__main__':
    # 设置守护主进程，主进程执行完成，则结束
    # 方式1
    work_process = multiprocessing.Process(target=work, daemon=True)
    # 方式2
    # work_process.daemon = True
    work_process.start()

    time.sleep(1)
    print("主进程执行结束")

Output result:
working. . . .
At work. . . .
At work. . . .
At work. . . .
End of main process execution

3.9 Case-Multi-process realization of video folder multi-task copyer

1. Requirement analysis:
1) Whether the target folder exists, if it does not exist, create it, if it exists, do not create it
2) Traverse all files in the source folder and copy them to the target folder
3) Use processes to achieve multitasking and complete the copy
2. Implementation steps:
1) Define the path where the source folder is located and the path where the target folder is located.
Create the target folder.
2) Obtain the file list in the source directory through os.listdir.
3) Traverse each file and define a function to implement specifically File copy
Use processes to achieve multi-tasking and complete high-concurrency copying

import os
import multiprocessing
def copy_file(file_name, source_dir, dest_dir):
    # 1 拼接源文件路径和目标文件路径
    source_path = source_dir + '\\' + file_name
    dest_path = dest_dir + '\\' + file_name
    # 2 打开源文件和目标文件
    with open(source_path, 'rb') as source_file:
        with open(dest_path, 'wb') as dest_file:
            # 3 循环读取源文件到目标路径
            while True:
                data = source_file.read(1024)
                if data:
                    dest_file.write(data)
                else:
                    break
if __name__ == '__main__':
    # 1 定义源文件夹和目标文件夹
    source_dir = r'F:\迅雷下载\视频-智能机器人从0到1系统入门课程\视频'
    dest_dir= r'F:\目标文件夹'

    # 2.创建目标文件夹
    try:
        os.mkdir(dest_dir)
    except:
        print("目标文件夹已经存在")
    # 3.读取源文件夹的文件列表
    file_list = os.listdir(source_dir)
    # 4.遍历文件列表实现拷贝
    for file_name in file_list:
        # copy_file(file_name, source_dir, dest_dir)
        # 5.使用多进程实现多任务拷贝
        sub_process = multiprocessing.Process(target=copy_file, args=(file_name, source_dir, dest_dir))
        sub_process.start()

Four, thread

4.1 Introduction to threads

1. Another form of multitasking
2. Thread is the smallest unit of program execution
3. Multiple threads belonging to the same process share all the resources owned by the process

4.2 Thread creation steps

1. Import the thread package

import threading

2. Create a process object through the thread class

线程对象 = threading.Thread()

Start threads to perform tasks

线程对象.start()

4.3 Create thread object through thread class

4.4 Thread usage is consistent with process usage, so I won’t repeat it here

import os
import threading
def copy_file(file_name, source_dir, dest_dir):
    # 1 拼接源文件路径和目标文件路径
    source_path = source_dir + '\\' + file_name
    dest_path = dest_dir + '\\' + file_name
    # 2 打开源文件和目标文件
    with open(source_path, 'rb') as source_file:
        with open(dest_path, 'wb') as dest_file:
            # 3 循环读取源文件到目标路径
            while True:
                data = source_file.read(1024)
                if data:
                    dest_file.write(data)
                else:
                    break
if __name__ == '__main__':
    # 1 定义源文件夹和目标文件夹
    source_dir = r'F:\迅雷下载\视频-智能机器人从0到1系统入门课程\视频'
    dest_dir= r'F:\目标文件夹'

    # 2.创建目标文件夹
    try:
        os.mkdir(dest_dir)
    except:
        print("目标文件夹已经存在")
    # 3.读取源文件夹的文件列表
    file_list = os.listdir(source_dir)
    # 4.遍历文件列表实现拷贝
    for file_name in file_list:
        # copy_file(file_name, source_dir, dest_dir)
        # 5.使用多线程实现多任务拷贝
        sub_thread = threading.Thread(target=copy_file, args=(file_name, source_dir, dest_dir))
        sub_thread.start()

4.5 The order of execution between threads:

Out of order, a thread is executed first by the CPU scheduling

Five, process and thread comparison

5.1 Relationship comparison

1. Threads are attached to the process. Without threads, there is no process
. 2. A process provides one thread by default, and the process can create multiple threads

5.2 difference to this

1. The resource overhead of creating a process is larger than that of creating a thread.
2. The process is the basic unit of operating system resource allocation, and the thread is the basic unit of CPU scheduling.
3. Threads cannot be executed independently and must exist in the process.

5.3 Comparison of advantages and disadvantages

1.
Advantages and disadvantages of the process: 1.1 Advantages: multi-core can be used
1.2 Disadvantages: large resource overhead
2. Thread advantages and disadvantages:
2.1 Advantages: low resource overhead
2.2 Disadvantages: multi-core not available

Reprinted at: https://blog.csdn.net/princezf/article/details/113110550