Netty series (1) Linux network IO model

1. Basic Concepts

Before we formally start talking about the Linux IO model, we will introduce 5 basic concepts.

1.1 User space and kernel space

Now the operating system uses virtual memory, so for a 32-bit operating system, its addressing space (virtual memory space) is 4G (2 to the 32nd power). The core of the operating system is the kernel, which is independent of ordinary applications and can access the protected memory space and also have all the permissions to access the underlying hardware devices . In order to ensure that user processes cannot directly operate the kernel and ensure the security of the kernel, the operating system divides the virtual space into two parts, one is the kernel space and the other is the user space . For the Linux operating system, the highest 1G bytes (from virtual address 0xC0000000 to 0xFFFFFFFF) are used by the kernel, which is called kernel space, and the lower 3G bytes (from virtual address 0x00000000 to 0xBFFFFFFF) are used for each Process use, called user space.

1.2 Process switching

In order to control the execution of a process, the kernel must have the ability to suspend a process running on the CPU and resume execution of a previously suspended process. This behavior is called process switching. Therefore, it can be said that any process runs with the support of the operating system kernel and is closely related to the kernel.

From the running of one process to running on another process, the following changes are made in this process:

Saves the processor context, including the program counter and other registers.
Update PCB information.
Move the PCB of the process into the corresponding queue, such as ready, blocked in a certain event queue.
Choose another process to execute, and update its PCB.
Update memory management data structures.
Restore the handler context.

Note: All in all, it is very resource-intensive.

1.3 Blocking of processes

The process being executed, because some expected events have not occurred, such as failure to request system resources, waiting for the completion of an operation, new data has not yet arrived or no new work to do, etc., the system automatically executes the blocking primitive (Block), Change itself from running state to blocking state. It can be seen that the blocking of the process is an active behavior of the process itself, so only the process in the running state (getting the CPU) can turn it into the blocking state. When the process enters the blocking state, it does not occupy CPU resources.

1.4 File descriptor fd

A file descriptor is a term in computer science, an abstraction used to express a reference to a file.

A file descriptor is formally a non-negative integer. In fact, it is an index value that points to the record table of the process's open files maintained by the kernel for each process . When a program opens an existing file or creates a new file, the kernel returns a file descriptor to the process. In programming, some low-level programming often revolves around file descriptors. However, the concept of file descriptors is often only applicable to operating systems such as UNIX and Linux.

1.5 Cache IO

Cached IO is also called standard IO, and the default IO operation of most file systems is cached IO. In the cache IO mechanism of Linux, the operating system will cache the IO data in the page cache (page cache) of the file system, that is to say, the data will be copied to the buffer of the operating system kernel first, and then will be loaded from the cache. The operating system kernel's buffers are copied into the application's address space.

Disadvantages of cached IO:

During the data transmission process, multiple data copy operations are required in the application address space and the kernel. The CPU and memory overhead caused by these data copy operations is very large.

2. Linux IO model

The essence of network IO is the reading of socket, socket is abstracted as stream in linux system, IO can be understood as the operation of convection. As I just said, for an IO access (take read as an example), the data will be copied to the buffer of the operating system kernel first, and then copied from the buffer of the operating system kernel to the address space of the application. So, when a read operation happens, it goes through two phases:

Stage 1: Waiting for the data to be ready.
Stage 2: Copying the data from the kernel to the process.

For socket streams,

The first step: usually involves waiting for a data packet on the network to arrive and then be copied to some buffer in the kernel.
Step 2: Copy the data from the kernel buffer to the application process buffer.

Network applications need to deal with nothing more than two types of problems, network IO, and data computing. Compared with the latter, the delay of network IO brings more performance bottlenecks to the application than the latter. The models of network IO are roughly as follows:

Synchronous model (synchronous IO)
Blocking IO (bloking IO)
Non-blocking IO (non-blocking IO)
Multiplexing IO (multiplexing IO)
Signal-driven IO (signal-driven IO)
Asynchronous IO (asynchronous IO)

Note: Since signal driven IO is not commonly used in practice, I only mention the remaining four IO Models.

1.2 Synchronous blocking IO (blocking IO)

2.1.1 Scenario description

After my girlfriend and I ordered the meal, we didn't know when it would be ready, so we had to sit in the restaurant and wait until it was ready, and then leave after eating. My girlfriend wanted to go shopping with me, but she didn't know when the meal would be ready, so she had to wait in the restaurant with me instead of going shopping. She couldn't go shopping until the meal was over, and the time spent waiting for cooking was wasted. . This is typical blocking.

2.1.2 Network Model

The synchronous blocking IO model is the most commonly used and the simplest model. In linux, all sockets are blocking by default. It conforms to the most common thinking logic of people. Blocking is when the process "is" rested, and the CPU is left to handle other processes.

In this IO model, a user-space application executes a system call (recvform), which causes the application to block and do nothing until the data is ready, and the data is copied from the kernel to the user process, which is then processed Data, in the two stages from waiting for data to processing data, the entire process is blocked. Cannot handle other network IO. The calling application is in a state where it no longer consumes CPU but simply waits for a response, so this is very efficient from a processing standpoint. When calling the recv()/recvfrom() function, the process of waiting for data and copying data occurs in the kernel, as shown in the following figure:

Figure 1-1 Synchronous blocking IO

2.1.3 Process Description

When the user process calls the recv()/recvfrom() system call, the kernel starts the first stage of IO: preparing data (for network IO, many times the data has not arrived at the beginning. For example, not yet Receive a complete UDP packet. At this time, the kernel will wait for enough data to arrive). This process needs to wait, that is to say, a process is required for data to be copied to the buffer of the operating system kernel. On the user process side, the entire process will be blocked (of course, the process itself chooses to block). The second stage: When the kernel waits until the data is ready, it will copy the data from the kernel to the user memory , and then the kernel returns the result, and the user process releases the block state and starts running again.

Therefore, the characteristic of blocking IO is that it is blocked in both stages of IO execution.

advantage:

Able to return data in time without delay;
This saves trouble for kernel developers;

shortcoming:

For users, waiting will pay the price of performance;

2.2 Synchronous non-blocking IO (nonblocking IO)

2.2.1 Scenario description

My girlfriend is not willing to wait here in vain, she wants to go shopping, and she is worried that the meal is ready. So we strolled around for a while, and came back to ask the waiter if the meal was ready. We went back and forth many times, and we were exhausted before we even had the meal. This is non-blocking. You need to keep asking if you are ready.

2.2.2 Network Model

Synchronous non-blocking is a polling method of "seeing the progress bar every once in a while". In this model, the device is opened in a non-blocking fashion . This means that the IO operation will not complete immediately, and the read operation may return an error code stating that the command cannot be satisfied immediately (EAGAIN or EWOULDBLOCK).

During network IO, non-blocking IO will also make a recvform system call to check whether the data is ready, which is different from blocking IO. "Non-blocking divides the blockage of a large whole piece of time into N many small blocks, so the process continues to There is a chance of being 'patched' by the CPU".

That is to say, after the non-blocking recvform system call, the process is not blocked, and the kernel returns to the process immediately. If the data is not ready, an error will be returned at this time. After the process returns, it can do something else and then issue the recvform system call. Repeat the above process and repeat the recvform system call. This process is often referred to as polling . Polling checks the kernel data until the data is ready, and then copies the data to the process for data processing. It should be noted that during the entire process of copying data, the process is still in a blocked state.

Under linux, you can make it non-blocking by setting the socket. When a read operation is performed on a non-blocking socket, the flow is as follows:

Figure 1-2 Synchronous non-blocking IO (nonblocking IO)

2.2.3 Process description

When the user process issues a read operation, if the data in the kernel is not ready, it will not block the user process, but immediately returns an error. From the perspective of the user process, after it initiates a read operation, it does not need to wait, but gets a result immediately. When the user process judges that the result is an error, it knows that the data is not ready, so it can send the read operation again. Once the data in the kernel is ready and the system call from the user process is received again, it immediately copies the data to the user memory and returns.

Therefore, the characteristic of nonblocking IO is that the user process needs to continuously ask the kernel data actively.

Synchronous non-blocking mode compared to synchronous blocking mode:

Advantages: can do other work while waiting for the task to complete (including submitting other tasks, that is, "background" can have multiple tasks executing at the same time).

Disadvantage: Increased response latency for task completion because read operations are polled every once in a while, and tasks may complete at any time between polls. This results in a reduction in overall data throughput.

2.3 IO multiplexing (IO multiplexing)

2.3.1 Scenario Description

Similar to the second solution, the restaurant installed an electronic screen to display the status of ordering, so that my girlfriend and I went shopping for a while, and when we came back, we didn’t have to ask the waiter, we could just look at the electronic screen. In this way, whether everyone's meal is ready, they can directly look at the electronic screen. This is a typical IO multiplexing.

2.3.2 Network Model

Because the synchronous non-blocking method requires continuous active polling, polling occupies a large part of the process, polling will consume a lot of CPU time, and there may be multiple tasks in the "background" at the same time, people think of looping query multiple The completion status of the task, as long as any task is completed, it is processed. It would be nice if the polling wasn't the user's process, but someone helped. Then this is called "IO multiplexing" . Select, poll, and epoll under UNIX/Linux do this (epoll is more efficient than poll and select, and does the same thing).

IO multiplexing has two special system calls select, poll, and epoll functions. The select call is at the kernel level. The difference between select polling and non-blocking polling is that the former can wait for multiple sockets, and can monitor multiple IO ports at the same time. When the data of any one of the sockets is ready, the It can be returned for reading, and then the process will make a recvform system call to copy the data from the kernel to the user process. Of course, this process is blocked . After the select or poll is called, the process will be blocked. Different from blocking IO blocking, the select at this time does not wait until all the socket data arrives before processing, but will call the user process to process when there is a part of the data . How do you know that some data has arrived? The monitoring thing is handed over to the kernel, and the kernel is responsible for the processing of data arrival. It can also be understood as "non-blocking".

The I/O multiplexing model uses select, poll, and epoll functions. These functions will also block the process, but unlike blocking I/O, these two functions can block multiple I/O operations at the same time. . Moreover, the I/O functions of multiple read operations and multiple write operations can be detected at the same time, and the I/O operation function will not be called until data is readable or writable (note that not all data is readable or writable). .

For multiplexing, that is, polling multiple sockets. Since multiplexing can handle multiple IOs, it brings new problems. The order between multiple IOs becomes uncertain , and of course, it can also be used for different numbers. The specific process is shown in the following figure:

Figure 2-3 IO multiplexing (IO multiplexing)

2.3.3 Process Description

IO multiplexing is what we call select, poll, and epoll. In some places, this IO method is also called event driven IO. The advantage of select/epoll is that a single process can handle the IO of multiple network connections at the same time. Its basic principle is that the function select, poll, epoll will continuously poll all the sockets it is responsible for. When a certain socket has data arriving, it will notify the user process.

When the user process calls select, the entire process will be blocked , and at the same time, the kernel will "monitor" all sockets that select is responsible for. When the data in any socket is ready, select will return. At this time, the user process calls the read operation again to copy the data from the kernel to the user process.

The feature of multiplexing is that a process can wait for IO file descriptors at the same time through a mechanism, the kernel monitors these file descriptors (socket descriptors), and any one of them enters the read ready state, select, poll, epoll functions to return. For monitoring methods, it can be divided into three methods: select, poll, and epoll.

The above diagram is not very different from the blocking IO diagram. In fact, it is even worse. Because two system calls (select and recvfrom) need to be used here, and blocking IO only calls one system call (recvfrom). However, the advantage of using select is that it can handle multiple connections at the same time.

Therefore, if the number of connections processed is not very high, the web server using select/epoll is not necessarily better than the web server using multi-threading + blocking IO, and the delay may be greater. The advantage of select/epoll is not that it can handle a single connection faster, but that it can handle more connections.

In the IO multiplexing Model, in practice, each socket is generally set to be non-blocking, but as shown in the figure above, the entire user process is actually blocked all the time. It's just that the process is blocked by the select function, not by socket IO. So IO multiplexing is blocked on system calls such as select and epoll, but not on real I/O system calls such as recvfrom.

After understanding the previous three IO modes, when the user process makes a system call, when they wait for the data to arrive, the processing method is different, directly waiting, polling, select or poll polling, two-stage process:

In the first stage, some are blocked, some are not blocked, and some can be blocked and not blocked.
The second stage is all blocking.

From the perspective of the entire IO process, they are executed sequentially, so they can be classified as synchronous. All processes actively wait and check the status with the kernel. 【This sentence is very important! ! ! 】

Programs with high concurrency generally use synchronous non-blocking methods instead of multi-threading + synchronous blocking methods. To understand this, it is first necessary to talk about the difference between concurrency and parallelism. For example, to do business in a certain department, you need to go to several windows in sequence. The number of people in the service hall is the number of concurrency, and the number of windows is the degree of parallelism. That is to say, the number of concurrency refers to the number of concurrent tasks (such as HTTP requests being served at the same time), and the number of parallelism is the number of physical resources that can work at the same time (such as the number of CPU cores). By rationally scheduling different stages of tasks, the number of concurrency can be far greater than the degree of parallelism, which is the mystery of how a few CPUs can support tens of thousands of concurrent requests from users. In such a high concurrency situation, creating a process or thread for each task (user request) is very expensive. The synchronous non-blocking method can throw multiple IO requests to the background, which can serve a large number of concurrent IO requests in one process.

Note: Whether IO multiplexing is a synchronous blocking model or an asynchronous blocking model, here is an analysis for you:

Synchronization is to actively wait for message notifications, while asynchronous is to passively receive message notifications and passively obtain messages through callbacks, notifications, and status. When IO multiplexing is blocked to the select stage, the user process actively waits and calls the select function to obtain the data ready status message, and its process status is blocked. Therefore, IO multiplexing is classified as synchronous blocking mode.

2.4 Signal-driven IO (signal-driven IO)

Signal-driven I/O: First of all, we allow Socket to perform signal-driven IO, and install a signal processing function, and the process continues to run without blocking. When the data is ready, the process will receive a SIGIO signal, and the I/O operation function can be called in the signal processing function to process the data. The process is shown in the figure below:

Figure 2-4 Signal-driven IO (signal-driven IO)

2.5 Asynchronous non-blocking IO (asynchronous IO)

2.5.1 Scenario Description

My girlfriend doesn't want to go shopping, and the restaurant is too noisy, so go home and have a good rest. So we ordered takeout, made a phone call to order food, and then my girlfriend and I could have a good rest at home, and the food was delivered to the house by the delivery man. This is a typical asynchronous, you just need to make a phone call, then you can do your own thing, and the meal will be delivered.

2.5.2 Network Model

Compared to synchronous IO, asynchronous IO is not executed sequentially. After the user process makes the aio_read system call, whether the kernel data is ready or not, it will be returned directly to the user process, and then the user mode process can do other things. When the socket data is ready, the kernel directly copies the data to the process, and then sends a notification from the kernel to the process. In both stages of IO, the process is non-blocking.

Linux provides AIO library functions for asynchronous implementation, but they are rarely used. There are many open source asynchronous IO libraries, such as libevent, libev, libuv. The asynchronous process is shown in the following figure:

Figure 2-4 Asynchronous non-blocking IO (asynchronous IO)

2.5.3 Process Description

After the user process initiates the aio_read operation, it can start doing other things immediately. On the other hand, from the perspective of the kernel, when it receives an asynchronous read, it will return immediately, so it will not generate any block for the user process . Then, the kernel will wait for the data to be ready, and then copy the data to the user memory. When all this is done, the kernel will send a signal to the user process or execute a thread-based callback function to complete this IO process , telling it The read operation is complete.

In Linux, the way of notification is "signal":

If the process is busy doing other things in user mode (for example, calculating the product of two matrices), then forcibly interrupt it and call the pre-registered signal handler, which can decide when and how to handle the asynchronous task . Since the signal processing function is suddenly intruded, there are many things that cannot be done like the interrupt handler. Therefore, to be on the safe side, the event is generally "registered" and put into the queue, and then return to what the process was doing. thing.
If the process is busy doing other things in the kernel state, such as reading and writing the disk in a synchronous blocking mode, it has to suspend the notification and wait until the kernel state is busy and is about to return to the user state. Signal notification.
If the process is suspended now, for example, there is nothing to do sleep, then wake up the process, and next time the CPU is idle, the process will be scheduled to trigger a signal notification.

Asynchronous APIs are easier said than done, mainly for the implementer of the API. Linux's asynchronous IO (AIO) support was introduced in 2.6.22, and there are many system calls that do not support asynchronous IO. Linux's asynchronous IO is originally designed for databases, so read and write operations through asynchronous IO will not be cached or buffered, which makes it impossible to use the operating system's caching and buffering mechanism.

Many people think of Linux's O_NONBLOCK as an asynchronous method, but in fact this is the synchronous non-blocking method mentioned earlier. It should be pointed out that although the IO API on Linux is slightly rough, each programming framework has a packaged asynchronous IO implementation. The operating system does less work and leaves more freedom to the user, which is the design philosophy of UNIX and one of the reasons why the programming frameworks on Linux flourish.

From the previous classification of IO models, we can see the motivation of AIO:

The synchronous blocking model requires the application to be blocked at the beginning of an IO operation. This means that it is impossible to overlap processing and IO operations at the same time.
The synchronous non-blocking model allows processing and IO operations to overlap, but requires the application to check the status of the IO operations against recurring rules.
This leaves asynchronous non-blocking IO, which allows processing to overlap with IO operations, including notification that the IO operation is complete.

IO multiplexing The function provided by the select function (asynchronous blocking IO) is similar to AIO except that it requires blocking. However, it blocks on notification events, not IO calls.

2.6 About asynchronous blocking

Sometimes our API only provides asynchronous notification, such as in node.js, but the business logic needs to do one thing after another, such as the database connection is initialized before it can start accepting user HTTP requests. Such business logic requires the caller to work in a blocking manner.

In order to simulate the effect of "sequential execution" in an asynchronous environment, it is necessary to convert synchronous code into asynchronous form, which is called CPS (Continuation Passing Style) transformation. BYVoid's continuation.js library is a tool for CPS transformation. The user only needs to write the code in a synchronous way that is more in line with human common sense, and the CPS converter will convert it into a form of asynchronous callbacks nested in layers.