1. The execution flow of an update Sql statement
The basic process of the update statement is the same as that of the query statement, that is, it must be processed by the parser and optimizer, and finally handed over to the executor. The difference lies in the operation after obtaining the qualified data.
1.1 Buffer Pool
First of all, for the InnoDB storage engine, data is stored on the disk. To operate the data, the storage engine must load the data from the disk into the memory before it can operate.
Here is a question, is it true that how much data we need, how much data do we load from disk to memory at once? For example, I want to read 6 bytes.
Disk I/O reads and writes are very slow compared to memory operations. If the data we need is scattered in different places on the disk, it means that there will be many I/O operations.
Therefore, whether it is an operating system or a storage engine, there is a concept of pre-reading . In other words, when a piece of data on the disk is read, it is very likely that the location near it will also be read immediately. This is called the principle of locality. So in this way, we simply read a little more each time instead of reading as much as we use.
InnoDB sets up a storage engine to read data from the disk to the smallest unit of memory, called a page. The operating system also has the concept of pages. The page size of the operating system is generally 4K, and in InnoDB, the smallest unit is 16KB by default. If you want to modify the size of this value, you need to clear the data and reinitialize the service.
There is a problem here. When manipulating data, it must be read from the disk to the memory (and then returned to the Server) every time.
Is there any way to improve efficiency? Or the idea of caching. Cache the read data pages.
InnoDB designed a memory buffer. When reading data, first judge whether it is in this memory area, if it is, read it directly, and then operate without loading it from the disk again. If not, write to the buffer of this memory after reading. This memory area has an exclusive name called Buffer Pool
When modifying data, it is also written to the buffer pool first, instead of directly writing to the disk. When the data pages of the memory are inconsistent with the disk data, we call them dirty pages. When will dirty pages be synchronized to disk?
There is a special background thread in InnoDB to write the data of the Buffer Pool to the disk, and write multiple changes to the disk at a time. This action is called flushing.
To summarize: The function of Buffer Pool is to improve the efficiency of reading and writing.
1.2 Redo log (for persistence)
Consider a question: because flushing is not real-time, if the dirty pages in the Buffer Pool are not flushed to the disk, the database is down or restarted, the data will be lost.
then what should we do? Therefore, the data in the memory must have a persistent measure.
In order to avoid this problem, InnoDB writes all modification operations on the page to a log file.
If there is data that has not been synchronized to the disk, when the database is started, it will recover from this log file (implement crash-safe). The D (persistence) in ACID of the transaction we are talking about is realized by it.
This log file is the redo log of the disk (called redo log)
Does any student have such a question: The same is writing to disk, why not directly write to dbfile? Why write the log first and then write to the disk?
What is the difference between writing to a log file and writing to a data file?
Let us first talk about the process of disk addressing. This is the structure of the disk. The disk of the magnetic disk keeps rotating, and the head will draw a circular track on the surface of the magnetic disk. This is called a track. There are many tracks with different radii from inside to outside. Then use the radius line to divide the track into sectors (the sectors within the two rays form sectors). If you want to read and write data, you must find the sector corresponding to the data. This process is called addressing.
- Random IO :
If the data we need is randomly scattered in different sectors of different pages on the disk, then we need to wait until the magnetic arm rotates to the specified page to find the corresponding data, and then the disk can find the corresponding sector before it can be found We need a piece of data. This process is performed at a time until all the data is found. This is random IO, and the speed of reading data is slow. - Sequential IO :
Assuming that we have found the first piece of data, and other required data is behind this piece of data, then there is no need to re-address, and we can get the data we need in turn. This is called sequential IO.
Flushing is random I/O, while logging is sequential I/O (continuously written). Sequential I/O is more efficient. It is essentially the difference between centralized storage and distributed storage of data. Therefore, the modification is written to the log file first. While the security of the memory data is ensured, the timing of flashing can be delayed, thereby increasing the system throughput.
The redo log is located in ib_logfile0 and ib_logfile1 under the /var/lib/mysql/ directory, and there are 2 files by default, each of 48M.
show variables like 'innodb_log%';
| parameter | meaning |
|---|---|
| innodb_log_file_size | Specify the size of each file, the default is 48M |
| innodb_log_files_in_group | Specify the number of files, the default is 2 |
| innodb_log_group_home_dir | Specify the path of the file, relative or absolute. If not specified, it is the datadir path |
Features of redo log
- The redo log is implemented by the InnoDB storage engine, not all storage engines have it. Support for crash recovery is a feature of InnoDB.
- The redo log does not record the status after the data page is updated, but records "what changes have been made on a certain data page". Belongs to the physical log.
- The size of the redo log is fixed, and the previous content will be overwritten. Once it is full, it will trigger the synchronization of the buffer pool to the disk in order to free up space to record future changes.
In addition to the redo log, there is also a log related to the modification, called undo logo. Redo log and undo log are closely related to transactions and are collectively referred to as transaction logs .
1.3. Undo log Undo log or rollback log
The undo log records the data status before the transaction occurs, and is divided into insert undo log and update undo log. If an exception occurs when modifying the data, undo log can be used to implement a rollback operation (to maintain atomicity).
It can be understood that the undo log records reverse operations . For example, insert records delete, and update records the original value of update, which is different from the physical page recorded by redolog, so it is called logical format log .
show global variables like '%undo%';
| parameter | meaning |
|---|---|
| innodb_undo_directory | The path of the undo file |
| innodb_undo_log_truncate | Set to 1, that is, open online recovery (shrink) undo log files |
| innodb_max_undo_log_size | If innodb_undo_log_truncate is set to 1, the truncate recovery (shrinking) action will be triggered when the size exceeds this size. If the page size is 16KB, the space after truncate is reduced to 10M by default 1073741824 bytes = 1G. |
| innodb_undo_logs | The number of rollback segments, the default is 128, this parameter is obsolete. |
| innodb_undo_tablespaces | Set the number of undo independent tablespaces, the range is 0-95, and the default is 0, which means that the independent undo tablespace is not turned on, and the undo log is stored in the ibdata file. This parameter is obsolete, |
1.4 Update process
With these logs, let's summarize the process of an update operation, which is a simplified process. The original value of name is xhc
update user set name = '中华第一帅' where id=1;
- At the beginning of the transaction, the data page containing this data is retrieved from the memory (buffer pool) or disk (data file) and returned to the executor of the Server;
- Server's executor modifies the value of this row of data on the data page to'China's first handsome';
- Record name=xhc to undo log;
- Record name=China's first handsome to redo log;
- Record bin log;
- Call the storage engine interface and record the data page to the Buffer Pool (modify name=China's first handsome);
- The transaction is committed.
1.5 bin log
What is binlog?
Binlog records the database table structure and table data changes, such as update/delete/insert/truncate/create. It will not record the select, because this does not make changes to the table.
Binlog records all DDL and DML statements in the form of events (because it records operations rather than data values, it belongs to logical logs), which can be used for master-slave replication and data recovery. Unlike redo log, its file content can be appended without a fixed size limit.
With the binlog function turned on, we can export binlog to SQL statements and replay all operations to achieve data recovery.
What does binlog look like? Binlog we can understand the SQL statement that stores each change.
What is binlog generally used for?
Two main functions: copy and restore data
-
When MySQL is used in the company, it has a one-master and multiple-slave structure. The slave server needs to keep the data consistent with the master server. This is achieved through binlog (master-slave replication of the database). Its principle is to read from the server. Take the binlog of the main server and execute it again.
-
The data of the database is killed, we can restore the data through binlog. Because binlog records the changes of database tables, we can use binlog to copy and restore data
After opening the bin log, let's take a look at how an update statement is executed (redo cannot be written at once)
Two-phase commit:
For example, a statement:
update teacher set name='靓仔' where id=1;
- This data is queried first, and if there is a cache, the cache will also be used.
- Change the name to "Jianzi", then call the API interface of the engine, write this line of data to the memory, and record the redo log. At this time, the redo log enters the prepare state, and then tells the executor that the execution is complete and can be submitted at any time.
- The executor records the binlog after receiving the notification, and then calls the storage engine interface to set the redo log to commit state.
- update completed.
The main point of this picture:
- Record to the memory first, and then write the log file.
- Recording redo log is divided into two stages.
- The storage engine and the server record different logs.
- Record redo first, then bin log
Why is a two-phase commit required?
Example:
If we change the name to "馬仔", if we finish writing the redo log, and before writing the binlog, MySQL restarts.
Because the redo log can be used to restore data when restarting, it is the "beautiful boy" that writes to the disk. But this logical log is not recorded in binlog, so if binlog is used to restore data or synchronize to the slave library, data inconsistency will occur.
So in the case of writing two logs, binlog acts as the coordinator of a transaction. Notify InnoDB to execute prepare or commit or rollback.
If step ⑥ fails to write to the binlog, it will not be submitted.
Simply put, there are two log writing operations, similar to a distributed transaction . Without a two-phase commit, there is no guarantee that both will succeed or fail.
During crash recovery, determine whether the transaction needs to be committed:
- Binlog has no record, redolog has no record: crash before redolog is written, recovery operation: roll back the transaction
- Binlog has no record, redolog status prepare: crash before binlog is written, recovery operation: rollback transaction
- Binlog has records, redolog state prepare: crash before the commit transaction after binlog is written, recovery operation: commit transaction
- Binlog has records, redolog status commit: normally completed transactions, no need to restore