This article describes the conventional log files (not case storage engine) from two aspects, the second is to speak the transaction log innodb storage engine.
First, the basic log files
1.1, the basic log files Category:
error log (error log)
slow query log log (slow query log)
binary log (binlog)
query log (general log)
1.2, the error log
Including start mysql, operation, shut down the process of recording.
By default, the hostname of the server named `hostname`.err, parameters can be --log-error = [file_name] specified
are deleted after the error log, error log does not automatically create a state run only when you refresh the log It will create a new error log file.
View error log: show variables like 'log_error';
Refresh Method Log:
MySQL> the FLUSH the LOGS;
the shell> mysqladmin the flush-logs
the shell> mysqladmin Refresh
1.3, the general query log
Query log is divided into general query log and slow query log, parameter long_query_time determination of its value specified time, if the query is complete within the parameters set time, for the general query log (proposed closure because too many), otherwise slow query log.
Parameter set # --general_log = {0 |. 1 }: General query log is turned --general_log_file = file_name: Specifies the general query log path hostname.log` default name ` log_output = {TABLE | the FILE | NONE}: set general query log and slow query log output format, respectively (table | file | not recorded)
1.4, slow query log
Variable long_query_time query exceeds the specified time value for the slow queries. mysql slow query log records are recorded only after the inquiry is finished and has been completely release the lock, and therefore slow query execution order and logging SQL query sequence may be inconsistent
View slow query log: show variables like '% slow_query_log%';
Execute the statement
mysql>select sleep(10);
And then view lgh3-slow.log logs cat:
If a large number of slow query log data, we can use the command to view mysqldumpslow slow analyze log files:
[root @ lgh3 MySQL] # mysqldumpslow - Help Usage: mysqldumpslow [OPTS ...] [LOGS ...] . the Parse and log the Summarize at The MySQL Query Options are SLOW - verbose verbose - Debug Debug --help the Write the this Standard Output to text - V verbose - D Debug -s to the ORDER What Sort by (Al, AT, Ar, C, L, R & lt, T), 'AT' IS default # default collation al: average lock time # average locking time ar: average rows sent # average return record time at: average query time # average query time c: count # number of records l: lock time # lock time r: sent rows # Returns the record number t: query time # query time - R & lt Reverse The Sort Order (Last INSTEAD of Largest First) # Decommitment - T Top NUM Just Show The n-n-Queries #topBack -a do not abstract all numbers to N and strings to 'S' # N Do not use replacement numbers when classified, S replacement string -n NUM abstract a numbers with the n-digits the WITHIN AT Least names -g the PATTERN grep: only the Consider stmts the include that the this String # match the query, like grep function -h hOSTNAME hostname of db Server for * -slow.log filename (CAN BE wildcard), default IS '*' , IE match All -i nAME Server instance name of (if using mysql.server startup script) -l don't subtract lock time from total time
Use mysqldumpslow simple view: obviously lost a lot of things omitted comments
Related parameters:
# Slow query parameters and set log_throttle_queries_not_use_indexes = 10 , the write enable recording per minute slow the log, 0 is the default, not to limit long_query_time = 10 # slow query specified timeout period (10 seconds by default), beyond the length of a single belonging to the slow query log_output = {TABLE | FILE | NONE} # define general query log and slow query log output format, the default for File log_slow_queries = {yes | nO} # if slow query log is enabled, not enabled by default slow_query_log = {1 | ON | 0 | OFF} # also whether to enable slow query log, this variable and modify one another while log_slow_queries change slow_query_log_file = / mydata / the Data / hostname-slow.log # default path for the library files directory host name plus - sLOW .log log_queries_not_using_indexes = OFF # query does not use the index when the whether is also credited slow query log
1.5, the binary log file
Record all operations carried out changes to the mysql database does not include such operations select and show, if you perform such an operation update and delete, but did not cause any changes in the database data can also be written to the binary log file.
Binary log is recorded in the form of an event, not a transaction log (but may also be based on the transaction record binary log), for operations in the transaction table, the binary log only write-once (innodb transaction-based binary log when the transaction is submitted ), each binary logging are submitted before the first cache, write the time of submission. For the operation of non-transactional tables, each executing the statement is written directly.
mysql also creates a binary log index file, the corresponding information is written to the file when the binary log file rotation time. Therefore, the file contains all the binary log file name used. The binary log file with the same filename By default, the extension '.index'. A file name can be specified by the parameter --log-bin-index [= file_name]
1.5.1, the role of the binary log file
1, Recovery: Users can restore through point-in-time binary
2, copy: The binary file to perform data recovery on remote machines
3. Audit: Can the binary log audit file, determine whether the database injection attacks
1.5.2, View binary file
1. Use the mysqlbinlog tool.
2. Use the show corresponding to the displayed information.
SHOW {BINARY | MASTER} LOGS # log file to see which
SHOW BINLOG EVENTS [IN 'log_name' ] [FROM pos] # View log operations performed
SHOW MASTER STATUS # Explicit master binary logs
mysqlbinlog using the format:
mysqlbinlog [option] log-file1 log-file2...
Common options:
-d, - Database = name: just view the log operation specified database -o, - offset = #: ignore the first n operations command log -r, - the Result-File = name: the output log information output to the specified file, too, can use redirection. -s, - Short - form: simple display format of the log records only some general statements will omit some additional information such as location and time information as well as log-based lines. Can be used to debug, production must not use the --set-charset = char_name: when output information to a log file, the first line of the file with names char_name SET --start-datetime, - STOP- datetime: Specifies the output start time and the end of all log information within the time --start-position # =, - = position-STOP #: all log information within the specified output start position and end position -v, -vv: display more detailed information based row of logs by default not show up, this time can be viewed using the -v or -vv
Use mysql-bin.000001日志进行查看 mysqlbinlog command, as follows:
[root@lgh3 mysql]# mysqlbinlog /data/binlog/mysql-bin.000001 /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #190908 21:18:32 server id 1000 end_log_pos 123 CRC32 0x052f8ffd Start: binlog v 4, server v 5.7.24-log created 190908 21:18:32 at startup ROLLBACK/*!*/; BINLOG ' aKh1XQ/oAwAAdwAAAHsAAAAAAAQANS43LjI0LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAABoqHVdEzgNAAgAEgAEBAQEEgAAXwAEGggAAAAICAgCAAAACgoKKioAEjQA Af2PLwU= '/*!*/; # at 123 #190908 21:18:32 server id 1000 end_log_pos 154 CRC32 0x927edf95 Previous-GTIDs # [empty] # at 154 #190908 21:19:15 server id 1000 end_log_pos 219 CRC32 0xb05f419c Anonymous_GTID last_committed=0 sequence_number=1 rbr_only=no SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/; # at 219 #190908 21:19:15 server id 1000 end_log_pos 313 CRC32 0xaafce462 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1567991955/*!*/; SET @@session.pseudo_thread_id=2/*!*/; SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/; SET @@session.sql_mode=1075838976/*!*/; SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; /*!\C utf8mb4 *//*!*/; SET @@session.character_set_client=45,@@session.collation_connection=45,@@session.collation_server=45/*!*/; SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create database test /*!*/; # at 313 #190908 21:19:53 server id 1000 end_log_pos 378 CRC32 0xdccfd8ec Anonymous_GTID last_committed=1 sequence_number=2 rbr_only=no SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/; # at 378 #190908 21:19:53 server id 1000 end_log_pos 497 CRC32 0x57a45727 Query thread_id=2 exec_time=0 error_code=0 use `test`/*!*/; SET TIMESTAMP=1567991993/*!*/; create table gxt1(id int , name varchar(128)) /*!*/; # at 497 #190908 21:20:17 server id 1000 end_log_pos 562 CRC32 0x316b41e9 Anonymous_GTID last_committed=2 sequence_number=3 rbr_only=yes /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/; # at 562 #190908 21:20:17 server id 1000 end_log_pos 634 CRC32 0x693ee549 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1567992017/*!*/; BEGIN /*!*/; # at 634 #190908 21:20:17 server id 1000 end_log_pos 684 CRC32 0xde906285 Table_map: `test`.`gxt1` mapped to number 108 # at 684 #190908 21:20:17 server id 1000 end_log_pos 734 CRC32 0xa0fc1d34 Write_rows: table id 108 flags: STMT_END_F BINLOG ' 0ah1XRPoAwAAMgAAAKwCAAAAAGwAAAAAAAEABHRlc3QABGd4dDEAAgMPAgACA4VikN4= 0ah1XR7oAwAAMgAAAN4CAAAAAGwAAAAAAAEAAgAC//wBAAAACAB6aGFuZ3NhbjQd/KA= '/*!*/; # at 734 #190908 21:20:17 server id 1000 end_log_pos 765 CRC32 0xc9208948 Xid = 10 COMMIT/*!*/; # at 765 #190908 21:30:18 server id 1000 end_log_pos 812 CRC32 0xda645def Rotate to mysql-bin.000002 pos: 4 SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/; DELIMITER ; # End of log file /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
Above blue font indicates the position of the red represents time.
SHOW BINARY LOGS ;
SHOW MASTER LOGS ;
show binlog events in 'mysql-bin.000001';
SHOW MASTER STATUS;
Represent a binary file it has been recorded in the fourth, position 418.
1.5.3, delete the binary files
1.reset master will delete all the logs and let the log file restarts from 000001.
mysql> reset master;
{BINARY 2.PURGE | MASTER LOGS} {the TO 'log_name' |} the BEFORE datetime_expr
purge Master logs to "binlog_name.00000X" will clear all log files before 00000X. Such as log files before deleting 000006.
MySQL> Master logs to purge "MySQL-bin.000006";
MySQL> binary logs to purge "MySQL-bin.000006";
Master and binary synonyms
purge master logs before 'yyyy-mm-dd hh: mi: ss' will remove all the log before the specified date. However, if the specified time is in use in a log file, you will not be able to purge.
3. Use --expire_logs_days = N option to specify how many days the log automatically expire cleared
1.5.4, the relevant parameters
log-bin = on | off | file_name # -bin or path specified = log / Data / the binlog / MySQL- bin opens and route setting, msyql5.7 turn this parameter must be assigned server_id, or mysql not start the server_id = 1000 # specify server_id max_binlog_size: # (default 1G) specifies the maximum value of a binary file, if more than this value, generate a new binary log file (taking into account the transaction can not span two binlog file, so sometimes the actual size of the file is not binlog will be equal to the set value) binlog_cache_size: # (default 32k) stores all the binary logging is not submitted for caching, when the transaction is submitted, it will be written to the binary log file, if this parameter is too small, probably) store all binary logging is not submitted for caching, when the transaction is submitted, it will be written to the binary log file, if this parameter is too small, may # is written to disk, affect performance, you can view the parameters binlog_cache_disk_use, represents the number of times the disk write sync_binlog: # By default, the binary log write will not always write binary Log file (it will be appreciated that the buffer) If a database is down, it will lack the presence of binary log files recorded sync_binlog parameters can be configured = [N], when N = representation synchronous write disk 1 write binary file, which is not using the operating system to write the log buffer. #N = 0 (default), although the set sync_binlog =1 theoretically ensures synchronization, but when innodb storage engine, there will be a binary recording operation, but the transaction is rolled back by the circumstances, # parameters can be innodb_support_xa = 1 to solve the problem, = innodb_support_xa 1 ensures binary file storage and file synchronization engine binlog_do - db: # records in the database binlog_ingnore - db: # ignore the database log_slave - Update: # If the slave role in the current database replication, will not get to the binary log from the master to write their own binary file, if required, you need to open # this parameter, if set up Master => slave => schema replication slave, you must turn on this parameter binlog_format: # can be set to a value #statement: binary log files are logic sql statement recorded #row: table row changes recorded case, if you set this parameter, you can innodb transaction isolation level of basic storage engine is set to READ COMMITTED, better concurrency, generally set row, although the disk and network overhead larger format to binary format. Use mysqlbinlog tool to view #mixed: mixed mode, use the default configuration statement
expire_logs_days: # specified time automatically deleted binary log, that log expiration time
log_bin_index: # mysql-bin.index specify the path to the file
max_binlog_cache_size: # specify the maximum size of the binary log cache, hard limit. Default 4G, big enough, it is recommended not to change
binlog_cache_use: using the cache write the number of the binary log (which is a real change in the statistics)
Binlog_cache_disk_use: use temporary files to write the number of the binary log, when the log exceeds the binlog_cache_size time will use temporary write the log file, if the variable value is not 0, then consider increasing the value of binlog_cache_size
1.5.5, the log reduction
Method 1: direct reduction
mysqlbinlog --stop-datetime = "2014-7-2 15:27:48 " /tmp/mysql-bin.000008 | mysql -u user -p password
Method two: into a first file and restore
the mysqlbinlog MySQL-bin.000001> /tmp/a.sql
MySQL -u the root password -p -e "Source / (TM)
Second, the transaction log
innodb transaction logs include redo log and undo log. redo log redo logs, to provide pre-roll operation, undo log is rolled back log, providing a rollback operation.
Reverse the process undo log is not a redo log, in fact, they are considered to restore the log:
1.redo log is usually physical log, modify the data record is the physical page, rather than a line or a few lines to modify how how, it is used to restore the physical data page after submission (to recover data pages, and can only be restored to the position of the last commit).
2.undo rows to roll back to a version. generally undo log logical logs recorded according to each recording line
2.1、redo log
mainly by the redo log redo log buffer (redo log buffer) and redo log files (redo log file) constitute. Data stored in the buffer memory when the machine is abnormal or abnormal service will result in loss of data buffering, data log files stored in the disk, the file is persistence.
Redo log buffer is written by first, and then flushed to disk operation, how to ensure that data is not lost it?
We can innodb_flush_log_at_trx_commit control parameter, the parameter has three values, namely redo log buffer and redo log files 0,1,2, prior to detailed understanding of this argument, we first understand how synchronization: to ensure that every time log can be written to the transaction log file will be called during each log in the log buffer is written to the log file in one operation fsync operating system (ie, fsync () system call). Because MySQL is working in user space, MySQL's log buffer in memory user space. To write to a log file on disk role (redo:: ib_logfile0 and ib_logfile1 documents, undo share tablespace or .ibd file), go through the middle os buffer operating system kernel space, call fsync () is the OS buffer log in to brush log file on disk. As shown below:
Then we continue to look at the significance of the value of innodb_flush_log_at_trx_commit:
When set to 1 when the transaction will be submitted to each log buffer is written to the log os buffer and call fsync () brush to the log file on disk. This way even if the system crashes will not lose any data, but because each commit are written to disk, IO's poor performance. (Recommended)
When set to 0, the transaction will not be submitted in the log buffer is written to the log os buffer, but writes per os buffer and call fsync () writes to the log file on disk. That is set to 0 (approximately) Refresh written to disk per second, when the system crashes, you lose 1 second of data.
When set to 2, and each submission are written only to os buffer, then the second is to call fsync () writes os buffer in the logs to the log file on disk.
redo log data is used for crash recovery, which is one of the important functions of the database to ensure data security, preserved innodb modify records in the table, it is also called the log file, in innodb storage engine, there are two log files by default after the new database, there will be two files named ib_logfile0 and ib_logfile1, if starting a database, two files do not exist, innodb is created according to the parameters.
Log files are used in a similar fashion cycle, when the first log file is full, it will continue to write to the second, when all log file is full, it will empty out some space in the first log file, providing write space, as shown below:
innodb log file to manage log group is a logical definition, it contains a number of log files, log file size is equal to the group size parameter set.
Innodb_log_files_group decision parameters, the default value is 2, each group decides how many log files.
Parameters innodb_log_group_home_dir, the default text of the mysql database directory, decided to catalog the log file is stored.
Each log file group has its own format, but also inside the same size page segmentation, where the page size is 512 bytes, the size of the mechanical hard block. This is designed to improve the throughput and efficiency of data writing. The logical transaction logs and database dispersing random write write order is converted to an integer multiple of 512 bytes, which greatly improves writing performance database, which is an important factor in a database management system different from the file management system.
Since that each log there are many 512-byte blocks, each log file our understanding of the configuration, as shown below:
As shown above, a log file contains a number of 512-byte blocks, and each block is divided into three parts, namely the head for the log, the log tail, and the specific contents of the log. Wherein the head occupies 12 bytes of the log, the log tail occupies 8 bytes, the remaining 492 bytes occupied by the log contents.
Part 4 log header comprising:
log_block_hdr_no: (4 bytes) position of the log block ID in the redo log buffer.
log_block_hdr_data_len: (2 bytes) the size of the log in the log Block recorded. When the log block is filled 0x200, it represents 512 bytes.
log_block_first_rec_group: (2 bytes) of the log in a log Block start offset.
lock_block_checkpoint_no: (4 bytes) location information checkpointed
Only a portion of the log tail: log_block_trl_no, and this value is equal to the block header log_block_hdr_no
Next we look at groups of files: In the first redo log file for each group, the first 2KB record four specific sections, from 2KB started after recording log block. Will be recorded in addition to the first redo log file, the other log file log group are not recorded in this 2KB, but it will make room for that of 2KB. as follows:
We know that after the redo log configuration and its role, redo log that is what point in time refreshed from the buffer to the log file it? Then take a look at the timing of logs brush plate.
Log brush time to sell are:
1, log buffer runs out of space,
2, Master main thread background refresh once per second
3, when operating in DML, will check log space is adequate, if you have reached the preset value, then flush the log to disk
4, when the checkpoint, guarantee LSV to brush the minimum of all the pages of data into the disk has been brushes,
5, transaction commit logic, controlled by parameter innodb_flush_log_at_trx_commit
It mentioned above are called logical sequence number LSN (log sequence number) logs in innodb storage engine, LSN 8 bytes. LSN value will increase as the log is written gradually increases.
According to LSN, you can get several useful information:
Version Information 1. Data page.
2. The total amount of the write log, the log can be calculated by the amount of writing start number and the end number LSN.
3. may know the location of the checkpoint.
In fact you can also get a lot of implicit information.
LSN not only in the redo log, it is also present in the data page, data in the header of each page, there is a record of the current page fil_page_lsn final LSN value is. Comparing the LSN value LSN value of the data page and redo log in, if the LSN value of the page is less than the redo log of the LSN value, then the data lost portion, this time can be restored to the redo log by recording the redo log state when the value recorded LSN
By: mysql> show engine innodb status \ G; View
among them:
log sequence number is the current redo log (in buffer) in the LSN;
log flushed up to brush the redo log file on disk in LSN;
Pages and the flushed up to that has been brushed to LSN on disk data pages;
Last checkpoint AT is the last check point as LSN
2.2、undo log
undo log has two functions: to provide a plurality of rows and version rollback control (MVCC)
Reference: Thanks the author of the article quality
https://www.cnblogs.com/f-ck-need-u/p/9010872.html#auto_id_12
https://www.cnblogs.com/f-ck-need-u/p/9001061.html
"Mysql operation and maintenance of internal control."
"Mysql Inside: innodb storage engine."