Outline
Preface
A MySQL bottleneck and countermeasures
Two specific mysql optimization parameters
Three mysql optimization practice
Four summary
references
Preface
1 Firstly, locate performance bottlenecks command Show
We can view the status of MySQL and variables show command, find the system bottleneck:
Mysql> show status - displaying status information (extended show status like 'XXX')
Mysql> show variables - System Variables (extended show variables like 'XXX')
Mysql> show innodb status - displays the status of the InnoDB storage engine
Mysql> show processlist - view the current SQL implementation, including the implementation of the state, such as whether the lock table
Shell> mysqladmin variables -u username -p password-- System Variables
Shell> mysqladmin extended-status -u username -p password-- display status information
View state variables and help:
Shell> mysqld –verbose –help
2 Turn on slow query log locate those slow sql
3 In fact, the best way is feeling the boss can not accept slow response must be optimized.
A MySQL bottleneck and countermeasures
MySQL is the bottleneck. When a single table MySQL data volume reached more than ten million level, in any case of MySQL optimization, how simple query, MySQL's performance will be significantly reduced. Take measures:
1) increase the value of MySQL configuration buffer and Cache, increasing the number of CPU and memory size of the server, this will largely deal with MySQL performance bottlenecks.
Performance optimization, the most significant effect, was undoubtedly the lowest cost hardware and server optimization, so this should be the first consideration.
2) Use a third-party engine or derived version.
As Percona in functionality and performance than MySQL has significantly improved; developed by MySQL founder of the free MariaDB performance on the InnoDB engine is also better than MySQL;
According to the official website, by number, another improved product TokuDB, performance of MySQL more than 10 times. These derivative version or a modified version of MySQL are mainly for MySQL InnoDB
engine. When InnoDB each submission of things, in order to ensure that the data has been persisted to disk (Durable), you need to call once told fsync file system, the data in the cache may be
flushed to the times, oh press. The fsync operation itself is very "expensive" in (to consume more I / O resources, slow response), if every time things do fsync submit a separate operation, this
will be a bottleneck in the system of TPS, so there Group Commit the concept. After MySQL 5.0, in consideration of a distributed architecture, in order to ensure that the internal InnoDB Commit
order and MySQL logs consistent, InnoDB forced to abandon support Group Commit. After that, it has not been a good solution. Until MariaDB appears, is rather perfect solution to
this problem.
3) migrate to other databases.
Post as open source and commercial heating SQL Oracle. Compared with Oracle and PostgreSQL, MySQL belongs to the threading model, and uses a plug-in engine architecture. This implementation
does have its own advantages: lightweight and fast, small consumption of system resources, support more concurrent connections, but the process model can more fully use CPU resources, an advantage in dealing with the complex business queries. For example,
under the premise of conventional optimization, Oracle, but do not experience performance bottlenecks at the level of 200 million the amount of data, far superior to MySQL. Moreover, in the relational query and built-in functions and other functions,
the Oracle database is MySQL victory. As another example, PostgreSQL database compared to MySQL, has a more powerful query optimizer will not rebuild indexes frequently, support for materialized views and other
advantages. Of course, migrate to other databases depends on the type of application, such as a biased application of OLTP (On-Line Transaction Processing, online transaction processing) or bias
OLAP (On-Line Analytical Processing, Online Analytical Processing) applications.
4) partitioning a database, sub-table, to reduce the volume of a single table.
5) Use NoSQL other auxiliary solutions such as Memcached, Redis.
6) use middleware for data split and distributed deployment, a typical case in this area have a database of open source middleware Alibaba Foreign Cobar.
7) database connection pool.
At 3:00, we have said, MySQL is the threading model, can support more concurrent connections. MySQL can support more than Oracle and PostgreSQL connection. So Oracle
And PostgreSQL applications typically multiplexed connection database connection pool. So why the need to use the MySQL database application of these common database connection pool technology? The reason is that
the MySQL mechanism is still not perfect, but some of the problems in the program are likely to lead MySQL database connection blocking, in large concurrent case, it will waste a lot of resources and repeatedly connected connection
consumption. Use the database connection pool, so that connections queuing and multiplexing, can ease the pressure of high concurrent connection to a certain extent.
MySQL bottleneck is real, but many large Internet companies are still using MySQL, and can be used well, this is because the company's technical strength is sufficient for MySQL secondary development and improvement, on the other hand have benefit from its sophisticated architecture. So, a good tool can not be used, the human factor accounts for a large proportion.
Two specific mysql optimization practice
The following first universal Gb GB and the difference
GB 0.1342 (gigabytes)
1.074 Gb (Gigabits)
of memory generally will not say Gb, only GB.
Here, in fact, is the difference between the famous Grand B and b are small.
Generally, the newsletter says, it is a small b, that is, bit, bit mean.
A byte is eight bits, called Byte, is the big B.
We generally say broadband, 4M, is a 4Mb, in addition to the 8, is the general use of computer B, the download speed is about 400KB.
Memory capacity does not say Gb, the bandwidth will be involved Gb.
Basically, b is the transmission rate will be related to the concept.
Summarizes size measurement capacity GB, the Gb size transmission rate communication;
1 transaction_isolation
Interpretation: transaction isolation levels, Oracle, SQL Server and other commercial database of well-known default level is READ-COMMITTED, while MySQL default is REPEATABLE-READ, which uses its own proprietary Gap Lock solve the "phantom read." But also because Gap Lock's sake, compared to the READ-COMMITTED level Record Lock, REPEATABLE-READ transaction concurrent insert performance is greatly limited.
Setting: choice of isolation level depends on the actual business requirements (safety and performance trade-offs), if not financial, telecommunications and other high-level requirements of business affairs, can set transaction_isolation = READ-COMMITTED.
2 innodb_buffer_pool_size
Interpretation: InnoDB buffer pool size, which determines how much data and indexes MySQL can be cached in memory, but not always read from the disk. We all know that Redis read and write very fast, the most important reason is that it's all data cached in memory. Imagine if innodb_buffer_pool_size large enough, MySQL all tables and indexes data can be cached, it would be kind of what experience?
Settings: If you are a dedicated MySQL server, usually set to about 75% of the operating system memory, but at least 2G of memory reserved for the operating system maintenance and MySQL exception processing.
3 innodb_buffer_pool_instances
Interpretation: InnoDB buffer pool number of instances, InnoDB buffer pool is managed by a whole linked list page (segment cluster, p) due to the presence of the mutex (the list of protected page), under high concurrent transactions, read and write pages on the need to lock the competition and waiting. By providing innodb_buffer_pool_instances, the entire list is divided into a plurality, each pool instance management pages and their mutually exclusive, thereby improving efficiency.
Set: If the pool is relatively large (8G above), can be set in accordance with innodb_buffer_pool_size / innodb_buffer_pool_instances = 1G, particularly if the buffer pool (32G or more), may be divided for each instance of 2 ~ 3G, the more the number of instances is not well, on behalf of multi-instance multi-threading, thread overhead (CPU, MEM) have to be considered.
4 innodb_log_file_size
Interpretation: InnoDB log file size (Redo Log), it matters to modify records stored in InnoDB tables ib_logfile0, ib_logfile1 in. innodb_log_file_size larger, dirty data buffer pool checkpoint is needed (the checkpoint) for a frequency less brush plate, thereby reducing disk IO to reduce the peak loads caused by high concurrent. However, the log file is not larger appointment, since the dirty data in the memory to reduce the frequency of the brush plate, once the abnormal database crash, when reading data from the database restart innodb_log_file performed longer recovery time.
Provided: General select the amount of log traffic peak of one hour as the standard, is calculated as follows:
# Command
MySQL-uroot--h172.26.96.146 -p -e 'Show Engine InnoDB Status \ G' | grep 'the Log Sequence Number The' \
&& SLEEP 60 \
&& MySQL-uroot--h172.26.96.146 -p -e ' Show Engine InnoDB Status \ G '| grep' the Log Sequence Number '
# output of
the Log Sequence Number 149 949 388 055
the Log Sequence Number 149 959 622 102
# is calculated
(149 959 622 102 - 149 949 388 055) / 1024/1024 = 1OM
10/60 * 3600 = 600M
600/2 = 300M
# Dictionary
log sequence number representative of the write operation so far InnoDB total number of bytes of the log, the thread between the two print sleep 60 seconds
to give the total amount of transaction log records within one minute 10M, converted to an hour total 600M
because ` ib_logfile0, ib_logfile1` two cycles to write the file, a file for the 300M
final, innodb_log_file_size = 300M
5 innodb_flush_log_at_trx_commit
Interpretation: InnoDB transaction log opportunity brush plate
When 0, the transaction commits to log buffer, a background thread Write every second of the buffer written to the system log buffer, the timing of actual physical log file is written depends on the operating system.
When 1, the transaction commits to log buffer, Master thread synchronization will be written to the physical log buffer log files directly, which is fully in line with the standard InnoDB ACID transactions, data is not lost.
When 2, the transaction submitted to the system buffer, Master thread every second buffer system log file written to the physical log.
: Security 1> 2> 0, speed 0> 2> 1, according to the actual needs of the business (safety and speed tradeoffs) selected brush disc a reasonable time.
6 innodb_file_per_table
Interpretation: InnoDB separate tablespaces, innodb_file_per_table = ON represent each table in a separate physical file (.ibd) for storing data and indexes, innodb_file_per_table = OFF means that all tables are shared space table i.e. a physical file (ibdata1). If by drop / truncate table operation, the physical storage will be a separate table space is immediately recovered (deletion / initialization), while the share table space will not be recovered and would only have been increased.
Provided: innodb_file_per_table = ON, but it should be noted that a separate table and index space only stores data, such as rolling log buffer (Undo Log), insert the index buffer (Insert Buffer), the second write buffer (Doublewrite Buffer) or the like discharge in the shared table space.
query_cache_size
Interpretation: query cache size, it is in order to track the data table does not change, this query before hitting query, so skip the parsing, optimization, execution stage, direct return data in the buffer pool. In actual OLTP systems, query cache hit rarely (if small changes in the frequency of data in the database), once the data has changed since the cache invalidation. And because query cache keeps track of all changes in the table, it will become a bottleneck for the entire database (resource competition point).
Setting: query_cache_size = 0, in conjunction with set query_cache_type = 0, MySQL5.7.20 above, MySQL8.0 will abandon all direct query cache configuration items.
max_connections
Interpretation: The maximum number of connections, when set too small max_connections (default 151), MySQL may be error Too many connections. When max_connections is set too large (more than 1000), the operating system may be busy switching between threads and unresponsive.
Provided: each connection will consume some memory, is calculated as follows:
Three went on to say mysql 5.7 Performance Optimization combat
Adjust Innodb_Buffer_Pool_size size must restart mysql process before they can take effect, now in MySQL5.7, the direct dynamic setting, a lot of convenience.
This function application scenarios:
- A large amount of data all data queries, need to be tuned.
- B machine to increase memory, DBA careless forget to turn up the Innodb_Buffer_Pool_size.
- C transfer of work, new to the DBA found Innodb_Buffer_Pool_size unreasonable former DBA settings.
The needs of local attention, during the adjustment Buffer_Pool, user requests will be blocked until the adjustment is completed, so please do not adjust during the day, adjust the low peak point in the morning 3-4.
When adjusted, the internal data page is moved to a new location, in units of blocks. If you want to increase the speed of movement, innodb_buffer_pool_chunk_size need to adjust the size of the parameter, the default is 128M.
Example (increase of the BP 128M 384M):
mysql> SELECT @@innodb_buffer_pool_size;
+---------------------------+
| @@innodb_buffer_pool_size |
+---------------------------+
| 134217728 |
+---------------------------+
1 row in set (0.00 sec)
mysql> SELECT @@innodb_buffer_pool_chunk_size;
+---------------------------------+
| @@innodb_buffer_pool_chunk_size |
+---------------------------------+
| 134217728 |
+---------------------------------+
1 row in set (0.00 sec)
mysql> SET GLOBAL innodb_buffer_pool_size=402653184;
Query OK, 0 rows affected (0.01 sec)
mysql> SELECT @@innodb_buffer_pool_size;
+---------------------------+
| @@innodb_buffer_pool_size |
+---------------------------+
| 402653184 |
+---------------------------+
1 row in set (0.00 sec)
innodb_buffer_pool_chunk_size size is calculated innodb_buffer_pool_size / innodb_buffer_pool_instances
For example, now is initialized innodb_buffer_pool_size 2G, innodb_buffer_pool_instances Examples 4, innodb_buffer_pool_chunk_size set to 1G, then automatically adjusting the innodb_buffer_pool_chunk_size 1G 512M, Example:
./mysqld --innodb_buffer_pool_size = 2147483648 = --innodb_buffer_pool_instances. 4
--innodb_buffer_pool_chunk_size = 1073741824;
mysql> SELECT @@innodb_buffer_pool_size;
+---------------------------+
| @@innodb_buffer_pool_size |
+---------------------------+
| 2147483648 |
+---------------------------+
1 row in set (0.00 sec)
mysql> SELECT @@innodb_buffer_pool_instances;
+--------------------------------+
| @@innodb_buffer_pool_instances |
+--------------------------------+
| 4 |
+--------------------------------+
1 row in set (0.00 sec)
# Chunk size was set to 1GB (1073741824 bytes) on startup but was
# truncated to innodb_buffer_pool_size / innodb_buffer_pool_instances
mysql> SELECT @@innodb_buffer_pool_chunk_size;
+---------------------------------+
| @@innodb_buffer_pool_chunk_size |
+---------------------------------+
| 536870912 |
+---------------------------------+
1 row in set (0.00 sec)
Monitoring Buffer Pool adjustment process
mysql> SHOW STATUS WHERE Variable_name='InnoDB_buffer_pool_resize_status';
+----------------------------------+----------------------------------+
| Variable_name | Value |
+----------------------------------+----------------------------------+
| Innodb_buffer_pool_resize_status | Resizing also other hash tables. |
+----------------------------------+----------------------------------+
1 row in set (0.00 sec)
Following commands to view alarms xing'i
MySQL> Show Represents warnings;
+ --------- + ------ + -------------------- + ----------------------------------------------
| Level | Code | the Message |
+ --------- + ------ + ---------------------------- + --------------------------------------
| Warning | 1292 | Truncated in here Incorrect innodb_buffer_pool_size value: ' 1073741824 '|
+ --------- + ------ + ----------------------------- + -------------------------------------
1 Row in the SET (0.00 sec)
appeared warned, look at actual size
MySQL> Show Variables like '%% innodb_buffer_pool_size';
+ -------- + ------------------------- + -----
| variable_name | Value |
+ ------------------------- + ------------ - +
| innodb_buffer_pool_size | 1073741824|
+-------------------------+-------------+
1 row in set (0.00 sec)
If there is a warning message, because, buffer pool size after the adjustment must meet the following conditions:
multiple innodb_buffer_pool_chunk_size * innodb_buffer_pool_instances, and that is a multiple of 128M.
Four summary
1 combat summary innodb_buffer_pool_size = 2G, innodb_buffer_pool_instances = 1 very good effect.
2 wide performance optimization, aspect relates more, according to the actual needs of the situation.
3 only index a limited effect on performance optimization.
References :
https://www.cnblogs.com/liugx/p/9935749.html
https://blog.csdn.net/nanyanglu/article/details/79109838
https://www.cnblogs.com/sandea/p/11695521.html
Ubuntu install MySQL 14.04 at http://www.linuxidc.com/Linux/2014-05/102366.htm
"MySQL The Definitive Guide (the original book version 2)" Chinese clear scan version PDF http://www.linuxidc.com/Linux/2014-03/98821.htm
Ubuntu 14.04 LTS installed Nginx LNMP \ PHP5 (PHP-FPM) \ MySQL http://www.linuxidc.com/Linux/2014-05/102351.htm
Set up under Ubuntu 14.04 from the main MySQL server http://www.linuxidc.com/Linux/2014-05/101599.htm
Ubuntu 12.04 LTS building distributed high-availability MySQL Cluster http://www.linuxidc.com/Linux/2013-11/93019.htm
Ubuntu 12.04 MySQL5.6 the source code and install-MySQLdb the Python http://www.linuxidc.com/Linux/2013-08/89270.htm
MySQL-5.5.38 Universal Binary installation http://www.linuxidc.com/Linux/2014-07/104509.htm
