background
In a production environment, the process of MySQL database instance is running, some of the SQL statement execution plan changes occur, leading to an increased risk of database stability here while there are several factors and scenarios, such as: With the change in the amount of table data, and automatically collect statistical information, CBO optimizer computed a lower cost plan, or if the table structure changes, additions and deletions of some index, or in the process of upgrading instances of migration, MySQL's own behavior and optimization algorithm has changed and so on. In order to be able to respond to online business planning and implementation of interventions SQL statement, AliSQL devised a method using the MySQL optimizer / index hint to stabilize the implementation of the plan, called the Statement outline, and provides a set of easy management interface (DBMS_OUTLN package), and publicly available on RDS MySQL 8.0 product.
Outline Design
AliSQL 8.0 outline MySQL 8.0 support for all officially supported hint types, divided into two categories:
- Optimizer Hint
The scope (query block) and hint objects, is divided into: Level Free Join hint, the Table / Index Level hint,
the Join Order hint like.
Details Reference: https://dev.mysql.com/doc/refman/8.0/en/optimizer-hints.html
- Index Hint
Depending on the type of main index hint (USE, FORCE, IGNORE) and scope (FOR JOIN, FOR ORDER BY ,
classification FOR GROUP BY).
Detailed syntax reference: https://dev.mysql.com/doc/refman/8.0/en/index-hints.html
To represent these abstract and Hint, and can persist outline, AliSQL 8.0 adds a system table mysql.outline, the following structure:
MYSQL.OUTLINE
CREATE TABLE `mysql`.`outline` (
`Id` bigint(20) NOT NULL AUTO_INCREMENT,
`Schema_name` varchar(64) COLLATE utf8_bin DEFAULT NULL,
`Digest` varchar(64) COLLATE utf8_bin NOT NULL,
`Digest_text` longtext COLLATE utf8_bin,
`Type` enum('IGNORE INDEX','USE INDEX','FORCE INDEX','OPTIMIZER') CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`Scope` enum('','FOR JOIN','FOR ORDER BY','FOR GROUP BY') CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT '',
`State` enum('N','Y') CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL DEFAULT 'Y',
`Position` bigint(20) NOT NULL,
`Hint` text COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`Id`)
) /*!50100 TABLESPACE `mysql` */ ENGINE=InnoDB
DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0 COMMENT='Statement outline'Columns
Digest/Digest_text
Outline matching the feature statement, this feature is Digest text, the hash calculation according to this Digest Text obtain
a 64-byte hash string. E.g:
Statement query: select * from t1 where id = 1
根据计算得到的Digest 和 Digest text 分别是：
Digest ： 36bebc61fce7e32b93926aec3fdd790dad5d895107e2d8d3848d1c60b74bcde6
Digest_text: SELECT * FROM `t1` WHERE `id` = ?
When the statement is complete parse, based on [schema + digest] as a hash key, matching query Outline.
Type
All optimizer hint for the unity of type OPTIMIZER
Index hint divided into three categories, namely:
- USE INDEX
- FORCE INDEX
- IGNORE INDEX
Scope
scope only for the Index hint, it is divided into four categories:
- FOR GROUP BY
- FOR ORDER BY
- FOR JOIN
If the empty string ALL
Position
Position is critical:
Optimizer hint in, position represents Query Block, because all optimizer hint must be applied to the Query Block,
where a determination is relatively simple, because the Optimizer hint only supports these types of keywords:
SELECT /*+ ... */ ...
INSERT /*+ ... */ ...
REPLACE /*+ ... */ ...
UPDATE /*+ ... */ ...
DELETE /*+ ... */ ...So, position from the beginning, hint role in the first few keywords anchor statement, is a few.
Index hint in, position represents the table position, but also from the beginning, hint role in the first few table anchor, is a few.
Hint
In the Index hint, the index shown here is a list of names, such as "ind_1, ind_2"
in the Optimizer hint, represented here is the complete hint string, such as: "/ + max_execution_time (1000) /"
The user interface
In order to facilitate the management Statement outline, AliSQL a DBMS_OUTLN package designed to manage, and offers five native procedure interfaces:
DBMS_OUTLN.add_index_outline(); 增加 index hint
DBMS_OUTLN.add_optimizer_outline(); 增加 optimizer hint
DBMS_OUTLN.preview_outline(); 预览某一个 SQL 语句命中 outline 的情况
DBMS_OUTLN.show_outline(); 展示内存中可用的所有 outline 及命中情况
DBMS_OUTLN.del_outline(); 删除内存和持久化表中的 outline
DBMS_OUTLN.flush_outline(); 刷新所有的 outline，从 mysql.outline 表中重新 loadIn order to facilitate the introduction of the use of DBMS_OUTLN, here using some test table:
CREATE TABLE `t1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`col1` int(11) DEFAULT NULL,
`col2` varchar(100) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `ind_1` (`col1`),
KEY `ind_2` (`col2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `t2` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`col1` int(11) DEFAULT NULL,
`col2` varchar(100) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `ind_1` (`col1`),
KEY `ind_2` (`col2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ADD_INDEX_OUTLINE
Syntax and parameters
CALL DBMS_OUTLN.add_index_outline(schema=>, digest=>, position=>, type=>,
scope=>, hint=>, query=>);
说明：
digest 和 query 可以选择其一， 如果填写了原始query语句，这个 proc 会计算 digest 和 digest text。Test case 1
Testing Statement
select * from t1 where t1.col1 =1 and t1.col2 ='xpchild';
Use the index ind_1
call dbms_outln.add_index_outline('outline_db', '', 1, 'USE INDEX', 'ind_1', '',
"select * from t1 where t1.col1 =1 and t1.col2 ='xpchild'");
查看 outline：
mysql> call dbms_outln.show_outline();
+------+------------+------------------------------------------------------------------+-----------+-------+------+-------+------+----------+------------------------------------------------------------------+
| ID | SCHEMA | DIGEST | TYPE | SCOPE | POS | HINT | HIT | OVERFLOW | DIGEST_TEXT |
+------+------------+------------------------------------------------------------------+-----------+-------+------+-------+------+----------+------------------------------------------------------------------+
| 30 | outline_db | b4369611be7ab2d27c85897632576a04bc08f50b928a1d735b62d0a140628c4c | USE INDEX | | 1 | ind_1 | 0 | 0 | SELECT * FROM `t1` WHERE `t1` . `col1` = ? AND `t1` . `col2` = ? |
+------+------------+------------------------------------------------------------------+-----------+-------+------+-------+------+----------+------------------------------------------------------------------+
1 row in set (0.00 sec)Verify Outline:
Verify that the Outline effect, there are two ways:
- dbms_outln.preview_outline () preview
- Directly explain viewing.
mysql> call dbms_outln.preview_outline('outline_db', "select * from t1 where t1.col1 =1 and t1.col2 ='xpchild'");
+------------+------------------------------------------------------------------+------------+------------+-------+---------------------+
| SCHEMA | DIGEST | BLOCK_TYPE | BLOCK_NAME | BLOCK | HINT |
+------------+------------------------------------------------------------------+------------+------------+-------+---------------------+
| outline_db | b4369611be7ab2d27c85897632576a04bc08f50b928a1d735b62d0a140628c4c | TABLE | t1 | 1 | USE INDEX (`ind_1`) |
+------------+------------------------------------------------------------------+------------+------------+-------+---------------------+
1 row in set (0.01 sec)
mysql> explain select * from t1 where t1.col1 =1 and t1.col2 ='xpchild';
+----+-------------+-------+------------+------+---------------+-------+---------+-------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+-------+---------+-------+------+----------+-------------+
| 1 | SIMPLE | t1 | NULL | ref | ind_1 | ind_1 | 5 | const | 1 | 100.00 | Using where |
+----+-------------+-------+------------+------+---------------+-------+---------+-------+------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
mysql> show warnings;
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Note | 1003 | /* select#1 */ select `outline_db`.`t1`.`id` AS `id`,`outline_db`.`t1`.`col1` AS `col1`,`outline_db`.`t1`.`col2` AS `col2` from `outline_db`.`t1` USE INDEX (`ind_1`) where ((`outline_db`.`t1`.`col1` = 1) and (`outline_db`.`t1`.`col2` = 'xpchild')) |
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)Test case 2
Testing statement:
select * from t1, t2 where t1.col1 = t2.col1 and t2.col2 ='xpchild'Testing using ind_2 index t2 table:
call dbms_outln.add_index_outline('outline_db', '', 2, 'USE INDEX', 'ind_2', '',
"select * from t1, t2 where t1.col1 = t2.col1 and t2.col2 ='xpchild'");
Verify Outline:
mysql> explain select * from t1, t2 where t1.col1 = t2.col1 and t2.col2 ='xpchild';
+----+-------------+-------+------------+------+---------------+-------+---------+-------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+-------+---------+-------+------+----------+-------------+
| 1 | SIMPLE | t1 | NULL | ALL | ind_1 | NULL | NULL | NULL | 1 | 100.00 | NULL |
| 1 | SIMPLE | t2 | NULL | ref | ind_2 | ind_2 | 303 | const | 1 | 100.00 | Using where |
+----+-------------+-------+------------+------+---------------+-------+---------+-------+------+----------+-------------+
2 rows in set, 1 warning (0.01 sec)
mysql> show warnings;
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Note | 1003 | /* select#1 */ select `outline_db`.`t1`.`id` AS `id`,`outline_db`.`t1`.`col1` AS `col1`,`outline_db`.`t1`.`col2` AS `col2`,`outline_db`.`t2`.`id` AS `id`,`outline_db`.`t2`.`col1` AS `col1`,`outline_db`.`t2`.`col2` AS `col2` from `outline_db`.`t1` join `outline_db`.`t2` USE INDEX (`ind_2`) where ((`outline_db`.`t2`.`col1` = `outline_db`.`t1`.`col1`) and (`outline_db`.`t2`.`col2` = 'xpchild')) |
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)ADD_OPTIMIZER_OUTLINE
Syntax and parameters
CALL DBMS_OUTLN.add_optimizer_outline(schema=>, digest=>, query_block=>
hint=>, query=>);
说明：digest 和 query 同样可以填其一，或者都填入。proc 会自动计算digest 和 digest text。Test case 1
Increase global MAX_EXECUTION_TIME / SET VAR optimizer hint;
CALL DBMS_OUTLN.add_optimizer_outline("outline_db", '', 1, '/*+ MAX_EXECUTION_TIME(1000) */',
"select * from t1 where id = 1");
CALL DBMS_OUTLN.add_optimizer_outline("outline_db", '', 1, '/*+ SET_VAR(foreign_key_checks=OFF) */',
"select * from t1 where id = 1");Verify Outline:
mysql> explain select * from t1 where id = 1;
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+--------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+--------------------------------+
| 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | no matching row in const table |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+--------------------------------+
1 row in set, 1 warning (0.01 sec)
mysql> show warnings;
+-------+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+-------+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Note | 1003 | /* select#1 */ select /*+ MAX_EXECUTION_TIME(1000) SET_VAR(foreign_key_checks='OFF') */ NULL AS `id`,NULL AS `col1`,NULL AS `col2` from `outline_db`.`t1` where multiple equal(1, NULL) |
+-------+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)Test case 2
Multi-table queries related tests: Nested-Loop join processing
CALL DBMS_OUTLN.add_optimizer_outline('outline_db', '', 1, '/*+ BNL(t1,t2) */',
"select t1.id, t2.id from t1,t2");Verify Outline:
mysql> explain select t1.id, t2.id from t1,t2;
+----+-------------+-------+------------+-------+---------------+-------+---------+------+------+----------+----------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+-------+---------+------+------+----------+----------------------------------------------------+
| 1 | SIMPLE | t1 | NULL | index | NULL | ind_1 | 5 | NULL | 1 | 100.00 | Using index |
| 1 | SIMPLE | t2 | NULL | index | NULL | ind_1 | 5 | NULL | 1 | 100.00 | Using index; Using join buffer (Block Nested Loop) |
+----+-------------+-------+------------+-------+---------------+-------+---------+------+------+----------+----------------------------------------------------+
2 rows in set, 1 warning (0.01 sec)
mysql> show warnings;
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Note | 1003 | /* select#1 */ select /*+ BNL(`t1`@`select#1`) BNL(`t2`@`select#1`) */ `outline_db`.`t1`.`id` AS `id`,`outline_db`.`t2`.`id` AS `id` from `outline_db`.`t1` join `outline_db`.`t2` |
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)Test case 3
In the case of the test subquery with the query block name
CALL DBMS_OUTLN.add_optimizer_outline('outline_db', '', 2, ' /*+ QB_NAME(subq1) */',
"SELECT * FROM t1 WHERE t1.col1 IN (SELECT col1 FROM t2)");
CALL DBMS_OUTLN.add_optimizer_outline('outline_db', '', 1, '/*+ SEMIJOIN(@subq1 MATERIALIZATION, DUPSWEEDOUT) */ ',
"SELECT * FROM t1 WHERE t1.col1 IN (SELECT col1 FROM t2)");
Verify Outline:
mysql> explain SELECT * FROM t1 WHERE t1.col1 IN (SELECT col1 FROM t2);
+----+--------------+-------------+------------+--------+---------------+------------+---------+--------------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+-------------+------------+--------+---------------+------------+---------+--------------------+------+----------+-------------+
| 1 | SIMPLE | t1 | NULL | ALL | ind_1 | NULL | NULL | NULL | 1 | 100.00 | Using where |
| 1 | SIMPLE | <subquery2> | NULL | eq_ref | <auto_key> | <auto_key> | 5 | outline_db.t1.col1 | 1 | 100.00 | NULL |
| 2 | MATERIALIZED | t2 | NULL | index | ind_1 | ind_1 | 5 | NULL | 1 | 100.00 | Using index |
+----+--------------+-------------+------------+--------+---------------+------------+---------+--------------------+------+----------+-------------+
3 rows in set, 1 warning (0.00 sec)
mysql> show warnings;
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Note | 1003 | /* select#1 */ select /*+ SEMIJOIN(@`subq1` MATERIALIZATION, DUPSWEEDOUT) */ `outline_db`.`t1`.`id` AS `id`,`outline_db`.`t1`.`col1` AS `col1`,`outline_db`.`t1`.`col2` AS `col2` from `outline_db`.`t1` semi join (`outline_db`.`t2`) where (`<subquery2>`.`col1` = `outline_db`.`t1`.`col1`) |
+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)PREVIEW_OUTLINE
dbms_outln.preview_outline () for using the specific SQL statement, Outline View Matching for manual verification.
The syntax and parameters:
CALL DBMS_OUTLN.preview_outline(schema=>, query=>);E.g:
mysql> call dbms_outln.preview_outline('outline_db', "select * from t1 where t1.col1 =1 and t1.col2 ='xpchild'");
+------------+------------------------------------------------------------------+------------+------------+-------+---------------------+
| SCHEMA | DIGEST | BLOCK_TYPE | BLOCK_NAME | BLOCK | HINT |
+------------+------------------------------------------------------------------+------------+------------+-------+---------------------+
| outline_db | b4369611be7ab2d27c85897632576a04bc08f50b928a1d735b62d0a140628c4c | TABLE | t1 | 1 | USE INDEX (`ind_1`) |
+------------+------------------------------------------------------------------+------------+------------+-------+---------------------+
1 row in set (0.00 sec)SHOW_OUTLINE
dbms_outln.show_outline outline showing where hit in the cache memory, there are two fields inside:
the HIT: The outline showing the number of hits
OVERFLOW: Description hint the outline query block is not found or the number of times corresponding table
E.g:
mysql> call dbms_outln.show_outline();
+------+------------+------------------------------------------------------------------+-----------+-------+------+-------------------------------------------------------+------+----------+-------------------------------------------------------------------------------------+
| ID | SCHEMA | DIGEST | TYPE | SCOPE | POS | HINT | HIT | OVERFLOW | DIGEST_TEXT |
+------+------------+------------------------------------------------------------------+-----------+-------+------+-------------------------------------------------------+------+----------+-------------------------------------------------------------------------------------+
| 33 | outline_db | 36bebc61fce7e32b93926aec3fdd790dad5d895107e2d8d3848d1c60b74bcde6 | OPTIMIZER | | 1 | /*+ SET_VAR(foreign_key_checks=OFF) */ | 1 | 0 | SELECT * FROM `t1` WHERE `id` = ? |
| 32 | outline_db | 36bebc61fce7e32b93926aec3fdd790dad5d895107e2d8d3848d1c60b74bcde6 | OPTIMIZER | | 1 | /*+ MAX_EXECUTION_TIME(1000) */ | 2 | 0 | SELECT * FROM `t1` WHERE `id` = ? |
| 34 | outline_db | d4dcef634a4a664518e5fb8a21c6ce9b79fccb44b773e86431eb67840975b649 | OPTIMIZER | | 1 | /*+ BNL(t1,t2) */ | 1 | 0 | SELECT `t1` . `id` , `t2` . `id` FROM `t1` , `t2` |
| 35 | outline_db | 5a726a609b6fbfb76bb8f9d2a24af913a2b9d07f015f2ee1f6f2d12dfad72e6f | OPTIMIZER | | 2 | /*+ QB_NAME(subq1) */ | 2 | 0 | SELECT * FROM `t1` WHERE `t1` . `col1` IN ( SELECT `col1` FROM `t2` ) |
| 36 | outline_db | 5a726a609b6fbfb76bb8f9d2a24af913a2b9d07f015f2ee1f6f2d12dfad72e6f | OPTIMIZER | | 1 | /*+ SEMIJOIN(@subq1 MATERIALIZATION, DUPSWEEDOUT) */ | 2 | 0 | SELECT * FROM `t1` WHERE `t1` . `col1` IN ( SELECT `col1` FROM `t2` ) |
| 30 | outline_db | b4369611be7ab2d27c85897632576a04bc08f50b928a1d735b62d0a140628c4c | USE INDEX | | 1 | ind_1 | 3 | 0 | SELECT * FROM `t1` WHERE `t1` . `col1` = ? AND `t1` . `col2` = ? |
| 31 | outline_db | 33c71541754093f78a1f2108795cfb45f8b15ec5d6bff76884f4461fb7f33419 | USE INDEX | | 2 | ind_2 | 1 | 0 | SELECT * FROM `t1` , `t2` WHERE `t1` . `col1` = `t2` . `col1` AND `t2` . `col2` = ? |
+------+------------+------------------------------------------------------------------+-----------+-------+------+-------------------------------------------------------+------+----------+-------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)DEL_OUTLINE
dbms_outln.del_outline () can remove the memory and the table a certain outline.
Syntax and parameters are as follows:
CALL DBMS_OUTLN.del_outline(outline_id=>);E.g:
mysql> call dbms_outln.del_outline(1000);
Query OK, 0 rows affected, 2 warnings (0.00 sec)
mysql> show warnings;
+---------+------+----------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------+
| Warning | 7521 | Statement outline 1000 is not found in table |
| Warning | 7521 | Statement outline 1000 is not found in cache |
+---------+------+----------------------------------------------+
2 rows in set (0.00 sec)FLUSH_OUTLINE
dbms_outln.flush_outline () to support clean-up cache in outline, and re-load from mysql.outline table.
If the user to directly modify the table to load the outline, you need to call to flush the cache.
E.g:
mysql> call dbms_outln.flush_outline();
Query OK, 0 rows affected (0.01 sec)