Oracle日志大表查询慢，本地随机生成千万亿级数据测试不同字段加索引对DML操作带来的影响

一、背景

​ 由于线上有一张Oracle亿级日志大表无索引查询慢的现象，故在本地创建亿级表和千万级表进行加索引测试，观察加索引前后查询耗时，DML插入更新删除操作耗时

二、操作

1. 创建亿级数据表和千万级数据表，设置id为主键

• 千万级表创建

CREATE TABLE TEST_LOG_DETAIL AS
SELECT
	ROWNUM AS ID,
	SYSDATE + ROWNUM / 24 / 3600 AS INSERT_TIME,
	TRUNC(DBMS_RANDOM.VALUE(0, 100)) AS MONEY,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS USERFROM,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PRODUCT,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS TAG,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PAY_CODE,
	DBMS_RANDOM.STRING('x',
	8) USERNAME,
	DBMS_RANDOM.STRING('x',
	16) ORDER_ID,
	DBMS_RANDOM.STRING('x',
	8) ADD_FROM
FROM xmltable('1 to 15000000');

• 亿级表创建

CREATE TABLE TEST1_LOG_DETAIL AS
SELECT
	ROWNUM AS ID,
	SYSDATE + ROWNUM / 24 / 3600 AS INSERT_TIME,
	TRUNC(DBMS_RANDOM.VALUE(0, 100)) AS MONEY,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS USERFROM,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PRODUCT,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS TAG,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PAY_CODE,
	DBMS_RANDOM.STRING('x',
	8) USERNAME,
	DBMS_RANDOM.STRING('x',
	16) ORDER_ID,
	DBMS_RANDOM.STRING('x',
	8) ADD_FROM
FROM xmltable('1 to 150000000');

• 增加主键

ALTER TABLE TEST_LOG_DETAIL
ADD CONSTRAINT TEST_LOG_DETAIL_PK PRIMARY KEY (ID)
ENABLE;

2. 测试DML前准备工作，设置主键自增，避免id主键插入失败

• 增加主键自增函数（设置开始自增数从id尾部开始自增）

CREATE SEQUENCE TEST_ID_SEQ
INCREMENT BY 1
START WITH 15000001
MAXVALUE 999999999
NOCYCLE
NOCACHE;

• 创建主键自增触发器（根据不同表修改）

CREATE OR REPLACE TRIGGER TEST_ID_SEQ_TRG
BEFORE INSERT ON "TEST_LOG_DETAIL"
FOR EACH ROW
WHEN (NEW."ID" IS NULL)
BEGIN
  SELECT TEST_ID_SEQ.NEXTVAL
  INTO :NEW."ID"
  FROM DUAL;
END;

3. 插入随机的3000条数据

INSERT
	INTO
	TEST_LOG_DETAIL (id,
	INSERT_TIME,
	MONEY,
	USERFROM,
	PRODUCT,
	TAG,
	PAY_CODE,
	USERNAME,
	ORDER_ID,
	ADD_FROM)
SELECT
	NULL,
	SYSDATE + ROWNUM / 24 / 3600 AS INSERT_TIME,
	TRUNC(DBMS_RANDOM.VALUE(0, 100)) AS MONEY,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS USERFROM,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PRODUCT,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS TAG,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PAY_CODE,
	'TESTADMIN' USERNAME,
	DBMS_RANDOM.STRING('x',
	16) ORDER_ID,
	DBMS_RANDOM.STRING('x',
	8) ADD_FROM
FROM
	DUAL
CONNECT BY
	LEVEL <= 3000

4. 更新3000条数据

UPDATE TEST_LOG_DETAIL SET MONEY  = 1001 WHERE USERNAME ='TESTADMIN'

5. 删除3000条数据

DELETE  TEST_LOG_DETAIL  WHERE  USERNAME ='TESTADMIN'

6. DQL查询操作

SELECT
	USERNAME,
	MONEY,
	TO_CHAR(insert_time, 'yyyy-MM-dd HH24:mi:ss') AS insert_time,
	ADD_FROM
FROM
	TEST_LOG_DETAIL
WHERE
	USERNAME = 'QUWO12TO'
	AND userfrom = 0
	AND insert_time >= TO_DATE('2022-12-21', 'YYYY-MM-DD')
	AND insert_time <= TO_DATE('2023-01-03', 'YYYY-MM-DD')+ 1
ORDER BY
	insert_time DESC

7. username字段创建索引，创建期间无法执行DML操作

create index idx_TEST_USERNAME on TEST_LOG_DETAIL(USERNAME);

8. 再次测试上文的DQL DML操作查看耗时

• 查询走索引耗时大幅降低
  

• 插入 索引字段username内容固定时耗时很快，非固定情况下耗时大幅度增加
  
  

• 更新走索引耗时大幅降低，不走索引耗时小幅降低

• 删除走索引耗时大幅降低，不走索引耗时小幅降低
  
  

9. 删除username索引

DROP index idx_TEST_USERNAME;

10. 在insert_time 字段增加索引

create index idx_TEST_INSERT_TIME ON TEST_LOG_DETAIL(INSERT_TIME);

11. 再次测试上文的DQL DML操作查看耗时

• 插入3000条数据，插入时间变化和固定耗时都大幅下降
  

INSERT
	INTO
	TEST_LOG_DETAIL (id,
	INSERT_TIME,
	MONEY,
	USERFROM,
	PRODUCT,
	TAG,
	PAY_CODE,
	USERNAME,
	ORDER_ID,
	ADD_FROM)
SELECT
	NULL,
	TO_DATE('2023-01-03 16:20:00','yyyy-mm-dd hh24:mi:ss')  AS INSERT_TIME,
	TRUNC(DBMS_RANDOM.VALUE(0, 100)) AS MONEY,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS USERFROM,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PRODUCT,
	TRUNC(1004) AS TAG,
	TRUNC(DBMS_RANDOM.VALUE(0, 9)) AS PAY_CODE,
	DBMS_RANDOM.STRING('x',
	8) USERNAME,
	DBMS_RANDOM.STRING('x',
	16) ORDER_ID,
	DBMS_RANDOM.STRING('x',
	8) ADD_FROM
FROM
	DUAL
CONNECT BY
	LEVEL <= 3000

• 查询走索引耗时大幅降低

  

• 更新走索引耗时大幅降低，不走索引耗时小幅降低

  

  

• 删除走索引耗时大幅降低，不走索引耗时增加
  
  

三、结论

数据库数据	无索引	username索引	insert_time索引
查询	9.458s	110ms	77ms
插入 3000条	523ms	479ms	402ms
更新 3000条	9.548s	12ms	5ms
删除 3000条	9.542s	35ms	75ms

增加索引后如果sql语句使用到了索引那么DQL，DML操作耗时都会大幅减少，如果没有使用到索引那么DML操作可能会有耗时增加，个人认为如果在日志大表中查询操作频繁且慢无法忍受的的话可以在常用的字段上加索引，但不要加过多索引，大多场景都和时间范围优化，索引最好加载时间字段上，同时需要注意在创建索引时表无法进行DML操作