服务器异常宕机引发的mysql故障

1：服务器异常宕机、mysql、mysql_safe都无法启动
查看mysql报如下错误：
2018-03-30 06:25:12 1649 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future r
elease. Please use the full name instead.
2018-03-30 06:25:12 1649 [Note] Plugin 'FEDERATED' is disabled.
2018-03-30 06:25:12 1649 [ERROR] Function 'innodb' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'innodb' with soname 'ha_innodb.so'.
2018-03-30 06:25:12 1649 [ERROR] Function 'federated' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'federated' with soname 'ha_federated.so'.
2018-03-30 06:25:12 1649 [ERROR] Function 'blackhole' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'blackhole' with soname 'ha_blackhole.so'.
2018-03-30 06:25:12 1649 [ERROR] Function 'archive' already exists
2018-03-30 06:25:12 1649 [Warning] Couldn't load plugin named 'archive' with soname 'ha_archive.so'.
2018-03-30 06:25:12 1649 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-03-30 06:25:12 1649 [Note] InnoDB: The InnoDB memory heap is disabled
2018-03-30 06:25:12 1649 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-03-30 06:25:12 1649 [Note] InnoDB: Memory barrier is not used
2018-03-30 06:25:12 1649 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-03-30 06:25:12 1649 [Note] InnoDB: Using Linux native AIO
2018-03-30 06:25:12 1649 [Note] InnoDB: Using CPU crc32 instructions
2018-03-30 06:25:12 1649 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-03-30 06:25:12 1649 [Note] InnoDB: Completed initialization of buffer pool
2018-03-30 06:25:35 1649 [ERROR] InnoDB: Tried to read 1048576 bytes at offset 1048576. Was only able to read 913408.
InnoDB: Fatal error: cannot read from file. OS error number 17.
2018-03-30 06:25:35 7fcc5aafc780 InnoDB: Assertion failure in thread 140515671525248 in file os0file.cc line 2694
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:25:35 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=400
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 175093 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x555930e2e94c]
/usr/sbin/mysqld(handle_fatal_signal+0x3c2)[0x555930b84e62]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fcc59c81330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fcc590c2c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fcc590c6028]
/usr/sbin/mysqld(+0x815583)[0x555930f3f583]
/usr/sbin/mysqld(_Z28buf_dblwr_init_or_load_pagesiPcb+0x272)[0x5559310105a2]
/usr/sbin/mysqld(+0x883b1c)[0x555930fadb1c]
/usr/sbin/mysqld(+0x7c5bbc)[0x555930eefbbc]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x555930ab3b28]
/usr/sbin/mysqld(+0x4f19e0)[0x555930c1b9e0]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x928)[0x555930c21608]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xa92)[0x555930aac9b2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fcc590adf45]
/usr/sbin/mysqld(+0x375c2d)[0x555930a9fc2d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

解决办法：
删除掉数据目录下ibdata*、ib_logfile*文件,

重启后可启动数据库，因数据库是非正常关闭引起的
****所有这类似的情况必须先备份数据库，避免人为操作造成的损坏
tar -zcf mysqlback20180329.tar.gz mysql
正常启动后mysql 可以正常访问，但是日志报错
2018-03-30 10:59:59 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 10:59:59 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2018-03-30 11:00:00 15939 [Warning] InnoDB: Cannot open table ambari/QRTZ_TRIGGERS from the internal data dictionary of InnoDB though the .frm file for the ta
ble exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.

所以直接退出，查看整个所有库大概有多少张表损坏
mysqlcheck -A -uroot -ppassworkd > /tmp/unnormaltable
**(查单个库和表：mysqlcheck zabbix users -uroot -ppassword)

通过上述命令输出然后在分析大概有哪些表需要做恢复
grep -v ":" /tmp/unnormaltable |awk -F"." '{print $1,$2}'|sort -rn |uniq -c

恢复zabbix库
grep -v ":" /tmp/unnormaltable|grep zabbix |awk -F"." '{print $2}' > tablezabbix

把zabbix库中的文件全部mv到其它目录，导入空表结构

mysql -uroot -ppassword < zabbix.sql

到出表空间，清空idb文件，只剩下frm文件
while read line;do mysql -uroot -ppassword -e "use zabbix;SET FOREIGN_KEY_CHECKS = 0 ;ALTER TABLE $line DISCARD TABLESPACE";done < tablezabbix

ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails ()
SET FOREIGN_KEY_CHECKS = 0

把之前备份的ibd文件重新导入到zabbix目录下
for i in `find myzabbix/ -name *.ibd `;do mv $i ./zabbix;done
把导入的ibd文件导入表空间
while read line;do mysql -uroot -ppassword -e "use zabbix;ALTER TABLE $line IMPORT TABLESPACE";done < tablezabbix

Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
导入的过程中如果有的表空间过大的话可能会报错，可以适当优化两个参数
wait_timeout = x 超时时间如600秒
max_allowed_packet = y 最大允许数据量

整体过程：
1、备份
2、确定哪些表损坏
3、移除损坏表的数据至备份目录
4、创建这些表
5、导出表空间
6、移入刚移出的这些表的ibd文件
7、导入表空间
8、测试

服务器异常宕机引发的mysql故障

猜你喜欢