PostgreSQL如何恢复误操作的数据，walminer关键时刻救你一命！

数据库 2024-11-01 14:30:44 阅读次数: 0

上一篇文章介绍了PostgreSQL如何恢复误操作的数据（详见：PG数据库误删除数据怎么办，还需要跑路吗？），由于使用了Walminer 4.8版本，这个需要购买license，才能使用。有读者私信，有没有开源的工具可以使用呢？当然了，Walminer有完全开源的3.0版，但它需要以插件的方式安装在数据库，支持PostgreSQL10及其以上版本，功能上也相比Walminer 4.x版本弱。如果你的需求不高，应该也能满足使用，下面就来说说walminer 3.0是怎么使用的。

WalMiner3.0安装

软件下载地址：https://gitee.com/movead/XLogMiner/tree/master/
它的安装方式主要有2种：

编译一：PG源码编译 如果你从编译pg数据库开始

  1. 将walminer目录放置到编译通过的PG工程的"…/contrib/"目录下

  2. 进入walminer目录

  3. 执行命令

    make && make install

编译二：依据PG安装编译 如果你使用yum或者pg安装包安装了pg

  1. 配置pg的bin路径至环境变量

    export PATH=/h2/pg_install/bin:$PATH

  2. 进入walminer代码路径

  3. 执行编译安装

    USE_PGXS=1 MAJORVERSION=12 make
#MAJORVERSION支持‘10’,‘11’,‘12’,‘13’,‘14’,‘15’,‘16’,‘17’
USE_PGXS=1 MAJORVERSION=12 make install

以我本机的pg 14.4安装为例：

--把下载的软件进行解压
[root@pgdkcs XLogMiner-walminer_3.0_stable]# unzip -q XLogMiner-walminer_3.0_stable.zip
[root@pgdkcs XLogMiner-walminer_3.0_stable]# cd walminer/
[root@pgdkcs walminer]# ls
datadictionary.c  expected             imagemanage.c  pagecollect.h  todo.md        wal2sql_ddl.h  wal2sql_spi.h      walminer_12.conf   walminer.c           walminer_contents.h  walminer_decode.h  wm_compatible.c  wm_utils.h
datadictionary.h  fetchcatalogtable.c  Makefile       sql            wal2sql.c      wal2sql.h      wallist.c          walminer--2.0.sql  walminer.conf        walminer.control     walminer_thread.c  wm_compatible.h
debug_info.md     image                pagecollect.c  temp.h         wal2sql_ddl.c  wal2sql_spi.c  walminer--1.0.sql  walminer--3.0.sql  walminer_contents.c  walminer_decode.c    walreader.c        wm_utils.c

[root@pgdkcs walminer]# USE_PGXS=1 MAJORVERSION=14 make
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o walminer.o walminer.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o wm_utils.o wm_utils.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o datadictionary.o datadictionary.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o fetchcatalogtable.o fetchcatalogtable.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o wallist.o wallist.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o walreader.o walreader.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o walminer_decode.o walminer_decode.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o imagemanage.o imagemanage.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o wal2sql.o wal2sql.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o walminer_contents.o walminer_contents.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o walminer_thread.o walminer_thread.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o wal2sql_spi.o wal2sql_spi.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o wal2sql_ddl.o wal2sql_ddl.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o pagecollect.o pagecollect.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -DPG_VERSION_14 -I. -I./ -I/usr/local/pg144/include/postgresql/server -I/usr/local/pg144/include/postgresql/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o wm_compatible.o wm_compatible.c
gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -shared -o walminer.so walminer.o wm_utils.o datadictionary.o fetchcatalogtable.o wallist.o walreader.o walminer_decode.o imagemanage.o wal2sql.o walminer_contents.o walminer_thread.o wal2sql_spi.o wal2sql_ddl.o pagecollect.o wm_compatible.o -L/usr/local/pg144/lib    -Wl,--as-needed -Wl,-rpath,'/usr/local/pg144/lib',--enable-new-dtags  
[root@pgdkcs walminer]# 
[root@pgdkcs walminer]# 
[root@pgdkcs walminer]# USE_PGXS=1 MAJORVERSION=14 make install
/usr/bin/mkdir -p '/usr/local/pg144/lib/postgresql'
/usr/bin/mkdir -p '/usr/local/pg144/share/postgresql/extension'
/usr/bin/mkdir -p '/usr/local/pg144/share/postgresql/extension'
/usr/bin/install -c -m 755  walminer.so '/usr/local/pg144/lib/postgresql/walminer.so'
/usr/bin/install -c -m 644 .//walminer.control '/usr/local/pg144/share/postgresql/extension/'
/usr/bin/install -c -m 644 .//walminer--3.0.sql  '/usr/local/pg144/share/postgresql/extension/'

创建walminer的extension：

postgres@[local]:5432=#\dx
                                            List of installed extensions
        Name        | Version |   Schema   |                              Description                               
--------------------+---------+------------+------------------------------------------------------------------------
 pg_stat_statements | 1.9     | public     | track planning and execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

postgres@[local]:5432=#create extension walminer;
CREATE EXTENSION
postgres@[local]:5432=#\dx
                                            List of installed extensions
        Name        | Version |   Schema   |                              Description                               
--------------------+---------+------------+------------------------------------------------------------------------
 pg_stat_statements | 1.9     | public     | track planning and execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
 walminer           | 3.0     | public     | analyse wal to SQL
(3 rows)

WalMiner使用步骤

有用过oracle的logminer挖掘归档日志，WalMiner使用方式大概类似：
（1）添加要解析的wal日志文件

-- 添加wal文件：
select walminer_wal_add('/usr/local/pg144/data/pg_wal');
-- 注：参数可以为目录或者文件

如果添加错误了，要Remove wal日志文件，可以使用下面的方法：

-- 移除wal文件：
select walminer_wal_remove('/usr/local/pg144/data/pg_wal');
-- 注：参数可以为目录或者文件

查看已添加的wal日志文件

-- 列出wal文件：
select walminer_wal_list();

（2）解析已添加的wal日志

普通解析，解析add的全部wal日志

--解析add的全部wal日志
select walminer_all();
或 select wal2sql();

精确解析：

--在add的wal日志中查找对应时间范围的wal记录
select walminer_by_time(starttime, endtime,'true'); 
或 select wal2sql(starttime, endtime,'true');

--在add的wal日志中查找对应lsn范围的wal记录
select walminer_by_lsn(startlsn, endlsn,'true'); 
或 select wal2sql(startlsn, endlsn,'true');

--在add的wal日志中查找对应xid的wal记录
select walminer_by_xid(xid,'true'); 
或 select wal2sql(xid,'true');

（3）解析结果查看

select * from walminer_contents;
-- 表walminer_contents 
(
 sqlno int, 		--本条sql在其事务内的序号
 xid bigint,		--事务ID
 topxid bigint,		--如果为子事务，这是是其父事务；否则为0
 sqlkind int,		--sql类型1->insert;2->update;3->delete(待优化项目)
 minerd bool,		--解析结果是否完整(缺失checkpoint情况下可能无法解析出正确结果)
 timestamp timestampTz, --这个SQL所在事务提交的时间
 op_text text,		--sql
 undo_text text,	--undo sql
 complete bool,		--如果为false，说明有可能这个sql所在的事务是不完整解析的
 schema text,		--目标表所在的模式
 relation text,		--目标表表名
 start_lsn pg_lsn,	--这个记录的开始LSN
 commit_lsn pg_lsn	--这个事务的提交LSN
)

注意：walminer_contents是walminer自动生成的unlogged表，在一次解析开始会首先创建或truncate walminer_contents表。

（4）结束walminer操作

select walminer_stop();

该函数作用为释放内存，结束日志分析，该函数没有参数。

测试案例

–创建测试表

create table walminer_t(id int,col1 varchar(8));
insert into walminer_t values(1,'w1'),(2,'w2');

postgres@[local]:5432=#select * from walminer_t ;
 id | col1 
----+------
  1 | w1
  2 | w2
(2 rows)

postgres@[local]:5432=#select pg_current_wal_lsn();
 pg_current_wal_lsn 
--------------------
 0/3B0533F0
(1 row)

postgres@[local]:5432=#select pg_walfile_name(pg_current_wal_lsn());
     pg_walfile_name      
--------------------------
 00000001000000000000003B
(1 row)

–添加需要解析的wal文件

我们可以添加wal目录，也可以添加指定的wal文件（默认情况下此步骤可以省略，因为会直接加载当前数据字典和当前wal路径下的所有wal文件,但不建议省略）。

select walminer_wal_add('/usr/local/pg144/data/pg_wal');
或
select walminer_wal_add('/usr/local/pg144/data/pg_wal/00000001000000000000003B');

–列出已添加的wal文件

postgres@[local]:5432=#select walminer_wal_list();
                    walminer_wal_list                    
---------------------------------------------------------
 (/usr/local/pg144/data/pg_wal/00000001000000000000003B)
(1 row)

–解析wal

postgres@[local]:5432=#select walminer_all();
NOTICE:  ===========================================
NOTICE:  ======OPEN SOURCE PROJECT WALMINER=========
NOTICE:  ======Author mail [email protected]========
NOTICE:  ===========================================
NOTICE:  Switch wal to 00000001000000000000003B on time 2024-10-28 21:31:00.728701+08
  walminer_all   
-----------------
 wal2sql success
(1 row)

–查看解析后的数据

walminer_contents是walminer自动生成的unlogged表，用来存储解析结果，在一次解析开始会先create或truncate此表，在表中我们看到这两条数据的xid、执行时间、执行操作、开始/结束lsn，甚至undo sql文本都写好了，很强大。

walminer_contents各字段解释：

(
sqlno int, --本条sql在其事务内的序号
xid bigint, --事务ID
topxid bigint, --如果为子事务，这是是其父事务；否则为0
sqlkind int, --sql类型1->insert;2->update;3->delete(待优化项目)
minerd bool, --解析结果是否完整(缺失checkpoint情况下可能无法解析出正确结果)
timestamp timestampTz, --这个SQL所在事务提交的时间
op_text text, --sql
undo_text text, --undo sql
complete bool, --如果为false，说明有可能这个sql所在的事务是不完整解析的
schema text, --目标表所在的模式
relation text, --目标表表名
start_lsn pg_lsn, --这个记录的开始LSN
commit_lsn pg_lsn --这个事务的提交LSN
)

postgres@[local]:5432=#select * from walminer_contents;
 sqlno | xid | topxid | sqlkind | minerd |           timestamp           |                         op_text                         |                       undo_text                        | complete | schema |  relation  | start_lsn  | 
commit_lsn 
-------+-----+--------+---------+--------+-------------------------------+---------------------------------------------------------+--------------------------------------------------------+----------+--------+------------+------------+-
-----------
     1 | 869 |      0 |       1 | t      | 2024-10-28 21:23:18.084329+08 | INSERT INTO public.walminer_t(id ,col1) VALUES(1 ,'w1') | DELETE FROM public.walminer_t WHERE id=1 AND col1='w1' | t        | public | walminer_t | 0/3B053310 | 
0/3B0533B8
     2 | 869 |      0 |       1 | t      | 2024-10-28 21:23:18.084329+08 | INSERT INTO public.walminer_t(id ,col1) VALUES(2 ,'w2') | DELETE FROM public.walminer_t WHERE id=2 AND col1='w2' | t        | public | walminer_t | 0/3B053350 | 
0/3B0533B8
(2 rows)


postgres@[local]:5432=#select undo_text from walminer_contents;
                       undo_text                        
--------------------------------------------------------
 DELETE FROM public.walminer_t WHERE id=1 AND col1='w1'
 DELETE FROM public.walminer_t WHERE id=2 AND col1='w2'
(2 rows)

我们可以直接执行undo_text 的语句，进行恢复数据。也可以通过以下方式先进行导出恢复的语句，然后在导入。

–导出数据

postgres@[local]:5432=#\pset footer
Default footer is off.
postgres@[local]:5432=#\pset t
Tuples only is on.
postgres@[local]:5432=#\o huifu.sql
postgres@[local]:5432=#select undo_text from walminer_contents;
postgres@[local]:5432=#\q

[postgres@pgdkcs ~]$ cat huifu.sql 
 DELETE FROM public.walminer_t WHERE id=1 AND col1='w1'
 DELETE FROM public.walminer_t WHERE id=2 AND col1='w2'

–结束walminer操作

select walminer_stop();

–恢复数据

\set AUTOCOMMIT off
\i huifu.sql
commit;

使用精确的解析方式，还可以用以下方式：

–基于时间范围解析

select walminer_by_time('2024-10-28 21:22:00','2024-10-28 21:23:18');

–基于lsn范围解析

select walminer_by_lsn('0/3B053310','0/3B0533B8');

–基于指定xid解析

select walminer_by_xid('869');

当然，walminer还有很多其它骚操作，有兴趣的可以去试试。

关注我，学习更多的数据库知识！
请添加图片描述

猜你喜欢

转载自blog.csdn.net/zhirongsu/article/details/143350045
今日推荐
周排行
TryParse的使用方法小结
Android之高仿手机QQ聊天
关于在java中关键字private能否用来修饰类的问题
去雾算法总结
前端面试题：事件防抖，函数节流，事件防抖和函数节流的区别
随堂小测
【OpenCV + Python】归一化函数cv2.normalize()的原理讲解
05: redis 主从复制
python3-基础5
持续更新-使用 Maven Module 搭建spring boot项目（整合Spring Security、Spring Social、spring OAuth）第一篇
每日归档
更多
2025-03-18(0)
2025-03-17(0)
2025-03-16(0)
2025-03-15(0)
2025-03-14(0)
2025-03-13(0)
2025-03-12(0)
2025-03-11(0)
2025-03-10(0)
2025-03-09(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022