数据大挪移之ora2pg使用指南

ora2pg是一款免费的小工具，用于将Oracle数据库迁移到PostgreSQL。ora2pg通过自动扫描oracle数据库获取表结构以及数据，然后生成PostgreSQL数据库的SQL脚本或者直接在PostgreSQL上创建表并插入数据。而且它还支持视图、序列以及分区表，其功能非常强大。这里有一个120G左右的oracle 11g测试库，因业务需求将其迁移至PostgreSQL 10.6。

一、安装ora2pg

1.1 安装OS依赖软件包

[root@pg07 ~]# yum install perl-DBD-Pg perl perl-devel perl-DBI  perl-CPAN bzip2 perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker

1.2 安装Oracle瘦客户端

从Oracle官方网站下载oracle 12c的瘦客户端，然后进行安装并配置。

[root@pg07 u02]# la /u02/software/oracle/
total 55M
drwxr-xr-x 2 root root 4.0K Dec  3 15:04 .
drwxr-xr-x 7 root root   76 Dec  3 15:04 ..
-rw-r--r-- 1 root root  51M Nov 28 14:34 oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 593K Nov 28 14:32 oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 1.5M Nov 28 14:32 oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 692K Nov 28 14:32 oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 922K Nov 28 14:35 oracle-instantclient12.2-tools-12.2.0.1.0-1.x86_64.rpm
[root@pg07 ~]# yum -y localinstall /u02/software/oracle/oracle-instantclient12.2-*.rpm
[root@pg07 ~]# mkdir -p /usr/lib/oracle/12.2/client64/network/admin

编辑root环境变量，加入以下内容：

[root@pg07 ~]# vi .bash_profile 
export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib
export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
export TNS_ADMIN=/usr/lib/oracle/12.2/client64/network/admin
export ORACLE_HOME=/usr/lib/oracle/12.2/client64/
export PATH=$PATH:$ORACLE_HOME/bin:$HOME/bin
[root@pg07 ~]# source .bash_profile

1.3 安装ora2pg

[root@pg07 ~]# wget https://sourceforge.net/projects/ora2pg/files/19.1/ora2pg-19.1.tar.bz2
[root@pg07 ~]# tar -xjf ora2pg-19.1.tar.bz2 
[root@pg07 ~]# cd ora2pg-19.1
[root@pg07 ora2pg-19.1]# perl Makefile.PL 
Checking if your kit is complete...
Looks good
Writing Makefile for Ora2Pg

Done...
------------------------------------------------------------------------------
Please read documentation at http://ora2pg.darold.net/ before asking for help
------------------------------------------------------------------------------
Now type: make && make install
[root@pg07 ora2pg-19.1]# make && make install
cp lib/Ora2Pg.pm blib/lib/Ora2Pg.pm
cp lib/Ora2Pg/GEOM.pm blib/lib/Ora2Pg/GEOM.pm
cp lib/Ora2Pg/PLSQL.pm blib/lib/Ora2Pg/PLSQL.pm
cp lib/Ora2Pg/MySQL.pm blib/lib/Ora2Pg/MySQL.pm
cp scripts/ora2pg blib/script/ora2pg
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg
cp scripts/ora2pg_scanner blib/script/ora2pg_scanner
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg_scanner
Manifying blib/man3/ora2pg.3
Installing /usr/local/share/perl5/Ora2Pg.pm
Installing /usr/local/share/perl5/Ora2Pg/GEOM.pm
Installing /usr/local/share/perl5/Ora2Pg/PLSQL.pm
Installing /usr/local/share/perl5/Ora2Pg/MySQL.pm
Installing /usr/local/share/man/man3/ora2pg.3
Installing /usr/local/bin/ora2pg
Installing /usr/local/bin/ora2pg_scanner
Installing default configuration file (ora2pg.conf.dist) to /etc/ora2pg
Appending installation info to /usr/lib64/perl5/perllocal.pod

1.4 安装DBD::Oracle

[root@pg07 ~]# perl -MCPAN -e shell
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9800)
Enter 'h' for help.
cpan[1]> get DBD::Oracle
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/01mailrc.txt.gz
Reading '/root/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/02packages.details.txt.gz
Reading '/root/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Wed, 28 Nov 2018 23:54:25 GMT
  HTTP::Date not available
.............
  New CPAN.pm version (v2.16) available.
  [Currently running version is v1.9800]
  You might want to try
    install CPAN
    reload cpan
  to both upgrade CPAN.pm and run the new version without leaving
  the current session.

...............................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/03modlist.data.gz
Reading '/root/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /root/.cpan/Metadata
Running get for module 'DBD::Oracle'
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/CHECKSUMS
Checksum for /root/.cpan/sources/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz ok
Scanning cache /root/.cpan/build for sizes
DONE
cpan[2]> quit
Terminal does not support GetHistory.
Lockfile removed.
[root@pg07 ~]# cd .cpan/build/DBD-Oracle-1.76-sU5yu5/
[root@pg07 DBD-Oracle-1.76-sU5yu5]# perl Makefile.PL
[root@pg07 DBD-Oracle-1.76-sU5yu5]# make && make install

1.5 创建ora2pg配置文件

首先从示例文件复制一份，然后根据实际情况更改配置文件相关参数：

[root@pg07 ~]# cp /etc/ora2pg/ora2pg.conf.dist /etc/ora2pg/ora2pg.conf
[root@pg07 ~]#  vi /etc/ora2pg/ora2pg.conf
--Oracle连接参数
ORACLE_HOME     /usr/lib/oracle/12.2/client64
ORACLE_DSN      dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
ORACLE_USER     system --如果这里的用户是sys、system，则user_grants选项必须设置为1
ORACLE_PWD      abcABC12
USER_GRANTS     1
ORA_INITIAL_COMMAND commit
--schema参数
EXPORT_SCHEMA   1
SCHEMA  XXKIN
CREATE_SCHEMA   0
COMPILE_SCHEMA  0
PG_SCHEMA   public
--导出对象(根据实际情况选择)
TYPE    TABLE COPY  VIEW TYPE 
DISABLE_COMMENT         0
EXTERNAL_TO_FDW         1
TRUNCATE_TABLE  0
USE_TABLESPACE          0
REORDERING_COLUMNS      0
--约束参数
KEEP_PKEY_NAMES         0
PKEY_IN_CREATE          0
FKEY_ADD_UPDATE         never
FKEY_DEFERRABLE 0
DEFER_FKEY      0
DROP_FKEY       1
--触发器参数
DISABLE_SEQUENCE        1
DISABLE_TRIGGERS 1
--对象修改参数
PRESERVE_CASE   0
INDEXES_RENAMING        0
USE_INDEX_OPCLASS       0
PREFIX_PARTITION        0
DISABLE_PARTITION       1
WITH_OID                0
ORA_RESERVED_WORDS      audit,comment
USE_RESERVED_WORDS      1
DISABLE_UNLOGGED        0
--PostgreSQL参数
PG_DSN          dbi:Pg:dbname=kkdb;host=192.168.120.149;port=5432
PG_USER         xxkin
PG_PWD          redhat
OUTPUT          xxkin.sql
OUTPUT_DIR     /tmp
BZIP2
FILE_PER_CONSTRAINT     0
FILE_PER_INDEX          0
FILE_PER_FKEYS          0
FILE_PER_TABLE  0
FILE_PER_FUNCTION       0
STOP_ON_ERROR           1
COPY_FREEZE             0
CREATE_OR_REPLACE       1
--TYPE参数
PG_NUMERIC_TYPE 1
PG_INTEGER_TYPE 1
DEFAULT_NUMERIC bigint
ENABLE_MICROSECOND      1
TO_NUMBER_CONVERSION    numeric
GEN_USER_PWD    0
FORCE_OWNER     0
FORCE_SECURITY_INVOKER  0
--数据参数
DATA_LIMIT      10000
NOESCAPE        0
TRANSACTION     serializable
STANDARD_CONFORMING_STRINGS     1
NO_LOB_LOCATOR  1
XML_PRETTY      0
LOG_ON_ERROR            0
TRIM_TYPE               BOTH
INTERNAL_DATE_MAX       49
FUNCTION_CHECK          1
--Export/Import性能参数
JOBS            1
ORACLE_COPIES   1
PARALLEL_TABLES 1
DEFAULT_PARALLELISM_DEGREE      0
PARALLEL_MIN_ROWS               100000
DROP_INDEXES    0
SYNCHRONOUS_COMMIT      0
--PLSQL参数
EXPORT_INVALID  0
PLSQL_PGSQL     1
NULL_EQUAL_EMPTY        0
EMPTY_LOB_NULL          0
PACKAGE_AS_SCHEMA       1
REWRITE_OUTER_JOIN      1
FUNCTION_STABLE         1
COMMENT_COMMIT_ROLLBACK 0
COMMENT_SAVEPOINT       0
USE_ORAFCE      0
AUTONOMOUS_TRANSACTION  1
--ASSESSMENT参数
ESTIMATE_COST           0
COST_UNIT_VALUE         5
DUMP_AS_HTML            0
TOP_MAX                 10
HUMAN_DAYS_LIMIT        5
--PostgreSQL特性参数
PG_SUPPORTS_WHEN        1
PG_SUPPORTS_INSTEADOF   1
PG_SUPPORTS_MVIEW       1
PG_SUPPORTS_CHECKOPTION 1
PG_SUPPORTS_IFEXISTS    1
PG_SUPPORTS_LATERAL     1
PG_SUPPORTS_PARTITION   1
BITMAP_AS_GIN           1
PG_BACKGROUND           0
PG_SUPPORTS_SUBSTR      1
PG_SUPPORTS_NAMED_OPERATOR      1
PG_SUPPORTS_IDENTITY    1
PG_SUPPORTS_PROCEDURE   0
--SPATIAL参数
AUTODETECT_SPATIAL_TYPE 1
CONVERT_SRID            1
DEFAULT_SRID            4326
GEOMETRY_EXTRACT_TYPE   INTERNAL
--FDW参数
FDW_SERVER      orcl
--MySQL参数
MYSQL_PIPES_AS_CONCAT           0
MYSQL_INTERNAL_EXTRACT_FORMAT   0

到此，整个ora2pg的安装配置已完成。

二、使用ora2pg

2.1 导出全部表结构

如果在配置文件里指定的type包含了COPY，导完表结构后，表的数据也会随之导出，加-t参数即可只导出表结构不导出数据。

[root@pg07 ~]# ora2pg -t table view
[========================>] 777/777 tables (100.0%) end of scanning.                 
[========================>] 777/777 tables (100.0%) end of table export.                  
[========================>] 0/0 views (100.0%) end of output. 

导出完成后，在PostgreSQL中创建数据库以及对应的表。

-bash-4.2$ psql
psql (10.6)
Type "help" for help.

postgres=# CREATE USER xxkin WITH password 'redhat' CREATEDB SUPERUSER replication bypa***ls createrole login;
postgres=# create database kkdb;
postgres=# alter database kkdb owner to xxkin;
postgres=# \c kkdb
You are now connected to database "kkdb" as user "postgres".
kkdb=# \i /tmp/output.sql

ora2pg有很多参数可供使用，具体请参考其帮助文件。

2.2 通过ora2pg迁移数据

之前已创建好数据库以及相关表结构，现在只需通过ora2pg将数据同步到PostgreSQL数据库上即可。

[root@pg07 ~]# ora2pg -d -t copy
[2018-11-30 11:38:51] Ora2Pg version: 19.1
[2018-11-30 11:38:51] Trying to connect to database: dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
[2018-11-30 11:38:51] Isolation level: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
[2018-11-30 11:38:51] DEBUG: executing initial command to Oracle: commit
[2018-11-30 11:38:51] Retrieving table information...
[2018-11-30 11:40:55] [1] Scanning table ABC_DEL_CLASS (1 rows)...
[2018-11-30 11:40:55] [2] Scanning table ACTIVITY_DATA (41 rows)...
[2018-11-30 11:40:55] [3] Scanning table ACTIVITY_GBEAN_STAT (243897 rows)...
......

如果在迁移过程中，需要排除某些表，加-e参数，如下:

[root@pg07 ~]# ora2pg -d -t copy -e usr_dx_tbs,hr_info_pg,op_log

总结：

以下是在使用过程遇到的问题，提出的建议：
1、先导出表结构，然后在同步数据；
2、在导出表结构的时候禁用分区表；
3、注意PostgreSQL的保留的关键字是否为oracle表字段名；
4、删除所有外键约束。
5、直接通过ora2pg迁移数据，不要将oracle表数据导出为csv格式，然后再导入PostgreSQL数据库。
参考文献：ora2pg DOCUMENTATION