1概述

顶新苏州IDC机房配置了四个Linux HA+ Oracle数据库集群。其配置信息如下：

1.1店会系统&FC-arap系统集群

集群资源组名称	ac_group	fc_group
集群资源文件挂载点	/ac-oradata	/fc-oradata
集群服务IP地址	fmpracora-vip/ 10.0.101.11	fmprfcora-vip/ 10.0.101.12
集群Listener	cvs_lsn	arap2_lsn
集群数据库	cvsdb	arap2db
优先运行的节点名称	fmpracora-priv	fmprfcora-priv
优先运行的节点IP	10.0.101.9	10.0.101.10
集群fence通讯IP	10.0.202.15	10.0.202.19

1.2订购-营收&主档-预算检讨-基盘集群

集群资源组名称	ma_group	po_group
集群资源文件挂载点	/ma-oradata	/po-oradata
集群服务IP地址	fmprmaora-vip/ 10.0.101.7	fmprpoora-vip/ 10.0.101.8
集群Listener	arpdb_lsn	yingshou_lsn
集群数据库	master_db, arpdb_db	dingoudb, yingshou_db
优先运行的节点名称	fmprmaora-priv	fmprpoora-priv
优先运行的节点IP	10.0.101.5	10.0.101.6
集群fence通讯IP	10.0.202.16	10.0.202.20

1.3IDC-SCM系统集群

集群资源组名称	idc_group
集群资源文件挂载点	/idc-oradata
集群服务IP地址	fmpridcora-vip/ 10.0.101.4
集群Listener	arpdb_lsn
集群数据库	fmidcdb
优先运行的节点名称	fmpridcora-priv
优先运行的节点IP	10.0.101.4
集群fence通讯IP	10.0.202.17

集群另外一个节点是10.0.101.3。

1.4JDE系统&JDE测试系统集群

集群资源组名称	pr_group	dr_group
集群资源文件挂载点	/jdepr-oradata	/jdedr-oradata
集群服务IP地址	fmprjdeora-vip/ 10.0.101. 15	fmdrjdeora-vip/ 10.0.101. 16
集群Listener	jde_lsn	jdedbdr_lsn
集群数据库	jde_db	jdedr_db
优先运行的节点名称	fmprjdeora-priv	fmdrjdeora-priv
优先运行的节点IP	10.0.101.13	10.0.101.14
集群fence通讯IP	10.0.202.18	10.0.202.22

2安装配置手册

RHEL 6.8 HA和RHEL 7.2 HA的安装过程不相同，但是集群的配置过程相同。我们将软件安装过程分开，配置过程放在一起描述。

2.1 . RHEL 6.8 HA软件安装

将RHEL 6.8 ISO文件挂载到/software/rh68目录。编辑/etc/yum.conf文件。

2.1.1 编辑/etc/yum.conf

[local]

name=local

baseurl=file:///software/rh68

enabled=1

gpgcheck=1

gpgkey=file:///software/rh68/RPM-GPG-KEY-redhat-release

[HighAvailability]

name=HighAvailability

baseurl=file:///software/rh68/HighAvailability

enabled=1

gpgcheck=0

2.1.2 安装软件

# yum install pcs pacemaker cman fence-agents

从RHEL 6.5以后，RHEL的HA通过pcs来管理，不再通过rgmanager管理。只有RHEL 6.8才有Oracle的Agent。

2.1.3 禁用和停止主机防火墙服务

禁用防火墙

#chkconfig iptables off

#chkconfig ip6tables off

停止防火墙服务

#service iptables stop

#service ip6tables off

For TCP: Ports 2224, 3121, 21064

For UDP: Ports, 5405

2.1.4 启动和配置服务pcsd

# chkconfig pcsd on

# service pcsd start

2.1.5 重置hacluster密码

hacluster是pcs 管理员账号，建议集群中的服务器设置相同的密码。

# passwd hacluster (hacluster1234)

更改用户 hacluster 的密码。

新的密码：

无效的密码：密码包含用户名在某些地方

重新输入新的密码：

passwd：所有的身份验证令牌已经成功更新。

2.2RHEL 7.2 HA软件安装

2.2.1 /etc/yum.conf

[HighAvailability]

name=HighAvailability

baseurl=file:///software/rh72/addons/HighAvailability

enabled=1

gpgcheck=0

[local]

name=local

baseurl=file:///software/rh72/

enabled=1

gpgcheck=1

gpgkey=file:///software/rh72/RPM-GPG-KEY-redhat-release

2.2.2 安装软件

# yum install pcs fence-agents-all

只有RHEL 7.2才有Oracle的HA Agent, RHEL 7.0和7.1都没有Oracle的HA Agent。在RHEL 7.2,将不再使用cman服务.

2.2.3 启动和配置服务pcsd

[root@fmprfcdb ~]# systemctl start pcsd.service

[root@fmprfcdb ~]# systemctl enable pcsd.service

ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'

检查pcsd服务能随机自动重新启动

[root@fmpracora dinxin]# systemctl list-unit-files |grep pcs

pcsd.service enabled

rpcsvcgssd.service static

2.2.4 禁用和停止主机防火墙服务

#systemctl stop firewalld

2.2.5 重置hacluster密码

hacluster是pcs 管理员账号，建议集群中的服务器设置相同的密码。

# passwd hacluster (hacluster1234)

更改用户 hacluster 的密码。

新的密码：

无效的密码：密码包含用户名在某些地方

重新输入新的密码：

passwd：所有的身份验证令牌已经成功更新。

2.3配置集群

RHEL 6.8和RHEL 7.2配置集群的步骤相同。将在以下介绍。

2.3.1 编辑/etc/hosts

[root@fmprpoora oracle]# cat /etc/hosts

127.0.0.1localhost localhost.localdomain localhost4 localhost4.localdomain4

::1localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.101.6 fmprmaora

10.0.101.5 fmprpoora

10.0.200.16 fmprpoora-priv

10.0.200.20 fmprmaora-priv

10.0.101.7fmprpoora-vip

10.0.101.8fmprmaora-vip

其中fmprpoora-priv，fmprmaora-priv为心跳IP地址对应的主机名，将用作集群间的通讯。

2.3.2 授权hacluster 访问集群中的节点

# pcs cluster auth fmprfcdb-priv fmpracdb-priv

Username: hacluster

Password:

fmprfcdb-priv: Authorized

fmpracdb-priv: Authorized

授权的令牌保存在以下目录中。

[root@fmprpoora oracle]# cat /var/lib/pcsd/tokens

{

"format_version": 2,

"data_version": 3,

"tokens": {

"fmprmaora-priv": "a2a19af3-584e-4f88-873b-c3bee27d137c",

"fmprpoora-priv": "33169580-5ba5-4cb3-9a1c-a4a87fd3d226"

}

2.3.3 创建cluster

[root@fmprfcora pcsd]# pcs cluster setup --start --name fmprfcac_cluster fmprfcora-priv fmpracora-priv

Shutting down pacemaker/corosync services...

Redirecting to /bin/systemctl stop pacemaker.service

Redirecting to /bin/systemctl stop corosync.service

Killing any remaining services...

Removing all cluster configuration files...

fmprfcora-priv: Succeeded

fmpracora-priv: Succeeded

Starting cluster on nodes: fmprfcora-priv, fmpracora-priv...

fmpracora-priv: Starting Cluster...

fmprfcora-priv: Starting Cluster...

Synchronizing pcsd certificates on nodes fmprfcora-priv, fmpracora-priv...

fmprfcora-priv: Success

fmpracora-priv: Success

Restaring pcsd on the nodes in order to reload the certificates...

fmprfcora-priv: Success

fmpracora-priv: Success

[root@fmprfcora pcsd]#

2.3.4 配置集群服务随主机启动而自动启动

[root@fmprfcora pcsd]# pcs cluster enable --all

fmprfcora-priv: Cluster Enabled

fmpracora-priv: Cluster Enabled

2.3.5 检查cluster服务状态

[root@fmpracdb pcsd]# pcs cluster status

Cluster Status:

Last updated: Sun Aug 7 11:07:59 2016 Last change: Sun Aug 7 11:06:44 2016 by hacluster via crmd on fmpracora-priv

Stack: corosync

Current DC: fmpracora-priv (version 1.1.13-10.el7-44eb2dd) - partition with quorum

2 nodes and 0 resources configured

Online: [ fmpracora-priv fmprfcora-priv ]

PCSD Status:

fmprfcora-priv: Online

fmpracora-priv: Online

2.3.6 配置集群的fence设备

#pcs stonith create imm-fmprjdeora fence_ipmilan action=reboot auth=password ipaddr=10.0.202.18 lanplus=1 login=USERID passwd=PASSW0RD delay=120cipher=1 pcmk_host_list=fmprjdeora-priv pcmk_host_check=static-list power_timeout=60

#pcs stonith create imm-fmdrjdeora fence_ipmilan action=reboot auth=password ipaddr=10.0.202.22 lanplus=1 login=USERID passwd=PASSW0RD cipher=1 pcmk_host_list=fmdrjdeora-priv pcmk_host_check=static-list power_timeout=60

查看当前fence设备配置的命令如下：

[root@fmprpoora oracle]# pcs stonith show --full

Resource: imm-fmprmaora (class=stonith type=fence_ipmilan)

Attributes: action=reboot auth=password ipaddr=10.0.202.20 lanplus=1 login=USERID passwd=PASSW0RD cipher=1 pcmk_host_list=fmprmaora-priv pcmk_host_check=static-list power_timeout=60

Operations: monitor interval=60s (imm-fmprmaora-monitor-interval-60s)

Resource: imm-fmprpoora (class=stonith type=fence_ipmilan)

Attributes: action=reboot auth=password ipaddr=10.0.202.16 lanplus=1 login=USERID passwd=PASSW0RD delay=5 cipher=1 pcmk_host_list=fmprpoora-priv pcmk_host_check=static-list power_timeout=60 debug=/tmp/fmprpoora.log

Operations: monitor interval=60s (imm-fmprpoora-monitor-interval-60s)

[root@fmprpoora oracle]#

2.3.7 测试fence

检查能否使用fence_ipmilan命令查看fence设备的状态

[root@fmpridcora oracle]# fence_ipmilan -a 10.0.202.22 -P -l USERID -p PASSW0RD -o status -v

Executing: /usr/bin/ipmitool -I lanplus -H 10.0.202.22 -U USERID -P PASSW0RD -p 623 -L ADMINISTRATOR chassis power status

0 Chassis Power is on

Status: ON

测试能否使用命令行将节点fence，fence之后的设备将自动重新启动

root@fmprpoora ~]# pcs stonith fence fmprmaora-priv

Node: fmprmaora-priv fenced

[root@fmprmaora ~]# pcs stonith fence fmprpoora-priv

Node: fmprpoora-priv fenced

2.3.8 创建浮动IP

创建浮动IP地址资源

[root@fmprfcora pcsd]# pcs resource create fmprfcora-vip ocf:heartbeat:IPaddr2 ip=10.0.101.12 cidr_netmask=24 op monitor interval=30s

[root@fmprfcora pcsd]# pcs resource create fmpracora-vip ocf:heartbeat:IPaddr2 ip=10.0.101.11 cidr_netmask=24 op monitor interval=30s

[root@fmprfcora pcsd]# pcs resource show --full

Resource: fmprfcora-vip (class=ocf provider=heartbeat type=IPaddr2)

Attributes: ip=10.0.101.12 cidr_netmask=24

Operations: start interval=0s timeout=20s (fmprfcora-vip-start-interval-0s)

stop interval=0s timeout=20s (fmprfcora-vip-stop-interval-0s)

monitor interval=30s (fmprfcora-vip-monitor-interval-30s)

Resource: fmpracora-vip (class=ocf provider=heartbeat type=IPaddr2)

Attributes: ip=10.0.101.11 cidr_netmask=24

Operations: start interval=0s timeout=20s (fmpracora-vip-start-interval-0s)

stop interval=0s timeout=20s (fmpracora-vip-stop-interval-0s)

monitor interval=30s (fmpracora-vip-monitor-interval-30s)

2.3.9 创建文件系统资源

文件系统和挂载点需要已经创建好，并在集群中的各节点依次挂载完成。接下来在集群的文件系统资源。

[root@fmpracdb pcsd]# pcs resource create ac_fs Filesystem \

>device="/dev/mapper/acvg-aclv" directory="/ac-oradata" fstype="ext4"

[root@fmpracdb pcsd]# pcs resource create fc_fs Filesystem \

>device="/dev/mapper/fcvg-fclv" directory="/fc-oradata" fstype="ext4"

显示文件系统资源的配置

[root@fmpracora dinxin]# pcs resource show ac_fs

Resource: ac_fs (class=ocf provider=heartbeat type=Filesystem)

Attributes: device=/dev/mapper/acvg-aclv directory=/ac-oradata fstype=ext4

Operations: start interval=0s timeout=60 (ac_fs-start-interval-0s)

stop interval=0s timeout=60 (ac_fs-stop-interval-0s)

monitor interval=20 timeout=40 (ac_fs-monitor-interval-20)

[root@fmpracora dinxin]# pcs resource show fc_fs

Resource: fc_fs (class=ocf provider=heartbeat type=Filesystem)

Attributes: device=/dev/mapper/fcvg-fclv directory=/fc-oradata fstype=ext4

Operations: start interval=0s timeout=60 (fc_fs-start-interval-0s)

stop interval=0s timeout=60 (fc_fs-stop-interval-0s)

monitor interval=20 timeout=40 (fc_fs-monitor-interval-20)

2.3.10 创建数据库lsnr和db资源

确认当前pcs版本支持Oracle数据库。以下表示支持oracle数据库

[root@fmpracora dinxin]# pcs resource list |grep ora

ocf:heartbeat:oracle - Manages an Oracle Database instance

ocf:heartbeat:oralsnr - Manages an Oracle TNS listener

确认oracle数据库和listener已经创建好，并且手工能够在集群中的节点正常启动到open状态。

确保能解析数据库sid,

[oracle@fmpracora admin]$ tnsping cvs

TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 23-AUG-2016 13:51:06

Used parameter files:

Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fmpracora-vip)(PORT = 1526)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = cvs)))

OK (0 msec)

[oracle@fmpracora admin]$ tnsping arap2

TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 23-AUG-2016 13:51:10

Used parameter files:

Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fmprfcora-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = arap2)))

OK (0 msec)

当前的oracle listener.ora和tnsnames.ora文件如下：

CVS =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = fmpracora-vip)(PORT = 1526))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = cvs)

)

ARAP2 =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = fmprfcora-vip)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = arap2)

)

[oracle@fmpracora admin]$ cat listener.ora

LSNR_arap2=

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = fmprfcora-vip)(PORT = 1521))

)

SECURE_REGISTER_LSNR_arap2 = (TCP)

LSNR_cvs=

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = fmpracora-vip)(PORT = 1526))

)

SECURE_REGISTER_LSNR_cvs = (TCP)

使用以下命令创建oracle数据库资源和listnener资源

[root@fmpracdb pcsd]# pcs resource create arap2db oracle sid=arap2 home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate

[root@fmpracdb pcsd]# pcs resource create arap2_lsn oralsnr sid=arap2 home=/home/app/11.2/db_1 user=oracle listener=LSNR_arap2

[root@fmpracdb pcsd]# pcs resource create cvsdb oracle sid=cvs home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate

[root@fmpracdb pcsd]# pcs resource create cvs_lsn oralsnr sid=cvs home=/home/app/11.2/db_1 user=oracle listener=LSNR_cvs2

使用以下命令查看当前数据库资源和listener资源配置

[root@fmpracora dinxin]# pcs resource show cvs_lsn

Resource: cvs_lsn (class=ocf provider=heartbeat type=oralsnr)

Attributes: sid=cvs home=/home/app/11.2/db_1 user=oracle listener=LSNR_cvs

Operations: start interval=0s timeout=120 (cvs_lsn-start-interval-0s)

stop interval=0s timeout=120 (cvs_lsn-stop-interval-0s)

monitor interval=10 timeout=30 (cvs_lsn-monitor-interval-10)

[root@fmpracora dinxin]# pcs resource show cvsdb

Resource: cvsdb (class=ocf provider=heartbeat type=oracle)

Attributes: sid=cvs home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate

Operations: start interval=0s timeout=120 (cvsdb-start-interval-0s)

stop interval=0s timeout=120 (cvsdb-stop-interval-0s)

monitor interval=120 timeout=30 (cvsdb-monitor-interval-120)

[root@fmpracora dinxin]# pcs resource show arap2db

Resource: arap2db (class=ocf provider=heartbeat type=oracle)

Attributes: sid=arap2 home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate

Operations: start interval=0s timeout=120 (arap2db-start-interval-0s)

stop interval=0s timeout=120 (arap2db-stop-interval-0s)

monitor interval=120 timeout=30 (arap2db-monitor-interval-120)

[root@fmpracora dinxin]# pcs resource show arap2_lsn

Resource: arap2_lsn (class=ocf provider=heartbeat type=oralsnr)

Attributes: sid=arap2 home=/home/app/11.2/db_1 user=oracle listener=LSNR_arap2

Operations: start interval=0s timeout=120 (arap2_lsn-start-interval-0s)

stop interval=0s timeout=120 (arap2_lsn-stop-interval-0s)

monitor interval=10 timeout=30 (arap2_lsn-monitor-interval-10)

[root@fmpracora dinxin]#

2.3.11 创建集群组

创建集群组，在组中添加浮动IP地址资源，文件系统资源，oracle listener资源，以及Oracle数据库资源，并设置资源间的依赖关系。组设置成功后，将作为一个整体进行节点间的切换。

# pcs resource group add ac_group fmpracora-vip

#pcs resource group add ac_group ac_fs --after fmpracora-vip

#pcs resource group add ac_group cvs_lsn --after ac_fs

#pcs resource group add ac_group cvsdb --after cvs_lsn

2.3.12 创建集群资源优先启动节点

集群节点将在优先节点启动。当优先节点因故障关机后，自动切换到其它节点。在优先节点开机重新启动后，将跟随cman,pcs服务启动而自动回切到优先节点。

#pcs constraint location fc_fs prefers fmprfcora-priv

#pcs constraint location fmprfcora-vip prefers fmprfcora-priv

#pcs constraint location cvs_lsn prefers fmprfcora-priv

#pcs constraint location cvsdb prefers fmprfcora-priv

#pcs constraint location fc_group prefers fmprfcora-priv

#pcs constraint location ac_fs prefers fmpracora-priv

#pcs constraint location fmpracora-vip prefers fmpracora-priv

#pcs constraint location apar2_lsn prefers fmpracora-priv

#pcs constraint location apar2db prefers fmpracora-priv

#pcs constraint location ac_group prefers fmpracora-priv

查看优先节点：

[root@fmpracora dinxin]# pcs constraint

Location Constraints:

Resource: ac_group

Enabled on: fmpracora-priv (score:INFINITY)

Resource: fc_group

Enabled on: fmprfcora-priv (score:INFINITY)

Resource: fmpracora-vip

Enabled on: fmpracora-priv (score:INFINITY)

Resource: fmprfcora-vip

Enabled on: fmprfcora-priv (score:INFINITY)

3RHEL 6.8 & 7.2 HA维护手册

3.1移动资源组到其它节点

[root@fmdrjdeora ~]# pcs resource move dr_group

Warning: Creating location constraint cli-ban-dr_group-on-fmdrjdeora-priv with a score of -INFINITY for resource dr_group on node fmdrjdeora-priv.

This will prevent dr_group from running on fmdrjdeora-priv until the constraint is removed. This will be the case even if fmdrjdeora-priv is the last node in the cluster.

服务迁移到其它节点。

[root@fmdrjdeora ~]# pcs resource

Resource Group: dr_group

fmdrjdeora-vip(ocf::heartbeat:IPaddr2):Started fmprjdeora-priv

dr_fs(ocf::heartbeat:Filesystem):Started fmprjdeora-priv

jdedbdr_lsn(ocf::heartbeat:oralsnr):Started fmprjdeora-priv

jdedr_db(ocf::heartbeat:oracle):Started fmprjdeora-priv

Resource Group: pr_group

pr_fs(ocf::heartbeat:Filesystem):Started fmprjdeora-priv

fmprjdeora-vip(ocf::heartbeat:IPaddr2):Started fmprjdeora-priv

jdedb(ocf::heartbeat:oracle):Started fmprjdeora-priv

jde_lsn(ocf::heartbeat:oralsnr):Started fmprjdeora-priv

且生成一个在节点fmdrjdeora-priv禁用

[root@fmdrjdeora oracle]# pcs constraint

Location Constraints:

Resource: dr_fs

Enabled on: fmdrjdeora-priv (score:INFINITY)

Resource: dr_group

Enabled on: fmdrjdeora-priv (score:300)

Enabled on: fmprjdeora-priv (score:100)

Disabled on: fmdrjdeora-priv (score:-INFINITY) (role: Started)

3.2回切该资源回到原优先节点

需要清除限制，否则该资源不会fmdrjdeora-priv运行。

[root@fmprjdeora ~]# pcs resource clear dr_group

清除资源后，constraint恢复正常。

[root@fmprjdeora ~]# pcs constraint

Location Constraints:

Resource: dr_fs

Enabled on: fmdrjdeora-priv (score:INFINITY)

Resource: dr_group

Enabled on: fmdrjdeora-priv (score:300)

Enabled on: fmprjdeora-priv (score:100)

Resource: fmdrjdeora-vip

Enabled on: fmdrjdeora-priv (score:300)

Resource: fmprjdeora-vip