准备环境的时 ，要4个对外IP，2个对内IP
不超过2T,，一般都用OCFS
高端存储适合用ASM
linux10G安装的时候，安装的机器时间要小于等于（如果是等于要严格等于）第二个机器的时间（只有linux 10G会有这个问题），不然会报错，拷贝空值什么的。
出现此错误，说明是时间问题，
hosts里面要注意的地方：
1）127.0.0.1 那一行不能删除。
2）主机名要小写，不能大写。
3）不能加下划线。
1.安装系统，把所有的开发包全部安装上
关掉防火墙和SELinux
yum -y install compat-libstdc++*
yum -y install elfutils-libelf-devel*
yum -y install elfutils-libelf-devel-static*
yum -y install gcc*
yum -y install gcc-c++*
yum -y install glibc*
yum -y install glibc-common*
yum -y install glibc-devel*
yum -y install glibc-headers*
yum -y install kernel-headers*
yum -y install ksh*
yum -y install libaio*
yum -y install libaio-devel*
yum -y install libgcc*
yum -y install libgomp*
yum -y install libstdc++*
yum -y install libstdc++-devel*
yum -y install make*
yum -y install sysstat*
yum -y install unixODBC*
yum -y install unixODBC-devel*
yum -y install libXp*
2.配置核心参数
vi /etc/sysctl.conf
kernel.core_uses_pid = 1
fs.file-max = 65536
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144
kernel.shmmni = 4096
kernel.sem = 500 64000 100 128
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
修改完执行此命令 ： sysctl -p
vi /etc/profile
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
之后，执行:$ulimit验证一下.
vi /etc/csh.login
if ( $USER == "oracle" ) then
limit maxproc 16384
limit descriptors 65536
umask 022
endif
vi /etc/pam.d/login
session required pam_limits.so
vi /etc/security/limits.conf
oracle soft nofile 655360
oracle hard nofile 635360
oracle soft nproc 10240
oracle hard nproc 16384
3.配置网络
127.0.0.1 linux1 localhost.localdomain localhost
需要将其删除成如下（注意上面的linux1被删除）：
127.0.0.1 localhost.localdomain localhost
如果 RAC 节点名出现在回送地址中，您在 RAC 安装期间将接收到以下错误信息：
ORA-00603:ORACLE server session terminated by fatal error
或
ORA-29702:error occurred in Cluster Group Service operation
在RAC1中，需要配置两个网段（RAC2也一样）
192.168.56.101 是公有网段
10.10.10.11 是私有网段
在RAC1中，配置两个文件可以达到修改IP的目的（RAC2中也如此类似操作）
vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=08:00:27:65:d4:be
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
IPADDR=192.168.56.101
NETMASK=255.255.255.0
GATEWAY=192.168.56.1
vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth1
BOOTPROTO=static
ONBOOT=yes
HWADDR=08:00:27:5f:b7:85
IPADDR=10.10.10.11
NETMASK=255.255.255.0
之后，重启IP，使之生效。
service network restart
vi /etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
#rac1
192.168.56.101 rac1
192.168.56.111 rac1vip
10.10.10.11 rac1priv
#rac2
192.168.56.102 rac2
192.168.56.222 rac2vip
10.10.10.22 rac2priv
4.创建用户，用户组
/usr/sbin/groupadd -g 1000 oinstall
/usr/sbin/groupadd -g 1200 dba
/usr/sbin/useradd -u 555 -g oinstall -G dba oracle
两边的用户id，组id 要一样
mkdir -p /u01/app/oracle/product/10.2/db_1
mkdir -p /u01/app/oracle/product/10.2/crs
chown -R oracle:oinstall /u01
chmod -R 775 /u01
vi /home/oracle/.bash_profile
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2/crs
export ORACLE_TERM=xterm
export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export LD_LIBRARY_PATH
export PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:$PATH
5 . 配置互信（一定要是oracle用户，两个节点都要做。）
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa
把4个文件合成一个文件
注意权限要是是644
chmod 644 authorized_keys
在节点RAC2进行的操作
[oracle@RAC2 ~]$ ls -a
. .bash_history .bash_profile .emacs .mozilla .viminfo
.. .bash_logout .bashrc .kde .ssh .Xauthority
[oracle@RAC2 ~]$ cd .ssh
[oracle@RAC2 .ssh]$ ls
id_dsa id_dsa.pub id_rsa id_rsa.pub
[oracle@RAC2 .ssh]$ scp id_dsa.pub rac1:/home/oracle/.ssh/id_dsa.pub2
oracle@rac1's password:
id_dsa.pub 100% 601 0.6KB/s 00:00
[oracle@RAC2 .ssh]$ ls
id_dsa id_dsa.pub id_rsa id_rsa.pub known_hosts
[oracle@RAC2 .ssh]$ scp id_rsa.pub rac1:/home/oracle/.ssh/id_rsa.pub2
oracle@rac1's password:
id_rsa.pub
再到RAC1节点
cat id_dsa.pub id_dsa.pub2 id_rsa.pub id_rsa.pub2 > authorized_keys
scp authorized_keys rac2:/home/oracle/.ssh
之后要测试一下（如果都不需要密码，得到时间后。就说明ssh 对等性配置成功。）
[oracle@rac1 ~]$ ssh rac1 date
[oracle@rac1 ~]$ ssh rac2 date
Sat Sep 14 16:03:58 CST 2013
[oracle@rac1 ~]$ ssh rac1priv date
Sat Sep 14 16:04:59 CST 2013
[oracle@rac1 ~]$ ssh rac2priv date
The authenticity of host 'rac2priv (10.10.10.22)' can't be established.
RSA key fingerprint is db:ad:62:77:5a:b5:d6:8f:c6:55:a9:64:61:9b:39:bc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac2priv,10.10.10.22' (RSA) to the list of known hosts.
Sat Sep 14 16:05:09 CST 2013
[oracle@rac2 ~]$ ssh rac1 date
Sat Sep 14 16:06:54 CST 2013
[oracle@rac2 ~]$ ssh rac2 date
Sat Sep 14 16:09:30 CST 2013
[oracle@rac2 ~]$ ssh rac1priv date
Sat Sep 14 16:09:55 CST 2013
[oracle@rac2 ~]$ ssh rac2priv date
The authenticity of host 'rac2priv (10.10.10.22)' can't be established.
RSA key fingerprint is db:ad:62:77:5a:b5:d6:8f:c6:55:a9:64:61:9b:39:bc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac2priv,10.10.10.22' (RSA) to the list of known hosts.
Sat Sep 14 16:10:03 CST 2013
6.创建共享裸设备
首先在rac3和rac4添加一块共享存储盘。
[root@rac3 ~]# fdisk -l
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End BlocksId System
/dev/sda1* 1 13 10439183 Linux
/dev/sda2 14 10448281507+ 8e Linux LVM
Disk /dev/sdb: 16.8 GB, 16862150656 bytes
255 heads, 63 sectors/track, 2050 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
[root@rac4 ~]# fdisk -l
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks IdSystem
/dev/sda1 *1 13 10439183 Linux
/dev/sda2 14 10448281507+ 8e Linux LVM
Disk /dev/sdb: 16.8 GB, 16862150656 bytes
255 heads, 63 sectors/track, 2050 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
[root@rac3 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 2050.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): d
No partition is defined yet!
Command (m for help): n
Command action
eextended
pprimary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2050, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-2050, default 2050): 300m
Command (m for help): n
Command action
eextended
pprimary partition (1-4)
p
Partition number (1-4): 2
First cylinder (301-2050, default 301):
Using default value 301
Last cylinder or +size or +sizeM or +sizeK (301-2050, default 2050): 600m //注意在这里一定要注意600这个值，这里的分区取值是600-前面
// 的300 ，所以是300，而不是600。
Command (m for help): n
Command action
eextended
pprimary partition (1-4)
p
Partition number (1-4): 3
First cylinder (601-2050, default 601):
Using default value 601
Last cylinder or +size or +sizeM or +sizeK (601-2050, default 2050):
Using default value 2050
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@rac3 raw]# fdisk -l
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End BlocksId System
/dev/sda1 *1 13 10439183 Linux
/dev/sda2 14 10448281507+ 8e Linux LVM
Disk /dev/sdb: 16.8 GB, 16862150656 bytes
255 heads, 63 sectors/track, 2050 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End BlocksId System
/dev/sdb1 1 3002409718+ 83 Linux
/dev/sdb2 301 6002409750 83 Linux
/dev/sdb3 601 205011647125 83 Linux
[root@rac4 raw]# fdisk /dev/sdb
The number of cylinders for this disk is set to 2050.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@rac4 raw]# fdisk -l
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End BlocksId System
/dev/sda1 *1 13 10439183 Linux
/dev/sda2 14 10448281507+ 8e Linux LVM
Disk /dev/sdb: 16.8 GB, 16862150656 bytes
255 heads, 63 sectors/track, 2050 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End BlocksId System
/dev/sdb1 1 3002409718+ 83 Linux
/dev/sdb2 301 6002409750 83 Linux
/dev/sdb3 601 205011647125 83 Linux
[root@rac3 ~]# cd /etc/udev/rules.d
[root@rac3 rules.d]# ls
05-udev-early.rules 60-pcmcia.rules 90-alsa.rules
40-multipath.rules 60-raw.rules 90-dm.rules
50-udev.rules 60-wacom.rules 90-hal.rules
51-hotplug.rules 61-uinput-stddev.rules 95-pam-console.rules
60-libsane.rules 61-uinput-wacom.rules 98-kexec.rules
60-net.rules 85-pcscd_ccid.rules bluetooth.rules
[root@rac3 rules.d]# vi 60-raw.rules
[root@rac3 rules.d]# cat 60-raw.rules
# Enter raw device bindings here.
#
# An example would be:
# ACTION=="add", KERNEL=="sda", RUN+="/bin/raw /dev/raw/raw1 %N"
# to bind /dev/raw/raw1 to /dev/sda, or
# ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="1", RUN+="/bin/raw /dev/raw/raw2 %M %m"
# to bind /dev/raw/raw2 to the device with major 8, minor 1.
ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdb2", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sdb3", RUN+="/bin/raw /dev/raw/raw4 %N"
KERNEL=="raw[1-9]", OWNER="oracle", GROUP="oinstall", MODE="0660"
[root@rac3 rules.d]# start_udev
Starting udev: [ OK ]
[root@rac3 rules.d]# ls -l /dev/raw
total 0
crw-rw---- 1 oracle oinstall 162, 1 Sep 17 10:21 raw1
crw-rw---- 1 oracle oinstall 162, 3 Sep 17 10:21 raw3
crw-rw---- 1 oracle oinstall 162, 4 Sep 17 10:21 raw4
[root@rac4 rules.d]# fdisk /dev/sdb
The number of cylinders for this disk is set to 2050.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@rac4 dev]# cd /etc/udev/
[root@rac4 udev]# ls
makedev.drules.d scripts udev.conf
[root@rac4 udev]# cd rules.d
[root@rac4 rules.d]# ls
05-udev-early.rules 60-net.rules 61-uinput-wacom.rules 95-pam-console.rules
40-multipath.rules 60-pcmcia.rules 85-pcscd_ccid.rules 98-kexec.rules
50-udev.rules 60-raw.rules 90-alsa.rules bluetooth.rules
51-hotplug.rules 60-wacom.rules 90-dm.rules
60-libsane.rules 61-uinput-stddev.rules 90-hal.rules
[root@rac4 rules.d]# vi 60-raw.rules
[root@rac4 rules.d]# more 60-raw.rules
# Enter raw device bindings here.
#
# An example would be:
# ACTION=="add", KERNEL=="sda", RUN+="/bin/raw /dev/raw/raw1 %N"
# to bind /dev/raw/raw1 to /dev/sda, or
# ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="1", RUN+="/bin/raw /dev/raw/raw2 %M %m"
# to bind /dev/raw/raw2 to the device with major 8, minor 1.
ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdb2", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sdb3", RUN+="/bin/raw /dev/raw/raw4 %N"
KERNEL=="raw[1-9]", OWNER="oracle", GROUP="oinstall", MODE="0660" //生产环境中一般都要加上这一句
[root@rac4 rules.d]# start_udev
Starting udev:[ OK ]
[root@rac4 rules.d]# cd /dev/raw
[root@rac4 raw]# ls
raw1 raw3raw4
[root@rac4 raw]# ll
total 0
crw-rw---- 1 oracle oinstall 162, 1 Sep 17 10:32 raw1
crw-rw---- 1 oracle oinstall 162, 3 Sep 17 10:32 raw3
crw-rw---- 1 oracle oinstall 162, 4 Sep 17 10:32 raw4
7，安装集群软件
注意：出现如下的时候，一定要注意改变一下目录的路径
改成如下所示（不能放在db_1，要放在crs）：
当出现如下所示时
要改成如下所示：
当出现如下所示时
要改成如下所示
注意如下OCR和VOTE的存放方式：
到下面所示，会有提示执行两个脚本，两个节点都要执行这两个脚本
执行的顺序是：RAC3执行第一个脚本，RAC4执行第个脚本，RAC3执行第二个脚本，RAC4执行第二个脚本
出现如下，为正常：
执行上面的脚本，先别急着点OK，必须先执行以下步骤
用root用户
cd /u01/app/oracle/product/10.2/crs/bin
vi vipca加一行（两个节点都要做 ）
unset LD_ASSUME_KERNEL
./oifcfg getif //查看是否有信息
./oifcfg setif -global eth0/192.168.56.0:public
./oifcfg setif -global eth1/10.10.10.0:cluster_interconnect
./vipca //在第一个节点上做
如果在第一个节点（第二个节点没事）出现下面所示，说明raw没有清除干净。（一定要用dd命令才能删除，不能用rm命令）
ddif=/dev/zero of=/dev/raw/raw1 bs=8192 count=100
8，安装数据库软件
9，升级集群软件和数据库软件
注意：这两个软件包是是同一个。
先升级集群软件，再升级数据库软件。
注意下面的一定要看清楚是集群软件还是数据库软件。
这里下面的是集群软件。我们选择先升级集群软件。
到这一步，一定要注意看清提示上说的，按照上面的步骤来做。
一定要看清楚里面说什么，不能退出。里面会提示执行脚本。
第一个节点执行完这两个脚本，然后再到第二个节点执行两个脚本。
10，升级完成，配监听（配完监听才能建库）
11,建库
dbca建库过程中， 遇到如下，注意，选 Select All
注意这里下面选择的是ASM方式。
注意下面这里的选择是pfile
注意下面这里的选择
