3.3 Oracle Automatic Storage Management Storage Configuration
Review the following sections to configure storage for Oracle Automatic Storage Management:
-
Configuring Storage for Oracle Automatic Storage Management
-
Configuring Oracle Automatic Storage Management Cluster File System
3.3.1 Configuring Storage for Oracle Automatic Storage Management
This section describes how to configure storage for use with Oracle Automatic Storage Management.
3.3.1.1 Identifying Storage Requirements for Oracle Automatic Storage Management
To identify the storage requirements for using Oracle ASM, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:
-
Determine whether you want to use Oracle ASM for Oracle Clusterware files (OCR and voting disks), Oracle Database files, recovery files, or all files except for Oracle Clusterware or Oracle Database binaries. Oracle Database files include data files, control files, redo log files, the server parameter file, and the password file.
Note:
You do not have to use the same storage mechanism for Oracle Clusterware, Oracle Database files and recovery files. You can use a shared file system for one file type and Oracle ASM for the other.If you choose to enable automated backups and you do not have a shared file system available, then you must choose Oracle ASM for recovery file storage.
If you enable automated backups during the installation, then you can select Oracle ASM as the storage mechanism for recovery files by specifying an Oracle Automatic Storage Management disk group for the Fast Recovery Area. If you select a noninteractive installation mode, then by default it creates one disk group and stores the OCR and voting disk files there. If you want to have any other disk groups for use in a subsequent database install, then you can choose interactive mode, or run ASMCA (or a command line tool) to create the appropriate disk groups before starting the database install.
-
Choose the Oracle ASM redundancy level to use for the Oracle ASM disk group.
The redundancy level that you choose for the Oracle ASM disk group determines how Oracle ASM mirrors files in the disk group and determines the number of disks and amount of free disk space that you require, as follows:
-
External redundancy
An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.
For Oracle Clusterware files, External redundancy disk groups provide 1 voting disk file, and 1 OCR, with no copies. You must use an external technology to provide mirroring for high availability.
Because Oracle ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use external redundancy with storage devices such as RAID, or other similar devices that provide their own data protection mechanisms.
-
Normal redundancy
In a normal redundancy disk group, to increase performance and reliability, Oracle ASM by default uses two-way mirroring. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.
For Oracle Clusterware files, Normal redundancy disk groups provide 3 voting disk files, 1 OCR and 2 copies (one primary and one secondary mirror). With normal redundancy, the cluster can survive the loss of one failure group.
For most installations, Oracle recommends that you select normal redundancy.
-
High redundancy
In a high redundancy disk group, Oracle ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.
For Oracle Clusterware files, High redundancy disk groups provide 5 voting disk files, 1 OCR and 3 copies (one primary and two secondary mirrors). With high redundancy, the cluster can survive the loss of two failure groups.
While high redundancy disk groups do provide a high level of data protection, you should consider the greater cost of additional storage devices before deciding to select high redundancy disk groups.
-
-
Determine the total amount of disk space that you require for Oracle Clusterware files, and for the database files and recovery files.
Use Table 3-5 and Table 3-6 to determine the minimum number of disks and the minimum disk space requirements for installing Oracle Clusterware files, and installing the starter database, where you have voting disks in a separate disk group:
Table 3-5 Total Oracle Clusterware Storage Space Required by Redundancy Type
Redundancy Level Minimum Number of Disks Oracle Cluster Registry (OCR) Files Voting Disk Files Both File Types External
1
300 MB
300 MB
600 MB
Normal
3
600 MB
900 MB
1.5 GBFoot 1
High
5
900 MB
1.5 GB
2.4 GB
Footnote 1 If you create a disk group during installation, then it must be at least 2 GB.
Note:
If the voting disk files are in a disk group, be aware that disk groups with Oracle Clusterware files (OCR and voting disk files) have a higher minimum number of failure groups than other disk groups.If you create a disk group as part of the installation in order to install the OCR and voting disk files, then the installer requires that you create these files on a disk group with at least 2 GB of available space.
A quorum failure group is a special type of failure group and disks in these failure groups do not contain user data. A quorum failure group is not considered when determining redundancy requirements in respect to storing user data. However, a quorum failure group counts when mounting a disk group.
Table 3-6 Total Oracle Database Storage Space Required by Redundancy Type
Redundancy Level Minimum Number of Disks Database Files Recovery Files Both File Types External
1
1.5 GB
3 GB
4.5 GB
Normal
2
3 GB
6 GB
9 GB
High
3
4.5 GB
9 GB
13.5 GB
-
Determine an allocation unit size. Every Oracle ASM disk is divided into allocation units (AU). An allocation unit is the fundamental unit of allocation within a disk group. You can select the AU Size value from 1, 2, 4, 8, 16, 32 or 64 MB, depending on the specific disk group compatibility level. The default value is set to 1 MB.
-
For Oracle Clusterware installations, you must also add additional disk space for the Oracle ASM metadata. You can use the following formula to calculate the disk space requirements (in MB) for OCR and voting disk files, and the Oracle ASM metadata:
total = [2 * ausize * disks] + [redundancy * (ausize * (nodes * (clients + 1) + 30) + (64 * nodes) + 533)]
Where:
-
redundancy = Number of mirrors: external = 1, normal = 2, high = 3.
-
ausize = Metadata AU size in megabytes (default is 1 MB)
-
nodes = Number of nodes in cluster.
-
clients - Number of database instances for each node.
-
disks - Number of disks in disk group.
For example, for a four-node Oracle RAC installation, using three disks in a normal redundancy disk group, you require an additional 1684 MB of space:
[2 * 1 * 3] + [2 * (1 * (4 * (4 + 1)+ 30)+ (64 * 4)+ 533)] = 1684 MB
To ensure high availability of Oracle Clusterware files on Oracle ASM, for a normal redundancy disk group, as a general rule for most installations, you must have at least 2 GB of disk space for Oracle Clusterware files in three separate failure groups, with at least three physical disks. To ensure that the effective disk space to create Oracle Clusterware files is 2 GB, best practice suggests that you ensure at least 2.1 GB of capacity for each disk, with a total capacity of at least 6.3 GB for three disks.
-
-
Optionally, identify failure groups for the Oracle ASM disk group devices.
If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.
To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.
Note:
Define custom failure groups after installation, using the GUI tool ASMCA, the command line toolasmcmd
, or SQL commands.If you define custom failure groups, then for failure groups containing database files only, you must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.
For failure groups containing database files and clusterware files, including voting disks, you must specify a minimum of three failure groups for normal redundancy disk groups, and five failure groups for high redundancy disk groups.
Disk groups containing voting files must have at least 3 failure groups for normal redundancy or at least 5 failure groups for high redundancy. Otherwise, the minimum is 2 and 3 respectively. The minimum number of failure groups applies whether or not they are custom failure groups.
-
If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:
-
All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.
-
Do not specify multiple partitions on a single physical disk as a disk group device. Each disk group device should be on a separate physical disk.
-
Although you can specify a logical volume as a device in an Oracle ASM disk group, Oracle does not recommend their use because it adds a layer of complexity that is unnecessary with Oracle ASM. In addition, Oracle RAC requires a cluster logical volume manager in case you decide to use a logical volume with Oracle ASM and Oracle RAC.
Oracle recommends that if you choose to use a logical volume manager, then use the logical volume manager to represent a single LUN without striping or mirroring, so that you can minimize the impact of the additional storage layer.
-
3.3.1.2 Creating Files on a NAS Device for Use with Oracle ASM
If you have a certified NAS storage device, then you can create zero-padded files in an NFS mounted directory and use those files as disk devices in an Oracle ASM disk group.
To create these files, follow these steps:
-
If necessary, create an exported directory for the disk group files on the NAS device.
Refer to the NAS device documentation for more information about completing this step.
-
Switch user to
root
. -
Create a mount point directory on the local system. For example:
# mkdir -p /mnt/oracleasm
-
To ensure that the NFS file system is mounted when the system restarts, add an entry for the file system in the mount file
/etc/fstab
.See Also:
My Oracle Support note 359515.1 for updated NAS mount option information, available at the following URL:https://support.oracle.com
For more information about editing the mount file for the operating system, refer to the
man
pages. For more information about recommended mount options, refer to the section "Checking NFS Mount and Buffer Size Parameters for Oracle RAC". -
Enter a command similar to the following to mount the NFS file system on the local system:
# mount /mnt/oracleasm
-
Choose a name for the disk group to create. For example:
sales1
. -
Create a directory for the files on the NFS file system, using the disk group name as the directory name. For example:
# mkdir /mnt/oracleasm/nfsdg
-
Use commands similar to the following to create the required number of zero-padded files in this directory:
# dd if=/dev/zero of=/mnt/oracleasm/nfsdg/disk1 bs=1024k count=1000 oflag=direct
This example creates 1 GB files on the NFS file system. You must create one, two, or three files respectively to create an external, normal, or high redundancy disk group.
-
Enter commands similar to the following to change the owner, group, and permissions on the directory and files that you created, where the installation owner is
grid
, and the OSASM group isasmadmin
:# chown -R grid:asmadmin /mnt/oracleasm # chmod -R 660 /mnt/oracleasm
-
If you plan to install Oracle RAC or a standalone Oracle Database, then during installation, edit the Oracle ASM disk discovery string to specify a regular expression that matches the file names you created. For example:
/mnt/oracleasm/sales1/
Note:
During installation, disk paths mounted on Oracle ASM and registered on ASMLIB with the string ORCL:* are listed as default database storage candidate disks.
3.3.1.3 Using an Existing Oracle ASM Disk Group
Select from the following choices to store either database or recovery files in an existing Oracle ASM disk group, depending on installation method:
-
If you select an installation method that runs Database Configuration Assistant in interactive mode, then you can decide whether you want to create a disk group, or to use an existing one.
The same choice is available to you if you use Database Configuration Assistant after the installation to create a database.
-
If you select an installation method that runs Database Configuration Assistant in noninteractive mode, then you must choose an existing disk group for the new database; you cannot create a disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.
Note:
The Oracle Automatic Storage Management instance that manages the existing disk group can be running in a different Oracle home directory.
To determine if an existing Oracle Automatic Storage Management disk group exists, or to determine if there is sufficient disk space in a disk group, you can use the Oracle ASM command line tool (asmcmd
), Oracle Enterprise Manager Grid Control or Database Control. Alternatively, you can use the following procedure:
-
View the contents of the
oratab
file to determine if an Oracle Automatic Storage Management instance is configured on the system:$ more /etc/oratab
If an Oracle Automatic Storage Management instance is configured on the system, then the
oratab
file should contain a line similar to the following:+ASM2:oracle_home_path
In this example,
+ASM2
is the system identifier (SID) of the Oracle Automatic Storage Management instance, with the node number appended, andoracle_home_path
is the Oracle home directory where it is installed. By convention, the SID for an Oracle Automatic Storage Management instance begins with a plus sign. -
Set the
ORACLE_SID
andORACLE_HOME
environment variables to specify the appropriate values for the Oracle Automatic Storage Management instance. -
Connect to the Oracle Automatic Storage Management instance and start the instance if necessary:
$ $ORACLE_HOME/bin/asmcmd ASMCMD> startup
-
Enter one of the following commands to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:
ASMCMD> lsdg
or:
$ORACLE_HOME/bin/asmcmd -p lsdg
-
From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.
-
If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.
Note:
If you are adding devices to an existing disk group, then Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.
3.3.1.4 Configuring Disks for Oracle ASM with ASMLIB
The Oracle Automatic Storage Management (Oracle ASM) library driver (ASMLIB) simplifies the configuration and management of the disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.
Without ASMLIB Linux 2.6 kernel and later, block device paths do not maintain permissions and path persistence unless you create a permissions
or rules
file on each cluster member node; block device paths that were /dev/sda
can appear as /dev/sdb
after a system restart. Adding new disks requires you to modify the udev
file to provide permissions and path persistence for the new disk.
With ASMLIB, you define the range of disks you want to have made available as Oracle ASM disks. ASMLIB maintains permissions and disk labels that are persistent on the storage device, so that label is available even after an operating system upgrade. You can update storage paths on all cluster member nodes by running one oracleasm
command on each node.
If you intend to use Oracle ASM on block devices for database storage for Linux, then Oracle recommends that you install the ASMLIB driver and associated utilities, and use them to configure the disks for Oracle ASM.
Caution:
On IBM: Linux on System z servers, due to a block size compatibility issue, you cannot use ASMLIB with SCSI storage devices and Fibre Channel Protocol (FCP) for Oracle Grid Infrastructure release 11.2.0.1 and later.
Workaround: use block device directly (for example, using paths similar to /dev/mapper/mpatha_part1
), or use DASD disks.
See Also:
My Oracle Support notes How to Manually Configure Disk Storage devices for use with Oracle ASM 11.2 on IBM: Linux on System z under SLES (Doc ID 1350008.1) and How to Manually Configure Disk Storage devices for use with Oracle ASM 11.2 on IBM: Linux on System z under Red Hat 5 (Doc ID 1351746.1), available at the following URL:
To use the Oracle Automatic Storage Management library driver (ASMLIB) to configure Oracle ASM devices, complete the following tasks.
-
Installing and Configuring the Oracle ASM Library Driver Software
-
Configuring Disk Devices to Use Oracle ASM Library Driver on x86 Systems
-
Configuring Disk Devices to Use ASM Library Driver on IBM: Linux on System z
Note:
To create a database during the installation using the Oracle ASM library driver, you must choose an installation method that runs ASMCA in interactive mode. You must also change the default disk discovery string to ORCL:*.
3.3.1.4.1 Installing and Configuring the Oracle ASM Library Driver Software
ASMLIB is already included with Unbreakable Enterprise Kernel packages, and with SUSE 11. If you are a member of the Unbreakable Linux Network, then you can install the ASMLIB rpms by subscribing to the Oracle Software for Enterprise Linux channel, and using up2date
to retrieve the most current package for your system and kernel. For additional information, refer to the following URL:
http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html
To install and configure the ASMLIB driver software manually, follow these steps:
-
Enter the following command to determine the kernel version and architecture of the system:
# uname -rm
-
Download the required ASMLIB packages from the Oracle Technology Network (OTN) Web site:
http://www.oracle.com/technetwork/server-storage/linux/downloads/index-088143.html
Note:
You must installoracleasm-support
package version 2.0.1 or later to use ASMLIB on Red Hat Enterprise Linux 5 Advanced Server. ASMLIB is already included with SUSE distributions.Tip:
My Oracle Support note 1089399.1 for information about ASMLIB support with Red Hat distributions:https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1089399.1
You must install the following packages, where
version
is the version of the ASMLIB driver,arch
is the system architecture, andkernel
is the version of the kernel that you are using:oracleasm-support-version.arch.rpm oracleasm-kernel-version.arch.rpm oracleasmlib-version.arch.rpm
-
Switch user to the
root
user:$ su -
-
Enter a command similar to the following to install the packages:
# rpm -ivh oracleasm-support-version.arch.rpm \ oracleasm-kernel-version.arch.rpm \ oracleasmlib-version.arch.rpm
For example, if you are using the Red Hat Enterprise Linux 5 AS kernel on an AMD64 system, then enter a command similar to the following:
# rpm -ivh oracleasm-support-2.1.3-1.el5.x86_64.rpm \ oracleasm-2.6.18-194.26.1.el5xen-2.0.5-1.el5.x86_64.rpm \ oracleasmlib-2.0.4-1.el5.x86_64.rpm
-
Enter the following command to run the
oracleasm
initialization script with theconfigure
option:# /usr/sbin/oracleasm configure -i
Note:
Theoracleasm
command in/usr/sbin
is the command you should use. The/etc/init.d
path is not deprecated, but theoracleasm
binary in that path is now used typically for internal commands. -
Enter the following information in response to the prompts that the script displays:
Prompt Suggested Response Default user to own the driver interface: Standard groups and users configuration: Specify the Oracle software owner user (for example, oracle
)Job role separation groups and users configuration:Specify the Grid Infrastructure software owner (for example,
grid
)Default group to own the driver interface: Standard groups and users configuration: Specify the OSDBA group for the database (for example, dba
).Job role separation groups and users configuration:Specify the OSASM group for storage administration (for example,
asmadmin
).Start Oracle ASM Library driver on boot (y/n): Enter y
to start the Oracle Automatic Storage Management library driver when the system starts.Scan for Oracle ASM disks on boot (y/n) Enter y
to scan for Oracle ASM disks when the system starts.The script completes the following tasks:
-
Creates the
/etc/sysconfig/oracleasm
configuration file -
Creates the
/dev/oracleasm
mount point -
Mounts the ASMLIB driver file system
Note:
The ASMLIB driver file system is not a regular file system. It is used only by the Oracle ASM library to communicate with the Oracle ASM driver.
-
-
Enter the following command to load the
oracleasm
kernel module:# /usr/sbin/oracleasm init
-
Repeat this procedure on all nodes in the cluster where you want to install Oracle RAC.
3.3.1.4.2 Configuring Disk Devices to Use Oracle ASM Library Driver on x86 Systems
To configure the disk devices to use in an Oracle ASM disk group, follow these steps:
-
If you intend to use IDE, SCSI, or RAID devices in the Oracle ASM disk group, then follow these steps:
-
If necessary, install or configure the shared disk devices that you intend to use for the disk group and restart the system.
-
To identify the device name for the disks to use, enter the following command:
# /sbin/fdisk -l
Depending on the type of disk, the device name can vary:
To include devices in a disk group, you can specify either whole-drive device names or partition device names.
Note:
Oracle recommends that you create a single whole-disk partition on each disk. -
Use either
fdisk
orparted
to create a single whole-disk partition on the disk devices.
-
-
Enter a command similar to the following to mark a disk as an Oracle ASM disk:
# /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1
In this example,
DISK1
is the name you assign to the disk.Note:
The disk names that you specify can contain uppercase letters, numbers, and the underscore character. They must start with an uppercase letter.If you are using a multi-pathing disk driver with Oracle ASM, then make sure that you specify the correct logical device name for the disk.
-
To make the disk available on the other nodes in the cluster, enter the following command as
root
on each node:# /usr/sbin/oracleasm scandisks
This command identifies shared disks attached to the node that are marked as Oracle ASM disks.
3.3.1.4.3 Configuring Disk Devices to Use ASM Library Driver on IBM: Linux on System z
-
If you formatted the DASD with the compatible disk layout, then enter a command similar to the following to create a single whole-disk partition on the device:
# /sbin/fdasd -a /dev/dasdxxxx
-
Enter a command similar to the following to mark a disk as an ASM disk:
# /etc/init.d/oracleasm createdisk DISK1 /dev/dasdxxxx
In this example,
DISK1
is a name that you want to assign to the disk.Note:
The disk names that you specify can contain uppercase letters, numbers, and the underscore character. They must start with an uppercase letter.If you are using a multi-pathing disk driver with ASM, then make sure that you specify the correct logical device name for the disk.
-
To make the disk available on the other cluster nodes, enter the following command as root on each node:
# /etc/init.d/oracleasm scandisks
This command identifies shared disks attached to the node that are marked as ASM disks.
3.3.1.4.4 Administering the Oracle ASM Library Driver and Disks
To administer the Oracle Automatic Storage Management library driver (ASMLIB) and disks, use the oracleasm
initialization script with different options, as described in Table 3-7.
Table 3-7 ORACLEASM Script Options
Option | Description |
---|---|
configure |
Use the # /usr/sbin/oracleasm configure -i To see command options, enter |
enable disable |
Use the # /usr/sbin/oracleasm enable |
start stop restart |
Use the # /usr/sbin/oracleasm restart |
createdisk |
Use the # /usr/sbin/oracleasm createdisk DISKNAME devicename |
deletedisk |
Use the # /usr/sbin/oracleasm deletedisk DISKNAME Caution: Do not use this command to unmark disks that are being used by an Oracle Automatic Storage Management disk group. You must delete the disk from the Oracle Automatic Storage Management disk group before you unmark it. |
querydisk |
Use the # /usr/sbin/oracleasm querydisk {DISKNAME | devicename} |
listdisks |
Use the # /usr/sbin/oracleasm listdisks |
scandisks |
Use the # /usr/sbin/oracleasm scandisks |
3.3.1.5 Configuring ASMLIB for Multipath Disks
Additional configuration is required to use the Oracle Automatic Storage Management library Driver (ASMLIB) with third party vendor multipath disks.
See Also:
My Oracle Support site for updates to supported storage options:
https://support.oracle.com
3.3.1.5.1 About Using Oracle ASM with Multipath Disks
Oracle ASM requires that each disk is uniquely identified. If the same disk appears under multiple paths, then it causes errors. In a multipath disk configuration, the same disk can appear three times:
-
The initial path to the disk
-
The second path to the disk
-
The multipath disk access point.
For example: If you have one local disk, /dev/sda
, and one disk attached with external storage, then your server shows two connections, or paths, to that external storage. The Linux SCSI driver shows both paths. They appear as /dev/sdb
and /dev/sdc
. The system may access either /dev/sdb
or /dev/sdc
, but the access is to the same disk.
If you enable multipathing, then you have a multipath disk (for example, /dev/multipatha
), which can access both /dev/sdb
and /dev sdc
; any I/O to multipatha
can use either the sdb
or sdc
path. If a system is using the /dev/sdb
path, and that cable is unplugged, then the system shows an error. But the multipath disk will switch from the /dev/sdb
path to the /dev/sdc
path.
Most system software is unaware of multipath configurations. They can use any paths (sdb
, sdc
or multipatha
). ASMLIB also is unaware of multipath configurations.
By default, ASMLIB recognizes the first disk path that Linux reports to it, but because it imprints an identity on that disk, it recognizes that disk only under one path. Depending on your storage driver, it may recognize the multipath disk, or it may recognize one of the single disk paths.
Instead of relying on the default, you should configure Oracle ASM to recognize the multipath disk.
3.3.1.5.2 Disk Scan Ordering
The ASMLIB configuration file is located in the path /etc/sysconfig/oracleasm
. It contains all the startup configuration you specified with the command /etc/init.d/oracleasm configure
. That command cannot configure scan ordering.
The configuration file contains many configuration variables. The ORACLEASM_SCANORDER
variable specifies disks to be scanned first. The ORACLEASM_SCANEXCLUDE
variable specifies the disks that are to be ignored.
Configure values for ORACLEASM_SCANORDER
using space-delimited prefix strings. A prefix string is the common string associated with a type of disk. For example, if you use the prefix string sd
, then this string matches all SCSI devices, including /dev/sda
, /dev/sdb
, /dev/sdc
and so on. Note that these are not globs. They do not use wild cards. They are simple prefixes. Also note that the path is not a part of the prefix. For example, the /dev/
path is not part of the prefix for SCSI disks that are in the path /dev/sd
*.
For Oracle Linux and Red Hat Enterprise Linux version 5, when scanning, the kernel sees the devices as /dev/mapper/
XXX
entries. By default, the 2.6 kernel device file naming scheme udev
creates the /dev/mapper/
XXX
names for human readability. Any configuration using ORACLEASM_SCANORDER
should use the /dev/mapper/
XXX
entries.
3.3.1.5.3 Configuring Disk Scan Ordering to Select Multipath Disks
To configure ASMLIB to select multipath disks first, complete the following procedure:
-
Using a text editor, open the ASMLIB configuration file
/etc/sysconfig/oracleasm
. -
Edit the ORACLEASM_SCANORDER variable to provide the prefix path of the multipath disks. For example, if the multipath disks use the prefix
multipath
(/dev/mapper/multipatha
,/dev/mapper/multipathb
and so on), and the multipath disks mount SCSI disks, then provide a prefix path similar to the following:ORACLEASM_SCANORDER="multipath sd"
-
Save the file.
When you have completed this procedure, then when ASMLIB scans disks, it first scans all disks with the prefix string multipath, and labels these disks as Oracle ASM disks using the /dev/mapper/multipath
X
value. It then scans all disks with the prefix string sd
. However, because ASMLIB recognizes that these disks have already been labeled with the /dev/mapper/multipath
string values, it ignores these disks. After scanning for the prefix strings multipath
and sd
, Oracle ASM then scans for any other disks that do not match the scan order.
In the example in step 2, the key word multipath is actually the alias for multipath devices configured in /etc/multipath.conf
under the multipaths
section. For example:
multipaths { multipath { wwid 3600508b4000156d700012000000b0000 alias multipath ... } multipath { ... alias mympath ... } ... }
The default device name is in the format /dev/mapper/mpath* (or a similar path).
3.3.1.5.4 Configuring Disk Order Scan to Exclude Single Path Disks
To configure ASMLIB to exclude particular single path disks, complete the following procedure:
-
Using a text editor, open the ASMLIB configuration file
/etc/sysconfig/oracleasm
. -
Edit the
ORACLEASM_SCANEXCLUDE
variable to provide the prefix path of the single path disks. For example, if you want to exclude the single path disks/dev sdb
and/dev/sdc
, then provide a prefix path similar to the following:ORACLEASM_SCANEXCLUDE="sdb sdc"
-
Save the file.
When you have completed this procedure, then when ASMLIB scans disks, it scans all disks except for the disks with the sdb
and sdc
prefixes, so that it ignores /dev/sdb
and /dev/sdc
. It does not ignore other SCSI disks, nor multipath disks. If you have a multipath disk (for example, /dev/multipatha
), which accesses both /dev/sdb
and /dev sdc
, but you have configured ASMLIB to ignore sdb
and sdc
, then ASMLIB ignores these disks and instead marks only the multipath disk as an Oracle ASM disk.
3.3.1.6 Configuring Disk Devices Manually for Oracle ASM
By default, the 2.6 kernel device file naming scheme udev
dynamically creates device file names when the server is started, and assigns ownership of them to root
. If udev
applies default settings, then it changes device file names and owners for voting disks or Oracle Cluster Registry partitions, corrupting them when the server is restarted. For example, a voting disk on a device named /dev/sdd
owned by the user grid
may be on a device named /dev/sdf
owned by root
after restarting the server.If you use ASMLIB, then you do not need to ensure permissions and device path persistency in udev
.
If you do not use ASMLIB, then you must create a custom rules file. When udev
is started, it sequentially carries out rules (configuration directives) defined in rules files. These files are in the path /etc/udev/rules.d/
. Rules files are read in lexical order. For example, rules in the file 10-wacom.rules
are parsed and carried out before rules in the rules file 90-ib.rules
.
When specifying the device information in the UDEV rules file, ensure that the OWNER, GROUP and MODE are specified before any other characteristics in the order shown. For example, if you want to include the characteristic ACTION on the UDEV line, then specify ACTION after OWNER, GROUP, and MODE.
Where rules files describe the same devices, on the supported Linux kernel versions, the last file read is the one that is applied.
To configure a permissions file for disk devices, complete the following tasks:
-
To obtain information about existing block devices, run the command
scsi_id
(/sbin/scsi_id
) on storage devices from one cluster node to obtain their unique device identifiers. When running thescsi_id
command with the-s
argument, the device path and name passed should be that relative to thesysfs
directory/sys
(for example,/block/device
) when referring to/sys/block/device
. For example:# /sbin/scsi_id -g -s /block/sdb/sdb1 360a98000686f6959684a453333524174 # /sbin/scsi_id -g -s /block/sde/sde1 360a98000686f6959684a453333524179
Record the unique SCSI identifiers of clusterware devices, so you can provide them when required.
Note:
The commandscsi_id
should return the same device identifier value for a given device, regardless of which node the command is run from. -
Configure SCSI devices as trusted devices (white listed), by editing the
/etc/scsi_id.config
file and addingoptions=-g
to the file. For example:# cat > /etc/scsi_id.config vendor="ATA",options=-p 0x80 options=-g
-
Using a text editor, create a UDEV rules file for the Oracle ASM devices, setting permissions to 0660 for the installation owner and the group whose members are administrators of the Oracle Grid Infrastructure software. For example, using the installation owner
grid
and using a role-based group configuration, with the OSASM groupasmadmin
:# vi /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="14f70656e66696c00000000", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd?2", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="14f70656e66696c00000001", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd?3", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="14f70656e66696c00000002", OWNER="grid", GROUP="asmadmin", MODE="0660"
-
Copy the
rules.d
file to all other nodes on the cluster. For example:# scp 99-oracle-asmdevices.rules root@node2:/etc/udev/rules.d/99-oracle-asmdevices.rules
-
Load updated block device partition tables on all member nodes of the cluster, using
/sbin/partprobe
devicename
. You must do this asroot
. -
Run the command
udevtest
(/sbin/udevtest
) to test the UDEV rules configuration you have created. The output should indicate that the block devices are available and the rules are applied as expected. For example:# udevtest /block/sdb/sdb1 main: looking at device '/block/sdb/sdb1' from subsystem 'block' udev_rules_get_name: add symlink 'disk/by-id/scsi-360a98000686f6959684a453333524174-part1' udev_rules_get_name: add symlink 'disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.887085-part1' udev_node_mknod: preserve file '/dev/.tmp-8-17', because it has correct dev_t run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17' run_program: '/lib/udev/vol_id' returned with status 4 run_program: '/sbin/scsi_id' run_program: '/sbin/scsi_id' (stdout) '360a98000686f6959684a453333524174' run_program: '/sbin/scsi_id' returned with status 0 udev_rules_get_name: rule applied, 'sdb1' becomes 'ocr1' udev_device_event: device '/block/sdb/sdb1' validate currently present symlinks udev_node_add: creating device node '/dev/ocr1', major = '8', minor = '17', mode = '0640', uid = '0', gid = '500' udev_node_add: creating symlink '/dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1' to '../../ocr1' udev_node_add: creating symlink '/dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085 -part1' to '../../ocr1' main: run: 'socket:/org/kernel/udev/monitor' main: run: '/lib/udev/udev_run_devd' main: run: 'socket:/org/freedesktop/hal/udev_event' main: run: '/sbin/pam_console_apply /dev/ocr1 /dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1 /dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085- part1'
In the example output, note that applying the rules renames OCR device
/dev/sdb1
to/dev/ocr1
. -
Enter the command to restart the UDEV service.
On Asianux, Oracle Linux 5, and RHEL5, the commands are:
# /sbin/udevcontrol reload_rules # /sbin/start_udev
On SUSE 10 and 11, the command is:
# /etc/init.d boot.udev restart
3.3.2 Using Disk Groups with Oracle Database Files on Oracle ASM
Review the following sections to configure Oracle Automatic Storage Management (Oracle ASM) storage for Oracle Clusterware and Oracle Database Files:
3.3.2.1 Identifying and Using Existing Oracle Database Diskgroups on Oracle ASM
The following section describes how to identify existing disk groups and determine the free disk space that they contain.
-
Optionally, identify failure groups for the Oracle ASM disk group devices.
If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.
To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.
Note:
If you define custom failure groups, then you must specify a minimum of two failure groups for normal redundancy and three failure groups for high redundancy.
3.3.2.2 Creating Diskgroups for Oracle Database Data Files
If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:
-
All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.
-
Do not specify multiple partitions on a single physical disk as a disk group device. Oracle ASM expects each disk group device to be on a separate physical disk.
-
Although you can specify a logical volume as a device in an Oracle ASM disk group, Oracle does not recommend their use because it adds a layer of complexity that is unnecessary with Oracle ASM. In addition, Oracle RAC requires a cluster logical volume manager in case you decide to use a logical volume with Oracle ASM and Oracle RAC.
3.3.3 Configuring Oracle Automatic Storage Management Cluster File System
Oracle ACFS is installed as part of an Oracle Grid Infrastructure installation (Oracle Clusterware and Oracle Automatic Storage Management) for 11g release 2 (11.2).
You can also create a General Purpose File System configuration of ACFS using ASMCA.
See Also:
Section 3.1.3, "General Information About Oracle ACFS" for supported deployment options
To configure Oracle ACFS for an Oracle Database home for an Oracle RAC database:
-
Install Oracle Grid Infrastructure for a cluster (Oracle Clusterware and Oracle Automatic Storage Management)
-
Change directory to the Oracle Grid Infrastructure home. For example:
$ cd /u01/app/11.2.0/grid
-
Ensure that the Oracle Grid Infrastructure installation owner has read and write permissions on the storage mountpoint you want to use. For example, if you want to use the mountpoint
/u02/acfsmounts/
:$ ls -l /u02/acfsmounts
-
Start Oracle ASM Configuration Assistant as the grid installation owner. For example:
./asmca
-
The Configure ASM: ASM Disk Groups page shows you the Oracle ASM disk group you created during installation. Click the ASM Cluster File Systems tab.
-
On the ASM Cluster File Systems page, right-click the Data disk, then select Create ACFS for Database Home.
-
In the Create ACFS Hosted Database Home window, enter the following information:
-
Database Home ADVM Volume Device Name: Enter the name of the database home. The name must be unique in your enterprise. For example:
dbase_01
-
Database Home Mountpoint: Enter the directory path for the mount point. For example:
/u02/acfsmounts/dbase_01
Make a note of this mount point for future reference.
-
Database Home Size (GB): Enter in gigabytes the size you want the database home to be.
-
Database Home Owner Name: Enter the name of the Oracle Database installation owner you plan to use to install the database. For example:
oracle
1 -
Database Home Owner Group: Enter the OSDBA group whose members you plan to provide when you install the database. Members of this group are given operating system authentication for the SYSDBA privileges on the database. For example:
dba1
-
Click OK when you have completed your entries.
-
-
Run the script generated by Oracle ASM Configuration Assistant as a privileged user (
root
). On an Oracle Clusterware environment, the script registers the ACFS as a resource managed by Oracle Clusterware. Registering ACFS as a resource helps Oracle Clusterware to mount the ACFS automatically in proper order when ACFS is used for an Oracle RAC database Home. -
During Oracle RAC installation, ensure that you or the DBA who installs Oracle RAC selects for the Oracle home the mount point you provided in the Database Home Mountpoint field (in the preceding example,
/u02/acfsmounts/dbase_01
).
See Also:
Oracle Automatic Storage Management Administrator's Guide for more information about configuring and managing your storage with Oracle ACFS
3.3.4 Upgrading Existing Oracle ASM Instances
If you have an Oracle ASM installation from a prior release installed on your server, or in an existing Oracle Clusterware installation, then you can use Oracle Automatic Storage Management Configuration Assistant (ASMCA, located in the path Grid_home
/bin
) to upgrade the existing Oracle ASM instance to 11g release 2 (11.2), and subsequently configure failure groups, Oracle ASM volumes and Oracle Automatic Storage Management Cluster File System (Oracle ACFS).
Note:
You must first shut down all database instances and applications on the node with the existing Oracle ASM instance before upgrading it.
During installation, if you are upgrading from an Oracle ASM release prior to 11.2, and you chose to use Oracle ASM and ASMCA detects that there is a prior Oracle ASM version installed in another Oracle ASM home, then after installing the Oracle ASM 11g release 2 (11.2) binaries, you can start ASMCA to upgrade the existing Oracle ASM instance. You can then choose to configure an Oracle ACFS deployment by creating Oracle ASM volumes and using the upgraded Oracle ASM to create the Oracle ACFS.
If you are upgrading from Oracle ASM 11g release 2 (11.2.0.1) or later, then Oracle ASM is always upgraded with Oracle Grid Infrastructure as part of the rolling upgrade, and ASMCA is started by the root scripts during upgrade. ASMCA cannot perform a separate upgrade of Oracle ASM from release 11.2.0.1 to 11.2.0.2.
On an existing Oracle Clusterware or Oracle RAC installation, if the prior version of Oracle ASM instances on all nodes is 11g release 1, then you are provided with the option to perform a rolling upgrade of Oracle ASM instances. If the prior version of Oracle ASM instances on an Oracle RAC installation are from a release prior to 11g release 1, then rolling upgrades cannot be performed. Oracle ASM on all nodes will be upgraded to 11g release 2 (11.2).