Oracle11g RAC 实施手册

一 系统网络资源

1.1网络分配：

主机名 公网(public) 专网(private) 虚拟网(vip)

Rac1 IP IP IP

Rac2 IP IP IP

1.2 硬件资源

Cpu ：Intel(R) Xeon(TM) MP CPU 3.16GHz*2

内存：1G

硬盘： 20G

共享磁盘：6G

1.3 操作系统安装

1.3.1 操作系统版本：Red Hat Enterprise Linux AS release 4 (Nahant Update 4)

1.3.2 系统分区

分区信息 分区类型 分区大小

/boot ext3 100M

/ ext3 10240M

Swap swap 2048

/oracle ext3 10240M

/opt ext3 5120M

/tmp ext3 1024M

1.3.3 系统语言

简体中文，ENGLISH（USA）

1.3.4 安装模式

完全安装

二 数据安装资源

2.1 数据库安装包

2.1.1 ORACLE 11G软件包：

下载地址：

Oracle Database 11g Release 1 (11.1.0.6.0) for Linux x86

Oracle Database 11g Release 1 Client (11.1.0.6.0) for Linux x86

Oracle Database 11g Examples (formerly Companion)

Oracle Clusterware Release 1 (11.1.0.6.0) for Linux x86

Oracle Database Gateways 11g Release 1 (11.1.0.6.0) for Linux x86

2.1.2 ORACLE 11G-ASMlib工具包：

下载地址：/technology/software/tech/linux/asmlib/

2.1.3 系统补丁的下载

下载地址：/projects/compat-oracle/files/RedHat/

三 系统安装配置

3.1 系统参数配置

3.1.1 关闭没必要的服务

chkconfig --level 35 autofs off

chkconfig --level 35 acpid off

chkconfig –level 35 sendmail off

chkconfig –level 35 cups-config-daemon off

chkconfig –level 35 cpus off

chkconfig –level 35 xfs off

chkconfig –level 35 lm_sensors off

chkconfig –level gpm off

chkconfig –level openibd off

chkconfig –level iiim off

chkconfig –level pcmcia off

chkconfig –level cpuspeed off

chkconfig –level nfslock off

chkconfig –level ip6tables off

chkconfig –level rpcidmapd off

chkconfig –level apmd off

chkconfig –level sendmail off

chkconfig –level arptables_jf off

chkconifg –level microcode_ctl off

chkconfig –level rpcgssd off

3.1.2设置主机表

vi /etc/hosts

127.0.0.1 omain localhost

IP rac1

IP racp1

IP racv1

IP rac2

IP racp2

IP racv2

3.1.3 修改内核参数

vi /etc/

= 536870912

= 4096

= 2097152

= 250 32000 100 128

-max = 65536

_local_port_range = 1024 65000

_default = 4194304

_max = 4194304

_default = 4194304

_max = 4194304

= 60

3.1.4设置oracle登录使用资源数

vi /etc/security/

oracle soft nofile 65536

oracle hard nofile 65536

oracle soft nproc 16384

oracle hard nproc 16384

vi /etc/pam.d/login –末行加入以下

session required pam_ multiple open

3.1.5建立hangcheck模块加载

检测是否安装了hangcheck模块

[root@rac1]# find /lib/modules -name ""

/lib/modules//kernel/drivers/char/

设置模块为自动加载

su –

echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/

或者

echo " /sbin/modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/

检测模块是否运行

su -

modprobe hangcheck-timer

grep Hangcheck /var/log/messages | tail -2

3.1.6 禁用selinux

getsebool

getsebool: SELinux is disabled

vi /etc/selinux/config

SELINUX=disabled

3.1.7 添加oracle用户和用户组

mkdir –p /oracle/oracle

groupadd –g 502 dba

groupadd –g 501 oinstall

useradd –g 501 –G 502 –d /oracle/oracle oracle

passwd oracle

chown -R ll /oracle/oracle

3.1.8 修改主机名

vi /etc/sysconfig/network

HOSTNAME=rac1

# hostname rac1

Rac 2 上同样执行

3.2 设置oracle用户

vi .bash_profile

umask 022

export ORACLE_BASE=/oracle/oracle

export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db

export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs

export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin

#export ORACLE_SID=rac2

export ORACLE_SID=rac1

export NLS_LANG=AMERICAN_16gbk

export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"

export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin

export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin

export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin

export ORACLE_TERM=xterm

export TNS_ADMIN=$ORACLE_HOME/network/admin

export ORA_NLS10=$ORACLE_HOME/nls/data

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib

export LD_ASSUME_KERNEL=2.4.19 # for RHEL AS 4

export CLASSPATH=$ORACLE_HOME/JRE

export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib

export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib

export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib

export THREADS_FLAG=native

export TEMP=/tmp

export TMPDIR=/tmp

if [ $USER = "oracle" ]; then

if [ $SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

fi

3.3 配置ASM服务

3.3.1安装asm包

rpm -Uvh

安装asm包一定要和系统的内核相对应

3.3.2 配置asm服务

cd /etc/init.d/

[root@localhost init.d]# ./oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets ('[]'). Hitting without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Fix permissions of Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: [ OK ]

Creating /dev/oracleasm mount point: [ OK ]

Loading module "oracleasm": [ OK ]

Mounting ASMlib driver filesystem: [ OK ]

Scanning system for ASM disks: [ OK ]

3.4 建立ssh等效性（在oracle用户操作）

3.4.1 建立相关目录生成密钥和公钥

su - oracle

mkdir -p ~/.ssh

chmod 755 ~/.ssh

[oracle@rac1 ~]$ /usr/bin/ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/oracle/oracle//.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /oracle/oracle//.ssh/id_rsa.

Your public key has been saved in /oracle/oracle//.ssh/id_.

The key fingerprint is:

16:2b:45:27:24:ee:97:46:5e:72:43:f1:9f:df:4b:dd oracle@rac1

[oracle@rac1 ~]$ /usr/bin/ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/oracle/oracle//.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /oracle/oracle//.ssh/id_dsa.

Your public key has been saved in /oracle/oracle//.ssh/id_.

The key fingerprint is:

0a:d5:e7:6a:a1:11:de:e4:72:74:fc:5f:b2:63:5b:6b oracle@rac1

同等在RAC2上建立公钥和私钥

3.4.2 建立等效密钥

在第一个主机上，以 oracle 用户身份登录

cat ~/.ssh/id_ >> ~/.ssh/authorized_keys

cat ~/.ssh/id_ >> ~/.ssh/authorized_keys

cp ~/.ssh/authorized_keys ~/.ssh/ized_keys

第二台主机上，以oracle用户登录

cat ~/.ssh/id_ >> ~/.ssh/authorized_keys

cat ~/.ssh/id_ >> ~/.ssh/authorized_keys

cp ~/.ssh/authorized_keys ~/.ssh/ized_keys

相互交换公钥

scp ~/.ssh/ized_keys oracle@IP:

scp ~/.ssh/ized_keys oracle@IP:

RAC2 上执行：

cat ized_keys >> ~/.ssh/authorized_keys

RAC1上执行：

cat ized_keys >> ~/.ssh/authorized_keys

建立等效性（每台机器都执行）

ssh rac1

ssh rac2

ssh racp1

ssh racp2

exec /usr/bin/ssh-agent $SHELL

/usr/bin/ssh-add

四 配置共享磁盘

4.1 建立ISCSI的共享磁盘

4.1.1 编译iscsi-target的源包

mkdir –p /usr/src/iscsi-target

wget /iscsi/RPMS/

rpmbuild --rebuild

rpm -Uvh iscsitarget-kernel-smp-0.4.12-6_2.6.9_

4.1.2 建立分区

fdisk /dev/sda

生成：/dev/sda8

fdisk /dev/sdb

生成：/dev/sdb1

4.1.3 配置共享盘

ISCSI服务端配置：

vi /etc/

注释掉所有的其他

Target la:1

IncomingUser gfs secretsecret

OutgoingUser

Lun 0 Path=/dev/sdb1,Type=fileio

Alias iDISK0

#MaxConnections 6

Target la:2

IncomingUser gfs secretsecret

OutgoingUser

Lun 0 Path=/dev/sda8,Type=fileio

Alias iDISK1

#MaxConnections 6

启动服务：service iscsi-target restart

加入开机启动：chkconfig –level 2345 iscsi-target on

Iscsi客户端配置：

vi /etc/

DiscoveryAddress=192.168.1.156

OutgoingUserName=gfs

OutgoingPassword=secretsecret

LoginTimeout=15

DiscoveryAddress=192.168.1.15

启动服务：service iscis restart

加入开机启动：chkconfig –level 2345 iscsi on

检查：iscsi共享磁盘：

[root@rac1 ~]# iscsi-ls

*******************************************************************************

SFNet iSCSI Driver Version ...4:0.1.11-3(02-May-2006)

*******************************************************************************

TARGET NAME : la:2

TARGET ALIAS :

HOST ID : 1

BUS ID : 0

TARGET ID : 0

TARGET ADDRESS : 192.168.1.156:3260,1

SESSION STATUS : ESTABLISHED AT Thu Oct 25 23:14:52 CST 2007

SESSION ID : ISID 00023d000001 TSIH 400

注：在从启系统时要注意iscsi-target服务和iscsi服务的启动顺序。先启动iscsi-target 在启动iscsi ，在/etc/rc3.d 下修改S后的数值，值约小启动级别越高

4.2 建立共享分区列表(一个节点操作即可)

分区信息 大小

/dev/sdc1 200M

/dev/sdc2 200M

/dev/sdc3 200M

/dev/sdc5 200M

/dev/sdc6 200M

/dev/sdc7 12000M

/dev/sdd1 2000M

/dev/sdd2 1000M

/dev/sdd3 1000M

/dev/sdd5 2600M

4.3 建立raw设备

vi /etc/sysconfig/rawdevices

/dev/raw/raw1 /dev/sdc1

/dev/raw/raw2 /dev/sdc2

/dev/raw/raw3 /dev/sdc3

/dev/raw/raw4 /dev/sdc5

/dev/raw/raw5 /dev/sdc6

启动服务：sevice rawdevices restart

开机启动：chkconfig –level 2345 rawdevices on

4.4 开机授权

vi /etc/

chown –R ll /dev/raw/raw*

chmod +755 /dev/raw/raw*

4.5建立asm磁盘卷组

service oracleasm createdisk FLASH /dev/sdd1

service oracleasm createdisk ARCH1 /dev/sdd2

service oracleasm createdisk ARCH2 /dev/sdd3

service oracleasm createdisk DATA /dev/sdd5

4.5.2

4.5.1 建立asm卷组（单机执行）

扫描asm卷组

service oracleasm scandisks

4.5.3 显示asm卷组

service oracleasm listdisks

五安装Cluster软件

注： 所有数据库软件都存放在/opt/oracle11g

5.1 cluster 硬件检测

./ stage -pre crsinst -n rac1,rac2 -verbose

5.1.1 解压cluster 软件包

unzip linux_11gR1_

安装补丁包

cd /opt/oracle11g/cluster/rpm

rpm -Uvh

安装系统补丁包

rpm -Uvh

第一步运行 $ ./runInstaller

su – oracle

cd /opt/oracle11g/cluster

./runInstaller

启动界面

指定crs主目录

检测系统资源

添加节点

设定网络使用（public、private）

指定OCR安装位置

指定Vo磁盘

指定安装内容开始安装

安装完毕

Root 在rac1、rac2 上依次运行

cd /oracle/oracle/

./

和

cd /oracle/oracle/product/11.1.0.6.0/crs

./

安装完毕

六安装数据库软件

6.1 安装数据库的软件

Su – oracle

Cd /opt/oracle11g/database

./runInstaller

安装界面

选择安装内容

安装目录的选择

集群模式安装

选择安装组建

选择操作组（默认）

安装软件

是否需求oracle注册管理

安装软件确定

开始安装

安装完毕

6.2 配置监听

su – oracle

netca

配置监听名（不能变）

七 建立实例、安装数据库、配置asm

su – oracle

dbca

7.1建立asm磁盘组

7.2 建立数据库实例

选择密码

集群文件系统的选择

选择asm卷组

数据区的选择

选择安装组建

指定SGA、字符集、语言、进程数、连接模式

开始建立数据实例

建立实例模板

建立实例

数据库建立完成指名emo位置

八.参数调整

8.1 集群命令的使用

8.1.1 crsctl

Usage: crsctl check crs - checks the viability of the Oracle Clusterware

crsctl check cssd - checks the viability of Cluster Synchronization Services

crsctl check crsd - checks the viability of Cluster Ready Services

crsctl check evmd - checks the viability of Event Manager

crsctl check cluster [-node ] - checks the viability of CSS across nodes

crsctl set css - sets a parameter override

crsctl get css - sets the value of a Cluster Synchronization Services

parameter

crsctl unset css - sets the Cluster Synchronization Services parameter to

its default

crsctl query css votedisk - lists the voting disks used by Cluster Synchronization

Services

crsctl add css votedisk - adds a new voting disk

crsctl delete css votedisk - removes a voting disk

crsctl enable crs - enables startup for all Oracle Clusterware daemons

crsctl disable crs - disables startup for all Oracle Clusterware daemons

crsctl start crs [-wait] - starts all Oracle Clusterware daemons

crsctl stop crs [-wait] - stops all Oracle Clusterware daemons. Stops Oracle Clusterware

managed resources in case of cluster.

crsctl start resources - starts Oracle Clusterware managed resources

crsctl stop resources - stops Oracle Clusterware managed resources

crsctl debug statedump css - dumps state info for Cluster Synchronization Services

objects

crsctl debug statedump crs - dumps state info for Cluster Ready Services objects

crsctl debug statedump evm - dumps state info for Event Manager objects

crsctl debug log css [module:level] {,module:level} ... - turns on debugging for Cluster

Synchronization Services

crsctl debug log crs [module:level] {,module:level} ... - turns on debugging for Cluster

Ready Services

crsctl debug log evm [module:level] {,module:level} ... - turns on debugging for Event

Manager

crsctl debug log res [resname:level] ... - turns on debugging for Event Manager

crsctl debug trace css [module:level] {,module:level} ... - turns on debugging for

Cluster Synchronization Services

crsctl debug trace crs [module:level] {,module:level} ... - turns on debugging for

Cluster Ready Services

crsctl debug trace evm [module:level] {,module:level} ... - turns on debugging for Event

Manager

crsctl query crs softwareversion [] - lists the version of Oracle Clusterware

software installed

crsctl query crs activeversion - lists the Oracle Clusterware operating version

crsctl lsmodules css - lists the Cluster Synchronization Services modules that can be

used for debugging

crsctl lsmodules crs - lists the Cluster Ready Services modules that can be used for

debugging

crsctl lsmodules evm - lists the Event Manager modules that can be used for debugging