Oracle Database 10gR2 on Centos (Sun virtualBox) RAC on ASM (shared storage ata over ethernet) ================================================ doc: * Database Oracle Clusterware and Oracle Real Application Clusters - Installation Guide, 10g Release 2 (10.2) for Linux * Database Oracle Clusterware and Oracle Real Application Clusters - Administration and Deployment Guide prepare VMs with centOS ======================= Website Name: mirror.centos.org CentOS Directory: centos/5.3/os/i386 Website Name: http://ftp.sh.cvut.cz/ CentOS Directory: MIRRORS/centos/5.3/os/i386 Virtual machines with 512MB (anaconda) - rac1 - rac2 - storage server (aoe) VBOX addons =========== install additions yum install gcc -y yum install kernel sources -y yum install kernel-devel -y sh VBoxLinuxAdditions.run time ==== yum install ntp chkconfig --level 235 ntpd on service ntpd stop ntpdate ntp.t-mobile.cz service ntpd start vi /etc/sysconfig/ntpd # Set to 'yes' to sync hw clock after successful ntpdate SYNC_HWCLOCK=yes # manual NTP sync service ntpd stop while true; do ntpdate ntp.t-mobile.cz sleep 5 done network ======= [root@rac1 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 # public 10.0.0.10 rac1.localdomain rac1 10.0.0.30 rac2.localdomain rac2 # # interconnect 10.0.10.10 rac1-priv.localdomain rac1-priv 10.0.10.30 rac2-priv.localdomain rac2-priv # # virtual 10.0.0.11 rac1-vip.localdomain rac1-vip 10.0.0.33 rac2-vip.localdomain rac2-vip os packages ============== Red Hat Enterprise Linux 4.0 and Asianux 2.0: binutils-2.15.92.0.2-13.EL4 compat-db-4.1.25-9 compat-libstdc++-296-2.96-132.7.2 control-center-2.8.0-12 gcc-3.4.3-22.1.EL4 gcc-c++-3.4.3-22.1.EL44 glibc-2.3.4-2.9 glibc-common-2.3.4-2.9 gnome-libs-1.4.1.2.90-44.1 libstdc++-3.4.3-22.1 libstdc++-devel-3.4.3-22.1 make-3.80-5 pdksh-5.2.14-30 sysstat-5.0.5-1 xscreensaver-4.18-5.rhel4.2 setarch-1.6-1 yum install binutils compat-db yum install compat-libstdc++* yum install gcc gcc-c++ glibc glibc-common yum install gnome-libs libstdc++ libstdc++-devel yum install compat-gcc-* yum install openmotif os user and groups ================== groupadd dba groupadd oinstall useradd -g oinstall -G dba oracle passwd oracle - test (id goup number has to fi on all nodes!) id oracle uid=501(oracle) gid=502(oinstall) groups=502(oinstall),501(dba) /etc/sysctl.conf ================ kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default = 1048576 net.core.rmem_max = 1048576 net.core.wmem_default = 262144 net.core.wmem_max = 262144 directories =========== $root mkdir -p /u01/app/oracle chmod -R 775 /u01/app/oracle chown -R oracle:oinstall /u01/app/oracle mkdir -p /u01/app/oracle/oradata chmod -R 775 /u01/app/oracle/oradata chown -R oracle:dba /u01/app/oracle/oradata su - oracle .bash_profile ============= export ORACLE_BASE=/u01/app export ORACLE_SID=rac1 su - root /etc/security/limits.conf ========================= oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 /etc/pam.d/login ================ session required /lib/securitls y/pam_limits.so session required pam_limits.so /etc/profile ============ if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi umask 022 ssh === su - oracle $ mkdir ~/.ssh chmod -R 700 ~/.ssh ssh-keygen -t rsa ssh-keygen -t dsa distribute to ~/.ssh/authorized_keys ~/.ssh/id_rsa.pub ~/.ssh/id_dsa.pub oracle@rac1: cat *.pub >> ./authorized_keys scp ./authorized_keys oracle@rac2:/home/oracle/.ssh oracle@rac2: cat *.pub >> ./authorized_keys scp ./authorized_keys oracle@rac1:/home/oracle/.ssh # create known hosts by calling ssh oracle@rac1 ssh oracle@rac2 STORAGE ======= prepare partitions for storage ============================== Identifying Clusterware, database files needs: ---------------------------------------------------- 1 x 1.4 GB OCR, vot doc: 256MB for each OCR and VOT disk 1 x 15 GB datafiles 1.2GB for data, 2Gb for FRA Oracle doc ATA over ethernet ================= http://aoetools.sourceforge.net - install vblade on storage server - install aoetolls on rac1, rac2 on storage ---------- vi /etc/rc.local # mapping local disk to ethernet address and slot vbladed 1 1 eth0 /dev/hdb1 vbladed 1 2 eth0 /dev/hdb2 on rac1, rac2 ------------- vi /etc/rc.local modprobe aoe sleep 10 test ==== reboot aoe-stat OCFS2 (for clusterware files, database, fra, OH shared possible) ====== doc: http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_users_guide.pdf # setup o2cb - ocfs cluster service # chose max. timeouts, stored in /etc/sysconfig/o2cb service o2cb configure # cluster membership for ocfs service /etc/ocfs2/cluster.conf mkfs.ocfs2 -L "oradata" /dev/etherd/e1.1 mkfs.ocfs2 -L "oracrs" /dev/etherd/e1.2 # rac1,rac2: vi /etc/rc.local mount -t ocfs2 /dev/etherd/e1.1 /u02/oradata -o datavolume,nointr,_netdev mount -t ocfs2 /dev/etherd/e1.2 /u02/oracrs -o datavolume,nointr,_netdev chown -R oracle:oinstall /u02 chmod -R 775 /u02 # clusterware files permissions mkdir /u02/oracrs chown oracle:oinstall /u02/oracrs chmod 775 /u02/oracrs # datafiles mkdir /u02/oradata chown oracle:oinstall /u02/oradata chmod -R 775 /u02/oradata # check storage by cluvfy (only for iSCSI) cd /u01/app/oracle/install/clusterware/cluvfy ./runcluvfy.sh comp ssa -n rac1,rac2 -s /u02/oradata,/u02/oracrs -verbose ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -c /u02/oracrs -q /u02/oracrs -verbose RAW devices (optional) =========== raw /dev/raw/raw1 /dev/etherd/e10.1 raw /dev/raw/raw2 /dev/etherd/e10.2 raw /dev/raw/raw3 /dev/etherd/e10.3 chmod 640 /dev/raw/raw2 #OCR chown root:dba /dev/raw/raw2 #OCR chmod 660 /dev/raw/raw3 #VOT chown oracle:oinstall /dev/raw/raw3 #VOT ASMlib (optional) ====== http://www.oracle.com/technology/software/tech/linux/asmlib/rhel5.html rpm -ivh oracleasm-support-2.1.3-1.el5.i386.rpm rpm -ivh oracleasm-2.6.18-128.el5-2.0.5-1.el5.i686.rpm rpm -ivh oracleasmlib-2.0.4-1.el5.i386.rpm service oracleasmlib configure service oracleasmlib status oracleasm createdisk VOL1 /dev/blockdevice -test reboot oracleasm listdisks on both nodes clusterware install =================== ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose run OUI bug RHEL5 http://forums.oracle.com/forums/thread.jspa?threadID=674181 /u01/app/crs/jdk/jre//bin/java: error while loading shared libraries: remove LD_ASSUME_KERNEL $ORA_CRS_HOME/bin/vipca $ORA_CRS_HOME/bin/srvctl any other errors? - cd $ORA_CRS_HOME/log/hostname/alert.log ./client/...logs - ls /tmp/* OCR problem? - metalink 367715.1 - Failed To Upgrade Oracle Cluster Registry Configuration When Running Root.Sh - auto repair - "ocrconfig -repair" (Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide) - run ocrschek binary # css misscount clusterware misscount - crsctl set css misscount 2147483647 clusterware deinstall ===================== metalink note 239998.1 - How to Clean Up After a Failed Oracle Clusterware (CRS) Installation 1. $ORA_CRS_HOME/install/rootdeinstall.sh 2. manual deletion rm -f /etc/oracle/* rm -f /etc/init.d/init.cssd rm -f /etc/init.d/init.crs rm -f /etc/init.d/init.crsd rm -f /etc/init.d/init.evmd rm -f /etc/rc2.d/K96init.crs rm -f /etc/rc2.d/S96init.crs rm -f /etc/rc3.d/K96init.crs rm -f /etc/rc3.d/S96init.crs rm -f /etc/rc5.d/K96init.crs rm -f /etc/rc5.d/S96init.crs rm -Rf /etc/oracle/scls_scr rm -f /etc/inittab.crs cp /etc/inittab.orig /etc/inittab 4. kill processes ps -ef | grep crs ps -ef | grep evm ps -ef | grep css 5. temp rm -f /var/tmp/.oracle/* rm -f /tmp/.oracle/* 6. ocr loc rm -fr /etc/oracle/ocr.loc 7. OUI deinstall (1-6 on all nodes - before you run OUI) 8.Remove the CRS install location: rm -Rf /u01/app/oracle/product/10.2.0/crs 9. Clean out the OCR and Voting Files with dd commands. Example: dd if=/dev/zero of=/dev/hdb2 bs=1M count=100 dd if=/dev/zero of=/dev/HDN3 bs=1M count=30 10.reboot node database software install ========================= oracle@rac1$ cd $ORA_CRS_HOME/bin ./cluvfy stage -pre dbinst -n rac1,rac2 -verbose OUI - runInstaller -------------------------------------------------------------------------------- centos performance hints ========================== chkconfig iptables off chkconfig ip6tables off chkconfig cups off chkconfig bluetooth off