Oracle RAC12c on OL 7 using Virtualbox

Assumptions:
– You have already installed OL7 on a virtual machine
– assigned at least 4G of memory, otherwise the CVU will complain.
– 2 network cards

=============== BEGIN ERRORS DESCRIPTION =======================

Note: Along the installation of the CLusterware 12cR1 I encountered couple of issues.

  1. This is actually the 2nd one, but should be fixed right at this step. RAC12cR1_error_2

Solution:

VBoxManage list vms
"rac1" {83ff3550-ced8-48be-8f8e-85d4363fb047}
"rac2" {3ae7500c-53b4-4ee6-aa8e-152869b6d473}

 VBoxManage setextradata 3ae7500c-53b4-4ee6-aa8e-152869b6d473 "VBoxInternal/CPUM/HostCPUID/Cache/Leaf" "0x4"
 VBoxManage setextradata 3ae7500c-53b4-4ee6-aa8e-152869b6d473 "VBoxInternal/CPUM/HostCPUID/Cache/SubLeaf" "0x4"
 VBoxManage setextradata 3ae7500c-53b4-4ee6-aa8e-152869b6d473 "VBoxInternal/CPUM/HostCPUID/Cache/eax" "0"
 VBoxManage setextradata 3ae7500c-53b4-4ee6-aa8e-152869b6d473 "VBoxInternal/CPUM/HostCPUID/Cache/ebx" "0"
 VBoxManage setextradata 3ae7500c-53b4-4ee6-aa8e-152869b6d473 "VBoxInternal/CPUM/HostCPUID/Cache/ecx" "0"
 VBoxManage setextradata 3ae7500c-53b4-4ee6-aa8e-152869b6d473 "VBoxInternal/CPUM/HostCPUID/Cache/edx" "0"
 VBoxManage setextradata 3ae7500c-53b4-4ee6-aa8e-152869b6d473 "VBoxInternal/CPUM/HostCPUID/Cache/SubLeafMask" "0xffffffff"

2. This is the first error that we’re going to get but this can be fixed while we’re doing the installation:

RAC12cR1_1st

Solution:
As soon as the installer has finished with the step “Copying files” do this:

cp $ORACLE_HOME/javavm/jdk/jdk6/lib/lib*12.a $ORACLE_HOME/lib/

This library (libjavavm12.a) wasn’t copied at the right location

==============================END ERRORS =============================

If you have cloned the VM (as I do), this is the command to change the name of the VM:
This is the syntax:

# hostnamectl set-hostname Your-New-Host-Name-Here
# hostnamectl set-hostname "Your New Host Name Here" --pretty
# hostnamectl set-hostname Your-New-Host-Name-Here --static
# hostnamectl set-hostname Your-New-Host-Name-Here --transient

And here is what I did:

[root@rac1 ~]# hostnamectl set-hostname rac1.home-oracle
[root@rac1 ~]# hostnamectl status
   Static hostname: rac1.home-oracle
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 2334d0c81bb84132a4f15d6b594a6647
           Boot ID: 8f7baf3c1ef14f52aa819ab822127fa5
    Virtualization: kvm
  Operating System: Oracle Linux Server 7.1
       CPE OS Name: cpe:/o:oracle:linux:7:1
            Kernel: Linux 3.8.13-55.1.6.el7uek.x86_64
      Architecture: x86_64

Let’s start with prereqs (all on the new VM):

    • – Download the Oracle Public Yum Repository Files
As root, change directory to /etc/yum.repos.d.

# cd /etc/yum.repos.d
Use the wget utility to download the repository configuration file that is appropriate for your system.

# wget http://public-yum.oracle.com/public-yum-release.repo
For Oracle Linux 7, enter:

# wget http://public-yum.oracle.com/public-yum-ol7.repo
The /etc/yum.repos.d directory is updated with the repository configuration file, in this example, public-yum-ol7.repo.

You can enable or disable repositories in the file by setting the value of the enabled directive to 1 or 0 as required.
    • – Install the oracle-rdbms-server preinstall on Oracle Linux 7
Install the oracle-rdbms-server-12cR1-preinstall package as the root user:
# yum install oracle-rdbms-server-12cR1-preinstall
 
Install the oracle-rdbms-server-11gR2-preinstall package as the root user:
# yum install oracle-rdbms-server-11gR2-preinstall

A best practice after the oracle-rdbms-server-*-preinstall RPM installation is to set the password for the oracle user that is created during the install. As root, type the following:

$ passwd oracle
Changing password for user oracle.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
$
  • Install the ASM required RPMs (you can also download them from here)
yum localinstall -y oracleasm-support-2.1.8-3.el7.x86_64.rpm oracleasmlib-2.0.12-1.el7.x86_64.rpm
    • – Disable iptables/firewall
[root@rac1 ~]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: inactive (dead) since Sun 2016-04-24 21:15:33 CEST; 14h ago
  Process: 669 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 669 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/firewalld.service

Apr 24 10:36:21 rac1.home-oracle systemd[1]: Starting firewalld - dynamic firewall daemon...
Apr 24 10:36:22 rac1.home-oracle systemd[1]: Started firewalld - dynamic firewall daemon.
Apr 24 21:15:27 rac1.home-oracle systemd[1]: Stopping firewalld - dynamic firewall daemon...
Apr 24 21:15:33 rac1.home-oracle systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@rac1 ~]# systemctl stop firewalld
[root@rac1 ~]# systemctl disable firewalld
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
[root@rac1 ~]# 
    • – Disable SELinux
Edit /etc/selinux/config. It should look like this:
[root@rac1 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disablede
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 
    • – Configure the /etc/hosts File

Mine is looking like this (and I do use a DNS server as a separate VM):

[root@rac1 ~]# cat /etc/hosts
127.0.0.1 localhost.home-oracle localhost
# Public
192.168.100.101 rac1.home-oracle rac1
192.168.100.102 rac2.home-oracle rac2
# Private
192.168.200.101 rac1-priv.home-oracle rac1-priv
192.168.200.102 rac2-priv.home-oracle rac2-priv
# Virtual
192.168.100.111 rac1-vip.home-oracle rac1-vip
192.168.100.112 rac2-vip.home-oracle rac2-vip
# SCAN
192.168.100.121 rac-scan.home-oracle rac-scan
192.168.100.122 rac-scan.home-oracle rac-scan
192.168.100.123 rac-scan.home-oracle tac-scan
    • – Configure SSH and X11 Forwarding

Install xauth and the xorg-x11-apps RPMs, as root type:

# yum -y install xorg-x11-apps xauth. 

Enable X11 forwarding in the /etc/ssh/sshd_config by adding “ForwardX11 yes”… something like this:

.........
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
.........

Now restart the sshd service

systemctl restart sshd.service
    • – Disable IPv6

At the end of /etc/sysctl.conf file add these lines:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
    • – Create the necessary directories
# mkdir -p /u01/app/oracle/product/12.1.0/dbhome_1
# mkdir -p /u01/app/oracle/12.1.0/grid/
# cd /u01
# chown -R oracle:oinstall app
# chmod -R 775 app/*
    • – Adapt ~/.bash_profile

Mine is looking like this

[oracle@rac1 ~]$ cat .bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

# Oracle variables
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=rac1.home-oracle; export ORACLE_HOSTNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/12.1.0/dbhome_1; export ORACLE_HOME
AGENT_HOME=$ORACLE_BASE/product/agentr4/agent_inst; export AGENT_HOME
OMS_HOME=$ORACLE_BASE/product/MiddlewareR4/oms; export OMS_HOME
ORACLE_INSTANCE=/u01/app/oracle/product/gc_inst/WebTierIH1; export ORACLE_INSTANCE
ORACLE_SID=ORCL; export ORACLE_SID
ORACLE_UNQNAME=ORCL1; export ORACLE_UNQNAME
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
    • Clone your RAC1 to RAC2

Note: Don’t forget to adapt the name and the network IPs

    • Run CVU to insure that your environment is complying with all prerequisites.

I used in the beginning the executable delivered within the 12c grid zip file which I have downloaded from OTN. I ran into these issues:

      • a) The CVU was failing at this point:
[oracle@rac1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose
.............

Check: TCP connectivity of subnet "192.168.200.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  rac1 : 192.168.200.101          rac1 : 192.168.200.101          passed          
  rac2 : 192.168.200.102          rac1 : 192.168.200.101          failed          

ERROR: 
PRVG-11850 : The system call "connect" failed with error "113" while executing exectask on node "rac2"
No route to host
  rac1 : 192.168.200.101          rac2 : 192.168.200.102          failed          

ERROR: 
PRVG-11850 : The system call "connect" failed with error "113" while executing exectask on node "rac1"
No route to host
  rac2 : 192.168.200.102          rac2 : 192.168.200.102          passed          
Result: TCP connectivity check failed for subnet "192.168.200.0"


WARNING: 
Could not find a suitable set of interfaces for VIPs

WARNING: 
Could not find a suitable set of interfaces for the private interconnect
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.100.0".
Subnet mask consistency check passed for subnet "192.168.200.0".
Subnet mask consistency check passed.

Result: Node connectivity check failed

Checking multicast communication...

Checking subnet "192.168.100.0" for multicast communication with multicast group "224.0.0.251"...
PRVG-11138 : Interface "192.168.100.101" on node "rac1" is not able to communicate with interface "192.168.100.101" on node "rac1" over multicast group "224.0.0.251"
PRVG-11138 : Interface "192.168.100.101" on node "rac1" is not able to communicate with interface "192.168.100.102" on node "rac2" over multicast group "224.0.0.251"
PRVG-11138 : Interface "192.168.100.102" on node "rac2" is not able to communicate with interface "192.168.100.101" on node "rac1" over multicast group "224.0.0.251"
PRVG-11138 : Interface "192.168.100.102" on node "rac2" is not able to communicate with interface "192.168.100.102" on node "rac2" over multicast group "224.0.0.251"
Checking subnet "192.168.200.0" for multicast communication with multicast group "224.0.0.251"...
PRVG-11138 : Interface "192.168.200.101" on node "rac1" is not able to communicate with interface "192.168.200.101" on node "rac1" over multicast group "224.0.0.251"
PRVG-11138 : Interface "192.168.200.101" on node "rac1" is not able to communicate with interface "192.168.200.102" on node "rac2" over multicast group "224.0.0.251"
PRVG-11138 : Interface "192.168.200.102" on node "rac2" is not able to communicate with interface "192.168.200.101" on node "rac1" over multicast group "224.0.0.251"
PRVG-11138 : Interface "192.168.200.102" on node "rac2" is not able to communicate with interface "192.168.200.102" on node "rac2" over multicast group "224.0.0.251"

Solution
In my case, it was the firewall which I missed to disable it (on both nodes):

[root@rac2 ~]# systemctl stop firewalld
[root@rac2 ~]# systemctl disable firewalld
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
[root@rac2 ~]# 
      • b) The CVU was constantly failing at this step:
Starting check for /dev/shm mounted as temporary file system ...

ERROR: 

PRVE-0426 : The size of in-memory file system mounted as /dev/shm is "2097152k" megabytes which is less than the required size of "2048" megabytes on node ""
PRVE-0426 : The size of in-memory file system mounted as /dev/shm is "20631552k" megabytes which is less than the required size of "2048" megabytes on node ""

Check for /dev/shm mounted as temporary file system failed

And this is my /dev/shm looking like:

[oracle@rac1 grid]$ cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Thu Apr 21 21:53:57 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=f323280a-75df-40a1-a180-654334b3fdfc /boot                   xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-tmp      /tmp                    xfs     defaults        0 0
/dev/mapper/ol-u01      /u01                    xfs     defaults        0 0
/dev/mapper/ol-usr      /usr                    xfs     defaults        0 0
/dev/mapper/ol-var      /var                    xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
tmpfs       		/dev/shm        	tmpfs   size=2048M      0 0

Now, the value of the shm is the one required by Oracle. As Oracle stated, this is a Bug related to a script (checktmpfs.sh) which is not processing correctly the size of the shm. It’s all described in this doc: Doc ID 1601354.1

In order to still check my environment I have downloaded the latest version on CVU from OTN, and all went just fine and I’m ready with all my prerequisite checks.

Configure ASM disk

I have created 5 shareable disks and attached them on both VMs.
I have formatted them and created 1 partition on each disk and this is the outcome:

[root@rac1]# fdisk -l | grep /dev/sd
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sdb1            2048    10485759     5241856   83  Linux
Disk /dev/sdd: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sdd1            2048    10485759     5241856   83  Linux
Disk /dev/sde: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sde1            2048    10485759     5241856   83  Linux
Disk /dev/sdc: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sdc1            2048    10485759     5241856   83  Linux

On the second rac execute this:

[root@rac2 ]# partprobe 
[root@rac2 ]# fdisk -l | grep /dev/sd
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sdb1            2048    10485759     5241856   83  Linux
Disk /dev/sdc: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sdc1            2048    10485759     5241856   83  Linux
Disk /dev/sdd: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sdd1            2048    10485759     5241856   83  Linux
Disk /dev/sde: 5368 MB, 5368709120 bytes, 10485760 sectors
/dev/sde1            2048    10485759     5241856   83  Linux

On RAC1 & RAC2 as root configure the ASMlib:

[root@rac1 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Scan for Oracle ASM disks on boot (y/n) [y]: 
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@rac1 ~]# 

On RAC1 as root create the ASM disks:

[root@rac1 ~]# oracleasm createdisk CRS /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm createdisk ASM1 /dev/sdc1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm createdisk ASM2 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm createdisk ASM3 /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac1 ~]# oracleasm listdisks
ASM1
ASM2
ASM3
CRS
[root@rac1 ~]# 

On RAC2 as root we scan and list the disks:

[root@rac2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "CRS"
Instantiating disk "ASM1"
Instantiating disk "ASM2"
Instantiating disk "ASM3"
[root@rac2 ~]# oracleasm listdisks
ASM1
ASM2
ASM3
CRS
[root@rac2 ~]# 

Just before running the GUI, I ran again a CVU check:

As oracle user on RAC1:

[oracle@rac1 bin]$ ./cluvfy stage -pre crsinst -n rac1,rac2 -fixup -verbose

Performing pre-checks for cluster services setup 
...............
Pre-check for cluster services setup was successful. 


NOTE: 
No fixable verification failures to fix
[oracle@rac1 bin]$ 

It looks good to go for the GUI 🙂

Install the clusterware release 12cR1

Go to the folder where you have unzipped the archives:

[oracle@rac1 ~]cd /media/sf_oracle_stuff/12cR1/grid
[oracle@rac1 grid]$ ./runInstaller 

RAC_12cR2_1

Choose the first option and click Next

RAC_12cR2_2

I configure a Standard cluster for fun

RAC_12cR2_3

For fun I chose the first option. Click next.

RAC_12cR2_4

Here I added also the second node to the configuration. Click next.

RAC_12cR2_5

Checked also the subnet. Click next.

RAC_12cR2_6

Put the correct Software location; As I like using ASM I chose the “Oracle Automatic Storage Management” option. I have set a very difficult password to guess. Don’t forget about the OSSASM group which should be the same as the one you have set while configuring the asm disks.

RAC_12cR2_7

Choose the disk(s) where you want to put the Clusterware configuration. Pay attention that you need to allocate more than 5G (you’re going to store 2 DB instances on it)

RAC_12cR2_8

Easy. Click next

RAC_12cR2_9

In 12cR1 you have the “luxury” to choose whether you want the installer to run the last “root.sh” file(s) on both nodes for you…or not. In the end I chose not to…..I was too curious to see what’s doing behind. But it’s good to know you have the option though.

RAC_12cR2_10

Now the installer is running the CVU script identifying all the problems that you may occur and for some, will offer you a fix.

RAC_12cR2_11

As expected, the CVU didn’t “forget” about SHM issue from the beginning of my RAC installation. Being a known problem/bug, I chose “Ignore All” and moved on.

RAC_12cR2_12

And this is the confirmation. I chose Yes.

RAC_12cR2_13

OK. We’re now ready to install the Clusterware 12cR1. Click Install.

RAC_12cR2_14

This is how the “progress” is looking like.

If you it does get done with “Copy files”, run the fix I told you in the beginning…. I mean this:

cp $ORACLE_HOME/javavm/jdk/jdk6/lib/lib*12.a $ORACLE_HOME/lib/

And you won’t get the image bellow :).

RAC_12cR2_15

Well…if you still get the image from above, you can still apply the fix and click “Yes”

Now, we got to the root script executions and this is how it looks in my case:

[root@rac1 ~]# /u01/app/grid/12.1.0/root.sh
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/grid/12.1.0
Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/grid/12.1.0/crs/install/crsconfig_params
2016/04/28 11:03:13 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/04/28 11:03:35 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/04/28 11:03:35 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2016/04/28 11:04:07 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
ASM created and started successfully.
Disk Group DATA created successfully.
CRS-2672: Attempting to start 'ora.crf' on 'rac1'
CRS-2672: Attempting to start 'ora.storage' on 'rac1'
CRS-2676: Start of 'ora.storage' on 'rac1' succeeded
CRS-2676: Start of 'ora.crf' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk a052a4e6e22d4fe2bf39cb34f155796f.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a052a4e6e22d4fe2bf39cb34f155796f (ORCL:ASM1) [DATA]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.storage' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.storage' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac1'
CRS-2676: Start of 'ora.storage' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac1'
CRS-2676: Start of 'ora.crf' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rac1
CRS-6016: Resource auto-start has completed for server rac1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/04/28 11:08:18 CLSRSC-343: Successfully started Oracle Clusterware stack
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
2016/04/28 11:09:27 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac1 ~]#

And this is for the second node:

[root@rac2 ~]# /u01/app/grid/12.1.0/root.sh
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/grid/12.1.0
Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/grid/12.1.0/crs/install/crsconfig_params
2016/04/28 11:09:40 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/04/28 11:10:02 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/04/28 11:10:03 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2016/04/28 11:11:13 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac2'
CRS-2676: Start of 'ora.storage' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac2'
CRS-2676: Start of 'ora.crf' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
CRS-6017: Processing resource auto-start for servers: rac2
CRS-2672: Attempting to start 'ora.net1.network' on 'rac2'
CRS-2676: Start of 'ora.net1.network' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rac2'
CRS-2676: Start of 'ora.ons' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-6016: Resource auto-start has completed for server rac2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/04/28 11:13:59 CLSRSC-343: Successfully started Oracle Clusterware stack
2016/04/28 11:14:13 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac2 ~]#

Now…we’re good to go further.

RAC_12cR2_16

Now is creating all the other stuff it needs such as the listener/tnsnames/ management instance…etc

RAC_12cR2_17

And ……we’re done.

 

CRS-4124: Oracle High Availability Services startup failed.

Some time ago I got the issue from below:

Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2011-05-11 02:20:52: Checking for super user privileges
2011-05-11 02:20:52: User has super user privileges
2011-05-11 02:20:52: Parsing the host name
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
CRS-4664: Node localhost successfully pinned.
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start: Inappropriate ioctl for device at /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 296.

Some possible solutions could be:

1. Look at these 2 documents:
– OHASD Failed to Start: Inappropriate ioctl for device [ID1069182.1]
– How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1]

2. You will have to follow the steps where you monitor the /var/tmp/.oracle directory for the file creation and run the following command as root user as soon as the file exist:

/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1

3. Be sure that you’re using the certified configurations. For instance, I got this error while I was using OEL6 and I was trying to install 11.2.0.4 Grid on it. When I realized, I have used 11.2.0.3 Grid version and all went smoothly.

If you have encountered other situations, please post them in the comments.