Adding and Deleting Cluster NodesStep By Step
The following are the step of deleting and adding cluster node in Oracle RAC 11gR2. It was happen in my company and fixed it successfully without down the system; I mean I can do online during customer do transaction and there is no one complain any issue during this.
Let me introduce my real case of this issue:
1. I have two node RAC, dbnode1 and dbnode2 working properly
2. Then There is a problem happen with OS on dbnode1, and file system /oracle was missed from OS and everything- grid and rdbms software- did not appear
3. Our system still running with dbnode2.
Now we are a DBA we need to fix it as soon as posible.
1. We need clean/remove/delete dbnode1 configuration from cluster(do it from dbnode2)
2. Run script post delete node and Pre add node.(do it from dbnode2)
3. Create nessary folder on dbnode1
4.Transfer binary of grid from dbnode2(do it from dbnode2)
5. Transfer binary of rdbms from dbnode2(do it from dbnode2)
6. Prepare db parameter on dbnode1
============
*******************************************************************
*Note: dbnode1 is new node which need to add to existing cluster.*
*******************************************************************
dbnode1:
mkdir -p /oracle/oraInventory
mkdir -p /oracle/11.2.0.3/grid
mkdir -p /oracle/product/11.2.0.3/dbhome_1
chown -R pitraib:oinstall /oracle
# cd /dev
# ls -l rhdisk*
# chown pitraib:oinstall rhdisk3 rhdisk4 rhdisk5 rhdisk6 rhdisk7 rhdisk8 rhdisk9 rhdisk10 rhdisk11
==========
dbnode2:=
==========
To delete a node from a cluster:
# olsnodes -s -t
#
if it pin then unpin
# crsctl unpin css -n node_to_be_deleted
# crsctl delete node -n dbnode1
CRS-4661: Node dbnode1 successfully deleted.
#
$ cd /oracle/11.2.0.3/grid/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=/oracle/11.2.0.3/grid "CLUSTER_NODES=dbnode2" CRS=TRUE
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4095 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/oraInventory
'UpdateNodeList' was successful.
$
---------------------------------------------------------------------------------
$ cluvfy stage -post nodedel -n dbnode1 -verbose
Performing post-checks for node removal
Checking CRS integrity...
Clusterware version consistency passed
The Oracle Clusterware is healthy on node "dbnode2"
CRS integrity check passed
Result:
Node removal check passed
Post-check for node removal was successful.
$
==============================
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE dbnode2
ora.LISTENER.lsnr
ONLINE ONLINE dbnode2
ora.OCRV.dg
ONLINE ONLINE dbnode2
ora.RECO.dg
ONLINE ONLINE dbnode2
ora.asm
ONLINE ONLINE dbnode2 Started
ora.gsd
OFFLINE OFFLINE dbnode2
ora.net1.network
ONLINE ONLINE dbnode2
ora.ons
ONLINE ONLINE dbnode2
ora.registry.acfs
ONLINE ONLINE dbnode2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE dbnode2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE dbnode2
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE dbnode2
ora.orcldb.db
1 ONLINE OFFLINE
2 ONLINE ONLINE dbnode2 Open
ora.cvu
1 ONLINE ONLINE dbnode2
ora.dbnode1.vip
1 ONLINE INTERMEDIATE dbnode2 FAILED OVER
ora.dbnode2.vip
1 ONLINE ONLINE dbnode2
ora.oc4j
1 ONLINE ONLINE dbnode2
ora.scan1.vip
1 ONLINE ONLINE dbnode2
ora.scan2.vip
1 ONLINE ONLINE dbnode2
ora.scan3.vip
1 ONLINE ONLINE dbnode2
$
$ cluvfy stage -post nodedel -n dbnode1 -verbose
$ cluvfy stage -pre nodeadd -n dbnode1 -verbose
----YOU HAVE TO BECAREFULL MAKE SURE WHAT U NEED TO IGNORE
$ export IGNORE_PREADDNODE_CHECKS=Y
$ cd /oracle/11.2.0.3/grid/oui/bin
$ ./addNode.sh -silent "CLUSTER_NEW_NODES={dbnode1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={dbnode1-vip}"
$ export IGNORE_PREADDNODE_CHECKS=Y
$ ./addNode.sh -silent CLUSTER_NEW_NODES={dbnode1}
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 3969 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes dbnode1,dbnode1 are available
............................................................... 100% Done.
SEVERE:The new nodes 'dbnode1' are already part of the cluster.
$ echo $ORACLE_HOME
/oracle/product/11.2.0.3/dbhome_1
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={dbnode2}"
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 3963 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/oraInventory
'UpdateNodeList' was successful.
$
$ export IGNORE_PREADDNODE_CHECKS=Y
$ cd /oracle/product/11.2.0.3/dbhome_1/oui/bin
$ ./addNode.sh -silent CLUSTER_NEW_NODES={dbnode1}
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 32768 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes dbnode1 are available
............................................................... 100% Done.
..
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /oracle/product/11.2.0.3/dbhome_1
New Nodes
Space Requirements
New Nodes
dbnode1
/oracle: Required 7.47GB : Available 86.38GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.3.0
Sun JDK 1.5.0.12.4
Installer SDK Component 11.2.0.3.0
Oracle One-Off Patch Installer 11.2.0.1.7
Oracle Universal Installer 11.2.0.3.0
Oracle USM Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.3.0
Oracle RAC Deconfiguration 11.2.0.3.0
Oracle Database Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.5.0.1
Oracle ODBC Driverfor Instant Client 11.2.0.3.0
LDAP Required Support Files 11.2.0.3.0
SSL Required Support Files for InstantClient 11.2.0.3.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.3.0
Oracle Database Vault J2EE Application 11.2.0.3.0
Oracle Label Security 11.2.0.3.0
Oracle Data Mining RDBMS Files 11.2.0.3.0
Oracle OLAP RDBMS Files 11.2.0.3.0
Oracle OLAP API 11.2.0.3.0
Platform Required Support Files 11.2.0.3.0
Oracle Database Vault option 11.2.0.3.0
Oracle RAC Required Support Files-HAS 11.2.0.3.0
SQL*Plus Required Support Files 11.2.0.3.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.3.0
Oracle SQL Developer 11.2.0.3.0
Oracle Application Express 11.2.0.3.0
XDK Required Support Files 11.2.0.3.0
RDBMS Required Support Files for Instant Client 11.2.0.3.0
SQLJ Runtime 11.2.0.3.0
Database Workspace Manager 11.2.0.3.0
RDBMS Required Support Files Runtime 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0
Enterprise Manager Repository Core Files 10.2.0.4.4
Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0
Enterprise Manager Grid Control Core Files 10.2.0.4.4
Enterprise Manager Common Core Files 10.2.0.4.4
Enterprise Manager Agent Core Files 10.2.0.4.4
RDBMS Required Support Files 11.2.0.3.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.3.0
Oracle Text Required Support Files 11.2.0.3.0
Parser Generator Required Support Files 11.2.0.3.0
Oracle Database 11g Multimedia Files 11.2.0.3.0
Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
Oracle Multimedia Annotator 11.2.0.3.0
Oracle JDBC/OCI Instant Client 11.2.0.3.0
Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
Precompiler Required Support Files 11.2.0.3.0
Oracle Core Required Support Files 11.2.0.3.0
Sample Schema Data 11.2.0.3.0
Oracle Starter Database 11.2.0.3.0
Oracle Message Gateway Common Files 11.2.0.3.0
Oracle XML Query 11.2.0.3.0
XML Parser for Oracle JVM 11.2.0.3.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.3.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.3.0
Oracle Quality of Service Management (Client) 11.2.0.3.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.3.0
Oracle Containers for Java 11.2.0.3.0
Perl Interpreter 5.10.0.0.2
Oracle Net Required Support Files 11.2.0.3.0
Secure Socket Layer 11.2.0.3.0
Oracle Universal Connection Pool 11.2.0.3.0
Oracle JDBC/THIN Interfaces 11.2.0.3.0
Oracle Multimedia Client Option 11.2.0.3.0
Oracle Java Client 11.2.0.3.0
Character Set Migration Utility 11.2.0.3.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.3.0
OLAP SQL Scripts 11.2.0.3.0
Database SQL Scripts 11.2.0.3.0
Oracle Locale Builder 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
SQL*Plus Files for Instant Client 11.2.0.3.0
Required Support Files 11.2.0.3.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.3.0
Oracle Notification Service 11.2.0.3.0
XML Parser for Java 11.2.0.3.0
Oracle Security Developer Tools 11.2.0.3.0
Oracle Wallet Manager 11.2.0.3.0
Cluster Verification Utility Common Files 11.2.0.3.0
Oracle Clusterware RDBMS Files 11.2.0.3.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.3.0
HAS Common Files 11.2.0.3.0
Precompiler Common Files 11.2.0.3.0
Installation Common Files 11.2.0.3.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.3.0
Buildtools Common Files 11.2.0.3.0
Assistant Common Files 11.2.0.3.0
Oracle Recovery Manager 11.2.0.3.0
PL/SQL 11.2.0.3.0
Generic Connectivity Common Files 11.2.0.3.0
Oracle Database Gateway for ODBC 11.2.0.3.0
Oracle Programmer 11.2.0.3.0
Oracle Database Utilities 11.2.0.3.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.3.0
Oracle Netca Client 11.2.0.3.0
Oracle Multimedia Locator 11.2.0.3.0
Oracle Call Interface (OCI) 11.2.0.3.0
Oracle Multimedia 11.2.0.3.0
Oracle Net 11.2.0.3.0
Oracle XML Development Kit 11.2.0.3.0
Database Configuration and Upgrade Assistants 11.2.0.3.0
Oracle JVM 11.2.0.3.0
Oracle Advanced Security 11.2.0.3.0
Oracle Internet Directory Client 11.2.0.3.0
Oracle Enterprise Manager Console DB 11.2.0.3.0
HAS Files for DB 11.2.0.3.0
Oracle Net Listener 11.2.0.3.0
Oracle Text 11.2.0.3.0
Oracle Net Services 11.2.0.3.0
Oracle Database 11g 11.2.0.3.0
Oracle OLAP 11.2.0.3.0
Oracle Spatial 11.2.0.3.0
Oracle Partitioning 11.2.0.3.0
Enterprise Edition Options 11.2.0.3.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Tuesday, October 4, 2016 1:07:48 PM ICT)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Tuesday, October 4, 2016 1:07:55 PM ICT)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Tuesday, October 4, 2016 1:31:16 PM ICT)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/oracle/product/11.2.0.3/dbhome_1/root.sh #On nodes dbnode1
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /oracle/product/11.2.0.3/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
=========
dbnode1=
=========
# /oracle/product/11.2.0.3/dbhome_1/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= pitraib
ORACLE_HOME= /oracle/product/11.2.0.3/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
=========
dbnode1=
=========
$ ps -ef|grep smon
root 20143 1 0 11:19 ? 00:02:38 /oracle/11.2.0.3/grid/bin/osysmond.bin
oracle 20277 1 0 11:19 ? 00:00:00 asm_smon_+ASM1
oracle 32192 32003 0 16:19 pts/0 00:00:00 grep smon
$ echo $ORACLE_SID
orcldb1
$ cd $ORACLE_HOME
$ cd dbs
$ ls -l
total 16
-rw-rw---- 1 pitraib oinstall 1544 Oct 04 08:31 hc_orcldb2.dat
-rw-rw---- 1 pitraib oinstall 1544 May 28 2014 hc_ROTEST12.dat
-rw-rw---- 1 pitraib oinstall 1544 May 28 2014 hc_ROTEST22.dat
-rw-r--r-- 1 pitraib oinstall 2851 Dec 12 2013 init.ora
-rw-r--r-- 1 pitraib oinstall 37 Dec 01 2015 initorcldb2.ora
-rw-r----- 1 pitraib oinstall 5120 Aug 16 16:00 orapworcldb2
-rw-r----- 1 pitraib oinstall 5120 Aug 16 15:59 orapworcldb2.16082016
-rw-r----- 1 pitraib oinstall 5120 Apr 04 2016 orapworcldb2.20160404
-rw-r----- 1 pitraib oinstall 2560 Aug 20 2014 orapworcldb2.bk
-rw-r----- 1 pitraib oinstall 5120 Jun 10 14:46 orapworcldb210062016
-rw-r----- 1 pitraib oinstall 14663680 May 20 00:02 snapcf_orcldb2.f
-rw-r----- 1 pitraib oinstall 7716864 May 28 2014 snapcf_ROTEST22.f
$ mv orapworcldb2 orapworcldb1
$ mv initorcldb2.ora initorcldb1.ora
$
$ rm hc_orcldb2.dat hc_ROTEST12.dat hc_ROTEST22.dat orapworcldb2.16082016
$ rm orapworcldb2.20160404 orapworcldb2.bk orapworcldb210062016 snapcf_orcldb2.f snapcf_ROTEST22.f
$
$ mkdir -p /oracle/product/admin/orcldb/adump
orapki wallet create -wallet "/oracle/product/admin/orcldb1/wallet" -auto_login_local
$ sqlplus / as sysdba
SQL> startup
ORACLE instance started.
Total System Global Area 1653518336 bytes
Fixed Size 2228904 bytes
Variable Size 1107299672 bytes
Database Buffers 536870912 bytes
Redo Buffers 7118848 bytes
Database mounted.
Database opened.
SQL>
$ crsctl stat res -t
Good Luck !!!!
No comments:
Post a Comment