Exadata half RACK Image Upgrade-Non-Rolling-Cell Server
As DMA its regular activity to patch exadata machine. There are two ways of patching Exadata box Rolling and Non-Rolling. In this blog, we will start with part 3 of the Exadata (Half Rack) Image Upgrade (Non-Rolling).
Precheck: Exadata Image Upgrade
- Oracle recommends to clear all the stateful alrts from all the cell nodes
[root@abcxyzadm01 ~]# dcli -g cell_group -l root "cellcli -e list alerthistory attributes name,beginTime,alertShortName,alertDescription,severity where alerttype=stateful and severity=critical" - Based on Exachk report check if Oracle finds any hardware failure which much be fixed before you proceed for the patching.
Cell Node/ Cell Server Patch Plan (Non-Rolling)
- update using the patchmgr utility.Unzip the update. It will extract into the patch_XXXXXX directory.
- Stop agents if running (Non-Rolling)
Check agents and restart if not running
ps -ef | grep agent | grep java | sed 's/\s\+/ /g' | cut -d " " -f 1,8 | sed 's/\/jdk.*//'
dcli -g dbs_group -l root 'su -l oracle -c "/u01/app/em13c/core/13.1.0.4.0/bin/emctl start agent"'
dcli -g dbs_group -l root 'su -l oracle -c "/u01/app/em13c/core/13.1.0.4.0/bin/emctl status agent"' | grep 'Agent is' - Check databases running before stopping CRS
/u01/app/12.1.0.2/grid/bin/crsctl status resource -t -w "TYPE = ora.database.type"
dcli -l root -g dbs_group "ps -ef | sort -k1.50 | grep pmon_ | grep -v grep && echo ''" - Stop and disable CRS
dcli -l root -g dbs_group /u01/app/12.1.0.2/grid/bin/crsctl disable crs /u01/app/12.1.0.2/grid/bin/crsctl stop cluster -all
dcli -l root -g dbs_group /u01/app/12.1.0.2/grid/bin/crsctl stop crs
dcli -l root -g dbs_group '/u01/app/12.1.0.2/grid/bin/crsctl check crs | grep online | wc -l | while read retval; do if [[ $retval -eq 0 ]]; then
echo CRS Stopped; elif [[ $retval -eq 4 ]]; then echo CRS Running; else echo CRS Not Ready; fi; done;' - Restart ILOM on all cell nodes (optional)
dcli -l root -g cell_group "ipmitool bmc reset cold" - Check uptime and reboot if needed
dcli -l root -g cell_group "uptime"
If cells up more than 7 days then reboot.
dcli -l root -g cell_group "cellcli -e alter cell shutdown services all"
Verify all cells stopped successfully. If not, repeat until all cells are stopped.
dcli -l root -g cell_group sync
dcli -l root -g cell_group sync
dcli -l root -g cell_group reboot - Stop services on all cell nodes
cd /u01/exa_img_upg/CELL/patch_XXXXXXX
dcli -l root -g cell_group "cellcli -e alter cell shutdown services all" - Cleanup space from any previous runs
the -reset_force command is only done the first time the cells are patched to this release.
It is not necessary to use the command for subsequent cell patching, even after rolling back the patch.
./patchmgr -cells cell_group -reset_force
Always use the -cleanup option before retrying a failed or halted run of the patchmgr utility.
./patchmgr -cells cell_group -cleanup - Run prerequisites check
cd /u01/exa_img_upg/CELL/patch_XXXXXXX
./patchmgr -cells cell_group -patch_check_prereq - Patch the cell nodes
nohup ./patchmgr -cells cell_group -patch & - Monitor the patch progress
Monitor the ILOM console for each cell being patched. You may want to download the ilom-login.sh script from note 1616791.1 for assisting in logging into the iloms.
cd /u01/exa_img_upg/CELL/patch_XXXXXXXX
tail -f nohup.out - Cleanup space
./patchmgr -cells cell_group -cleanup - Post Checks
dcli -l root -g cell_group imageinfo -version
dcli -l root -g cell_group imageinfo -status
dcli -l root -g cell_group "uname -r"
dcli -l root -g cell_group cellcli -e list cell
dcli -l root -g cell_group /opt/oracle.cellos/CheckHWnFWProfile - Additional checks (if there were problems)
cd /u01/exa_img_upg/CELL/patch_XXXXXXXX
cat patchmgr.stdout
cat _wip_stdout file
ssh <cell-node>
cd /var/log/cellos
grep -i 'fail' validations.log
grep -i 'fail' vldrun*.log
cat validations.log
cat vldrun.upgrade_reimage_boot.log
cat vldrun.first_upgrade_boot.log
cat CheckHWnFWProfile.log
cat cell.bin.install.log
cat cellFirstboot.log
cat exachkcfg.log
cat patch.out.place.sh.log
cat install.sh.log - Start agents and verify that they are running
Check agents and restart if not running
ps -ef | grep agent | grep java | sed 's/\s\+/ /g' | cut -d " " -f 1,8 | sed 's/\/jdk.*//
dcli -g dbs_group -l root 'su -l oracle -c "/u01/app/em12c/core/12.1.0.4.0/bin/emctl start agent"
dcli -g dbs_group -l root 'su -l oracle -c "/u01/app/em12c/core/12.1.0.4.0/bin/emctl status agent"' | grep 'Agent is' - Enable and start CRS
dcli -l root -g dbs_group /u01/app/12.1.0.2/grid/bin/crsctl enable crs
dcli -l root -g dbs_group /u01/app/12.1.0.2/grid/bin/crsctl start crs
dcli -l root -g dbs_group '/u01/app/12.1.0.2/grid/bin/crsctl check crs | grep online | wc -l | while read retval; do if [[ $retval -eq 0 ]]; then echo CRS Stopped; elif [[ $retval -eq 4 ]]; then echo CRS Running; else echo CRS Not Ready; fi; done;' - Verify databases started
/u01/app/12.1.0.2/grid/bin/crsctl status resource -t -w "TYPE = ora.database.type"
dcli -l root -g dbs_group "ps -ef | sort -k1.50 | grep pmon_ | grep -v grep && echo ''"
Rolling Back Successfully Patched Exadata Cells
(This section describes how to roll back successfully patched Exadata Cells. Cells with incomplete or failed patching cannot be rolled back.)
(This section describes how to roll back successfully patched Exadata Cells. Cells with incomplete or failed patching cannot be rolled back.)
Do not run more than one instance of the patchmgr utility at a time in the deployment.
Check the prerequisites using the following command:
./patchmgr -cells cell_group -rollback_check_prereq [-rolling]
Perform the rollback using the following command:
./patchmgr -cells cell_group –rollback [-rolling]
Check the prerequisites using the following command:
./patchmgr -cells cell_group -rollback_check_prereq [-rolling]
Perform the rollback using the following command:
./patchmgr -cells cell_group –rollback [-rolling]
Click on to for compute node image upgrade.
Click on to for Switch Firmware upgrade.
Please check our other blogs for Exadata.
Comments
Post a Comment
Please do not enter any spam link in comment Section suggestions are Always Appreciated. Thanks.. !