Exadata(Half Rack) Image Upgrade (Rolling) Part-3-Compute Node Image Upgrade

As DMA its regular activity to patch exadata machine. There are two ways of patching Exadata box Rolling and Non-Rolling. In this blog, we will start with part 3 of the Exadata (Half Rack) Image Upgrade (Rolling).

Step 1: Check image version

cd /root

 dcli -l root -g dbs_group imageinfo -version

 dcli -l root -g dbs_group imageinfo -status

 dcli -l root -g dbs_group uname -r

--Verify dbnodeupdate script version
Download latest version of dbnodeupdate script from patch 21634633
Download dbserver.patch.zip which contains dbnodeupdate.zip and patchmgr for dbnodeupdate orchestration via patch 21634633.

cd /u01/exa_img_upg/YUM

 unzip -o p21634633_122110_Linux-x86-64.zip

 Should be at least version 5.151022

 ./dbnodeupdate.sh -V

 ver=$(./dbnodeupdate.sh -V | awk '{print $3}'); if (( $(echo "$ver < 5.151022" | bc -l) )); then echo -e "\nFAIL: dbnodeupdate version too low. Update before proceeding.\n"; elif (( $(echo "$ver > 5.151022" | bc -l) )); then echo -e "\nPASS: dbnodeupdate version OK\n"; else echo -e "\nWARN: dbnodeupdate minimum version ($ver) detected. Check if there is a newer version before proceeding.\n"; fi

 dbnodeupdate script is updated frequently (sometimes daily). If not current then download updated version.

Step 2: Check databases running before stopping CRS

crsctl status resource -t -w "TYPE = ora.database.type"

ps -ef | grep pmon_ | grep -v grep

Step 3: Stop the CRS (Rolling)

Execute on the Compute Node for which you are going for Image Upgrade.

crsctl disable crs

crsctl stop crs

crsctl check crs

ps -ef | grep grid | grep -v grep

Reboot server and reset ILOM

uptime

If uptime more than 7 days then reboot server

reboot

Reset the ilom

ipmitool bmc reset cold

Step 4: Unmount NFS partitions

umount -a -t nfs -f -l

Step 5: Run precheck

cd /u01/exa_img_upg/YUM/dbserver_patch_5.170420

./dbnodeupdate.sh -u -l /u01/exa_img_upg/YUM/IMAGE.zip -t PATCHNUMBER  -v

Step 6: Perform backup and upgrade

==========================

 Make sure to check known issues section above prior to executing dbnodeupdate.sh

        cd /u01/exa_img_upg/YUM/dbserver_patch_5.170420

 nohup ./dbnodeupdate.sh -u -l /u01/exa_img_upg/YUM/IMAGE.zip -t PATCHNUMBER -q &

Step 7: Monitor the reboot

==================

Monitor the reboot of each node by logging into the ilom console.

After reboot completes

Before running the completion step, run the CheckHWnFWProfile script to make sure it passes. If not, shut the system down and power cycle it from the ilom ( stop /SYS, wait 5 minutes, start /SYS)

 /opt/oracle.cellos/CheckHWnFWProfile

 cd /u01/exa_img_upg/YUM

 umount -a -t nfs -f -l

        cd /u01/exa_img_upg/YUM/dbserver_patch_5.170420

 ./dbnodeupdate.sh -t PATCHNUMBER -c

 mount -a

 Verify fuse RPMs are Installed

 yum list installed | grep fuse

 

Step 8: Check version and status

========================

cd /root

 dcli -l root -g dbs_group imageinfo -version

 dcli -l root -g dbs_group imageinfo -status

 dcli -l root -g dbs_group uname -r

Enable CRS

 crsctl enable crs

 crsctl check crs

If CRS is not already started then start the CRS on this Image Upgraded node.

crsctl start crs

Step 9: Post checks

===========

 /u01/app/12.1.0.2/grid/bin/crsctl status resource -t -w "TYPE = ora.database.type"

Additional checks (if there were problems)

 ssh <database-node>

 cd /var/log/cellos/

 cat dbnodeupdate.log

 cat dbserver_backup.sh.log

 cat CheckHWnFWProfile.log

 cat exadata.computenode.post.log

 cat cellFirstboot.log

 cat exachkcfg.log

 cat vldrun.each_boot.log

 cat validations.log

--------------------------------------------------------------------------------------

Rollback Steps in case required

 ===============================

 1. Rolling back the update with the dbnodeupdate.sh utility:

 ./dbnodeupdate.sh -r

 2. Reboot the server using the reboot command.

 # reboot

 3. Run the dbnodeupdate.sh utility in 'completion mode' to finish post patching steps

 Similar like with regular updates or One-Time updates, when switching OS binaries with the same Oracle Home, the database kernel should be relinked, so the 'post completion' step needs to be performed.

 ./dbnodeupdate.sh -c

Click on to for Switch Firmware upgrade.

Click on to for Cell node image upgrade.

You can learn in detail on Exadata from book Expert Oracle Exadata

 ==========================================================

Please check our other blogs for Exadata.