From Exadata oracle have replaced infini Switch with RoCE Switch. In this blog , We will show the steps to patch Exadata RoCE switch.
Pre-Check:
Collect below information from the RoCE switches:
show running-config
show lldp neighbor
dir bootflash:
show interface status
show version
Patch RoCE Switch:
Patch should be uncompressed by a non root user (dbmadmin in this example) and in a directory owned by the same non root user.
For this example /u01/install is used.Patchmanager should be executed by a non root user on pre 20.1.0.0.0 . dbmadmin is used for this example.
1) Set up Directories owned by user "dbmadmin" and copy/unzip patch file.
[root@node_name u01]#cd /u01
[root@node_name u01]# mkdir install
[root@node_name u01]# chown dbmadmin install
[root@node_name u01]# ls -l
total 0
drwxr----- 2 root root 6 Sep 24 11:42 crashfiles
drwxr-xr-x 2 dbmadmin root 6 Jan 30 13:19 install
[root@node_name u01]# cd install
[root@node_name install]# mkdir scratchpad
[root@node_name install]# chown dbmadmin scratchpad
[root@node_name /]# cd /u01
[root@node_name u01]# ls
crashfiles install p30547087_193000_Linux-x86-64.zip
[root@node_name u01]# mv p30547087_193000_Linux-x86-64.zip install <-----------------------File previously SCP to this directory /u01
[root@node_name u01]#
[root@node_name u01]# su dbmadmin <----------------------dbmadm user account is created by default
[dbmadmin@node u01]$ cd install
dbmadmin@node install]$unzip p30547087_193000_Linux-x86-64.zip
[dbmadmin@node install]$ ls -l
total 2773844
-rwxr-xr-x 1 root root 2840400433 Jan 30 13:26 p30547087_193000_Linux-x86-64.zip
drwxrwxr-x 6 dbmadmin dbmadmin 4096 Feb 1 04:52 patch_switch_19.3.3.0.0.191211.1
-rwxr-x--- 1 dbmadmin root 30 Jan 30 13:28 roceswitches.lst <------------Create this file as described in the documentation
drwxr-xr-x 4 dbmadmin root 4096 Feb 1 04:52 scratchpad
2) Configure ssh Equivalency
[dbmadmin@node patch_switch_19.3.3.0.0.191211.1]$ ./roce_switch_api/setup_switch_ssh_equiv.sh xxxIPxxx,xxxIPxxx
enter switch 'admin' password: <--------------------Enter the RoCE switch password
generating key pair for user 'dbmadmin'... [OK]
checking if 'xxxIPxxx' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'xxxIPxxx'... [OK]
checking if 'xxxIPxxx' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'xxxIPxxx'... [OK]
3) Verify the Configuration
[dbmadmin@node install]$ cd patch_switch_19.3.3.0.0.191211.1
[dbmadmin@node patch_switch_19.3.3.0.0.191211.1]$ time ./patchmgr --roceswitches /u01/install/roceswitches.lst --verify-config --log_dir /u01/install/scratchpad/
2020-02-01 05:55:41 -0800 :Working: Initiate config verify on RoCE switches from . Expect up to 6 minutes for each switch
[NOTE ] Password equivalency is NOT setup for user 'dbmadmin' to switch-rocea0 from 'node.#####.com'. Set it up? (y/n): y
enter switch 'admin' password: <--------------------Enter the RoCE switch password if prompted
checking if 'switch-rocea0' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'switch-rocea0'... [OK]
[NOTE ] Password equivalency is NOT setup for user 'dbmadmin' to switch-roceb0 from 'node.#####.com'. Set it up? (y/n): y
enter switch 'admin' password: <--------------------Enter the RoCE switch password if prompted
checking if 'switch-roceb0' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'switch-roceb0'... [OK]
2020-02-01 05:56:42 -0800 1 of 2 :Verifying config on switch switch-rocea0
2020-02-01 05:56:42 -0800: [INFO ] Dumping current running config locally as file: /u01/install/scratchpad//run.switch-rocea0.cfg
2020-02-01 05:56:44 -0800: [SUCCESS ] Backed up switch config successfully
2020-02-01 05:56:44 -0800: [INFO ] Validating running config against template [1/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch.cfg
2020-02-01 05:56:44 -0800: [INFO ] Validating running config against template [2/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 05:56:44 -0800: [INFO ] Config matches template: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 05:56:44 -0800: [SUCCESS ] Config validation successful!
2020-02-01 05:56:44 -0800 2 of 2 :Verifying config on switch switch-roceb0
2020-02-01 05:56:44 -0800: [INFO ] Dumping current running config locally as file: /u01/install/scratchpad//run.switch-roceb0.cfg
2020-02-01 05:56:45 -0800: [SUCCESS ] Backed up switch config successfully
2020-02-01 05:56:45 -0800: [INFO ] Validating running config against template [1/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch.cfg
2020-02-01 05:56:45 -0800: [INFO ] Validating running config against template [2/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 05:56:46 -0800: [INFO ] Config matches template: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 05:56:46 -0800: [SUCCESS ] Config validation successful!
2020-02-01 05:56:46 -0800 :SUCCESS: Config check on RoCE switch(es)
2020-02-01 05:56:46 -0800 :SUCCESS: Completed run of command: ./patchmgr --roceswitches /u01/install/roceswitches.lst --verify-config --log_dir /u01/install/scratchpad/
2020-02-01 05:56:46 -0800 :INFO : config attempted on nodes in file /u01/install/roceswitches.lst: [switch-rocea0 switch-roceb0]
2020-02-01 05:56:46 -0800 :INFO : For details, check the following files in /u01/install/scratchpad/:
2020-02-01 05:56:46 -0800 :INFO : - updateRoceSwitch.log
2020-02-01 05:56:46 -0800 :INFO : - updateRoceSwitch.trc
2020-02-01 05:56:46 -0800 :INFO : - patchmgr.stdout
2020-02-01 05:56:46 -0800 :INFO : - patchmgr.stderr
2020-02-01 05:56:46 -0800 :INFO : - patchmgr.log
2020-02-01 05:56:46 -0800 :INFO : - patchmgr.trc
2020-02-01 05:56:46 -0800 :INFO : Exit status:0
2020-02-01 05:56:46 -0800 :INFO : Exiting.
real 1m21.878s
user 0m5.055s
sys 0m0.970s
[dbmadmin@node patch_switch_19.3.3.0.0.191211.1]$
4) RoCE Switch Upgrade Precheck
[dbmadmin@node patch_switch_19.3.3.0.0.191211.1]$time ./patchmgr --roceswitches /u01/install/roceswitches.lst --upgrade --roceswitch-precheck --log_dir /u01/install/scratchpad/
[NOTE ] Password equivalency is NOT setup for user 'dbmadmin' to switch-rocea0 from 'node.#####.com'. Set it up? (y/n): y
enter switch 'admin' password: <--------------------Enter the RoCE switch password if prompted
checking if 'switch-rocea0' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'switch-rocea0'... [OK]
[NOTE ] Password equivalency is NOT setup for user 'dbmadmin' to switch-roceb0 from 'node.#####.com'. Set it up? (y/n): y
enter switch 'admin' password: <--------------------Enter the RoCE switch password if prompted
checking if 'switch-roceb0' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'switch-roceb0'... [OK]
2020-02-01 05:59:17 -0800 :Working: Initiate pre-upgrade validation check on 2 RoCE switch(es).
2020-02-01 05:59:18 -0800 1 of 2 :Updating switch switch-rocea0
2020-02-01 05:59:19 -0800: [INFO ] Switch switch-rocea0 will be upgraded from nxos.7.0.3.I7.6.bin to nxos.7.0.3.I7.7.bin
2020-02-01 05:59:19 -0800: [INFO ] Checking for free disk space on switch
2020-02-01 05:59:20 -0800: [INFO ] disk is 96.00% free, available: 113169051648 bytes
2020-02-01 05:59:20 -0800: [SUCCESS ] There is enough disk space to proceed
2020-02-01 05:59:21 -0800: [INFO ] Found nxos.7.0.3.I7.7.bin on switch, skipping download
2020-02-01 05:59:21 -0800: [INFO ] Verifying sha256sum of bin file on switch
2020-02-01 05:59:34 -0800: [SUCCESS ] sha256sum matches: dce664f1a90927e9dbd86419681d138d3a7a83c5ea7222718c3f6565488ac6d0
2020-02-01 05:59:34 -0800: [INFO ] Performing FW install pre-check of nxos.7.0.3.I7.7.bin (eta: 2-3 minutes)
2020-02-01 06:01:35 -0800: [SUCCESS ] FW install pre-check completed successfully
2020-02-01 06:01:35 -0800 2 of 2 :Updating switch switch-roceb0
2020-02-01 06:01:37 -0800: [INFO ] Switch switch-roceb0 will be upgraded from nxos.7.0.3.I7.6.bin to nxos.7.0.3.I7.7.bin
2020-02-01 06:01:37 -0800: [INFO ] Checking for free disk space on switch
2020-02-01 06:01:38 -0800: [INFO ] disk is 96.00% free, available: 112907247616 bytes
2020-02-01 06:01:38 -0800: [SUCCESS ] There is enough disk space to proceed
2020-02-01 06:01:38 -0800: [INFO ] Found nxos.7.0.3.I7.7.bin on switch, skipping download
2020-02-01 06:01:38 -0800: [INFO ] Verifying sha256sum of bin file on switch
2020-02-01 06:01:52 -0800: [SUCCESS ] sha256sum matches: dce664f1a90927e9dbd86419681d138d3a7a83c5ea7222718c3f6565488ac6d0
2020-02-01 06:01:52 -0800: [INFO ] Performing FW install pre-check of nxos.7.0.3.I7.7.bin (eta: 2-3 minutes)
2020-02-01 06:03:52 -0800: [SUCCESS ] FW install pre-check completed successfully
2020-02-01 06:03:52 -0800 :Working: Initiate config verify on RoCE switches from . Expect up to 6 minutes for each switch
2020-02-01 06:03:54 -0800 1 of 2 :Verifying config on switch switch-rocea0
2020-02-01 06:03:54 -0800: [INFO ] Dumping current running config locally as file: /u01/install/scratchpad//run.switch-rocea0.cfg
2020-02-01 06:03:55 -0800: [SUCCESS ] Backed up switch config successfully
2020-02-01 06:03:55 -0800: [INFO ] Validating running config against template [1/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch.cfg
2020-02-01 06:03:55 -0800: [INFO ] Validating running config against template [2/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:03:55 -0800: [INFO ] Config matches template: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:03:55 -0800: [SUCCESS ] Config validation successful!
2020-02-01 06:03:55 -0800 2 of 2 :Verifying config on switch switch-roceb0
2020-02-01 06:03:55 -0800: [INFO ] Dumping current running config locally as file: /u01/install/scratchpad//run.switch-roceb0.cfg
2020-02-01 06:03:56 -0800: [SUCCESS ] Backed up switch config successfully
2020-02-01 06:03:56 -0800: [INFO ] Validating running config against template [1/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch.cfg
2020-02-01 06:03:56 -0800: [INFO ] Validating running config against template [2/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:03:57 -0800: [INFO ] Config matches template: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:03:57 -0800: [SUCCESS ] Config validation successful!
2020-02-01 06:03:57 -0800 :SUCCESS: Config check on RoCE switch(es)
2020-02-01 06:03:57 -0800 :SUCCESS: Initiate pre-upgrade validation check on RoCE switch(es).
2020-02-01 06:03:57 -0800 :SUCCESS: Completed run of command: ./patchmgr --roceswitches /u01/install/roceswitches.lst --upgrade --roceswitch-precheck --log_dir /u01/install/scratchpad/
2020-02-01 06:03:57 -0800 :INFO : upgrade attempted on nodes in file /u01/install/roceswitches.lst: [switch-rocea0 switch-roceb0]
2020-02-01 06:03:57 -0800 :INFO : For details, check the following files in /u01/install/scratchpad/:
2020-02-01 06:03:57 -0800 :INFO : - updateRoceSwitch.log
2020-02-01 06:03:57 -0800 :INFO : - updateRoceSwitch.trc
2020-02-01 06:03:57 -0800 :INFO : - patchmgr.stdout
2020-02-01 06:03:57 -0800 :INFO : - patchmgr.stderr
2020-02-01 06:03:57 -0800 :INFO : - patchmgr.log
2020-02-01 06:03:57 -0800 :INFO : - patchmgr.trc
2020-02-01 06:03:57 -0800 :INFO : Exit status:0
2020-02-01 06:03:57 -0800 :INFO : Exiting.
[dbmadmin@node patch_switch_19.3.3.0.0.191211.1]$
5) UPGRADE the RoCE Switches
[dbmadmin@node patch_switch_19.3.3.0.0.191211.1]$ time ./patchmgr --roceswitches /u01/install/roceswitches.lst --upgrade --log_dir /u01/install/scratchpad/
[NOTE ] Password equivalency is NOT setup for user 'dbmadmin' to switch-rocea0 from 'node.#####.com'. Set it up? (y/n): y
enter switch 'admin' password: <--------------------Enter the RoCE switch password if prompted
checking if 'switch-rocea0' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'switch-rocea0'... [OK]
[NOTE ] Password equivalency is NOT setup for user 'dbmadmin' to switch-roceb0 from 'node.#####.com'. Set it up? (y/n): y
enter switch 'admin' password: <--------------------Enter the RoCE switch password if prompted
checking if 'switch-roceb0' is reachable... [OK]
setting up SSH equivalency for 'dbmadmin' from node.#####.com to 'switch-roceb0'... [OK]
2020-02-01 06:16:32 -0800 :Working: Initiate upgrade of 2 RoCE switches to 7.0(3)I7(7) Expect up to 15 minutes for each switch
2020-02-01 06:16:32 -0800 1 of 2 :Updating switch switch-rocea0
2020-02-01 06:16:33 -0800: [INFO ] Switch switch-rocea0 will be upgraded from nxos.7.0.3.I7.6.bin to nxos.7.0.3.I7.7.bin
2020-02-01 06:16:33 -0800: [INFO ] Checking for free disk space on switch
2020-02-01 06:16:34 -0800: [INFO ] disk is 96.00% free, available: 113169051648 bytes
2020-02-01 06:16:34 -0800: [SUCCESS ] There is enough disk space to proceed
2020-02-01 06:16:35 -0800: [INFO ] Found nxos.7.0.3.I7.7.bin on switch, skipping download
2020-02-01 06:16:35 -0800: [INFO ] Verifying sha256sum of bin file on switch
2020-02-01 06:16:48 -0800: [SUCCESS ] sha256sum matches: dce664f1a90927e9dbd86419681d138d3a7a83c5ea7222718c3f6565488ac6d0
2020-02-01 06:16:48 -0800: [INFO ] Performing FW install pre-check of nxos.7.0.3.I7.7.bin (eta: 2-3 minutes)
2020-02-01 06:18:49 -0800: [SUCCESS ] FW install pre-check completed successfully
2020-02-01 06:18:49 -0800: [INFO ] Performing FW install of nxos.7.0.3.I7.7.bin on switch-rocea0 (eta: 3-7 minutes)
2020-02-01 06:21:59 -0800: [SUCCESS ] FW install completed
2020-02-01 06:21:59 -0800: [INFO ] Waiting for switch to come back online (eta: 6-8 minutes)
2020-02-01 06:29:59 -0800: [INFO ] Verifying if FW install is successful
2020-02-01 06:30:02 -0800: [SUCCESS ] switch-rocea0 has been successfully upgraded to nxos.7.0.3.I7.7.bin!
2020-02-01 06:30:02 -0800 2 of 2 :Updating switch switch-roceb0
2020-02-01 06:30:04 -0800: [INFO ] Switch switch-roceb0 will be upgraded from nxos.7.0.3.I7.6.bin to nxos.7.0.3.I7.7.bin
2020-02-01 06:30:04 -0800: [INFO ] Checking for free disk space on switch
2020-02-01 06:30:05 -0800: [INFO ] disk is 96.00% free, available: 112907247616 bytes
2020-02-01 06:30:05 -0800: [SUCCESS ] There is enough disk space to proceed
2020-02-01 06:30:06 -0800: [INFO ] Found nxos.7.0.3.I7.7.bin on switch, skipping download
2020-02-01 06:30:06 -0800: [INFO ] Verifying sha256sum of bin file on switch
2020-02-01 06:30:19 -0800: [SUCCESS ] sha256sum matches: dce664f1a90927e9dbd86419681d138d3a7a83c5ea7222718c3f6565488ac6d0
2020-02-01 06:30:19 -0800: [INFO ] Performing FW install pre-check of nxos.7.0.3.I7.7.bin (eta: 2-3 minutes)
2020-02-01 06:32:20 -0800: [SUCCESS ] FW install pre-check completed successfully
2020-02-01 06:32:20 -0800: [INFO ] Checking if previous switch switch-rocea0 is fully up before proceeding (attempt 1 of 3)
2020-02-01 06:32:22 -0800: [SUCCESS ] switch-rocea0 switch is fully up and running
2020-02-01 06:32:22 -0800: [INFO ] Performing FW install of nxos.7.0.3.I7.7.bin on switch-roceb0 (eta: 3-7 minutes)
2020-02-01 06:35:29 -0800: [SUCCESS ] FW install completed
2020-02-01 06:35:29 -0800: [INFO ] Waiting for switch to come back online (eta: 6-8 minutes)
2020-02-01 06:43:29 -0800: [INFO ] Verifying if FW install is successful
2020-02-01 06:43:33 -0800: [SUCCESS ] switch-roceb0 has been successfully upgraded to nxos.7.0.3.I7.7.bin!
2020-02-01 06:43:33 -0800 :Working: Initiate config verify on RoCE switches from . Expect up to 6 minutes for each switch
2020-02-01 06:43:35 -0800 1 of 2 :Verifying config on switch switch-rocea0
2020-02-01 06:43:35 -0800: [INFO ] Dumping current running config locally as file: /u01/install/scratchpad//run.switch-rocea0.cfg
2020-02-01 06:43:36 -0800: [SUCCESS ] Backed up switch config successfully
2020-02-01 06:43:36 -0800: [INFO ] Validating running config against template [1/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch.cfg
2020-02-01 06:43:36 -0800: [INFO ] Validating running config against template [2/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:43:36 -0800: [INFO ] Config matches template: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:43:36 -0800: [SUCCESS ] Config validation successful!
2020-02-01 06:43:36 -0800 2 of 2 :Verifying config on switch switch-roceb0
2020-02-01 06:43:36 -0800: [INFO ] Dumping current running config locally as file: /u01/install/scratchpad//run.switch-roceb0.cfg
2020-02-01 06:43:38 -0800: [SUCCESS ] Backed up switch config successfully
2020-02-01 06:43:38 -0800: [INFO ] Validating running config against template [1/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch.cfg
2020-02-01 06:43:38 -0800: [INFO ] Validating running config against template [2/3]: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:43:38 -0800: [INFO ] Config matches template: /u01/install/patch_switch_19.3.3.0.0.191211.1/roce_switch_templates/roce_leaf_switch_multi.cfg
2020-02-01 06:43:38 -0800: [SUCCESS ] Config validation successful!
2020-02-01 06:43:38 -0800 :SUCCESS: Config check on RoCE switch(es)
2020-02-01 06:43:38 -0800 :SUCCESS: upgrade 2 RoCE switch(es) to 7.0(3)I7(7)
2020-02-01 06:43:38 -0800 :SUCCESS: Completed run of command: ./patchmgr --roceswitches /u01/install/roceswitches.lst --upgrade --log_dir /u01/install/scratchpad/
2020-02-01 06:43:38 -0800 :INFO : upgrade attempted on nodes in file /u01/install/roceswitches.lst: [switch-rocea0 switch-roceb0]
2020-02-01 06:43:38 -0800 :INFO : For details, check the following files in /u01/install/scratchpad/:
2020-02-01 06:43:38 -0800 :INFO : - updateRoceSwitch.log
2020-02-01 06:43:38 -0800 :INFO : - updateRoceSwitch.trc
2020-02-01 06:43:38 -0800 :INFO : - patchmgr.stdout
2020-02-01 06:43:38 -0800 :INFO : - patchmgr.stderr
2020-02-01 06:43:38 -0800 :INFO : - patchmgr.log
2020-02-01 06:43:38 -0800 :INFO : - patchmgr.trc
2020-02-01 06:43:38 -0800 :INFO : Exit status:0
2020-02-01 06:43:38 -0800 :INFO : Exiting.
real 28m7.475s
user 0m10.331s
sys 0m1.567s
6) Log into all switches and confirm that the firmware is updated
show running-config
show lldp neighbor
dir bootflash:
show interface status
show version
NOTE: Downgrade is the same procedure . Just replace "--upgrade" in the command line to "--downgrade"
Comments
Post a Comment
Please do not enter any spam link in comment Section suggestions are Always Appreciated. Thanks.. !