[maintenance] Update FW on isar cluster nodes as available

[dmoessne]

Using the maintenance week for updating

• iDRAC/ILO/CIMC
• NIC FW
• BIOS

as available.

Completed already for ceph1[012]

[dmoessne]

inf4

[root@inf4 ~]# cd /home/core/                                                                                                                                                                                                                         [479/479]
[root@inf4 core]# ll                                                                                                                                                                                                                                           
total 268768                                                                                                                                                                                                                                                   
-rw-r--r--. 1 core core  30908459 Dec 17 13:28 BIOS_9M80P_LN64_2.25.0.BIN                                                                                                                                                                                      
-rw-r--r--. 1 core core  35763469 Dec 17 13:28 Network_Firmware_234W1_LN_23.0.8_A00_01.BIN                                                                                                                                                                     
-rw-r--r--. 1 root root     10755 May 14  2024 foo                                                                                                                                                                                                             
-rw-r--r--. 1 core core 208523599 Dec 17 13:28 iDRAC-with-Lifecycle-Controller_Firmware_VP556_LN64_7.00.00.183_A00.BIN                                                                                                                                         
[root@inf4 core]# chmod +x *.BIN                                                                                                                                                                                                                               
[root@inf4 core]# ./Network_Firmware_234W1_LN_23.0.8_A00_01.BIN                                                                                                                                                                                                
Collecting inventory...                                                                                                                                                                                                                                        
..                                                                                                                                                                                                                                                             
Running validation...                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                               
Intel(R) Ethernet 10G 4P X710 SFP+ rNDC                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                               
The version of this Update Package is newer than the currently installed version.                                                                                                                                                                              
Software application name: Intel(R) Ethernet 10G 4P X710 SFP+ rNDC                                                                                                                                                                                             
Package version: 23.0.8                                                                                                                                                                                                                                        
Installed version: 18.3.6                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                               
Continue? Y/N:Y                                                                                                                                                                                                                                                
Executing update...                                                                                                                                                                                                                                            
WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER PRODUCTS WHILE UPDATE IS IN PROGRESS.                                                                                                                                                                       
THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!                                                                                                                                                                                                        
.......................................................................................................................................................................................                                                                        
Device: Intel(R) Ethernet 10G 4P X710 SFP+ rNDC                                                                                                                                                                                                                
  Application: Intel(R) Ethernet 10G 4P X710 SFP+ rNDC                                                                                                                                                                                                         
  Update success.                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                               
Would you like to reboot your system now?                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                               
Continue? Y/N:N                                                                                                                                                                                                                                                
The system should be restarted for the update to take effect.                                                                                                                                                                                                  
[root@inf4 core]#

After that there have been some issues with applying the updates via cli regarding Temp full.
Switched to applying BIOS/iDRAC/NIC FW via iDRAC
Completed for inf4/5/6:

• NIC: 23.0.8
• BIOS Version 2.25.0
• iDRAC Firmware Version 7.00.00.183

[dmoessne]

Firmware updates for storm servers (of Isar cluster) have all the latest available FW installed

storm2,3,5,6
============
BIOS Version       2.19.0       (2024-03-18)
Firmware Version   2.86.86.86   (2020-04-03)
LC Firmware        2.86.86.86   (2020-04-03) 
Nic FW             19.5.12      (2020-02-14)
PERC H730 Mini     25.5.9.0001  (2024-05-03)

[dmoessne]

updating ucs5[567]:

• ucs55 has already the latest available FW updates installed 4.1(2m)
• ucs56 and ucs57 are at 4.1(2k)

[dmoessne]

using update tool -> ucs56 completed

[dmoessne]

ucs57 - update has Problem with 12G SAS Modular RAID controller --> Upgrade failed
Also visible in errors

[dmoessne]

searching for this issue suggests there is cached data for already removed disks pending ..

entered the raid controller (seems no disks configured at all) and cleared pending data. Trying to update FW again --> successful

[dmoessne]

last issue we had on ucs57 is a waring because of an Intel NVME not properly recognized --> removed --> warning gone

[dmoessne]

With this the isar cluster FW update is completed