Patch and Update for 16G

Introduction

Microsoft Azure Stack Hub operators are faced with the enormous challenge of keeping their solution both secure and functional. They must ensure that the solution is not vulnerable to threats, external or internal, while maintaining negotiated service-level agreements.

Solution overview

As a Microsoft Azure Stack Hub systems provider, Dell Technologies is committed to ensuring that your Azure Stack Hub environment is both secure and functional. In Dell Technologies releases, you will find firmware and driver updates for both the Hardware Lifecycle Host (HLH) and the scale unit (SU) nodes that are contained in the OEM package updates. Dell Technologies delivers these updates quarterly, and you can apply them independently of Microsoft updates.

There are two other types of update packages for the Azure Stack Hub integrated systems: software updates and hotfixes.

For more information, see Update package types - Azure Stack Hub.

Audience

This patch and update guide is for Azure Stack Hub 16th-generation (16G) operators who intend to perform lifecycle operations (Patch and Update) for:

Firmware updates for the Hardware Lifecycle Host (HLH)
Operating System updates for the HLH (Windows Server 2022)
Driver updates for the HLH (Windows Server 2022)
Windows Defender Application Control (WDAC) policy updates for the HLH Operating System
Firmware updates for the Top-of-Rack (ToR) switches (S5248F-ON)
Firmware updates for the scale unit (SU) nodes
Driver updates for the scale unit nodes (Windows Server 2022)

Dell Integrated System for Microsoft Azure Stack Hub Lifecycle Manager overview

Dell Integrated System for Microsoft Azure Stack Hub Lifecycle Manager provides an automated way of upgrading HLH firmware, software, and driver components.

Lifecycle Manager will perform the following tasks:

Update firmware on the Hardware Lifecycle Host
Update drivers on the Hardware Lifecycle Host
Install Windows updates for the Hardware Lifecycle Host Operating System
Update the Windows Defender Application Control (WDAC) policy on the Hardware Lifecycle Host

NOTE

Firmware updates for the Top-of-Rack (ToR) switches are not included in the Lifecycle Manager update process and must be updated manually from the Hardware Lifecycle Host (HLH) using SCP and SSH commands. See the section Manually Updating the Switch Firmware from the Hardware Lifecycle Host (HLH) for more information.

Firmware and driver updates for scale unit (SU) nodes are not included in the Lifecycle Manager update process and must be updated manually using the Azure Stack Hub administrator portal. See the section OEM package updates for more information.

Lifecycle Manager should take between 1-2 hours to complete.

Lifecycle Manager is available for download on the Integrated System for Microsoft Azure Stack Hub 16G > Drivers & Downloads product support website.

OEM package updates

The Azure Stack Hub OEM package update contains firmware, device drivers, and hardware management tools in a specialized format used by Azure Stack Hub during initial deployment and update.

Dell Technologies is responsible for the end-to-end servicing lifecycle for the hardware-related firmware and driver update packages.

In addition, Dell Technologies will own and maintain guidance for all software and hardware on the HLH.

OEM package updates are available for download on the Integrated System for Microsoft Azure Stack Hub 16G > Drivers & Downloads product support website.

Prerequisites

Perform the following steps before proceeding with the Lifecycle Manager update process. There are some steps which are a one-time-activity and some that you must perform in every release.

Download the Lifecycle Manager zip file

To download the latest Lifecycle Manager zip file, perform the following steps:

Steps

Go to the Dell Technologies Support website at Integrated System for Microsoft Azure Stack Hub 16G > Drivers & Downloads and download the latest Lifecycle Manager zip file.
Log in to the HLH with the HLHAdmin default administrator account or a customer specified administrative user account.
- The HLH can be accessed via Remote Desktop Protocol (RDP) from a Permitted Network or directly via the iDRAC Direct port (USB to Micro-USB) connection.
  - See Permitted Networks - Azure Stack Hub for additional details.
On the HLH, navigate to the E:\ drive and create a folder named Temp.
Download the Lifecycle Manager zip file to the HLH E:\Temp directory.

Download the OEM Update Package zip file

To download the corresponding OEM package update zip file, perform the following steps:

Steps

Go to the Dell Technologies Support website at Integrated System for Microsoft Azure Stack Hub 16G > Drivers & Downloads and download the latest OEM package update zip file.

WARNING

The Lifecycle Manager zip file version must match the version of the OEM package zip file, otherwise the Lifecycle Manager will not pass the pre-checks.

For example, if the Lifecycle Manager zip file is 2502, then the OEM package zip file must also be 2502.
Log in to the HLH with the HLHAdmin default administrator account or a customer specified administrative user account.
- The HLH can be accessed via Remote Desktop Protocol (RDP) from a Permitted Network or directly via the iDRAC Direct port (USB to Micro-USB) connection.
  - See Permitted Networks - Azure Stack Hub for additional details.
Download the OEM package update zip file to the HLH E:\Temp directory.

Extract Lifecycle Manager and OEM package update zip files

Lifecycle Manager zip and OEM package update zip files that were downloaded to the HLH from the Dell support website will need to be extracted into their own directories.

To extract Lifecycle Manager and OEM package update zip files, perform the following steps:

Steps

Before beginning the extraction process, first the files in the E:\Temp directory will need to be unblocked. Open a PowerShell terminal as administrator and type the following command:
```
Get-ChildItem -Path "E:\Temp" -Recurse | Unblock-File
```
WARNING
Lifecycle Manager zip file contents must be unblocked; otherwise, the Lifecycle Manager will not pass the pre-checks.
Once the contents within the E:\Temp directory have been unblocked, you can begin to extract the zip files into their directories.
On the HLH, navigate to the E:\ drive and create a folder named LCM.
On the HLH, navigate to the E:\ drive and create a folder named OEM.
Extract the Lifecycle Manager zip file from the E:\Temp directory to the E:\LCM directory.

Extract the OEM package update zip file from the E:\Temp directory to the E:\OEM directory.

Hardware Lifecycle Host (HLH)

Running Lifecycle Manager

NOTE

Lifecycle Manager will reboot the HLH multiple times during the upgrade process. You will need to log back into the HLH manually after each reboot; otherwise, the upgrade will not continue until you do.

As an optional step you can enable the Auto Logon feature on the HLH. This will allow the HLH to automatically log back in after each reboot.

For more information on how to enable the Auto Logon feature, see Turn on automatic logon in Windows.

To run Lifecycle Manager on the HLH, perform the following steps:

Steps

WARNING

Never use PowerShell ISE at any time during the use of this document.

From the HLH, open a PowerShell console window as an administrator.
You will need to export the BitLocker recovery keys before proceeding with the upgrade. Type the command below into your PowerShell prompt to export your BitLocker recovery keys.
```
(Get-BitLockerVolume -MountPoint D, E).KeyProtector | Where-Object -FilterScript {$_.KeyProtectorType -eq 'RecoveryPassword'} | Select-Object -Property KeyProtectorId,RecoveryPassword | Out-File -FilePath "E:\Temp\BitLockerRecoveryKeys.txt"
```
CAUTION

Once the BitLocker recovery keys have been exported into the E:\Temp folder, ensure they are moved to a secure location outside of the HLH.

Failure to do so may result in an inaccessible HLH due to firmware or hardware issues during the firmware upgrade process.

For more information see BitLocker Operations Guide.

Change the current directory to the E:\LCM location that the Lifecycle Manager zip file was extracted to in a previous step:
```
Set-Location -Path "E:\LCM"
```
Run Lifecycle Manager by using the following command:
```
.\Invoke-DellAzSHubLifecyclemanager.ps1 -OEMPackagePath "E:\OEM\AzS-Dell-<version>.zip"
```
You will be prompted to provide credentials for the following endpoints:
- Hardware Lifecycle Host OS administrative account credential
- iDRAC administrative account credential for the Hardware Lifecycle Host
You will then be prompted to confirm that you have backed up the BitLocker recovery keys. Press Y to confirm if you have backed up the BitLocker recovery keys; otherwise, press N to cancel the update process and back up your BitLocker recovery keys.

Once the update process begins, continue to monitor Lifecycle Manager while it runs. While the automation is running there will be multiple reboots, in order to keep the upgrade process progressing you must log back into the HLH manually; otherwise, the upgrade will not continue until you do.

Some updates are grouped together to limit the amount of reboots. The HLH may reboot for any of the following reasons:

Update Type	Reboot Type
Firmware updates	Separate reboot
Driver updates	Grouped reboot
Operating System updates	Grouped reboot
Windows Defender Application Control (WDAC) policy updates	Grouped reboot

Upon completion, the output should look similar to the image shown below. This shows a list of all completed tasks from Lifecycle Manager, as well as the total elapsed time of the upgrade.

If the upgrade was successful, you will see the Dell AzS Hub Lifecycle Manager completed successfully message in the output.

Finally, you can gather logs by navigating to the directory E:\LCM\Logs on the HLH. These logs can be used for troubleshooting purposes if needed.
- Additionally, C:\MASLogs will contain logs related to the firmware update process.

Top-of-Rack (ToR) Switches

Manually Updating the Switch Firmware from the Hardware Lifecycle Host (HLH)

You must manually install the OS10 software image from the HLH using SCP and SSH commands.

WARNING

To maintain the integrity of Dell Azure Stack Hub Integrated System, it is required to perform the switch firmware update in conjunction with the Lifecycle Manager update process.

To update switch firmware, perform the following steps:

Steps

WARNING

Never use PowerShell ISE at any time during the use of this document.

From the HLH, open a PowerShell console window as an administrator.
- The HLH can be accessed via Remote Desktop Protocol (RDP) from a Permitted Network.

Change directory to E:\OEM\AzS-Dell-<version>\Firmware\Switches\msftdell

Set-Location -Path "E:\OEM\AzS-Dell-<version>\Firmware\Switches\msftdell"

Run the following command to copy the switch firmware from the extracted OEM package update zip file on the HLH to the ToR switches.

For example, if upgrading the firmware on ToR-1 with an IP address of 10.128.164.98 and a switch administrative user account of azsadmin-gtbzp, you would run the following:
```
scp -o MACs=hmac-sha2-512 .\PKGS_OS10-Enterprise-<version>-installer-x86_64.bin azsadmin-gtbzp@10.128.164.98:
```
Repeat the above step for ToR-2.

For example, if upgrading the firmware on ToR-2 with an IP address of 10.128.164.99 and a switch administrative user account of azsadmin-gtbzp, you would run the following:
```
scp -o MACs=hmac-sha2-512 .\PKGS_OS10-Enterprise-<version>-installer-x86_64.bin azsadmin-gtbzp@10.128.164.99:
```

The below steps will be repeated on each ToR switch (e.g. ToR-1 and ToR-2) ONLY after fully completing the first switch (e.g. ToR-1).

Run the following commands to connect to the ToR switch.

For example, if upgrading the firmware on ToR-1 with an IP address of 10.128.164.98 and a switch administrative user account of azsadmin-gtbzp, you would run the following:
```
ssh -o MACs=hmac-sha2-512 azsadmin-gtbzp@10.128.164.98
```
- Enter the password of the switch administrative user account when prompted.
Run the following command to move the SCP copied firmware file to its intended destination (/var/os10-image):
```
system "sudo mv ./PKGS_OS10-Enterprise-<version>-installer-x86_64.bin /var/os10-image"
```
- Enter the password of the switch administrative user account when prompted.

Verify the image is now located in the image directory.

dir image

The output should look similar to the below example:

Directory contents for folder:
Date (modified)        Size (bytes)  Name
---------------------  ------------  ------------------------------------------
2025-03-18T18:37:56+00:667040481     PKGS_OS10-Enterprise-<version>-installer-x86_64.bin

To verify that an installation is not already in progress, run the following command:

show image status

The output should look similar to the below example:

Image Upgrade State:     idle
==================================================
File Transfer State:     idle
--------------------------------------------------
  State Detail:          No download information available
  Task Start:            -001-11-30T00:00:00+00:00
  Task End:              -001-11-30T00:00:00+00:00
  Transfer Progress:     0 %
  Transfer Bytes:        0 bytes
  File Size:             0 bytes
  Transfer Rate:         0 kbps

Installation State:      idle
--------------------------------------------------
  State Detail:          No install information available
  Task Start:            -001-11-30T00:00:00+00:00
  Task End:              -001-11-30T00:00:00+00:00

Run the following command to install the downloaded image to the standby partition:
```
image install image://PKGS_OS10-Enterprise-<version>-installer-x86_64.bin
```

To view the image install progress, run the following command:

show image status

The output should look similar to the below example:

Image Upgrade State:     idle
==================================================
File Transfer State:     idle
--------------------------------------------------
  State Detail:          No download information available
  Task Start:            0000-00-00T00:00:00Z
  Task End:              0000-00-00T00:00:00Z
  Transfer Progress:     0 %
  Transfer Bytes:        0 bytes
  File Size:             0 bytes
  Transfer Rate:         0 kbps

Installation State:      install
--------------------------------------------------
  State Detail:          In progress: Configure filesystem (3 of 9)
  Task Start:            2025-03-17T21:15:16Z
  Task End:              0000-00-00T00:00:00Z

Keep checking until the image installation is complete. Then check to verify if the image is installed to the standby partition by running the following command:

show boot

The output should look similar to the below example:

Current system image information:
===================================
Type      Boot Type      Active                   Standby                 Next-Boot
-----------------------------------------------------------------------------------
Node-id 1 Flash Boot     [B] 10.5.6.5            [A] 10.6.0.1            [B] active

Set the next boot partition to the standby partition where the downloaded image is installed by running the following command:
```
boot system standby
```
Save the currently running configuration and then reload to the configured image by running the following commands:
```
write memory
reload
```
After the switch reloads, copy the new active image to the standby boot partition by running the following command:
```
image copy active-to-standby
```

Verify the status of the BGP connections on the switches by running the following command:

show ip bgp summary

Verify that BGP is established for all neighbors as indicated by the Up/Down column showing a connection timer. The output should look similar to the below example:

BGP router identifier 10.128.164.24 local AS number 64910
Neighbor                                      AS          MsgRcvd     MsgSent     Up/Down           State/Pfx
10.128.165.12                                 64912       11          12          00:07:01          25
10.128.165.13                                 64912       11          12          00:07:01          25
10.128.165.33                                 64912       10          10          00:07:01          25
10.128.164.1                                  64900       11          12          00:07:06          1
10.128.164.9                                  64900       11          12          00:07:06          1
10.128.164.30                                 64910       12          14          00:06:23          8
10.128.164.34                                 64910       12          14          00:07:06          8

Verify that the switch firmware version was updated by running the following command:

show version

The output should look similar to the below example:

Dell SmartFabric OS10 Enterprise
Copyright (c) 1999-2024 by Dell Inc. All Rights Reserved.
OS Version: 10.6.0.1
Build Version: 10.6.0.1.35
Build Time: 2024-12-02T17:55:39+0000
System Type: S5248F-ON
Architecture: x86_64
Up Time: 1 day 00:11:14

Original Equipment Manufacturer (OEM) package updates

Servicing Policy

Updates for the hardware components in the form of driver updates, firmware updates, and security patches are provided by Dell Technologies. These updates are referred to as OEM package updates. It is recommended to be within N-2 of the current available OEM package for Azure Stack Hub.

WARNING

Azure Stack Hub OEM package updates must be installed in sequential order.

It is not supported to skip an OEM package update version.

Identify the OEM version applied to the Azure Stack Hub

To identify the version of the Original Equipment Manufacturer (OEM) package currently applied to the Azure Stack Hub, perform the following steps:

Steps

Open the Azure Stack Hub administrator portal.
Open the Microsoft Azure Stack Hub - Administration Dashboard and click Dashboard.
Click Update.

The Updates blade will appear as shown below. The Updates blade displays your current Azure Stack Hub version and the latest deployed version of the OEM package update.

Import and Install an OEM package update

In this step you will be uploading and installing the OEM package update utilizing the Azure Stack Hub administrator portal.

CAUTION

Before you apply an OEM package update, always apply the latest Azure Stack Hub hotfix available for your Azure Stack Hub system.

For additional information about Azure Stack Hub updates, please see:

Steps

Access the Azure Stack Hub administrator portal of the system you wish to perform the OEM package update on.
Navigate to Storage accounts > updateadminaccount > Containers.

Create a container for the OEM package update by clicking +Container.
In the Name field on the right navigation pane enter the name for your container e.g. 2408oemupdate and then click Create.

Once the container is created, click the name of the container to access it.

Once you have clicked on the container click Upload at the top of the page.

Dell OEM package updates are in a zip format. The zip file contains the oemMetadata.xml file and the OEM package update payload. You will need to upload both the zip file for the OEM package update and the oemMetadata.xml that is associated with it.

Once uploaded to the storage account container, they should appear in the container that was created in step 3.
Navigate back to the Dashboard and then click the Update tile.

The Updates blade should now show an available OEM package update that can be installed.
Click the update and then click Install to install the OEM package update.
After clicking install on the OEM package update, the update run details can be seen by clicking on the update in the Updates menu once the update has entered into a Preparing or Installing state.

Once completed, Updates blade should display Applied successfully and in the Update history menu the update state should be marked as Installed.

If the update fails, the Update blade reports Needs attention. Click on the failed update and use the Download summary button to get a high-level status of where the update failed.

Last modified April 10, 2025: Additional line breaks (#143) (49156875)