Having never pushed a BFB image or worked with DPUs, can you confirm the steps here after I confirm in rshim mode to serve as a mini-guide here? Thank you - sincerely appreciate it.
Np. Your starting point is:
https://developer.nvidia.com/doca-downloads I believe 3.4.0 is the latest one (I could be wrong but it will give you a fresh 24.04 ubuntu image.) I only mentioned 3.2.2 because that was my starting point so you can just go with 3.4.0 for the latest image and firmware.
Downloading drivers and BFB images:
1. Host-Server >DOCA-Host > Linux > x86_64 > doca-all > Ubuntu > 24.04 > deb (local) -- (this is my host system, choose the right version for your host) -- this installs the drivers as instructed.
2. Bluefield > BF-Bundle > Ubuntu > 24.04 > BFB
I would download the BFB image to a working folder since you will need to also create config file as well. I just put the BFB in a folder called bluefield in my home dir. In this directory where the BFB image sits, run:
printf "ubuntu_PASSWORD='%s'\n" "$(openssl passwd -6)" >> bf.cfg
it will prompt for a password, enter one and this will be your password to the ubuntu DPU after imaging.
Now open bf.cfg, you should see a hashed version of your password. Now add the following to bf.cfg:
BMC_USER="root"
BMC_PASSWORD="<same hash>"
This should also reset the Redfish BMC password if it's not secure-locked by the previous owner. If it is then you're SOL with resetting BMC password. Also the default is root // 0penBmc (that's a zero).
UPDATE: This card does not have a BMC -- only the *-Cxxx cards. Ignore the BMC part of the config.
You can also customize the hostname by adding this in the
bf.cfg below ubuntu_Password:....
Code:
bfb_modify_os()
{
echo "bf2-card-0" > /mnt/etc/hostname
sed -i "s/127.0.1.1.*/127.0.1.1\tbf2-card-0/" /mnt/etc/hosts
# keep cloud-init from overwriting it on first boot
echo "preserve_hostname: true" > /mnt/etc/cloud/cloud.cfg.d/99-preserve-hostname.cfg
}
** Update bf2-card-0 to your desired hostname
Now run rshim (or at least make sure it's running):
sudo systemctl start rshim # start it now
sudo systemctl enable rshim # start on boot
sudo systemctl status rshim # verify it's running
ls -d /dev/rshim* (you should see at least 1 if you only have one device).
Before continuing, install screen and pv:
sudo apt install -y pv screen
Open another terminal and attach to screen to watch the card's serial console (this is handy to see the install status):
sudo screen /dev/rshim0/console 115200
In the other terminal where you're in the bluefield folder run (assuming you only have 1 bluefied-2 card and it's rshim0):
echo "SW_RESET 1" | sudo tee /dev/rshim0/miscs in the other terminal where you're running screen, watch the reboot for this screen:
Code:
Mellanox BlueField-2 [A1] A72(D08) 8 Cores r1p0
Memory Information:
DDR4 Memory Available: 16 GB Operating Frequency: 3200 MT/s
Current Secure Boot State: enabled
Secure Boot Mode : User Mode
PK is configured
Redfish enabled
Make sure the terminal with screen is in the foreground and press 'esc' twice, do a it a few times to make sure it registers. You'll be prompted for a password. If you're lucky and it wasn't changed then
bluefield is the default password. You'll be asked to change it. I just went with
HelloNvidia3D! since that was in the documentation. You're going to need to turn off
Secure Boot (notice that it's turned on. If it's turned off already then you won't need to do this).
**NOTE: I didn't mention this part in the documentation earlier because I did do this the first time around but didn't realize that it was related to why rshim flashing kept failing.
But the documentation does say that when Secure Boot is on then both emmc are write protected. Also after updating the BFB (which automatically updates everything else, firmware, etc). I re-ran rshim bfb-install and it worked even with Secure Boot on. Maybe there was some quirk with firmware my card was on before the upgrade? Probably related to certificates that gets installed when imaging.
** NOTE: If you go into the Password Settings here, you can change the password back to legacy mode with 'bluefield' as the password just so you don't forget. Earlier firmware versions seems to have had more relaxed password rules whereas trying to change the password via the normal change password settings won't let you use less than 12 characters.
This is where you can exit out of the UEFI menu by going back to the main screen and choose reset. If done correctly, you'll see this after it reboots:
Code:
Mellanox BlueField-2 [A1] A72(D08) 8 Cores r1p0
Memory Information:
DDR4 Memory Available: 16 GB Operating Frequency: 3200 MT/s
Current Secure Boot State: disabled
Secure Boot Mode : User Mode
PK is configured
Redfish enabled
At this point you can go back to the other terminal where your bluefield folder containing the BFB image and run (you do not need to force BOOT_MODE 0 like I mentioned in the previous post, bfb-install will write part of the file to one of the emmc partition and the boot process will recognize that there's an image to receive and flash):
sudo bfb-install --bfb bf-bundle-*.bfb --config bf.cfg --rshim rshim0
You'll notice your upload speed crawl from KiB/s to MiB/s speed and the imaging will start after the bfb image finishes uploading. The whole flashing process take a long time, like 30 minutes long. Make sure you don't lose power during the process where it updates your firmware after wiping and imaging the new Ubuntu image.
The serial terminal should also say this before imaging starts:
Code:
FmpDxe: EFI Capsule Authentication Successful, Status: Success.
[PMI] Boot image update started.
Trusted Board Boot is enabled (GA)
Note: Installed image will be filtered from 7406600 bytes to 3489280 bytes
Size check good, 3489280 <= 4194304
Verify Trusted Board Boot FW certificate.
Check boot image, Status: Success
PmiInstallBootImageInternal: RuntimeUpgrade 0
...Filtering unneeded executables...
...Preparing boot image...
...Generating boot image...
...Writing the boot image, Size: 3489792 bytes...
...Verify boot image...
Write boot image to primary partition 1, Status: Success
...Filtering unneeded executables...
...Preparing boot image...
...Generating boot image...
...Writing the boot image, Size: 3489792 bytes...
...Verify boot image...
Write boot image to backup partition 2, Status: Success
[PMI] Boot Image update completed, Status: Success
[PMI] Total number of updates: 1
[PMI] Errors during updates : 0
CapsuleRuntimeDxe: ProcessCapsuleImage 0, Status: Success
Mellanox BlueField-2 [A1] A72(D08) 8 Cores r1p0
Memory Information:
DDR4 Memory Available: 16 GB Operating Frequency: 3200 MT/s
Current Secure Boot State: disabled
Secure Boot Mode : User Mode
PK is configured
Redfish enabled
Once flashing is done and the DPU reboots and you see your ubuntu login prompt with your custom hostname, the last thing to do is a shutdown and pull the power. It seems that the old firmware is still in memory and persists until after all power is cut. A shutdown isn't enough.
** NOTE: If you want to turn on Secure Boot after imaging the new image (and automatic firmware upgrade) just follow the steps above to go back into the UEFI menu and turn Secure Boot back on. You don't need to reflash the bfb image.
**EDIT: I think the reason why bfb-install fails with secure boot on is because of the certificate that's installed with the existing image. The new image has a different set of certificates and mismatches what the system is expecting. I don't think it has anything to do with the file system being locked down or some quirk with the older firmware. Once the new image is written, subsequent flashing with the same image works even with Secure Boot on because the certificates are now on the system with the old ones wiped.
