HP Blade ESXi Driver/firmware updates

markpower28

Active Member
Apr 9, 2013
405
103
43
There is a reason HP split the business division away from the consumer division. With the pressure from UCS, both Dell and HP are try to adapt similar approach. It's one of the benefit of attending conference from MS, VMware, you have the option to direct engage with product managers or channel managers. I am sure they will be more than happy to assist.

*I did a UCS based VSAN solution build with assistant directly from UCS VSAN product team, worked out great.
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
Yeah that is where we are at. Out lab stack has been switched over to UCS VSAN. Unfortunately I did not get to attend VMWorld but am looking forward to it for next year.
 

Lupu

New Member
Sep 13, 2015
6
0
1
40
Just curious,
How does cisco gear help with firmware/esxi updates if you cannot take the blades off-line?

You should describe a little more your upgrade procedure that took 18mo because the
bottleneck isn't obvious, like `TuxDude` said one guy would have no problem
doing these upgrade in under a month using the SPP iso.
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
One guy can update 8700 servers in a month? Please explain the process you have that you think would allow one guy to do that.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,287
439
83
All depends on what your dependencies are. If your architecture is as such that you can reboot 8700 servers all at once, then it'll take about an hour to do all 8700 servers. If your architecture is as such that you can only reboot any two at a time then yes, it'll take you 18 months.
 

Lupu

New Member
Sep 13, 2015
6
0
1
40
modder man: I was referencing the earlier post from TuxDude which explained a typical(realistic) upgrade procedure. Since you didn't state how many blades you manage and didn't correct the 300 servers he mentioned I assumed it was at least in the same ballpark.

With this new information in mind I agree that using the HP way of doing FW upgrades with the SPP-iso is nothing but a bad joke.

I am still genuinely curious how Cisco gear is helping you solve this problem.
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
I currently have no experience with the Cisco gear. From what the other guys here say it makes it easier to set a standard at the management level and the hosts just take the new standard any time they reboot. You are correct in saying that it does not solve all of the problems but it does allow us to not manually sit in from of two servers at a time watching SPP do its job slower than molasses.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,287
439
83
Hopefully you don't mean you let each blade boot the ISO and wait for it to do its glacial magic...? That does indeed take forever. Extract the ISO onto a windows/linux workstation, create your hosts in dependency groups, run the inventory beforehand and when the time comes let HPSUM on the workstation push out the updates and the reboot commands. That way you'll have only one machine doing the endless baseline inventorying, and it'll be doing that over local storage rather than the network.
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
Have any of you used Oneview 2.0 Integrated with Vcenter? IS it worth looking into at all? We have a 1.2 oneview deployment that I am told did not work at all but wondering if maybe it is better in 2.0.
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
I take it no one has used oneview 2.0? Also is it possible to add firmware to the SPP when making a custom ISO?
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
I am revisiting this as it has become a pressing project for me again. Does anyone have any outside the box ideas for firmware upgrades?

I have an idea i am going to chase after. I am going to use powershell commands against vcenter and ilo to accomplish rolling upgrades through a cluster.
 
Last edited:

Jerry Renwick

Active Member
Aug 7, 2014
202
37
28
40
As far as I am concerned, update your firmware and drivers to HP qualified baselines, you must ensure optimum reliability with smart components, SPP and associated supplements and HP SUM, which typically takes a long time to achieve the upgrade.
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
Our drivers are always updated to HP's current standard...the problem is with as many servers as we have if it is done manually we never reach the current version before newer versions come out. If there is a critical fix there is no way to roll it out in a timely manner.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,287
439
83
Is HPSUM not an option...? It's clunky as hell to be sure but its a damn site quicker and more reliable than our experiences with OneSpew.

If you want to do HP baselines and ad-hoc updates then I think you need to look into automating your process outside of HP tools since they're really not designed for that kind of usage. Sometimes we'll be able to roll an update into VUM for our ESX estate but nine times out of ten its quicker and more reliable to do that sort of stuff via ssh/psexec/powershell/language-o'choice. What's your existing solution for patch management?
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
I was never able to get HPSUM to work in a way I was happy with. that said I did not put a whole lot of effort into it as I could not find the documentation that I wanted for it. We have used psexec/powershell in the past and it works pretty well the problem that we have run into is that I do not believe it is possible to update the NIC firmware in this manner. I am currently using powershell to mount the SPP before powershell kicks off a remediation with Update manager.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,287
439
83
The way it handles NIC firmware is pretty shambolic, yeah; it's certainly possible to do but fraught with errors as half the time it'll update the NIC, drop offline and then refuse to do the rest. We usually copy it locally and run the update process in a background task (or via schtasks) or - more commonly now we're finally rid of our broadcom NICs - don't bother updating it at all unless we have to.
 

Lupu

New Member
Sep 13, 2015
6
0
1
40
Have you considered building a live PXE linux image with a automated fw update process?
- booting the pxe-image can be triggered trough a simple script that controls the dhcp server
- changing the boot order can be automated through ilo3/ilo4 (one time)
- downtime would be around 10-15min
 

modder man

Active Member
Jan 19, 2015
644
70
28
29
Well an update to this thread. I do have a powerCLI script that has been working very well for us to get firmware updates done. The script will MM a host and mount the SPP ISO to the host using HP powershell cmdlets. It will reboot the host and then when it comes back kickoff a VUM remediation to update the OS install itself. Once the host is back online it will then start the next host in the cluster. That whole process has been quite smooth. The issue we are currently running into is that it seems that VUM doesn't really do the "clean" install that we were hoping for, it is leaving behind old VIB versions which have actually been causing PSOD's for us. Since our upgrade a month or so ago we have seen about 25 PSOD's all related to old VIB versions.

Wondering if you have an better experience or other ideas on ways to update the host other than using VUM?