Where will you be installing the U.2 drives? Does your unit have the newer SIOCs with the U.2 ports? I thought those were only on the S3260M5 chassis.
AVOID - 4U, 2x Node, 4x E5 V3/V4, 56x LFF SAS3 3.5" bay - $299 - CISCO UCS C3260
- Thread starter eduncan911
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
You can install U.2 drives underneath the RAID mezzanine card. I have a couple of M.2 to U.2 adapters that allow me to use some NVMe SSDs I had.
To everyone else, there is a newer SOIC version (the device that has the 2x 40G ports) that adds two additional 2.5" 7mm U2 locations hidden under the SOIC, in addition to an x8 PCI half-height slot! Model number UCS-S3260-PCISIOC.
However, as stated by the Cisco docs, it's only compatible when using the newer M5 nodes (LGA 3647 Skylake). And no, you can't mix nodes either. Has to be both M4s or M5s.
Yes, you can boot from the U.2 device. You'll need to make sure you're booting in UEFI mode to boot from NVMe SSDs. Set the PCI cards to run in UEFI only mode and allow UEFI boot options.
I am using this adapter to convert regular M.2 to U.2:
We're running 2x E5-2680v4 chips with a TDP in ours, no issues at all. It's in a datacenter, so it's kept consistently cool.
Ordered mine this week! Haven't decided on what os I'm going to use for the storage node, either truenas, unraid or openmediavault. 2nd node I might just use for Nomad but not sure yet. Only have 2x qsfp on my primary switch, so will probably use break out cables or qsfp to sfp+ transceivers.
Well, threw a P4510 NVMe drive in it and now the fans are all stuck at 100%. Seems the temperature being reported to BMC is sky high (80C never varies) but inside the OS its 30C. These are Intel OEM, not Cisco branded.
Does your M.2-adapted NVMes show the correct temperature in the CIMC? Fans are normal?
Does your M.2-adapted NVMes show the correct temperature in the CIMC? Fans are normal?
Back early in the thread, someone reported the same problem. Your drive's firmware must support a temperature reading, even if it is fake (*cough* OCZ *cough*).
Once they updated the drive's firmware, the system's fans went back to purring.
OK, I tried that before posting using the Intel firmware utility, had a big leap from the version it was on but same problem. I see that Cisco uses the exact same NVMe in their servers so fired up the HUU and, naturally, doesn't detect the drives. Likely because they are on Intel firmware and not Cisco.
Debating what to do, send these back and pay $$$ for Cisco NVMe drives or try to find a working combination that doesn't cause it to constantly be in a "critical fault" state.
Debating what to do, send these back and pay $$$ for Cisco NVMe drives or try to find a working combination that doesn't cause it to constantly be in a "critical fault" state.
I don't see temps reported in CIMC for the NVMe SSD, but I do see them in the OS:
Fans are running high:
I'd love to see those fans ramp down some, but it may not be possible with a non Cisco U.2 drive installed. Power profile probably also has something to do with RPM. Additionally, I only have 2 drives installed in the chassis so far so there are a lot of open spaces.
Yea, same problem then. My temps are OK in the Chassis sensors but under the Compute node tab:
Installed the Intel utilities in Windows and extracted the Cisco firmware out of the HUU, lets see what I can break today.
Installed the Intel utilities in Windows and extracted the Cisco firmware out of the HUU, lets see what I can break today.
If the NVMe firmware is up to date via the OS then update CIMC.
There was an issue with M5 boards where below a certain version if a non-Cisco piece of hardware was detected it would ramp the fans. They fixed that bug and you can now turn them down.
I was the one with the NVMe firmware. I'm 99% that we updated via the Windows tool.
There was an issue with M5 boards where below a certain version if a non-Cisco piece of hardware was detected it would ramp the fans. They fixed that bug and you can now turn them down.
I was the one with the NVMe firmware. I'm 99% that we updated via the Windows tool.
Humm, Cisco firmware is going to suck. I wonder if anyone has ever done a pentest on the chassis firmware. Perhaps we can as round around any signatures and edit
@Slothstronaut have you updated all of the firmware for the chassis, bmc, everything Cisco? maybe even install Windows real quick and run through its cisco manager to update any additional firmwares?
Thanks! Yeah, was going to find who it was and ping. Humm, this seller in the OP is selling M4 nodes only though.
@Slothstronaut have you updated all of the firmware for the chassis, bmc, everything Cisco? maybe even install Windows real quick and run through its cisco manager to update any additional firmwares?
I'm not sure the firmware is Cisco specific. We have purchased after-market parts that are the same model but HPE or Dell branding and Cisco picks up the SKU and thinks it's a Cisco part.
When using UCS Manager there is a compatibility catalog that has every part Cisco can 'see'. In my experience as long as the part is the same model and the firmware above a certain threshold everything works fine.
Specifically for drives Cisco sources from everyone. There is no white-labeling of anything. Any drives purchased from Cisco are HGST/Toshiba/Intel with the OEM branding very clear and a little UCS sticker affixed to it. If you look at the spec sheets there are details on what types of drives can be mixed and matched. One of the reasons we keep buying their hardware is because it accepts anything..
When using UCS Manager there is a compatibility catalog that has every part Cisco can 'see'. In my experience as long as the part is the same model and the firmware above a certain threshold everything works fine.
Specifically for drives Cisco sources from everyone. There is no white-labeling of anything. Any drives purchased from Cisco are HGST/Toshiba/Intel with the OEM branding very clear and a little UCS sticker affixed to it. If you look at the spec sheets there are details on what types of drives can be mixed and matched. One of the reasons we keep buying their hardware is because it accepts anything..
Ah, ok. You updated the firmware, of the SSD, through the Windows management software (I'll try to remember "UCS Manager" from now). I thought it was more of the Chassis firmware.
Which starts to beg the question: Can all firmware can be updated outside of Windows? E.g. via Linux/fwupdmgr? Just another thing to document...
Going to dig that up again. I saw it before; but, was hoping it was not that proprietary.
So, what happens when you use drives not on that list though? Can you use non-spec'd items? It looks like a few reports are showing excessively high fan speeds when adding something not on the list (I'm tracking down some private reports some have sent me as well).
I put in an Intel P3500 series NVMe SSD and updated the firmware via their Linux CD boot disk. I was going to use Windows, but saw they had a boot disk and didn't need to run the Windows utility for firmware updates. Now the temperature is reported at 0C/30F in CIMC and the fans are at the high 7000's for RPM which is much more reasonable. There is a fault on the SSD firmware not being compatible, but it's a "Major" level fault.
That aside, the OS sees the SSD and reports the correct temperature and usage info:
managed to get into the CIMC but can't find where to adjust the fan speed.
@Slothstronaut @pcmantinker @Digital Spaceport were you able to find anything related to fan control?
furthermore, looking at the video from digital's post, in the storage area mine shows these:
anything past 10 drive installed will not show under OS and im not sure why. it doesn't show server 1 or server 2 like in he video, every drive just says none. when trying to assign a drive past 10th drive to server 1, it gives that error message. any ideas?
Sorry took a break from this project after a few hundred failed attempts to get things working smoothly. A few updates with what Ive been doing and what didnt work.
1. Installed a P4510 NVMe in each chassis, fans went to 100% and got a critical fault on every blade that the NVMe was overheating (it wasnt). Updated the firmware with Intels utility, same issue. Then went down the rabbit hole that is Cisco vs OEM firmware and tried a few thousand things. The HUU for the S3260 does NOT contain firmware for the P4510 because it was never offered on this platform, only the C480M5 HUU (and a few others) have the firmware needed to get these drives to show up correctly to CIMC. So, lets try a few things:
a. Tried to modify the HUU ISO to include the correct firmware and edit the Catalog.json file to detect this drive. No dice, using 6 different ISO editing tools resulted in either no boot at all, partial boot or (best so far) it boots all the way up then throws an error after the HUU is loaded that the image cant be mounted. Two likely issues here, the ISO is built using some tool im unfamiliar with and re-writing it is breaking something. The HUU is checking file consistency somehow (more on that later).
b. Instead of editing the ISO with a utility, I hex edit the C480M5 HUU ISO directly and change the machine string to match my S3260M4. Still fails at the intial boot that the HUU is only valid for X when Y detected.
c. Dig further into the ISO and find out that the Catalog.json file is a red herring and isn't actually used. That file does nothing and the real Catalog.json is compressed inside initrd0.gz, I am unable to edit that one because anytime I re-write the ISO i run into corruption issues with HUU when it tries to launch.
d. OK, fine, ill extract the firmware files directly. Extract the Intel firmware from the ISO, but unable to find a utility that can write the NVMe firmware directly. Only the Intel utility but it comes with the firmware baked-in to a DLL file thats signed. Great, I don't know of any utility that can write NVMe firmware outside of the vendor. Decoding how the HUU does it, it uses a utility that comes with it but I can't find a reliable way to launch it.
e. I have a bunch of non-Cisco 10TB HE10 drives. HUH721010AL4204. These are the same as the Cisco HUH721010AL42C0. Tried to use several tools to write the cisco firmware to them, ended up with 1 paperweight that I'm trying to recover now...
So far, using non-Cisco 12TB drives has not had any bad experience. I am seeing some "Unexpected Sense" in MegaRAID but the drives are happily churning along copying my data over for a load test. I have one S3260 set up 50/50 with 30 drives per blade. One blade is fully populated with 30x 12TB SAS HUH721212AL4201 drives, and the other blade has been my test bed for tinkering with a handful off oddball SAS drives. Had a bit of a whoopsie-daisy with this setup as Node1 was copying data at ~500MBs, i installed the paperweight drive from the above comment into the SECOND blade mapped port, so not mapped to the first node at all that was doing the copy, should only impact the second node right? Wrong, backplane had some kind of hiccup or brain freeze that knocked about half the 12TB drives offline on the other server and threw a huge fit. Rebuilt the array, started over. Removed said dead drive from the second node, again, first node has storage explosion. Long story short, these servers are bit sensitive to faulted drives, working on one can absolutely impact the other!!
To answer your questions, the fan control does exactly what it wants, you can tell it to chill out but it will ignore your preference if it detects any anomaly with the devices. E.g. an NVMe reporting incorrect temperature, a HDD not reporting temperature at all will absolutely cause the fans to ramp up and there is no override for the override.
@jtaj it looks like your node(s) are not being detected properly by the chassis so it cant make changes to the SAS expander mapping. Boot both blades up, make sure they are ALL on the same firmware, refresh the CIMC and try again. You can run into odd behavior if you have one blade powered off and make changes to it "offline". I just booted them both up, hit F2 to go into BIOS and then went into CIMC to tinker. At that point all the devices will be initialized and CIMC can talk to the SIOCs/SAS expanders/RAID cards to do any changes needed.
1. Installed a P4510 NVMe in each chassis, fans went to 100% and got a critical fault on every blade that the NVMe was overheating (it wasnt). Updated the firmware with Intels utility, same issue. Then went down the rabbit hole that is Cisco vs OEM firmware and tried a few thousand things. The HUU for the S3260 does NOT contain firmware for the P4510 because it was never offered on this platform, only the C480M5 HUU (and a few others) have the firmware needed to get these drives to show up correctly to CIMC. So, lets try a few things:
a. Tried to modify the HUU ISO to include the correct firmware and edit the Catalog.json file to detect this drive. No dice, using 6 different ISO editing tools resulted in either no boot at all, partial boot or (best so far) it boots all the way up then throws an error after the HUU is loaded that the image cant be mounted. Two likely issues here, the ISO is built using some tool im unfamiliar with and re-writing it is breaking something. The HUU is checking file consistency somehow (more on that later).
b. Instead of editing the ISO with a utility, I hex edit the C480M5 HUU ISO directly and change the machine string to match my S3260M4. Still fails at the intial boot that the HUU is only valid for X when Y detected.
c. Dig further into the ISO and find out that the Catalog.json file is a red herring and isn't actually used. That file does nothing and the real Catalog.json is compressed inside initrd0.gz, I am unable to edit that one because anytime I re-write the ISO i run into corruption issues with HUU when it tries to launch.
d. OK, fine, ill extract the firmware files directly. Extract the Intel firmware from the ISO, but unable to find a utility that can write the NVMe firmware directly. Only the Intel utility but it comes with the firmware baked-in to a DLL file thats signed. Great, I don't know of any utility that can write NVMe firmware outside of the vendor. Decoding how the HUU does it, it uses a utility that comes with it but I can't find a reliable way to launch it.
e. I have a bunch of non-Cisco 10TB HE10 drives. HUH721010AL4204. These are the same as the Cisco HUH721010AL42C0. Tried to use several tools to write the cisco firmware to them, ended up with 1 paperweight that I'm trying to recover now...
So far, using non-Cisco 12TB drives has not had any bad experience. I am seeing some "Unexpected Sense" in MegaRAID but the drives are happily churning along copying my data over for a load test. I have one S3260 set up 50/50 with 30 drives per blade. One blade is fully populated with 30x 12TB SAS HUH721212AL4201 drives, and the other blade has been my test bed for tinkering with a handful off oddball SAS drives. Had a bit of a whoopsie-daisy with this setup as Node1 was copying data at ~500MBs, i installed the paperweight drive from the above comment into the SECOND blade mapped port, so not mapped to the first node at all that was doing the copy, should only impact the second node right? Wrong, backplane had some kind of hiccup or brain freeze that knocked about half the 12TB drives offline on the other server and threw a huge fit. Rebuilt the array, started over. Removed said dead drive from the second node, again, first node has storage explosion. Long story short, these servers are bit sensitive to faulted drives, working on one can absolutely impact the other!!
To answer your questions, the fan control does exactly what it wants, you can tell it to chill out but it will ignore your preference if it detects any anomaly with the devices. E.g. an NVMe reporting incorrect temperature, a HDD not reporting temperature at all will absolutely cause the fans to ramp up and there is no override for the override.
@jtaj it looks like your node(s) are not being detected properly by the chassis so it cant make changes to the SAS expander mapping. Boot both blades up, make sure they are ALL on the same firmware, refresh the CIMC and try again. You can run into odd behavior if you have one blade powered off and make changes to it "offline". I just booted them both up, hit F2 to go into BIOS and then went into CIMC to tinker. At that point all the devices will be initialized and CIMC can talk to the SIOCs/SAS expanders/RAID cards to do any changes needed.
@Slothstronaut those are some pretty high level stuff you're willing to try to go that far. normally i'd just try for a supported NVMe drive or give up. I managed to solve the issue with assigning under storage section so thats working now but still unable to find the fan control.
I dont have anything that janks it way up to 100% but where did you find the fancontrol? I looked around in the CIMC anywhere and can't find them, could you share with me a screenshot?
my final step would be to get internet working on this thing, does this really require the cisco cable? even so my routher/switch dont even have 10g port.. one RJ45 is management the other RJ45 says console but no idea what it does.
I dont have anything that janks it way up to 100% but where did you find the fancontrol? I looked around in the CIMC anywhere and can't find them, could you share with me a screenshot?
my final step would be to get internet working on this thing, does this really require the cisco cable? even so my routher/switch dont even have 10g port.. one RJ45 is management the other RJ45 says console but no idea what it does.