AVOID - 4U, 2x Node, 4x E5 V3/V4, 56x LFF SAS3 3.5" bay - $299 - CISCO UCS C3260

[oneplane]

It looks like the drive in there is set as a single-drive RAID0 so it is all passed through the controller in RAID mode, meaning you cannot do anything in Windows until the special driver is loaded.

I didn't see a controller mode switch in the screenshots, but perhaps those don't show up until the virtualdrive/drivepool are deleted?

 

[eduncan911]

To grab the drivers, Cisco requires you to sign up for an account. Once in your account, you can search for the system model number and download the drivers.

Yeah there had to be to configure HBA mode - it's just we haven't seen it yet in the screenshots.

Please do delete the Virtual RAID0 drive, and check options again to see if a new option pops up to configure HBA mode.

 

[jtaj]

It looks like the drive in there is set as a single-drive RAID0 so it is all passed through the controller in RAID mode, meaning you cannot do anything in Windows until the special driver is loaded.

I didn't see a controller mode switch in the screenshots, but perhaps those don't show up until the virtualdrive/drivepool are deleted?

To grab the drivers, Cisco requires you to sign up for an account. Once in your account, you can search for the system model number and download the drivers.

Yeah there had to be to configure HBA mode - it's just we haven't seen it yet in the screenshots.

Please do delete the Virtual RAID0 drive, and check options again to see if a new option pops up to configure HBA mode.

I dont see the HBA mode specifically but I have option to not use virtual raid and use JBOD, JBOD also doesnt show up either so probably needs a driver.

how do people use the front loading bays without drivers then?

 

[eduncan911]

I dont see the HBA mode specifically but I have option to not use virtual raid and use JBOD, JBOD also doesnt show up either so probably needs a driver.

how do people use the front loading bays without drivers then?

Could you switch it to JBOD and snap more pics? Also, pull the driver and get used to loading the driver during these Windows tests. You'll most likely need the driver regardless.

But also, could you help us as well? Many of us is dying to know: Is the raw disk details presented straight to the OS in JBOD mode? I know 7 that will sell right now, if the answer is yes. Only way to know is to set as JBOD, load the driver in Windows, and see what it shows.

Btw, what are your drive name and model numbers - as well as their installed locations? That would help with some of the screenshots.

 

[oneplane]

I dont see the HBA mode specifically but I have option to not use virtual raid and use JBOD, JBOD also doesnt show up either so probably needs a driver.

how do people use the front loading bays without drivers then?

It is possible that people cannot use windows and the front bays without drivers. On Linux the drivers are built in most of the time so they can work as-is.

Considering it has a "JBOD" mode, maybe that is their terminology for HBA mode. Vendors are highly inconsistent in their choice of words but it should be simple enough to check.

Get the drivers from Cisco's website and then put them on a USB drive, then during Windows setup you can select them in the installer application. Alternatively, it might be wise to simply install Proxmox or ESXi and then run windows as a virtual machine on top of that.

 

[eduncan911]

Take a look at Cisco's manual on this M4 system as well (scroll way down for the storage portion):

Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note

Provides information on LEDs and replacing components within the M4 server node.

www.cisco.com

Something to note that there are two "raid" controllers available for this system:

■ For hardware SAS MegaRAID— Avago Technologies/LSI 12 Gb/s MegaRAID SAS Software User’s Guide, Rev. F
■ For embedded software MegaRAID— LSI Embedded MegaRAID Software User Guide

You either have one, or the other, but not both. The manual states that when you have the hardware MegaRAID controller, there is no Embedded Sotware MegaRAID. That should help you when finding which "driver" to load during Windows setup.

So, there is a manual - which seems quite extensive in configuring the hardware SAS MegaRAID controller:

Avago Technologies/LSI 12 Gb/s MegaRAID SAS Software User’s Guide, Rev. F

There's lots of JBOD information there, and how to configure the drives for boot.

@oneplane: fyi, when poking around that PDF above, I noticed that it states there is an EFI "storage driver" available which allows you to access the drivers from an EFI shell. I think I recall us discussing that earlier, just FYI.

 

[oneplane]

@oneplane: fyi, when poking around that PDF above, I noticed that it states there is an EFI "storage driver" available which allows you to access the drivers from an EFI shell. I think I recall us discussing that earlier, just FYI.

Indeed, that was exactly what we were thinking about. Without that, some RAID controllers cannot boot in UEFI because it wouldn't be able to access the ESP or any other partition for that matter. I wonder if Cisco was smart enough to include the driver for their selected RAID card in their own firmware builds

 

[jtaj]

It is possible that people cannot use windows and the front bays without drivers. On Linux the drivers are built in most of the time so they can work as-is.

Considering it has a "JBOD" mode, maybe that is their terminology for HBA mode. Vendors are highly inconsistent in their choice of words but it should be simple enough to check.

Get the drivers from Cisco's website and then put them on a USB drive, then during Windows setup you can select them in the installer application. Alternatively, it might be wise to simply install Proxmox or ESXi and then run windows as a virtual machine on top of that.

Take a look at Cisco's manual on this M4 system as well (scroll way down for the storage portion):

Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note

Provides information on LEDs and replacing components within the M4 server node.

www.cisco.com

Something to note that there are two "raid" controllers available for this system:

■ For hardware SAS MegaRAID— Avago Technologies/LSI 12 Gb/s MegaRAID SAS Software User’s Guide, Rev. F
■ For embedded software MegaRAID— LSI Embedded MegaRAID Software User Guide

You either have one, or the other, but not both. The manual states that when you have the hardware MegaRAID controller, there is no Embedded Sotware MegaRAID. That should help you when finding which "driver" to load during Windows setup.

So, there is a manual - which seems quite extensive in configuring the hardware SAS MegaRAID controller:

Avago Technologies/LSI 12 Gb/s MegaRAID SAS Software User’s Guide, Rev. F

There's lots of JBOD information there, and how to configure the drives for boot.

@oneplane: fyi, when poking around that PDF above, I noticed that it states there is an EFI "storage driver" available which allows you to access the drivers from an EFI shell. I think I recall us discussing that earlier, just FYI.

@eduncan911 those are some good find. I have attached some more photos mostly in the raid controller settings ctrl + R right after post.
@oneplane so I set everything in that PCI LOS w/e screen to legacy, and then set virtual group (the SSD) to bootable in raid controller, and the PCI 00 raid card adapter boot option appears. so you are most correct that it requires some sort of driver so I can install OS on it.

now, how to obtain said driver is another issue. unless you guys can help me out, I do have an account on cisco, they do allow you to download driver without being an existing customer with warranty. So can simply register, issue is when I look up c3160 or S3260 (not sure which) there are a bunch of them and not very user friendly, and I am no tech junky.

 

[jtaj]

also another discovery, is that I reset CMOs battery on the M3 node, swap out the M4 and put in the M3 instead and viola, installed OS onto SSD and the bays are recognized without needing any driver (but both uses avago mega raid).

so not sure why drivers aren't needed for M3 but required for M4

 

[eduncan911]

so not sure why drivers aren't needed for M3 but required for M4

Because your M3 node has a different Raid controller card, perhaps?

 

[jtaj]

Because your M3 node has a different Raid controller card, perhaps?

they both go through raid controller screen (ctrl + R) and both are avago mega raid. the raid card isn't in the chassis but part of the node right? or is it on the SOIC?

btw, do you know if the full LFF bays will work with a single node or does it require both nodes populated?

 

[Samir]

View attachment 22504

I think you've already got around this issue by using the M3, but in this picture with the virtual drive, see if there's a bootable option under the 'advanced' options.

 

[eduncan911]

they both go through raid controller screen (ctrl + R) and both are avago mega raid. the raid card isn't in the chassis but part of the node right? or is it on the SOIC?

The raid card is part of the node itself, yes. It's attached in the center.

Is it perhaps your M3 node has the JBOD version of the controller card? When you place the M3 and M4 nodes side by side, does the center card look identical? Snap a pic maybe of both nodes' internals?

btw, do you know if the full LFF bays will work with a single node or does it require both nodes populated?

The manual link for the C3260 I posted earlier answers this. There's a dedicated section for node Population rules and how drives are assigned with different setups. Then there's the controller manual I posted, which goes over the assignment details of the dual channel SAS drives (if you have em).

Take a long read of those links we posted earlier because at this point, you're going to need to read the manual.

In short: all LFF drives are assigned to one node if only populating 1 node.

 

[jtaj]

In short: all LFF drives are assigned to one node if only populating 1 node.

I would need some help on that. I spent nearly 3 hours going through page by page, ended up giving up and go for search terms which I think would fall under drive bay population but to no avail. couldn't find anything remote to it, if you could point me to which page approxmiately and I'd start there.

on the bright side, I did install the driver and got the M4 to work. though I still hate the idea that I can't boot off of the U.2 PCIe SSD inside the node, bios simply lacking that option. window sees the drive but can't install OS nor boot from it..

 

[Digital Spaceport]

I would need some help on that. I spent nearly 3 hours going through page by page, ended up giving up and go for search terms which I think would fall under drive bay population but to no avail. couldn't find anything remote to it, if you could point me to which page approxmiately and I'd start there.

on the bright side, I did install the driver and got the M4 to work. though I still hate the idea that I can't boot off of the U.2 PCIe SSD inside the node, bios simply lacking that option. window sees the drive but can't install OS nor boot from it..

Can you give us an idea as to the noise level and also the wattage you are seeing and amount of drives you have populated so far?

 

[jtaj]

Can you give us an idea as to the noise level and also the wattage you are seeing and amount of drives you have populated so far?

no idea of wattage havent had time to measure them. can probably install hwinfo but it would not be fully accurate and I currently dont have enough drives to populate them, will eventually migrate from old servers to it tho.

for noise, on idle its around 57 to 58db at lowest. not sure why the manual says 38 but I am probably missing something, currently testing with 1 SIOC, 1 node and 1 PSU, 1 CPU socket populated only. going to 2 or 4 PSU doesnt change noise level.

some interesting thing is that you can actually remove all the fans or leave 1 of 4 fans running and it'll only give fan LED error but server still runs. possibly a fan mod is doable.

 

[oddball]

I'd ditch the BIOS modification directly. You want to use CIMC to manage the BIOS options and setup. It will overrule whatever is set in the BIOS anyways.

I've installed windows on a LOT of UCS servers with NVMe. As others have noted you need to be in UEFI mode, your error is showing you're in legacy boot mode.

In the UEFI shell list the devices, you should be able to see your NVMe drive. UEFI is similar to a lspci shell, you can probe devices.

In 100% of the cases where I had issues the hardware was detected fine but the software wasn't setup correctly to see things.

CIMC unlocks a lot of other BIOS settings that you won't have access to. You can change boot order, set the boot loader path, change network configs etc. This is how you setup to boot from the bottom SSD's as well.

You can download drivers from the Cisco support portal. They are free without a support contract, you want the c-series stand alone managed servers. They have a driver pack for Windows, Linux, VMWare, Xen, they're about 3GB each. There is also a firmware pack that's a few gigs as well.

 

[oddball]

In terms of noise, if you have 'supported' hardware and the firmware is up to date these are very quiet. I had an NVMe drive with out of date firmware and it caused the fans to ramp.

Look in CIMC at the logs and you'll see why it isn't quiet. Ours are in a DC and around 40db, but with that funky drive were extremely loud.

Regarding "supported" these things will run anything. But for Intel NVMe drives you need certain firmware levels to hit that metric.

If you have CIMC 4.2 you can modify the fan level and set it to low.

 

[NablaSquaredG]

I would kill for a bunch of those in Europe... But sadly shipping them from USA to Germany wouldn't be worth it

 

[eduncan911]

I'd ditch the BIOS modification directly. You want to use CIMC to manage the BIOS options and setup. It will overrule whatever is set in the BIOS anyways.

I've installed windows on a LOT of UCS servers with NVMe. As others have noted you need to be in UEFI mode, your error is showing you're in legacy boot mode.

In the UEFI shell list the devices, you should be able to see your NVMe drive. UEFI is similar to a lspci shell, you can probe devices.

In 100% of the cases where I had issues the hardware was detected fine but the software wasn't setup correctly to see things.

CIMC unlocks a lot of other BIOS settings that you won't have access to. You can change boot order, set the boot loader path, change network configs etc. This is how you setup to boot from the bottom SSD's as well.

I agree with this as well, as I've only seen the full options in LSI controllers via the CIMC.

You can download drivers from the Cisco support portal. They are free without a support contract, you want the c-series stand alone managed servers. They have a driver pack for Windows, Linux, VMWare, Xen, they're about 3GB each. There is also a firmware pack that's a few gigs as well.

Are the Linux drivers really necessary? That's a Yikes, if 3 GB for a Linux driver.

Last question we are all trying to answer:

Does the MegaRAID LSI controller present the true "raw" drive to the OS, when the controller is set to Pass-through mode? No RAID0 Volumes.

This is killing me and a few others from buying these machines and setting them up for ZFS.