AVOID - 4U, 2x Node, 4x E5 V3/V4, 56x LFF SAS3 3.5" bay - $299 - CISCO UCS C3260

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
@oddball or anyone have this server, pls help. we got this server a couple months back and we can't seem to boot from the 2.5 sata SSD. the SSDs gets picked up by the MEGA raid controller and is seen in bios or raid menu, but bios does not show them as bootable option which differs from the old M3.

screen looks like this:

only network, PCI raid adapter and UEFI shell. setting up virtual or jbod via raid controller still wont have those array/drive as boot option in bios. what seems to be the issue?

View attachment 22412

View attachment 22413
Just curious... Did you install an OS yet? As that would pickup the old legacy boot sector on the blue menu.

Also, have you tried UEFI? Perhaps it only boots EFI kernels from a boot partition. Some HPs I had was like that.

And just checking, the SSDs are in the rear slots, at the back of the machine, correct?
 
  • Like
Reactions: Samir

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
@oddball we got this server a couple months back and we can't seem to boot from the 2.5 sata SSD. the SSDs gets picked up by the MEGA raid controller and is seen in bios or raid menu, but bios does not show them as bootable option which differs from the old M3.
In the manual, it says:

NOTE: 2.5-inch form-factor NVMe SSDs are bootable in UEFI mode; legacy booting is not supported
That's for NVMe drives and you got sata. But maybe it's the same restriction?
 
  • Like
Reactions: Samir

jtaj

Member
Jul 13, 2021
74
38
18
Just curious... Did you install an OS yet? As that would pickup the old legacy boot sector on the blue menu.

Also, have you tried UEFI? Perhaps it only boots EFI kernels from a boot partition. Some HPs I had was like that.

And just checking, the SSDs are in the rear slots, at the back of the machine, correct?
well we can't even get the SSD to show as a bootable in bios, so we can't install OS because no drive shows up when booting from USB, windows server (with 1 exception, answered below)


In the manual, it says:

That's for NVMe drives and you got sata. But maybe it's the same restriction?
we got both sata and nvme u.2 ssd installed in their respective slots.

the only NVMe SSD that does show up when installing window is the one we installed in the node itself (not the sata 2.5 near the bottom) but we also can't install on it because window says it will not allow installing on not recognized bootable device.

so basically SSD can be seen once booted with window install USB drive but its still not seen as a bootable device in bios because the boot menu is what Ive shown, NVMe SSD isnt one of them.

I have tried to turn into legacy and try restart to check boot menu, but NVMe boots only from UEFI
 
Last edited:
  • Like
Reactions: Samir

oddball

Active Member
May 18, 2018
206
121
43
42
Can you see the drives in the UEFI shell? There are a ton of commands to do from the shell. You can navigate the drives and look for a boot loader.

We put the drives in, then booted from a vKVM image and installed Windows Server. Windows put a boot loader on the drive we selected and it's booted from that by default since.
 
  • Like
Reactions: Samir

jtaj

Member
Jul 13, 2021
74
38
18
Can you see the drives in the UEFI shell? There are a ton of commands to do from the shell. You can navigate the drives and look for a boot loader.

We put the drives in, then booted from a vKVM image and installed Windows Server. Windows put a boot loader on the drive we selected and it's booted from that by default since.
hey thanks for replying.

I can't see the drive in bios and I do not know how to use UEFI shell, simply wanted to install via USB flash drive. in the M3 seems to be the opposite, it shows SATA options and can be installed via USB but no mega raid even with the same chassis and SIOC
 
  • Like
Reactions: Samir

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
hey thanks for replying.

I can't see the drive in bios and I do not know how to use UEFI shell, simply wanted to install via USB flash drive. in the M3 seems to be the opposite, it shows SATA options and can be installed via USB but no mega raid even with the same chassis and SIOC
It seems you are mixing up some terms here; to know if a disk is usable or not it doesn't really matter what a boot menu thinks about it, that stuff can be modified as you see fit anyway.

There are two ways a disk attached to a controller can be used:

1. As a raw disk where the controller doesn't interfere with the communication between the host OS and the disk, effectively meaning the controller plays the role of a Host Bus Adaptor (HBA)

2. As a volume managed by the controller, which in some vendor terminology means different things. This is not a directly accessible disk and all you'll see in the firmware (be it legacy BIOS or UEFI) is the controller presenting itself with volume options

There are two ways your current machine can boot:

1. Legacy BIOS where it just runs a random bit of code on the first few bytes of a disk, hoping that someone put a boot loader in there. (the boot loader or boot manager then uses either a loader specific configuration or looks for the 'active' partition to boot from - some firmwares used to look for the 'active' partition by itself)

2. UEFI boot manager where a persistent list of paths are configured in the UEFI interface that point to specific locations on specific devices.

Before you do either of those you'd need to know what the controller is doing to your disk, and you'd need to know if your OS can actually talk to the controller at all. In some cases the controller isn't usable at all from BIOS of UEFI land, and can only be used inside a full OS or post-bootloader.

If you are doing something like trying to install a desktop OS like windows server, you are likely going to need a driver disk during setup before it can see anything. If you are using Linux, it's possible that the RAID controller needs special drivers too if it doesn't use a standard SATA or SCSI command set to interact with it. Lastly there is the vendor-specific disk configuration, perhaps your controller calls an unused disk a RAW disk, which isn't the same as passing the disk through to the OS in raw form. Sometimes it's pretending JBOD isn't JBOD but a single disk RAID0 (or the other way around). Sometimes it has a real HBA mode. The problem is not really which one it is, but mainly selecting the right option.

In the rare case where the vendor thought it would be fun to not include hardware drivers in UEFI for that RAID card's storage, you can sometimes only see the OptionROM for it over the bus, but not the disks (be it physical or virtual LUNs) at which point you can launch the config but never boot from it because the UEFI can never read the storage directly. This is generally not really what happens but it's still something I come across from time to time on onder and 'pretend-proprietary' setups (like whitebox HP and Dell based systems).

So before continuing, try to get a picture of the way the controller is currently configured.
 
  • Like
Reactions: Samir and jtaj

jtaj

Member
Jul 13, 2021
74
38
18
If you are doing something like trying to install a desktop OS like windows server, you are likely going to need a driver disk during setup before it can see anything.
those are a lot of good information, I am not as technical so it is difficult for me to understand some of them, I think I know what you're trying to say and theres probably a lot that I am not understanding.

question on the quoted part, if the 2.5" SATA SSDs near the bottom of chassis are going through controller/raid then it'd make sense why I need driver before I can see them to install window server on it, but this doesn't explain why I can see the NVMe SSD that I installed in the node. NVMe SSD does show up under windows installation, it just won't let me install due to window says its not a bootable device, which it isn't seen as a bootable option under BIOS.

I have had many servers and desktop, if they support NVMe boot via UEFI they will always show as a boot option in bios, but I do not have that option, do you know why?

furthermore, shouldn't SATA be native and be able to boot off of it? I dont understand why cisco would have it route through raid controller. and even in BIOS I create virtual array via raid controller option, set them to bootable under raid card "operation" and still cant see them as a boot option so windows install still can't see it.

on M3 the SATA seems native and I could install it just fine.

is there a way to make the 2.5" slots not go through controller?
 
Last edited:
  • Like
Reactions: Samir

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
I can see the NVMe SSD that I installed in the node
With your eyes, or are we talking about a program, operating system, firmware or controller option ROM here?

NVMe SSD does show up under windows installation, it just won't let me install due to window says its not a bootable device, which it isn't seen as a bootable option under BIOS.
Those are two very different things. Windows just pulls some text out of its ass with no real meaning as to what the actual issue is. The bootable options in the EFI Boot Manager (not the BIOS - You don't have a BIOS) are either set by the OS, added by the user, or generated for the CSM module to emulate Legacy BIOS Booting within UEFI.

Windows usually complains about this if the partition table format doesn't match the boot method. It also doesn't allow bare NVMe booting on Legacy BIOS boot modes. It also can't boot from a drive that has no room for a partition but needs an ESP or windows boot partition. Which one of those it is is really hard to determine. It's also hard to see what Windows thinks it sees without more information.


I have had many servers and desktop, if they support NVMe boot via UEFI they will always show as a boot option in bios, but I do not have that option, do you know why?
I have no idea. No two UEFI systems are the same. The boot options in the UEFI boot manager (not BIOS, it's not a BIOS) can be set to any arbitrary data. You can manually create an entry called "Banana" and have to boot off of "fs0:/the_third_moon_of_mars.efi" and that would be a totally valid boot entry (but it's unlikely to actually find the EFI file it points to).

If you want to boot from NVMe storage, make sure of the following:

- GPT partition scheme
- UEFI boot, no legacy boot; disable CSM to be sure
- UEFI-capable bootloader
- UEFI-capable OS

furthermore, shouldn't SATA be native and be able to boot off of it? I dont understand why cisco would have it route through raid controller. and even in BIOS I create virtual array via raid controller option, set them to bootable and still cant see them as boot option so windows install still can't see it.
There is no reason a specific bus is or isn't available for booting. Any device can be configured to ignore/disable buses at boot. Then again, if the devices can be seen in the EFI Shell they can also be configured in the EFI boot manager. But if they are connected to the RAID Controller, and that controller cannot present them as real disks or fake disks (as a logical volume) then UEFI can't do anything with them because it can't see them.

Windows probably can't see it because even in its current iteration it needs special help from a driver disk to be able to see most hardware raid configurations. This is fine, but you have to prepare the this and load the driver before Windows can see it.

on M3 the SATA seems native and I could install it just fine.
It's a different device so I suppose that's out of scope here.

is there a way to make the 2.5" slots not go through controller?
I don't think so, unless the backplane can be disconnected and connected to a different bus. Maybe if there is some chipset bus or a plug-in HBA you can attach it to, but I think it might make more sense to first configure the system the right way and then create a UEFI install disk.
 
  • Like
Reactions: Samir

jtaj

Member
Jul 13, 2021
74
38
18
If you want to boot from NVMe storage, make sure of the following:

- GPT partition scheme
- UEFI boot, no legacy boot; disable CSM to be sure
- UEFI-capable bootloader
- UEFI-capable OS
I am bad when comes to UEFI boot as im stuck with older generation computing, I insert disk make sure they can be seen in bios as bootable device and off I go, booting off of a USB drive and install on disk, then boot off of disk to run ubuntu or windows. I should have tried to learn more about EFI back in the days.

- on the quoted part, at first NVMe was GPT and window installer would not let me install saying it needs to be MBR. I change it back to MBR and this time around it gives error message that it isn't seen as a bootable device hence windows can't install on it. what is wrong here?

- how and where in bios can I disable CSM? also how do I turn it back on if I need it down the road?

- UEFI-capable bootloader, no idea what I needed here. is this on the USB flash drive? cause it can clearly boot off my USB stick

- server 2012r2 is UEFI capable I think so I have got that

included some screenshots:

d.jpga.jpgb.jpgc.jpge.jpg
 
Last edited:
  • Like
Reactions: Samir

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
Now, switch to UEFI. If it has SecureBoot, turn that on. Some BIOSes comes with a "Certified Windows" or "Windows Secure Boot" or something related to Windows that forces CSM to No.

In addition, I echo 100% what @oneplane said. You must know how the raid controller is configured, and in what mode it is in (PassThrough or RAID) and how you've configured any groups (or maybe the previous owner still has old groups and failover configs that haven't been erased). Without this info, there's not much to point to.

Also, on that Windows install screen that says it won't let you install due to non-bootable controller ... That is a screen to load the driver as well (besides the F6 option).

Are you loading the driver at all? The install guide for the C3260 had a step by step procedure for how to install Windows Server using the driver disk. It was nothing we all haven't done dozens of times for servers 10 to 20 years ago (so much easier than 3.5" floppies or CDR burns of ISO driver disks from back in the 90s). Windows doesn't include certified drivers for all RAID cards. You still need to load drivers from time to time.
 
  • Like
Reactions: jtaj

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
While the information definitely clears some things up, there are a few unknown items:

- Is the USB disk booted in UEFI or in legacy mode? I recall PMAP devices being legacy boot options?
- Is CSM/Legacy boot still on?
- Is "UEFI Boot Options" an item you can access in the configuration setup?
- SBNVMe seems to be the NVMe drive (the one not attached to the controller?) that cannot be used, apparently the UEFI you have doesn't know how to deal with it, or it's in legacy mode which needs to be disabled first.

The RAID controller seems to talk about 1 JBOD and 1 Virtual Device. That virtual device is the "PCI RAID Adapter" you see in your boot menu. I'm not 100% sure if this firmware groups JBOD in with the virtual device map, but it sure smells like it. On the other hand a Samsung and Micron device are detected so it's possible that one is JBOD and the other Virtual, but since neither are actually usable I think the entire RAID config is broken.

I recommend enabling Secure Boot as well since this forces CSM off in almost all firmwares which will help a lot. If your USB disk suddenly doesn't get detected anymore that means it was created in legacy-only mode and not dual or UEFI mode.
 

jtaj

Member
Jul 13, 2021
74
38
18
Are you loading the driver at all? The install guide for the C3260 had a step by step procedure for how to install Windows Server using the driver disk
yea I prob should, M3 was native and sees drive I just assumed M4 doesnt need them. do you know where I can find the driver for raid card on M3 and M4?

In addition, I echo 100% what @oneplane said. You must know how the raid controller is configured, and in what mode it is in (PassThrough or RAID) and how you've configured any groups (or maybe the previous owner still has old groups and failover configs that haven't been erased). Without this info, there's not much to point to.
thats something I have to check into, I did however set it to default config or clear config but maybe thats not enough

- Is the USB disk booted in UEFI or in legacy mode? I recall PMAP devices being legacy boot options?
- Is CSM/Legacy boot still on?
- Is "UEFI Boot Options" an item you can access in the configuration setup?
- SBNVMe seems to be the NVMe drive (the one not attached to the controller?) that cannot be used, apparently the UEFI you have doesn't know how to deal with it, or it's in legacy mode which needs to be disabled first.
- the USB has installed UEFI machines so it definitely is UEFI capable

- I have no idea, the bios tab Main, Security and Server management dont have much. but what you said about secure boot forces disable CSM I'll check and give it a try if theres such an option.

- in UEFI boot options when I click enter, there is only UEFI shell and thats it.

- theres a bunch of option under that section, let me screenshot few more.


I recommend enabling Secure Boot as well since this forces CSM off in almost all firmwares which will help a lot. If your USB disk suddenly doesn't get detected anymore that means it was created in legacy-only mode and not dual or UEFI mode.
oh at one point, USB did disappear after I switch some settings so maybe it is not dual.
 
  • Like
Reactions: Samir

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
Could you add a screenshot of the RAID configuration utility? (bottom option in your first screen)
 
  • Like
Reactions: Samir

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
Uh oh... I see a lot of RAID0 under this Virtual Drive setup. That does not spell good for ZFS.

Is anyone else very surprised to see the Megaraid configuration - in the BIOS?! Usually you wait during the POSTing of the bios to press CTRL-K or R or alike to get into the Rail card bios. I believe I even recall the setup docs talking about the normal raid utility.

@jtaj I see now what you meant about resetting the controller to defaults, via that option in the screenshots.

Are you sure you don't have a boot menu that shows up for the rail controller cars?
 
  • Like
Reactions: Samir

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
The first thing to look for is, what mode is the controller in. HBA or RAID.

That's the main thing missing here that will determine what driver to load, as well as how to configure the drives. If in raid, it may be forcing that Virtual Device RAID0 (which is not what we want, if not using raid). If in HBA, then less to do.
 
  • Like
Reactions: Samir and jtaj

jtaj

Member
Jul 13, 2021
74
38
18
I would try turning the option on for 'controller bios' and see if that makes the virtual drive show up.
it doesnt, by default it is enabled and I disabled it to see whta it does but doesnt seem to change bios

Are you sure you don't have a boot menu that shows up for the rail controller cars
by putting the PCIe menu portion back to normal "enabled" rather than "UEFI only" for the PCIe slot for raid card which pictured below then it will show another boot option but I can't install on it. the option also pictured below which says "Dev 00 PCI raid adapter", but I have no idea how to install windows into it.

you mentioned driver before, where can I get the driver for the raid card I can try install them to see if window can pick up the volumes created via raid card. (theres virtual raid, or jbod which I think does have passthrough and passthrough somehow still requires driver because DL380 gen10 is like that, needs driver to see them even on HBA).

20220411_221010_1.jpg
c.jpg


That's the main thing missing here that will determine what driver to load, as well as how to configure the drives. If in raid, it may be forcing that Virtual Device RAID0 (which is not what we want, if not using raid). If in HBA, then less to do.
as for driver it probably wouldnt matter if I have all of them, but right now I have zero drivers donno where to get them. if I have a bunch I can simply try them all till find one that works.
 
  • Like
Reactions: Samir