Sanity Check: Advice on Next Server Steps
- Thread starter ere109
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
A question: doesn't this NVME use Oculink? I know the Oculink ports are capable of 4 SATA drives per - so are you losing two drives potential? Make up for it in speed?
One Oculink has 4 lanes. Assuming the host/HBA supports such, that can be 4x SAS/SATA or one PCIe x4 SSD. In this case, the drive cage has 2x Oculink inputs, so you'd need 2 x4 ports (or one x8 port) fully dedicated to it. Nothing would be wasted.
You do run into this issue with the SAS/SATA 2x drive cages, since a typical SAS HBA/expander port would have 4 lanes.
Okay. So this is what I have that works:
After I set the x4x4 bifurcation in the BIOS settings, it wouldn't boot. It just looped at the first "DXE phase start" screen. I popped out the card and just set JP1,2,3 to all be in the 2-3 position instead of the default 1-2 position. It started right up then.
- MCP-220-82619-0N NVMe kit (fits 826 chassises)
- MCP-220-83608-0N SATA/SAS kit (fits 836 chassises)
- AOC-SLG3-2E4T Retimer card
- 2 OcuLink cables
After I set the x4x4 bifurcation in the BIOS settings, it wouldn't boot. It just looped at the first "DXE phase start" screen. I popped out the card and just set JP1,2,3 to all be in the 2-3 position instead of the default 1-2 position. It started right up then.
I have had a hell of a time getting the MCP-220-82619-0N NVMe kit (RE 81) NVMe kit working as a replacement for the 2 x SATA drives in my SSG-6049P-E1CR60L+ server, so I thought I would write down what I have and have not gotten working (mainly for my future self).
I also have a MCP-220-82617-0N (ID A0, Rev 3) NVMe kit which I have tried but *not* managed to get that one working under any configuration. The 82617 part seems to flash all of the blue, red and green LEDs on power on while the 82619 part only flashes blue and red (does it have green LEDs?). The other ways to tell the backplanes apart are;
The SSG-6049P-E1CR60L+ server has 4 x NVMe OCulink ports on the motherboard (all connected to PE1 port of CPU1. These are normally connected to the optional NVMe cage (MCP-220-94607-0N - SC946S NVMe kits w/ cage, tray, cable, BPN OCuLink v.91,INT,PCIe NVMe SSD, 55CM,34AWG) via Oculink cables. The NVMe cable however has 6 slots. The final 2 x NVMe slots are normally provided by a AOC-SLG3-2E4R-O, which is a PCIe card with retimers. This card is placed in the 8x PCIe slot (driven by PE1 on CPU2) and connected to the NVMe cage via Mini-SAS HD (SFF 8643) connectors to OCulink cables.
My original plan was as follows;
For the PCIe switch card, I ended up using a “DiLiVing 8 Port MiniSAS HD to PCI Express x16 SFF-8639 NVMe SSD Adapter”, part number LRNV9349-8I (LRNV9349-8I), which uses SFF-8643 (Mini SAS HD) connectors and the PLX8749 IC (which is a 48-Lane, 18-Port PCI Express Gen 3 (8 GT/s) Switch).
The first challenge was getting cables that would work between the NVMe cage and the MiniSAS HD card. It turns out that there are many different types of cables which have MiniSAS HD on one end and Oculink cables on the other. This mess seems to be documented in the SFF-9402 standard at https://members.snia.org/document/dl/27380. One of the primary issues is that the cables are unidirectional - a MiniSAS HD host to Oculink backplane cable can not be used to connect an Oculink host to MiniSAS HD backplane cable. This makes it super easy to end up with the wrong cable type even after checking the seller's description carefully.
Given that I have only ended up using 4 of the 8 connectors on the PCIe card I probably should have gone with one of the two options below as they use OCulink connectors and it would have simplified the cabling issue (and probably been cheaper).
I was able to get the PCIe card<->MCP-220-82619-0N NVMe kit working. It was super useful to be able to demonstrate that the PCIe card<->NVMe Cage<->U.2 NVMe SSD pathway was working before moving the same set of cables & U.2 NVMe SSDs to the MCP NVMe kit. This is what eventually led me to discover that the NVMe SSD devices were not fully mating with the U.2 connectors on the backplane. I’m still unsure what is causing this but it might be because of slightly different mechanical dimensions between the various “orange” NVMe caddies.
I have not yet been able to get the motherboard NVMe Oculink connectors working with the MCP-220-82619-0N NVMe kit. I have tried;
The last thing I’m going to try is using the PCIe 4.0 Oculink cables from MicroSATA cables (PCIe Gen4 16GT/s Oculink (SFF-8611) to Oculink 4 Lane Cable) which apparently have an inbuilt PCIe redriver.
If that doesn’t work, I’m probably going to just end up using using the following configuration which seems to work but has a lot longer cable runs;
One thing I’ve yet to understand is what the “NVMe connected to CPU1” and “NVMe connected to CPU2” configuration options actually do. Why does what CPU the NVMe devices are connected to matter? I assume it has something to do with the sideband / I2C / hotplug type operations? Another potential option is something around the PCIe reference clock? Anyone know?
I did find this page of documentation at https://static.nix.ru/images/Supermicro-MCP-220-82619-0N-4938022247.jpg which does warn about issues with NVMe compatibility...
Supermicro support shared the following information with me regarding the jumpers on the NVMe cage backplane which might also be useful to some people in the future;
And the following pictures around the MCP-220-82619-0N NVMe kit,
Hope this information helps someone in the future!
Tim 'mithro' Ansell
I also have a MCP-220-82617-0N (ID A0, Rev 3) NVMe kit which I have tried but *not* managed to get that one working under any configuration. The 82617 part seems to flash all of the blue, red and green LEDs on power on while the 82619 part only flashes blue and red (does it have green LEDs?). The other ways to tell the backplanes apart are;
- The white sticker on the Lattice CPLD (which I assume does sideband and LED signalling) says “ID A0 Rev 3” for the 82617 (the 82619 says “RE 81”).
- The backplane rev & part number on the bottom near J4 is directly on the PCB with Rev on the top and BPN on the bottom for the 82617 (the 82619 had a white sticker with BPN on the top and Rev on the bottom).
The SSG-6049P-E1CR60L+ server has 4 x NVMe OCulink ports on the motherboard (all connected to PE1 port of CPU1. These are normally connected to the optional NVMe cage (MCP-220-94607-0N - SC946S NVMe kits w/ cage, tray, cable, BPN OCuLink v.91,INT,PCIe NVMe SSD, 55CM,34AWG) via Oculink cables. The NVMe cable however has 6 slots. The final 2 x NVMe slots are normally provided by a AOC-SLG3-2E4R-O, which is a PCIe card with retimers. This card is placed in the 8x PCIe slot (driven by PE1 on CPU2) and connected to the NVMe cage via Mini-SAS HD (SFF 8643) connectors to OCulink cables.
My original plan was as follows;
- Replace the 2 x 2.5 inch SATA drives in the back panel with 2 x NVMe using the MCP-220-82619-0N NVMe kit connected to 2 x OCulink NVMe ports on the motherboard.
- Use the other 2 x OCulink NVMe ports on the motherboard with the NVMe cage.
- Use a PCIe card with a PCIe switch to populate the remaining 4 x NVMe ports on the NVMe cage.
For the PCIe switch card, I ended up using a “DiLiVing 8 Port MiniSAS HD to PCI Express x16 SFF-8639 NVMe SSD Adapter”, part number LRNV9349-8I (LRNV9349-8I), which uses SFF-8643 (Mini SAS HD) connectors and the PLX8749 IC (which is a 48-Lane, 18-Port PCI Express Gen 3 (8 GT/s) Switch).
The first challenge was getting cables that would work between the NVMe cage and the MiniSAS HD card. It turns out that there are many different types of cables which have MiniSAS HD on one end and Oculink cables on the other. This mess seems to be documented in the SFF-9402 standard at https://members.snia.org/document/dl/27380. One of the primary issues is that the cables are unidirectional - a MiniSAS HD host to Oculink backplane cable can not be used to connect an Oculink host to MiniSAS HD backplane cable. This makes it super easy to end up with the wrong cable type even after checking the seller's description carefully.
Given that I have only ended up using 4 of the 8 connectors on the PCIe card I probably should have gone with one of the two options below as they use OCulink connectors and it would have simplified the cabling issue (and probably been cheaper).
- LRNV9224-4I (LRNV9224-4I) which has 4 x OCulink connectors (rather than the SFF-8643 connectors) and uses the smaller PLX 8724 IC (which is 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch).
- Supermicro AOC-SLG3-4E2P (AOC-SLG3-4E2P | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.) which also has 4 x OCulink connectors and PEX 9733 IC (which is 32-Lane, 18-Port PCI Express Gen 3 (8 GT/s) Switch).
I was able to get the PCIe card<->MCP-220-82619-0N NVMe kit working. It was super useful to be able to demonstrate that the PCIe card<->NVMe Cage<->U.2 NVMe SSD pathway was working before moving the same set of cables & U.2 NVMe SSDs to the MCP NVMe kit. This is what eventually led me to discover that the NVMe SSD devices were not fully mating with the U.2 connectors on the backplane. I’m still unsure what is causing this but it might be because of slightly different mechanical dimensions between the various “orange” NVMe caddies.
I have not yet been able to get the motherboard NVMe Oculink connectors working with the MCP-220-82619-0N NVMe kit. I have tried;
- Using the first two NVMe ports and the second two NVMe ports. All these ports work correctly with the same cables and U.2 NVMe SSDs in the NVMe cage.
- Using different vendor brands and types of U.2 NVMe SSDs.
- Dropping the PCIe speed back to Gen1/Gen2 rather than Gen3 in the BIOS.
- I've tried the jumpers in positions 1-2 and 2-3.
The last thing I’m going to try is using the PCIe 4.0 Oculink cables from MicroSATA cables (PCIe Gen4 16GT/s Oculink (SFF-8611) to Oculink 4 Lane Cable) which apparently have an inbuilt PCIe redriver.
If that doesn’t work, I’m probably going to just end up using using the following configuration which seems to work but has a lot longer cable runs;
- 4 x NVMe Oculink from motherboard to the NVMe cage.
- 2 x NVMe MiniSAS HD from PCIe card to the NVMe cage.
- 2 x NVMe MiniSAS HD from PCIe card to the MCP NVMe slots
One thing I’ve yet to understand is what the “NVMe connected to CPU1” and “NVMe connected to CPU2” configuration options actually do. Why does what CPU the NVMe devices are connected to matter? I assume it has something to do with the sideband / I2C / hotplug type operations? Another potential option is something around the PCIe reference clock? Anyone know?
I did find this page of documentation at https://static.nix.ru/images/Supermicro-MCP-220-82619-0N-4938022247.jpg which does warn about issues with NVMe compatibility...
Supermicro support shared the following information with me regarding the jumpers on the NVMe cage backplane which might also be useful to some people in the future;
And the following pictures around the MCP-220-82619-0N NVMe kit,
Hope this information helps someone in the future!
Tim 'mithro' Ansell
Attachments
Wish I had better news for you. This is all based on my own inferences - it might be wrong. If anyone knows better, please correct me.
You can connect an NVMe device through one of four main ways:
1. Via a normal PCIe slot (possibly via a redriver or retimer)
2. Using an onboard NVMe header
3. Using a switch card
4. Using an HBA that exposes the drive as something else entirely (e.g. LSI/Broadcom 9400)
For #1, the problem is that a plain old PCIe slot usually lacks the necessary hotplug support and sideband stuff. So the reason you have that cable that connects from the redriver/retimer card to the JNVI2C header on the motherboard is to hook into the sideband signal for bay management and hotplug.
For #2, it's mostly the same, but the sideband connection is built in.
Now for #3 and #4, the card itself would originate the sideband signals if it supports them. Sort of like how you can manage a backplane with a SAS expander without using sideband signals, because the expander itself is responsible for the management rather than the HBA.
As for the jumper settings on the backplane, I'm not 100% sure what the correct way to do it is and I haven't gotten a clean answer from SMC support. To make the example easy, consider a 4-port backplane.
- If you're using redrivers/retimers/direct ports, then set the jumpers according to how you have the drives hooked up to CPUs. I think if all of them are connected to CPU2, but the backplane doesn't have a jumper combination for that, you just set it to "All drives connected to CPU1".
- If you're using a single 4+-port PCIe switch or HBA to do all four ports, set it to "All drives connected to CPU1".
- If you're using 2x 2-port switches/HBAs, set it to "2 drives on CPU1, 2 drives on CPU2".
It should also be noted that there is a pretty big difference between SAS/SATA backplanes and NVMe backplanes.
Almost all NVMe backplanes are closer to fancy cables than what I would consider a "backplane", they route the high speed signals in a passive manner. This means you need one connection for each device.
Almost all NVMe backplanes are closer to fancy cables than what I would consider a "backplane", they route the high speed signals in a passive manner. This means you need one connection for each device.
Sorry to necropost and my appologies to the original poster whose thread was hijacked.
The SuperMicro rear NVMe bays are a confusing mess. The lack of pictures adds to the problem.
WiredZone sells many of them but no pictures.
The thinking is that NUMA (CPU to CPU interconnects) would provide added latency over QPI links (CPU internal interconnects).
So consider placing all NVMe/controllers on one CPU for minimal latency.
My thanks for the pictures @ramicio
I would like to see a similar pictorial size comparison of the whole lot.
MCP-220-82609-0N
MCP-220-82617-0N
MCP-220-82619-0N
MCP-220-94607-0N
Which cages do these backplane boards works on???
They look older style by the hole pattern.
The SuperMicro rear NVMe bays are a confusing mess. The lack of pictures adds to the problem.
WiredZone sells many of them but no pictures.
I am not sure where you are finding these settings but generally I like to keep all NVMe on one CPU and its assigned slots.(as found in manual)
The thinking is that NUMA (CPU to CPU interconnects) would provide added latency over QPI links (CPU internal interconnects).
So consider placing all NVMe/controllers on one CPU for minimal latency.
My thanks for the pictures @ramicio
I would like to see a similar pictorial size comparison of the whole lot.
MCP-220-82609-0N
MCP-220-82617-0N
MCP-220-82619-0N
MCP-220-94607-0N
Which cages do these backplane boards works on???
They look older style by the hole pattern.
Supermicro BPN-SAS3-F424-A2N2 Backplane SAS / NVMe Connectors | eBay
Manufacturer Supermicro. If this happens, don't worry, we will stand behind our products!
www.ebay.com
That is indeed a different style. I think those are for certain servers like this one (it lists it in the parts list).Sorry to necropost and my appologies to the original poster whose thread was hijacked.
The SuperMicro rear NVMe bays are a confusing mess. The lack of pictures adds to the problem.
WiredZone sells many of them but no pictures.
I am not sure where you are finding these settings but generally I like to keep all NVMe on one CPU and its assigned slots.(as found in manual)
The thinking is that NUMA (CPU to CPU interconnects) would provide added latency over QPI links (CPU internal interconnects).
So consider placing all NVMe/controllers on one CPU for minimal latency.
My thanks for the pictures @ramicio
I would like to see a similar pictorial size comparison of the whole lot.
MCP-220-82609-0N
MCP-220-82617-0N
MCP-220-82619-0N
MCP-220-94607-0N
Which cages do these backplane boards works on???
They look older style by the hole pattern.
Supermicro BPN-SAS3-F424-A2N2 Backplane SAS / NVMe Connectors | eBay
Manufacturer Supermicro. If this happens, don't worry, we will stand behind our products!www.ebay.com
82609 is the unmanaged SATA/SAS BP for 826/847/etc.
82616 is the same but managed.
82617 is allegedly an NVMe option for 826/847, but I have never seen it.
82619 is the typical NVMe option for 826/847.
Not sure about 94607.
It would be very unusual to have one drive on each CPU for a two-drive cage. You'd need a single x4 from one CPU and one from the other, but AFAICT, Supermicro only sells redrivers/retimers/switches with 2+ ports.
You generally don't want to just put them on one CPU - you'd want to make sure that the workload that they'd handle is also on the same CPU. For example, if the purpose of the machine is to be a Ceph OSD node, then you might want the NVMe drives and network interface to be on one CPU (and pin the OSD process to that CPU as well).
If you put the NVMes on CPU0 and the network interface on CPU1, then you'd take the inter-CPU latency hit no matter what.
Also, fwiw: some of the newer backplanes no longer have the CPU selector jumpers. Not sure if they've switched to some other management method entirely, or what the situation is.
Last edited:
Necro all you want. NVME seems to be an unexplained solution waiting for a problem, at least in SuperMicro. I got lost when people started talking about retimers. As of now, I've just left both Oculink ports empty and fully maxed my SATA ports. Extra storage ports or cache drives would be nice, but it doesn't sound like they're ready for the average home tinkerer. Though it's certainly a waste, it'll probably also add a few years to my life not to stress it.