How to run PCIe-based storage devices with only one lane?

[Partofthesolution]

What I would like to do is unconventional and a idea to reduce cost for hardware, space and power consumption.

Typically, a M.2 or U.2 drive uses four PCIe lanes. I would like to operate M.2 or U.2 drives with only one or two lanes per drive instead of four, so I can attache 16 NVME drives to a PCIe x16 slot or to a HBA with 16 channels. I want to do this, because PCIe 4.0 supports 2GB/s per lane and PCIe5.0 4GB/s per lane - netto. Thats enough bandwith for a lot of homeserving use cases and much faster than SATA 6G (netto 600MB/s) and SAS 12G (netto 1,2GB/s) is per channel.

Most (all?) M.2 storage devices with NVME supports x1 connectivity as fallback mode. Some also support x2 operation. There are also some single-board computer and some cheap boards with x1 or x2 M.2 connecters. I think it should not be a problem for most drives to run in x1 or x2 mode.

At the moment, I see two possible solution to reach this goal: CPU or chipset based bifurcation or with a HBA.

First idea is bifurcation:

Bifurcation would be a nice way to reach this goal, because additional active hardware/HBA-chipset is not needed. But I see these problems:

1. A mainboard which can bifurcate a PCIe slot to x1x1x1x1x1x1x1x1x1x1x1x1x1x1x1x1 or x2x2x2x2x2x2x2x2 is needed.

2. Additional adaption (riser, breakoutcables, wireing harness, adapters) to lead out every single or pairs of PCIe-lanes to a single M.2 or U.2 connectors is needed.

I think modern CPUs or chipsets can do this, but have you ever seen a mainboard UEFI which allows you to do this? I think this problem can only be solved by the mainboardvendor or someone who can modify a UEFI/BIOS.

The second problem can be solved with available hardware. What you need is:

1x https://de.aliexpress.com/item/1005005237599668.html
4x https://de.aliexpress.com/item/1005004274364890.html
16x https://de.aliexpress.com/item/1005003610538234.html
16x https://de.aliexpress.com/item/4001030510953.html

Please note, that this is not USB protocol. Its PCIe over physical USB connectors.

I knot that this parts list is a bit wild, but its only to demonstrate a concept. Maybe some of you have a better idea to solve the connection issue.

I also though about using a card like this one and use a selfmade, customized wireing harness, but this will not work because every device needs its own RefClock signal:
DS320-SLIMSAS-EVM Evaluation board | TI.com
Best way to use such a card would be with a backplane, which supports 16 U.2 devices with x1 connectivity.



Second idea is HBA-based:

Use an industrial standard HBA. I see the following problems here:

3. There are a lot of HBAs which support up to 8 or 16 SATA/SAS drives, because they have 8 or 16 channels. Officially its only possible to attache two or four PCIe-based drives like M.2 or U.2 because both usually use four lanes/channels. But, what if you connect 16 NVME-based drives to a HBA with only one lane/channel per drive? Will it work to run 16 NVME drives? Maybe the HBA chipset could do this job, but I guess the firmware is not ready for it.

4. Same problem like point 2. A wireing harness is needed to connect 16 drives to a 16 channel HBA. But in this case, you might need one with a miniSAS connector on the one side, and U.2 on the other side.

5. Every PCIe device needs its own RefClock connection. Not sure if a HBA has 8 or 16 individual clock pins available?



Does anyone have experience with this topic?

 

[ano]

pci gen5 doesnt support 1x, because intel doesnt support it apparantly

 

[heromode]

would a Icy Dock ToughArmour MB873MP-B with a SSD7580A be something worth studying?

that would be 2 pcie4 lanes per nvme device, but how common are pcie4/5 x8 slots on motherboards? They all seem to be x16.

a single lane would require a pcie4 x16 controller with 8x oculink cables connected to 2x of those icy docs?

 

[ano]

but... epycs has enough lanes to run x4 to 24x devices on a single cpu

 

[mattventura]

#1: Not sure if such a thing exists. x4x4x4x4 is usually the lowest you can go.
As such, #2 is unlikely to exist.

As for your list of components, the only problem I see is that it has a massive bottleneck - that M.2 to 4x1 switch card would work with a motherboard that can bifurcate x4x4x4x4 (as it is a switch rather than a bifurcation adapter), but the upstream port is only x1 - so you'd have 1/4 the bandwidth.

For the rest - there's two different types of NVMe "HBA".
Sometimes they're just a redriver or retimer card which would depend on the motherboard's bifurcation settings.
Others are a PCIe switch - this might be the closest you could get, as you'd be able to get 8 x4 connections out of a single x16.
Then there's tri-mode HBAs that expose it as a SAS drive. Supposedly, there's a way to get some Broadcom HBAs to break out to x2 links, but I'm not sure what all that requires.

However, if you're going for low power, it seems better to just get a larger drive with a wider link width.

 

[heromode]

As a note for large amount of nvme drives, i would forget about motherboard bifurcation, and focus on a pcie PLX based solution, for reliability and especially the ability to run it on any system.

Based on a quick glance of available hardware, and also the issue of available cables, seems to me that a pcie4 x16 PLX adapter divided to 8x pcie4x2 lanes is the most plausible solution ATM

 

[Partofthesolution]

pci gen5 doesnt support 1x, because intel doesnt support it apparantly

Oh, realy? Didnt know that. Does that mean that there is no more physical PCIe x1 slot on 5.0 boards anymore? Or does this also mean, that its not possible to attache onboard chips like Audio, NIC, IPMI etc. with only one lane on 5.0 boards?

but... epycs has enough lanes to run x4 to 24x devices on a single cpu

Yes, of course. If I would buy a EPYC System, I think I would buy this mainboard:

ASRock Rack > ROMED4ID-2T

www.asrockrack.com

But the thing is, I have a relative up to date ryzen system which is very power saving while EPYCs are not.

Powersaving is one of the goals why I would like to discuss the topic.

I would like to remove the LSI HBA with HDDs from the allways on system and replace it with flash drvies.
The HBA and the HDDs will take place in the backup system which must not be online 24/7.

would a Icy Dock ToughArmour MB873MP-B with a SSD7580A be something worth studying?

that would be 2 pcie4 lanes per nvme device, but how common are pcie4/5 x8 slots on motherboards? They all seem to be x16.

a single lane would require a pcie4 x16 controller with 8x oculink cables connected to 2x of those icy docs?

This enclosure looks nice. Unfortunately, my homeserver does not have 5.25" bay yet. New Enclosure is not out of budget and I will definetly keep this in mind.
Do you have one? How loud are the fans? Can the fan speed be adjusted?

I have a Asrock Rack x570d4u-2l2t Mainboard and here is my current build: https://forums.servethehome.com/ind...0d4u-2l2t-build-for-esxi-and-nas-usage.30993/

I have never heared of oculink and my mainboard do not have these ports. Looks like SFF to oculink adapters and cables are available. So, no showstopper here as far as I can see And if so, maybe its time for this HighPoint HBA

Edit: OMFG. One of these highpoint adapters costs like 1500EUR / 1.627,33USD.

Edit2: The ToughArmour MB873MP-B costs 750EUR. -_-

#1: Not sure if such a thing exists. x4x4x4x4 is usually the lowest you can go.
As such, #2 is unlikely to exist.

As for your list of components, the only problem I see is that it has a massive bottleneck - that M.2 to 4x1 switch card would work with a motherboard that can bifurcate x4x4x4x4 (as it is a switch rather than a bifurcation adapter), but the upstream port is only x1 - so you'd have 1/4 the bandwidth.

For the rest - there's two different types of NVMe "HBA".
Sometimes they're just a redriver or retimer card which would depend on the motherboard's bifurcation settings.
Others are a PCIe switch - this might be the closest you could get, as you'd be able to get 8 x4 connections out of a single x16.
Then there's tri-mode HBAs that expose it as a SAS drive. Supposedly, there's a way to get some Broadcom HBAs to break out to x2 links, but I'm not sure what all that requires.

However, if you're going for low power, it seems better to just get a larger drive with a wider link width.

Thanks for this informations.

Oh, yes, you are totaly right. There is a ASMedia chipset on the M2 to 4x1 card. I missed that point. Damn... It ruins the whole plan.

I will investigate your informations regarding the Broadcom HBA with x2 breakouts information.

As a note for large amount of nvme drives, i would forget about motherboard bifurcation, and focus on a pcie PLX based solution, for reliability and especially the ability to run it on any system.

Based on a quick glance of available hardware, and also the issue of available cables, seems to me that a pcie4 x16 PLX adapter divided to 8x pcie4x2 lanes is the most plausible solution ATM

Yes, seems so. Is you last sentence based on mattventuras information or do you have more information about which HBA supports 8x2 ? Or anyone else?

 

[heromode]

This enclosure looks nice. Unfortunately, my homeserver does not have 5.25" bay yet. New Enclosure is not out of budget and I will definetly keep this in mind.
Do you have one? How loud are the fans? Can the fan speed be adjusted?

I have never heared of oculink and my mainboard do not have these ports. Looks like SFF to oculink adapters and cables are available. So, no showstopper here as far as I can see And if so, maybe its time for this HighPoint HBA

Edit: OMFG. One of these highpoint adapters costs like 1500EUR / 1.627,33USD.

Edit2: The ToughArmour MB873MP-B costs 750EUR. -_-

Yes that Icy Dock is extremely expensive, but i haven't come across any other manufacturer making them yet.. I don't have one because i'm poor. STH actually reviewed the 12 port SATA version here

That that highpoint controller card is just an example i took from Icy Dock's webpage. There are several other much cheaper options available. There is an extensive thread in this forum listing lots of PLX based cards, and there are i think two manufacturers that sell on Aliexpress etc.. So the controller will not be 1500 EUR. Look for Ceacent and Linkreal

Oculink is just a new version of the SFF connectors, so cables are no issue.

To sum it up, for a cheaper solution you could look at a pcie3x16 PLX Adapter to 8x nvme drives, 2 pcie3 lanes per device. More expensive would be the same with pcie4.

The biggest issue would be where to host the drives. I guess there are PLX Adapters with placeholders for 8x nvme M.2 drives, but i imagine combined with the PLX logic they are expensive. (edit: and that also raises the issue of 75W watt limit of the pcie slot)

 

[mattventura]

Yeah, most NVMe mobile racks are offensively overpriced, especially for an unmanaged/dumb backplane. Wouldn't bother with that stuff since the increase in price would immediately kill the benefit of any power savings or whatever you're going for.

 

[unwind-protect]

What is this controller presenting on the drive side? x2 or x4?

NVMe RAID AIC Adapter | HighPoint Technologies，Inc.

HighPoint's performance focused PCIe 4.0 & 3.0 NVMe RAID AICs, for M.2, U.2/U.3 & E1.S SSD, come with advanced cooling systems to prevent performance throttling.

www.highpoint-tech.com

 

[i386]

Theoretically it would be possible to bifurcate down to 16* x1 lanes, but i think there are a few problems:
- how to power these devices
- cooling
- signal integrity (espcially with newer/faster pcie versions)

 

[ano]

also remember nvme enterprise devices LOVES power

signal integrity is why we are seeing pci gen 4. on gen 5 capeable hw apparantly

 

[mattventura]

What is this controller presenting on the drive side? x2 or x4?

NVMe RAID AIC Adapter | HighPoint Technologies，Inc.

HighPoint's performance focused PCIe 4.0 & 3.0 NVMe RAID AICs, for M.2, U.2/U.3 & E1.S SSD, come with advanced cooling systems to prevent performance throttling.

www.highpoint-tech.com

Looks like 8x4. Though you'd also probably want the non-RAID version of that.

also remember nvme enterprise devices LOVES power

signal integrity is why we are seeing pci gen 4. on gen 5 capeable hw apparantly

That, and you also have to make sure that whatever HBA/switches you use doesn't use too much power itself.


Oh, you know what, I forgot about something. Some manufacturers have mining-oriented boards that have a ton of PCIe x1 slots. They're not necessarily good slots - they might only be gen2 or gen3, and might all be bottlenecked on a switch chip, but it's something to explore.

 

[Mithril]

pci gen5 doesnt support 1x, because intel doesnt support it apparantly

What Intel does and does not support only really matters for lanes coming from the CPU. PCIe 5.0 is defined by PCI-sig, and is fully backwards compatible, which includes existing lane width options. PCI is also required to support fewer lanes than present (a 1x, 2x, 4x or 8x device in a 16x electrical slot).

 

[Mithril]

also remember nvme enterprise devices LOVES power

signal integrity is why we are seeing pci gen 4. on gen 5 capeable hw apparantly

Which is funny as I remember several people claiming that Gen 5 would actually be better for that due to allowable db drop and encoding, oh well

 

[ano]

what intel does, also reflects what a lot of the vendors do

little gen5 stuff is actually out though, we will see, but.. we have so many pcie lanes on cpu now, it really helps

 

[Mithril]

what intel does, also reflects what a lot of the vendors do

little gen5 stuff is actually out though, we will see, but.. we have so many pcie lanes on cpu now, it really helps

Except for internal-only traces I don't think I've ever seen CPUs support 1x standalone. In consumer land you've got your 16/20/24 lanes + 4 lanes for the "chipset", and maybe a few lanes effectively dedicated for certain onboard things. Bifurcation generally only goes down to 4x (this is not a new gen5 thing). It's usually the chipsets that 1x lanes hang off of. However you could bifurcate to 4x4x4x4x and stick 4 single lane devices. It would work, and waste 75% of the lanes.

IMHO the "right" way is to just use a PLX (well whatever they are called now) to do x4 to 4 4x

 

[heromode]

Except for internal-only traces I don't think I've ever seen CPUs support 1x standalone. In consumer land you've got your 16/20/24 lanes + 4 lanes for the "chipset", and maybe a few lanes effectively dedicated for certain onboard things. Bifurcation generally only goes down to 4x (this is not a new gen5 thing). It's usually the chipsets that 1x lanes hang off of. However you could bifurcate to 4x4x4x4x and stick 4 single lane devices. It would work, and waste 75% of the lanes.

IMHO the "right" way is to just use a PLX (well whatever they are called now) to do x4 to 4 4x

i agree. your options are basically a 4 or 8 port pcie 3/4 PLX card in a pcie 3/4 x16 slot, giving you 8x 2x pcie3/4 or 4x 4x pcie3/4. Anything else is not viable for real world use atm.

edit: a 8 port PLX card introduces lots of additional costs, cabling issues, and hassles for those 8x 2x lanes, it's not worth it. A better idea is getting a motherboard with enough pcie slots to run 2x cheap 4 port PLX cards instead.

 

[Partofthesolution]

edit: a 8 port PLX card introduces lots of additional costs, cabling issues, and hassles for those 8x 2x lanes, it's not worth it. A better idea is getting a motherboard with enough pcie slots to run 2x cheap 4 port PLX cards instead.

Now, that its clear that there is no "simple" solution, can we please discuss how its possible to add more than four NVMEs to a 9500-16e/i card (which can handle up to 32NVMEs) or up to aight NVMEs to a Rocket 1580 from Highpoint (which can handle up to aight NVME drives) in a consumerlike systems?

I know, that professional server cases come with backplanes which have a PLX on it, but how can one add such a PLX into a consumerlike system?


I am currently in contact with the guy behind these products, but unfortunately, it looks like he cant help much:

c-payne

c-payne.com

c-payne.com

Edit:

I think i just found a solution with the Highpoint 1580 card. Here is the Datasheet:

https://filedn.com/lG3WBCwKGHT7yNuTsFCwXy0/HighPoint-Download/Document/Datasheet/Rocket/R1000/R1580_U2_U3_Datasheet_v1.02_23_03_20.pdf

And here is the last page of the sheet. Its possible to attache aight drives to this card with the shown cables.

The cards is very expensive (1500EUR per piece). The 9500 cards are much cheaper (750EUR per piece). But I still dont know how to attache more than four NVMEs to it.