Multi-NVMe (m.2, u.2) adapters that do not require bifurcation

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Shonk

Member
Nov 25, 2016
63
35
18
74
I have a PCIE 8x ASM2824 with 4 x M.2 Coming from aliexpress for my server

It will be fine as its going in an 8x slot in the server and each drive will get a 4x link
though i could only saturate two drives at a time before im out of bandwidth on the slot

anyway before i fit it in the server i will test it in the 4x slot of my 14900KS
as its hard to find out if the m.2 drives will be forced to 2x due to the PCIE 4x link

I suspect all drives will link at 4x but ofc i can only saturate one drive at a time to not affect the 4x link

will reply in a week or so when it turns up from aliexpress


Images of the Card
1.png2.png
 

mack

New Member
Jul 17, 2022
20
6
3
Can anyone provide the idle power consumption for cards with PEX8724 or PEX8748(9) chips? I currently have ASM2824 and for it it is about 8W in idle mode with or without disks. I am looking for a switch that will allow me to add 4 nvme on x8 pcie and will use little power in idle mode.
 

kryten

Member
Apr 10, 2023
78
12
8
i need to be able to test 4x u.2 drives without bifurication.

i have tried pcie board that you place the nvme on but i found not all u.2 drives were detected using it.
High end chipsets like the samsung 1733a werent seen, but a cheap micron was.
I have tried a DL380 G10 but that doesnt seem to detect all types of u.2 drives.

Bifurication cards just seem to hit an miss as to where they work and the broadcom tri-mode are no good either.

The best solution so far is a supermicro 1u server that has onboard oculink, but there hard to come by.
 

nexox

Well-Known Member
May 3, 2023
1,410
657
113
Hello, in x4 slot it means they working with x1 speed each? Just curious, as this card will be good way to expand my fast storage, PSU can't handle more drives xD
Unless the switch chip is really lame it should share the x4 slot bandwidth with the drives that are doing active IO, and it's full duplex, so for example you could read at x4 from one drive and write at x4 to another drive, or read from two drives at approximately x2, etc.
 
  • Like
Reactions: Josu

Josu

New Member
Nov 19, 2024
2
0
1
Unless the switch chip is really lame it should share the x4 slot bandwidth with the drives that are doing active IO, and it's full duplex, so for example you could read at x4 from one drive and write at x4 to another drive, or read from two drives at approximately x2, etc.
ah cool, thanks
but how this drive will show up in system? as one or multiple? I am asking for Unraid perspective, I want to create lancache
 

nexox

Well-Known Member
May 3, 2023
1,410
657
113
ah cool, thanks
but how this drive will show up in system? as one or multiple? I am asking for Unraid perspective, I want to create lancache
A PCIe switch is mostly transparent, you can find it and see that the drives are connected to it if you poke around, but at a higher level it just looks like you have connected each drive directly, so it shows up as four raw devices that you can format or partition or whatever.
 
  • Like
Reactions: Josu

Shonk

Member
Nov 25, 2016
63
35
18
74
Hello, in x4 slot it means they working with x1 speed each? Just curious, as this card will be good way to expand my fast storage, PSU can't handle more drives xD
no its a PCIe switch chip

Upstream will link at 8x in an 8x electrical slot & 4x in a 4x electrical slot
then each drive will get a 4x link to the switch chip

It works in the exact same way as a Z690/Z790/Z890 PCH (As that is also a PCIe switch chip with extra's like usb and such)
 
Last edited:
  • Like
Reactions: KevinR

hooman64

New Member
Dec 23, 2024
4
1
3
Hi!

Searched this thread if anyone has tried a 4xNVMe in an HP Microserver gen10 plus, no hit.
Anyone? And if so, which card?
This looks promising though: Linkreal PCIe 3.0 X16 to Quad M.2 NVMe SSD Swtich Adapter Card for Servers-LRNV9547L-4I
I know it's been over year, but did you end up getting that card? I just ordered one and wanted to see if it worked with your Gen10+. I have the v2.
 

hooman64

New Member
Dec 23, 2024
4
1
3
I can confirm that LinkReal LRNV9547L-4I works. I have it installed on a HPE Proliant Microserver Gen10 Plus v2 with 4 NVMes and I can see all drives on my system. I'm also able to boot from one of the drives on the card. The card say it's low-profile. It's half height for sure and comes with both half and full brackets, but I think it's full length tho. I didn't have any issues installing it on my system, it pushes a bit on the HDD cables but not too much that's concerning. I haven't tested any of the drives to see what speeds I get tho.

Edit: here is a link to the card.
 

ribroc

ebay hardware hobbyist
Feb 16, 2022
26
25
13
I can confirm that LinkReal LRNV9547L-4I works.
I've picked two up. My notes:
  • The card "works" and I can see all the drives. It presents itself as "PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch"
  • On the sample size of two cards I have, both present the same problem. From my basic testing, bandwidth is full 3.0x4 for the nvme devices without issue or error, it appears to me that both cards I have suffer from a single bad slot. Possibly bad luck on my end.
  • Tested populated with 4.0x4 nvme devices, and they predictably operated at similar 3.0x4 speed to the 3.0x4 devices I tested.
  • If placed in slot 1 of my motherboard (typically a GPU slot), my test system doesn't boot. My only guess is that there is something with the x16 lane negotiation at boot that causes this, but this is strange for me to see. I wonder if the bad slot behavior on my cards are the cause.
  • Happily, if placed in an under-provisioned slot (pcie 4.0x1), the card properly negotiates and presents all four nvme's
    • Code:
       2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:03:11.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
If the caveats are acceptable, at ~115USD it's a winner of a card to me, but it does seem like it has some considerations before purchase.
 
Last edited:

ca3y6

Active Member
Apr 3, 2021
189
96
28
By the way, I think I experienced some compatibility issues between the caecent PCIe switch card and HGST u.2 drives (SN100 and SN200). Either the drives are not booting (no light on) or are sometimes dropping during use. The other brands of drives I tried do not have this problem (Micron and Intel). Those same HGST drives are stable in the same system connected to a u.2 adapter using motherboard pcie bifurcation.
 

hooman64

New Member
Dec 23, 2024
4
1
3
The card "works" and I can see all the drives. It presents itself as "PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch"
Interesting that yours show up as that, mine shows up as SLOT1 Drive[1-4] NVMe[0-3] + name of the manufacturer and the size in BIOS (except one drive that wouldn't show it's actual name even when connected directly to PCIe interface). and in my system when I run lsblk they show up as NVMe[0-3].

I'm in the infancy stages of building my system and will let you know how the drives react, but I'm planning on hosting the VMs+LXCs on the drives. Are you using yours as part of your storage pool?

I returned 2 of the drives I had since I'm going to use only 2 drives for now (I might expand to 4 in the future). I can run some test if you or anyone is interested in that. currently I'm mirroring the 2 drives as RAID1 and there seems to be no issues.

On the sample size of two cards I have, both present the same problem. If one/two nvme slots are filled, bandwidth is full 3.0x4 for the nvme devices without issue or error, but if any more than two slots are filled, we begin to see millions of correctable errors presented, performance degrades significantly, and this causes excess CPU use. This is fixed via OS ASPM, which means that passing the nvme pci device through to a VM doesn't really work. Sad news for the truenas VM use case. (Happy to be corrected if people have tips)
how did you test this? I'm new to this thing and can try and reproduce this on my system as well.
 

ribroc

ebay hardware hobbyist
Feb 16, 2022
26
25
13
I've updated my above post. I think I actually got two different bad slots on the two different cards. I'm going to order another one and see if the issue was just poor luck.
 

nexox

Well-Known Member
May 3, 2023
1,410
657
113
Interesting that yours show up as that, mine shows up as SLOT1 Drive[1-4] NVMe[0-3] + name of the manufacturer and the size in BIOS (except one drive that wouldn't show it's actual name even when connected directly to PCIe interface). and in my system when I run lsblk they show up as NVMe[0-3].
BIOS and lsblock are just showing you storage devices, you need to use lspci or similar to see the PLX device.
 
  • Like
Reactions: TRACKER

Prophes0r

Member
Sep 23, 2023
85
75
18
East Coast, USA
By the way, I think I experienced some compatibility issues between the caecent PCIe switch card and HGST u.2 drives (SN100 and SN200). Either the drives are not booting (no light on) or are sometimes dropping during use. The other brands of drives I tried do not have this problem (Micron and Intel). Those same HGST drives are stable in the same system connected to a u.2 adapter using motherboard pcie bifurcation.
I'm going to guess it's a power thing.

x4 and x8 slots only provide 25w per the spec.
x16 slots spec is 75w.

Those HGST drives are HUNGRY, requiring up to 20w(?) each.

I have an x16 card that mounts 4x U.2 drives and has an external SATA power connection for another ~50w of overhead.

I know that when I was testing those drives in combination with Micron and Intel ones, I forgot to plug the extra power in a few times and got sketchy behavior.

NOTE: The x16 card is...fine. Mine came with the wrong kind of drive screws. It's also a PITA to keep the drives cool and required me to DIY a crazy shroud with separate fans blowing THROUGH the drives, and alongside them. If I was shopping again I would bite the bullet and get a card with a PCIe switch and MCIO ports, and shell out for cables. But it does the job and was BY FAR the least expensive option.
 
  • Like
Reactions: nexox