Multi-NVMe (m.2, u.2) adapters that do not require bifurcation

[unphased]

So you are dealing with an ITX board with 3x M.2s? Even cheap ITX boards should provide 4xSATA, but an LSI card can give 8 or 16x SATA. lots of extra cables to route though, but nothing hard to deal with. I would personally want to put an HBA off of one M.2, that handles all your SATA stuff, I wouldn't waste NVMe space running RAID1, but I would set up a robust backup system to prevent data loss (which could target spinning rust if your data movements are not constant and furious). Then I would put the fastest NVMe on the fastest M.2 slot and the rest off a PLX card off the remaining M.2 slot.

But if you have so many storage peripherals it begs the question of if any discrete GPU present there truly needs to hog all x16 lanes on the main x16 slot as your storage seems important enough to start wanting to consume some of that. If you have 4x NVMe you may as well go all-in with a x4x4x4x4 M.2 card and make it a storage monster (note pitfalls like the 5600G i have only giving x8 lanes on that slot!)

I recently reconfigured my 5950x workstation rig from running 12 HDDs and 2x 3090's into two separate machines, the 5950x hosts 3x 3090s now and a 5600G on an ITX board is on dedicated NAS duty. it's excellent since the NAS now no longer needs to burn any discrete gpu power. I would have liked 4 ddr4 slots to give to the NAS but it's a non issue and I'll take more ram for the GPU box actually, yes, when it needs it it will need it. ZFS can use lots of ram but it doesn't NEED need it.

 

[ptf]

So you are dealing with an ITX board with 3x M.2s?

The ASRock board has two - the MSI board has three but one is on the back of the motherboard with zero airflow - one of the things I wanted to do was get away from having an M.2 mounted there - currently my hottest drives are 45°C and the motherboard drives low 30's; prior to the old motherboard failing it would be more like 75°C for the rear mounted drive.

Even cheap ITX boards should provide 4xSATA

Some boards which are definitely not "cheap" have fewer - eg the ASRock Z790 PG-ITX/TB4 only has three and the Asus ROG STRIX Z790-I only has two), but both of the motherboards that I have handy do, indeed, have four. I was kind-of trying not to use the motherboard ports as they are a bit of a pain to use in this case (it's pretty tight) but I think I will have to, along with a single ASM1166 or JMB585 based expansion board.

I wouldn't waste NVMe space running RAID1

Well, that's debatable MVME drives can still fail, they do so at a lower rate than hard drives but not that much. It's about 60% of the failures per unit time compared with HDDs - and I've certainly had HDDs fail.

For now, at least, my NVME drives are all in RAID1 pairs.

but I would set up a robust backup system to prevent data loss

Backup and RAID solve two different problems - I have backup, both on and off site for critical bits of data.

put the fastest NVMe on the fastest M.2

On the ASRock both are Gen4 x4, on the MSI it's two Gen4x4 and one Gen3x4 - but the second Gen4 x4 is the one on the back of the motherboard that I don't want to use.

any discrete GPU [...] needs to hog all x16 lanes on the main x16 slot

The motherboard is ITX - so there's only one slot. In any case the server doesn't need fancy graphics so the system uses the integrated graphics leaving the PCIe slot free for M.2 and/or SATA adapters..

If you have 4x NVMe you may as well go all-in with a x4x4x4x4 M.2 card and make it a storage monster (note pitfalls like the

Well, that's the intent

But I can only do it with a PEX87xx (or 880xx) based PCIe splitter - intel 12th/13th/14th gen is limited to x16 or x8x8 bifurcation on the CPU PCIe lanes which means a passive splitter will not support four drives. I did run a passive splitter for a while with one of the M.2's and one of the ASM1166 cars and the system ran quite stably like that.

I have a feeling that is the configuration I might be stuck with.

5950x

I've never really been into AMD - almost every time I've needed a new motherboard and CPU I've looked at AMD and every time I have come to the conclusion that Intel is better for what I want to do.

Currently the CPU is an i7-12700T - 8P cores and 4 E cores for 20 threads and a 35W TDP. If I look at AMD I can have more threads - your 5950x is 32 I believe, but it also has a TDP of 105W which is too toasty as I can only fit an ultra low profile cooler in the case. If I go to 35W then I'm limited to 8 cores and 16 threads.

I would have liked 4 ddr4

Again mini-ITX is a bit limiting here but thanks to the old motherboard also blowing up one of my 48GB sticks I now have 128 which should be enough for now. It's really not the time to be forced to buy large amounts of RAM

For now the plan is to try a firmware update (which seems to have been successful but I don't yet know if it has fixed anything) or the JMB585 based SATA adapters. If neither work it will be a passive splitter with one SATA adapter and the other M.2 card in a USB adapter.

 

[unphased]

get away from having an M.2 mounted there

This slot is best for putting M.2 to PCIe risers. Two birds with that stone for sure. I always do this on my SFF builds because it gets hot there for sure. I don't notice NVMe throttling except when i was testing stuff in WSL2. The filesystem access is very inefficient there and this is one way i know. I regularly use m.2 -> pcie -> back to m.2 for NVMe's to put them in a more amenable location for cooling, and this also makes it super convenient to swap that for NICs and other stuff as the need arises.

I've certainly had HDDs fail.

I've had SSDs fail as well, but I still couldn't stomach 0.5 capacity ratio to get redundancy for NVMe. Optane might be up your alley for the use case though it may be hard to source now.

intel 12th/13th/14th gen is limited to x16 or x8x8 bifurcation on the CPU PCIe lanes

Color me spoiled with AMD then. Wow that is shocking.

Good luck! You really just want a pcie 4 version of the 4x m.2 ceacent plx card I have, except mine is gen 3 and would bottleneck (but not too bad if you have all the lanes!), perhaps that is what you say you already have.

 

[ptf]

This slot is best for putting M.2 to PCIe risers.

No place to put another PCIe card in the system, a M.2 extender could go there. But another issue the front side M.2's on the MSI board are stacked vertically which interferes with anything sticking out from the PCI card on that side (and there's not much room on the other side of the PCIe card in the U-NAS case). Overall the ASRock is a better board physically - the two M.2's are side by side with a heat spreader over both of them, the NVMEs seem to run much cooler than they do on the MSI board.#

Good luck! You really just want a pcie 4 version of the 4x m.2 ceacent plx card I have, except mine is gen 3 and would bottleneck (but not too bad if you have all the lanes!), perhaps that is what you say you already have.

but I still couldn't stomach 0.5 capacity ratio to get redundancy for NVMe

Yes, it's a pain but think of it as a constantly updated backup.

RAID5 or 6 are also possibilities but I appreciate there's a debate to be had about RAID on top of NVME.

Color me spoiled with AMD then. Wow that is shocking.

Last gen Intel is rubbish on that score. 15th Gen allows 8x4x4x, as did 10th/11th Gen - but IIRC you have to go back to 9th Gen for x4x4x4x4 (and then only if supported by the motherboard).

Good luck! You really just want a pcie 4 version of the 4x m.2 ceacent plx card I have, except mine is gen 3 and would bottleneck (but not too bad if you have all the lanes!), perhaps that is what you say you already have.

Yes, it's a no name Aliexpress PEX8749 one though https://www.aliexpress.com/item/1005009972741405.html

PCIe Gen 4 cards based on the PEX880xx are also available, but pricier https://www.aliexpress.com/item/1005009298996028.html or Broadcom PEX88048 Array Card/8 PCIe 4.0 x16 M.2 NVMe with Cooling 4010 Ball Fan/2*SFF-8654(8i)/Support Soft array/2230 2260 2280 22110 M.2NVMe /Dual Power Supply: Amazon.co.uk: Computers & Accessories

There are also cards based on the ASMedia ASM2824 https://www.amazon.co.uk/PCIe-ASM2824-Adapter-Expansion-Converter/dp/B0872YW1BL

I'm hoping not to have to buy every variant before I find a solution that works

Flashing the ASM1166 didn't help - well it might have a bit, under heavy I/O load it took about five minutes to lock the system (compared with a few seconds on the older firmware - though it could just have been luck).

 

[panchovix]

Just received a PEX/PLX88048 switch. It is X16 4.0 to 12*X4 4.0 (8 NVMe slots and 2 Slimsas 8i downstream that I think are 4x4 each).

I got this one: https://es.aliexpress.com/item/1005009881058457.html

I will put it on a X4 5.0 slot for now (so X4 4.0) to do some tests, not storage but M2 NVMe to PCIe adapters.

I have a SlimSAS X16/X8 host card but not sure if it will work with the switch set at X4/X4 on the SlimSAS ports.

The host SlimSAS I mean looks like this, but not sure if it will work:

My other PLX88096 switches should arrive next week or next next week.

 

[ptf]

Just received a PEX/PLX88048 switch. It is X16 4.0 to 12*X4 4.0 (8 NVMe slots and 2 Slimsas 8i downstream that I think are 4x4 each).

Hope it goes well - do keep us updated, I for one am very interested to see if this works to a general PCIe card on the riser at the same time as nvme drives on the board itself..

Can't say I have found these cards a positive experience, using a pair of JMB585 based M.2->SATA adapters was better but it has still locked the system solid twice in 48 hours and seems to make booting unreliable as well - not good for a server, especially if I'm away from home and can't get console access.

I'd love to fill one with 8x8TB nvme drives but that is too much investment of £££ if it is not going to be reliable.

For now I have thrown enough money and time at the problem - I'm going back to a single ASM1166 adapter - it's in a passive splitter set to x8x8 which gives me the option of adding an extra NVME drive though that is not that useful to me at the moment, so it's just the SATA adapter. the two nvme drives that I *do* want in the system are in USB caddies.

I note no one in the thread or in response to my original query has actually popped up and said "yes this works for me" - at this point I don't know if the issue is the splitter, the motherboard or the SATA adapters but I have been through two of each in various combinations and nothing has worked reliably except a passive splitter - but I'm stuck with a very limited choice in what bifurcation modes are supported on the Intel platform.

Will let you all know if I ever sort this. I have a couple of smaller capacity spare drives so I might put together a scratch system and see what happens, either with nvme drives + the SATA adapters or just nvme. In which case I will report back.

A parting thought - mine had a little microprocessor daughter board containing a STM32G030C8T6 - several listings show this, typically labelled as "improving drive compatibility" - anyone know what it actually does?

 

[panchovix]

@ptf on my case it worked fine! I used 5 M2 to PCIe adapters, so connected 5 GPUs on that PLX88048 switch, and also moved 4 SSDs to a PLX88024 switch.

Topology looks like this

pancho@fedora:/run/media/pancho/MX500/ChatIAs/SillyTavern$ lspci -tv
-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Root Complex
           +-01.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
           +-01.1-[01]--+-00.0  NVIDIA Corporation GB202 [GeForce RTX 5090]
           |            \-00.1  NVIDIA Corporation GB202 High Definition Audio Controller
           +-01.2-[02-1a]----00.0-[03-1a]--+-00.0-[04-0d]----00.0-[05-0d]--+-00.0-[06]--
           |                               |                               +-04.0-[07]--
           |                               |                               +-08.0-[08]--+-00.0  NVIDIA Corporation AD102 [GeForce RTX 4090]
           |                               |                               |            \-00.1  NVIDIA Corporation AD102 High Definition Audio Controller
           |                               |                               +-0c.0-[09]--
           |                               |                               +-10.0-[0a]--+-00.0  NVIDIA Corporation AD102 [GeForce RTX 4090]
           |                               |                               |            \-00.1  NVIDIA Corporation AD102 High Definition Audio Controller
           |                               |                               +-14.0-[0b]--+-00.0  NVIDIA Corporation GA102GL [RTX A6000]
           |                               |                               |            \-00.1  NVIDIA Corporation GA102 High Definition Audio Controller
           |                               |                               +-18.0-[0c]--+-00.0  NVIDIA Corporation GA102 [GeForce RTX 3090]
           |                               |                               |            \-00.1  NVIDIA Corporation GA102 High Definition Audio Controller
           |                               |                               \-1c.0-[0d]----00.0  NVIDIA Corporation GA102GL [A40]
           |                               +-04.0-[0e-13]----00.0-[0f-13]--+-10.0-[10]--
           |                               |                               +-14.0-[11]--
           |                               |                               +-18.0-[12]--
           |                               |                               \-1c.0-[13]--
           |                               +-08.0-[14-15]----00.0-[15]--
           |                               +-0c.0-[16-19]----00.0-[17-19]--+-14.0-[18]--
           |                               |                               \-15.0-[19]--
           |                               \-1c.0-[1a]----00.0  Broadcom / LSI PEX880xx PCIe Gen 4 Switch
           +-02.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
           +-02.1-[1b-34]----00.0-[1c-34]--+-00.0-[1d-26]----00.0-[1e-26]--+-00.0-[1f-25]----00.0-[20-25]--+-00.0-[21]----00.0  ADATA Technology Co., Ltd. LEGEND 850 LITE NVMe SSD (DRAM-less)
           |                               |                               |                               +-04.0-[22]----00.0  Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD [E18]
           |                               |                               |                               +-08.0-[23]----00.0  ADATA Technology Co., Ltd. LEGEND 850 LITE NVMe SSD (DRAM-less)
           |                               |                               |                               +-0c.0-[24]----00.0  Silicon Motion, Inc. SM2269XT (DRAM-less) NVMe SSD Controller
           |                               |                               |                               \-10.0-[25]--
           |                               |                               \-1c.0-[26]----00.0  Broadcom / LSI PEX880xx PCIe Gen 4 Switch
           |                               +-04.0-[27]--
           |                               +-05.0-[28]--
           |                               +-06.0-[29]--
           |                               +-08.0-[2a-32]----00.0-[2b-32]--+-00.0-[2c]----00.0  Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
           |                               |                               +-04.0-[2d]--
           |                               |                               +-06.0-[2e]----00.0  Intel Corporation Ethernet Controller I225-V
           |                               |                               +-07.0-[2f]--
           |                               |                               +-08.0-[30]--
           |                               |                               +-0c.0-[31]----00.0  Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller
           |                               |                               \-0d.0-[32]----00.0  Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller
           |                               +-0c.0-[33]----00.0  Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller
           |                               \-0d.0-[34]----00.0  Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller
           +-02.2-[35]--+-00.0  NVIDIA Corporation GB202 [GeForce RTX 5090]
           |            \-00.1  NVIDIA Corporation GB202 High Definition Audio Controller
           +-03.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
           +-04.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
           +-08.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
           +-08.1-[36]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon Graphics]
           |            +-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller
           |            +-00.2  Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP
           |            +-00.3  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI
           |            +-00.4  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI
           |            \-00.6  Advanced Micro Devices, Inc. [AMD] Ryzen HD Audio Controller
           +-08.3-[37]----00.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 2.0 xHCI
           +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
           +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
           +-18.0  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0
           +-18.1  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1
           +-18.2  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2
           +-18.3  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3
           +-18.4  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4
           +-18.5  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5
           +-18.6  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6
           \-18.7  Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7

P2P looks like this:

pancho@fedora:/run/media/pancho/MX500/ChatIAs/SillyTavern$ nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-23    0               N/A
GPU1    PHB      X      PIX     PIX     PIX     PIX     PHB     PHB     0-23    0               N/A
GPU2    PHB     PIX      X      PIX     PIX     PIX     PHB     PHB     0-23    0               N/A
GPU3    PHB     PIX     PIX      X      PIX     PIX     PHB     PHB     0-23    0               N/A
GPU4    PHB     PIX     PIX     PIX      X      PIX     PHB     PHB     0-23    0               N/A
GPU5    PHB     PIX     PIX     PIX     PIX      X      PHB     PHB     0-23    0               N/A
GPU6    PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     0-23    0               N/A
NIC0    PHB     PHB     PHB     PHB     PHB     PHB     PHB      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx4_0

So pretty happy for now so far!

In some weeks I will get those PLX88096 switches to do more tests.

This one:



And this one (which has a switch for X16,X8/X8 or X4/X4/X4/X4 for all slots)

 

[ptf]

@ptf on my case it worked fine!

Ha, I'm jealous

Though this is with GPU's - has anyone found it stable with SATA adapters?

 

[nexox]

Though this is with GPU's - has anyone found it stable with SATA adapters?

At some point you're going to be better off with a SAS HBA and an expander, since a PCIe switch and a pile of SATA adapters will use a similar amount of power, but with apparently less reliability and fewer people who can help you with the setup.

 

[ptf]

At some point you're going to be better off with a SAS HBA and an expander, since a PCIe switch and a pile of SATA adapters will use a similar amount of power, but with apparently less reliability and fewer people who can help you with the setup.

Yeah, I'd agree if I wanted to stay with SATA drives long term - the catch here being that I don't, a HBA is not going to drive a bunch of NVME drives and I only have one PCIe slot.

As above, the long term aim is be to transition to all solid state, until then I have 23T of data (in about 45T of filesystem), half of which *is* already on solid state but 870 QVO's (i.e. SATA). I'd love to stick 8x 8TB nvme drives on one of the splitter cards and format a RAID6 array on it (48GB usable) but I don't want to part with roughly £5.3k on drives if the platform is not solid. Nor do I really want to throw away £2.5k worth of 870 QVO drives in the process, though to be fair I could use the M/B SATA ports as it's just the four drives and I'd probably get 60% of their purchase price back on eBay.

 

[panchovix]

Got the PLX88096 switch/plate! It arrived and tested it shortly at X16 4.0, worked without issues.

I removed it temporary as I got a Gen 5 100 lane switch from C-payne (the microchip switchec one, not arrived yet) and I will connect that one to a X16 5.0, and one downstream connect this switch and other one, the pending PLX88096 Slimsas switch that haven’t arrived yet.

Some pictures.

Man my experience with AliExpress have been so good lol.

 

[panchovix]

Got this beauty today, PCIe Gen 5 switch with 100 lanes.

I could test it with a passive MCIO host adapter to it, but if I had to guess it won’t work really good without a retimer.

So gonna wait until I get the retimer (sadly since I want to use it so much right now lol)

EDIT: Also I wonder, which ones are the 2 MCIO X8 uplinks?

 

[panchovix]

Christian answered me with the updated c-payne tool for anyone that has any of his switches (including PCIe 4.0 ones)

https://c-payne.com/c-payne-tool

He told me he added all of my configs and a bit more.

So I requested this for those interested:

* Default: All X16 (so 5*X16 5.0 downstream)
* All X16 (so 5*X16 5.0 downstream)
* All X8 (so 10*X8 5.0 downstream)
* All X4 (so 20*X4 5.0 downstream)
* X16 for 3 pairs of MCIO, and the rest at X8 (so i.e. MCIO0 to MCIO5 at X16, MCIO6 to MCIO9 at X8)
* X16 for 4 pairs of MCIO, and the remaining pair at X8 (so i.e. MCIO0 to MCIO7 at X16, MCIO8 and 9 at X8)
* X16 for 2 pairs of MCIO, 2 pairs at X8 and 1 pair at X4 (so MCIO0 to MCIO3 at X16, MCIO4 to MCIO7 at X8, MCIO8 and 9 at X4)
* X16 for 3 pairs of MCIO, 1 pair at X8 and 1 pair at X4 (so MCIO0 to MCIO5 at X16, MCIO6 and 7 at X8, and MCIO8 and 9 at X4)

MCIO 11 and 12 by default are upstream (if you check the manual I posted here), the rest are downstream. You can change any port to other port to be upstream and downstream, including the X4 one.

 

[panchovix]

And another PLX88096 one, this one took a while to get here but for the price (400USD) I can’t complain much.

I thought it would be a scam lol, but it works just fine, tested just quickly at X16 and X8X8, works without issues. But I'm on standby until some cables and adapters arrive.

No more switches for now I think, until PLX89048 ones drop a little maybe (there is one for 1K USD on AliExpress)

(And also sorry for so many comments from me)

 

[potrebitel]

And another PLX88096 one, this one took a while to get here but for the price (400USD) I can’t complain much.

This is one of the most interesting cards - in theory it should be able to be programmed on different modes for mixing GPU + NVMe drives, but haven't got the time to really check.

Can you please share the link for this one and the gen5 ones ? I've seen several places on AE, but they are way overpriced and seems its a "front" for taobao dropshippers (hence the high price on AE).

Out of curiosity - what do you do with so many GPUs ? Training or inference ?

 

[panchovix]

This is one of the most interesting cards - in theory it should be able to be programmed on different modes for mixing GPU + NVMe drives, but haven't got the time to really check.

Can you please share the link for this one and the gen5 ones ? I've seen several places on AE, but they are way overpriced and seems its a "front" for taobao dropshippers (hence the high price on AE).

Sure, here are the links.

The PLX88096 with switches above: https://es.aliexpress.com/item/1005010306067997.html (though if you search for PLX88096 on aliexpress, maybe there's other cheaper ones. This one was cheaper some weeks ago)
Microswitch Switchec 100 lane Gen 5: PCIe gen5 MCIO Switch 100-Lane - Microchip Switchtec PM50100
The PLX89048 one from aliexpress: https://es.aliexpress.com/item/1005010616929159.html, this one is about 1000USD.

If you buy from Ali right now you can have some coupons on the app, like 60USD off on 500USD, or 55USD off on 450USD or something like that.

EDIT: Also for sure they're cheaper on taobao or alibaba, but I'm not sure how to buy there .

Out of curiosity - what do you do with so many GPUs ? Training or inference ?

For now I only have done inference, and training on 2x5090 (at X8/X8 5.0). But got the switch to be able to train with the P2P driver (GitHub - aikitoria/open-gpu-kernel-modules: NVIDIA Linux open GPU with P2P support) with the 5090s at X16/X16 5.0, or 4090s at X16/16 4.0, or the A40/A6000 at X16/X16 4.0. The P2P driver let you direct PCIe connection instead of going by the CPU root complex on any GPU (Ampere and newer).

 

[UhClem]

Only PCIe Gen3, but GREAT price (while/if it lasts).
PCIe x16 host to 8 x4 targets (via 4x SFF-8654_8i ports).
Search aliexpress.com for >>CNS44PE16<<
Sold by Ceacent_Store (the manufacturer). Price: US$50 (+10ship)

I've been using one for 4+ years; excellent results.
Just bought another! (At $50, "Better to have and not need, than to need and not have", right?)

 

[panchovix]

Only PCIe Gen3, but GREAT price (while/if it lasts).
PCIe x16 host to 8 x4 targets (via 4x SFF-8654_8i ports).
Search aliexpress.com for >>CNS44PE16<<
Sold by Ceacent_Store (the manufacturer). Price: US$50 (+10ship)

I've been using one for 4+ years; excellent results.
Just bought another! (At $50, "Better to have and not need, than to need and not have", right?)

I searched that but only found one for 100USD and one for 200USD. Can you share the link?

 

[nexox]

I searched that but only found one for 100USD and one for 200USD. Can you share the link?

https://www.aliexpress.us/item/3256805484706547.html for the US, anyway, but AliExpress tends to vary products and prices a lot by region.

Edit: I'm tempted to grab one even though my boards all support bifurcation and I already have a spare 8 port switch...

 

[panchovix]

https://www.aliexpress.us/item/3256805484706547.html for the US, anyway, but AliExpress tends to very products and prices a lot by region.

Edit: I'm tempted to grab one even though my boards all support bifurcation and I already have a spare 8 port switch...

Ohh I can see it now. It doesn't ship to Chile, that's why haha. But amazing price! I would get 2 if I could lol.