External PCIE Bifurcation

Ryan Knightley

New Member
Jul 15, 2017
7
0
1
29
Hello all,

So I recently got it into my head after seeing NVIDIA open up GPU passthrough for GeForce cards that I wanted to try something a little unusual, though perhaps not so much on this forum.

I'm a 3d artist and do a fair bit of rendering but am also an avid tinkerer and like messing around with combining odd bits and ends and seeing what I can make of it. As a result, I came up with this idea to build an external PCIE expansion box on my own, but not only that, I wanted it to use the Threadripper platform and be supported by bifurcation so I could maximize my slot density. The Asus Zenith Extreme motherboard has 2 x16 slots that can be split to 4 each (supposedly, I did put a hyper m.2 card in one so I know that much works) 2 x8 slots that can also be split, a x4 lot and an x1 slot. So of these, it looks like I can possibly get 13 x4 slots in on my board.

In a nutshell, I bought some of these intriguing risers from a website called maxcloudon, you can find them here. They claimed their risers support bifurcation some x8x8 and some x4x4x4x4 and a few in between. I went ahead and ordered some from them and much to my surprise I was able to get two old quadros I had lying around to be recognized on the bottom most slot of my Zenith Extreme motherboard (x8 or x4+u.2 or x4x4 aka pcie raid as Asus calls it). Great, easy enough.

These risers appear to use sff-8087 instead of the USB connectors that you would typically see on something like a mining riser. So it got me thinking, why not get a sff-8087 to sff-8088 adapter card and hook them up to see if I could relocate the cards instead to a separate enclosure where I could better control my thermals and possibly set it up for water-cooling.

So I ordered a pair of each the sff-8088 cables and 8088-8087 adapters from Amazon. They arrived and I went to connect up a single known working riser to the new hardware. Lo and behold, no signal from the card and it's no longer recognized in the device manager. However, I did note that my boot sequence was longer than normal as if the board realized something was different and needed to reconfigure.

So my question to those of you that have experience in one or more of the technologies used here is where did I mess up or where should I start looking? I already tried every combination of new cabling with new adapters to rule that out (though if I have more than one piece of DOA hardware I really don't have any other way to test) and I did plug the riser back up in it's default configuration to make sure I didn't fry something somewhere during my testing and the card is once again recognized when connected directly. I'm assuming it's not due to signal loss, as I've seen before a ridiculous number of daisy chained pcie risers courtesy of Linus Tech Tips.

Thanks for taking the time to look through this!
 
Last edited:

UhClem

Member
Jun 26, 2012
88
34
18
NH, USA
The devil is in the details. If you check the product description of the cable component[Link] of the kit you linked, you see:
Uses custom pinout in order to comply with PCI-e standards - should NOT be replaced with default SFF-8087 cable
In addition, while the standard sff-8087 connector is stated to be 36-pin, the standard sff-8088 connector is stated to be 28-pin, giving Murphy a 2nd bite at your apples :) .

Hey, you can't have everything [where would you put it?]
 
Last edited:

kapone

Well-Known Member
May 23, 2015
1,006
573
113
Interesting project... :)

Yup, as @UhClem stated, the SAS 8087 connectors on that riser are wired differently than standard SAS.

while the standard sff-8087 connector is stated to be 36-pin, the standard sff-8088 connector is stated to be 28-pin
That's by design. The 8088 connector does not carry sideband signals (there are 8 pins for that), because the expectation is that the device at the end of a 8088 cable is a storage enclosure type of device, which has one of the (SES/SES-2/SGPIO etc) enclosure management protocols built-in. No dedicated pins are required for that.

@Ryan Knightley - If you wanna do what you're trying to do, you'll have to use the cables provided by the riser. However...nothing prevents you from using using external pcie adapters to get the whole pcie bus out to a different box, and then splitting it using the risers you mentioned. That's not a cheap endeavor and will get expensive real fast for multiple pcie slots. :)

If I wanted to do this, I'd most likely go with a Magma expansion chassis (they even make one that is like 18 slots or something), but use two of them (since each splits one x16 slot into typically 7 slots). They use PLX based switching, so no bifurcation support is needed on the motherboard side. That would be the "cleaner" solution, but again not cheap. These expansion chassis run ~$2K. Hell, I could build 10 systems for that much, each with an x16 slot. :)
 
  • Like
Reactions: legopc and UhClem

Ryan Knightley

New Member
Jul 15, 2017
7
0
1
29
Thank you both @UhClem and @kapone!

The devil really is in the details! This really does scupper all the plans I had for this enclosure. Though I don't think it scraps them entirely. I'll just have to rethink having them as two independent boxes.

UhClem you'll have to forgive me as I'm still pretty green to all this, when it says custom pinouts should not be replaced by a standard cable, in the instance of adapting why does this matter if the end cable is also one of the originals provided by the company? Like say for example, I were to buy a pair of 8087 couplers put a standard 8087 in-between and the maxcloudon cables on either side. Would that effectively convert the pinout appropriately so the right wires are all connected? Murphy needs to quit taking bites of my pie because I want to have it and eat it too!

That's by design. The 8088 connector does not carry sideband signals (there are 8 pins for that), because the expectation is that the device at the end of a 8088 cable is a storage enclosure type of device, which has one of the (SES/SES-2/SGPIO etc) enclosure management protocols built-in. No dedicated pins are required for that.
Kapone, thank you for that nugget of information. Really appreciate you taking the time to explain that one because I was wondering how the signal would be kept if the number of pins was different. In the end, I (incorrectly) assumed that perhaps the pins were unused and would not interfere with my plans. I had seen the solution from Magma years ago when I was working for an architectural design company and the pricing was just plain eye watering. I was hoping that perhaps in there intervening years the tech had become more affordable/accessible. Much to my dismay, while things like nvme drives have come down significantly in price, external PCIE still appears to remain a pretty niche sector.

Even if the coupling idea were to work, those aren't cheap at ≈$50 each so to do that for all 13 cards that's still $650. At which point, you're right it just makes more sense to just build a second machine entirely. The only point that might still make the single box idea better is if software licensing exceeds the cost of all of this. Just the rendering application alone is roughly $500 a license not to mention the host application being roughly $4500.

With all that in mind, are the options pretty much still either paying a ton for a pcie expansion card setup or building multiple boxes?
 

Blinky 42

Active Member
Aug 6, 2015
604
222
43
45
PA, USA
Yes the magma chassis are pretty pricey (we used to use them at $job) but they do work well as long as you power cycle things in the correct order.
I went down a bit of a occulink to pcie google rabbit hole since I was messing aroudn with occulink cables last week for NVMe.
Found this guy's shop Slimline Device Adaptors based on a Hardforum thread PCIE Bifurcation You will probably want to go for shielded cables like the magma stuff has if it isn't a very short cable run however to help with the signalling noise. You may need re-timers or an active bridge because the initial training on each pcie link can fail pretty easy when you get too crazy with cables and connectors in the signal path.
 
Last edited:

kapone

Well-Known Member
May 23, 2015
1,006
573
113
How about one of those "mining" type of chassis that can take ~8 GPUs? Would that work?

It'll still be a mess cabling wise (with risers and extenders etc) but at least it'll in one machine...food for thought.
 

gkanev@maxcloudon

New Member
Apr 9, 2021
1
0
1
Hi @Ryan Knightley,

I'm Gabriel and I part of the team behind MaxCloudOn. Our connectors are custom and not exactly "sff-8087". That is why experimenting with other cables can damage the risers and your hardware. Soon we will start selling longer cables (for now we have tested successfully up to 3 meters and so far it is working flawlessly). If you want you can subscribe to our newsletter and you will know when the cables are up for grab.
 

Ryan Knightley

New Member
Jul 15, 2017
7
0
1
29
@Blinky 42, I think I came across those same risers when I found this blog post detailing the process. Though the issue I took with the occulink stuff was that since they were all multicard risers, you were limited in what kinds of cards could be put in one or at least it made it more complicated. If I were to single slot convert and water cool every single card those might work fine though.

@kapone I think a mining form factor chassis might be fine so long as the board can still use the bifurcation risers. I think most of the mining boards specific to those chassis are typically x1 which is a no go for rendering tasks where x8 is preferred and x4 is where you start to get minimal performance degradation.

@gkanev@maxcloudon Ah that's great! Yes I think some longer cables would be great for single box setups. Would it be possible to at some point develop an external connector to achieve the result that I was going for in this build? I know that it's a pretty niche build, but I have some ideas that having the external cabling would be very beneficial for. I'm imagining a product similar to the one I've already bought but instead of the connectors on the pcie card being internal, there are four of them facing outward from the pcie slot like a normal pcie card that would then connect to another card in a separate machine that would then connect to your existing risers.
 

UhClem

Member
Jun 26, 2012
88
34
18
NH, USA
... That's by design. The 8088 connector does not carry sideband signals (there are 8 pins for that), because the expectation is that the device at the end of a 8088 cable is a storage enclosure type of device, which has one of the (SES/SES-2/SGPIO etc) enclosure management protocols built-in. No dedicated pins are required for that.
Thanks, I appreciate the enlightenment; iit helps flesh out my understanding of SAS & expanders. [On topic, it also might explain something else ... see my following post.]

In spelling out the 36-pin & 28-pin, my intent was to guide the OP to realize that, even if he acquired a second set (of 4) "special" cables, it was likely still a no-go for his external RenderBox.
 

UhClem

Member
Jun 26, 2012
88
34
18
NH, USA
The devil really is in the details! This really does scupper all the plans I had for this enclosure. Though I don't think it scraps them entirely. I'll just have to rethink having them as two independent boxes.
Or, **MAYBE** just rethink a few details about those two boxes. [Basic premise: we stick to standard cables/connectors.] This is pure conjecture, so treat accordingly ... At minimum, you want PCIe Gen3 x4, but you might want to grow into x8, right? [I hear that signal specs on PCIe Gen4 are significantly more stringent than Gen3, so that may be a different/later design.] So, instead of sff-8087 as a "source" connector, let's consider sff-8643. It is commonly used for U.2/NVMe (usually with a sff-8639 "target" connector). But your goal is to go external. [see edit in following post!] Now -8643 has -8644 as its external counterpart, but just like -8088, it's missing some pins. What I don't know (yet) is whether those pins are "needed"--I haven't looked at the pinout specs. But, consider that PCIe x4 only needs 21 pins (for signal; the first 11 pins are power, which will be supplied at the final target point [the slot]). Iff all of those 21 pins from a sff-8643, in use as a NVMe conveyance, are connected to the -8644, then very cool! If not, there may be a newer external sff spec that is also used , in practice, with -8643, that does convey the required pins. So, don't give up too soon--never underestimate the future value of modularity. [cf: Unix / pipes]
UhClem you'll have to forgive me as I'm still pretty green to all this, when it says custom pinouts should not be replaced by a standard cable, in the instance of adapting why does this matter if the end cable is also one of the originals provided by the company? Like say for example, I were to buy a pair of 8087 couplers put a standard 8087 in-between and the maxcloudon cables on either side. Would that effectively convert the pinout appropriately so the right wires are all connected?
[You think you're green? Hah! [I'm fossilized!] I retired in the late 90's and stopped being hardcore in the late 70's [swtch()'d from Kernel to User]. But, strangely ironic, regarding your primal subject here--3D rendering--I passed up the opportunity to be involved (as the Unix kernel specialist) at the very cutting edge of that--when I turned down a job offer from Ed Catmull in summer of 1975. Se la vie!]

Again, I haven't looked at the specs (and I'm not sure I would grok the pertinent details--I'm a software type), but I wouldn't be surprised if those extra 8 conductors, being only used for sideband purposes, were not spec'd to the same standard that the "real" PCIe signal conductors are. However, if you used the couplers without the "standard 8087" it shoudl/could be effective. Seek counsel from a hardware type.
Murphy needs to quit taking bites of my pie because I want to have it and eat it too!
I think (hope!) you only caught half my drift. You certainly got the Murphy part, but my use of the word "your" (apples) was in the "most personal" possessive, sir. If you DID get the full drift, well, all I can say is: "Just coffee for me--no dessert!" :)
 
Last edited:

UhClem

Member
Jun 26, 2012
88
34
18
NH, USA
...Now -8643 has -8644 as its external counterpart, but just like -8088, it's missing some pins. What I don't know (yet) is whether those pins are "needed"--I haven't looked at the pinout specs. But, consider that PCIe x4 only needs 21 pins (for signal; the first 11 pins are power, which will be supplied at the final target point [the slot]). Iff all of those 21 pins from a sff-8643, in use as a NVMe conveyance, are connected to the -8644, then very cool! If not, ...
Mea culpa!! The above should be replaced with:
Now -8643 has -8644 as its external counterpart, but unlike -8088, sff-8644 is NOT missing any pins. It is a perfect external counterpart to -8643. Ergo very cool.

@Ryan Knightley , note that there are bifurcating x16-to-4x-of-x4 pcie gen3 cards using sff-8643 connectors. There are pcie brackets with 2x8643-to-2x8644 connectors, and there are pcie gen3 x4 risers (w x16 phy slot) with a sff-8643 (+ power) inputs. Plus, of course, the obvious, and standard, cables. Do you want to walk the walk?

And, no scare stories about:
Our connectors are custom and not exactly "sff-8087". That is why experimenting with other cables can damage the risers and your hardware.
 

Ryan Knightley

New Member
Jul 15, 2017
7
0
1
29
@UhClem Just read through your responses! Really appreciate you taking the time to dig through all this!

I did some quick Googling to see what I could find in regards to the risers you mentioned in your latest post, I managed to find these that support bifurcation via the SFF-8643 36 pin connector, so that would (in theory) work for the motherboard side. I also found this adapter bracket to go to 8644 external from 8643 internal. I did however run into a snag though when hunting around for a riser for the cards to be plugged into. I found what might have been the right riser save the fact that it would not have appropriate power delivery. As far as I know, the spec for the power to be delivered from the slot is around 70W from the slot and pushing that kind of power over sata is a no go as it's a fire hazard. Granted that info was found on a mining subreddit so I'm not sure if it also applies to rendering (I'd assume so though, wattage is wattage regardless) Unless of course I've missed something on how the power delivery works with the SFF connectors and some power is distributed through that while the rest is supplemented through the sata connector? The Maxcloudon riser has a dedicated 6pin power connector for this reason. I'd love to look through whatever product links you've got! I'm no stranger to odd hardware most certainly not being used for it's original intended purpose :D

I think (hope!) you only caught half my drift. You certainly got the Murphy part, but my use of the word "your" (apples) was in the "most personal" possessive, sir. If you DID get the full drift, well, all I can say is: "Just coffee for me--no dessert!" :)
I'll admit I think I only caught the first half there! Now that you've said that, I've tried figuring it out and all I came up with was a post from Lost in Suburbia, which I think can apply in this situation but I'm not sure. Either way, now I want a chocolate lava cake.

[You think you're green? Hah! [I'm fossilized!] I retired in the late 90's and stopped being hardcore in the late 70's [swtch()'d from Kernel to User]. But, strangely ironic, regarding your primal subject here--3D rendering--I passed up the opportunity to be involved (as the Unix kernel specialist) at the very cutting edge of that--when I turned down a job offer from Ed Catmull in summer of 1975. Se la vie!]
Very green in comparison, possibly not even green, more like a newly germinated beansprout. I can't even imagine getting a job offer from a giant such as Ed Catmul! I thought it was cool enough being at the same school as Lasseter's kid hehe, think I met him like once at a party. I completed my degree in VFX with a concentration in Crowd/FX simulation and a minor in Technical Direction at SCAD. Though since graduating the majority of the work I've done has been in Light/Shading/Rendering and figuring out how to build the best possible machines for VFX work. What got me into the hardware side so heavily was actually tinkering with infiniband and quad nvme raid setups.
 

UhClem

Member
Jun 26, 2012
88
34
18
NH, USA
I did however run into a snag though when hunting around for a riser for the cards to be plugged into. I found what might have been the right riser save the fact that it would not have appropriate power delivery. As far as I know, the spec for the power to be delivered from the slot is around 70W from the slot and pushing that kind of power over sata is a no go as it's a fire hazard.
This time, I hope that the devil angel is in the details: The riser I found has a Sata power connector on it. But that doesn't mean that you must use a Sata power cable into the connector. As long as the 12V edge pin(s) on the connector, and the PCB trace(s) to the slot, can handle the (max spec) 5.5A, and you are able to feed the connector those 66W (max draw) (and the 3.3V/3A) appropriately, I'd think you'd be fine. Again, though, seek counsel from a hardware type. (For this project, I recommend that you cultivate a rapport with a HW heavy at your firm [BAH, right?] .)
...I managed to find these that support bifurcation via the SFF-8643 36 pin connector, so that would (in theory) work for the motherboard side.
I'll PM you a better deal on a similar card. (Supply is low, so best to minimize the chance you'd get sniped.)
I also found this adapter bracket to go to 8644 external from 8643 internal.
Looks fine, and a better deal than I found! It's too bad that nobody seems to make a 4x bracket/adapter.
I'll admit I think I only caught the first half there! Now that you've said that, I've tried figuring it out ... but I'm not sure. Either way, now I want a chocolate lava cake.
There's nothing at all wrong with apple pie, but, as much as I support free will, I really hope you won't use your apples.
... I can't even imagine getting a job offer from a giant such as Ed Catmul!
To fully grasp that (and my decision), you need to put it in its chronological context. #5 :rolleyes: Like I said, Se la vie.
 

Ryan Knightley

New Member
Jul 15, 2017
7
0
1
29
@UhClem Thanks! And unfortunately, I'm not at BAH anymore, really need to update that on LinkedIn at some point. I jumped ship over to a green architectural engineering firm right before the pandemic was in full swing. So now my resources in that department are a bit more limited, but overall the work is more interesting heh. I've got a few more technical friends over at Pixar that I could probably nudge and and see if they have any contacts over in the Bay Area that might have some experience in hardware. That's interesting to know about the edge pins, makes me wonder why the mining risers don't just take advantage of a connector like that and instead use usb. I suppose it's cheaper and they get make back a higher margin that way.

And thanks for the PM, I'll check that in a sec, my wife has given up on trying to keep track of the revolving door of hardware I've been having come in and out lately with this project! She rolled her eyes at me when I got the router bit attachment for the dremel and a "face shield" from home depot so I could mock up some mods before buying the thicker acrylic/aluminum and tapping kits needed from harbour freight. I'm quite excited to see if I can make something like this work as I've seen some amazing systems put together that house 8-12 monster NVIDIA GPUs. I've got this longterm goal in mind of at some point seeing if I can build a GeoExchange loop and silently cool my growing home lab by dumping all the heat back into the earth rather than having an ear shattering number of fans in the server room.

Looks fine, and a better deal than I found! It's too bad that nobody seems to make a 4x bracket/adapter.
I did find I think one, but it was something in the neighborhood of $400 so that was a hard pass.

There's nothing at all wrong with apple pie, but, as much as I support free will, I really hope you won't use your apples.
Ah I was wondering if that was what you were referring to but I wasn't sure with the desserts reference! Guess I had it right at first!