4090 blocking PCIE slots

[techfan32]

Hi All,

I am planning a dual 4090 GPU build and am seeking advice about not blocking PCIE slots with these large cards. I would like to use a Supermicro H13SSL-NT motherboard in a Sliger CX4728 chassis.

I was considering the following riser cables in the hopes that they can fit underneath the 4090s to make use of the blocked slots:

https://www.amazon.com.au/Riser-Extender-Reverse-Vertical-Gaming/dp/B0CC9B5WBL/ref=mp_s_a_1_4?crid=ULBBV84L6YBJ&dib=eyJ2IjoiMSJ9.20Jg-DMhqAYvkkwXNq6jawBVINFqFVOlvLyxhANkl-5bHKrCpWSwKRaophVfzqw6mydGYf01KYDwb7Qg7GXYVHNuCqlWEXAkNaig1roYeBcgk9BfvTV1HH7qA93AE3ZmionhZ54Sis5pfLvUDQR_EldIHDxeCEFcEzblQNLpMGuf593frWC-9fwjjzRM7Qprvxuu_CdFx_vNIJ4KIk_j0Q.8tv7rOUCtIjTh571oxwTBWZoFN32wF6K23ukVsQfP-g&dib_tag=se&keywords=pcie+4+extension+cable+thin&qid=1715504751&sprefix=pcie+4+extension+cable+%2Caps%2C757&sr=8-4

But I’m not sure if that is feasible or if it would cause performance/thermal issues.

Has anyone else encountered this problem or have suggestions?

Thanks a lot for your help!

 

[Chriggel]

I mean, sure, this is a common problem.

Two options:
1. Move the GPUs with riser cables
2. Route the remaining PCIe slots to other locations (what you want do to)

Both come with their own problems and by looking at that case, probably neither is going to work. Even if you can fit these under the GPUs, which is probably already unlikely, where do you want to go with them?

These risers appear to be 180°, so the mainboard will immediately be in your way if you're not using them on the last slot on the board's edge. Depending on the amount of slots that are blocked and you still want to use, you'd need several risers. If you move the GPUs, you only need two, but it's the same problem. There's no place to go where you can fit the GPUs instead.

There are two solutions I can think of at the moment, but neither includes that case.

A big tower case would allow for more flexible and custom mounting positions of the GPUs or other PCIe cards. Having one GPU hanging over the edge of the board is also popular.
For a 19" solution, server vendors have cases specifically for GPUs that still allow using other PCIe slots, for example with front mounted GPU instead of drive cages.

 

[techfan32]

Thanks for your feedback, looks like this is a harder problem than I was hoping... I have already purchased the chassis so am a bit stuck with it unfortunately. By getting the deeper 4728 version, I hoped there would be more options to work with.

I was considering mounting a 4090 on each of the first and last slots. This should hopefully leave the middle slot available, and then I want to find a riser cable that can liberate the two blocked slots (one per card). These blocked slots would be used for SAS and NVME expander cards and require no external connections. In that case, is it possible that I can mount these cards somewhere internally (eg. even with some custom mount/3D printed thing so they don't electrically short out) since they don't need external cable access and their thermal requirements should not be too extreme?

Another idea is to route at least one of the 4090s with a riser cable and find somewhere custom to mount that in the case, thus only blocking one slot at least. I have heard vertical mounting of GPUs is possible, for example, but am not sure if this makes sense in my chassis?

 

[Chriggel]

Thanks for your feedback, looks like this is a harder problem than I was hoping... I have already purchased the chassis so am a bit stuck with it unfortunately. By getting the deeper 4728 version, I hoped there would be more options to work with.

Since I was unfamiliar with it, I googled it and the 4728 appears to be a lightly longer version of the 4712. It didn't look promising in my opinion, but actually trying to make it fit with the actual case in front of you will give you a much better perspective. Maybe you can work something out.

In that case, is it possible that I can mount these cards somewhere internally (eg. even with some custom mount/3D printed thing so they don't electrically short out) since they don't need external cable access and their thermal requirements should not be too extreme?

Could be possible. Again, since you're going to have that case in front of you, you have the best overview of the situation. From experience and by looking at it, I'd say no. But that shouldn't discourage you, there's no substitute for sticking your head in and look around in the live thing.

Another idea is to route at least one of the 4090s with a riser cable and find somewhere custom to mount that in the case, thus only blocking one slot at least. I have heard vertical mounting of GPUs is possible, for example, but am not sure if this makes sense in my chassis?

Vertical mounting means the GPU is parallel to the board), instead of the usual horizontal position (90° angled slotted into the board). This is usually done for style, not for functionality. Some tower cases allow for a half style, half functional vertical mounting. For example, the Fractal Design Meshify 2 XL allows for vertical mounting of a 3-slot GPU while still allowing full height PCIe cards to be put in behind it. Another case, I think from Lian Li, allows vertically mounting the GPU in a completely different position, not overlapping with the board at all. But for most cases, it actually means mounting the GPU as the only PCIe card, with a riser, slap bang in the middle of the PCIe slot area, effectively blocking ALL slots. But this is all academic anyway, as it doesn't apply in your case (pun intended).

In some 19" cases, functional vertical mounting of GPUs is actually the default, because GPUs don't fit upright in 2U cases, for example. You can usually get about 4 2-slot GPUs in an industry standard 19" 2U server from Dell and the other usual suspects. Supermicro might have dedicated GPU cases for standard form factor components that do the same thing. But again, nothing of that is of value to you since you're already getting that case.

Two additional options I can think of:
1. 1-slot water blocks for the GPU, but that means you need to fit a pump, a reservior and radiators. Some of it might have to be external.
2. Talking about external, external GPU cases.

I guess none of that is really what you imagined with this build.

 

[techfan32]

Since I was unfamiliar with it, I googled it and the 4728 appears to be a lightly longer version of the 4712. It didn't look promising in my opinion, but actually trying to make it fit with the actual case in front of you will give you a much better perspective. Maybe you can work something out.

Could be possible. Again, since you're going to have that case in front of you, you have the best overview of the situation. From experience and by looking at it, I'd say no. But that shouldn't discourage you, there's no substitute for sticking your head in and look around in the live thing.

Yeah, it does not look too straightforward and, unfortunately, this is my first build, so I am not so familiar with just how this would work... Attached is a photo of the inside of the case with the dimensions of the usable mounting space for reference in case you have any ideas!

My hope is that I could fit the NVME and SAS cards (which need no external connections) somewhere in that region, or even in two of the eight card mounting slots on the front of the chassis. I guess this will depend on having good PCIE riser cables that can fit under the 4090s, are flexible enough, and coming up with a mount that won't electrically short the cards. Are there any standard parts that people would use for this sort of thing that might make it simpler?

Vertical mounting means the GPU is parallel to the board), instead of the usual horizontal position (90° angled slotted into the board). This is usually done for style, not for functionality. Some tower cases allow for a half style, half functional vertical mounting. For example, the Fractal Design Meshify 2 XL allows for vertical mounting of a 3-slot GPU while still allowing full height PCIe cards to be put in behind it. Another case, I think from Lian Li, allows vertically mounting the GPU in a completely different position, not overlapping with the board at all. But for most cases, it actually means mounting the GPU as the only PCIe card, with a riser, slap bang in the middle of the PCIe slot area, effectively blocking ALL slots. But this is all academic anyway, as it doesn't apply in your case (pun intended).

Interesting, thanks for the explanation! I will seriously be considering these tower cases in future. For now, I am just hoping to find some non ideal approach that will at least get me up and running... I have not yet bought the 4090s and motherboard so I can't even put them in place to work out how things will look which is a challenge.

Two additional options I can think of:
1. 1-slot water blocks for the GPU, but that means you need to fit a pump, a reservior and radiators. Some of it might have to be external.
2. Talking about external, external GPU cases.

I guess none of that is really what you imagined with this build.

Yeah, the water cooling does seem like it might be the way forward. I am really trying to avoid this to avoid potential points of failure and maintenance (as unlikely as that probably is, it still concerns me). But it may well be necessary if there just isn't a way of mounting these cards...

I have not considered the external GPU option so will look into this more. But again, my preference is just to get this working within the current chassis somehow...

Thanks for your ideas!

 

[Chriggel]

Oh, you see, this is only something you see on the real thing (or good pictures). I also didn't look at the the given dimensions on their website. This case is much taller than I first thought. You can put the GPUs (maybe) or the other PCIe cards above the full height PCIe slots, much like vertical mounting on a desktop case.



You "only" need to figure out clearance to the top cover, especially if you put the GPUs there and if their fans want to pull in air from that direction. Same if you put in other cards, you need to make sure that they get access to fresh air. You'll need some sort of support structure to hold the cards there, but that shouldn't be too difficult.

 

[techfan32]

That's great to hear!!

The question then becomes how to proceed next. Do you have a sense from this photo as to whether mounting the GPUs up there or mounting the NVME and SAS cards up there makes more sense?

My gut feeling is that mounting the SAS and NVME cards would be easier. It would also allow access to the video outputs of the GPUs in the usual way if they are not hidden up there (not that this is a major concern for my ML setup, just a nice to have).

I am using one of these NVME expanders:

PCIE 4.0 to 4x U.2 Expansion Card SFF-8639 SFF-8643 NVME PCIE X16 SSD Adapter | eBay

This product can only be used on PCIE slots that support PCIe split. Slot: PCIE X4 X8 X16. Expansion Slot: SFF-8643. This product is not plug and for Need machine support, BIOS settings, kind of complicated!

www.ebay.com.au

and one of these SAS expanders:

Fujitsu 9300-8I LSI SAS3008 12G HBA IT Mode ZFS FreeNAS UnRAID 2*SFF-8643 Cable | eBay

We will do our best and help you provide the solution. ※ All faulty due to improper installation or improper use will NOT be covered by the warranty.

www.ebay.com.au

They do not seem to need active cooling so hopefully locating them up there won't be too bad.

It just then comes back to the challenge of finding a suitable PCIE riser cable for them that can fit under a 4090. Would the part that I initially proposed make sense based on this photo? Or perhaps another type of cable?

 

[Chriggel]

The SAS HBA absolutely needs at least some airflow. They only have passive heatsinks, but are meant for well ventilated cases.

I also think that mounting the SAS HBA and the U.2 adapter there will be easier, but mounting the GPUs there makes more sense. I don't think that you'll be able to pull PCIe risers out from under the GPUs. And even if you manage to do that, they'll come up in front of the fans and block airflow. That's not good.

I can't tell by the photo if the GPUs will be a good fit for that space, it also depends on the GPUs you'll be getting. Measure the clearance and compare them to the dimensions of the GPUs. You could also mount the GPUs upside down with the fans facing the board, so that they will have it easier at pulling in fresh air coming from the center case fans.

To use the video outputs, well, you need to make a cutout. Probably not what you want. But this whole operation has strong Frankenstein vibes anyway, so maybe it's an option.

 

[techfan32]

There are reports online of certain riser cables fitting under 4090s, but your point about blocking the airflow is a good one and I would like to avoid this.

I measured the height of the space that you marked on your sub photo and it is about 50 mm. I have not yet bought the 4090s, so can go with any model. But from what I can tell, the smallest are closer to 60 mm eg. Founders Edition. So I guess that won't work... from the bottom of the case to the top it is about 170 mm, but it's not clear where to mount them still.

I am now considering a riser cable (eg. one like that in my initial post) for a 4090 on the far left (relative to the photo I posted) slot of the motherboard and routing it into the far left slot on the chassis (of the 8 available slots). The other 4090 could go on the far right slot (blocking one slot) but leaving 2 slots for my SAS and NVME cards. Does this seem feasible from what you can see? Without having the motherboard dimensions I can't measure for sure if this would work, but to my limited understanding it makes sense...

 

[UhClem]

... but leaving 2 slots for my SAS and NVME cards. ...

Are you ignoring the 3 MCIO (x8) connectors for a reason?

 

[Chriggel]

Oh boy, I haven't seen them either. Supermicro has listed them neither under "Key Features" nor "Expansion Slots". They're under "I/O" and all the way at the bottom.

 

[techfan32]

Good observation, thanks! I recall seeing them a while back and since I am not familiar with the connection type, I was just leaving them for future expansion. I got the impression they were for connecting NVME drives. But is there such a thing as an MCIO to PCIE adapter, that I could then use to connect the NVME x4 expander and SAS x8 expander cards to?

This does still mean that multiple of my PCIE lanes would be blocked by the GPU's which is disappointing. I assume my previously outlined plan is not feasible/preferable still for some reason?

If this is the best path forward then I am prepared to make this compromise. Let me know what you reckon!

 

[tinfoil3d]

Just throwing this in as a general idea, you may be able to use m.2 or any other pcie slots and convert them to a traditional pcie for your nvme(usually needs 4 lanes) and SAS(if you're willing to give up half of bandwidth and run it with 4 out of 8 lanes).
You may want to consider these as a workaround solutions to install nvme and hba.

 

[tinfoil3d]

Not rly shilling aliexpress but amazon is pretty much aliexpress just more expensive. And on ali there's a ton of various cheap converters like these A/E to nvme key cards, m.2 to 8643, m.2 ribbon cables for extending, oculink, and whatnot, there's a ton of things you can swap around to get your pcie lanes from wherever you can technically get them. I was happy to upgrade all my TMMs with 2230 nvmes in their wifi slots(i dont use wifi). For something more solid like when you truly need pcie 4 i'd recommend going witha known brands, but for pcie 3, those are way less sensitive to signal loss so something short and minor like adaptor card would do the trick almost all the time.

 

[reasonsandreasons]

The MCIO should mean that you don't need to bother with the NVMe expander at all--you can just buy a cable like this and hook two U.2 drives up to each of your MCIO ports. If you need SFF-8643--for a hot swap bay, for example--there are other cables on Aliexpress that will do that.

On the SAS expander, are you actually using SAS drives? One of the board's MCIO connectors should be able to be used as an additional 8 SATA ports; just buy a cable like this.

 

[techfan32]

Thanks for all the suggestions!

Using these MCIO ports does seem like the best solution for now. I hope that I can work out a way to access the PCIE slots in future somehow since it is quite a waste to have them blocked, but this is enough for me to proceed in the short term.

Yes, I only need to connect to SATA not SAS drives. So I will use your suggested cable:

https://www.aliexpress.us/item/3256806451230723.html?spm=a2g0o.productlist.main.1.18725ae2Lh9lZQ&algo_pvid=a0fb63ba-2db6-4054-8409-f9594eb6098a&algo_exp_id=a0fb63ba-2db6-4054-8409-f9594eb6098a-0&pdp_npi=4%40dis%21USD%2132.53%2126.03%21%21%21234.22%21187.42%21%402101ea7117157868842121359e52a3%2112000037890774916%21sea%21US%210%21AB&curPageLogUid=Zf2SYAi3GhCM&utparam-url=scene%3Asearch%7Cquery_from%3A&gatewayAdapt=4itemAdapt

I will be connecting the 4x NVME drives to an Icydock MB699VP-B. In that case I am thinking I will use something like this:

‌MCIO x8 74 Pin ( NVMe) to 2 X Mini SAS SFF-8643 Cable - 50 CM 611851926779 | eBay

Find many great new & used options and get the best deals for ‌MCIO x8 74 Pin ( NVMe) to 2 X Mini SAS SFF-8643 Cable - 50 CM at the best online prices at eBay!

www.ebay.com.au

Are there any speed/reliability implications for using these MCIO cables and adapters, as compared with the PCIE slots and expansion cards?

 

[UhClem]

... I will be connecting the 4x NVME drives to an Icydock MB699VP-B. In that case I am thinking I will use something like this:
(a MCIO x8 to 2xSFF-8643 cable)

That enclosure (the initial version; w/8643 connectors) was/is a connectivity nightmare. Icydock really screwed up! [you've been warned: [disclaimer: which does not obligate me to provide you the gory details, which might then require me to educate you on the specifics, etc. etc. ...]]

Separately, fyi, if your 4090s are the standard triple-slot-width, only one can have an x16 slot (and, hence, full bandwidth) on the mobo you've mentioned.

 

[reasonsandreasons]

Are there any speed/reliability implications for using these MCIO cables and adapters, as compared with the PCIE slots and expansion cards?

For the SATA I wouldn't expect any issues; it's basically just a way of increasing port density. It's possible you might get a bum cable, of course, but that's not unique to MCIO.

NVMe might be a bit dicier--what PCIe gen are your drives? My impression is that basically all well-made cabling solutions will do gen 3, some will do gen 4, and very few non-OEM solutions will do gen 5 yet.

 

[techfan32]

That enclosure (the initial version; w/8643 connectors) was/is a connectivity nightmare. Icydock really screwed up! [you've been warned: [disclaimer: which does not obligate me to provide you the gory details, which might then require me to educate you on the specifics, etc. etc. ...]]

That is ominous… I have already bought it and plugged the drives in, and I do note that some of them felt weird to push in and I’m not certain they are seated correctly despite a few attempts. I have not powered it on yet.

if you are willing to elaborate/explain your comment I’d be very interested to hear. I guess worse case I just have to get rid of it, assuming it won’t cause damage to other components?

Separately, fyi, if your 4090s are the standard triple-slot-width, only one can have an x16 slot (and, hence, full bandwidth) on the mobo you've mentioned.

I have not bought any 4090s yet, so I can be flexible on that, but yes the smallest I have found are 3 slot width. I do not know how they sit relative to the card slots, but was hoping that they would fit in the first and last slot which I thought were both x16…

If that is not the case, is it such that x8 PCIE 5 is about the same as x16 PCIE 4? Which to my understanding, the 4090 cannot saturate anyway? So this at least should not be an issue for me?

NVMe might be a bit dicier--what PCIe gen are your drives? My impression is that basically all well-made cabling solutions will do gen 3, some will do gen 4, and very few non-OEM solutions will do gen 5 yet.

I’m using four Samsung PM9A3 drives. These should be PCIE 4. Do you happen to know which cables would be suitable for these PCIE 4 drives?

 

[techfan32]

NVMe might be a bit dicier--what PCIe gen are your drives? My impression is that basically all well-made cabling solutions will do gen 3, some will do gen 4, and very few non-OEM solutions will do gen 5 yet.

I have not been able to find cables that specifically say they support my PCIE 4 drives. So I guess I will just give the ones I posted above a try and see how I go.