SXM2 over PCIe

[CyklonDX]

then you didn't read correctly.

This setup doesn't require special motherboard with nvlink on it - like with most sxm supporting servers.

What i described is as follows:
We had issues with those ports failing... and we were able to connect them onto normal pcie bus as on the picture... (red failing, and we were able to use raisers to connect those pcie ports there on green...)

The oculink/s, and bridge/s were always required ~ and they are connected by standard in any of those servers using this board; It just they do not have to be a special pcie at all... and potentially all motherboard with enough pcie lanes can connect that AOM-SXM board.


and it does work (i blacked out any identifiers)

 

[gsrcrxsi]

maybe something is getting lost in translation.

since the red ports are physically a proprietary size. how exactly did you connect the bridge connector (the red side on the AOM-SXMV) to the green ports? I'm asking how you yourself did it. not the other guy who used the RSC-G-6 riser. you said you used a "long riser", but what riser did you use? where did you get it?

but the pic you posted supports my point about GPU impingement. not sure he can install a GPU in the 3rd GPU slot because to put another RSC-G-6 in the other port would block that GPU. maybe not even able to put a GPU in GPU1 either for the same issue.

 

[CyklonDX]

In our servers? We custom ordered pcie extension cable for that sc port, and unplugged raiser from one of the ports (green) that we could as its the same pcie connector as in red.

by long raiser i meant (cable) ~ as such (note this std pcie) img just an example


There's nothing special on that long proprietary port from sc. Its normal pcie just soldered in a bit opposite way... so you can have male-male long sc pcie extender cable/raiser to normal pcie without any real issue.


The whole point is to make this work on non-specific hardware/motherboard - it was intended to work with. This is as far as i know only single board that can be connected to non-vendor specific mobo as it doesn't have nvlink on the motherboard itself - but on sxm board.

 

[gsrcrxsi]

it's normal PCIe "electrically" but physically it is not standard. a normal PCIe riser will not fit.

look at what the RSC-G-6 adapter looks like:

you can clearly see the physical difference in both size and pin layout between the "sc port" fingers and the PCIe slot it adapts to.

that's why I was asking for a source on the cable. but if you had to have it custom made, that seems like a road block for most people.

btw, what do you mean by "sc port"? is that the actual name for this connector type?

 

[CyklonDX]

We ordered 50 or so propitiatory very long male-to-male sc pcie shielded cables from ebay seller - we initialized conversation with few ebay sellers that were selling their own raisers, and stuff as such for mining rigs (one of them obliged us). Still this is beyond the point what we used in servers... if you had same class server you could just connect it normal way unless 2 ports failed for you too after warranty.


Getting a raiser as such, and normal male-to-male pcie extension cable is within most people... raiser is like 50 usd, and cable is like 20-50 usd.
Any pro electrical shop that deals with cables/mining rigs will be able to take that raiser rsc-g-6 and make you raiser cable you want.

 

[gsrcrxsi]

so you can have male-male long sc pcie extender cable/raiser to normal pcie without any real issue.

this is the part I'm trying to clarify. if you know of a specific cable that can connect the "sc port" from the AOM-SXMV to a normal PCIe slot, I'd like to hear it. it seems that the RSC-G-6 is the only way to achieve this without a custom order. is that right?

but the RSC-G-6 seems to have physical interference if you try to use all 4 SXM GPUs. that's the problem I see.

 

[CyklonDX]

is that right?

yes, thats right.

but the RSC-G-6 seems to have physical interference if you try to use all 4 SXM GPUs. that's the problem I see.

There are few other raisers from sc with this sc pcie port to normal pcie; i haven't looked personally for them, but they should exist somewhere for purchase -> might be rare tho. (quite few sc servers use this sc pcie port of different form factor for raisers)

 

[gsrcrxsi]

so if you just want this board to run 4x V100s as compute cards without the special network access, the oculink cable ports are not needed at all?

I was looking to try this out sometime because I was under the impression that I could connect them via ONLY the oculink ports and still use them for compute. having to get some special connector cable makes this much more difficult.

btw, what does "sc" mean in "sc port"? is that an acronym for something? knowing the right thing to search for will help tracking down the right cable that's needed.

 

[CyklonDX]

the oculink cable ports are not needed at all?

I've never tested without them; the servers come in with both oculinks and sxm bridges installed. (2 active raisers come with oculink ports)

that I could connect them via ONLY the oculink ports and still use them for compute.

I heard from a friend working with just dell hardware that their sxm boards only have oculinks (but they have dell drivers required before it sees the gpu's over them.) so likely you won't get anywhere - oculinks as i recall are used for GPUDirect RDMA.


(so i'm guessing its only useful if you utilize both PLX switches with 4x v100, and emulate it as single device with 2 or more cpu's or even network)

what does "sc" mean in "sc port"

sc as in supermicro

 

[line]

where heatsink at? | and you didn't connect your pcie port... (the lack of bridge pcie port won't prevent you from booting the system - but you will need it -ex. you can use this sc raiser SUPERMICRO RSC-G-6 to connect that pcie with your motherboard pcie)
View attachment 29325


the power connectors should be your 12V 8-pin GPU power supply connectors. (std pcie connectors)

I have a question, how do I connect the PCIE on the RSC-G-6 to the PCIE on the motherboard?

 

[line]

then you didn't read correctly.

This setup doesn't require special motherboard with nvlink on it - like with most sxm supporting servers.

What i described is as follows:
We had issues with those ports failing... and we were able to connect them onto normal pcie bus as on the picture... (red failing, and we were able to use raisers to connect those pcie ports there on green...)
View attachment 29328
The oculink/s, and bridge/s were always required ~ and they are connected by standard in any of those servers using this board; It just they do not have to be a special pcie at all... and potentially all motherboard with enough pcie lanes can connect that AOM-SXM board.


and it does work (i blacked out any identifiers)
View attachment 29329

Can I see how it is connected to the motherboard?? The picture is a bit small and hard to see. I would like to refer to it.

 

[gsrcrxsi]

You would need the RSC-G-6 (or maybe the RH version has better clearance?) and then you will need a male-to-male PCIe ribbon riser cable.

AOM-SXMV -> RSC-G-6 -> male-male riser -> motherboard PCIe slot.

 

[line]

You would need the RSC-G-6 (or maybe the RH version has better clearance?) and then you will need a male-to-male PCIe ribbon riser cable.

AOM-SXMV -> RSC-G-6 -> male-male riser -> motherboard PCIe slot.

Is only the RSC-G-6 card usable? I couldn't find it for sale on shopping websites. I would like to know if there are any other adapters that could be used instead.

 

[CyklonDX]

Here's couple (RSC-G-6's)

SUPERMICRO RSC-G-6 – RackmountNet

www.rackmountnet.com

Supermicro RSC-G-6 - Supermicro Riser Card - 1 x PCI Express 3.0 x16 PCI Express | eBay

Find many great new & used options and get the best deals for Supermicro RSC-G-6 - Supermicro Riser Card - 1 x PCI Express 3.0 x16 PCI Express at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[line]

Here's couple (RSC-G-6's)

SUPERMICRO RSC-G-6 – RackmountNet

www.rackmountnet.com

Supermicro RSC-G-6 - Supermicro Riser Card - 1 x PCI Express 3.0 x16 PCI Express | eBay

Find many great new & used options and get the best deals for Supermicro RSC-G-6 - Supermicro Riser Card - 1 x PCI Express 3.0 x16 PCI Express at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Can one RSC-G-6 work normally, or do I need to have two of them?

 

[line]

Here's couple (RSC-G-6's)

SUPERMICRO RSC-G-6 – RackmountNet

www.rackmountnet.com

Supermicro RSC-G-6 - Supermicro Riser Card - 1 x PCI Express 3.0 x16 PCI Express | eBay

Find many great new & used options and get the best deals for Supermicro RSC-G-6 - Supermicro Riser Card - 1 x PCI Express 3.0 x16 PCI Express at the best online prices at eBay! Free shipping for many products!

www.ebay.com

oh,no, The region option for www.rackmountnet.com does not include China. As for eBay, China is listed under the exclusion list.

 

[CyklonDX]

For china i would recommend you look at ali express, and your own internal markets; It should be much easier to get custom made parts there - maybe you have to ask sellers from shenzen (-> a decade ago, they were quite helpful making pcie raiser splitter with plx chip on it) maybe someone can make a cable for you without a raiser at all.

 

[line]

For china i would recommend you look at ali express, and your own internal markets; It should be much easier to get custom made parts there - maybe you have to ask sellers from shenzen (-> a decade ago, they were quite helpful making pcie raiser splitter with plx chip on it) maybe someone can make a cable for you without a raiser at all.

OK，Thanks a lot!

 

[CyklonDX]

while at it; some people may find this guy quite useful
(EU based - ships to US)

C-Payne PCB Design

C-Payne PCB Design

c-payne.com

He's not cheap, but does quality work. May be willing to do custom stuff. (but again he's not cheap)

 

[gsrcrxsi]

I did think about him while pondering where to get a custom cable made. I have some of his other risers.

I think the biggest problem will be communicating the exact spec of the custom PCIe slot. Not know what it’s actual spec name is will be hard for them to find what they need to be making an adapter for (physical connectors, pinout, etc). Same for Chinese suppliers. Outside of sending them an RSC-G-6 or something so they can figure it out themselves.

After they figure out the specifications of what they are actually making I think the best solution would be for someone to make an adapter card from the Supermicro custom port straight to an Oculink or similar cable connection. Then it would be easy for the end user to adapt that to a PCIe slot with whatever length cables they need.