SXM2 over PCIe

[d450d112]

I would recommend getting at least the 2U copper colored heatsinks. the heatsinks you have now (pictured before) are 1U, but the airflow direction is transverse to the airflow in a normal chassis, because they are meant for 90deg perpendicular installed position relative to how it's installed on the AOM-SXMV. unless you put the board in sideways, it will likely have bad airflow.

I'm not very concerned with idle power consumption since I run it 24/7 for the most part. but from what i remember, the board and GPUs were idling around 140-150W from the wall depending on fan speed i set. 40-50 per GPU at idle sounds normal-ish. it could be a little elevated if the GPUs aren't dropping down to P8 state at idle.

I might have to give that a try. I installed the board into my 2U case, and currently, I have the server blower fans positioned perpendicular to the heatsink fins. Without load, each GPU runs at around 45-50°C, and under load, the temperature rises to about 80°C.

On a different note, I'm inventorying my lab and struggling to determine the value of my cluster. How would you appraise an AOM-SXMV + 4x V100 build? I feel that simply adding up the costs might undervalue it.

 

[gsrcrxsi]

Value? Hard to say. $1500?

 

[gsrcrxsi]

Got both my boards rack mounted now (had to use the shelf-type rail things). Yes you can use two of these boards on the same system. Not super unexpected, but I don’t think anyone else did it to confirm. Ignore the dusty rack

the setup is quite nice. Using two sets of 70mm PCIe extension cables (one end is specific to the AOM-SXMV custom port) that I got from one of the sellers on Xianyu. I would have preferred to use locally sourced PCIe cables and the AOM-SVMV-PIO adapters, but I couldn’t find any long enough that were stable and not ridiculously expensive. Both GPU boards are connected to an H11DSi-NT motherboard (2x 7742, watercooled). I made sure to plug the PCIe cables from both boards into the four PCIe ports closest to the CPUs, as all four of these are connected to CPU1. The 5th PCIe port is connected to CPU2 and I didn’t want any issues or latency from that (maybe won’t matter, but I feel better about it this way). This does mean that each board has two GPUs connected over just PCIe 3.0 x8, but that does not affect my personal use case. You could easily use another board like AsRock EPYCD8/ROMED8 with four full x16 links.

 

[RGL]

Now I have produced sxm2 pcie adapter cards and installed them in batches.

 

[mtg]

Now I have produced sxm2 pcie adapter cards and installed them in batches.
View attachment 38218View attachment 38219View attachment 38220
View attachment 38221View attachment 38222

Wow, those look great - are you selling these?

 

[RGL]

Wow, those look great - are you selling these?

If anyone wants to buy it. A set of adapter card weighs about 1.2kg.
Superbuy - The Best Taobao Agent Help You Shop,Shipping From China
https://h5.m.taobao.com/awp/core/detail.htm?ft=t&id=822328333239

 

[RGL]

I got a batch of sxm2 to pcie engineering adapter cards. The quality is not perfect but they can be used normally. The main thing is that the price is low and could be cheaper if the quantity is large. This is the link of Taobao and Xianyu, and Superbuy.
https://m.tb.cn/h.gmVHyLl5mWsg6Z9?tk=zZvF3Vs0bGq
https://m.tb.cn/h.gmVG0KE?tk=XvPZ3Vs1l9W
Superbuy - The Best Taobao Agent Help You Shop,Shipping From China
Superbuy - The Best Taobao Agent Help You Shop,Shipping From China

 

[phantasmagoria]

I got a batch of sxm2 to pcie engineering adapter cards. The quality is not perfect but they can be used normally. The main thing is that the price is low and could be cheaper if the quantity is large. This is the link of Taobao and Xianyu, and Superbuy.
https://m.tb.cn/h.gmVHyLl5mWsg6Z9?tk=zZvF3Vs0bGq
https://m.tb.cn/h.gmVG0KE?tk=XvPZ3Vs1l9W
Superbuy - The Best Taobao Agent Help You Shop,Shipping From China
Superbuy - The Best Taobao Agent Help You Shop,Shipping From China
View attachment 38439 View attachment 38440

View attachment 38441

"Engineering" - is that why they're so cheap? Anything in particular you notice with them? What fans are those, and did you have to buy the heatsink separately?

Might pick a few of these up... jeez

 

[gsrcrxsi]

"Engineering" - is that why they're so cheap? Anything in particular you notice with them? What fans are those, and did you have to buy the heatsink separately?

Might pick a few of these up... jeez

FYI, these are about $75/ea, not the $7/ea that shows on the superbuy link. The $7 is for JUST the metal back plate, not the adapter

 

[phantasmagoria]

FYI, these are about $75/ea, not the $7/ea that shows on the superbuy link. The $7 is for JUST the metal back plate, not the adapter

Thanks for the catch - not so familiar with Taobao/Superbuy.

Did some more looking around and it appears 22GB 2080tis can be had for about US$350 before shipping... I saw some people debating this earlier in the thread but I'll probably just rent out some systems from vast.ai to test my workloads in the event I want to expand. Right now 4xSXM2 P100s are more than enough, but the X10DGQ mobo has plenty of PCIe x16 slots to spare if I want to expand at some point.

 

[efschu3]

I got a batch of sxm2 to pcie engineering adapter cards. The quality is not perfect but they can be used normally. The main thing is that the price is low and could be cheaper if the quantity is large. This is the link of Taobao and Xianyu, and Superbuy.
https://m.tb.cn/h.gmVHyLl5mWsg6Z9?tk=zZvF3Vs0bGq
https://m.tb.cn/h.gmVG0KE?tk=XvPZ3Vs1l9W
Superbuy - The Best Taobao Agent Help You Shop,Shipping From China
Superbuy - The Best Taobao Agent Help You Shop,Shipping From China
View attachment 38439 View attachment 38440

View attachment 38441

Are they compatible with V100? If yes, some of you interested in a group buy? (I would be )

 

[RGL]

Are they compatible with V100? If yes, some of you interested in a group buy? (I would be )

Yes, all GPUs with sxm2 interface are compatible.

 

[mtg]

Are they compatible with V100? If yes, some of you interested in a group buy? (I would be )

Yeah I’d be interested in a few.

 

[gsrcrxsi]

I can probably run a group buy. But there will probably have to be substantial interest to make it worth it.

the seller only offered a 2% discount on a quantity of 10. So not really much savings in bulk order unless we’re ordering like 100 or 1000

 

[Snurt]

Heya. Just picked up an AOM-SXMV and 4xV100's.
Probably going to run my 6900xt from the M.2 CPU slot via a PCIe adapter, and plug the board into two x8 slots.

 

[frystal]

browsing XianYu again last night. I came across some 8x SXM2 boards. the listing didnt say what they were from or really any information about them. they had green PCB, 4x pcie x16 slots (grouped together on front/left side), 8x SXM2 spots, and 8x slimSAS-8i or similar connectors which I assume are the main PCie outputs from the board. with lots of 8-pin power on the back edge of the board. and 4x black heatsinks over the 4x PEX9797 bridges.

some quick googling looks like these are almost identical the Microsoft HGX-1 platform circa 2018-ish? there's a STH article about that looks identical : https://www.servethehome.com/microsoft-hgx-1-at-the-ai-hardware-summit/

anyone know if these GPU boards are essentially unlocked and could be used on a normal computer like the AOM-SXMV can with the appropriate PCIe adapter cables? the seller I messaged did not reply.

I got one. But i am still struggling to figure out how to make it work.

 

[mtg]

I got one. But i am still struggling to figure out how to make it work. View attachment 38790

This might be helpful - Project Olympus HGX-1 (PDF warning) From Server/ProjectOlympus - OpenCompute
They are designed as expansion chassis according to that wiki, so in theory they shouldn't need special sauce.

I think the SXM2 adapters we are seeing are based on the pinouts from the documents on this OCP project actually, the schematic has the SXM2 socket pinouts.

 

[gsrcrxsi]

I got one. But i am still struggling to figure out how to make it work. View attachment 38790

I think you’ll have to use the four flat oculink-like
Connectors just behind the four black heat sinks. And of course you’ll need to power the boards power connectors and attach GPUs

 

[frystal]

This might be helpful - Project Olympus HGX-1 (PDF warning) From Server/ProjectOlympus - OpenCompute
They are designed as expansion chassis according to that wiki, so in theory they shouldn't need special sauce.

I think the SXM2 adapters we are seeing are based on the pinouts from the documents on this OCP project actually, the schematic has the SXM2 socket pinouts.

That is the main problem. This board adopts a little bit different design compared with the HGX-1. It replaces the 2 oculinks with one SFF-8654-16i connector. I try to plug a SFF-8654-8i cable since it shares the same pin map as part of the SFF-8654-16i. However, I can not see any PLXs from lspci.

 

[mtg]

That is the main problem. This board adopts a little bit different design compared with the HGX-1. It replaces the 2 oculinks with one SFF-8654-16i connector. I try to plug a SFF-8654-8i cable since it shares the same pin map as part of the SFF-8654-16i. However, I can not see any PLXs from lspci.

Is it? It looks exactly the same: