AMD EPYC 7351P + GIGABYTE MZ31-AR0 = 7 GPUS?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Jan 12, 2017
89
6
8
Cambridge, UK
Hi Guys,

I just saw the review on the AMD EPYC 7351P CPU but it did not really go into GPU usage so I thought I would do a post in here to get some feedback and or advice on this.

We are looking to build a 7 GPU rack using the EPCY so we can take advantage of the 128 PCIE lanes, basically running 4 x16 Gen3 and 3 x8 Gen3 GPUs in total.

So far the mobo that seems to be able to do this is the GIGABYTE MZ31-AR0, and the EPYC 731P CPU seems to be the lowest spec, and have more than enough speed and cores/threads to run this fine.

Q) Am I right in thinking that if we were are only running one SATA HD and USB devices then we would have full use of the 128 PCIE lanes for all 7 PCIE slots?

Q) Is this the lowest price/spec EPYC processor for single socket usage?

Q) Am I correct in thinking we would need to run 8x memory sticks to get full oct-channel usage?

Q) When is this mobo going to have full release as it only seem available from a few places in the US and one or two suppliers in Germany?!

Thanks for any help on this.
 
Last edited:
  • Like
Reactions: Subsonic
Jan 12, 2017
89
6
8
Cambridge, UK
Thanks Subsonic I was not aware of that, or the 7251.

Right now we have stopped doing Folding@home (which requires minimum x8 databus for GPU) so we will probably put off the EPYC rack until more components/boards are released and the price drops abit.
 

Erick

New Member
Nov 16, 2017
28
20
3
Saint Paul, MN
Does Folding @ Home really need that much bus speed between the system and the cards? For example, many 7+ card crypto currency mining rigs use simple USB PCIX extenders as all the work is done on the card and doesn't require much bandwidth between the cards and the rest of the system.
 
Jan 12, 2017
89
6
8
Cambridge, UK
We run 4x 1080 Ti's and 5x 1080's onboard, and using x1 PCIE risers.

BOINC, and all the crypto currency mining that we have done has no problem with x1 Gen2 or even Gen3.

F@H on the other hand reduced PPD (points per day) even on x16 Gen2 which is pretty close to x8 Gen3. I asked about and knowledgeable people said the lowest you want to run is x8 Gen3, which seems to be backed up by what we found on Gen2.

In truth it depends on the algorithm being used, and how much it is using the processor and, the amount of interaction going on between the GPU, CPU, Memory, and Hardrive (etc). Also as other elements on the mobo also use PCIE lanes this would also have a factor.

This is why we intend to future proof for all data processing we do, be it F@H, BOINC, or Crypotmining, and only use EPYC GPU Racks. In our opinion it makes sense to have a setup that can never (any time soon) be bottlenecked due to the EPYC's CPUs 128 PCIE lanes. It just gives the flexibility to run 7 GPUs using only x8 and x16 Gen3 risers.
 
Last edited:

Cinegy

New Member
Feb 21, 2018
1
0
1
54
Hi Guys,

I just saw the review on the AMD EPYC 7351P CPU but it did not really go into GPU usage so I thought I would do a post in here to get some feedback and or advice on this.

We are looking to build a 7 GPU rack using the EPCY so we can take advantage of the 128 PCIE lanes, basically running 4 x16 Gen3 and 3 x8 Gen3 GPUs in total.

So far the mobo that seems to be able to do this is the GIGABYTE MZ31-AR0, and the EPYC 731P CPU seems to be the lowest spec, and have more than enough speed and cores/threads to run this fine.

Q) Am I right in thinking that if we were are only running one SATA HD and USB devices then we would have full use of the 128 PCIE lanes for all 7 PCIE slots?

Q) Is this the lowest price/spec EPYC processor for single socket usage?

Q) Am I correct in thinking we would need to run 8x memory sticks to get full oct-channel usage?

Q) When is this mobo going to have full release as it only seem available from a few places in the US and one or two suppliers in Germany?!

Thanks for any help on this.
A bit of a warning from my side. This motherboard has some serious GOTCHAs depending on what you want to do with it.

I bought this board as I thought - WOW - 128 PCIe lanes let's cram four GTX 1080 in there. The SuperMicro single processor boards where not available yet, so this was the one and only choice.

Then I received this MZ31-AR0 thing finally last week and what a let down.

Whoever designed this motherboard is an utter tool. Nice specs on paper but try to use it.

Of the theoretical 5x PCIe x16 slots you may have thought you could put a full size graphics card into only one physically fits.

ONE.

Just ONE.

That is not an issue of the case I have chosen, but the motherboard itself.

Three of the PCIe x16 slots are directly in line with the RAM sockets of the CPU. So unless you want to just plug in half-length PCIe x16 cards(?), and who would want to do that with a motherboard / CPU combination like this?

Leave the RAM out on that side of the CPU and halve the RAM channels used in sake of using the PCIe slots? Not an option as the sockets even without RAM modules installed are too tall to allow the installation of full or 3/4 length cards.

Leaves theoretically two PCIe x16 slots. BUT the very first one is placed close to the very edge of the motherboard and my SuperMicro case sits flush with the motherboard. No space for a two slot wide graphics card like a GTX1080. There is one single slot width GTX1070 I could buy to fit in there, but why bother.

On my desk I have the following cards I wanted to pile in there: 2x GTX1080, 1x Highpoint SSD7101A RAID controller or alternatively 1x ASUS Hyper M.2 X16 card. So far I managed to install exactly one GTX1080.

So the motherboard is useless unless you want to e.g. use loads of half length PCIe x16 cards - like small, castrated graphics cards - but for what? Dual 10G Ethernet is already on the motherboard and works fine.

So what to use this "masterpiece" of engineering for? I am just sorry for myself not having taken much more time to study the photos of the motherboard to have an epiphany thinking "WTF?".

I ran some benchmarks with the Epyc processor I can say: yes, nice performance, just a shame that this motherboard is USELESS for me. I bought it to use the abundance of 128x PCIe 3.0 lanes of which 88 are being exposed via PCIe connectors.

A different case would possible give me two usable PCIe x16 slots, but that is still a far way from where I wanted to be - have my cake and eat it -> dual GTX1080 and 2x 12 GB/s SSD performance.

I should have waited for the SuperMicro H11SSL-i which at least looking at the pictures that are now up has 3x fully usable PCIe 3.0 x16 slots. I would want more, but three in my math is more than one.

In the meantime until that arrives it is back to Intel. The ASUS WS C422 PRO/SE with a cheapo Xeon W processor gives me really 4x usable PCIe x16 slots even though when using all four two of them then would be just x8 electrically or alternatively use just three PCIE x16 at full x16.

To my astonishment none of the reviews I have read so far on the Gigabyte MZ31-AR0 has mentioned that the RAM / PCIe slot layout is quite "unusual" which is an euphemism for "demented".
 
Last edited:

Erick

New Member
Nov 16, 2017
28
20
3
Saint Paul, MN
I'm sorry you are having issues. I agree, it kinda sucks, but it is what it is and we all knew that going into it so I'm not too bummed. It's pretty clear on the product page the limitations for full length graphics cards. They even show where a full length, double-slot card would go. So you ask, what is this board for? Well it works well for many people, including me. In fact, it supports double the amount of memory as the Super Micro board you referenced, and that's why it has this particular layout. It's a high density solution for workloads that require lots of memory. It would be near impossible to fit the amount of ram slots + room for full length cards on to one E-ATX board... that's just one of the limitations of the form factor.

If you want more than 1 (or 3) double slot full length GPU's on one board without risers, you are looking at the wrong type of board all together. You need to look at a more custom, much larger, Super Micro or Tyan board that has a completely different layout. However; if you have some ingenuity and a large enough case (4u or more) you can make it work with a little bit of customizing and some short risers... not too hard to do. ;)
 
Jan 12, 2017
89
6
8
Cambridge, UK
Hi Cinegy,

Thanks for you your reply.

I completely agree with what you say; for a motherboard producer to use this CPU, and then put so many PCIE slots on the board, when infact modern GPUs cannot be used is pretty if not totally misleading.

I am gad you found a temp solution with x8 slots on the W Xeon.

I am not sure what you have done with the board but if you wanted to re-sell it do let me know as I am still after one of these, however probably only if you are based in UK/EU due to shipping and import costs.

Eric, for you this board is about RAM; however myself, the original OP is about GPUs, so for Cinegy to fully share his experience within this area is EXACTLY why I wrote this post. This is the ONLY EPYC board that we have seen with 7 GPU slots.

Whereas any more than 32GB of RAM to us is fairly irrelevant as we do not need anymore than this.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
Not that I'd want to go near Gigabyte server kit, but isn't it pretty obvious from the board layout that full-length graphics cards aren't going to fit...? Naturally the supermicro has the better layout in terms of PCIe slots

Most servers, at least the sort I deal with, will never see a graphics card in their life so I don't think outfitting a basic server with 5+ (presumably single-slot?) graphics cards is that common of a usage scenario, nor I think using top-end gaming graphics cards in a mid-range server. I'd echo Erick's comment that this is precisely the wrong sort of board for such endeavours. But then I'm not sure what the "right" board for 7x PCIe 8x GFX slots is...
 
Jan 12, 2017
89
6
8
Cambridge, UK
There are very few deccent GPUs in life that are not normal sized. And even if you used these, you would not put 7 of them in. So I do think my original assertion of this mobo being slightly misleading probably holds true.

The problem with SuperMicro is most of their mobos (with large numbers of PCIE slots) are proprietary sized which then causes issues when trying to find an non-SuperMicro chasis - which is probably why they do this.

Personally, as I soon as I saw the mobo I could see that normal sized GPUs would not fit, and even if they did they would hit it the RAM Slots. But we are running data processing servers and racks and hence look for things that perhaps others would not.

In truth I think that most people would assume that if a mobo has N slots, then they could use N modern graphics cards in it; which seems quite fair.

The reason why I really like this mobo is because it is PERFECT for data processing. It allows us to run 7x GPUS, 4 @ x16, and 3 @x18, all at Gen3. No other CPU can offer this amount of PCIE lanes, and so far no other mobo has this offering of slots. We need a minimum of x8/x16 at Gen3 for our requirements.

We run with Risers so spacing is not an issue for us, however it must have a form factor based on ATX eg E-ATX so that is very important as the GPU rack stand off screw holes are predrilled; which normally rules out SuperMico mobos.
 
Last edited:

vcandy

New Member
Mar 19, 2018
1
0
1
33
Anyone have a recommendation for a case and riser setup to utilise 7 GPUs with this board?
I thought of using cheap 200mm cable risers (x8-x16 and x16-x16), but they don't look like they would fit with any enclosed 4U cases.
 
Jan 12, 2017
89
6
8
Cambridge, UK
You might want to actually read the thread as that Supermicro server rack solution does not even come close.

Wrong CPU not enough PCIE lanes
Propertarory motherboard
FE GPUs
Server rack not Mining rig
Overpiced
Underspec'd
etc etc
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
I did read the thread. You are looking for a unicorn.

If you need proper rackmount chassis with 6+ double width GPU, it is not physically possible with standard form factor motherboards and chassis. These will have to be specialized solutions.

Open rigs, mining rigs etc are different and you can make it work with extenders, risers etc. But I didn't think you were looking for that and wanted a clean solution.

You can follow the link I posted and look through the various offerings available to see if anything suits you. I just posted an example of they type of system you are describing. It's up to you whether you want to follow up and do more research in that line. Supermicro has not yet released any AMD based GPU servers yet that I know of.

There were a couple of main site posts describing a 10-11 GPU setup with the SYS-4028GR-TR2 server. you should be able to find them searching for deeplearning10 and 11.
 
Jan 12, 2017
89
6
8
Cambridge, UK
Please stop:

(a) trolling as you clearly have not even read the title of this post, let alone my posts, as OP

(b) trying to Hijack this thread to 'why you should buy Supermicro'
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
I have no reason to do (a) or (b). ¯\_(ツ)_/¯

I’ll move off YOUR thread and go troll somewhere else
 

TomUK

Member
Aug 30, 2017
67
12
8
38
Might be worth looking at the SuperMicro H11SSL series, they can do 3 x16 and 3 x8 slots and have started to crop up in the UK/EU, still one x16 slot short of what you were after, but if your still considering Epyc it might be worth a look, board layout looks Ok for full length GPU's
 

Damien_MENIGAUX

New Member
Sep 12, 2018
8
0
1
Hi. I'm planning on using this card to do a 7 gpu deep learning rig. I didn't quite get if this is possible with those contradicting comments :s

Can I do that and assure all my cards use all the pcie leanes (x8 minimum for each card) with risers ? I was planning on using an open air mining rig that's flexible and doesn't obstrudes the pcie plugs of the gpus.
 

Erick

New Member
Nov 16, 2017
28
20
3
Saint Paul, MN
Hi. I'm planning on using this card to do a 7 gpu deep learning rig. I didn't quite get if this is possible with those contradicting comments :s

Can I do that and assure all my cards use all the pcie leanes (x8 minimum for each card) with risers ? I was planning on using an open air mining rig that's flexible and doesn't obstrudes the pcie plugs of the gpus.
Yes, it should be possible. I built a 4 x 1080ti deep learning rig a couple of months ago using the board. I used an EPYC 7251 as it was the lowest cost option.

Although there are five 16x slots on the board, only four of them are wired at 16x PCIe speeds. The other one is 8x. You'll have to get riser cables for all slots (not cheap!), including smaller 8x to 16x cables for the smaller slots.

Also, you will definitely want to plan on populating all 8 memory channels with DIMM's. I was running into issues with inconsistent PCIe speeds with only 4 of the 8 channels populated.

Erick