DeepLearning12 NVLink

Patrick

Administrator
Staff member
Dec 21, 2010
11,908
4,871
113
Getting there. These things are quite unpleasant to install compared to CPUs. I have had members here, people I have spoken to over the past few weeks, and even someone walking through the data center today tell me not to overtighten because of how fragile these are. Doubly so for the heatsinks.
DeepLearning12 8x Tesla P100 GPUs in Gigabyte G481-S80.jpg
 

nitrobass24

Moderator
Dec 26, 2010
1,083
127
63
TX
This system supports the new Optane DIMMs right? Any chance you have some of those going into this build?
 

Revrnd

Member
Jan 2, 2018
30
2
8
AU
How's your Deep Learning 12 project coming along Patrick? Really looking forward to seeing how well it goes compared to the 1080Ti powered DL11.
 

Jaket

Member
Jan 4, 2017
76
15
8
Seattle, New York
purevoltage.com
The major benefits are that you save 15-20% on the initial installation per GPU and some on the ongoing costs since you are using more GPUs per chassis.

I am really interested in the build-out of that Gigabyte server. It is a dual root design so are you planning to use 2x Mellanox cards and avoid the NUMA penalty?

Hey Patrick,
Sorry for the long delay it took our supplier a while to get this system built out.
It's our first dual root design for we didn't go the 2x Mellanox route with it yet however it has room if needed.

As we've only ran single root systems in the past I've got a question for you.

Our system seems to only show 4 GPU's when running nvidia-smi topo -m

ubuntu@4U-GPU:~# nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 CPU Affinity
GPU0 X PIX PIX PIX 0-11,24-35
GPU1 PIX X PIX PIX 0-11,24-35
GPU2 PIX PIX X PIX 0-11,24-35
GPU3 PIX PIX PIX X 0-11,24-35

Not sure if there is anything extra special we need to do in order to show the other cards on the system.

Any help would be greatly appreciated as I couldn't find anything online about this while trying the Googler.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,908
4,871
113
@Jaket so you only have 4 GPUs and they are all connected to a 12 core CPU0. You have another CPU1 that does not have GPUs according to that map.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,908
4,871
113
Patrick, I just think about this platform and beside power consumption can you tell me how loud is that? Mine let's say "server room" is near my workplace so I'm just curious how loud is that? I look on the web about any noise info about this but without success.
They are very loud under load. Put this in a data center.