DeepLearning12 NVLink

Discussion in 'Machine Learning, Deep Learning, and AI' started by Patrick, Jul 1, 2018.

  1. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
    Getting there. These things are quite unpleasant to install compared to CPUs. I have had members here, people I have spoken to over the past few weeks, and even someone walking through the data center today tell me not to overtighten because of how fragile these are. Doubly so for the heatsinks.
    DeepLearning12 8x Tesla P100 GPUs in Gigabyte G481-S80.jpg
     
    #22
    Jaket, niekbergboer, ideabox and 5 others like this.
  3. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    672
    Likes Received:
    216
    WOW !!!
     
    #23
  4. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,072
    Likes Received:
    125
    This system supports the new Optane DIMMs right? Any chance you have some of those going into this build?
     
    #24
  5. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
    Not a released product yet and requires Cascade Lake CPUs.
     
    #25
  6. Revrnd

    Revrnd New Member

    Joined:
    Jan 2, 2018
    Messages:
    27
    Likes Received:
    1
    How's your Deep Learning 12 project coming along Patrick? Really looking forward to seeing how well it goes compared to the 1080Ti powered DL11.
     
    #26
  7. ideabox

    ideabox Member

    Joined:
    Dec 11, 2016
    Messages:
    68
    Likes Received:
    23
    Mondays are usually boring.. Not today though!

    NVLink Time!
     
    #27
    vv111y and Patrick like this.
  8. Robert Fontaine

    Robert Fontaine Active Member

    Joined:
    Jan 9, 2018
    Messages:
    112
    Likes Received:
    26
    I suspect that picture is as close as I will ever get to this tech.
     
    #28
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
    upload_2018-8-22_17-13-37.png
     
    #29
    vv111y and William like this.
  10. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
    Using docker and xmrig-nvidia in case anyone wanted to know

    upload_2018-8-24_11-58-33.png
     
    #30
    William likes this.
  11. Jaket

    Jaket Member

    Joined:
    Jan 4, 2017
    Messages:
    63
    Likes Received:
    11

    Hey Patrick,
    Sorry for the long delay it took our supplier a while to get this system built out.
    It's our first dual root design for we didn't go the 2x Mellanox route with it yet however it has room if needed.

    As we've only ran single root systems in the past I've got a question for you.

    Our system seems to only show 4 GPU's when running nvidia-smi topo -m

    ubuntu@4U-GPU:~# nvidia-smi topo -m
    GPU0 GPU1 GPU2 GPU3 CPU Affinity
    GPU0 X PIX PIX PIX 0-11,24-35
    GPU1 PIX X PIX PIX 0-11,24-35
    GPU2 PIX PIX X PIX 0-11,24-35
    GPU3 PIX PIX PIX X 0-11,24-35

    Not sure if there is anything extra special we need to do in order to show the other cards on the system.

    Any help would be greatly appreciated as I couldn't find anything online about this while trying the Googler.
     
    #31
  12. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
    @Jaket so you only have 4 GPUs and they are all connected to a 12 core CPU0. You have another CPU1 that does not have GPUs according to that map.
     
    #32
  13. Jaket

    Jaket Member

    Joined:
    Jan 4, 2017
    Messages:
    63
    Likes Received:
    11
    Correct for some reason the others are not showing up.
    The system has 2x Intel Xeon Gold 6126 Processor's in it and 8x 1080Ti's however only 4 seem to show up on the system.
     
    #33
  14. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
    Can any other tools see them?
     
    #34
  15. Jaket

    Jaket Member

    Joined:
    Jan 4, 2017
    Messages:
    63
    Likes Received:
    11
    Which other tools would you suggest for trying to view it?
     
    #35
  16. Jaket

    Jaket Member

    Joined:
    Jan 4, 2017
    Messages:
    63
    Likes Received:
    11
    Little update on this, we had to RMA the whole barebone, once that was done everything worked as it should have.
     
    #36
    gigatexal likes this.
Similar Threads: DeepLearning12 NVLink
Forum Title Date
Machine Learning, Deep Learning, and AI NEW OEM NVIDIA Tesla NVLink P100 SXM2 16GB CoWoS HBM2 Aug 31, 2018

Share This Page