DeepLearning12 NVLink

Discussion in 'Machine Learning, Deep Learning, and AI' started by Patrick, Jul 1, 2018.

  1. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,900
    Likes Received:
    3,836
    I have been getting restless to do another deep learning build. Today I invested in some Tesla P100 16GB GPUs.

    Instead of going with PCIe cards, I decided SXM2 with NVLink.

    Next items:
    1. Need to do some research on whether I can put P100 in V100 trays. The V100 I believe has 6x 50gb/s links. The P100 was 4x 40. That is a big difference but if if they work with both, I will want the newer V100 trays.
    2. I think this is going to be Skylake based. It looks like the E5 generations were less expensive because CPUs were less expensive. Also, motherboards were less expensive.
    3. Skylake is somewhat strange. If you want 2x GPU memory, and each P100 has 16GB that is 64GB in a 4x GPU system or 128GB in an 8x GPU system. That means, at 2x is 128 or 256GB of system RAM. With Skylake the options are really 96GB, 192GB, or 384GB. With E5 128 or 256GB would be easier.
    4. CPUs. What to use?

    Many questions. Likely a few weeks from answers.
     
    #1
  2. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,679
    Likes Received:
    745
    popcorn time
     
    #2
  3. Jaket

    Jaket Member

    Joined:
    Jan 4, 2017
    Messages:
    59
    Likes Received:
    10
    I would love to see how AMD's CPU's work with AI, Deep learning etc. We've been building a lot of Intel systems with 8x 1080TI's however nothing with AMD as of yet.
     
    #3
  4. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,900
    Likes Received:
    3,836
    @jacket are you doing single root or dual root? Tried 10x 1080 Ti yet?
     
    #4
    Jaket likes this.
  5. Jaket

    Jaket Member

    Joined:
    Jan 4, 2017
    Messages:
    59
    Likes Received:
    10
    We haven't tried running 10x 1080 Ti's yet it's mostly for one of our clients and they've only requested 8 cards so far. We have mostly used SM for them however this is the next system we will be building out.
    G481-HA0 (rev. 100) | High Performance Computing System - GIGABYTE B2B Service

    All of the storage options in this system seems like a great option for their requirements.

    Have you found it being a big advantage using 10 cards over 8? Might be interesting to bring up to our client.
     
    #5
  6. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,900
    Likes Received:
    3,836
    The major benefits are that you save 15-20% on the initial installation per GPU and some on the ongoing costs since you are using more GPUs per chassis.

    I am really interested in the build-out of that Gigabyte server. It is a dual root design so are you planning to use 2x Mellanox cards and avoid the NUMA penalty?
     
    #6
  7. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    775
    Likes Received:
    52
    Block diagram shows you are stuck with a built in X550-AT2 on CPU1. Only a non-GPU x16 slot off CPU0. Spec page suggested it's designed for dual Omni-Path CPUs.
     
    #7
    Patrick likes this.
  8. Revrnd

    Revrnd New Member

    Joined:
    Jan 2, 2018
    Messages:
    22
    Likes Received:
    0
    If you had a really good use case and some extra cash laying around you could always opt for one of these...

    Nvidia DGX-2

    On a side note, I'd love to see how these would go rendering some really intense scenes like in Ready Player One or some other CGI intense movie.

    But on a serious note, just out of interest, do you guys hire these things out? Or do you use them for data analytics etc?

    Love your work on the other Deep Learning machines though Patrick. Keep up the good work.
     
    #8
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,900
    Likes Received:
    3,836
    @Revrnd we are testing allowing people to hire the big GPU systems
     
    #9
    Revrnd likes this.
  10. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,900
    Likes Received:
    3,836
    DeepLearning12 update 8x NVIDIA SXM2 16GB 800px.jpg
     
    #10
    cesmith9999 and K D like this.

Share This Page