Colocation Number 3 Build Out Pictures (Cab 2)

Discussion in 'DIY Server and Workstation Builds' started by Patrick, Sep 5, 2015.

  1. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,496
    Likes Received:
    4,440
    Working on this a bit today and wanted to share an update. This is the #2 cabinet we have. #1 is ultra ugly at the moment as @Diavuno is working on the longer term networking.

    Here is a picture with the Quanta QuantaMesh T3048-LY9 (48 port 10Gbase-T + 6x QSFP+ 40GbE switch):
    20150904_164412.jpg

    Here is the "beast" Supermicro dual Xeon E5 v3 server with 256GB RAM and 8x AMD FirePro S9150 GPUs (GPU specs here):
    20150904_164424.jpg

    For those wondering, the setup does pull a lot of power. 710w on this view (idling):

    20150904_185006.jpg
    And we can now see power consumption over time by power supply (this 8x GPU box has 4x power supplies):
    upload_2015-9-5_13-17-34.png

    Here's a view of the overall load from the APC 8441 by outlet right now.
    upload_2015-9-5_13-26-32.png

    Outlets 4-7 are the Supermicro 8x GPU server @William is testing. Outlets 22 and 23 are the QuantaMesh T3048-LY9 switch redundant power supplies.
     
    #1
    T_Minus, neo, William and 1 other person like this.
  2. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    742
    Likes Received:
    227
    The guts will be posted soon :)
     
    #2
  3. Diavuno

    Diavuno Active Member

    Joined:
    Jan 31, 2014
    Messages:
    839
    Likes Received:
    108
    I know you can "see it(ICMP)" but how do you SEE it (RBG)

    XD
     
    #3
  4. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,496
    Likes Received:
    4,440
    Guts of the colo or guts of the 8x GPU system?

    In terms of the colo, not much more in that rack right now other than that 8x GPU system and the Quanta 48 port 10Gbase-T switch you can see (top) in that rack. The other one next to it you can see is in complete disarray but is more fluid in terms of gear and networking now. You can actually see a set of cable organizers on the ground in the rack to the right.
     
    #4
    T_Minus likes this.
  5. TangoWhiskey9

    TangoWhiskey9 Active Member

    Joined:
    Jun 28, 2013
    Messages:
    402
    Likes Received:
    59
    I've seen racks more full and more disorganized than that first pic for sure.
     
    #5
  6. Martin Jørgensen

    Martin Jørgensen New Member

    Joined:
    Jun 3, 2015
    Messages:
    28
    Likes Received:
    6
    Whats do you need the gpu for? :D
     
    #6
  7. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,496
    Likes Received:
    4,440
    Thank goodness @William is now on a 208V circuit! I was at the datacenter this evening and saw the box above was using a lot of power (he was benchmarking it.)
    APC During Bills 8x GPU test.jpg

    For those wondering 10.3A * 208v = over 2.1kw.
     
    #7
    rnavarro, T_Minus and William like this.
  8. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    742
    Likes Received:
    227
    Yeah that was great not having to worry about popping breakers here and no noise HAHA.
    Its really coming together in the data center !!
     
    #8
  9. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    742
    Likes Received:
    227
    Going over the data it looks like this pulled 2,613watts under full load.
     
    #9
    T_Minus and Patrick like this.
  10. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,734
    Likes Received:
    1,440
    Holy heater batman!

    How's cooling at this colo work
     
    #10
  11. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,496
    Likes Received:
    4,440
    upload_2015-9-7_18-29-14.png

    Hmmm.... what is now on outlet 9-12?
     
    #11
  12. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,496
    Likes Received:
    4,440
    Hey @William guess what is getting prepped: upload_2015-9-7_19-16-11.png
     
    #12
    William likes this.
  13. Martin Jørgensen

    Martin Jørgensen New Member

    Joined:
    Jun 3, 2015
    Messages:
    28
    Likes Received:
    6
  14. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,496
    Likes Received:
    4,440
    Can I borrow 48 monitors real quick?
    ASRock Rack 8x AMD FirePro W9100 front.jpg
     
    #14
    nry, Dajinn, ultradense and 6 others like this.
  15. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    742
    Likes Received:
    227
    HAHA might make for a nice game of BF4 with 48 displays :)
    Or it can actually run 8x 4K displays, now that's nice :)
     
    #15
  16. Hank C

    Hank C Active Member

    Joined:
    Jun 16, 2014
    Messages:
    642
    Likes Received:
    66
    is that asrock build?
     
    #16
  17. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    742
    Likes Received:
    227
    ASRock Rack 3U8G-C612 8x AMD W9100's
     
    #17
  18. Martin Jørgensen

    Martin Jørgensen New Member

    Joined:
    Jun 3, 2015
    Messages:
    28
    Likes Received:
    6
    what could you possibly need that for? o_O
     
    #18
  19. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    742
    Likes Received:
    227
    At trade shows and other places where they need a very large display setup they would normally use something like this.
    Supermicro 7048GR-TR (Intel C612) Workstation Tower System Review
    That can also hold 4x W9100's with 6 displays on each card or 1 4K display.

    This machine could do an even bigger display with 8x W9100's

    The machine we have installed has been certified for workstation deities also, picture a Thunderbolt setup and running this rig remotely that way. In our case we will have it setup for number crunching using remote access to operate. But we will also throw some workstation type benchmarks on it to see what it does. Sometimes these benchmark programs have issues with this many cards so we will see what works and what doesn't.

    I am expecting something like 41.92 TFLOPS of Single precision crunching power !
     
    #19
    b3nz0n8 likes this.
  20. neo

    neo Well-Known Member

    Joined:
    Mar 18, 2015
    Messages:
    672
    Likes Received:
    359
    Have you bought a ServerLIFT yet?


    [​IMG]
     
    #20
Similar Threads: Colocation Number
Forum Title Date
DIY Server and Workstation Builds STH Colocation Number 2 Build Log Apr 23, 2015
DIY Server and Workstation Builds Minimal colocation build ideas wanted Jun 13, 2017
DIY Server and Workstation Builds Upgrading 96TB Colocation. Need some ideas for making higher IOPS Feb 2, 2017
DIY Server and Workstation Builds Thoughts on making money from colocation? Apr 26, 2012

Share This Page