Colocation Number 3 Build Out Pictures (Cab 2)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Working on this a bit today and wanted to share an update. This is the #2 cabinet we have. #1 is ultra ugly at the moment as @Diavuno is working on the longer term networking.

Here is a picture with the Quanta QuantaMesh T3048-LY9 (48 port 10Gbase-T + 6x QSFP+ 40GbE switch):
20150904_164412.jpg

Here is the "beast" Supermicro dual Xeon E5 v3 server with 256GB RAM and 8x AMD FirePro S9150 GPUs (GPU specs here):
20150904_164424.jpg

For those wondering, the setup does pull a lot of power. 710w on this view (idling):

20150904_185006.jpg
And we can now see power consumption over time by power supply (this 8x GPU box has 4x power supplies):
upload_2015-9-5_13-17-34.png

Here's a view of the overall load from the APC 8441 by outlet right now.
upload_2015-9-5_13-26-32.png

Outlets 4-7 are the Supermicro 8x GPU server @William is testing. Outlets 22 and 23 are the QuantaMesh T3048-LY9 switch redundant power supplies.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Shame you didn't post guts, if anything those pictures are always seen somewhere else on internet.
Guts of the colo or guts of the 8x GPU system?

In terms of the colo, not much more in that rack right now other than that 8x GPU system and the Quanta 48 port 10Gbase-T switch you can see (top) in that rack. The other one next to it you can see is in complete disarray but is more fluid in terms of gear and networking now. You can actually see a set of cable organizers on the ground in the rack to the right.
 
  • Like
Reactions: T_Minus

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Thank goodness @William is now on a 208V circuit! I was at the datacenter this evening and saw the box above was using a lot of power (he was benchmarking it.)
APC During Bills 8x GPU test.jpg

For those wondering 10.3A * 208v = over 2.1kw.
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Yeah that was great not having to worry about popping breakers here and no noise HAHA.
Its really coming together in the data center !!
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,624
2,043
113
Holy heater batman!

How's cooling at this colo work
 

William

Well-Known Member
May 7, 2015
789
252
63
66
HAHA might make for a nice game of BF4 with 48 displays :)
Or it can actually run 8x 4K displays, now that's nice :)
 

William

Well-Known Member
May 7, 2015
789
252
63
66
At trade shows and other places where they need a very large display setup they would normally use something like this.
Supermicro 7048GR-TR (Intel C612) Workstation Tower System Review
That can also hold 4x W9100's with 6 displays on each card or 1 4K display.

This machine could do an even bigger display with 8x W9100's

The machine we have installed has been certified for workstation deities also, picture a Thunderbolt setup and running this rig remotely that way. In our case we will have it setup for number crunching using remote access to operate. But we will also throw some workstation type benchmarks on it to see what it does. Sometimes these benchmark programs have issues with this many cards so we will see what works and what doesn't.

I am expecting something like 41.92 TFLOPS of Single precision crunching power !
 
  • Like
Reactions: b3nz0n8