This is my first post here and my first computer build (as will soon become apparent ). I work in deep learning and, after renting various servers for several years, have built the following machine for a specific workflow centered around cuDNN:
Build’s Name: Server-1
Operating System/ Storage Platform: Ubuntu Server 18.04
CPU: Threadripper 2990WX
Motherboard: X399 Taichi
Chassis: ?? (need advice)
Drives: Samsung SSD 970 EVO 1 TB M.2
RAM: Corsair Vengeance LPX 4x16 GB 3200 MHz
Graphics Cards: Quad (4x) GigaByte RTX 2080 Ti WindForce
Power Supply: Corsair AX1600i
For now I have it running on an open test bench in my office (pictures attached). My plan is to put it in a rack mount case and move it to a co-location facility once everything is done, but I have not yet purchased the rack mount case or case fans.
My problem is with cooling the graphics cards. I made the mistake of getting "open-air"-style graphics cards instead of blower-style and now I am experiencing significant thermal throttling and would like some advice about what to do.
When I run the graphics cards with their fans and shields on, and point an external 16" house fan running at max speed over the open-air case, I get the following performance:
GPU 0 (on the end, fans facing open air, backplate facing GPU 1): 100% (ie. same as running just one GPU)
GPU 1 (surrounded on both sides): 71% of one GPU
GPU 2 (surrounded on both sides): 82%
GPU 3 (on the end, fans facing GPU 2, backplate open): 91%
When I remove the fans and shields of GPUs 1-3, as shown in the pictures, and use the same external house fan to get air flow, I get the following improved performance:
GPU 0: 100%
GPU 1: 77%
GPU 2: 89%
GPU 3: 96%
So, removing the fans and shields from the graphics cards clearly helps. The performance still isn't great, but it's (I guess) acceptable. However, I am not sure if the airflow will be as good inside a case.
My questions are:
1. What rack mount server case would be best for the above components?
2. Is there anything worth doing to try to improve the GPU cooling?
It seems to me that I have the following options:
1. Put this system inside something like a Rosewill RSV-L4500 case, get the best case fans (maybe Delta PFB1212UHE-F00 ??), and accept a 10% or so loss of performance.
2. Split out the graphics cards using a PCI-e x16 extender, put the WindForce GPU fans back on, and move the cards to a separate section of the case where they have more spacing.
3. Keep the graphics cards on the motherboard (where they have dual-slot spacing) but replace the GPU cooling to either water cooling, a better passive solution, or a custom blower.
I am leaning towards #1 but am not sure. I also haven't been able to find any after-market passive or blower style cooling solutions for the RTX 2080 Ti.
Build’s Name: Server-1
Operating System/ Storage Platform: Ubuntu Server 18.04
CPU: Threadripper 2990WX
Motherboard: X399 Taichi
Chassis: ?? (need advice)
Drives: Samsung SSD 970 EVO 1 TB M.2
RAM: Corsair Vengeance LPX 4x16 GB 3200 MHz
Graphics Cards: Quad (4x) GigaByte RTX 2080 Ti WindForce
Power Supply: Corsair AX1600i
For now I have it running on an open test bench in my office (pictures attached). My plan is to put it in a rack mount case and move it to a co-location facility once everything is done, but I have not yet purchased the rack mount case or case fans.
My problem is with cooling the graphics cards. I made the mistake of getting "open-air"-style graphics cards instead of blower-style and now I am experiencing significant thermal throttling and would like some advice about what to do.
When I run the graphics cards with their fans and shields on, and point an external 16" house fan running at max speed over the open-air case, I get the following performance:
GPU 0 (on the end, fans facing open air, backplate facing GPU 1): 100% (ie. same as running just one GPU)
GPU 1 (surrounded on both sides): 71% of one GPU
GPU 2 (surrounded on both sides): 82%
GPU 3 (on the end, fans facing GPU 2, backplate open): 91%
When I remove the fans and shields of GPUs 1-3, as shown in the pictures, and use the same external house fan to get air flow, I get the following improved performance:
GPU 0: 100%
GPU 1: 77%
GPU 2: 89%
GPU 3: 96%
So, removing the fans and shields from the graphics cards clearly helps. The performance still isn't great, but it's (I guess) acceptable. However, I am not sure if the airflow will be as good inside a case.
My questions are:
1. What rack mount server case would be best for the above components?
2. Is there anything worth doing to try to improve the GPU cooling?
It seems to me that I have the following options:
1. Put this system inside something like a Rosewill RSV-L4500 case, get the best case fans (maybe Delta PFB1212UHE-F00 ??), and accept a 10% or so loss of performance.
2. Split out the graphics cards using a PCI-e x16 extender, put the WindForce GPU fans back on, and move the cards to a separate section of the case where they have more spacing.
3. Keep the graphics cards on the motherboard (where they have dual-slot spacing) but replace the GPU cooling to either water cooling, a better passive solution, or a custom blower.
I am leaning towards #1 but am not sure. I also haven't been able to find any after-market passive or blower style cooling solutions for the RTX 2080 Ti.
Attachments
-
553.5 KB Views: 35
-
493.6 KB Views: 33
-
481.2 KB Views: 35
Last edited: