Hello everyone,
I am constructing a multi-GPU server for a small lab and am looking for good suggestions for vendors and specific chassis models.
There are a few constraints that we are under:
1) We are limited on price, and therefore purchasing equipment completely decked out is not possible (want just 1x GPUs, few RAM sticks & CPUs)
2) Needs to *support* 8x GPUs (A100s or H100s) - i.e. the future goal (when more funds come later)
3a) If the motherboard uses PCIe card slots for GPUs, it *must* be water-cooled (the cards will likely melt with 8x consumer grade PCIe card GPUs lined up)
3b) If the motherboard supports/uses SXM sockets, fan cooling is acceptable (not PCIe) (but recall [1] still applies)
4) Unfortunately, the team does not feel confident in setting up liquid cooling systems, so the liquid cooling solution needs to be as 'plug-and-play' as possible. (Like a corsair H100i CPU cooler - preferably not taking apart the GPU)
These requirements have felt pretty restrictive, as this seems like an area when most simply by 12x DGX Superpods for $mega-dollars and go about their business. A single DGX A100 80GB or a DGX H100 is pretty much what we are looking for, but with a few GPUs missing to lower the price to what we can afford.
I know that the SXM GPUs come more 'chip-like' (without the fan/radiator/pcie on them) which *may* make a closed system GPU cooler more easy to use, but we don't have any experience with these yet.
The [STH video about liquid cooling](https://www.youtube.com/watch?v=4Np1HnWiHb4) was very intriguing due to the fact that the heat exchange occurs *outside* the server (thus freeing up space, and allowing any unexpected issues to be external to the server components).
There are several decent options that look decent for chassis [see below 1,2,3,4], but we have not found how to *actually purchase* any of these items directly.
Reference chassis systems:
1) G492-ZL2 G492-ZL2 (rev. A00) | GPU Servers - GIGABYTE Global
- Liquid cooled, but *pre-setup*
2) Gigabyte G492-ZD0 (from 2020) G492-ZD0 (rev. 100) | GPU Servers - GIGABYTE Global
- Air cooled, but with SXM socket
3) Navion 4U HGX A100 Navion 4U 8 GPU NVIDIA A100 Server with NVLink® - NVIDIA HGX™ A100 - Microway
- Air cooled, but with SXM socket
These are just examples, but we're open for any suggestions.
I like (1) since it seems that components could be purchase and simply added without much fuss, and retains the far superior cooling system to get enormous gains in performance.
However, does anyone know of a good vendor to purchase *just the bare-bones chassis* for example?
I am constructing a multi-GPU server for a small lab and am looking for good suggestions for vendors and specific chassis models.
There are a few constraints that we are under:
1) We are limited on price, and therefore purchasing equipment completely decked out is not possible (want just 1x GPUs, few RAM sticks & CPUs)
2) Needs to *support* 8x GPUs (A100s or H100s) - i.e. the future goal (when more funds come later)
3a) If the motherboard uses PCIe card slots for GPUs, it *must* be water-cooled (the cards will likely melt with 8x consumer grade PCIe card GPUs lined up)
3b) If the motherboard supports/uses SXM sockets, fan cooling is acceptable (not PCIe) (but recall [1] still applies)
4) Unfortunately, the team does not feel confident in setting up liquid cooling systems, so the liquid cooling solution needs to be as 'plug-and-play' as possible. (Like a corsair H100i CPU cooler - preferably not taking apart the GPU)
These requirements have felt pretty restrictive, as this seems like an area when most simply by 12x DGX Superpods for $mega-dollars and go about their business. A single DGX A100 80GB or a DGX H100 is pretty much what we are looking for, but with a few GPUs missing to lower the price to what we can afford.
I know that the SXM GPUs come more 'chip-like' (without the fan/radiator/pcie on them) which *may* make a closed system GPU cooler more easy to use, but we don't have any experience with these yet.
The [STH video about liquid cooling](https://www.youtube.com/watch?v=4Np1HnWiHb4) was very intriguing due to the fact that the heat exchange occurs *outside* the server (thus freeing up space, and allowing any unexpected issues to be external to the server components).
There are several decent options that look decent for chassis [see below 1,2,3,4], but we have not found how to *actually purchase* any of these items directly.
Reference chassis systems:
1) G492-ZL2 G492-ZL2 (rev. A00) | GPU Servers - GIGABYTE Global
- Liquid cooled, but *pre-setup*
2) Gigabyte G492-ZD0 (from 2020) G492-ZD0 (rev. 100) | GPU Servers - GIGABYTE Global
- Air cooled, but with SXM socket
3) Navion 4U HGX A100 Navion 4U 8 GPU NVIDIA A100 Server with NVLink® - NVIDIA HGX™ A100 - Microway
- Air cooled, but with SXM socket
These are just examples, but we're open for any suggestions.
I like (1) since it seems that components could be purchase and simply added without much fuss, and retains the far superior cooling system to get enormous gains in performance.
However, does anyone know of a good vendor to purchase *just the bare-bones chassis* for example?
