AI build.

[spyroot]

Hi Folks,

I spent weeks sizing, calculating, figure out the power distribution. I guess the most complicated one.
Unlike all PDB you can find now, this one can power 12 GPU 600Watt each. But if you want to do something like that,
you need to change all the power cables in the house.

Massive thank you, RolloZ170. He helped me debug all these crazy issues with the Gigabyte board.

I use PPR II Island Universal Board For Redundancy Or Combined Power | Patent Pending - Parallel Miner
For all power distribution. It allows you to feed in 4 server PSU.

* You first need to ensure you are using very good cables.
* Be careful when dealing with 100 Amper +, so you can blow up your investment.
* You will need to make your ESPv12 cable. Use 14AWG and split to 4x12v and 4xGround
* You need 5v/3v source. So either use ATX or GaN PSU ( you can feed 3 wire 12v to 800Watt for example)

This system is mainly for AI Researches who don't have access to a 100k system to confirm experiments
or idea.

 

[rtech]

I am missing something here?
Where do the 12V EPS custom cables fit in.

 

[spyroot]

The breaker board is 6 PIN, so if you want to get 800-1000W watts from a server PSU
you will need 2x8 PIN EPSv12 to 6 PIN. First, I would not recommend anyone buy cable
from Amazon; most EPSv12 cables to Molex 18 AWG and another half of the cables
sold on aws use the wrong pins or 18 AWG cables.

So if you want to get all 12v from the server PSU, you need this cable. Miners mostly
use 4xPIN CPU cables to motherboards, so they are in a ~250 Watt budget.

So PIN 5/6/7/8 12v and confirm with a tester or use a PSU tester (highly recommend)
before plugging anything into a motherboard or using ATX PSU for the motherboard.
(you still need 5v for the pump etc.), Also, note that you can use a separate server PSU with
a separate board provides you SATA and Molex (i.e., 5v), so technically, you
can wire pumps separately.

This will give 5V. If you only want to use server PSU for 5v.

SATA Power Archives - Parallel Miner

SATA Power Archives - Parallel Miner |

www.parallelminer.com

more advanced breaker box miner use provides you 24 PIN ~ but bounded to
250 watts max, so more is needed.

Use decent 8 PIN x 16 AWG on one end and solder to 14 AWG on another end with 3x12V (yellow)
on another end to 6 PIN. In my case, I use Sapphire Raids so that it can go all the way to 1000 watts.

These folks also make boards for miners.

HP 1500w/1200w/750w Breakout Board DITM-W-12-75

This breakout board will automatically power on once a AC power cord is plugged into the server power supply. This board is compatible with the following models. HP 1200 watt: DPS-1200FB 438202-002 440785-001 441830-001 HSTNS-PD11 HSTNS-PD19 570451-101 570451-001 579229-001. HP 750 watt...

www.deepinthemines.com

So you can take 14 AWG cables from that and make custom EPSv12 or use boards.

4090 can push 600 Watt, = 6 GPU x 6. It is already 3600 watts.
(we need at least 20%/30% + margin, so PSU stays quiet and is not loaded
all the time at max)

So you are already on a 4000 Watt add-on top motherboard, etc. Hence we are on a budget 7000-8000 Watt system,
which is not a joke. That way, I said first to change the cable for the 48 Amp socket, etc.
(You can idle with 6 GPUs (about 50 watts per GPU max, but if you push all 6 GPUs to 100 utilization).

Lastly, I recommend liquid-cooled, and I have separate radiators per GPU.

Even two GPU 4090 wired to the same loop after 48 training at 100 utilization push a liquid system
to the max so the pump will work non-stop. So you need either increase the surface
of the radiator. i.e., use something like MORA.

* Separate RAD per GPU.
* Separate rad for CPU.

(That is my view) but I learn many lessons the hard way.

 

[smstewart_9554]

Just curious, why aren't symmetrical RAM slots used?

 

[RolloZ170]

Just curious, why aren't symmetrical RAM slots used?

 

[piranha32]

Another server power supply project, unfortunately 12v only. The good news is that it allows to connect multiple units in parallel, making multi-kW configurations possible: Building an extremely high powered 1-12V lab power supply on the cheap – Matt's Tech Pages

 

[smstewart_9554]

View attachment 30468

Thanks. Interesting board design.

 

[RolloZ170]

Thanks. Interesting board design.

???
G F E - A C D
edit: SPR XCC uses 4 tiles with each two mem.channels.
6 DIMM means unbalanced in any case. this kind of population is better than optical symetrical, but not without options.
SM X13SEI

 

[rtech]

Why did you wanted to power motherboard with server PSU and not with the ATX PSU?

 

[RolloZ170]

@rtech i think it is cheaper, and 4800Watts ATX PSU is hard to find

 

[rtech]

My question is actually followup. I was wondering why did made PCIE > EPS adapter.
Following on this why did he did power via breakout boards through PCIE > EPS not via ATX which is on picture.

 

[RolloZ170]

I was wondering why did made PCIE > EPS adapter.

probably he thinks 8pin is always EPS and 6pin is PCIe.

 

[tinfoil3d]

This is cool, I've only built 2-GPU directly-attached systems. Also used a dell-board for 2.4kw PSU from PM.

 

[spyroot]

Folks,

you can use EPSv12 to 6 PIN, and I encourage people to share trusted vendors who make cables EPSv12
to 6 PIN ( straight)

But

a) first, check a pinout,
b) don't use the PSU CPU cable; in 9 out of 10 cases, it is a different pin out with server boards.
c) make sure it is 16 AWG. (you are only using 3 PINs, 12v, and three ground, don't forget about that)

( server breakout board can feed 600 Watt easily over that 6 PIN) but remember this is not a standard,
hence cable, connector, etc., must be solid. I only did see a decent CPU cable from Supermicro, but I
I couldn't find pinouts to the PDB side and thus made it myself.

---

Regarding why I use only server breakout.

If you notice, I showed a board that can aggregate four server boards.
If you check docs, that board will shut down if one of the PSUs sources fails. Why do I believe it is important?

Consider your ATX feed 12/5/3v to the motherboard. Now the miner uses the Razer card. All this
work on X1 speed; all get separate 12v directly from the same board. (i.e., GPU and Razer card have
same power source)

But if you are using PCI 5.0 or 4.0, you need all lanes ( for AI, it matters, really matters)
because you are constantly moving data from CPU to GPU. (It one of the reasons high-end systems
try to bypass the motherboard etc, and use interconnect).

Hence, you want to connect GPU directly to the PCI slot or get a very expensive PCI backplane.
As far as I know, PCI 5.0 backplane is very hard to get now. ( Broadcom has a chip,
but I don't think you can get it at the moment)

Correct me if wrong here.

----

Given that, now consider if one server PSU will fail, i.e., the one that provides 12v to GPU.
Now GPU "potentially" might try to draw more than 75 watts from the PCI slot from the motherboard
and it's dangerous. ( because the motherboard has a different power source,
even if synced, it is still a different source). Hence If you are using ATX. Because of the way it works,
ATX provides a signal over Molex to a breaker box. (i.e., it is a one-way signal)

If ATX PSU feeds 12v to the motherboard while server PSU will go down.
It means GPU one leg connects to the motherboard via PCI.

So what do you want ( to be on the safe side with GPU that costs 1.5k-3k) is that the
entire system shutdown
if one of the PSU goes down.

Safety first.

If anyone knows in detail does modern GPUs have protection for this case,
please correct me.

---

Lastly, if you can find any PDB that can feed 6 GPUs with a budget 600Watt per GPU,
go get it; it will simplify your life a lot. ( But based on my research, there are no PDB like that)

The PDB designed for 6/8 GPU is all for custom motherboard layouts.
If you carefully check, most super micro motherboards are sized at 300 watts per GPU.
(this is based on an A100 card).

For example, if you check the latest 4U SuperTower and you want to put
4x4090, you will be power limited.

Again, please correct me if I am wrong here.

---

I sent several emails to a few vendors and Supermicro, PDB matrices I got dated for 2019.
So it is unclear the PDB rev for this year, how sized.

 

[spyroot]

Another server power supply project, unfortunately 12v only. The good news is that it allows to connect multiple units in parallel, making multi-kW configurations possible: Building an extremely high powered 1-12V lab power supply on the cheap – Matt's Tech Pages

Yes, I think the parallel and decent PDB would be the best approach but I couldn't find any that you can buy.
Plus I sizing systems for 2-3 years and I don't expect AMD/Intel/Nvidia will reduce TDP only increase.

 

[spyroot]

@rtech i think it is cheaper, and 4800Watts ATX PSU is hard to find

You can't get 4800 ATX PSU, and I don't think you will see any. Remember, in a regular home in Europe/Asia
with 220-240v, sized 16A, the circuit breaker, cable, etc., are all sized with 16A max. (in the US I'm
100 sure, but I think it even half with 110v)

That way, I said if you are doing what I'm doing, change the first cable and circuit breaker to handle 40Ams first.
If you are using two different sockets ( technically, you can, but, i.e., one server PSU to one socket and circuit
brake and another to another), But in case you might have an issue with the difference in the ground.

 

[spyroot]

My question is actually followup. I was wondering why did made PCIE > EPS adapter.
Following on this why did he did power via breakout boards through PCIE > EPS not via ATX which is on picture.

I posted more detailed motivation, but in short, considering you have ATX and breakout if the server PSU fails, GPU is still connected one leg to Motherboard. Now it is not a miner setup, so GPU still draws 75 watts from the PCI slot from the motherboard, so if GPU lost all 6 / 8 PIN, it might draw more than it should from the motherboard. The breakout I showed has four inputs from 4 different breakouts fed in via five cables to this board. If one PSU fails, the entire system shuts down, so it is much safer.

Generally, you want to have a single power source. i.e., the same as you normally do via PDB boards.

(you might say GPU will not do that). Well, I confirmed my experiment with the GPU motherboard and simulated this, and it burned
the entire PCI cable. )) (note that GPU survived)

server PSU -> GPU -- 75 Watt -- PCI slot -- motherboard <-- ATX
In this experiment, I shut down the server PSU and GPU, drew more than it should,
and immediately burned the PCI cable. ( if you ask me, GPU should have protection for this and should not
draw more then it should, but it gamble)

 

[rtech]

How much it did cost you?

 

[spyroot]

The server PSU is more or less cheap (of course, used). But
you can get Delta 2400 for 100-200 USD. That PSU is ten times better
than any ATX. If you are using Dell/HP, you probably want to replace the fan with
a nocturne to reduce noise. (. additional cost)

The breakout board cost 45 USD, so it is cheap as well, or use the one I show without any board,
just a connector and use your cable.

Power Supply Kit Archives - Parallel Miner

Customized server power supply kit for ASIC miners and GPU rigs.

www.parallelminer.com

Deep In The Mines LLC

We are a Biostar authorized distributor. We sell motherboards, power supplies, cables, and breakout boards used for cryptocurrency mining

www.deepinthemines.com

Cables I ordered from China 14 AWG /16 AWG. You
must wait a bit, but you will get good-quality cables.

Check review people who buy just cable check quality five times : )

https://www.aliexpress.com

This tool does magic

https://www.amazon.com/Capri-Tools-20010-Precision-Stripper/dp/B01018CVM0

The most expensive part motherboard, costs 1100-1400 USD.
(I would not recommend using 790 chipsets, so stay with c741,
it gambles on CPU support). (Consider that 12 months from now,
you will get a Platinum Xeon CPU on eBay for a price of 200-300 USD)

DDR5 RDIMM you can find for a reasonable price.
Actually, you can find DDR5 memory for servers cheaper than "sexy" DDR5
overclocked DDR5 for gamers.

GPU, I'm it a rabbit hole, really... ( I have done this for many years, so I have many GPUs for research systems
so it does not require a single time investment).

I guess investing makes more sense if you need to confirm your research, do many runs,
and don't have access ( free to TPU/GPU resources)

Lastly, I did a lot of benchmarks and comparisons; if you want to use ADA,
it beats the A100 that I tested every single time.

Now PUMP, Rads, etc. I just moved, but It was expensive.
I love this pump ( but I hate SATA behind). ( note the nice part that the pump
has integration with Linux so that you can get all the data from the lm-sensor)

ULTITUBE D5 100 PRO LEAKSHIELD Ausgleichsbehälter mit D5 NEXT Pumpe und Leckageschutzsystem - Aqua Computer

ULTITUBE D5 100 PRO LEAKSHIELD Ausgleichsbehälter mit D5 NEXT Pumpe und Leckageschutzsystem: Die Ausgleichsbehälter der ULTITUBE-Serie sind mit einer Röhre aus Borosilikatglas ausgestattet, das im Gegensatz zu dem häufig für Röhren-Ausgleichsbehälter eingesetzten Plexiglas eine sehr hohe Härte...

shop.aquacomputer.de

airplex radical

airplex radical

shop.aquacomputer.de

Rads are expensive, but it does magic. ( note that the quality of the build,
you will buy once for the next ten years)

For the CPU, I use this. Expensive but not crazy

13090 Alphacool ES Jet LGA 4677 2U (SP5 optional)

Server and workstation optimized CPU cooler for Intel Xeon with 4 nozzles technology for high performance and reliable cooling

shop.alphacool.com

So the main cost is GPUs.

P.S also forgot

8020 T-Slot Product Resources | Extruded Aluminum Building Systems

Learn about 80/20 t-slot profiles and parts, including fasteners, panels, casters, accessories, & more. Visit our site for more info on your build needs.

8020.net

Source for all crazy types of nuts, t-slots , brackets etc : )))

 

[Stephan]

Insane. I like it.