Supermicro Chassis or Power Supply to GPUs

Discussion in 'Cryptocurrency Mining and Markets' started by nkw, Jan 3, 2018.

  1. nkw

    nkw Active Member

    Joined:
    Aug 28, 2017
    Messages:
    130
    Likes Received:
    44
    I have more SC846 chassis and SuperMicro 1k power supplies sitting around than I am really utilizing and am wondering if anyone has tried repurposing these for GPU mining?

    Several people are selling breakout boards to take HP server power supplies and break them into 6-pin or 8-pin PCIE power connections for GPUs (like this: DPS-1200FB/QB A Power Supply Breakout Board & 10 Cable 6 pin for Ethereum Mining | eBay ), but I haven't seen anything similar for Supermicro. Looks like some people in the past have toyed with it - Dmitry Pakhatov: Supermicro Power Supply PWS-1K41P-1R rev1.1 Occasion and Inexpensive 1400W Power supply

    I thought maybe Supermicro already had a PDU replacement that might include a number of PCIE power connections, but it appears even for their GPU servers they provide the PCIE power connections on the motherboard. e.g. https://www.supermicro.com/manuals/superserver/1U/MNL-1810.pdf

    It might not be cost effective to do something like this, but I setup a couple of GPU mining systems with consumer power supplies and missed having access to the PMBUS info that you get with a real server. (Although I did splurge and use X10SRL-F motherboards + E5 CPUs instead of the cheapest Celeron because I really don't like having systems around I can't manage with IPMI.)
     
    #1
  2. marcoi

    marcoi Active Member

    Joined:
    Apr 6, 2013
    Messages:
    880
    Likes Received:
    138
  3. nkw

    nkw Active Member

    Joined:
    Aug 28, 2017
    Messages:
    130
    Likes Received:
    44
    Hmm.. That looks promising. I can't find anywhere they list how many molex/PCIE connectors it has. Maybe 4 PCIE 6-pin from looking at the picture?
     
    #3
  4. marcoi

    marcoi Active Member

    Joined:
    Apr 6, 2013
    Messages:
    880
    Likes Received:
    138
    it comes on this 4u box SC747TG-R1400B-SQ | 4U | Chassis | Products | Super Micro Computer, Inc.
    im looking at the manual now

    update
    The Supermicro power distributor PDB-PT747-4648 connects 27-pairs input to (1) power 24-pin ATX, (2) power 8-pin +12v, (6) legacy 4-pin and (1) l2C

    edit 2: also they have this bad boy
    Supermicro PDB-PT418-B6824 27-Pairs Power Distributor For Redundant Power Supply
    The Supermicro power distributor PDB-PT418-B6824 connects 27-pairs input to (1) power 24-pin ATX, (6) power 8-pin +12v, (13) legacy 4-pin and (1) l2C.
     
    #4
    nkw likes this.
  5. britinpdx

    britinpdx Active Member

    Joined:
    Feb 8, 2013
    Messages:
    342
    Likes Received:
    145
    It's called out in the Supermicro Power Supply Cable List .. 4x 6pin PCI-E and 4x 8pin PCI-E
     
    #5
    nkw and StammesOpfer like this.
  6. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    486
    Likes Received:
    210
    Garbage.

    A pair of 1620w running on 208v is needed to keep 800w of GPUs running stable on that. It's a joke.

    "PDB 826" can be used to upgrade a 2u 825 chassis to a decent amount of power delivery. Then add in an 8-pin EPS -> 2x 8-pin PCIe converter cable (expect it to be stable up to a gpu needing 8+6 level of power). Power the motherboard off the 4+4 connector (one 4-pin for each cpu -- just stay away from TDP > 115W). Then get the "2 molex -> 2x 8-pin pcie" adapters. "Easy" enough to power 3x 1080ti's on this arrangement plus have 2 molex left for drives. Or 4x 1080ti if you know what you're doing regarding balancing the molex loads.

    Can draw about 1000w off an 826 if you do it right (i.e. buy $40 in power cables) -- using a single 1280w psu or pair of 740's if you want to. Good luck pulling that off with the PT747 without a pair of 1620's using 208v power.
     
    #6
  7. Fzdog2

    Fzdog2 Member

    Joined:
    Sep 21, 2012
    Messages:
    82
    Likes Received:
    11
    Anybody done anymore digging on this? I have a pair of PWS-902-1R PSU left over from a 4u build that could be used for GPUs if it had a proper breakout board. The PDB-PT846-8824 breakout board just has a 24 pin and 2x 8 pin EPS plugs with a bunch of 4 pin molex.
     
    #7
  8. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    486
    Likes Received:
    210
    theres an 8-pin EPS to pcie cable made by supermicro.
    also 2x molex -> pcie is easy to come by

    problem is the older pdb's perform badly. by that I mean, unreliable when used at wattages well below spec of the connectors and psu.
     
    #8
  9. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    486
    Likes Received:
    210
    Some more details:

    PDB-PT826-8824 is the "good" model of PDB.

    It has thicker gauge wires than the "825" model. It has one 8-pin EPS plug and one 2x4-pin EPS plug. What I normally do for GPU servers, is a few things:

    ----------

    First, I take the 2x4pin connector, and connect 4 pins to one motherboard 8 pin connector, and the other 4 pins to the second 8 pin motherboard connector.

    Based on what I could find online, everything I read said this would definitely be fine for CPUs <100W TDP, probably OK for 115w TDP CPUs, and probably NOT OK for 130w+ CPUs or overclocked CPUs. I've had no issues with 115w TDP Intel chips (E5-2670v1 / E5-2680v2) with this configuration.

    That leaves you with an available 8-pin EPS plug coming off the PDB. This can be converted to a 2x 8-pin PCIe cable.
    Supermicro part number CBL-0424L -- I've bought these from wiredzone.com or superbiiz.com. I've used quite a few of these and they work great.

    superbiiz.com: CBL-0424L Supermicro 40cm 8Pin to Dual 6+2Pin GPU Power Extension Cable - Internal Power Cable - SuperBiiz.com
    wiredzone.com: Supermicro CBL-0424L Power Cables

    Looks like both are out of stock right now, but they usually have them.

    I would say, expect to power no more than 8+6 off the 8+8 cable. One exception is the Vega, as I draw quite a bit less than 200 watts on those, even though it has 2x 8-pin PCIe, this works fine. 1080ti has 1x 8-pin + 1x 6-pin. That has worked fine as well. However, powering 2 1080ti's, using both 8-pins on this to power the 8-pins on each of 2x 1080ti, and then connecting up something else for the 6-pins on those 1080ti's, I've not had that work out, so don't expect it to. I don't know if I've tried powering two cards that each need an 8-pin connector -- it might work, but don't count on it.

    So that gets you part of the way there. Works fine for 1-gpu servers.

    ----------

    Second, is make use of the molex plugs.

    See this cable:
    https://www.amazon.com/Athena-Power-4-pin-PCI-Express-Cable-PCIE4628/dp/B0032MT0OU

    It takes 2x molex plugs, and converts to 2x 8-pin pcie. A bit on that in a minute.

    As to the product itself, these are usually good, but I've had a few of them break. Make sure that the molex pins are securely seated inside the molex connector, that the wires are not damaged, and that the molex plug you are attaching it to is firmly seated. Finally, these aren't "delicate", but, I've had a couple break or not give a solid connection if you are too rough with your bend radius on these. If you keep those things in mind, the product itself is good.

    Ok, so you can totally go and draw 300+ watts from one of these right? Maybe -- I don't recommend it. Try to stick to the rule for the EPS converter: -- 8 + 6 pcie is your target on these (i.e. 250w draw max), or a vega 8+8 so long as you're undervolting it. Also, whether this works for your use depends upon the PDB you're using, the wattage of the PSU, and the gauge of the wires you're connecting these to.

    To keep things as simple as I can, here's how you wire it up:

    ----

    Scenario 1 -- You have an old style "825" PDB that supports a maximum of 920w power supplies:

    Connect one molex socket on this cable to a molex connector on one "molex string" on your PDB, and the other molex socket on this cable to a molex connector on a different molex string on your PDB. This advice assumes you plan to connect a 1080ti or other 250w TDP card (8+6 pin card) to this cable (or equivalent power draw). If you are connecting a lower wattage card (say, a 1070 or 1080 -- maximum of a single 8-pin pcie), then you can probably get away with powering both molex sockets from connectors on the same molex string.

    But wait -- this uses up both strings! How can I connect 3+ GPUs to this PDB?
    Answer: You can't. Sorry. It's a terrible PDB. You'll be lucky to get away with 2x 1080ti set to 200w TDP on this PDB. One GPU should definitely work, 2 if you're lucky, and not more than that.

    What power supply to use?

    1x 1080ti GPU -- Real draw is likely 300 - 500 watts, depending upon drives and cpu utilization -- 740w 80+ platinum is my recommendation (2 for redundancy)

    2x 1080ti GPUs -- Real draw is likely 500 - 700 watts, depending upon drives and cpu utilization. 2x 920w 80+ platinum if you have them -- it's the highest wattage you can get into this PDB. Being over-specc'ed might give you a little more fudge factor due to the lousy PDB, but this might still not work.

    ----

    Scenario 2 -- You have a newer style "826" PDB that supports 1600w power supplies:

    This is a little more flexible. You have 3 molex strings to work with. You've also got that 8 pin EPS cable you can use. I've got 2 servers with 4x 1080ti powered by this PDB (plus a single CPU), but it was a little dicey. I have no problem suggesting that 3x 1080ti + dual cpus will work fine if your power supplies are beefy enough. Here's how you do that:

    So, the first molex connector on each "molex string" has a thicker gauge wire than the second molex connector. If possible, wire up at least one of the two molex sockets on this cable to the "thicker wire" (first molex connector) on one of the strings. You can wire up the second molex socket on this cable to the "second molex connector" on a string -- that's fine. Looking at the cable I linked to, you'll notice that one molex connector has one yellow wire and two black wires. And the other molex connector has two yellow wires (on the same pin) and 1 black wire. My recommendation is to use the "2 yellow wires on one pin + 1 black wire on 1 pin" connector in combination with the "first molex connector on the string". Reason being, this connector is going to draw more power per wire than the other connector, so it makes sense to connect this one to the beefier molex connector.

    From what I was reading online, the specs of the molex connector is such that the thicker gauge wire + 1 molex connector supports something like 125 watts, and the thinner gauge wire + 1 molex connector supports 100 watts. (If I am remembering correctly). So wiring up one molex to the thick wiring, and the other to the thin wiring, it should be within spec at 225w usage maximum. a 250w TDP card is supposed to draw 25w from the pcie slot and 225w from the pcie cables, so this should be within spec.

    Anyway, this PDB has 3 strings, 6 molex connectors. 2 strings are medium length, and one is short. For a 3 GPU setup, I use the two longer strings to connect 2x 1080ti, and use the 8-pin EPS connector (with adapter cable) to power the third 1080ti. This leaves you with 2 molex connectors you can use for drives. The backplane on a 2u supermicro needs 3 molex connectors, so if you want to use that, you'll need a molex splitter / Y adapter. I recommend connecting the splitter to the first molex connector on the short molex string. Thicker gauge splitter cables are better -- I think 16 AWG is the best you'll find -- and this will work fine. Based on my prior math of "the better molex connector is good for 125w and the less-good-one is good for 100w", this means the backplane could hypothetically draw 225w. For a 12-drive backplane using 3.5" 7200rpm sata drives, 80-120w is the likely draw fully populated, so this is no issue.

    What power supply to use?

    1x 1080ti GPU -- 1x 740w platinum is fine, or anything higher. 2 PSUs for redundancy.

    2x 1080ti GPUs -- 1x 740w platinum is probably fine, but you might want to go with 920w or higher to be on the safe side. 2 PSUs for redundancy.

    3x 1080ti GPUs -- With 12 drives, 2 cpus at full utilization, and 3x 1080ti at 100% TDP, you could be drawing over 1100w from the PSU (roughly 1200w from the wall). 1280w and 1600w Supermicro PSUs are rated for 1000w on 120v power and only rated for their "name plate capacity" when operating on 208v or 240v. If you are using 120v power, get a pair of 740w or higher 80+ platinum PSUs and you should be fine (this will not provide redundancy under full load). If you are using 208v power, you want a 1280w or 1600w PSU, or two for redundancy, under full load. Or you can again get by with 2 smaller PSUs without redundancy.

    ----------

    Ok, so that's what I've done for most of these. The "better 826 PDB", when you can find it, is between $30 and $100. It's a bit tricky to install into a 2u supermicro 825 chassis (requires use of a hammer in my experience) -- instructions are floating around this forum. If you don't want to use it with a chassis and just want to make use of a supermicro power supply, I guess that's an option. But the PDB's are hard to find, and the cables they give you aren't amazing, both in terms of the connectors you get and how much power you can draw from them.

    If anyone wants 1280w 80+ supermicro power supplies, I've got a ton of those I could sell you, or you can get them on ebay easily enough for under $100. Then each 8+8 cable is easily another $10, whether you're using the EPS -> 8+8, or you're using the 2x molex -> 8+8. Either way you slice it, it's not "cheap" compared to the HP power supplies + sketchy-looking pcie breakout boards you see crypto miners using. But if you can find the PDB, it is a step up from using desktop power supplies, in terms of wattage and 80+ rating vs price.

    ==

    One last thing:

    Another option (I --don't-- recommend) is the Supermicro PDB-PT747-4648 power distribution board. This is an upgrade for 4u supermicro chassis. And it does technically qualify as an upgrade -- the PDB that comes stock with a lot of these 4u chassis is terrible -- good luck powering more than 1x 1080ti on it. However, this PDB is not much better. It does have a large number of PCIe power cables -- no adapters needed. But it can't supply much power for some reason.

    If you use 2x 1600w 80+ platinum PSUs, both on 208v power, you can probably power 3 GPUs with this. That's 3200w of power supplies for a roughly 1000w maximum load!

    On 120v power, 1000w+ PSUs are derated to 1000w each, which means 2x 1280 or 2x 1600 should be able to supply 2000 watts -- good luck with that. The ~1000w draw of a 3-GPU server will cause reboots as soon as you engage the GPUs, even with 2000w of power supply capacity attached and running. On 120v power, a pair of 1280w or higher PSUs will probably be reliable to power 2 GPUs with this PDB.

    ==
     
    #9
  10. Fzdog2

    Fzdog2 Member

    Joined:
    Sep 21, 2012
    Messages:
    82
    Likes Received:
    11
    Wow, serious thanks for taking the time to write that up.

    My intended use is in a new 8 GPU box i ordered that will require 10 PCIe 6 or 8 pins.
    2x 6 pins to power the motherboard
    8x 6 or 8 pins to power the 8 GPUs, depending on if I put in GTX 1060's or GTX 1070ti's

    I'm thinking the 2 motherboard 6 pins could be powered using a 24pin to 2x 6pin adapter.
    The 2x 8 pin EPS plugs could do 3x 6 pin PCIe plugs each.
    The remaining 2 6 pins would come easily from molex adapters.

    Getting to 8x 8 pin PCIe plugs would probably not be as easy, or safe.
    GTX 1070ti's with 1x 8 pin are doing about 130w while mining. Is 260w too much for the EPS plug?
     
    #10
  11. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,368
    Likes Received:
    192
  12. Fzdog2

    Fzdog2 Member

    Joined:
    Sep 21, 2012
    Messages:
    82
    Likes Received:
    11
    I asked a supplier if the Supermicro PSUs were compatible with the widely available HP breakout boards and they said they were not.

    Unless you were saying to just buy an HP PSU and breakout board, which yeah if I can't get it going with the Supermicro's I will probably go that route.
     
    #12
  13. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    486
    Likes Received:
    210
    Yes, I was saying there is a good reason the HP psu's are popular in crypto. Seems the Supermicros, the manufacturer PDB's are not great and/or hard to get. Still, I don't know that I trust the chinese "crypto grade" HP breakout boards : P
     
    #13
  14. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    486
    Likes Received:
    210
    #14
  15. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    486
    Likes Received:
    210
    The GPU should draw 25w from the pcie slot, so you can subtract that. So a pair of 1070ti's will probably draw the same power from the 8-pins as a single 1080ti. I would expect the eps converter to power 2 of those cards. You'll need to test it to be sure. If there's one thing I learned about all this, is it's pretty rare that anything works the way you'd expect it to.
     
    #15
  16. DanP

    DanP New Member

    Joined:
    Jan 8, 2016
    Messages:
    10
    Likes Received:
    4
    It would be cool to see someone release some cheap breakout boards for SM supplies (like the HP ones). I get why the HP ones are so popular, the mass production of them / they where everywhere. We have hundreds if not a thousand left over 920/1200 plat sm supplies laying around so these would be handy. One of our guys are toying around with just rigging up a setting to pull 12v off it for the gpu side of things (reusing part of the HP breakout board).
     
    #16
  17. Godfr33

    Godfr33 Member

    Joined:
    Jan 20, 2018
    Messages:
    92
    Likes Received:
    16
    4U Superworkstation 900w Gold. 2 E5-2620

    1 Vega, rx580, 1060. 1 6pin powered riser hanging the rx in the back of the thermaltake 900.

    Going to get a 2Psu adapter and run a total of 6 cards.

    Add pics tomorrow!

    518 W at wall with killawatt meter
     
    #17
  18. Fzdog2

    Fzdog2 Member

    Joined:
    Sep 21, 2012
    Messages:
    82
    Likes Received:
    11
    #18
    StammesOpfer likes this.
Similar Threads: Supermicro Chassis
Forum Title Date
Cryptocurrency Mining and Markets Mining with Dual Socket Motherboard. Supermicro H11DSI-NT. Feb 25, 2018
Cryptocurrency Mining and Markets Ubuntu Nvidia Driver issue with Supermicro motherboard Aug 30, 2017

Share This Page