Need recommendation on server-grade pcie m.2 bifurcation card

[nutsnax]

I'm getting a little (and I do mean little) bit of budget to build a new database server for my company and I need something incredibly fast to deal with increasing demand that is coming for what this server is going to be doing. I will also have to ship this thing across the country so it has to run flawlessly. The current db server is running on old Dell hardware with a 7200 RPM SAS drive.

I'm looking for a bullet-proof PCI-E bifurcation card. This will be used on a Supermicro H12DSI motherboard most likely running ESXi 7. I was thinking of using the Supermicro x8 cards but I'd like to be able to also use the x16 slots for this as well.

Any input would be helpful thanks!

 

[Sean Ho]

the cheap ($60) consumer ASUS 4x m.2 bifurcation cards work just fine; they're all passive components and not much to break. Another option is an x16-to-4x8643 bifurcation card, plus cables to U.2. Are you planning on zfs raid10 (pool of mirrors)? If you want it to be "bulletproof" you need to plan on drive failure as well as card failure.

 

[ReturnedSword]

Keep in mind that hitting consumer/prosumer M.2 drives will increase failure rate by quite a bit. They’re just not designed for that kind of DWPD. Going up to 22110 enterprise drives will cost a pretty penny.

If there’s room in the chassis, U.2 drives may be much better. Would a Supermicro AOC-SLG3-4E4R work? Each x16 slot would be able to support 4 x U.2 drives that way.

 

[nutsnax]

the cheap ($60) consumer ASUS 4x m.2 bifurcation cards work just fine; they're all passive components and not much to break. Another option is an x16-to-4x8643 bifurcation card, plus cables to U.2. Are you planning on zfs raid10 (pool of mirrors)? If you want it to be "bulletproof" you need to plan on drive failure as well as card failure.

I was thinking of an asus card as there didnt seem like much to break and theyre readily available. I have such a low budget i probably cant run RAID-10. As far as drives the only prosumer drive i was contemplating was the firecuda 530. Otherwise i was going to try to get a PM9A3 but i dont know how to inexpensively mount 2.5" nvme in a tower chassis.

...Which leads me to the next issue... This will have to be in a tower chassis unfortunately... Plus my budget constraints mean that big 3u or 4u with a sas3/NVME backplane turned on its side and dressed up to look like a tower will likely cost too much unless im missing something.

is there a server-type bifurcation pci-e card (like an asus) that will do 2.5" nvme drives?

 

[nutsnax]

Keep in mind that hitting consumer/prosumer M.2 drives will increase failure rate by quite a bit. They’re just not designed for that kind of DWPD. Going up to 22110 enterprise drives will cost a pretty penny.

If there’s room in the chassis, U.2 drives may be much better. Would a Supermicro AOC-SLG3-4E4R work? Each x16 slot would be able to support 4 x U.2 drives that way.

Do you know of an inexpensive way to mount those 2.5" nvme drives in a tower chassis? Also im looking for pcie 4.0 and dont need RAID. Im going to run a SATA RAID-1 as the ESXi OS drive and for backups.

I think the best prosumer drive is likely the firecuda 530 from what i read and that is the only one id even contemplate using unless there are others? Otherwise I was going to go for PM9a3 2.5" or a Micron MAX series though the pm9a3 looks like the better value. Four Firecuda 530's on asus cards will be my "poverty-tier" solution if it comes to that. RAM is my main focus.

 

[nabsltd]

Do you know of an inexpensive way to mount those 2.5" nvme drives in a tower chassis?

Exactly how you would mount an SSD, assuming the chassis doesn't have a backplane, but has "bare" 2.5" mounting points.

You would then plug a cable like this into the U.2 connector on the motherboard or add-in card, connect an SATA power cable from your power supply to the SATA power on the cable, and plug the remaining connector into the U.2 drive. Note that this is just an example cable so you can see what they look like...they can be had more cheaply. You could also get something like this PCIe U.2 carrier if you have at least a 3U chassis.

Enterprise U.2 NVMe drives tend to be cheaper than the M.2 or PCIe card equivalent, at least second-hand on eBay, since they do require more work to plug in.

 

[nutsnax]

Exactly how you would mount an SSD, assuming the chassis doesn't have a backplane, but has "bare" 2.5" mounting points.

You would then plug a cable like this into the U.2 connector on the motherboard or add-in card, connect an SATA power cable from your power supply to the SATA power on the cable, and plug the remaining connector into the U.2 drive. Note that this is just an example cable so you can see what they look like...they can be had more cheaply. You could also get something like this PCIe U.2 carrier if you have at least a 3U chassis.

Enterprise U.2 NVMe drives tend to be cheaper than the M.2 or PCIe card equivalent, at least second-hand on eBay, since they do require more work to plug in.

Thanks i didnt realize those existed.

Now i need a small-ish tower case with redundant power supply that will handle dual epyc.

 

[Sean Ho]

U.2 is great and works just fine without backplane as nabsltd described. There are also plenty of older enterprise 22110 drives that fit in the ASUS Hyper card; I have a few Timetec EPX and PM983 (predecessor to PM9a3).

 

[ReturnedSword]

It depends if you want to mount the U.2 in nice hotswap trays, in which case Icy Dock makes a line of U.2 to 5.25” hotswap adapters. They are an expensive brand though. If you don’t need to hotswap, then simple 3.5” or 5.25” to 2.5” brackets will work. It may be useful to make sure the drives have some airflow over them.

 

[nutsnax]

U.2 is great and works just fine without backplane as nabsltd described. There are also plenty of older enterprise 22110 drives that fit in the ASUS Hyper card; I have a few Timetec EPX and PM983 (predecessor to PM9a3).

I think i want to go the pm9a3 route though my main focus now is getting a super inexpensive chassis with redundant power supplies. I was looking at Intel 4000 type chassis but unsure if an epyc board will bolt in.

 

[nabsltd]

I was looking at Intel 4000 type chassis but unsure if an epyc board will bolt in.

Having gotten a deal on an Intel 4000 chassis, I absolutely do not recommend them unless you like a lot of tinkering.

• The fans use 6-pin connectors that will not work if the motherboard fan headers are too cramped. Also, the cables are short, and you can't use extensions because of the 6-pin connectors. I bought this fan plug kit and replaced all the 6-pin connectors with standard 4-pin. The extra two pins are for the fan fail LED on the hot-swap carrier. Since this will never work unless you have a motherboard with BMC support for the extra pins, it's perfectly safe to just clip the wires and toss them.
• The fans draw so much power that you cannot plug more than one fan into a single header. So, unless you have at least 6 fan headers (5 for the fans and one for CPU cooler), you'll have to use some kind of splitter with power insertion.
• The fans are 80x80x35mm and they are held in place by pressure alone, so you can't just buy replacement 80x80x25mm fans. And, even if you do something like 3D print a spacer, the fans are hard-wired into the hot-swap bracket connector. It is bizarre that it requires no tools to insert/remove the fan from the hot-swap carrier, but you need very special tools to actually connect/disconnect the wires.
• The chassis connector for the motherboard power/reset/HDD LED/etc. is essentially Intel proprietary. Technically, it's an SSI standard, but nobody but Intel seems to use it, so you'll have to either re-wire the connector to match your motherboard, or build a converter cable.

But, it will accept any ATX standard motherboard up to E-ATX size.

 

[nutsnax]

Having gotten a deal on an Intel 4000 chassis, I absolutely do not recommend them unless you like a lot of tinkering.

• The fans use 6-pin connectors that will not work if the motherboard fan headers are too cramped. Also, the cables are short, and you can't use extensions because of the 6-pin connectors. I bought this fan plug kit and replaced all the 6-pin connectors with standard 4-pin. The extra two pins are for the fan fail LED on the hot-swap carrier. Since this will never work unless you have a motherboard with BMC support for the extra pins, it's perfectly safe to just clip the wires and toss them.
• The fans draw so much power that you cannot plug more than one fan into a single header. So, unless you have at least 6 fan headers (5 for the fans and one for CPU cooler), you'll have to use some kind of splitter with power insertion.
• The fans are 80x80x35mm and they are held in place by pressure alone, so you can't just buy replacement 80x80x25mm fans. And, even if you do something like 3D print a spacer, the fans are hard-wired into the hot-swap bracket connector. It is bizarre that it requires no tools to insert/remove the fan from the hot-swap carrier, but you need very special tools to actually connect/disconnect the wires.
• The chassis connector for the motherboard power/reset/HDD LED/etc. is essentially Intel proprietary. Technically, it's an SSI standard, but nobody but Intel seems to use it, so you'll have to either re-wire the connector to match your motherboard, or build a converter cable.

But, it will accept any ATX standard motherboard up to E-ATX size.

Do all of the ATX/E-ATX holes line up properly?

Thanks for the fan info. I just got some Dell fans that I re-pinned to match standard 4-pin PWM.

Also is there a 2.5" SAS-3 cage available that you know of? Thats probably asking way too much

I would really prefer a redundant power supply in this box.

 

[nabsltd]

Do all of the ATX/E-ATX holes line up properly?

I had no issue with a Supermicro motherboard.

Also is there a 2.5" SAS-3 cage available that you know of?

Spare 8x2.5 Hot-Swap Drive Cage Kit FUP8X25HSDKO (in 5.25in cage area) - Product Specifications | Intel

Spare 8x2.5 Hot-Swap Drive Cage Kit FUP8X25HSDKO (in 5.25in cage area) quick reference with specifications, features, and technologies.

www.intel.com

There is no expander on this cage, so it should work fine with SAS3.

 

[nutsnax]

I had no issue with a Supermicro motherboard.

Spare 8x2.5 Hot-Swap Drive Cage Kit FUP8X25HSDKO (in 5.25in cage area) - Product Specifications | Intel

Spare 8x2.5 Hot-Swap Drive Cage Kit FUP8X25HSDKO (in 5.25in cage area) quick reference with specifications, features, and technologies.

www.intel.com

There is no expander on this cage, so it should work fine with SAS3.

Thanks for that info i appreciate it.

Looking at the fan connectors in one listing it looks like they might not be a standard 4-pin fan pinout.

Also i do see the issue with the power and reset button connectors...... That is really lame! Im going to have to find a different case

 

[nabsltd]

Also i do see the issue with the power and reset button connectors...... That is really lame!

I'm right in the middle of re-pinning now. I have pulled and cut all the truly useless wires, and marked each remaining wire with its pin position number using binary...red for a one and black for a zero, so 5 marks on each wire. Tonight, I'll pull all the remaining wires, then put them back so I can connect to a Supermicro motherboard.

 

[nutsnax]

I'm right in the middle of re-pinning now. I have pulled and cut all the truly useless wires, and marked each remaining wire with its pin position number using binary...red for a one and black for a zero, so 5 marks on each wire. Tonight, I'll pull all the remaining wires, then put them back so I can connect to a Supermicro motherboard.

This box would be going across the country and I may not even be building it so I'd need something that just worked