Sanity Check: Advice on Next Server Steps

[Pete.S.]

What's there to be cautious about? I don't understand the sentence about M.2 and Optane being an oxymoron. I have 5 PCIe slots. 1 is for a 40g NIC (x8), 1 is for a GPU (x16), 1 is for an HBA (x8), and 1 holds 4 SSDs (x16).

I could use literally any drive to test this, but I'm thinking to move over 84+ TB of data just to see how large my metadata drive needs to be is a waste. And I couldn't just use any U.2 drives. I have no solution to mount them anywhere.

You should be cautious about not getting the performance you think you will get.
I was just assuming you wanted maximum performance since you're talking about Optane drives.

You have two bottle necks.

• M.2 can't to do sustained heavy workloads because of lack of heat dissipation causing it to throttle down performance.
• 4 x M.2 drives that each uses 4 PCIe lanes, mounted on a PCIe 8x card can only reach half of it's potential transfer speed.

Why not just order the NVMe drive rear option? If it's vaporware or out of stock and not manufactured anymore you wont get it.
And if you get it, your problem is solved.

 

[ramicio]

You should be cautious about not getting the performance you think you will get.
I was just assuming you wanted maximum performance since you're talking about Optane drives.

You have two bottle necks.

• M.2 can't to do sustained heavy workloads because of lack of heat dissipation causing it to throttle down performance.
• 4 x M.2 drives that each uses 4 PCIe lanes, mounted on a PCIe 8x card can only reach half of it's potential transfer speed.

Why not just order the NVMe drive rear option? If it's vaporware or out of stock and not manufactured anymore you wont get it.
And if you get it, your problem is solved.

The M.2s would be in a server chassis with air blowing over them, and they aren't going to be doing sustained heavy workloads, so not concerned there. As far as I'm aware, yes and no on your second claim. If all 4 drives together exceed what a PCIe x8 bus can do, then the PCIe x8 bus will be the limiting factor. They aren't magically going to have their performance cut in half just because of a PLX in the middle. For example, if the drives were rated for 100 MB/s, they aren't going to get cut down to 50 MB/s while the x8 bus (3.0) can do 7.88 GB/s. And they would be for metadata. It would still be tons faster than having that on spinning disks, especially with the factor of latency more than raw throughput.

Why not order it?
1. It's obviously not available in my country.
2. No idea how to even connect it even if it existed here. Because they deny it exists, thus there is zero documentation about it, so I have no specs on anything with this issue.

So how am I supposed to mount two drives?

And this is all besides the point. How am I supposed to determine how much ZFS needs for metadata? I haven't found ANY info on how to even check how much storage metadata is using. ZERO info. Regardless, can it even be transferred to SSDs AFTER the fact that I build the pool?

 

[Markess]

I provided them with part numbers as well. They replied to me twice that they don't make any rear NVMe rear cage kits. So now I am trying to figure out what else to do.

Not to take the thread even more off track, but thought I'd ask what department at Supermicro you contacted concerning these parts?

I know from experience that for chassis option parts, going through the parts department/eStore will often yield no results. There's several folks on the fourms that have successfully gone through the RMA department with a part number and were able to purchase the parts they needed. I tried it myself a couple months back for a chassis rear window and they got me a quote in a day. Sounds odd to go through RMA, but if it works it works.

Sometimes you need to get creative though, and some folks have recommended that you tell them you own a system/chassis that lists the part as an option and you now want that option part. Its worked for others here before, but YMMV.

Searching the two part numbers for Rear NVMe that @Pete.S. listed above, I got some hits on them as optional parts for several systems/chassis on Supermicro.com. For example:




Here's a few other hits:

MCP-220-82619-0N 2x 2.5" hot-swap tool-less NVMe U.2 rear kit for 216B/826B/847B w/o cables

• SC847BE1C12-R1K68LPB4 | 4U | Chassis | Products | Supermicro
• SSG-540P-E1CTR36H | 4U | SuperServer | Products | Supermicro
• SYS-520P-WTR | 2U | SuperServer | Products | Supermicro
• and more...

MCP-220-82617-0N 2x 2.5" hot-swap NVMe U.2 rear kit for 216B/826B/847B w/o cables

• Super Micro Computer, Inc. - SYS-7089P-TR4T Storage Options
• Super Micro Computer, Inc. - SYS-7088B-TR4FT Storage Options

Cheers.

 

[ramicio]

Ok, thank you to all for your knowledge here. I have ordered the MCP-220-82619-0N off of eBay. One card I found that I believe will work is the AOC-SLG3-2E4T. I will have to power down tomorrow and pop it open to get a rough idea of how long of the OcuLink cables I will need.

As far as storage, this is off topic, but if someone could help me, maybe through PMs? One test I did do was to copy 1/100 of my data, from sizes small to large, to a ZFS pool on a second server set up with a metadata special device. Zdb shows 1.38G (or 1.48 in decimal), so I'd figure multiply that by 100 to get what size devices I'd need to store the entire set of data.

Also, does anyone have info on the max height of drives that will fit in this cage? Thank you.

 

[Pete.S.]

Also, does anyone have info on the max height of drives that will fit in this cage? Thank you.

The 2.5" NVMe trays are physically the same size as the 2.5" SATA/SAS trays. So all thickness of 2.5" disks fit.

I think the standard for 2.5" drives says up to 19mm thickness (about 3/4").

 

[ramicio]

The 2.5" NVMe trays are physically the same size as the 2.5" SATA/SAS trays. So all thickness of 2.5" disks fit.

I think the standard for 2.5" drives says up to 19mm thickness (about 3/4").

Excellent, thank you.

 

[Pete.S.]

The 2.5" NVMe trays are physically the same size as the 2.5" SATA/SAS trays. So all thickness of 2.5" disks fit.

I think the standard for 2.5" drives says up to 19mm thickness (about 3/4").

@ramicio
Sorry, I have to correct myself here.

All thickness of currently available 2.5" drives fit. But it's 15mm thickness max. FAQ Entry | Online Support | Support - Super Micro Computer, Inc.

The form factor standard actually allows up to 19mm. https://members.snia.org/document/dl/25851

The U.2 drives that are older are 15mm. For example Intel DC P3520.
Some of the newer drives on the market are thinner, for example 7mm thickness on Samsung PM9A3 U2.

I've never heard of any drives that are thicker than 15mm. Dell and others also have 15mm max thickness in their bays.

 

[ramicio]

@ramicio
Sorry, I have to correct myself here.

All thickness of currently available 2.5" drives fit. But it's 15mm thickness max. FAQ Entry | Online Support | Support - Super Micro Computer, Inc.

The form factor standard actually allows up to 19mm. https://members.snia.org/document/dl/25851

The U.2 drives that are older are 15mm. For example Intel DC P3520.
Some of the newer drives on the market are thinner, for example 7mm thickness on Samsung PM9A3 U2.

I wound up ordering two 960GB U.2 15mm drives. They'll fit then.

 

[ramicio]

This is all a bunch of nonsense. No, the kits for the 826 and 836 do not use the same backplanes. The 826 kits are shorter in height. I ordered the MCP-220-82619-0N and it DOES NOT fit the 836 chassis. It's about 1/4" too wide and probably 3/8" too short in height. Now what the hell am I supposed to do? How am I supposed to mount two U.2 drives in this chassis?????? How?????

 

[Pete.S.]

This is all a bunch of nonsense. No, the kits for the 826 and 836 do not use the same backplanes. The 826 kits are shorter in height. I ordered the MCP-220-82619-0N and it DOES NOT fit the 836 chassis. It's about 1/4" too wide and probably 3/8" too short in height. Now what the hell am I supposed to do? How am I supposed to mount two U.2 drives in this chassis?????? How?????

Well, that sucks! Sorry to hear that. Obviously someone was mistaken saying the 826 and 836 are the same in the back.

I'm trying to think of solutions to mount the U.2 drives but...

• You use too many PCIs slots already to put the two U.2. drives on two PCIe adapter cards. We have that on two or three servers and that works well.
• Your 836 chassis has no empty 3.5" or 2.5" bays without a backplane, only a slim DVD I think. So you can't mount the U.2 drives in regular non-hotswap bays and connect with cables. I've done that on the 825 chassis and it works well.
• You don't want to screw anything in your chassis so you can't mount them on a 2x2.5" bracket inside. Usually it's easy to find a spot somewhere and there is always the lid as a last resort.
• It a hack like the previous point so it's not for you but you could potentially order a SATA rear kit for your chassis that fits mechanically. Unscrew the sata backplane and connect the U.2 drives directly with cables.
• You said you want to keep your 836 chassis otherwise you could upgrade to another chassis where Supermicro says your rear kit fits - 216B, 826B or 847B. Or just get another server that has NVMe slots built in. Move over your drives, cards, memory and CPUs.

Sorry but I can't think of any other options.

 

[ramicio]

Well, that sucks! Sorry to hear that. Obviously someone was mistaken saying the 826 and 836 are the same in the back.

I'm trying to think of solutions to mount the U.2 drives but...

• You use too many PCIs slots already to put the two U.2. drives on two PCIe adapter cards. We have that on two or three servers and that works well.
• Your 836 chassis has no empty 3.5" or 2.5" bays without a backplane, only a slim DVD I think. So you can't mount the U.2 drives in regular non-hotswap bays and connect with cables. I've done that on the 825 chassis and it works well.
• You don't want to screw anything in your chassis so you can't mount them on a 2x2.5" bracket inside. Usually it's easy to find a spot somewhere and there is always the lid as a last resort.
• It a hack like the previous point so it's not for you but you could potentially order a SATA rear kit for your chassis that fits mechanically. Unscrew the sata backplane and connect the U.2 drives directly with cables.
• You said you want to keep your 836 chassis otherwise you could upgrade to another chassis where Supermicro says your rear kit fits - 216B, 826B or 847B. Or just get another server that has NVMe slots built in. Move over your drives, cards, memory and CPUs.

Sorry but I can't think of any other options.

Mounting a single U.2 drive to a PCIe card is a waste of an x8 slot. Plus, any cards out there are dubious.

I already have a SAS/SATA rear drive kit. It's nowhere close to the same. It's like half of a drive taller, and part of that difference is built between the drives, so the backplane's connector spacing is way off from what this NVMe kit is. This NVMe kit is probably the thickness of the case's sheet metal wider and it just won't slide in past about an inch. There's no way to make it work without doing some drilling, cutting, and welding of paper-thin metal.

There is nowhere inside my case to mount 2.5" drives. I'm not going to just leave them sitting loose up in the area where a slim optical drive would go. I might as well just stop this whole game and build a server in some gamer tower with gamer hardware.

There is absolutely no room inside to mount this kit.

Yes, I want to keep the 836. I don't want a 2u or a 4u. I only need 2 drives, not a server that has all drive bays NVMe (which would be 2u).

I just want to know what this MCP-220-82618-0N part from the dubious presentation really is. Can it be found? I'll buy it. These "new product brief" things I found were all Russian. I'm willing to bet that I can't find anything in a search on that part because there's a Western embargo on all things Russian right now (search results omitted, obviously not going to be able to buy directly from there right now). There has to be NOS of them somewhere there that someone could import for me.

 

[ramicio]

timramich.com/20230224_215636.jpg
timramich.com/20230224_215640.jpg
timramich.com/20230224_215653.jpg
timramich.com/20230224_215702.jpg
timramich.com/20230224_215721.jpg

 

[ere109]

To take back over MY thread, the new X11 Scaleable arrived this week. This first photo is of my current "second" server with a stable dual X9 motherboard inside. E-ATX boards do fit so well inside these 836 chassis, but server time marches on. I need to get a picture of my new X11 board in this chassis.


A picture of both chassis mounted inside my half-rack. The top chasis is my production machine. Note that I was able to fit three SSD drives in the slim DVD and floppy slots, by removing the serial/usb panel.


I began a major upgrade over the holidays, and am more than halfway through. I've got two 5-drive pools, two SSD pools, and the rear 2x2.5" slot housing my boot drives.
And a shot of my bezels on, for nice clean lines:

@ramicio, would you mind posting some photos of the rear of your chassis with the 826 2x2.5" There may be a way to make them work. If not, I've seen the listings for the NVME drives, as well, and believe they're really out there - though possibly harder to find.

 

[Pete.S.]

I already have a SAS/SATA rear drive kit. It's nowhere close to the same. It's like half of a drive taller, and part of that difference is built between the drives, so the backplane's connector spacing is way off from what this NVMe kit is. This NVMe kit is probably the thickness of the case's sheet metal wider and it just won't slide in past about an inch. There's no way to make it work without doing some drilling, cutting, and welding of paper-thin metal.

I understand. What I meant was that:

1. Take out the 4? screws and remove the sata backplane on your old rear drive kit.
2. Mount the U2. drives in your SATA or NVMe bays (they should fit) and insert into the old drive kit. Now your drives back end should be exposed.
3. If you don't have one, get a Oculink to NVMe drive cable and connect to the drives. Supermicro has them.

You won't have the NVMe backplane and it's a little inconvenient but you can hook up the drives without it.

 

[i386]

I feel bad because I was the one who said that the rear kits share the same backplanes ._.

My information/assumptions are based on pictures of the rear kits in some shops (this information could be inaccurate due stock photos etc) and pdfs from supermicro.
826/216 rear kit MCP-220-82616: https://www.supermicro.com/manuals/chassis/drive_kit/MCP-220-82616-0N.pdf
836 rear kit MCP-220-83608: https://www.supermicro.com/manuals/chassis/drive_kit/MCP-220-83608-0N.pdf
Both of these kits use the BPN-SAS3-826TQ-B2B backplane.

I assumed from that that the other bacckplanes have the same dimensions & mounting points

 

[ramicio]

To take back over MY thread, the new X11 Scaleable arrived this week. This first photo is of my current "second" server with a stable dual X9 motherboard inside. E-ATX boards do fit so well inside these 836 chassis, but server time marches on. I need to get a picture of my new X11 board in this chassis.
View attachment 27373

A picture of both chassis mounted inside my half-rack. The top chasis is my production machine. Note that I was able to fit three SSD drives in the slim DVD and floppy slots, by removing the serial/usb panel.
View attachment 27374

I began a major upgrade over the holidays, and am more than halfway through. I've got two 5-drive pools, two SSD pools, and the rear 2x2.5" slot housing my boot drives.
And a shot of my bezels on, for nice clean lines:
View attachment 27375
@ramicio, would you mind posting some photos of the rear of your chassis with the 826 2x2.5" There may be a way to make them work. If not, I've seen the listings for the NVME drives, as well, and believe they're really out there - though possibly harder to find.

I know they make a kit for the optical drive area and the floppy area, but I wasn't aware they made one for the USB/serial area. Or does the optical drive one fit there as well? That's cool.

My chassis is an 836 and this kit for the NVMe won't even fit into the hole under the power supplies, so there's no picture I can provide. The pictures I did provide compare the NVMe kit with the SAS/SATA kit made for the 836. It looks like it would fit, but it doesn't. Even if I ground away those 2 pressed standoffs near the front and cut off that little tab on the side, it wouldn't. It's too wide only by the thickness of the metal itself. The only way this could ever go in is to drill out those 3 rivets on the side, cut off the tabs that stick up from the bottom, and somehow weld that thin sheet metal together.

 

[ramicio]

I understand. What I meant was that:

1. Take out the 4? screws and remove the sata backplane on your old rear drive kit.
2. Mount the U2. drives in your SATA or NVMe bays (they should fit) and insert into the old drive kit. Now your drives back end should be exposed.
3. If you don't have one, get a Oculink to NVMe drive cable and connect to the drives. Supermicro has them.

You won't have the NVMe backplane and it's a little inconvenient but you can hook up the drives without it.

The whole point of having enterprise stuff like this is to have hot-swap stuff that doesn't need cabling to be messed with. To be able to keep the machine on and not have to turn it off because you have to pop lids, go inside and pop out fans and air ducts, etc. I might as well just stuff this all in some gamer case with gamer hardware at that point. Plus, that area of the case is absolutely infuriating to do anything to.

 

[ramicio]

I feel bad because I was the one who said that the rear kits share the same backplanes ._.

My information/assumptions are based on pictures of the rear kits in some shops (this information could be inaccurate due stock photos etc) and pdfs from supermicro.
826/216 rear kit MCP-220-82616: https://www.supermicro.com/manuals/chassis/drive_kit/MCP-220-82616-0N.pdf
836 rear kit MCP-220-83608: https://www.supermicro.com/manuals/chassis/drive_kit/MCP-220-83608-0N.pdf
Both of these kits use the BPN-SAS3-826TQ-B2B backplane.

I assumed from that that the other bacckplanes have the same dimensions & mounting points

You know what? You're not wrong. My kit is the MCP-220-83605-0N. It's been discontinued for some time. Looking at the MCP-220-83608-0N, it looks to bring the drives together and have a smaller backplane (versus having the drives spaced apart on the old 83605). Ugh, more purchasing, because I think that cage may be the ticket to put this NVMe backplane onto. Rant: they make the cages for SAS, but the cages don't care what drives are in there. They make the backplanes for NVMe, and they look like they'd fit the cages. The drive sleds all interchange. So why don't they just sell the NVMe kit for the 836? Just combining parts they already make.

 

[ere109]

Or does the optical drive one fit there as well?

Yes, the optical drive model fits in the center. I had to grind the rivets down a bit before it would slide in, but it's an exact fit otherwise. I also built a USB header that I was planning to attach to the front, but never got around to it.
The difference between the 83605 and 83608 is SAS2 vs SAS3 backplane, if my research stands true.
If you do decide to try the 83608, please update this thread, as I'm interested in the Oculink/NVME 2x2.5 option, as well. If you were to buy the 83608 and find that it didn't work, I might be interested in it for my chassis.

 

[ramicio]

Yes, the optical drive model fits in the center. I had to grind the rivets down a bit before it would slide in, but it's an exact fit otherwise. I also built a USB header that I was planning to attach to the front, but never got around to it.
The difference between the 83605 and 83608 is SAS2 vs SAS3 backplane, if my research stands true.
If you do decide to try the 83608, please update this thread, as I'm interested in the Oculink/NVME 2x2.5 option, as well. If you were to buy the 83608 and find that it didn't work, I might be interested in it for my chassis.

I ordered one yesterday (Saturday). I am pretty sure it will fit the NVMe backplane. I cut apart the 82619 (for 826 cases) up to try to make it fit and it doesn't really. I got it to slide in enough to work temporarily until the 83608 comes. I'll try to snap photos of some stuff then.