SMCI NVMe Rear Hot Swap Drive Kit - Any Experience with Thermals?

[svtkobra7]

I currently have the non-NVMe / 12G iteration of the Supermicro Rear Hot Swap 2 x 2.5" NVMe kit and would like to swap in the NVMe variant, but have some concerns over what NVMe drive temps would look like. Two questions for the community:

• I'm curious if anyone has NVMe drive temp figures for the Supermicro MCP-220-82619-0N, and if so what does your cooling scenario look like (x fans @ y RPM, etc)?
• What is the best way to connect the MCP-220-82619-0N (2 x Oculink) to an X9 board? [There are a number of ways (SFF-8643 PCIe adapter w/ or w/o switch+ Oculink to SFF-8643 cable | Oculink PCI adapter + Oculink cable, etc. - haven't reviewed various options for cost effectiveness yet)]

Unfortunately, I'm not currently using either existing kit, so I don't even have a "baseline" SSD temp, off which I could try to guess what the thermals for what higher wattage NVMe drives could look like. A bit more below (as to why I ask) ...

1. Both servers have 3 x FAN-0126L4 (7000 RPM) / 72.5 CFM / 1.09 in H20 static pressure) in them behind the HDD backplane (and also both have 2 x 2U SMCI active CPU coolers, if I'm being thorough). Fan mode (X9 board) is controlled via script based on max 3.5" HDD temp (of 12), but even at a high ambient (78F), Optimal keeps HDD temps under 40C. I prefer to stay at Optimal as Standard becomes audible and Full is untenable (the poor servers are mounted in a condo "server closet" and upside down at that - they must think I hate them). I don't want NVMe drives added in this manner to be considered in fan mode control and want to ensure they will be sufficiently cooled @ Optimal (which I suspect may be a pipe dream).

2. Looking at the design of the kit, I fail to see how the cutout in the backplane can provide sufficient airflow for NVMe drives (backplane is nearly identical for NVMe and non-NVMe kits) ...

This cutout provides enough air flow for two 18W - 25W NVMe drives ???


Further the engineering looks whack - it hits one at the top / one at the bottom of drive ...


The air shroud would aid air flow a bit ... but enough?


[ I posted this query in the STH Main Site Posts forum (Supermicro SuperStorage SSG-5029P-E1CTR12L 2U Storage Server Review); however, it may be more prudent to ask the question here ... ]

 

[Aestr]

I have a few of these sitting around that I've used before. I'll share what I can.

Temperature Experience:

• SC826 Chassis with 4x80mm FAN-0104L4 low noise fans running around 3,000 RPM
  • Removed stock fanwall to allow for 4 fans
• 2 x P3605 1.6TB getting standard homelab type usage as datastores
• As expected they ran warm. I'm not currently using them but my recollection is in the mid 40's. As seen above my setup was for balanced cooling/noise so not a best case scenario.

Connectivity Experience:

• OCuLink cable connected to a X11SPH-nCTF worked
• OCuLink cable connected to AOC-SLG3-4E4T worked (This card will not work with all motherboards)
• OCuLink -> MiniSAS HD cable connecter to AOC-SLG3-2E4 did not work
  • This card is works connected to MiniSAS HD NVME backplane

 

[svtkobra7]

I have a few of these sitting around that I've used before. I'll share what I can.

• I really appreciate you weighing in with your experience here - thank you!

Temperature Experience:

• SC826 Chassis with 4x80mm FAN-0104L4 low noise fans running around 3,000 RPM
  • Removed stock fanwall to allow for 4 fans
• 2 x P3605 1.6TB getting standard homelab type usage as datastores
• As expected they ran warm. I'm not currently using them but my recollection is in the mid 40's. As seen above my setup was for balanced cooling/noise so not a best case scenario.

• Mid 40s is reassuring actually, especially given lower RPM / CFM / static pressure fans in your scenario (vs. mine = FAN-0104L4).
  • Actually, its not just reassuring - I'm impressed - as my first SMCI build, an 836, shipped with 3 x FAN-0094-L4s mid-chassis and it took me forever to figure out the reason the chassis could barely hold the HDDs at 40C (78F ambient) at 100% duty was because they were rated @ 0.85 in H20 vs. the 126s @ 1.09 in. H20 (which it should have shipped with).
  • And there is a massive difference between the '94 @ 0.14 in. H20 vs. '126 @ 1.09 in H20 and since we are referring to the ability to provide airflow through such a tiny slice in the NVMe kit backplane, I would have thought that rating to be of utmost importance.
• Questions: (a) Which 826 model, i.e 2.5" or 3.5"? (b) Did you leave the air shroud in place?

Connectivity Experience:

• Again, your reply is really appreciated, especially the thoroughness you go into here.

Given your direct experience with the kit and the fact that you aren't afraid to mod your chassis config, I'm curious for your feedback (if you care to share) for an idea I came up with after I posted my initial query ...

• In my case, hot swap is a "nice to have" but by no means requisite.
• Further, with a dated X9 motherboard, Oculink is more of a PITA than anything.
• Considering those points and the fact I'd need to run 2 cables to PCIe slots anyway ... my idea = essentially convert the existing hot swap kit into a "drive cage" (and nothing more).

1. Keep the 12G kit in place.
2. Remove the backplane.
3. Connect power and U.2 connectors to each drive.

Advantages:

• Entire surface area of the "interface side" of the drive would now be exposed to airflow = lower drive temps.
  
• Save a few bucks by reusing existing 12G kit. Save a few additional bucks by not needing Oculink cables (I think they carry a slight premium).

Disadvantages:

• Loss of hot swap ability. Chassis lid would need to be opened to remove a drive.
• Additional cabling required (power cables to each drive, but I already have a big 4 pin plugged into the existing kit o/c, so I would be easy to add a Y adapter to that).

 

[Aestr]

Questions: (a) Which 826 model, i.e 2.5" or 3.5"? (b) Did you leave the air shroud in place?

12x3.5 and I do not use the air shroud

my idea = essentially convert the existing hot swap kit into a "drive cage" (and nothing more).

Other than the disadvantages that you list I don't see a reason not to. As you point out you're really just creating a static mounting point once you remove the backplane.

Hope it works out well for you!

 

[svtkobra7]

Hope it works out well for you!

Many thanks for your assistance - I'll circle back with some reasons.

I forgot to ask previously - apologies - info on the SAS2/3 backplane kits seems to be much more readily available and its very easy to discern what cables come with that kit ... may I ask what, if any, data cables this kit comes with, please?

Example on the SAS3 kit: https://store.supermicro.com/mcp-220-83608-0n-rear-drive-kit.html

 

[Aestr]

I'm pretty sure that no cables come with the kit(power or OCuLink). It's actually very weird to buy as they randomly included instructions for the SAS rear drive kit for the SC846 which isn't even the same form factor. I'm guessing this is still a low volume part for them so they don't have the process ironed out.

 

[svtkobra7]

Connectivity Experience:

• OCuLink cable connected to a X11SPH-nCTF worked
• OCuLink cable connected to AOC-SLG3-4E4T worked (This card will not work with all motherboards)
• OCuLink -> MiniSAS HD cable connecter to AOC-SLG3-2E4 did not work
  • This card is works connected to MiniSAS HD NVME backplane

Any clue why the Oculink to mini-SAS SFF-8643 didn't work? I would think it would given it has the PLX chip ... (also, I wasn't quite sure I followed the comment about working when connected to "MiniSAS HD NVME backplane" - but I'm sure it is me and not having exposure to the tech)

If you have the spare PCIe slots ... is there any reason to discard the SMCI AOC solutions and simply get one of the below Oculink to PCIe adapters? (other than being decidedly horrible to look at!) At roughly half the cost of the AOC-SLG3-2E4 - which from my understanding of your message wouldn't work anyway - it seems like a winner to me?

Note: I think I'm limited to AOC cards with PLX chips as I'm still unsure if my X9 board supports bifurcation ...

PCIe Gen 3/4 Lane to Oculink SFF-8612 Adapter

 

[azev]

Note: I think I'm limited to AOC cards with PLX chips as I'm still unsure if my X9 board supports bifurcation ...

I have a few different X9 motherboard and since the latest 3.3 bios update, they all seems to support bifurcation in the bios.
However I have not tried using any of those PCI express break out card from supermicro so I am not sure if it actually works.