....Getting Into More Trouble....

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
Because I have yet to remove the vertical 2U struts and implemented a method to properly support the upper tray, I went ahead and test fit the guts from the GS7200 on the upper shelf.. Not sure you could ask for a better fit:




With the amount of available, I likely either build a secondary enclosure into the lower 2U portion of the SC846 to house the switch, or create a two level shelf to house the Gnodal PSU's cleanly on a separate level.

It would appear that there will be plenty of clearance to implement new airflow ducting and an actively cooled heat sink for the chip to suit the new enclosure... I'm thinking a Noctua with a custom mounting bracket?
 
  • Like
Reactions: Logan and whitey

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
At the same time, it would also be nice to reduce the height of the tray to create a 3U on 1U configuration... this would allow for better primary processor cooling.

A third alternative considering that the mid section of the chassis floor already has standoffs installed might be to go with a small form factor motherboard, leave the GS7200 up top, and install a 24 2.5" disk backplane in the lower back 2U area... This would get the chassis to 72 hot swappable disks... The only caveat would be relocating the PSU's.

Norco 120mm fan walls and Noctua IPPC 3000 RPM fans should be arriving shortly.

This build is probably going to get the X10DAC's and 2x 2696V4's.... Now what to do with the X10DRU-i+ I managed to grab today.. Would have to take some measurements, but maybe I could stuff it into this chassis.. with the correct ultra and add on cards it should easily support 24x NVMe drives. A custom back plate would be necessary to accommodate the riser cards though (I suppose we'll already be building one for the switch; might as well throw a little extra material on the CNC machine if we could make it work cleanly.
 
Last edited:

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
Hmm.. So here are the NVMe related parts lists from Supermicro.com I really want to know more about the NVMe re-timer (AOC-SLG3-4E4T-O) which is included in the 48x system but not the 24x system. These have a PLX chip and are only about $250.

 

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
You know we've all been waiting for an update...give it up!!!
Haven't had a chance to do much with it yet today.. I really have two separate discussions going on in this thread lol...

Discussion one is putting the Gnodal GS7200 internals in the SC847 chassis.
Discussion two is implementing 24x into a single case (and my X10DRU-i+ MB)

The X10DRU-i+ could never be mounted on the motherboard tray in this case due to the PSU's... I haven't measured, but I doubt it could go into this case at all (even mounted below the tray in the lower 2U section); this was never my intent. I was, however, surprised by how closely the S2600WT2 and X10DRU-i+ motherboards lined up physically.. Having just finished up an SW2600 build a few days ago and being really impressed with the quality of the R2600WTXXX case (new off ebay for $250), this was intriguing.
 

Rand__

Well-Known Member
Mar 6, 2014
4,549
901
113
It it's simply a pass through, it should be. See the end of post #1 where the Supermicro parts list and the NVMe backplane manual seem to use the SAS and NVMe model numbers interchangeably (they've just tacked on NVME capable expanders).
Have you ever been able to determine whether this assumption was correct?
TQ might be an issue due to missing PCI-e capable cables I'd guess and I have not seen a SAS3 (HD connector) -A version yet
Edit: just saw those -A on a 'great deals' of you - question is where to get pcie capable SAS HD-SAS HD cables from...
 
Last edited:

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
Rand,

Yes... I totally missed the boat on the original post. As you pointed out, the tq backplanes are not pcie capable (more pins on the pcie sff-8639 connector relative to Sata/SAS).

The confusion came from the SAS backplane listed in the manual which is for the rear two drives rather than the primary backplane. I was learning ;)

After scanning this thread again a few days ago I was inspired to do something with the X10DRU-i and 847 chassis I still had laying around. Custom DRU-i/847 is about 70% done ✅

I have a number of NVMe related cards and other misc laying around at this point, so I was thinking I might temporarily use this machine as a compatibility test bench. Would like to pick up the SM x16 oculink retimer and a couple of the new AOC’s as well.

It looks like Intel has now released their nvme capable pcie switch/oculink cards, HP has some new stuff out, and the Sun/Oracle x12 drive 2U NVMe pass though backplanes I’ve seen floating around lately are looking mighty tempting..

In theory you could get quite a bit more bandwidth out of the direct attach backplane configuration assuming you had the lanes available, no?
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
4,549
901
113
Ok,
some more info about the various option surly would be welcome:)
And yes, if there was proper cableing then would think the A might still be capable of providing nvme support-
you might be able to test this on the Intel Server of yours - use the NVME cable on a non nvme capable drive slot and attach to the nvme card.
The intel backplane seems to be -A style (as a separate expander is available)
 

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
Ok,
some more info about the various option surly would be welcome:)
And yes, if there was proper cableing then would think the A might still be capable of providing nvme support-
you might be able to test this on the Intel Server of yours - use the NVME cable on a non nvme capable drive slot and attach to the nvme card.
The intel backplane seems to be -A style (as a separate expander is available)
The backplane would appear the be passthrough but the cards are not x16, so the kit is not direct attach..
 

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
A couple of teasers:





Haven’t decided exactly what I want to do here yet.. set it up as a x60 drive storage server, or do something a little more off the wall like dump the 10G switch internals back in.

This whole project got pushed off to the side after I found 40mm pwm fans for the Gnodal which alleviated the sound concern. At this point combining the switch into this case would be purely for grins.

Any suggestions/other wild, but interesting ideas for the extra space? Secondary independent system for backup? High powered water cooling setup w/a drip lip added into the case floor to isolate the front backplane?
 
Last edited:
  • Like
Reactions: Patrick and T_Minus

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
So I know we were all drooling over the x16 HighPoint RocketRaid NVMe card a while back, but has anyone looked into their more recent offerings? ...Particularly the m.2 stuff? I've not really spent a lot of time researching true availability (vaporware), buut:



Maybe they'll come out with an m.2 pcie backplane that could be packed full of mid grade m.2 NVMe :)
 

AJXCR

Active Member
Jan 20, 2017
562
95
28
32
Ok,
And yes, if there was proper cableing then would think the A might still be capable of providing nvme support-
Unless I'm missing something, I don't think the "A" or "TQ" backplane will ever be able to support NVMe drives regardless of cabling. The actual connectors on the drive side are missing pins to support a PCIe capable drive..



I would assume that the lack of pins on the connector would also almost guarantee the absence of traces on the board... So I think we're shit out of luck on this one. I've never really looked at an electrical diagram of a pass through board, but if all you're looking for is convenient mounting and dismounting of the drive, a pcb which simply converts sff-8643 connectors to equally spaced sff-8639 connectors sounds pretty straight forward to me? (I'm sure someone with more knowledge than me on the subject can explain to me what I'm missing).

If all of the traces are 1:1, couldn't someone have a small run (maybe qty 100) manufactured relatively easily). I'm being lazy here, but do sff-8639 and sff-8643 connectors have the same number of pins? Is everything 1:1?

At the same time, the amount of NVMe capable hardware seems to be increasing exponentially atm.. so I'm sure we'll start to see all kinds of options in the very near future. I'm ready for it. At the moment I own only 6 non-NVMe drives.. 4 HE8's for backups and two 512GB 850 Pro's that I was using as mirrored boot drives. The headaches associated with NVMe are a PITA!! (At least when you start trying to run high qty's in a single system)

But like I said, all sorts of stuff popping up at the moment:

Tyan:


Intel:


Oracle:





The other piece of tech I'm dying to see become available is OLED notebook screens. I've been running a 55" OLED monitor and it's quite literally one of my favorite pieces of tech to date; total game changer.

I'd kill to have the same display quality in a notebook, but as far as I'm aware, the only machines currently offering it are the Lenovo Thinkpad X1 Yoga and an Alienware machine..

The day OLED is offered on a P51/Zbook G4/Precision 5220/Fujitsu, I'm going to have one in the mail!! ...Hell, I'd even suck it up and convert to a Macbook Pro for an OLED :D
 
Last edited:

oooyyyeee

New Member
Apr 25, 2018
1
0
1
39
Hi,

Did you ever get the NVME working with a LSI card?
I have 2 BPN

BPN-NVME3-216N-S4
BPN-NVME3-216N-N4

My board is more complicated... SMCI Quad CPU board

One is direct attached to 24x NVME and the other one has Expander that connect to NVME 24 bay.

Trying to find out what worked for you.




.....So I'm trying to get a deal worked out on one of the Supermicro boards that supports 24+ NVME drives..

Before you guys start hammering on me:
1. I know they use a proprietary form factor
2. I've seen Supermicro's disclaimer:

"Due to the complexity of integration, this product is sold as completely assembled systems only (with minimum 2 CPU, 4 DIMM and 6 NVMe). Please contact your Supermicro sales rep for special requirements."

3. I recognize that whatever it is I'm doing probably doesn't require 24x+ NVMe drives...
4. etc. etc. etc.....

That being said, SM's high end NVMe storage solutions appear to accomplish this using the following method/components:

For 24x NVMe:
-1x BPN-SAS3-826TQ-B2B (obtainable): "2-port 2U SAS3 12Gbps backplane, support up to 2x 2.5-inch SAS3/SATA3 HDD/SSD"

-1x BPN-NVME3-216EB (No Source Found): "BPN-NVMe3-216A-S4 Backplane Base Board"

-1x AOC-2UR6N4-i4XT-P (obtainable): "2U Ultra Riser with 4-port 10GbE RJ45 (10GBase-T)"
http://www.supermicro.com/a_images/products/Accessories/AOC-2UR6N4-i4XT.jpg


-2x BPN-NVME3-216EL (No Source Found): "PCIe Gen3x16 input to PLX9765 to support 12x NVMe port" ...Although this simply looks like an NVMe expander similar to what you'd see on a BPN-SAS3-216EL1, but for NVMe

-4x CBL-SAST-0819 (Obtainable): "OCuLink v.91,INT,PCIe NVMe SSD, 65CM,34AWG"

-4x CBL-SAST-0820 (Obtainable): "OCuLink v.91,INT,PCIe NVMe SSD, 85CM,34AWG"

-1x RSC-U2N4-6 (Obtainable ~$80 new): "2U Ultra Riser Card with 4 NVME and PCI-Ex16,RoHS/REACH"

-1x RSC-R1UW-E8R (Obtainable ~$40 new): "RSC-R1UW-E8R-O-P"

....They also make this 1u ultra riser which supports 6x oculink NVMe ports (AOC-URN6-i2XT)...


As well as this AOC which includes 4x oculink, plx, and is both cheap and available (AOC-SLG3-4E4T):



And this AOC w/2x oculink.. even cheaper and available (AOC-SLG3-2E4T):


Regarding the 1x BPN-NVME3-216EB and 2x BPN-NVME3-216EL: There are several commonly available SuperMicro 2.5" combo back planes available which support both SAS3 and some quantity of NVMe (e.g. the BPN-SAS3-216A-N4 2U). This is a similar approach to what Intel seems to currently offer, but in a less modular approach..

Then I stumbled across the following user's manual for the (BPN-NVMe3-216A-N4 2U):
https://www.supermicro.com/manuals/other/BPN-NVMe3-216A-N4.pdf

The nomenclature for the other combo back planes usually specifies the number of NVMe slots using the "-N*" at the end of the PN which threw me off a bit in this case...

An interesting excerpt from the manual:

....I found it very interesting that the parts list for the pre-configured SM chassis list:
1x BPN-SAS3-826TQ-B2B
1x BPN-NVME3-216EB
2x BPN-NVME3-216EL

Whereas the user manual suggests that 1x BPN-NVME3-216EB + 2x BPN-NVME3-216EL = 1x BPN-NVMe3-216A-N4...

Could it be that, because the BPN-SAS3-826TQ-B2B is simply a TQ pass through, it's just being relabeled/re-purposed in combination with the BPN-NVME3-216EB & BPN-NVME3-216EL cards? If so, I see the BPN-SAS3-826TQ's steeply increasing in value.. This might also lead one to believe that the 216EB and 216EL cards (or possibly the 9400-16i/3840a) might work in combination with other SAS3 TQ backplanes...

May have to start an initiative to corner the 2nd had TQ market! :D