NVMe on Dell Precision Workstation T7910

[galvesribeiro]

Hello folks!

I've recently acquired two T9710 which are arriving soon so they can be a home lab VMWare Cluster.

I'm adding to each machine 8x SATA3 consumer level SSDs using SAS breakout cables. Those are capacity disks.

I also need to add 4x NVMe M.2 SSDs. However, the T7910 doesn't have M.2 slots so I need a PCIe x16 card to make this to work. Most of the cards I've seen requires PCIe bifurcation and as far as I know, the T7910 system board doesn't support it.

I've seeing some posts in this forum about some cards that has some sort of "switch" and doesn't require that the motherboard to support PCIe bifurcation but I've been struggling on finding one that for sure work and/or is available to purchase.

Can someone shed a light on a PCIe x16 card that can accommodate those 4x NVMe SSDs without requiring the bifurcation? Also, is there any performance impact on using those cards?

Thank you!

 

[BLinux]

What makes you think T7910 doesn't support bifurcation?

 

[galvesribeiro]

What makes you think T7910 doesn't support bifurcation?

The machines are getting here tomorrow so I haven't tested it yet. However, I've searched around the web and everything I found says it doesn't support it. Only T9720 has the option to enable it.

Also note that I mentioned the "Workstation T7910" which is the Tower/Desktop product. Dell also have a rack mountable 7910, which is sold as just a "rack option" but in reality it is a PowerEdge motherboard. That rack model has a BIOS update which enable the option to enable the bifurcation which apparently doesn't exist in the Tower motherboard.

 

[BLinux]

@galvesribeiro I've seen reports of people using Dell P/N 80G5N for T7910 workstation? That requires bifurcation.

 

[galvesribeiro]

Yeah that is the thing.

I've seen those around and yeah it requires bifurcation, but I only saw people using T7920 or the 7910 rack model. I also saw one report somewhere that the T7910 doesn't have the option to enable/disable bifurcation on the BIOS, however, it does it "automagically". I thought it was suspicious as no other reports said the same.

I guess I'll have to test it out and see what happens :/

 

[BLinux]

Yeah that is the thing.

I've seen those around and yeah it requires bifurcation, but I only saw people using T7920 or the 7910 rack model. I also saw one report somewhere that the T7910 doesn't have the option to enable/disable bifurcation on the BIOS, however, it does it "automagically". I thought it was suspicious as no other reports said the same.

I guess I'll have to test it out and see what happens :/

It is true that T7910 can do the "automagically" if I recall. My T7910 is in use right now so I can't reboot it, but I recall seeing the option to manually set the bifurcation too. And there are a lot of people using the Dell quad M.2 adapter in T7910 workstations. Perhaps you've just encountered a lot of "fake news" LOL

 

[galvesribeiro]

I hope you are right and it just works

 

[galvesribeiro]

I bought couple of cheap Chinese cards that requires the bifurcation. Not going to keep it for "production" but since they are extremely cheap and are available here in Brazil, I ordered just to test. If that works by at least the drivers are detected by VMWare, then I'll invest more in the Dell cards.

Fingers crossed.

 

[galvesribeiro]

Alright! Just getting back to you.

I got the machine to work. The BIOS has indeed no option to enable bifurcation even after updating to latest version (2020). However, when I added the Chinese card I mentioned with 2 NVMe, they just worked and appeared on the OS!

(ignore the language, it was the OS that was installed on it, just booted)



Now time to find our how to convert the LSI SAS 3008 to HBA mode and will all be ready

Thanks for the help!

 

[BLinux]

great! To convert the onboard controller to IT mode, i made this video:

 

[galvesribeiro]

Oh! So you are the guy from the art of server

I just followed it after a while trying to find the proper firmware. Now it is on IT mode and VMWare can see all disks on it!

Thank you very much for your help!

 

[galvesribeiro]

Now trying to decide wether to pick one of this MB038SP-B_8 Bay 2.5" SAS/SATA HDD/SSD Mobile Rack for 5.25" Bay or go nuts and all the way NVMe and use this MB873MP-B V2_8 Bay M.2 NVMe SSD PCIe 4.0 Mobile Rack Enclosure for External 5.25" Drive Bay (8 x OCuLink SFF-8612)

I guess the the NVMe route would be awesome with vSAN ESA BUT, there is no OCuLink on this motherboard and I don't think PCI lanes for 8 OCuLink connections either I wish it was an EPYC CPU

 

[Zedicus]

Now trying to decide

in case you have not decided yet, the Athena is always on sale. i use them.

Athena Power BP-15827SAC an 1 x 5.25" External Drive Bay to 8 x 2.5" SSD / HDD Hot-Swap SAS / SATA Backplane Module - Transfer Rate Up to 6 Gbps SAS (I, II) / SATA (I, II, III) - Newegg.com

Buy Athena Power BP-15827SAC an 1 x 5.25" External Drive Bay to 8 x 2.5" SSD / HDD Hot-Swap SAS / SATA Backplane Module - Transfer Rate Up to 6 Gbps SAS (I, II) / SATA (I, II, III) - Drive Thickness Up to 7mm - HDD / SSD Accessories with fast shipping and top-rated customer service. Once you...

www.newegg.com

 

[galvesribeiro]

in case you have not decided yet, the Athena is always on sale. i use them.

Oh! Nice! Thanks!

This bay takes 8x 2.5" disks while the bottom of the case has 4x 3.5" bays. With that price, I may get 2 of those Athena and put the second one somehow in the back of the case where you show on your video that the official Dell kit has the fans (ofc if somehow I can secure it there with some screws) since it is a full waste of empty spaces.

That would bring me to 20 SATA disks in total. Since the on-board controller has only 2x SATA connectors which would give me only support for 8 disks only, which (cheap) HBA you would recommend that accept that IT firmware?

Thanks!

 

[Zedicus]

8i cards are dirt cheap

Inspur LSI SAS3008 12Gbps SAS HBA RAID Controller Card P/N:YZCA-00424-101 Tested | eBay

Item is in good working condition unless otherwise stated. Any parts not photographed WILL NOT be included. We understand if you encounter any setbacks.

www.ebay.com

if you need a 16i card that is more difficult.

 

[chrgrose]

The dell 'Ultra Speed' Quad M.2 enclosures absolutely work in the T7910. Just be sure to populated them in ascending order, or some drives will not be visible.

 

[galvesribeiro]

Another question that came up is about the Memory.

I've got a pair of E5-2699A v4 which will arrive soon. Now I'm looking to increase the memory. Right now it has 128gb as 4x 32GB 2Rx4 PC4-2400T-RB1-11.

In the owner manual it says the maximum would be 512Gb ECC. I've found out this site which say it can go as high as 1Tb if I can find 64Gb sticks but what is confusing me is if it support those sticks with regular ECC or if it requires the LRDIMM.

Would something like this supported? The memory I got here is 2Rx4, while those are 4Rx4. Would that work?

Thanks!

 

[chrgrose]

I've used 4DRx4 64GB modules in the T7910 with no issues. I'm not certain but I doubt you would have issues with registered modules.

 

[galvesribeiro]

I've used 4DRx4 64GB modules in the T7910 with no issues. I'm not certain but I doubt you would have issues with registered modules.

The ones that came are RDIMM, not LRDIMM, so I can't mix them, but according to the docs LRDIMM works. I was just concerned about the 4Rx4 but if it worked for you I think I'll be good.

Thanks!