Dell R640 Fan High speed issue with front-loaded 2.5 U.2 Enclosure/NVME Drives

[5bngr]

Sorry I to repost my previous post. I had to correct a couple of typos/mistakes and it keeps deleting my post when I do. Not sure what that's about..

Hi All,
Good to be here. To sum up my situation, I bought a R640 off ebay. Here is my configuration:
Native:
- R640 10 Bay Chassis
- H740p mini controller
- (1) NDC - (2) 1G Eth / (2) 10G Eth
- M3/M4 NVME Cables to enable NVME on Slots 6-9

Third Party equipment:
- (1) 3rd party Quad NVME card in PCIe slot #1
- (4) sk hynix p31 1TB NVME - Installed in Quad NVME card
- (6) Kingston 480G SATA (from Old build) - front loaded in slots 0-5
- (3) sk hynix 2TB NVME - front loaded in slots 6-8 via 3rd party 2.5 U.2 enclosure

The Good:
- It all Works. Proxmox sees all the drives
- Tested all 4 p31s simultanously with fio.
- Reads -> min= 2070, max= 2376, per=100.00%, avg=2255.03
- Writes -> min= 1657, max= 3096, per=100.00%, avg=1880.30

The Bad:
- I can't see the NVME drives in the iDRAC/BIOS so I can't boot from them natively - I can live with this.

The Ugly
- As soon as I insert the 2.5 U.2 enclosure into any slots, in the range 6-9, i get this in the logs:
- PR7 New device detected: Disk (PCIe SSD in Slot X in Bay 1) X being 6-9
- PCI3030 New PCI card(s) have been detected in the system. Fan speeds may have changed to add additional cooling to the cards.
- Few seconds later the Fan goes crazy - This, I can't live with.

The crazy thing is Fan speeds were normal and didn't change when I installed an actual PCIe card in slot 1 (Quad NVME card). But the minute I front load a 2.5' U.2 enclosure with (or without nvme drive in it!!) i get the above logs and Fan goes crazy.
Any ideas on:
- How to ramp down the fan speeds on an R640? or
- How to update OEM/Standard NVMEs to a firmware where iDRAC/BIOS recognizes the NVME and it doesn't mess with the Fans?
- I really don't want to buy dell drives. To expensive for what is available elsewhere.

Thanks for reading/listening!

 

[CyklonDX]

- I can't see the NVME drives in the iDRAC/BIOS so I can't boot from them natively - I can live with this.

You can only see nvme's with dell's nvme backplane, or boss cards. Normally connected nvme -> idrac won't show - or non-dell cards won't appear there.

To change - fan control on idrac > system > cooling > configure cooling (and thats all you get)

- How to update OEM/Standard NVMEs to a firmware where iDRAC/BIOS recognizes the NVME and it doesn't mess with the Fans?

You typically can't.

- I really don't want to buy dell drives. To expensive for what is available elsewhere.

 

[5bngr]

You can only see nvme's with dell's nvme backplane, or boss cards. Normally connected nvme -> idrac won't show - or non-dell cards won't appear there.

To change - fan control on idrac > system > cooling > configure cooling (and thats all you get)


You typically can't.

Thanks for the reply. Appreciated. So.. I ran into this youtube vid (referring to a R720 I think) but I wonder if it maybe still applicable to an R640? The guy is all over the place and keeps fumbling the commands, but you get the idea after watching the vid. If it is doable, I'm thinking of downgrading to pre iDRAC 3.34.x.x and give it a try. here is the link.

 

[mattventura]

It's a long shot, but you can try using some Kapton tape or similar to cover the sideband pins on the drive. I have a supermicro system with a WD drive that reports a normal temperature via SMART (or whatever the nvma equivalent is called), but reports 127C on the sideband. Plugging the drive in will cause the fans to go up because the system thinks the drive is boiling, but covering up those pins causes the BMC to have no idea what the drive's temperature is.

 

[CyklonDX]

Thanks for the reply. Appreciated. So.. I ran into this youtube vid (referring to a R720 I think) but I wonder if it maybe still applicable to an R640? The guy is all over the place and keeps fumbling the commands, but you get the idea after watching the vid. If it is doable, I'm thinking of downgrading to pre iDRAC 3.34.x.x and give it a try. here is the link.

r640 series idrac is completely different than that in rX20, X30 servers. You cannot "downgrade" to them.

At best idrac won't be functional, at worst you'll brick your post (as idrac needs to be within certain version range to bios.)
idrac sideloads firmware on to all connected supported devices during post *(this includes some cpu microcode).

 

[5bngr]

It's a long shot, but you can try using some Kapton tape or similar to cover the sideband pins on the drive. I have a supermicro system with a WD drive that reports a normal temperature via SMART (or whatever the nvma equivalent is called), but reports 127C on the sideband. Plugging the drive in will cause the fans to go up because the system thinks the drive is boiling, but covering up those pins causes the BMC to have no idea what the drive's temperature is.

I will look into this. Thanks for the info.

 

[5bngr]

r640 series idrac is completely different than that in rX20, X30 servers. You cannot "downgrade" to them.

At best idrac won't be functional, at worst you'll brick your post (as idrac needs to be within certain version range to bios.)
idrac sideloads firmware on to all connected supported devices during post *(this includes some cpu microcode).

Gotcha. I wash hopeful since he said it works across vendors i thought it would at least work across some the poweredge line of servers. Wishful thinking I guess. I'll check the minimum version of the R640 iDRAC I can downgrade to and give that a shot. At least I'll mimimize briking my system by staying within the R640 iDRAC versions.

 

[CyklonDX]

Stay within 4 minor idrac revisions of bios as a safe rule of thumb to have fully functional idrac/bios. *Those things won't make difference either
way. Its best to keep up-2-date idrac and bios.

if your bios is 2.22.2
you want to stay within range of those idrac's revisions.

 

[5bngr]

I will look into this. Thanks for the info.

Hey. I was looking further into this, as it maybe the easiest path to try, based on what is actually happening on my chassis. To recap, as soon as I insert the 3rd party U.2 enclosure in slots 6-9, the fans go crazy. Just inserting the empty enclosure without any NVME drives inside it, does the same thing. I'm now suspect of the NMVE enclosure I'm using. Based on my research, mine doesn't seem to have a smbus eeprom. Could that be the root cause of the issue? No proper sideband communication support? Anyway, as reference, below is the pinout diagram for a U.2 connector for NVME drives. In terms of the hack you suggested, masking the sideband pins you mentioned before, are those pins E23-24 in the diagram? Or some other pins? The idea here is to either confirm/fix proper sideband support on the enclosure I'm using or, if that fails, just mask the sideband pin as suggested by your hack. Thanks again for pointing me down this path as it may be the root cause of the issue.

 

[CyklonDX]

thats how its designed to work;
no signal / no temp data = 100% fan in case there is an issue.

 

[5bngr]

thats how its designed to work;
no signal / no temp data = 100% fan in case there is an issue.

That being the case, help me understand the logic behing the hack you proposed. How does it work if it is masking a signal? Wouldn't masking the signal be the same as no signal? Pardon the dumb question. I know I'm missing something.

 

[CyklonDX]

What i mean - it won't help on dell's idrac.

 

[frogtech]

Dell gutted manual fan control on 14G (Rx40+) servers and beyond I'm pretty sure.

Not sure if you can still disable the "3rd party pci cooling" iDRAC property anymore either.

 

[frogtech]

That being the case, help me understand the logic behing the hack you proposed. How does it work if it is masking a signal? Wouldn't masking the signal be the same as no signal? Pardon the dumb question. I know I'm missing something.

Those pins seem to be for sideband yes.

My understanding is you're blocking the out of band enclosure management (iDRAC/BMC) from even seeing reported information. So blocking these pins may not show it as connected in iDRAC under backplane/enclosure. Not sure what the behavior would be ultimately but it's definitely worth trying. You should be able to even do a test run with some painters tape until you get kapton.

 

[5bngr]

Dell gutted manual fan control on 14G (Rx40+) servers and beyond I'm pretty sure.

Not sure if you can still disable the "3rd party pci cooling" iDRAC property anymore either.

Yup. Gutting is the right word for sure. Can't find anything via IPMI/RACADM/REDFISH APIs. For those interested, You still can (I tested it via RACADM) disable the "3rd party pci cooling" behaviour for PCIe slots 1-3, so that's something. For me it didn't do crap, since it is the front-loaded NVME SSDs that are triggering the Fan speed change. The irony is not lost on me, the fact that I have a Quad 3rd party PCIe card with 4 NVMEs installed in PCIe slot #1 and that's working fine and the Fans are happy. Crazy!

 

[5bngr]

Those pins seem to be for sideband yes.

My understanding is you're blocking the out of band enclosure management (iDRAC/BMC) from even seeing reported information. So blocking these pins may not show it as connected in iDRAC under backplane/enclosure. Not sure what the behavior would be ultimately but it's definitely worth trying. You should be able to even do a test run with some painters tape until you get kapton.

Makes sense. I'm getting some Kapton and testing your theory. Keep u posted. Thanks again.

 

[imivanov]

I had almost the same issue : Fans up to 76% after inserting NVME disks. I've tried all workarounds that I've found without any success. After high level of depression, I've changed the fans from standard performance fans to high performance fans. And miracle - after iDRAC reload - fans are stable at 26%. So I can confirm that on Dell R640(2x Xeon Gold 6262) with two NVME U.3 disks(Micron MTFDKCC1T9TGP-1BK1DABYY) bought from Senetic and high performance fans, you will have quiet R640.

And after that I've found this:
The manufacturer requires High Performance fans if the processor’s TDP exceeds 200W or if NVMe drives are installed.
Using standard fans in such configurations will cause them to run at maximum speed, which may increase noise levels.

So, if you choose the right, not so expensive non-dell branded NVMEs, there is a chance for quiet Dell R640

 

[Tasoss]

Hello i have a dell xc640 with non dell nvmes same issue fans at 75%. Can anyone else verify that high performance fans lower the speed of the fans??

 

[Marc_]

Hello i have a dell xc640 with non dell nvmes same issue fans at 75%. Can anyone else verify that high performance fans lower the speed of the fans??

They do indeed lower the fan RPM. High TDP processors or NVMe drives without high performance fans will run the standard fans at near max.

 

[Tasoss]

I am just curious what is the difference at performance fans I mean they probably ran at higher rpm to have more air flow ? Is it just a dell safe mechanism ?