Dell PowerEdge T640 as a Workstation : Powerful and Silent.

[ElCoyote_]

For anyone considering a Dell T640 (14th gen) as a workstation, I'd like to report good success (with some gotchas) with the T640.

Here's the current config:
- Operating system: RHEL 7.9 (but will move to RHEL 8.4 later)
- Dell PowerEdge T640 chassis. 18 x 3.5 HDD Hotswap bays.
- 2 x Xeon Platinum Scalable 2nd gen 8269CY @ 2.5GHz (26C/52T ea), total of 104 Threads.
- 16 x 64Gb DDR4 Hynix LRDIMM@2666Mhz, 8 slots still available (the machine has 24).
- 1x1100W PSU (see below)
- 1 x X710-DA4 ( 4x 10G) NIC
- H730P (2G) connected to the backplane.
- 2 x Samsung Evo 860 Sata SSD
- 2 x HGST SN260 7.68Tb NVMe
- NVidia Quadro P2200 (see below) @ 3840x21600
- Dell UltraSharp U4320Q
- Aten CS-2944DP DisplayPort KVM switch (shared between two T630's and a T3431 SFF)

Summary: I scavenged the CPUs from China, bought the RAM on eBay in Chicago, got the NVMe from Thailand and bought the X710 NICs in Australia. This is an international machine (I live in Canada). It serves as a daily workhorse for KVM

The good:
- with the stock fans (2x120mm and 4x90mm), the machine is silent enough to stand near it all day long. The fans don't even need to be adjusted outside of the iDRAC anymore (and Dell took that away from us anyway in iDRAC 3.21+).
- I've settled with those fan settings:


Which gives me:



The Gotchas:
- Even though the system idles under 260W (or 364W with 20 VMs) and I never saw it above 560W, the Platinum processors + memory didn't like my 750W PSU (it would prompt for keyboard override at boot). I got a 1100W PSU and it's all good on a single PSU now.

- The T630 didn't care if I had GPUs with external power but the T640 didn't like the GTX 1660 Ti I had initially gone with: It worked fine but the system would start requesting an 'External PCI Fan' - a box which covers your PCI backplates in the back of the machine. Not having the External fan made the other fans go 100% (think of a 747 at takeoff).

I had even obtained the 'Right External Fan' and it worked OK but that fan was noisier and covered the PCI backplates. Since I was using a GPU to drive a Monitor and not to do AI, this got in the way and for two weeks I had the External fan hanging on the back of the machine with tape since I couldn't fit it.
After some hassle trying to make it work or 'fake' the external Fan, I relocated the GTX 1660 Ti to another machine, got a Quadro P2200 (it only uses PCI-E power from the x16 it's in), and removed the external GPU Board + External fans altogether.

Again, I like the machine now but it took some patience to get there (The T640 is to the right of the picture, the T630 is not powered on):

 

[TrumanHW]

For anyone considering a Dell T640 (14th gen) as a workstation, I'd like to report good success (with some gotchas) with the T640.

Here's the current config:
- Operating system: RHEL 7.9 (but will move to RHEL 8.4 later)
- Dell PowerEdge T640 chassis. 18 x 3.5 HDD Hotswap bays.
- 2 x Xeon Platinum Scalable 2nd gen 8269CY @ 2.5GHz (26C/52T ea), total of 104 Threads.
- 16 x 64Gb DDR4 Hynix LRDIMM@2666Mhz, 8 slots still available (the machine has 24).
- 1x1100W PSU (see below)
- 1 x X710-DA4 ( 4x 10G) NIC
- H730P (2G) connected to the backplane.
- 2 x Samsung Evo 860 Sata SSD
- 2 x HGST SN260 7.68Tb NVMe
- NVidia Quadro P2200 (see below) @ 3840x21600
- Dell UltraSharp U4320Q
- Aten CS-2944DP DisplayPort KVM switch (shared between two T630's and a T3431 SFF)

Summary: I scavenged the CPUs from China, bought the RAM on eBay in Chicago, got the NVMe from Thailand and bought the X710 NICs in Australia. This is an international machine (I live in Canada). It serves as a daily workhorse for KVM

The good:
- with the stock fans (2x120mm and 4x90mm), the machine is silent enough to stand near it all day long. The fans don't even need to be adjusted outside of the iDRAC anymore (and Dell took that away from us anyway in iDRAC 3.21+).
- I've settled with those fan settings:
View attachment 16588

Which gives me:
View attachment 16589


The Gotchas:
- Even though the system idles under 260W (or 364W with 20 VMs) and I never saw it above 560W, the Platinum processors + memory didn't like my 750W PSU (it would prompt for keyboard override at boot). I got a 1100W PSU and it's all good on a single PSU now.

- The T630 didn't care if I had GPUs with external power but the T640 didn't like the GTX 1660 Ti I had initially gone with: It worked fine but the system would start requesting an 'External PCI Fan' - a box which covers your PCI backplates in the back of the machine. Not having the External fan made the other fans go 100% (think of a 747 at takeoff).

I had even obtained the 'Right External Fan' and it worked OK but that fan was noisier and covered the PCI backplates. Since I was using a GPU to drive a Monitor and not to do AI, this got in the way and for two weeks I had the External fan hanging on the back of the machine with tape since I couldn't fit it.
After some hassle trying to make it work or 'fake' the external Fan, I relocated the GTX 1660 Ti to another machine, got a Quadro P2200 (it only uses PCI-E power from the x16 it's in), and removed the external GPU Board + External fans altogether.

Again, I like the machine now but it took some patience to get there (The T640 is to the right of the picture, the T630 is not powered on):

View attachment 16590

Okay -- how much did you pick it up for .. ?

 

[ElCoyote_]

@TrumanHW I was able to pick a barebones T640 Direct from Dell for about 1200 USD. I sourced/scavenged the parts on eBay (buying cpus, RAM and storage from Dell would have been extremely expensive). On most of the items, Dell wanted 5x-6x the price I paid on eBay.

 

[MountainDew]

For anyone considering a Dell T640 (14th gen) as a workstation, I'd like to report good success (with some gotchas) with the T640.

Here's the current config:
- Operating system: RHEL 7.9 (but will move to RHEL 8.4 later)
- Dell PowerEdge T640 chassis. 18 x 3.5 HDD Hotswap bays.
- 2 x Xeon Platinum Scalable 2nd gen 8269CY @ 2.5GHz (26C/52T ea), total of 104 Threads.
- 16 x 64Gb DDR4 Hynix LRDIMM@2666Mhz, 8 slots still available (the machine has 24).
- 1x1100W PSU (see below)
- 1 x X710-DA4 ( 4x 10G) NIC
- H730P (2G) connected to the backplane.
- 2 x Samsung Evo 860 Sata SSD
- 2 x HGST SN260 7.68Tb NVMe
- NVidia Quadro P2200 (see below) @ 3840x21600
- Dell UltraSharp U4320Q
- Aten CS-2944DP DisplayPort KVM switch (shared between two T630's and a T3431 SFF)

Summary: I scavenged the CPUs from China, bought the RAM on eBay in Chicago, got the NVMe from Thailand and bought the X710 NICs in Australia. This is an international machine (I live in Canada). It serves as a daily workhorse for KVM

The good:
- with the stock fans (2x120mm and 4x90mm), the machine is silent enough to stand near it all day long. The fans don't even need to be adjusted outside of the iDRAC anymore (and Dell took that away from us anyway in iDRAC 3.21+).
- I've settled with those fan settings:
View attachment 16588

Which gives me:
View attachment 16589


The Gotchas:
- Even though the system idles under 260W (or 364W with 20 VMs) and I never saw it above 560W, the Platinum processors + memory didn't like my 750W PSU (it would prompt for keyboard override at boot). I got a 1100W PSU and it's all good on a single PSU now.

- The T630 didn't care if I had GPUs with external power but the T640 didn't like the GTX 1660 Ti I had initially gone with: It worked fine but the system would start requesting an 'External PCI Fan' - a box which covers your PCI backplates in the back of the machine. Not having the External fan made the other fans go 100% (think of a 747 at takeoff).

I had even obtained the 'Right External Fan' and it worked OK but that fan was noisier and covered the PCI backplates. Since I was using a GPU to drive a Monitor and not to do AI, this got in the way and for two weeks I had the External fan hanging on the back of the machine with tape since I couldn't fit it.
After some hassle trying to make it work or 'fake' the external Fan, I relocated the GTX 1660 Ti to another machine, got a Quadro P2200 (it only uses PCI-E power from the x16 it's in), and removed the external GPU Board + External fans altogether.

Again, I like the machine now but it took some patience to get there (The T640 is to the right of the picture, the T630 is not powered on):

View attachment 16590

Do you plan to add more drives? Looks like you only had 4 to start. I'd be curious if noise levels changed when adding more drives.

 

[ElCoyote_]

Do you plan to add more drives? Looks like you only had 4 to start. I'd be curious if noise levels changed when adding more drives.

I think that will ultimately depend on the drive type. I've added a bunch of Samsung evo's 860 and 870 and the T640 didn't care. But I am sure that if I were to add WD RED's, it would probably notice and bump up the fans a bit. But then again it's been a long time since I've tested a WD RED on a 14th gen Dell.

 

[MountainDew]

I think that will ultimately depend on the drive type. I've added a bunch of Samsung evo's 860 and 870 and the T640 didn't care. But I am sure that if I were to add WD RED's, it would probably notice and bump up the fans a bit. But then again it's been a long time since I've tested a WD RED on a 14th gen Dell.

Thanks for responding. The post where you're sharing fan settings and current temps - is that in iDRAC? and if so, is it the enterprise version? I'm contemplating buying straight from Dell, so I wanna make sure whatever settings you have are available.

 

[ElCoyote_]

Thanks for responding. The post where you're sharing fan settings and current temps - is that in iDRAC? and if so, is it the enterprise version? I'm contemplating buying straight from Dell, so I wanna make sure whatever settings you have are available.

Hi, Yes, that's from the iDRAC. Get the cheapest iDRAC option when ordering and then find yourself an enterprise license on ebay for 25USD..
(the price they charge at point -of-sale for that license is just crazy)

 

[MountainDew]

Hi, Yes, that's from the iDRAC. Get the cheapest iDRAC option when ordering and then find yourself an enterprise license on ebay for 25USD..
(the price they charge at point -of-sale for that license is just crazy)

so you do have some control of the fans, but not as much with the previous iDrac versions?

 

[ElCoyote_]

so you do have some control of the fans, but not as much with the previous iDrac versions?

Yes, that's correct. If you'd like to know what it 'sounds' like, PM me a private message and we could setup a google meet/bluejeans/whatever meeting so that you'd hear the 'absence' of noise. (you can hear it's on but it doesn't bug me enough to notice)

 

[rafale77]

Reading this thread with great interest as I am looking at getting one of these as a VM server with 1 or 2 GPUs for computation and NAS.
I just got a used 8 swap bay chassis and dual 1100W PSU barebone for a mere $700. I am probably not going to need that much CPU power so will likely only populate 1 Xeon silver CPU for now. I have yet to find out a number of details about what I bought. (not sure what RAID card and NIC it has). I am seeing on the Dell website that there are 2 versions of the motherboard. What is the difference between them?

 

[ElCoyote_]

@rafale77 be aware that if you populate only one of the two sockets, then not all PCIe slots get enabled.

 

[MountainDew]

more info on fan speeds

 

[rafale77]

Thanks for the info guys. It sounds like this machine got a little too "smart" for its own good and as a result, not that great of a candidate for the secondary market. Well at least the drives are not blocked.
I am of course aware that the PCIe ports are linked to their corresponding CPUs. I will see what I get...

I am having a hard time deciding on CPUs at the moment...

 

[ElCoyote_]

@rafale77 if the 14th gen reaction to the non-enterprise HDDs bother you, then perhaps consider a T630. I also have one and this one doesn't freak out with my non-dell non-enterprise HDDs (WD RED Plus) so I've made the T630 a storage node and the T640 a hypervisor.

 

[rafale77]

@ElCoyote, Thanks! Not too worried about noise at the moment since I am pretty flexible in terms of location (I may just put it in the garage). I am more focused on CPU compatibility. Does it handle ES CPUs well? The Dell website seems to indicate it comes with 2x10GB NIC?
It seems like I will be getting an H330 card with mine...

 

[ElCoyote_]

Yes, the T640 comes with 2x10G. I'd advise to prefer QS cpus over ES cpus if you're into that kind of cpus. I used to have a pair of QS 6240C's in mine but sold them to buy 2 x 8269CY from my contact in Shanghai (a very reliable lady, please PM if you'd like her contact info).

ES cpus will typically run at a lower clock rate and/or have issues that neither QS or Production CPUs have, This is also why ES (Engineering Sample) cpus are so low-price. That being said, I'd expect 2nd gen or 3rd gen Xeon Scalable cpus to be somewhat decent even though they're ES but don't go for ES cpus from the 1st gen Xeon Scalables (my 2c).

 

[rafale77]

I learned something today. I knew the ES had lower clocks but never looked at the QS. Thank you. I don't have a massive budget and am looking at progressively replacing a QNAP has my do it all server. I run my home automation (including some deep learning models for video processing), pfsense, NVR, media server amongst many other things on it so I will explore the QS options. PM sent!

 

[ElCoyote_]

Any ES cpu -might- be fine, or it may not. It may also show issues in the long term. It's just about the right balance between the risk of running into issues and the price you're willing to pay for those cpus.
I don't go for ES. QS might be fine but I'm not ready to pay full price from Dell for new shiny Official CPUs at 5-6x the price. So it's really about finding your sweet spot. I use my T640 mostly as an hypervisor so I needed the best chips I could get within my budget. If you're building a DIY NAS, your conclusions/specs will most likely be quite different from mine.

 

[rafale77]

Diving right into it with @ElCoyote_ 's precious help. Got a pair of Platinum 8272CL QS off of ebay because... they were relatively cheap compared to the alternatives I considered... Hopefully they work out. Noticed that my used unit comes with 2x low wattage heatsinks which got me to go chase high wattage ones. They have 8 heatpipes instead of 4. There is a confusion about the P/N I am still trying to figure out.

On the Dell website,
The low wattage ones are
Manufacturer Part# : N4GJN | Dell Part# : 412-AANI
The high wattage:
Manufacturer Part# : 0VX3D | Dell Part# : 412-AANJ

But search on ebay gave me 0489KP and 0KN2PJ respectively... Confusing.

 

[ElCoyote_]

Yes, I looked at the pictures I took of my high-wattage heatsinks and they do have 8 pipes, I'll be opening the machine shortly to take a picture.