Dell PowerEdge T640 as a Workstation : Powerful and Silent.

[nilaykmrsr]

Hello @rafale77 , @ElCoyote_ I just purchased a barebones PowerEdge T640 for building a computational node (running physics based simulations) out of it for home use. I would like to know how's the performance of your 8272CL /8269CY on your machines?
I bought a pair of 8273CLs from eBay hoping they would work but they do not, the server just goes into a boot loop and was never able to post completely. I'm wanting to stay as close as possible to the 28 core/CPU count without going bankrupt. Thanks

 

[ElCoyote_]

Hello @nilaykmrsr , I am very happy with my 8269CY's (I'm getting about 2200 on LINPACK Linux for the T640 with those chips).
I think @rafale77 had trouble with his @8272CL's he got from ebay (or maybe I'm mistaken).
When you say the machine goes into a boot loop, do you see anything relevant in the iDRAC's log? What timezone are you in? perhaps we could do a debugging session together... (if you can share the iDRAC's dashboard). Several of my colleagues are using a T640 as a workstation/server with cpus we got from China but none of us are using a 'CL' version..

 

[nilaykmrsr]

Hello @nilaykmrsr , I am very happy with my 8269CY's (I'm getting about 2200 on LINPACK Linux for the T640 with those chips).
I think @rafale77 had trouble with his @8272CL's he got from ebay (or maybe I'm mistaken).
When you say the machine goes into a boot loop, do you see anything relevant in the iDRAC's log? What timezone are you in? perhaps we could do a debugging session together... (if you can share the iDRAC's dashboard). Several of my colleagues are using a T640 as a workstation/server with cpus we got from China but none of us are using a 'CL' version..

Hello @ElCoyote_ Thanks for the response. I would have loved to do a debugging session with you (I’m in EST) but I have already removed the CPUs from the server and plan on returning the CPUs. I did login to iDRAC yesterday and it just said “BIOS halted” in critical errors. It did not detect the 8273CL CPUs at all. The eBay seller had also previously warned me that 8273CL does not work on any of the 14th Gen PowerEdge servers, I wanted to try these out just in case a newer BIOS may end up supporting it. Apparently it’s some hardware white listing at play by Dell.

I almost purchased the 8272CL this morning on eBay but since you mentioned @rafale77 has had issues with his 8272CL I am cancelling that as well.

I think my only option is to get the tried and tested 8269CY. Can you please share the contact details of the vendor from where you got the 8269CYs over DM?

 

[ElCoyote_]

Hi @nilaykmrsr I'm EST too so please keep my offer in mind if you need help in the future. I don't think it has to do with cpu whitelisting (all of the cpus we have in our T640's are from China cloud providers so I doubt it's whitelisting at play). I think @rafale77 had some trouble with those SKUs too and he returned them too. We (all 4 of us) got our cpus from Wenjie in Shanghai. PM me if you wish to have her information.

 

[ElCoyote_]

Also, 'BIOS halted' is not something I've ever seen on my T640 (and I tried quite a few cpu types)

 

[ElCoyote_]

So far, we have those SKUs confirmed to work 100% fine in our T640's:
- Platinum 8269CY
- Gold 6240C
- Gold 6271(C)
- Platinum 8255C (strange LINPACK numbers on these, though, only about half of the 8269CY performance)
- Gold 6267

 

[nilaykmrsr]

Thanks a lot @ElCoyote_ for the information you provided and for going out of your way to help me. I will try to get in touch with the supplier to see if I can get my hands on a 8269CY. My ideal CPUs would be Gold 6258R or simply the Platinum 8280.

 

[ElCoyote_]

You're most welcome @nilaykmrsr ! If you get one of these cpus (150W and above), please make sure to get a pair of high-wattage heatsinks for your beast. The P/N's are:
* Heatsinks
- 0VX3D ( 412-AANJ ) : High Wattage heatsink (165W + ) / KN2PJ

489KP (412-AANI) is the standard < 150W heatsink.

 

[nilaykmrsr]

You're most welcome @nilaykmrsr ! If you get one of these cpus (150W and above), please make sure to get a pair of high-wattage heatsinks for your beast. The P/N's are:
* Heatsinks
- 0VX3D ( 412-AANJ ) : High Wattage heatsink (165W + ) / KN2PJ

489KP (412-AANI) is the standard < 150W heatsink.

Thanks for the part numbers @ElCoyote_ . I had already ordered the OEM High wattage heat sinks from China. I had been following your posts on this forum for the past couple of months before signing up for STH Forums so I was aware about the requirement for high wattage heat sinks. I hope I’m able to get some CPUs.

 

[nilaykmrsr]

Hello @ElCoyote_. I hope you have been doing good all along. The 8269CYs work great on my T640. I have gotten to the point of installing GPUs now for trying some GPU based computations. I am planning to install two RTX2070 Super GPUs that I had lying around. I bought the required external fans but iDRAC would give me fan configuration errors. I reached out to Dell support today and they advised that I have to use the High-Performance Mid fans (Dell P/N: 0KJN71) to correct that problem. These fans are super expensive and are not available easily in North America (eBay etc.). I am also seeing Dell P/N: 06KK42 as a GPU compatible mid fan. I was curious to know what part number of mid tray fans you have on your T640, I assume that you are running GPUs on it as per your blog? Any other information you may have is deeply appreciated.

 

[ElCoyote_]

Hi @nilaykmrsr my T640 came with the mid-case fan tray (it happens when you order 18 x 3.5 configs). The mid-tray I have is 06KK42.
If you are using GPUs which require external power, you will also need the GPU Power board that goes underneath the main motherboard (it's a delicate process to do that). Here is a picture: Dell Poweredge T640 GPU Chassis Upgrade Kit (those things used to sell for 60USD).
For my own T640, because I found the external fan quite noisy, I've simply removed the GPU power board and setlled on a Quadro P2200 GPU (without external power). I don't do GPU computations, only desktop at 4k.
There is a Dell video of the process to add a GPU PowerBoard to the T640 here:

To be honest, it's a lot easier to add GPUs to the T630. The T630 still requires the GPU PIB but no external fans are needed.

 

[nilaykmrsr]

Thanks @ElCoyote_ for your response. I have already installed the GPU PDU Board with the external fans, although I was not aware about the high-performance mid tray fan requirements if you are using a GPU. These fans are practically unobtanium here in NA as of now and some vendors in CN want north of $600-700 for the fans and mid tray. Just got a quote from Dell and ordered them (P/N: 0KJN71) for $440 (still cheaper than eBay surprisingly). I just checked the part number on my existing mid tray, it says its 06KK42 as well, but it has the standard (0CKY79) fans on it.

 

[ElCoyote_]

Hi @nilaykmrsr I have an extra T640 mid-fan tray (ordered in April of 2020 before my T640 came -with- its own mid-tray) if you'd like to buy it.
I don't think the high-performance fans would be required given the size of the machine.. I'd really go with standard fans for the mid-case tray and bump up the minimum speed for fans in the iDRAC if needed.

 

[nilaykmrsr]

Hello @ElCoyote_ . I already have the mid tray with my T640, it's just that the fans in it are "standard" low CFM 10.6W fans. I had already tried changing the fan profile using iDRAC (while using standard fans in the mid tray) and it did not fix the issue. The extra cooling is overkill in my opinion even with dual RTX2070 supers but iDRAC won't accept the standard fans with GPUs installed (Ref: Dell's GPU guide for T640). I have ordered the fans directly from Dell, hopefully this will fix the issue.

 

[ElCoyote_]

Yeah, stupid over-zealous over-cautious T640 iDRAC.. Sorry, I had not realized you already had the mid-tray fans and that the iDRAC demanded the high-speed fans. I agree with your analysis, the high-speed fans are probably not needed, except to make the iDRAC happy..
Good luck, I hope it resolves your problem.

 

[dmitry.n.medvedev]

hi,

first of all, this is super insightful post enjoying it a lot. Also, it looks that this is the only post on T640 as a workstation.

I have a couple of questions about a similar machine. Let me start with the context first.

The Context: Dell T640 8x3.5" chassis. Two silver CPUs. A bit of RAM. A backplane. Two 1100 gold PSUs. No controllers, no GPU. I am also trying to build a workstation from this server. And I would like this workstation to be fast.

The Problem: I thought it could be great to use NVME U.2 drives in the machine. Unfortunately, it seems like I am out of luck in regards to NVMEs. The community has splitted in those who believe it is totally not possible to have U.2 NVMEs in the chassis given, and those who aren't that certain, but aren't sure how to achieve this.

The Question: do you think it is possible to physically install nvme expansion backplane meant to be used in another T640 chassis ( and connect it to something like HBA330 or HBA350i )? I have just a single chassis at hand and cannot tell if an nvme backplane could be attached to the front-top part of it.

best regards,
Dmitry

 

[ElCoyote_]

@dmitry.n.medvedev If your T640 did not come with the NVMe backplane and you have the 8 x 3.5 bays version you could buy the NVMe conversion kit but last I checked it retailed around 700-800USD.
So for U.2 and U.3 drives in my T640, I went with U.3 carriers from Linkreal bought on alibaba:

this was a lot cheaper and the T640 has a ton of PCIe slots anyway

 

[CyklonDX]

You can try getting those (its much cheaper than icy dock) After all disk caddies are more fancy
(the con that i can see is that its large, i think 3x 5.2")

https://www.aliexpress.us/item/3256804903201011.htm

 

[dmitry.n.medvedev]

@dmitry.n.medvedev If your T640 did not come with the NVMe backplane and you have the 8 x 3.5 bays version you could buy the NVMe conversion kit but last I checked it retailed around 700-800USD.
So for U.2 and U.3 drives in my T640, I went with U.3 carriers from Linkreal bought on alibaba:
View attachment 32309
this was a lot cheaper and the T640 has a ton of PCIe slots anyway

I found this: DELL POWEREDGE T640 8 BAY SSD PCIE HDD NVME EXPANDER BACKPLANE, CAGE, AND CABLES | eBay

is it the conversion kit you mentioned? also, I have contacted the seller, but he is certain the kit at the link isn't going to fit into the chassis of my server I am a little bit sceptical on the ebay sellers' knowledge tho ( or just would like him to be wrong for obvious reasons ).

 

[dmitry.n.medvedev]

good morning, nice community!

one more question please: I would like to find out the part number of the GPU Air Shroud for a T640.

I have contacted Dell community, but a Dell representative urges me to call official ( payed ) support which I would like to avoid for obvious reasons.

best regards,
Dmitry