EPYC3251D4I-2T build

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nasi

Member
Feb 25, 2020
64
20
8
My memory modules are 2666, too and they are running on 2400.
Was the same to me but I have successfully "overclocked" them in the BIOS. I only have changed the clock speed to 1333MHz and left everything else on auto. Now my Proxmox test installation shows 2666MT/s (via dmidecode on command line).
I'm already using new 1.20 BIOS.
I've installed 2x Crucial CT16G4RFD4266 (which are not on Asrock Memory QVL but the bigger 32GB version).
 

csp-guy

Active Member
Jun 26, 2019
372
153
43
Hungary, Budapest
I am sad about the late lrdimm support. I almost bought 64GB modules for a very cheap price last year when I bought the motherboard, after testing they doesnt work, so i bought 4x32Gb RDIMM modules. Now, they have LRDIMM support, damn. :-(
 

x_interupt

New Member
Dec 6, 2020
4
1
3
Where did you get the OCULINK -> 4x sata cable? I wanted to buy one, too, but currently I resolved the lack of the cables with a Lenovo M1015 card -IT mode - installed.
i found them in australia, they'll ship to the USA, good price, with currency exchange rate, ordered on a friday, got them the following wednesday, and Asrock Rack branded.
OCuLink to 4 SATA Cable - CBL-SAST-0933| Shop SFP Products
had a hell of a time trying to find them domestic, they seem to be the only supplier for them that i can see.
 

andersenep

New Member
Dec 24, 2020
6
2
3
I have one of these boards, and it runs pretty hot. Anyone have any idea if one of these would work?

Supermicro M11SDV-8C-LN4F 60x25mm CPU fan holder by vihapuu

It's for a "Supermicro M11SDV-8C-LN4F motherboard (or any similar model with the same heatsink dimensions)". Someone earlier said the heatsink screws on the asrock board are 60mm x 60mm. Does anyone know if the Supermicro board is the same? Seems like it would be a cheap/elegant solution to cooling these things down a bit....

edit: checked the .stl file and it is indeed 60x60mm, so why not? if anyone has ideas for some sort of adhesive to glue the ABS plastic to the heatsink screws, I'd love to hear them....
 
Last edited:

andersenep

New Member
Dec 24, 2020
6
2
3
I'll give it a shot. Will probably dab a blob of jb-weld on 2 screws and mount it thus if everything is working.

I am debating whether or not to remove the passive cooler and apply some better thermal paste first? I am not super comfortable with doing this on such a cpu. It looks like no spreader/directly on-die application...Have you or anyone else done so?

This thing will crank up to 80-90C just posting...Once an OS boots it turns itself down a bit, but obviously active cooling is needed in my case (which was more or less expected).
 

andersenep

New Member
Dec 24, 2020
6
2
3
Turned out pretty well. Attached with some decent double-sided tape. We'll see how that goes (board sits vertically). Temps are much better now. Can't push it much past 60C now even when I try.

IMG_20210112_144705.jpgIMG_20210112_145323.jpgIMG_20210112_145729.jpg
 
  • Like
Reactions: nasi

andersenep

New Member
Dec 24, 2020
6
2
3
AsRock was kind enough to send me a beta BMC firmware (L1.32.00) that adds fan control options to the IPMI webUI.

I had a ton of Card Side Temperature Upper Non-critical - going high - Asserted/Deasserted events in the IPMI Event log due to the fact that all my case fans would only spin up based on the CPU temperature sensor. So when the CPU sat at idle, the case fans weren't pushing enough airflow to keep that Card Side Temperature sensor cool. The onboard LAN sensors also got rather uncomfortably warm (70C) while idle, but at least that wasn't blowing up my IPMI log. With the case fans manually set to 80% (~1100RPM), things are much cooler and still quiet.

Here is the link to the 1.32 BMC firmware
 
  • Like
Reactions: nasi

csp-guy

Active Member
Jun 26, 2019
372
153
43
Hungary, Budapest
AsRock was kind enough to send me a beta BMC firmware (L1.32.00) that adds fan control options to the IPMI webUI.

I had a ton of Card Side Temperature Upper Non-critical - going high - Asserted/Deasserted events in the IPMI Event log due to the fact that all my case fans would only spin up based on the CPU temperature sensor. So when the CPU sat at idle, the case fans weren't pushing enough airflow to keep that Card Side Temperature sensor cool. The onboard LAN sensors also got rather uncomfortably warm (70C) while idle, but at least that wasn't blowing up my IPMI log. With the case fans manually set to 80% (~1100RPM), things are much cooler and still quiet.

Here is the link to the 1.32 BMC firmware
Thanks, it is very useful.
 

andersenep

New Member
Dec 24, 2020
6
2
3
Thanks, it is very useful.
Glad others can benefit from it. It really really makes a difference on temps overall. I am not certain why they didn't have the option to set manual fan speeds in the BIOS as described in TFM, page 42, but this works just fine. Everything is nice and frosty now. Beast of a board.
 

andersenep

New Member
Dec 24, 2020
6
2
3
@csp-guy What are the temps on this board at idle and full load (with the fan)?
@wazoo42 here are my numbers before/after running stress-ng --matrix 0 for an hour if they are helpful (FAN2 is the noctua NF-A6x24 PWM on the CPU]:

[edit ambient room temperature is about 22-23C]

idle temps before:

CPU1 Temp: 28
MB Temp: 36
Card Side Temp: 45
DDR4_A1 Temp: 34
DDR4_A2 Temp: 33
FAN1: 1100
FAN2: 900
FAN3: 1200

after one hour stress-ng:

CPU1 Temp: 65
MB Temp: 46
Card Side Temp: 51
DDR4_A1 Temp: 46
DDR4_A2 Temp: 43
FAN1: 1100
FAN2: 2200
FAN3: 1200

Here's a graph of how the CPU dialed back down to idle:

post_graph.jpg
 
Last edited:

nasi

Member
Feb 25, 2020
64
20
8
I've build a watercooled server based on this nice little mainboard.
  • Bitfenix Prodigy case with insulation material
  • Asrock Rack EPYC3251D4I-2T
  • 2x Kioxia SATA SSD 1.92TB
  • 2x Seagate IronWolf 6TB 5400rpm
  • Intel Optane 800p 58GB (not in use yet, with Corsair cooler)
  • external 12V PSU, MeanWell GST120A with max. 102W
  • Alphacool Eisstation DC-LT pump+reservoir (running on 7V)
  • modified Aquacomputer Cuplex Kryos CPU cooler
  • custom made Anfitec VRM cooler (you can order it by writing them an email, maybe it will be available in their onlineshop soon Anfi-tec )
  • Watercool HTSF2 1x120 LT radiator
  • Fans:
    • Bitfenix Spectre Pro 230mm (running on 7V)
    • Noiseblocker NB-eLoop 120mm PWM
    • Scythe 80mm PWM
I did modify the case a bit - cut some more holes in drive cage for better air flow, made a faceplate with power socket and water temperature display instead of ATX-PSU, tried to acoustic damp every free space, added better dust filter in front.

1.JPG2.JPG3.JPG4.JPG



Temperatures right after starting minimal Debian are very low:
CPU1 Temp: 15°C (!!! - guess that's a failure)
MB Temp: 30°C
Card Side Temp: 44°C
DDR4_A1 Temp: 32°C
DDR4_B1 Temp: 34°C
Onboard LAN Temp: 49°C
Power usage: 43 Watts

after 1hour idling:
CPU1 Temp: 18°C
MB Temp: 33°C
Card Side Temp: 46°C
DDR4_A1 Temp: 35°C
DDR4_B1 Temp: 37°C
Onboard LAN Temp: 52°C

after 1hour stress (with s-tui, running workers for CPU and memory):
CPU1 Temp: 41°C
MB Temp: 40°C
Card Side Temp: 52°C
DDR4_A1 Temp: 57°C
DDR4_B1 Temp: 61°C
Onboard LAN Temp: 57°C
Power usage: 102 Watts average

HDDs: 37~38°C
SSDs: 33°C
 

csp-guy

Active Member
Jun 26, 2019
372
153
43
Hungary, Budapest
I've build a watercooled server based on this nice little mainboard.
  • Bitfenix Prodigy case with insulation material
  • Asrock Rack EPYC3251D4I-2T
  • 2x Kioxia SATA SSD 1.92TB
  • 2x Seagate IronWolf 6TB 5400rpm
  • Intel Optane 800p 58GB (not in use yet, with Corsair cooler)
  • external 12V PSU, MeanWell GST120A with max. 102W
  • Alphacool Eisstation DC-LT pump+reservoir (running on 7V)
  • modified Aquacomputer Cuplex Kryos CPU cooler
  • custom made Anfitec VRM cooler (you can order it by writing them an email, maybe it will be available in their onlineshop soon Anfi-tec )
  • Watercool HTSF2 1x120 LT radiator
  • Fans:
    • Bitfenix Spectre Pro 230mm (running on 7V)
    • Noiseblocker NB-eLoop 120mm PWM
    • Scythe 80mm PWM
I did modify the case a bit - cut some more holes in drive cage for better air flow, made a faceplate with power socket and water temperature display instead of ATX-PSU, tried to acoustic damp every free space, added better dust filter in front.

View attachment 18186View attachment 18187View attachment 18188View attachment 18189



Temperatures right after starting minimal Debian are very low:
CPU1 Temp: 15°C (!!! - guess that's a failure)
MB Temp: 30°C
Card Side Temp: 44°C
DDR4_A1 Temp: 32°C
DDR4_B1 Temp: 34°C
Onboard LAN Temp: 49°C
Power usage: 43 Watts

after 1hour idling:
CPU1 Temp: 18°C
MB Temp: 33°C
Card Side Temp: 46°C
DDR4_A1 Temp: 35°C
DDR4_B1 Temp: 37°C
Onboard LAN Temp: 52°C

after 1hour stress (with s-tui, running workers for CPU and memory):
CPU1 Temp: 41°C
MB Temp: 40°C
Card Side Temp: 52°C
DDR4_A1 Temp: 57°C
DDR4_B1 Temp: 61°C
Onboard LAN Temp: 57°C
Power usage: 102 Watts average

HDDs: 37~38°C
SSDs: 33°C
Hey, Nasi, very nice build, congratulations!
 
  • Like
Reactions: nasi

csp-guy

Active Member
Jun 26, 2019
372
153
43
Hungary, Budapest
Update 18/May/2021:

I decided to rebuild and reorganize my configuration:

Major changes:

- storage will be replaced with ssd drives. Instead of 2x2 5TB disk mirror + 2x500GB ssd mirror plus Optane 900P slog disk:
2x3.84TB S4510 in mirror (1 drive net capacity)
2xintel s4510 1.92TB 2xintel s4500 1.92TB, all in raidz1 (3 drive net capacity)
1xs3700 800GB + 2x S3710 800GB (2 drive net capacity)

- Optane 32GB boot drive - Optane 900P SLOG used from old boot and storage pool.
- front coolers has been locked with drilling a screw hole, and very good cable management using zipping ties.
- Cable management to reduce space used by cables.
- After some drilling inside the case -> 2 ssd is hidden under the 3.5 bays, 1 ssd is near the PSU.

10 drive in zfs ssd pool, plus boot ssd, plus SLOG need to be configured.

My mobo is still in RMA, after more, than 1 months we didnt get motherboard back from the distributor (Thank you, Asbis Hungary, for nothing, it is outrageously slow trader attitude, it was only VGA connector soldering problem). Finally they give the money back. :-(

New motherboard ordered from other distributor, i am waiting for the arrival.


Will attach photos, during the build.

Update 27/05/2021

2x3.84TB S4510 in mirror (1 drive net capacity) 2x3.84TB S4510 SSD + 1x3.84 DELLEMC Intel S4510 (2 drive net capacity) :)

Total storage available: 15.04TB net capacity.

And, finally tomorrow arrives the new motherboard. :-D
I am happy.
 

Attachments

Last edited:

nasi

Member
Feb 25, 2020
64
20
8
Hey, Nasi, very nice build, congratulations!
Thank you. I appreciate that very much.

In the meanwhile I have upgraded to 2x32GB RAM and Optane is now SLOG/L2ARC for the HDDs.
System is running since a while. I've installed Proxmox and have some Containers and VMs for Nextcloud, Tor Relay, Fileserver, Win10 Gameserver, Torrentserver, InfluxDB and Grafana. Temperatures are doing fine - I didn't see water temperature higher than 33°C yet (unfortunately not logging it), HDDs and SSDs are below 40°C, other temperatures inconspicuous. Power draw is ~50W. Together with all network stuff I've got ~96W which is nice. Loudness of the server is very low but not inaudible (yet ;)).
 
  • Like
Reactions: Senses3