Newbie needs to buy a high-end server - advice please!

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Tagaravabha

New Member
Feb 10, 2018
5
1
3
45
Dear server experts! I am a researcher looking into buying a high-end server for some memory-intensive simulations (mostly finite-element method for those that are interested). I know that I'll need a minimum of 1TB of RAM, as many physical cores as I can get (for parallel computing), and an array of 15K rpm hard drives, but I have no idea of what else might be needed. I came across the following specs for a *refurbished* server and wanted some advice on anything I might be missing that will be necessary for its smooth operation, or anything I might need to watch out for. I have some comments in-line with the specs and some additional questions that I've included at the end of this message, and would really appreciate your advice! Thank you in advance:


  • Dell PowerEdge R920 server 4U rack, supports up to 6TB of memory in 96 slots (I know this is discontinued and replaced with the R940)
  • 4 x Intel Xeon 6-core E7-8893 v2 3.4GHz 37.5MB cache/processor (are there faster/cheaper newer processors, e.g. i5, out there? Do they offer the same amount of cache?)
  • 64 x 16GB registered server RAM (does the brand of RAM really matter, e.g. between Hynix, Samsung, Kingston?)
  • 10 x DELL 600GB 15K rpm 2.5" SAS Hard Drives (is there any need for a RAID configuration if my priority is speed of writing and reading? Should some of the drives, e.g. for the OS, be excluded from the RAID setup? What kind of RAID would you recommend?)
  • 24 x 2.5" SFF SAS/SATA/SSD BAYS
  • PERC H730P 2GB NV cache RAID controller
  • IDRAC7 express (Is there a big difference between IDRAC7 and IDRAC8? Would the latter even be supported on the R920?)
  • No optical drive (is there ever a need for an optical drive?)
  • Redundant 1100W power supply (does "redundant" here mean two supplies with a total power of 1100W, or 1100W each? Is this enough for what I want to put in here?)
  • Broadcom 5720 1GB quad port daughter card
  • Up to 10 PCI-E slots: 8 x PCI-E 3.0 + 1 RAID slot + 1 NDC slot + 2 optional PCI-E slots (what are these, and what can I do with them?)
  • E-VGA NVIDIA GEForce GTX 980 TI 6GB (Will this card even work on this system? Can the power supply handle it? If not, what would be a good supported alternative? I don't need GPUs for computation, but will it help when accessing the server remotely?)
  • Rail Kit
  • No bezel


Some additional questions:


  1. What would be a decent price for the above system (please note, it is refurbished)?
  2. Are there any components that are not included, but which will definitely be needed? If so, what are they?
  3. I heard that Intel will stop supporting/selling these processors after 2019. Is this a big problem?
  4. Any suggestions on a good-quality reliable UPS for this server?


Thank you!

TRB
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
As someone from a similar field (CFD) with a little experience in FEM here are my two cents: what matters the most here is memory bandwidth, not CPU cores.
So you will certainly need a few cores for performance, but more importantly: enough memory bandwidth to feed the cores. Otherwise scaling will be bad and money on "faster" CPUs will be wasted.
Since you say you need 1TB of RAM, the only cost-effective option is older hardware that uses cheap DDR3 reg ECC. Speed matters here, not the brand. Cheap refurbished servers are often equipped with DDR3-1066 or even slower memory although the CPUs support up to DDR3-1600. For your applications, memory speed will affect performance almost 1:1. So make sure to check which memory type you get for your money. The CPUs themselves are a pretty decent pick for this type of applications: medium core count, high clock speed. More cores probably won't help much due to the lack of memory bandwidth and increasing parallelization overhead.
 
  • Like
Reactions: Patrick

Tagaravabha

New Member
Feb 10, 2018
5
1
3
45
Thank you! I believe the memory is 1600MHz. I'd be very interested to know your responses to my other questions - thanks again.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Any chance you can go SSD instead of those 600GB 15K rpm drives? For example: HGST WD s840 Enterprise Commercial SAS SED 2TB 6GB/s SSD 2.5" Z16IZF2E-2TBUCZ | eBay --- different seller but potentially a thread worth reading https://forums.servethehome.com/index.php?threads/2tb-hgst-s840-enterprise-sas-sed-ssd.18245/

You do not need a GPU if you just need 2D output. These servers have them built-in.

Agreed with @alex_stief that if you are looking for 1TB inexpensively, DDR3 is the answer.

You could also do 32x32GB DDR4 in a dual AMD EPYC server to hit 1TB, but you will have lower memory speeds and be buying more expensive DDR4.

I usually buy RAM in an order that goes something like: Samsung->Micron or SK.Hynix-> Others (only if I have to)
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
Thank you! I believe the memory is 1600MHz. I'd be very interested to know your responses to my other questions - thanks again.
I can only weigh in on stuff I know about. Questions left out indicate that I have no idea.
are there faster/cheaper newer processors, e.g. i5, out there? Do they offer the same amount of cache?
There are lots cheaper and faster processors. I5 and the likes are not an option since you probably need a shared memory system and these processors do not support more than one CPU per motherboard. Xeon E5-2xxx are only dual-CPU capable, E5-4xxx have less QPI links, i.e. slower interconnect for shared memory. The Xeon E7 v2 are about as good as it gets for what you need and possibly want to spend.
10 x DELL 600GB 15K rpm 2.5" SAS Hard Drives (is there any need for a RAID configuration if my priority is speed of writing and reading? Should some of the drives, e.g. for the OS, be excluded from the RAID setup? What kind of RAID would you recommend?)
Might be ok if you can't find a cheaper offer with less HDDs included. I strongly recommend SSD-based storage for pre- and post-processing. And depending on the FEM software, also for solving. Some commercial FEM packages are notorious for heavy disk-I/O during solving even if the model fits into RAM.
Redundant 1100W power supply (does "redundant" here mean two supplies with a total power of 1100W, or 1100W each? Is this enough for what I want to put in here?)
1100W each. The power output should be sufficient, Don't know if it has the PCIe connectors you need for a graphics card.
I don't need GPUs for computation, but will it help when accessing the server remotely?
AFAIK, usually not.
Any suggestions on a good-quality reliable UPS for this server?
Are you absolutely sure that you need one? Are power failures a regular occurrence in your area?
 

Tagaravabha

New Member
Feb 10, 2018
5
1
3
45
Thanks all for your help! In the end, I did purchase this server with a few modifications -- no HDDs (I bought 3 x 2TB Micron SSDs instead) and no graphics card. I also ensured that the RAM was Samsung and 1600MHz. It cost me CAD $14K.
 
  • Like
Reactions: T_Minus

BobbyB

Member
Dec 26, 2016
33
10
8
74
I also ensured that the RAM was Samsung and 1600MHz. It cost me CAD $14K.
Check the configured, actual speed of the memory once you get it.
The DIMMs themselves might be 1600, likely they will only run at 1066 or 1333 in 2DPC (dimms per channel). IIRC LRDIMMs (32GB and up in these R920s) can do 1333 always, 2-3DCP with ordinary RDIMMs are often limited to 1066 even with 1.5V in these R920s. Dell owners manual has the info.
Again, check what speed you can set.
Had a quick look on some R9x0's in work - R920 with 32GB LRDIMM 1600 runs at 1333. R930 E7v4 with 32GB LRDIMM DDR4 2400 runs at 1600 and R940 with Skylake-SP has 64GB LRDIMM 2666 runs at 2666. Each box with all slots populated.
 

Tagaravabha

New Member
Feb 10, 2018
5
1
3
45
Check the configured, actual speed of the memory once you get it.
The DIMMs themselves might be 1600, likely they will only run at 1066 or 1333 in 2DPC (dimms per channel). IIRC LRDIMMs (32GB and up in these R920s) can do 1333 always, 2-3DCP with ordinary RDIMMs are often limited to 1066 even with 1.5V in these R920s. Dell owners manual has the info.
Again, check what speed you can set.
Had a quick look on some R9x0's in work - R920 with 32GB LRDIMM 1600 runs at 1333. R930 E7v4 with 32GB LRDIMM DDR4 2400 runs at 1600 and R940 with Skylake-SP has 64GB LRDIMM 2666 runs at 2666. Each box with all slots populated.
Thanks - the RAM modules I've ordered are 16GB each (64 x 16GB); are you saying these are not supported by the R920?? If they are, will I still see the same slowdown? I was told the model of the memory would probably be Samsung M393B2G70QH0-YK0...
 

BobbyB

Member
Dec 26, 2016
33
10
8
74
Thanks - the RAM modules I've ordered are 16GB each (64 x 16GB); are you saying these are not supported by the R920?? If they are, will I still see the same slowdown? I was told the model of the memory would probably be Samsung M393B2G70QH0-YK0...
They should work, but likely the speed will be set to either 1333 or 1066. Check the R920 owner's manual, it should list the memory speed settings with the combinations or RDIMM/DPC/voltage/mode. I honestly don't know if it will let you override it to 1600, maybe it does.
 

Tagaravabha

New Member
Feb 10, 2018
5
1
3
45
Thanks - I checked the manual and it looks like 16GB RDIMM/DPC=3/1.5V will run at 1066MHz. It says that 1.35V is "not applicable" -- does that mean it wont work in this configuration?