Cheap numbercrunching workstation

  • Thread starter Deleted member 6328
  • Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
D

Deleted member 6328

Guest
Hi,

I haven't built a system for over ten years now and there has certainly been a lot of progress since then. Enough to where I feel I really don't have a good overview of the market anymore. It used to be simple!

This will be my typical workload:

  • ESXi platform
  • One Win7 Professional instance running scientific CAD software such as Ansys.
  • One instance running Ubuntu 14.04 with typical tasks being compiling, GNU Radio development, etc.
I have a Dell Precision M6600 workstation now and it's great. 32GB ram and 2760QM CPU. Three SSD drives.

Working with CAD on VMWare under Ubuntu is OK but a bit painful. I need more cores and RAM also.

So what about:
  • Dual E5-2670
  • Motherboard with DP and 16 DDR sockets. Probably start with 64GB but might want to upgrade to 128GB later.
  • Dual grapics card. I want to send each card via VT-d to Linux and Win7 respectively.
  • 1TB, or dual 512GB, SSD drives for OS installation.
I have dual boot on the M6600 but find myself always booting into Linux and working from there. It's just too much of a hassle to reboot and I need both environments concurrently.

I have a case that I can use. Can someone comment on feasibility? I'm happy with the 2670QM single thread performance and from what I understand it's been mostly incremental performance upgrades last couple of years. I do need AVX though so nothing less than Sandy Bridge will do. It does make a big difference in DP matrix solve performance as well as FFTW3 what I find myself waiting for.

Comments?

/K
 
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,517
5,812
113
Does the Ubuntu instance need a GPU? Running Windows 7 Pro with Hyper-V would allow you to run Ubuntu 14.04. GPU pass-through would be the harder thing to use there.

Duel E5-2670 is inexpensive and very nice.
 
D

Deleted member 6328

Guest
Diavuno said:
Do you need a mobile workstation?

Going from a Second gen I chip to a 5th or 6th gen will blow your mind. it's pretty drastic!
Click to expand...
I was under the impression that later generations actually didn't improve performance much and especially not considering that I can get a used E5-2670 for USD100. This is pretty unheard of in terms of price/performance.

Yes, I need a mobile workstation but I already have it. The Dell Precision M6600. So now I want more cores and more memory thus my planned build! I can always benchmark against my HPC cluster at work which runs E5-2680 v3 DP w/ 256GB RAM on each node.

Keljian said:
What's your budget?
Click to expand...
A very relevant question of course. I checked ebay and taobao and came to the conclusion that since I have a case with PSU I should be able to get:
  • 2 x E5-2670 = USD200
  • 8 * 16GB ECC RDIMM Samsung = USD720 OR 8 * 8GB ECC RDIMM Samsung = USD200
  • Motherboard: I don't know but perhaps an ASUS X9 model will be ok? Some Intel C602? Unsure here.
  • GPUs: I don't require a lot of horsepower but would like to run both at the same time.
So trying to stay under a USD1200 or maybe USD600 if going with 64GB.

Patrick said:
Does the Ubuntu instance need a GPU? Running Windows 7 Pro with Hyper-V would allow you to run Ubuntu 14.04. GPU pass-through would be the harder thing to use there.

Duel E5-2670 is inexpensive and very nice.
Click to expand...
Ideally I would like both instances to have their own physical GPU via VT-d if that is at all possible.

Yes, the E5-2670 seems perfect for this.
 
D

Deleted member 6328

Guest
Just a very interesting update... I ran some real world benchmark on the FEM code that I intend to run. This is commercial code so beyond control as far as recompiling is concerned.

Code: 
Name          i7-2670QM        i7-4600M      E5-2680v3
Base               2.4          2.9          2.5    
Turbo              3.5          3.6          3.3
Total            06:29          06:33         07:33
Freq              :37             :41            :47
This is single threaded performance. This code loves cores so that's great but certain parts will always run single threaded and besides it's the only way to compare across CPUs so that's what I'm sticking to.

The i7-2670QM Sandy Bridge has the newer architectures beat! I'm going with E5-2670 for sure now.
 
K

Keljian

Active Member
Sep 9, 2015
428
71
28
Melbourne Australia
I would be very interested to know how it runs on broadwell.. (I7-5xxx or Xeon e3-1xxx v4.) 128 meg of local cache could make a big difference..
My desktop is an i7-5775c w/32 gig of ram if you would like me to run some tests.
 
  • Like
Reactions: Patriot and gigatexal
Patriot

Patriot

Moderator
Apr 18, 2011
1,451
792
113
Kazam said:
Just a very interesting update... I ran some real world benchmark on the FEM code that I intend to run. This is commercial code so beyond control as far as recompiling is concerned.

Code: 
Name          i7-2670QM        i7-4600M      E5-2680v3
Base               2.4          2.9          2.5   
Turbo              3.5          3.6          3.3
Total            06:29          06:33         07:33
Freq              :37             :41            :47
This is single threaded performance. This code loves cores so that's great but certain parts will always run single threaded and besides it's the only way to compare across CPUs so that's what I'm sticking to.

The i7-2670QM Sandy Bridge has the newer architectures beat! I'm going with E5-2670 for sure now.
Click to expand...
That V3 is most certainly newer than the V1 ... Sandybridge is from 2011.
However single threaded code runs best at highest clock, I would check to see what frequencies those chips are actually holding, probably not getting full clocks on the V3 if it is losing. You would need to weight the time spent in single vs multithreaded to make the best choice.
 
D

Deleted member 6328

Guest
Patriot said:
That V3 is most certainly newer than the V1 ... Sandybridge is from 2011.
However single threaded code runs best at highest clock, I would check to see what frequencies those chips are actually holding, probably not getting full clocks on the V3 if it is losing. You would need to weight the time spent in single vs multithreaded to make the best choice.
Click to expand...
Well, both the HPC cluster and the code is beyond my control. I can certainly check what frequencies I'm getting on the E5 v3. It should turbo to 3.3GHz. However, even if I discover I'm getting less I can't do much about it. HPC is more about being able to solve huge problems than solving small problems fast. I usually run 200+ cores for 72+ hours to solve the problems I encounter in my work.

That's not the focus for the home build of course, there I just want something more capable than the 4 cores 32G RAM I have in my M6600. 16/128 looks like the sweetspot and with these performance numbers I would be quite happy to "settle" for Sandy Bridge. Obviously AVX2 doesn't really help. I have noticed that the code is compiled against Intel MKL so I would be surprised if that's the problem.

Keljian said:
I would be very interested to know how it runs on broadwell.. (I7-5xxx or Xeon e3-1xxx v4.) 128 meg of local cache could make a big difference..
My desktop is an i7-5775c w/32 gig of ram if you would like me to run some tests.
Click to expand...
Do you have access to an Ansys license server?
 
K

Keljian

Active Member
Sep 9, 2015
428
71
28
Melbourne Australia
No I don't have access to an Ansys license server, sorry..but it does look like the student version is a free download, I am sure Ansys wouldn't mind if testing was the aim
 
G

Gnodu

Active Member
Oct 10, 2015
115
37
28
49
Consider the Intel w2600cr2 (uses sandy bridge or ivy bridge; ivy requires bios update) from oemxs-- they sell a nice 4u case version (don't remember exact model- but can look it up if you want) that is larger than you likely need, but the mobo is rock solid in my experience, and the ability to expand is nice too. I think I paid 225 for the barebones one-- was cheap enough that I bought a second as a backup.

I think they are on vacation until early January, but if you can wait a couple days, might be worth considering.

Lesson learned on Intel stuff from them:
1) motherboards may require a sandy bridge chip to boot and update bios before you can swap in an ivy bridge.
2) Intel motherboards are not willing to take ES chips if stepping less than the commercial/marketed versions end up being. (In the CR2/IP4 bios notes, there is actually a point where the update says "no steppingstone below C0), so if planning to go ES, be very careful.
3) I have never worked with better made cases/hardware/physical stuff than Intel puts out- at least in terms of being solid/feeling like it is going to withstand any abuse I can throw at it.
4) modular drive slots like Intel offers are mighty convenient-- BUT on the 3.5in backplane, make sure you move the jumper in the corner to run drives at full speed (odd, and easy to miss)
5) learn to edit the FRUSDR (esp the SDR) file to get fan speed to a manageable level. Mine is super quiet and runs cool even at load-- but as delivered from the factory... Was more like a jet engine.

Just my two cents- best of luck!

Note: I am not running ESXi, so cannot speak to that. However, I hear that the IP4 (server not workstation board) is good with it.
 
  • Like
Reactions: T_Minus
D

Deleted member 6328

Guest
Checked out the W2600CR2 and it doesn't support VT-d. :(

The S2600IP4 does however so that seems a good candidate. Have to find a good source though.
 
G

Gnodu

Active Member
Oct 10, 2015
115
37
28
49
The board manuals are the same (literally) for both... I have both machines. I bought the IP4 in a 2u config as well as two CR2 (the 4u I mentioned above. Checking eBay history:
P4304CR2LFGN - $225 each - this is the one with no drive bays built in, but another $200 in parts built out a really nice setup
R2312IP4lHPC - $600, includes raid daughter board, a sas expander, very complete setup for LFF/3.5 drives. I think the orig/listed price was 1200 or 1400?

I have no idea what they are accepting/priced at now, as their eBay store is closed until 1/4 (says they are on vacation)... But they are, bar-none (again, in my opinion) the cheapest place for Intel parts/servers.

Couple other thoughts:
The CR2s I got only came with one power supply- extra one for $25-
The CR2 and IP4 boards share the same manuals and docs, but there are definitely differences. The IP4, for example, has no USB3 built in, but the CR2 has sound on the motherboard (IP4 does not)
Tech product specs (page 25- not attached) says supported, but they may be referring to the e5-26xx series in general. However, page 48 (attached) looks like a solid "it's supported" to me. (Will be home this weekend and can look at the bios)
**just to be clear- since I sound like a walking commercial for the OEMxs people, I am NOT related to them in any way- am just a very happy customer.
Your question on the GL board- I remember looking at that one, but opting to NOt consider it, but looking back, I don't remember why. It also has a close relative, the GZ version. May have been the rest of the server config though, I don't recall- I was looking at the whole package, not just the motherboard at the time.

Best of luck- would love to hear how it works out for you in terms of speed/improvement!image.jpg
 
D

Deleted member 6328

Guest
Gnodu said:
The board manuals are the same (literally) for both... I have both machines. I bought the IP4 in a 2u config as well as two CR2 (the 4u I mentioned above. Checking eBay history:
P4304CR2LFGN - $225 each - this is the one with no drive bays built in, but another $200 in parts built out a really nice setup
R2312IP4lHPC - $600, includes raid daughter board, a sas expander, very complete setup for LFF/3.5 drives. I think the orig/listed price was 1200 or 1400?

I have no idea what they are accepting/priced at now, as their eBay store is closed until 1/4 (says they are on vacation)... But they are, bar-none (again, in my opinion) the cheapest place for Intel parts/servers.

Couple other thoughts:
The CR2s I got only came with one power supply- extra one for $25-
The CR2 and IP4 boards share the same manuals and docs, but there are definitely differences. The IP4, for example, has no USB3 built in, but the CR2 has sound on the motherboard (IP4 does not)
Tech product specs (page 25- not attached) says supported, but they may be referring to the e5-26xx series in general. However, page 48 (attached) looks like a solid "it's supported" to me. (Will be home this weekend and can look at the bios)
**just to be clear- since I sound like a walking commercial for the OEMxs people, I am NOT related to them in any way- am just a very happy customer.
Your question on the GL board- I remember looking at that one, but opting to NOt consider it, but looking back, I don't remember why. It also has a close relative, the GZ version. May have been the rest of the server config though, I don't recall- I was looking at the whole package, not just the motherboard at the time.

Best of luck- would love to hear how it works out for you in terms of speed/improvement!
Click to expand...
I think the odd form factor of the GL4 works against it. Requires a rack mount style case with some special PSU interconnect if I interpret things correctly. This is the reason I'm not going for it at least!

Great find on the CR2, this is definitely on my short list now and looks just about perfect. Thanks for the input.

Read the article here on STH on why there's a lot of E5-2670's flooding the market now. Thanks Facebook! I will look out for the C2 stepping.

I did talk to the Ansys KAM about speed differences and he ran it across their FAE. Turns out solving sparse matrices for multiple RHS is all about AVX and clock speed. Not a lot of improvement with AVX2. Hyper Threading should always be turned off, and I certainly always do. It degrades performance for these heavily parallelized codes.

When you find yourself needing to go to XEON and server grade components for HPC you very often have one of two use cases:
  • The problem you're trying to solve is ginormous and simply won't fit in the available RAM on any one node. You need to distribute over nodes to make use of multiples of 256GB or 512GB.
  • You need to run several hundred smaller problems in a parameter sweep each one requiring a few minutes and a few GBs of RAM.
In either one of theses cases shaving off say 20% solution time isn't really the main driving force. Either you can solve it or you can't. Since Case 1 will take several days to solve anyway another few hours won't matter.

In case 2, which is the one I benchmarked above, you will wait for the solver a few minutes. It doesn't matter so much if it's 5 or 6 minutes. But if in that timeframe you get 1 solution or 100 solutions that's definitely something that will influence your workflow and improve your design.

I digress but this is interesting stuff! For me at least... :)

So, to conclude I'm totally blown away by the price/performance ratio the dual E5-2670 allows for this particular workload and I will try to get two full setups going with 16 x 8GB ECC 1600MHz RAM and dual E5-2670's. Only need one 512GB SATA3 drive to hold Server 2012 HPC and application software.

Will check out OEMxs!

As far as student version of Ansys is concerned that's not really an option for benchmarking. Very limited.

Will keep you posted on progress. Thanks!
 
K

Keljian

Active Member
Sep 9, 2015
428
71
28
Melbourne Australia
Last edited:
D

Deleted member 6328

Guest
Keljian said:
Combine ANSYS Mechanical* with the Intel[emoji768] Xeon Phi[emoji769] coprocessor for dramatic performance gains in both Windows* and Linux* environments | Intel[emoji768] Developer Zone

This might be one time you want a Xeon phi coprocessor board, (they have AVX, in spades) and they're going cheap...


Intel Xeon Phi Coprocessor 31S1P Special Price Promotion of $195*

Could passthrough to a VM...
Click to expand...
Not supported for Ansys FEM solvers, and really not applicable since FDTD codes make better use of memory throughput. However, there's always CST and they are supporting Phi:
Xeon Phi | CST

Wasn't aware that these things were going so cheap. What happened? Might pick up a few for that price... Thanks for the tip!
 
Boddy

Boddy

Active Member
Oct 25, 2014
772
144
43
Kazam said:
Wasn't aware that these things were going so cheap. What happened? Might pick up a few for that price... Thanks for the tip!
Click to expand...
FYI. Purchases of 10 or more are $125 each.

Hopefully there is a group buy in progress if you are interested. Cheers
 
D

Deleted member 6328

Guest
Boddy said:
FYI. Purchases of 10 or more are $125 each.

Hopefully there is a group buy in progress if you are interested. Cheers
Click to expand...
Wow. Yes, I'm interested in two for that price. I'm in Europe but I'm sure that can be arranged. Thanks!
 
You must log in or register to reply here.
Top Bottom