buying suggestions for DDR3 era workstations...

[Twice_Shy]

Okay so a few comments there are useful. A note to prefer hp/dell due to cost which is one of the big factors for my use case.

So when I analyze or think about things, yes it doesn't matter if something is a custom form factor, or case, or even PSU as long as spare parts OR adapters are available inexpensively. What i'm mostly looking for are will be lots of RAM slots and an abundance of PSU power so i'm not upgrading that just to put in a bunch of drives or a newer gpu.

Full size ATX or 7 slots (with 2 pci x16 if possible) might be needed too, even if it's not a standard form factor - I just mean it's about available slots because I need to plug in a number of x1 cards and probably an SAS LBA along with a gpu. Proxmox needs to pass thru some dedicated USB cards and such.

More than one cpu socket (dual and even quad considered for some reasons or experiments - obviously not low wattage ones ) especially if it doesnt require all sockets filled is a plus... or maybe they all do that? I'd heard the HP Z-series can use cheaper xeons intended for quad socket boards in the dual or even single socket ones... unless others do that?


A bit newer, I learned apparently Lenovo is locking ryzen cpu's to specific computers making the motherboard/workstation un-upgradeable and a used cpu unusable outside Lenovo(!). Linus tech tips had a video on it a year ago. This will be worth knowing in the not far off future but i'm taking notes now.

I'm also seeing lenovo 'workstations' of even only 7 years ago (like thinkstation p310) are looking about as uselessly tiny and lightweight as whats driving me away from nearly all desktops - 250w psu and cramped microATX cases. Maybe this is new or maybe this is just them but it lets me know what not to scan craigslist and such for. Thats a workstation in name only to me.

I'd heard Fujitsu workstations have hard to find parts.. i'm just compiling things i'm learning in case it's useful to anyone else.

 

[zachj]

In my opinion the sweet spot for legacy systems would be the hp z440.

the z420 is ddr3 sandy bridge/ivy bridge and can be had for a song but those are more than a decade old at this point—you’re asking for trouble investing in something so old. My z420 and z620 both started acting really weird last year just out of the blue.

The z620/z640 both offer a 2nd cpu on a riser card with additional ram slots and aren’t much more expensive that the z4x0 siblings, so that makes them look appealing. But realistically you can run a minimum of 256GB of ram on the z4x0 machines and 10+ cpu cores on the z4x0 machines; if your use case can genuinely take advantage of more horsepower than that then I would argue strongly you should be looking at Xeon scalable or epyc.

the z820/840 really only differ from the z620/z640 in the number of memory slots. See above for why I just don’t think you ought to care about this. The z8x0 machines also sell for a premium over the z6x0 machines that I just don’t think can be justified.

 

[TRACKER]

zachj,
" the z820/840 really only differ from the z620/z640 in the number of memory slots. See above for why I just don’t think you ought to care about this. The z8x0 machines also sell for a premium over the z6x0 machines that I just don’t think can be justified. "

That's not actually true i have both z620 and z820 and they differ big time

1) z820 can use LRDIMM, z620 cannot - so my z620 is maxed out on 192GB and my z820 on 512GB RAM
2) z620 pcie lanes are only from 1st cpu which means 40 max. z820 on other hand uses pcie lanes from both cpus

 

[zachj]

zachj,
" the z820/840 really only differ from the z620/z640 in the number of memory slots. See above for why I just don’t think you ought to care about this. The z8x0 machines also sell for a premium over the z6x0 machines that I just don’t think can be justified. "

That's not actually true i have both z620 and z820 and they differ big time

1) z820 can use LRDIMM, z620 cannot - so my z620 is maxed out on 192GB and my z820 on 512GB RAM
2) z620 pcie lanes are only from 1st cpu which means 40 max. z820 on other hand uses pcie lanes from both cpus

hp doesn’t officially support lrdimms on the z420/z620 but they work just fine. There’s a STH forum user doing so with 64gb lrdimms right now…I myself had 32gb non-lrdimms also working fine. Sure the z820 supports more dimms but most people won’t need more than 512gb of ram on a single host so it’s a bit of a moot point—especially given how damn cheap 64gb ddr3 lrdimms have become.

NOTE: this requires the rev2 motherboard with support for ivy bridge; everything sucks quite a bit more on the rev1 motherboard :-(

yes I know the PCIe lanes are different on the 820 but keep me honest:

1. On the z820 some of the slots (specifically slots #3 and #4) don’t work without a 2nd cpu installed, whereas on the z420/620 all slots work regardless of cpu quantity
2. Yes the 820 gives you an extra physical slot compared to the 420/620 but for many users that doesn’t justify the price premium nor make it easier to accept just how effing big the z820 is dimensionally (jeez!)

In short the z620 is only inferior to the z820 if one wants more than 512gb ram or if one wants more than 5 pcie slots—that’s not very many people. I guess it’s fair to say I glossed over the pcie slot differences but I only did so because I don’t consider them to be material.

in my mind: z820 is almost identical to a dl380 g7 in a tower chassis with the only big technical difference being that the server has a lot more bios options and supports 3 dimms per channel (which again seems moot given how cheap 64gb lrdimms are—3dpc only matters if you want more than 1tb ram).

 

[Twice_Shy]

Bringing everyone up to speed...

And also trying to show i'm doing my own research/not just making others do it for me. I can post more info of where I found or what I read (if anyone else who doesn't know is curious/if they're learning with me) but I don't want to make it too long to read.

I'm PRETTY much narrowed it down to the Dell Precision 5810 line and those 3nd gen HP Z series. If I found the 2nd gen locally for nothing esp the higher ones like Z620/820 i'd take that too.

The Dell Precision 5810 as near as I can tell is the CHEAPEST viable workstation of earliest DDR4 era computing, will take up to 256gig, and has a very stout 825 watt PSU apparently. (it's wierd but later models i'm looking at they seem to seriously scale this down? However I may not understand Dell's numbering and product lines - if anyone can fill me in better?) Dual x16 gpu slot support. Although i'm more likely to shove an SAS LBA in the 2nd slot - but that's not as viable in a consumer desktop board since the 2nd slots are usually x4 electrical slowing it down.


The HP Z Workstations are what i'd heard about before - but they seem to go for a bit more than the Dell Precisions. However there are some things that I like about them, most notably: E5-46xx v4 processors work in the Z440, Z640 and Z840 workstations! you can use the E5-46xx v4 cpu line for 100% sure instead of only the E5-26xx and E5-16xx cpu lines. These cpu's have lower demand used because as near as I can tell they do NOT work for most of those guys "using cheap xeons" - although lower speed there are 18 cores for $55 in one socket available. I don't even know if it will work in the Dell Precision. It may (I found one suggestion it's possible, possibly more of a limit on number of cores to 14 unlike HP at 22) but not more than that, so I have more faith in HP Z-series being able to use this.


I would even consider the newer Z8 Z6 Z4 skylake/coffee lake xeon supporting generation - those seem to be on ebay for not that much more. What I don't like is they have smaller physical cases - I want to pass thru extra USB x1 cards in Proxmox in one or two (more if it works) future projects.


Some people have even hackintoshed the Z440 which is of interest to me - it's less clear if there are as many successful Dell Precision hackintosh builds.


If I find supercheap Z420/620/820 locally for nothing for DDR3 i'll consider it, but i'm getting more swayed to just entering the DDR4 era even tho I plan bigger future RAM needs for editing 8k video at some point. DDR4 ram should continue to go down until it reaches parity with DDR3 which is probably when i'll go past 64-128gig. And even if I upgrade the workstation to a Z8 Z6 Z4 the ram would go with, and should go newer than that in the future if the ram speed keeps up.


I've chosen to skip Lenovo and Fujitsu workstations entirely (are there other major makers for the US?) since they don't sound competitive used - and if anyone can fill me in on alternate Dell Precision numbering-scheme models worth eyeing in addition to, or which might even be better than, i'm all ears still. Not locked in stone, just getting a clearer picture of what to seek.

PS - Zachj - double thanks for that factoid on LRDIMMS, thats' an example of the kind of specific info I was looking for to help me distinguish or choose one workstation over another. (are they just less expensive used? Or...) Hopefully what I shared about E5-46xx cpus in HP's is useful to someone.

 

[zachj]

Ddr3 lrdimms are dirt cheap because they can only be used on motherboards that were produced prior to like 2014(?) so there’s just nobody buying them and meanwhile data centers are dumping them.

that’s the same reason why low-speed ddr4 rdimms are also cheap.

id encourage you to look at the z440/640/840 over the z420/620/820 because the z*40 machines support pcie bifurcation and (as you said) any ddr4 you buy for it would be reusable with a later upgrade to z6/z8.

 

[paf]

About the Dell 5810: There are two different versions.
One with a C610 chipset (V3 Xeons only).
One with a C612 chipset (V3 and V4 Xeons).

The Dell 7810 has the C612 chipset and supports V3 and V4 Xeons.

The C612 chipset is the one used on the HP Z440, Z640, Z840 and Lenovo P410 machines (among others).

 

[Twice_Shy]

Is ebay going to be the best prices on bulk LRDIMMS for both DDR3 and DDR4? This sounds like a must have feature for me since i'm looking to go to alot of ram for multiple workstations to get several people set up. [EDIT: Though if it's only mobos thru 2013 even for DDR4 then that RAM wont be usable in newer machines by definition. It may still get me to 64-256 the cheapest but has the same limited re-use as DDR3 meaning DDR3 era machines are still on the table. ]

Also another thanks to paf for that bit of info - I know very little about Dells (or any other workstation outside HP's series) other than i'd seen that specific model number repeatedly mentioned as 'the cheapest viable workstation' and seeing prices on ebay that reflected that. There could also be other numbers or bits of useful info I dont know yet.


[EDIT2:] Does anyone have info about whether or not LRDIMM's (assumedly ddr4) are supported in xeon chips of Skylake, Kaby Lake, or Coffee Lake? I'm having a hard time finding definitive info. Looking around I see an article even from here even https://www.servethehome.com/intel-skylake-ddr4-rdimm-support/ saying skylake supports RDIMM (does that mean LRDIMM automatically, I assume yes?), and intel suggests Coffee Lake is back to UDIMM (Is Registered ECC Memory Compatible with Intel® Xeon®... though its specifically on the 2000 series, unless other series are different) though I can find nothing talking about Kaby Lake.

I'm trying to figure out how long my RAM might last me if I migrate LRDIMM DDR4 to newer systems, maybe thru the older 4th generation (current seems 5th gen) HP Z4 Z6 Z8 series if i'm researching right, which i'd be happy with, just sounds like it may need careful selection for compatibility with both older and newer.

 

[zachj]

Xeon scalable (look for the words “bronze,” “silver,” “gold” and “platinum”) first/second/third generation all support lrdimm ddr4.

if you buy ddr4 lrdimm it should be compatible with haswell/broadwell/skylake/cascade lake/ice lake Xeon processors. So essentially any cpu made between ~2013 and 2022.

Be mindful that older ddr4 will be slower speed so you’d definitely be hobbling an ice lake Xeon by trying to use 2133mhz memory. That said I’ve never checked to see what is the slowest ddr4 speed supported by the newer xeons—I assume they’ll support down to sub-2000mhz but I don’t know so.

 

[Twice_Shy]

Oh, so by ddr4 lrdimms 'also being cheap' like ddr3 you didn't mean they were only for equally earlier motherboards, just that lrdimms in general are cheap, and the ddr3 ones are stuck in the past. I was confused seeing apparent lrdimm support on newer mobos but wondering if something else had changed..

That makes it even more useful...

EDIT: I realize where part of my confusion was, Is Registered ECC Memory Compatible with Intel® Xeon®... suggests coffee lake e3-2xxx dont support rdimm, so this is part of my confusion with the xeon scalable era.

Searching ebay and seeing lrdimms of ddr3 for less than half of what ddr4 are - I can go either way at this second but it's clear ddr4 will be the way forward then.

My immediate use is absolutely RAM bottlenecked - it's not that things like faster cpu's are also faster, it's that the fastest bottleneck i'll hit with desktop boards is just running out of RAM. Premiere builds often go to 128gig now, After Effects to 256gig or even more. I need to run this software but nobody is being paid by the hour (yet or for awhile) for their work - this is just to enable people to run AV software on the cheap and volunteer some time to show what they can do or/and learn for awhile.

If me and those friends get out of the equipment shortage hole it will hopefully shift from hobby to paid work just enabling the buying of modern gear to do proper jobs. That may take a few years though just like an actor cant predict when (if) they get a big break.


My last remaining confusion, more for the future at this point, is sockets. There's so many changes into the xeon scalable era, which is probably by design. I might just make a separate topic because we've expanded beyond ddr3 workstations by now.

 

[nexox]

My last remaining confusion, more for the future at this point, is sockets. There's so many changes into the xeon scalable era, which is probably by design.

Only the first gen scalable in the LGA3647 socket are affordable at all, second gen CPUs fit but they're still somewhat more expensive, later sockets are still too new and don't have the quantities of used parts available that leads to $14 xeons.

 

[zachj]

Only the first gen scalable in the LGA3647 socket are affordable at all, second gen CPUs fit but they're still somewhat more expensive, later sockets are still too new and don't have the quantities of used parts available that leads to $14 xeons.

E3 xeons are not in the “scalable” family; they’re basically rebadged consumer processors with ecc enabled but they only support udimms.

what you’re looking for is exclusively socket 2011, socket 2011-3 or socket 3647. Note that “2011” refers to 2011 pins, not to a calendar year

any workstation/server board you buy with either of those three sockets will support rdimm. Most will work with lrdimm but you’ll probably find a lot of manufacturers only claiming official support for rdimm; of the manufacturers claiming official support for lrdimm few are claiming support for 128gb+ modules and even fewer have a QVL with more than like two such modules. In other words if you want to use lrdimms you’re probably going to find yourself having to hunt this forum to find proof that your specific combo is going to work.

as an example HP doesn’t claim ddr3 lrdimms work on the z420/z620–they only claim support for them on the z820–but they work just fine. And HP also doesn’t claim support for 64gb lrdimms even on the z820–they only claim support for 32gb lrdimms—but the larger modules do indeed work on all three workstations. This lack of claimed official support is due to a combination of market segmentation—HP wants you to buy up to the more expensive models—and a simple lack of testing (which is probably motivated both by a desire to get you to move to a newer model and the fact that availability of 64gb ddr3 memory modules trailed availability of the z-series workstations by several years and hp could be forgiven for not wanting to validate them on a by-then 2-3 year old platform).

 

[Twice_Shy]

That's super interesting and useful... both comments... so Socket 3647 will be my way forward at some point when I move on from like Z440 machines, at least up until Cascade Lake on that socket - looks like that goes all the way up to the top of the line xeon platinums for compatibility? It's ok if theyre not affordable yet - i'm probably good a few years with some big ram Z440's. What about Ice Lake and 3rd gen Xeon era I guess thats after? That looks like another new socket LGA 4189 though I imagine it take what 5-6 years to become affordable? (maybe someone more familiar with this for longer can guesstimate)

Browsing List of Intel Xeon processors - Wikipedia I also see E3 v5/v6 chips for LGA1151 - so those must be rebranded desktop parts just given a xeon name but not really a xeon? (I had been confused reading somewhere that intel was going to stop allowing the same socket to be used for consumer and server after haswell, and then v5/v6 for lga1151 seemed to do exactly the same thing, but i'm assuming that was the end of the line and those parts arent much cheaper than used consumer parts due to the compatibility)

I'd still been curious if E7 chips of later haswell could be found both cheaper and usable on any lesser hardware (it says lga2011-1 - which I assume isn't compatible with lga2011-3) than some high end 8 socket server.

 

[nexox]

You can actually occasionally find a cheap 4 socket E7 system, depending on configuration they can take either ddr3 or ddr4, but you don't want to have to pay for power on one of those things, and they have a bunch of odd parts that would be difficult to replace.

 

[zachj]

with respect I think at this point you should graduate from admiring the problem to making a decision

there isn’t a dramatic savings in E7 vs E5 xeons. E3 xeons don’t meet your needs (they only support UDIMM memory). Ice lake xeons aren’t in your budget and neither are cascade lake xeons.

As per everything you’ve said your choice is narrowed to:

1) sandy bridge E5-2600
2) ivy bridge E5-2600v2
3) haswell E5-2600v3
4) broadwell E5-2600v4
5) skylake 1st generation scalable bronze/silver/gold/platinum

within the HP workstation ecosystem that means you’re choosing between:

1) z420/620/820
2) z440/640/840 (ddr4)
3) z4/z6/z8 (ddr4)

of those the 4/6 variants make most sense and are cheaper than the 8 variants and #1 isn’t dramatically cheaper than #2 so I don’t see much sense in choosing #1.

If you narrow it down to #2 or #3 then, since you are so price conscious, I think your choice should come down to how many cores you think you need; if you need more than 8 cores #2 will quickly be cheaper than #3.

 

[alaricljs]

I've got an X9DR3-F w/ 2x E5-2630Lv1 and 16x16GB DDR3-10600 ECC sitting around waiting for me to get rid of it.
I've got an X10DRH-CT w/ 2x E6-2660v3 and 4x32GB DDR4-2400 ECC in the midst of being decommed...

In case you're interested

 

[Twice_Shy]

with respect I think at this point you should graduate from admiring the problem to making a decision

there isn’t a dramatic savings in E7 vs E5 xeons. E3 xeons don’t meet your needs (they only support UDIMM memory). Ice lake xeons aren’t in your budget and neither are cascade lake xeons.

If you narrow it down to #2 or #3 then, since you are so price conscious, I think your choice should come down to how many cores you think you need; if you need more than 8 cores #2 will quickly be cheaper than #3.

I've already made decisions, this is more figuring out longer term plans for what makes either a good upgrade plan, or things to watch for. Like with shipping being about $30/computer at cheapest, if I find some more local cache of HP or Dell workstations of certain models, which are worth grabbing a dozen of to sit on because I can use them eventually?

Your logic is not wrong. My first buying decision has been primarily about inexpensive RAM. The next decision was actually about PSU's I dont have to immediately upgrade - picking those workstation models that can run like a few heavier weight GPU's. (including older stuff like K80 Teslas for some people to learn on or experiment with) Everything I want or could plan around is definitely in HP's product line - i've mostly been comparing against that, like Dells seem cheaper at times.

My reason for doing all the research is i'm sort of the shoe-in unexpected sysadmin for a group of people. I'm building a few test machines, if they work well, I get everyone else set up and between us I expect to buy a terabyte or more of RAM easily.

8 cores is enough for something like Premiere, but Adobe After Effects will take anything you throw at it - Puget Systems suggests like 32 cores (!) is a good idea that it can use with max efficiency keeping them all loaded and it will happily use 64 too! So at some point if i'm buying 4 or 8 cpu's per workstation savings starts to add up, especially in those situations where two sockets with half the desired cores each costs less than a single chip.

But at some point modern core architechture becomes faster per-clock and something like 16 Ryzen cores at the same clockspeed will equal 32 older sandy bridge Xeon cores while using notably less power. (though probably not support the RAM, again that is always my _first_ bottleneck 100% of the time right now)

At some point the power bill starts to become relevant - only for PC's that are up 24/7 for awhile. Some of this is about building cheap backup PC's too - i've had a PC die in the middle of doing a job and decided I need to have a backup, I don't care if it wastes power, just able to finish a job I start if the unexpected happens before it finishes.

 

[Twice_Shy]

Well if someone in the minneapolis area has DDR3 workstations and ram theyre willing to throw out or is a dont wanter dumping for below market rates vs ebay i'd still happily take them off their hands and experiment with them. :^)

 

[UhClem]

Not workstations (but local):
9x HP ProLiant DL380 Gen9 Mixed Specifications W/ CPU RAM PSU Lot of 9

 

[Twice_Shy]

Not workstations (but local):
9x HP ProLiant DL380 Gen9 Mixed Specifications W/ CPU RAM PSU Lot of 9

That actually makes me wonder... I dont suppose there's any uber-deals in the server space, where I could just use a used server as an effective workstation for less than a used workstation would cost?

For that matter i'm already planning to use one or two workstations as servers already in the future as I don't think i'm excessively needful of things like cpu power.


Anyone in the future who maybe sees this thread in the next, I dunno, 6-9 months? And might have "usable ddr 3 workstations" to dump for below market or barely over scrap value/aka a don't wanter in minnesota can PM me privately. (this month excepted - i'm in a bind needing car repair first) Not try to take advantage of anyone - if you can get a better value elsewhere, by all means, sell it for money. Just trying to make use of stuff that might otherwise be thrown out as e-waste.

I only say that because i'm pretty sold by everyone on the idea of investing in DDR4 instead of DDR3 - even with cheaper ddr3 ram by the time I buy half a terabyte or more could be a year off anyways, I mostly need to get some machines up to 64gigs for the interim, the big upgrades come later.


I'll leave one standing request - anyone who can give insight into Dell's numbering systems if they could explain any of it to me on figuring out what other search terms to use on ebay would be appreciated. HP is pretty straightforward z420 z440 z640 z8 whatever, but Dell has everything from ultrasmall form factors that they call workstations to compact workstations to fullsize ones like the 5810 and i'm wondering what else is like the 5810 (heavy duty PSU's 700+ watts, dual x16 gpu slots, ram capacity 128gb plus) MIGHT also be in the running.

If I start buying HP I will probably stick with HP but if I found a lot buy of a bunch of appropriate Dells all at once for hundreds less I probably wouldn't pass it by.