No love for Intel LGA3647?

[AveryFreeman]

Hey. I haven't heard a lot about this socket since they first came out with the weird PHI CPUs that were apparently a dud. Is there a reason they're so frequently passed over? They look super inexpensive on the used market now, basically starting at next to nothing, but the only real disadvantage I see in them is 6 instead of 8-channel memory support.

Anyone used them, and is there anything I'm missing?

 

[nexox]

I'm using them in my workstation and file/container/vm server, I rather like them, but the cheap Skylake CPUs aren't really that much better than Broadwell, and LGA2011-3 motherboards are somewhat cheaper than LGA3647. If you can find a good deal on a board that suits your needs then they're not bad, and one day Cascade Lake CPUs will get cheaper so there's a bit of an upgrade path, but Intel kind of fumbled the whole socket, particularly with delays getting to 10nm, so quite a few people go with AMD offerings from that time period instead.

 

[Micro]

I use Cascade Lake-X CPUs (8260Ls ES, all cores running @ 3.1 Ghz) in my workstation with 48 Gb memory running @ 2933mhz.
While there are a lot of faster things out there, for now they suffice just fine for 4K and 8K video work I do.
Besides, who doesn't like seeing 96 logical processors working

If you do decide to go with 3647s and go with ES or QS units, be very selective to get the closest to production units.
For instance, my 8260L ES are stepping 7, revision B1/L1, the same as production units except they can also clock down to 700 Mhz.

 

[AveryFreeman]

I use Cascade Lake-X CPUs (8260Ls ES, all cores running @ 3.1 Ghz) in my workstation with 48 Gb memory running @ 2933mhz.

Those look beasty. I'm noticing the single-CPU benchmarks are pretty similar to my laptop, though, it's got an Ultra 7 155H. Seems like the newer chips are really becoming a lot more efficient. Obviously, there's no SMT/NUMA options, so doesn't really help you any, but from a compute power per core basis, the 155H clocks about 2x more power per core (~2000 vs ~4000).

It'll be really interesting to see what Intel comes out with for multi-processor platforms, the new socket 1851 for this line just came out and will be available for purchase starting tomorrow. Apparently they're down to 3nm fabs now. Stunning.

 

[Stephan]

I am a bit of a resident 3647 "super" user on STH. 8259CL CPUs these days are a bit over 100 fiat out of China. Really no need to use QS or ES anymore. My servers have six 32 GB DDR4 2666 DIMMs and most have two 512 GB Optane PMEM100. I like that you can still patch the BIOS to eliminate e.g. TDC limit on Supermicro boards. Almost nobody dares to reprogram the VRM controller for TDC 255A so boards and CPUs stay nicely cheap.

Never used a 3647 Phi. Yesterday I modded an RTX A4000 with a 2 slot, 1 fan after market cooler, 2 slot bezel from A4500, 8-pin PCIe power socket, and a Noctua 92mm fan. Dropped GPU and VRAM temps to mid 70 deg C under load. Anyhow that's what I use for massively parallel computation tasks.

 

[zachj]

I’ve got a Xeon Gold 6268CL with 4x512gb optane pmem 100 DIMMs and 2x128GB ddr4 2666 DIMMs. It’s got a bunch of 3.84tb U.2 storage in it too.

I think it’s a fabulous rig.

AMD Epyc Rome and Milan are better and more modern platforms (faster memory, more memory channels, PCIe gen4, twice as many PCIe lanes) but the motherboards for SP4 socket are outrageously expensive and there’s no arguing that you can get a shitload more memory for the money thanks to optane.

If you’re looking to get a workstation today and you don’t want the latest and greatest I would say you really can’t go wrong with the Xeon unless there’s something SP4 Epyc gives you that you outright need.

 

[AveryFreeman]

I’ve got a Xeon Gold 6268CL with 4x512gb optane pmem 100 DIMMs and 2x128GB ddr4 2666 DIMMs. It’s got a bunch of 3.84tb U.2 storage in it too.

I think it’s a fabulous rig.

AMD Epyc Rome and Milan are better and more modern platforms (faster memory, more memory channels, PCIe gen4, twice as many PCIe lanes) but the motherboards for SP4 socket are outrageously expensive and there’s no arguing that you can get a shitload more memory for the money thanks to optane.

If you’re looking to get a workstation today and you don’t want the latest and greatest I would say you really can’t go wrong with the Xeon unless there’s something SP4 Epyc gives you that you outright need.

Yeah, that's definitely along the lines of what I used to go for when I bought my broadwell E5s. Now that I think about it a bit longer, I am with you 100% on why bother buying something if I don't need it. I like looking at different stuff, but realistically haven't needed more compute power than an I7-8700K machine I scaled down to when I moved. I will probably get along fine with a single socket machine, and will likely lean more on the side of power efficiency, rather than scalability. I am really impressed with the numbers from new performance + efficiency core CPUs coming out, for example, or even the "prosumer" Epyc 9004 line.

But that stuff is still new and expensive, so I was shocked af when I saw there are e.g. Z6 G4s starting at $200, random 3647 motherboards at $90, and 3647 CPUs starting under $10, though, and thought maybe there was something "wrong" with the platform (the inspiration for this thread).

So IMO, even if 6151 CPUs have 1/2 the per-core performance per watt of an Ultra 7 265, if they cost 1/5th to 1/3rd of the price, would still be a great value, and cheaper to get into (irrespective of having additional PCIe lanes, etc.). If running a single-socket, what's your idle power consumption at the wall?

Edit: correction, I think I was analyzing the 8171M, not the 6151

 

[nexox]

Idle power isn't great, not better than Broadwell E5s so far as I can tell, and just the IPMI on my Supermicro X11 boards consumes a decent fraction of the idle power of an entire desktop i7 system. I don't have data recorded for the bare minimum hardware, probably around 50W (but my boot NVMe doesn't support ASPM, so that could maybe go lower.) After I added some SAS hardware and a fast NIC the lowest I got with a single 4114 was 110W idle, I switched to a 5119T, which had slightly better idle power, but at the same time I also added more drives and now I'm up to around 135W idle, plus ~25W more when I re-install my FusionIOs.

 

[Styp]

I use an LGA3647 Xeon Gold 6212u with dual 4080 super as my ML engineering and research rig daily.

I run 192 GB RAM and 2 GPUs—stable as fu**. The machine ran for 2 weeks while I was on vacation doing heavy ML training, and when I came back, it was still running. I really like the platform. I am starting to feel an itch to upgrade, but more for expandability reasons (I want 4 u.2 Raid 0 discs for 20TB datasets).

 

[Stephan]

thought maybe there was something "wrong" with the platform

From memory...

I did a Linux kernel compile benchmark with my trash 1 CPU Cascade Lake 8259CL system recently and found that on an updated Arch Linux you lose around 10% performance going from mitigations=off to turning everything back on as the kernel sees fit. That number might be alot worse with old toolchains, which do not emit machine instructions geared towards performance mitigation. In this regard "alot" is wrong with the CPU. But at the current price point, even a 400 HP car only delivering 360 HP finds a buyer. Newer CPUs have other defects, which do not apply to Cascade Lakes surprisingly.

The Lewisburg chipset is a space heater, easy 5-10 watts gone just with that. Might be the quad 10 Gbps 722 NICs in this, sipping power for nothing even on quad 1 Gbps boards, or disabled power saving, or something else. Should be cooled separately, just like DDR4 DIMMs or SAS cards or your Mellanox NIC.

An "empty" 3647 system will idle with proper settings at around 50-60 watts. That is alot more than some 10-20 watts people like Wolfgang of Wolfgang's YT channel get with low power desktop CPUs or desktop Xeons.

Linux kernel and rasdaemon instrumentation that tells you if something is wrong is very stable and nicely chatty compared to desktop CPUs. I like hardware with diagnostics so I can make an educated guess and be right the first time when swapping a component.

By now most functional bugs are known and worked around in software or microcode. This must have been thousands of engineer hours while this chip was in use at hyperscalers or supercomputing clusters.

I trust the Intel 14nm++ process to be more reliable than all their newer nodes.

Alot of PCIe lanes and slots. I always run out with desktop chips.

No stupid license keys to enable silicon accelerators. Alot more engineering and whole-system support like in IBM System z has to happen, before such license schemes start to make sense for the customer. And who uses these accelerators really? General purpose x64 code will also run on EPYC and not be locked-in to some Intel Xeon generation. And if you really, really need something for massively parallel compute, go on ebay and pick up a GPU for 200 bucks.

 

[zachj]

I’ve got an epyc Milan desktop right next to my cascade lake Xeon. Both are rock solid stable.

Both idle at over 100W but both are slammed full of memory and storage and 10gb copper NICs and GPUs (my Epyc has two teslas in it).

I agree with your summary that even at half the IPC and twice (or more) power consumption compared to newer desktop platforms, the per dollar performance is hard to beat.

Go to eBay and look at the price of 32gb or 64gb RDIMMs. Look at the price of standalone 10gb NICs. Look at the price of PCIe bifurcation cards (or, worse, cards with a PCIe switch on them). By the time you get done kitting out a desktop system you’re in it for hundreds of dollars more. And the power consumption of desktop platforms goes to hell when you start adding all the expansion cards.

If all you need is an 8700k then that’s just fine. For daily driving it’ll be snappier than a 3647 Xeon because the clock speeds are higher. But if you’re looking for more _____ the 3647 platform is pretty damn appealing.

Epyc Rome processors are getting cheap but the motherboards aren’t. For the same money I’d definitely prefer Epyc Rome over a 3647 Xeon—but it ain’t the same money. Not even close.

 

[UhClem]

...
If all you need is an 8700k then that’s just fine. For daily driving it’ll be snappier than a 3647 Xeon ...

Don't overlook the Xeon W-32xx. E.g., a W-3235 QS is ~$100; and the W's have 64 PCIe lanes [on the right mobo's]
[ Passmark: CcTt single multi
8700K 6c12t 2726 13624
W-3235 12c24t 2642 25578 ]

 

[RolloZ170]

E.g., a W-3235 QS is ~$100; and the W's have 64 PCIe lanes

extra 16 lanes used only in rare set of motherboards.

 

[zachj]

I don’t see a cost or power argument to choose the Xeon W over the 3647 xeons.

The only argument in their favor might be clockspeed but at equal core counts I don’t see enough difference there to make it compelling.

 

[drdepasquale]

The LGA 3647 platform was introduced around the same time as Socket SP3 for EPYC. EPYC provides a stronger socket, more memory bandwidth, more PCIe lanes, and more cores for a much lower price.

 

[chinesestunna]

The LGA 3647 platform was introduced around the same time as Socket SP3 for EPYC. EPYC provides a stronger socket, more memory bandwidth, more PCIe lanes, and more cores for a much lower price.

I believe this was true at launch and when new, however it appears most large players still bought xeons and current used market is flooded by them hence price/performance may actually make these 3647 platforms more attractive?

 

[drdepasquale]

I believe this was true at launch and when new, however it appears most large players still bought xeons and current used market is flooded by them hence price/performance may actually make these 3647 platforms more attractive?

It took some time, but the prices have improved on the used market

 

[Styp]

My reason for 3647 over Epyc SP3 back when I bought it:

- Asus C621 WS Workstation board, with Sleep, Audio, and usable bootup times
- AVX-512 -> and the Intel-optimized software stack
- stability and peace of mind - Intel is still, in eco-system terms, the more complete platform solution

Currently, I could imagine buying into a Threadripper or a Ryzen 9000 with all the AMD benefits, but back then, Intel had just a more mature offering. I am not an Intel fanboy, but since the first couple of Ryzen generations had some weird issues forcing me to opt for an Intel solution...

 

[Stephan]

I have Chips and Cheese in my RSS Feeds because of their in-depth reviews of functional blocks of new and older CPUs. This popped up:

He says with Arrow Lake systems, you throw in a GPU from NVIDIA or AMD and then it will no longer boot. Or funny Xeon stuff Patrick found out. No such problems in 3647 land, I just added a RTX A4000 to one of my server boards for experiments, next to a SAS2008 controller, next to a ConnectX3.

 

[NablaSquaredG]

I do use a lot of LGA3647, but my main issue with them is the lack of lanes in single socket.

ASRock Rack EPC621D8A gives you 44 lanes (2x16 + 8 + 4) which is about the maximum you can get.. And this is not a lot.
Primary NIC with x16, NVMe with x16, HBA with x4 (or x8) and secondary NIC with x8 (or x4) and you're out of lanes.

(Yes, ASRock Rack SHOULD have made the x4 slot an x8 like the X11SPI, but they didn't for an unknown reason - the platform supports it an they're wasting 4 precious lanes!)