Because I see 18 cores slain by 4 cores. That is unacceptable. Xeon should be the King of power because servers are badass. Yet i've seen 18 core Xeon's get beaten up by the 7700k. DAMN THEM.
There are a few reasons for this but it comes down to single thread performance which is dictated by IPC (Instructions Per Clock) and the actual clock speed of the chips.
You will find that the Core i7 7700K is Kaby-Lake which is one of the fastest architectures Intel has. Current XEON's only use Skylake-EP architecture which not only has lower IPC than Kaby-Lake but it also uses a different core to core interconnect which is a grid structure while previous XEON's used a ring structure which was more preformative in certain tasks.
Combined with this XEON's often have much lower clock speeds than consumer parts. The 7700K has a clock speed of 4.2GHz which rises to 4.5GHz under single core load scenarios. Most XEON's don't touch 4GHz.
For example the 18 core XEON you're probably thinking of is the one in the iMac Pro for example which has a 2.3GHz base clock and a single core boost clock of 4.5GHz.
Clearly the 7700K will win against this CPU in tasks that use between 1 and 4 threads because it has twice the base clock speed and is a newer architecture. But it cannot compete with the 18 core CPU once you throw a multithreaded load at it such as video transcoding, audio transcoding, scientific work, webserver with lots of connections, virtual machine hosting, sound production with lots of instruments etc
That's where those 18 slower cores really come out of their shell.
To put this another way. Think of the 7700K like a motorcycle. Really fast with a single individual on it, you could beat a four door saloon car very easily in a race. But that saloon car could move 4 people to the end of the finish line faster than a motorcycle could because it has a wider capacity for people. You give up some acceleration to gain capacity and that's what the 18 Core XEON is great at, wide workloads that can make use of all 18 cores.
Now to answer why doesn't Intel sell XEON's with higher clock speeds? Well they have to fit the CPU within what we call a power envelope. The 7700K has a 91 Watt thermal design power or TDP. So it must stay within 91 Watts of heat dissipation so that it can be manageably cooled. Since it's only 4 cores those cores can run at quite a high base frequency of 4.2GHz and sitll stay within that TDP.
The 18 core XEON however while having a 150 Watt TDP also has 18 cores which is 14 more than the Core i7. So Intel cannot run them as fast because it would go over the TDP and be unmanageable to cool with air cooling (and even some water coolers would struggle over 250 watts etc).
The last thing you may ask is, why has Intel chosen to launch new XEON's using Skylake-EP when the mainstream consumer parts are now on Coffee Lake which is 2 generations ahead and thus higher in IPC. Only Intel can really answer this, but I think it's a combination of factors.
1. The XEON market must have stable products with the fewest erreta (errors) so it's better for Intel to release them to consumers first, find the bugs and fix them before they release the XEON versions. Basically this is Intel hoping that the workstation and server markets will get a more stable product by having consumers test the architectures first.
2. Intel used to use a tick-tock strategy to chip fabrication where you would release a new design architecture then a year later release a lower fabrication node to make that chip thus lowering power allowing for higher clock speeds. This strategy has only recently started to unravel in the past 4 years.
The reason this impacts XEON's is because Intel likes to have newer nodes tested on low power parts first. Think notebooks, phones, small desktops like the NUC. The reason for this is that these devices use smaller processor dies so they allow more chips to be produced per wafer. The wafer is the thing in the factory that the chips are actually created on.
So when you have a new node that is untested you want to be able to use a small die so that if there is a problem you lose the fewest amount of chips from each wafer produced. If for example you have a small die you could get 200 chips from a wafer if 25% of those are defective you only lost 50 chips and could still sell 150. But if your die is quite large and only allows 80 chips to be produced per wafer and you lose 25% of those that's a loss of 20 chips and you now only have 60 to sell.
Generally the larger the die is, the more defects you're going to have due to the dies having more surface area on the wafer. This is also why larger chips often cost more, they take up more room on the wafer, the wafer price is static so the smaller the chip, the less it costs you to produce.
So for Intel producing the smallest dies on newer process nodes first is economically the right idea and that may be why we're seeing the XEON's use 2 architectures behind because they were intending to release newer architectures on newer nodes but due to the delays in 14nm and now 10nm we're seeing delays in getting those bigger chips on the latest nodes.
For example Kaby Lake and even Coffee Lake were both meant to be 10nm parts but instead are 14nm. It's conceivable that Intel would like to launch Coffee Lake-EP on the XEON side on 10nm eventually.
This post is already pretty long so I'll stop here, I hope this gives you a good idea of why things are like they are. If you look at the AMD side they're doing it slightly differently using one single die across their entire product stack from consumer desktop to enthusiast desktop to server. Using multiple small dies together takes out the wafer risk I mentioned above and allows them to keep their entire product stack on the same architecture.