And so it begins... First AMD Ryzen AM4 server motherboard.
- Thread starter McKajVah
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Ryzen itself is limited on PCIe lanes; the chips have 24 lanes themselves, of which 16x usually go to the graphics card, 4x for NVME and 4x to the chipset. Other PCIe lanes vary depending on the chipset (IIRC X470 provides an additional 8x PCIe 2.0) but ASRock helpfully provide a block diagram on page 13 of the manual (PDF warning or you can see an image of it here); you can see they've split the 16x into two 8x and the 4x goes into a slot rather than into NVME.
The remaining 8 lanes from the chipset all look to be used which makes me somewhat sceptical of a 10GbE version being possible - TTBOMK there's no 10Gb MAC on the Ryzen die and you need PCIe 2.0 2x to max out a single 10GbE connection, so one of the other peripherals would need to be chopped... and at that point why not just tell people to use a card in one of the PCIe slots?
Certainly going to be interesting to see how this behaves with Ryzen 3000 chips and potentially PCIe 4.0 - especially if 32GB ECC UDIMMs work with it.
The remaining 8 lanes from the chipset all look to be used which makes me somewhat sceptical of a 10GbE version being possible - TTBOMK there's no 10Gb MAC on the Ryzen die and you need PCIe 2.0 2x to max out a single 10GbE connection, so one of the other peripherals would need to be chopped... and at that point why not just tell people to use a card in one of the PCIe slots?
Certainly going to be interesting to see how this behaves with Ryzen 3000 chips and potentially PCIe 4.0 - especially if 32GB ECC UDIMMs work with it.
The lack of PCIe lanes is a constraint of a consumer platform vs an enterprise one. Intel's enthusiast platform (Z390) only appears to have "more" lanes, but if we take a closer look they're not real PCIe lanes.
AMD X470 with Zen/Zen+
AMD X470 with Zen/Zen+
- CPU: PCIe 3.0 x16 or PCIe 3.0 x8/x8 or PCIe 3.0 x8/x4/x4 or PCIe 3.0 x4/x4/x4/x4
- CPU: PCIe 3.0 x4
- CPU -> X470 chipset: PCIe 3.0 x4 interconnect @ 3.94 GB/s
- X470: Configurable 8x PCIe 2.0 lanes
- CPU: PCIe 3.0 x16 or PCIe 3.0 x8/x8
- CPU -> Z390 chipset: 4 lane DMI 3.0 @ 3.93 GB/s
- Z390: Configurable 24x PCIe 3.0 lanes
Some notes:
The CPU Zen dies has 32 PCIe Lanes, not 24. AM4 doesn't expose 8 lanes from the SERDES Controller (For PCIe, SATA or 10G NIC), and always wastes 4 for the Chipset, which is the main reason why I consider Embedded Ryzen/EPYC to be superior (Note that Embedded Ryzen is a Raven Ridge APU and it is missing 8 PCIe Lanes from the pure PCIe Controller, but those are missing in AM4 too. Is 32 or 24 vs 24 or 16 on CPU and APU, respectively). Also, the embedded Zens DOES expose the 10G NICs, which takes two lanes each.
Zen also can bifurcate each 16 lanes controller to up to 16 1x, but I'm not sure if AM4 Ryzens can do that. You could theorically have 8x/4x/1x/1x/1x/1x if you want to.
Intel consumer CPUs can do 8x/4x/4x. Note that for PCI Passthrough scenarios they require hacks since the Processor PCIe Controller does NOT support PCIe ACS, whereas AMD supports it.
While Intel has much better Chipset connectivity, a single high end NVMe drive can saturate DMI. Ryzen can have it hooked directly to it, albeit chances are that you are using the Chipset in some way (Moving NVMe SSD data to the Chipset NIC?). Actually, a full 32 lanes Embedded EPYC should be able to pull 16x or 8x/8x, 2 10G NICs, 2 NVMe and 4 SATAs with no multiplexing. A serious I/O beast.
The CPU Zen dies has 32 PCIe Lanes, not 24. AM4 doesn't expose 8 lanes from the SERDES Controller (For PCIe, SATA or 10G NIC), and always wastes 4 for the Chipset, which is the main reason why I consider Embedded Ryzen/EPYC to be superior (Note that Embedded Ryzen is a Raven Ridge APU and it is missing 8 PCIe Lanes from the pure PCIe Controller, but those are missing in AM4 too. Is 32 or 24 vs 24 or 16 on CPU and APU, respectively). Also, the embedded Zens DOES expose the 10G NICs, which takes two lanes each.
Zen also can bifurcate each 16 lanes controller to up to 16 1x, but I'm not sure if AM4 Ryzens can do that. You could theorically have 8x/4x/1x/1x/1x/1x if you want to.
Intel consumer CPUs can do 8x/4x/4x. Note that for PCI Passthrough scenarios they require hacks since the Processor PCIe Controller does NOT support PCIe ACS, whereas AMD supports it.
While Intel has much better Chipset connectivity, a single high end NVMe drive can saturate DMI. Ryzen can have it hooked directly to it, albeit chances are that you are using the Chipset in some way (Moving NVMe SSD data to the Chipset NIC?). Actually, a full 32 lanes Embedded EPYC should be able to pull 16x or 8x/8x, 2 10G NICs, 2 NVMe and 4 SATAs with no multiplexing. A serious I/O beast.
I think you're all arguing apples and oranges.
This is targeted at replacing aging entry level compute/storage servers. Read SMB (nowhere near enterprise).
EPYC/Ryzen embedded is embedded, which is a whole other ballgame for use cases and taget market. The price tags alone will tell that story.
The Intel E-2100 series is actually a closer compare, but, likewise, still far off. E-2100 simply doesn't offer the raw compute power (read core count) and flexibility of these devices.
Hint:
There is a reason that savy buyers arent running out for those E-2100s: the value vs used last gen equipment just isn't there.
The Ryzen (socketed) CPUs on the other hand offer so much more value per $ NEW it's tantalizing! Heck, I can swap this board into my 3U Supermicro 836, swap over my HBA and 10Gb NIC, and not sweat pennies or ever second guess performance. Getting even better compute for less than half the power usage as a bonus.
This is targeted at replacing aging entry level compute/storage servers. Read SMB (nowhere near enterprise).
EPYC/Ryzen embedded is embedded, which is a whole other ballgame for use cases and taget market. The price tags alone will tell that story.
The Intel E-2100 series is actually a closer compare, but, likewise, still far off. E-2100 simply doesn't offer the raw compute power (read core count) and flexibility of these devices.
Hint:
There is a reason that savy buyers arent running out for those E-2100s: the value vs used last gen equipment just isn't there.
The Ryzen (socketed) CPUs on the other hand offer so much more value per $ NEW it's tantalizing! Heck, I can swap this board into my 3U Supermicro 836, swap over my HBA and 10Gb NIC, and not sweat pennies or ever second guess performance. Getting even better compute for less than half the power usage as a bonus.
I think you're right there @Mam89. This platform would compete with the Xeon E-21xx. Quite interesting for homelabbing or SMB.
Here's the French hardware site link referring to a variant by this summer that would have 10 GbE:
Google Translate
They have a review up as well, but it's paywalled:
Google Translate
You guys nailed it. These NEW components are nicer in terms of price / performance / efficiency / cost vs last gen used stuff of similar tier.
FYI:
DDR4 ECC Unbuffered UDIMM / EUDIMM Crucial - 16 GB (1 x 16 GB) DDR4-2666 Memory (CT16G4WFD8266) - PCPartPicker now at $117. The price has been dropping nicely, I'm waiting until they're sub $99 myself.
Re: AMD Embedded :
Sapphire Unveils Motherboard With AMD Ryzen Embedded V1000 APU
UDOO BOLT - UDOO
Around $350 - $450 for a a powerful small form factor (5" x 5")
FYI:
DDR4 ECC Unbuffered UDIMM / EUDIMM Crucial - 16 GB (1 x 16 GB) DDR4-2666 Memory (CT16G4WFD8266) - PCPartPicker now at $117. The price has been dropping nicely, I'm waiting until they're sub $99 myself.
Re: AMD Embedded :
Sapphire Unveils Motherboard With AMD Ryzen Embedded V1000 APU
UDOO BOLT - UDOO
Around $350 - $450 for a a powerful small form factor (5" x 5")
Shame there are no 32GB EUDIMM available yet. If 4 x DR EUDIMMs are used the memory speed drops to 1866 MHz. The motherboard can do 2400 MHz with 2 x DR EUDIMM.
No word on whether this board and a current Ryzen CPU will work with 32GB ECC UDIMMs though, unofficially. Officially, it's limited to 4x16GB EUDIMMs.
The spec sheet says it supports up to 64GB EUDIMM. Not sure if that's going to the limited by the Ryzen/Ryzen+ IMC or not. Officially Ryzen/Ryzen+ supports up to 64GB. I haven't been able to find any information on configuration. It would be nice if it supports 2 x 32GB DR.
A minor point but, as I've been eyeing this board as a potential replacement for my current hardware when the Zen2 chips come out; no new BIOS releases or updates on the memory compatibility yet, but they are at least confirming it'll work with "7mm Ryzen" chips; when last I looked they were only showing compatibility with the Pinnacle Ridge (Zen+) chips.
Then:
AMD Ryzen™ (Pinnacle Ridge) 2nd Generation Series Processors
Now:
AMD AM4 Socket Ryzen Series CPUs (Raven Ridge and Pinnacle Ridge) and Ryzen 7nm CPUs
They're available in the UK now but I'm likely going to be waiting until boards with a compatible BIOS are in the channel, and it'd be nice to see an update on memory compatibility as well. Sadly the Zen2 IMC still only seems to be rated for 16GB DIMMs so still limited to 64GB max memory officially. Eagerly awaiting benches and especially TDP numbers of the new chips as well, the 3700X is looking like an ideal sweet-spot for me.
Then:
AMD Ryzen™ (Pinnacle Ridge) 2nd Generation Series Processors
Now:
AMD AM4 Socket Ryzen Series CPUs (Raven Ridge and Pinnacle Ridge) and Ryzen 7nm CPUs
They're available in the UK now but I'm likely going to be waiting until boards with a compatible BIOS are in the channel, and it'd be nice to see an update on memory compatibility as well. Sadly the Zen2 IMC still only seems to be rated for 16GB DIMMs so still limited to 64GB max memory officially. Eagerly awaiting benches and especially TDP numbers of the new chips as well, the 3700X is looking like an ideal sweet-spot for me.
Same here on Zen2 @EffrafaxOfWug.
I’ve switched all my workstations over to Zen/Zen+ and it’s been great so far. My only Intel PC left is my Dell XPS 9350, and with all the previous vulnerabilities + the new vulnerabilities out this month recommending disabling HT, it’s meh for the blue team.
My main gripe about Zen/Zen+ is lack of downstream PCIe connectivity. Though as I highlighted before that in terms of raw CPU-Chipset interconnect bandwidth it’s basically the same for Intel vs AMD consumer platforms, it’s somewhat more convenient for the chipset to act as a PCIe switch for “more” lanes. Looks like the X570 chipset basically fixes this issue.
I can’t be sure, but based on observation it looks like for the Ryzen 3000 AM4 is taking much more seriously by the motherboard manufacturers. I wonder what sort of effort it would take to update this motherboard to the X570, beyond the chipset being in-house AMD, and needing redrivers for the PCIe_2 and onward slots?
I’ve switched all my workstations over to Zen/Zen+ and it’s been great so far. My only Intel PC left is my Dell XPS 9350, and with all the previous vulnerabilities + the new vulnerabilities out this month recommending disabling HT, it’s meh for the blue team.
My main gripe about Zen/Zen+ is lack of downstream PCIe connectivity. Though as I highlighted before that in terms of raw CPU-Chipset interconnect bandwidth it’s basically the same for Intel vs AMD consumer platforms, it’s somewhat more convenient for the chipset to act as a PCIe switch for “more” lanes. Looks like the X570 chipset basically fixes this issue.
I can’t be sure, but based on observation it looks like for the Ryzen 3000 AM4 is taking much more seriously by the motherboard manufacturers. I wonder what sort of effort it would take to update this motherboard to the X570, beyond the chipset being in-house AMD, and needing redrivers for the PCIe_2 and onward slots?
The problem with the mainstream platforms has been the low pin count on the socket both for AMD and Intel. They need to up their game in that department but I'm sure it costs too much. People criticize the HEDT platform as a cash grab but it was necessary to provide more lanes and memory channels. If it was just consumer variant of the server platform so be it.
Seems like we are stuck with this setup of some lanes coming directly from the CPU and some via the chipset for the forseeable future. There has been no mention of any other layout for mainstream like just having a beefy link between CPU and chipset and having good PCIe switching capability in the chipset where everything connects including that GTX 2080 Ti.
Seems like we are stuck with this setup of some lanes coming directly from the CPU and some via the chipset for the forseeable future. There has been no mention of any other layout for mainstream like just having a beefy link between CPU and chipset and having good PCIe switching capability in the chipset where everything connects including that GTX 2080 Ti.
As it is, I can get away with the connectivity restraints of the Ryzen desktop platform - I only need PCIe slots for an HBA and maybe a 10Gb card. IO is expensive both in terms of power and additional complexity so I don't begrudge the additional cost of the proper workstation/server platforms. The vast majority of Ryzen purchasers will at most use one PCIe card, an NVME drive and maybe a SATA drive or two.
My only disappoint is has the IMC, given that the Epyc can comfortably handle 128GB DIMMs but I guess that's more a function of registered ECC than anything else.
If there'd been something like an Epyc 3000 in a mATX format - lots of PCIe capability and support for much more memory being the huge advantages there - I'd have leapt on that in a shot. But as it is, no-one seems to have anything like that in their roadmaps and I'm in the market for some more high-performance cores for not a huge amount of money; this board in conjunction with an 8+ core Ryzen2/Ryzen+ fits the bill perfectly even with the constraints. HEDT boards just don't have the IPMI or anything like it.
My only disappoint is has the IMC, given that the Epyc can comfortably handle 128GB DIMMs but I guess that's more a function of registered ECC than anything else.
If there'd been something like an Epyc 3000 in a mATX format - lots of PCIe capability and support for much more memory being the huge advantages there - I'd have leapt on that in a shot. But as it is, no-one seems to have anything like that in their roadmaps and I'm in the market for some more high-performance cores for not a huge amount of money; this board in conjunction with an 8+ core Ryzen2/Ryzen+ fits the bill perfectly even with the constraints. HEDT boards just don't have the IPMI or anything like it.
You want exactly what I want, I have ranted out loud to see if I can generate interest in such a product.
EPYC Embedded does a far better use of the Zen SoCs that what AM4 + Chipset does. Supermicro has Motherboards that use it almost the way that I want, but they're all mITX, so at most one slot, and they don't have consumer/workstation features that we're used to like Azalia audio. There is nothing stopping a Motherboard manufacturer to try to make a Ryzen Embedded or EPYC Embedded prosumer facing product.
Zen SoC has a single weakness: It has only 4 USB Ports. Even Supermicro EPYC Embedded Motherboards have some USB Hubs integrated to fanout a single USB 3.0 to two USB 2.0, it is really that starved. I hope that now that they have a dedicated I/O die, AMD can put at least eight, as that would make it far more viable.
@zir_blazer might be worth looking at Asrock X399m? Has 3 full NVME M2 slots that support NVME RAID and 3 full x16 slots in M-ATX. Reasonably cheap, supports ECC and HCC Threadripper - only downside is it's limited to 64 GB of RAM total (though this might go up to 128 with a new BIOS when 32gb Udimms become more common)
New 10Gb version...
ASRock Rack > X470D4U2-2T
And a Threadripper monster...
ASRock Rack > X399D8A-2T
ASRock Rack > X470D4U2-2T
And a Threadripper monster...
ASRock Rack > X399D8A-2T
The 10GbE version of the X470D4U2-2T has been kicking around for a while, it was shown in their 2019 catalogue which was published a few weeks back.
That threadripper board... wasn't. That's one helluva board, sadly doesn't scratch my itch for a home server board though.
That threadripper board... wasn't. That's one helluva board, sadly doesn't scratch my itch for a home server board though.
ThreadRipper based platforms should balloon too much the price, whereas AMD price for EPYC Embedded series seems to be comparable the consumer AM4 parts (The 8C/16T 3251 was like, 330 U$D, same than R7 1700 on release) but with more SoC features, that is why I find them more attractive. Besides, TR on mATX has pretty much the same issue than single die EPYC Embedded on mITX: Too big for that size to use all its I/O. The X99D8A-2T posted before suits better my tastes.
Sadly, first world tastes, third world wallet...