A friend is currently in USA and he wants to build a new Workstation. Since the budget allows for it, I decided to recommend him to go for a true Workstation with a Xeon with ECC RAM. Additionally, he wants two GeForce 1080, which will be used for GPGPU purposes. The system will also be used as a gaming machine, but that is not reelevant.
This is a small list of the parts which I have pretty much nailed down.
The full list (With outdated prices) is HERE: Part List - Pastebin.com
PROCESSOR - LGA 2011-3
Intel Xeon E5-1620v4 - Broadwell-E / 4C-8T / 3.5-3.8 GHz / 40 PCIe Lanes - 306 U$D https://www.amazon.com/dp/B01GUAJQ08
Intel Xeon E5-1650v4 - Broadwell-E / 6C-12T / 3.6-4.0 GHz / 40 PCIe Lanes
Intel Core i7 6800K - Broadwell-E / 6C-12T / 3.4-3.6 GHz / 28 PCIe Lanes
Intel Core i7 6850K - Broadwell-E / 6C-12T / 3.6-3.8 GHz / 40 PCIe Lanes
The Xeon E5-1620v4 wins hands down due price and features. It is actually an excellent alternative to high end LGA 1151 (Core i7 6700K), and I would say anyone building such machine to consider going entry level LGA 2011-3 with that Xeon.
The Ci7 6800K is around 100 U$D more expensive, has 6 Cores and can be overclocked (Not important since he said he doesn't like to do that), but sacrifices ECC RAM support and PCIe Lanes.
The 6 Cores 40 Lanes Processors are around 600 U$D, too expensive.
HEATSINK - LGA 2011-3
Cooler Master Hyper 212 EVO - RR-212E-20PK-R2
Will be purchased locally since there is little difference between Amazon and some local vendors.
MOTHERBOARD - LGA 2011-3
AsRock X99 Taichi - ATX / X99 / 12 VRM / 2 BIOS / Realtek ALC1150 / Intel i218v + Intel 211-AT / 8x DDR4 DIMM / 3x PCIe 16x / 2x PCIe 1x / 2x M.2 Key M - 220 U$D https://www.amazon.com/dp/B01ITOLDQI
Gigabyte GA-X99P-SLI - ATX / X99 / 2 BIOS / Realtek ALC1150 / Intel NIC? / 8x DDR4 DIMM / 4x PCIe 16x / 2x PCIe 1x / 1x M.2 Key M / Thunderbolt 3
Supermicro X10SRA - ATX / C612 / 8 VRM / Realtek ALC1150 / Intel i210-AT / 8x DDR4 DIMM / 4x PCIe 16x / 2x PCIe 1x
Supermicro X10SRA-F - ATX / C612 / 8 VRM / Realtek ALC1150 / Intel i210-AT / 8x DDR4 DIMM / 4x PCIe 16x / 2x PCIe 1x / IPMI / VGA
The initial idea was using a Supermicro X10SRA or X10SRA-F, but after the launch of "X99 Refresh" Motherboards for Broadwell-E, I find these to be too dull and expensive. I still love Supermicro Motherboards and the fact that their Manuals always have a diagram with all the I/O topology so you know exactly what goes connected where, something which all the other manufacturers lack.
After searching a while, I found the AsRock X99 Taichi, which seems to be simple, solid, cheap, and has great reviews, superceding the Supermicros as my first choice. It has 3 PCIe 16x slots (16x/8x/8x) and two M.2 Key M PCIe 4x coming from the Processor, filling the 40 PCIe Lanes. The topology seems to be quite simple and devoid of lane switches, which is how I like Motherboards.
The disadvantage of Haswell-E Motherboards is that they may NOT work out of the box with a Broadwell-E Processor without flashing the Firmware, since they could be leftover parts with older Firmware. Since chances are that he travels back here without testing the parts (He will not be able to assemble a full system), this means that we're risking to have to procure here a 200-300 U$D Haswell-E Processor JUST for flashing (Unless there are alternatives like removing the Firmware chip and flashing it with a Flash programmer, but I would have to purchase that too). The AsRock X99 Taichi being a X99 Refresh works with Broadwell-E out of the box, that gives me a lot of peace of mind.
The advantage of the Supermicro X10SRA-F with the BMC is that it doubles as a basic integrated VGA, which may be useful if he ever decides to set up his system to do VGA Passthrough as I do with mine. Otherwise, he would have to set it up headless. IPMI itself doesn't serve any purpose for him.
I considered the Gigabyte GA-X99P-SLI because having an Alpine Ridge for ThunderBolt 3 makes it special, and is just slighty more expensive than the AsRock X99 Taichi. It also has four PCIe 16x slots, but the specifications doesn't really make it clear about the topology (It can't be 8x/16x/16x/8x, more likely 8x/16x/8x/8x?). The M.2 Key M Slot should also be PCIe 2.0 4x since all the Processor PCIe Lanes should go to the PCIe 16x slots.
Finally, there is the issue of DDR4 RAM type. I read elsewhere that AsRock support claims that some of their X99 Motherboards works with ECC RAM but only if its x8 wide (1Rx8 or 2Rx8), as it fails to POST with x4. The Supermicro X10SRA/X10SRA-F also seems to be picky since the Manual says that it supports non-ECC UDIMM ONLY if there is a Core i7 plugged in, with Xeons it requires RDIMM. I'm not sure it is just "not supported or validated to work" or it means that it will not POST at all.
Moreover, I recently make a Thread because there doesn't seem to be hard info about if ECC works as intended on X99 Motherboards (Or other consumer Chipsets in LGA 1150/1151) or Intel restrict it to C Series Chipsets only.
RAM MEMORY
As mentioned previously, this depends on both Processor (If Core i7 or Xeon) and how picky the Motherboard is with x4, x8, and whatever. So far, assuming I'l go for Xeon and that ECC works in X99 and thus the extra price is justified, I was going to use either DDR4 UDIMM ECC or RDIMM ECC.
Everything I found with ECC (Both UDIMM and RDIMM) seems to be either 2133 or 2400 MHz, there is nothing higher than that.
Since with 8 slots you can go up to 128 GiB using 8 * 16 GiB and these are available as ECC UDIMMs (Which could also be used for a LGA 1151 Xeon E3 V5) I don't know if going for the slighty slower and less compatible RDIMM is worth it. I doubt that he ever upgrades the RAM. 64 GiB seems already overkill, so depending price, 4 * 8 GiB could be enough, or 4 * 16 GiB could fit just for the sake of spending the budget.
I also found that RAM prices has been on the rise in the last few months and that a 16 GiB ECC UDIMM module could be found for 80 U$D some months ago, they're 100 U$D or more currently. That's sad.
Overally, I'm rather clueless about what RAM to buy and would take suggestions, since I can't decide between UDIMM or RDIMM. I prefer first party modules (Those that produces the DRAM and also assemble modules with it, like Samsung, Hynix, Micron), but they are hard to find and highly expensive...
DDR4 UDIMM with ECC
Kingston ValueRAM KVR24E17D8/16 - 16 GiB x 1 - 2400 MHz / 17-17-17 / 1.2V / 2Rx8 / 2Gx72 / (1Gx8)x18 - 110 U$D https://www.amazon.com/dp/B01FM3GBC0/
Crucial CT16G4WFD824A - 16 GiB x 1 - 2400 MHz / 17 / 1.2V / 2Rx8 / 2Gx72 - 100 U$D PROVANTAGE: Crucial Technology CT16G4WFD824A Crucial CT16G4WFD824A 16Gb Ddr4-2400 Ecc Udimm
Micron MTA18ASF2G72AZ-2G3 - 16 GiB x 1 - 2400 MHz / 17-17-17 / 1.2V / 2Rx8 / 2Gx72 / (1Gx8)x18 - 102 U$D MTA18ASF2G72AZ-2G3 Micron DDR4-2400 16GB/2Gx72 ECC CL17 Server Memory - DDR4 SDRAM Server 288 Pins - SuperBiiz.com
VIDEO CARDs
The plan is to use two GeForce GTX 1080. The question is... which one?
I'm assuming that all the GeForces are designed with gaming in mind, something that you at most do 12 hours daily, and in conditions that may not actually max out the GPU at all. GPGPU instead means 24/7 computing that will actually put them in Full Load, as if it was a continuous stress test. This means that I prefer beefed up models with strong VRMs and cooling. Founder Edition seems to be the worst of the lot in that regard.
There is also the issue that most 1080 were suffering from coil whine, but I can't fix that...
I used this as reference for PCBs: Comparison of custom GeForce GTX 1080 PCBs | VideoCardz.com
ZOTAC GeForce GTX 1080 AMP Edition - 1822 MHz - 8+3? - 570 U$D https://www.amazon.com/dp/B01GCAVRSU
MSI GEFORCE GTX 1080 ARMOR 8G OC - 1797 MHz - 10? - 599 U$D (Plus 20 U$D rebate) https://www.amazon.com/dp/B01GXOWUDQ
EVGA GeForce GTX 1080 CLASSIFIED GAMING ACX 3.0 08G-P4-6386-KR - 1860 MHz - 14+3 - 700 U$D Micro Center - Computers and Electronics
I will urge him to buy the Zotac NOW since they were 620 U$D yesterday, and is currently by far the most beefed up model for less than 600 U$D. Doubt than that can be beat. The EVGA Classified is supposed to be the best, but also the most expensive one.
The Zotac suffers from overheating, but that may be fixed: Zotac 1080 amp overheating fix • /r/nvidia
The other component that will be purchased in USA that I'm missing is the SSD, will edit what I find later. I don't know if the Intel 750 PCIe is still the top dog. I could also use a Samsung 960 PRO M.2, and maybe with a M.2-to-PCIe adapter to put a beefy heatsink on it.
Power Supply will be procured locally since you can't freely import those. It will probabily be a high end Seasonic model (Platinum 1000W or such). Case will be a Tower, probabily a rather sad one. I'm starting to like the idea of going for a 4U Rack as Tower replacement...
Well, that's all. I hope that other indecisive people can use all my research for reference, but I prefer at the moment if other people solves my indecisions first
This is a small list of the parts which I have pretty much nailed down.
The full list (With outdated prices) is HERE: Part List - Pastebin.com
PROCESSOR - LGA 2011-3
Intel Xeon E5-1620v4 - Broadwell-E / 4C-8T / 3.5-3.8 GHz / 40 PCIe Lanes - 306 U$D https://www.amazon.com/dp/B01GUAJQ08
Intel Xeon E5-1650v4 - Broadwell-E / 6C-12T / 3.6-4.0 GHz / 40 PCIe Lanes
Intel Core i7 6800K - Broadwell-E / 6C-12T / 3.4-3.6 GHz / 28 PCIe Lanes
Intel Core i7 6850K - Broadwell-E / 6C-12T / 3.6-3.8 GHz / 40 PCIe Lanes
The Xeon E5-1620v4 wins hands down due price and features. It is actually an excellent alternative to high end LGA 1151 (Core i7 6700K), and I would say anyone building such machine to consider going entry level LGA 2011-3 with that Xeon.
The Ci7 6800K is around 100 U$D more expensive, has 6 Cores and can be overclocked (Not important since he said he doesn't like to do that), but sacrifices ECC RAM support and PCIe Lanes.
The 6 Cores 40 Lanes Processors are around 600 U$D, too expensive.
HEATSINK - LGA 2011-3
Cooler Master Hyper 212 EVO - RR-212E-20PK-R2
Will be purchased locally since there is little difference between Amazon and some local vendors.
MOTHERBOARD - LGA 2011-3
AsRock X99 Taichi - ATX / X99 / 12 VRM / 2 BIOS / Realtek ALC1150 / Intel i218v + Intel 211-AT / 8x DDR4 DIMM / 3x PCIe 16x / 2x PCIe 1x / 2x M.2 Key M - 220 U$D https://www.amazon.com/dp/B01ITOLDQI
Gigabyte GA-X99P-SLI - ATX / X99 / 2 BIOS / Realtek ALC1150 / Intel NIC? / 8x DDR4 DIMM / 4x PCIe 16x / 2x PCIe 1x / 1x M.2 Key M / Thunderbolt 3
Supermicro X10SRA - ATX / C612 / 8 VRM / Realtek ALC1150 / Intel i210-AT / 8x DDR4 DIMM / 4x PCIe 16x / 2x PCIe 1x
Supermicro X10SRA-F - ATX / C612 / 8 VRM / Realtek ALC1150 / Intel i210-AT / 8x DDR4 DIMM / 4x PCIe 16x / 2x PCIe 1x / IPMI / VGA
The initial idea was using a Supermicro X10SRA or X10SRA-F, but after the launch of "X99 Refresh" Motherboards for Broadwell-E, I find these to be too dull and expensive. I still love Supermicro Motherboards and the fact that their Manuals always have a diagram with all the I/O topology so you know exactly what goes connected where, something which all the other manufacturers lack.
After searching a while, I found the AsRock X99 Taichi, which seems to be simple, solid, cheap, and has great reviews, superceding the Supermicros as my first choice. It has 3 PCIe 16x slots (16x/8x/8x) and two M.2 Key M PCIe 4x coming from the Processor, filling the 40 PCIe Lanes. The topology seems to be quite simple and devoid of lane switches, which is how I like Motherboards.
The disadvantage of Haswell-E Motherboards is that they may NOT work out of the box with a Broadwell-E Processor without flashing the Firmware, since they could be leftover parts with older Firmware. Since chances are that he travels back here without testing the parts (He will not be able to assemble a full system), this means that we're risking to have to procure here a 200-300 U$D Haswell-E Processor JUST for flashing (Unless there are alternatives like removing the Firmware chip and flashing it with a Flash programmer, but I would have to purchase that too). The AsRock X99 Taichi being a X99 Refresh works with Broadwell-E out of the box, that gives me a lot of peace of mind.
The advantage of the Supermicro X10SRA-F with the BMC is that it doubles as a basic integrated VGA, which may be useful if he ever decides to set up his system to do VGA Passthrough as I do with mine. Otherwise, he would have to set it up headless. IPMI itself doesn't serve any purpose for him.
I considered the Gigabyte GA-X99P-SLI because having an Alpine Ridge for ThunderBolt 3 makes it special, and is just slighty more expensive than the AsRock X99 Taichi. It also has four PCIe 16x slots, but the specifications doesn't really make it clear about the topology (It can't be 8x/16x/16x/8x, more likely 8x/16x/8x/8x?). The M.2 Key M Slot should also be PCIe 2.0 4x since all the Processor PCIe Lanes should go to the PCIe 16x slots.
Finally, there is the issue of DDR4 RAM type. I read elsewhere that AsRock support claims that some of their X99 Motherboards works with ECC RAM but only if its x8 wide (1Rx8 or 2Rx8), as it fails to POST with x4. The Supermicro X10SRA/X10SRA-F also seems to be picky since the Manual says that it supports non-ECC UDIMM ONLY if there is a Core i7 plugged in, with Xeons it requires RDIMM. I'm not sure it is just "not supported or validated to work" or it means that it will not POST at all.
Moreover, I recently make a Thread because there doesn't seem to be hard info about if ECC works as intended on X99 Motherboards (Or other consumer Chipsets in LGA 1150/1151) or Intel restrict it to C Series Chipsets only.
RAM MEMORY
As mentioned previously, this depends on both Processor (If Core i7 or Xeon) and how picky the Motherboard is with x4, x8, and whatever. So far, assuming I'l go for Xeon and that ECC works in X99 and thus the extra price is justified, I was going to use either DDR4 UDIMM ECC or RDIMM ECC.
Everything I found with ECC (Both UDIMM and RDIMM) seems to be either 2133 or 2400 MHz, there is nothing higher than that.
Since with 8 slots you can go up to 128 GiB using 8 * 16 GiB and these are available as ECC UDIMMs (Which could also be used for a LGA 1151 Xeon E3 V5) I don't know if going for the slighty slower and less compatible RDIMM is worth it. I doubt that he ever upgrades the RAM. 64 GiB seems already overkill, so depending price, 4 * 8 GiB could be enough, or 4 * 16 GiB could fit just for the sake of spending the budget.
I also found that RAM prices has been on the rise in the last few months and that a 16 GiB ECC UDIMM module could be found for 80 U$D some months ago, they're 100 U$D or more currently. That's sad.
Overally, I'm rather clueless about what RAM to buy and would take suggestions, since I can't decide between UDIMM or RDIMM. I prefer first party modules (Those that produces the DRAM and also assemble modules with it, like Samsung, Hynix, Micron), but they are hard to find and highly expensive...
DDR4 UDIMM with ECC
Kingston ValueRAM KVR24E17D8/16 - 16 GiB x 1 - 2400 MHz / 17-17-17 / 1.2V / 2Rx8 / 2Gx72 / (1Gx8)x18 - 110 U$D https://www.amazon.com/dp/B01FM3GBC0/
Crucial CT16G4WFD824A - 16 GiB x 1 - 2400 MHz / 17 / 1.2V / 2Rx8 / 2Gx72 - 100 U$D PROVANTAGE: Crucial Technology CT16G4WFD824A Crucial CT16G4WFD824A 16Gb Ddr4-2400 Ecc Udimm
Micron MTA18ASF2G72AZ-2G3 - 16 GiB x 1 - 2400 MHz / 17-17-17 / 1.2V / 2Rx8 / 2Gx72 / (1Gx8)x18 - 102 U$D MTA18ASF2G72AZ-2G3 Micron DDR4-2400 16GB/2Gx72 ECC CL17 Server Memory - DDR4 SDRAM Server 288 Pins - SuperBiiz.com
VIDEO CARDs
The plan is to use two GeForce GTX 1080. The question is... which one?
I'm assuming that all the GeForces are designed with gaming in mind, something that you at most do 12 hours daily, and in conditions that may not actually max out the GPU at all. GPGPU instead means 24/7 computing that will actually put them in Full Load, as if it was a continuous stress test. This means that I prefer beefed up models with strong VRMs and cooling. Founder Edition seems to be the worst of the lot in that regard.
There is also the issue that most 1080 were suffering from coil whine, but I can't fix that...
I used this as reference for PCBs: Comparison of custom GeForce GTX 1080 PCBs | VideoCardz.com
ZOTAC GeForce GTX 1080 AMP Edition - 1822 MHz - 8+3? - 570 U$D https://www.amazon.com/dp/B01GCAVRSU
MSI GEFORCE GTX 1080 ARMOR 8G OC - 1797 MHz - 10? - 599 U$D (Plus 20 U$D rebate) https://www.amazon.com/dp/B01GXOWUDQ
EVGA GeForce GTX 1080 CLASSIFIED GAMING ACX 3.0 08G-P4-6386-KR - 1860 MHz - 14+3 - 700 U$D Micro Center - Computers and Electronics
I will urge him to buy the Zotac NOW since they were 620 U$D yesterday, and is currently by far the most beefed up model for less than 600 U$D. Doubt than that can be beat. The EVGA Classified is supposed to be the best, but also the most expensive one.
The Zotac suffers from overheating, but that may be fixed: Zotac 1080 amp overheating fix • /r/nvidia
The other component that will be purchased in USA that I'm missing is the SSD, will edit what I find later. I don't know if the Intel 750 PCIe is still the top dog. I could also use a Samsung 960 PRO M.2, and maybe with a M.2-to-PCIe adapter to put a beefy heatsink on it.
Power Supply will be procured locally since you can't freely import those. It will probabily be a high end Seasonic model (Platinum 1000W or such). Case will be a Tower, probabily a rather sad one. I'm starting to like the idea of going for a 4U Rack as Tower replacement...
Well, that's all. I hope that other indecisive people can use all my research for reference, but I prefer at the moment if other people solves my indecisions first
