Since there's a constant stream of people landing on this post (wazzup Google...) a bit of a followup. Meanwhile I built a fourth 3647 system with same 8259CL CPU, but X11SPL-F mainboard. Switched to Micron DDR4-2666. Stayed with CX3. Idle power more like 60 watts.
The good: Inexpensive board, enough PCIe slots. My custom kernel compiles in 11 minutes. Supermicro produces updates for BIOS and IPMI, Asrock Rack while it works, produced nothing. Stable platform, couldn't crash it. Supermicro and Intel go way back and you can tell. Have to keep temperatures in view though, especially RAM and the CPU voltage regulator. I have two 140mm PWM fans blowing straight over PCIe cards, RAM, and under CPU heatsink towards the VRM. Also one more 140mm Noctua industrial with 2000rpm on the back of the tower case. No longer a job for quiet 1000rpm Noctuas I am afraid. But, PWM fans are ramped nicely by the IPMI. Machine is very quiet when doing nothing but will sound like a mid-sized colony of angry bees when doing something intensive. VRM peaks at around 75 deg C in European summer heat. Tested one Optane PMEM 100 on the board, also working. Two Intel i210 ethernet chips. Ya, don't even have to turn off half the acceleration features on these, so they don't mess up connectivity.
The bad: While only costing a few %, no x16 slot for GPU. VRM needed MCP2221a I2C reprogramming treatment to run a 255A TDC CPU, and BIOS also needed a TDP patch. I knew that before buying. Not sure what super-important secret code Intel is running in the C62x series of chipsets, but they're hot and wasteful space heaters.
The ugly: Nothing yet.
I also benchmarked a non-Pro ConnectX3 using Linux kernel namespace tricks to force traffic out of one physical port and back into the other. With an FDR (56 Gbps capable) cable and at four threads ntttcp reports 44-48 Gbps thoughput. If you read old articles from 2011 like these Mellanox forges switch-hitting ConnectX-3 adapters you will see they're talking about "below 3 watts" per port. Recent 2.5 Gbps cards struggle to achieve that efficiency, 12 years later. Huge heatsink and a fan for only 2.5 Gbps. For anything beyond 1 Gbps, if you can, skip twisted pair ethernet and go straight for SFP+ QSFP etc. and 25/40/56/100/200/400 Gbps.
The good: Inexpensive board, enough PCIe slots. My custom kernel compiles in 11 minutes. Supermicro produces updates for BIOS and IPMI, Asrock Rack while it works, produced nothing. Stable platform, couldn't crash it. Supermicro and Intel go way back and you can tell. Have to keep temperatures in view though, especially RAM and the CPU voltage regulator. I have two 140mm PWM fans blowing straight over PCIe cards, RAM, and under CPU heatsink towards the VRM. Also one more 140mm Noctua industrial with 2000rpm on the back of the tower case. No longer a job for quiet 1000rpm Noctuas I am afraid. But, PWM fans are ramped nicely by the IPMI. Machine is very quiet when doing nothing but will sound like a mid-sized colony of angry bees when doing something intensive. VRM peaks at around 75 deg C in European summer heat. Tested one Optane PMEM 100 on the board, also working. Two Intel i210 ethernet chips. Ya, don't even have to turn off half the acceleration features on these, so they don't mess up connectivity.
The bad: While only costing a few %, no x16 slot for GPU. VRM needed MCP2221a I2C reprogramming treatment to run a 255A TDC CPU, and BIOS also needed a TDP patch. I knew that before buying. Not sure what super-important secret code Intel is running in the C62x series of chipsets, but they're hot and wasteful space heaters.
The ugly: Nothing yet.
I also benchmarked a non-Pro ConnectX3 using Linux kernel namespace tricks to force traffic out of one physical port and back into the other. With an FDR (56 Gbps capable) cable and at four threads ntttcp reports 44-48 Gbps thoughput. If you read old articles from 2011 like these Mellanox forges switch-hitting ConnectX-3 adapters you will see they're talking about "below 3 watts" per port. Recent 2.5 Gbps cards struggle to achieve that efficiency, 12 years later. Huge heatsink and a fan for only 2.5 Gbps. For anything beyond 1 Gbps, if you can, skip twisted pair ethernet and go straight for SFP+ QSFP etc. and 25/40/56/100/200/400 Gbps.
