Well, I've hit some partial success in coaxing my EVGA motherboard to turbo a bit more. It turns out that while the BIOS isn't setting up the MSRs quite right, it's leaving them unlocked (as is the SuperMicro for that matter), so you can happily modify them from Linux after boot.
The MSR that has the short and long term TDP limits is 0x610 (PKG_RAPL_POWER_LIMIT). The bit description is on page 57 of ``xeon-e5-1600-2600-vol-1-datasheet.pdf''. The first 32-bits are the long term TDP bits, the second are the short term bits. In those 32-bits, bits 0-14 are the Power Limit, bit 15 is the Power Limit Enable bit, bit 16 is the Clamp Mode bit (not sure what that does!), and 23-17 is the Control Time Window.
I was able to simply `rdmsr 0x610' on the SuperMicro and `wrmsr 0x610' that value on the EVGA and I was able to run mprime at full 3.0GHz turbo for increased time.
The default value of MSR 0x610 on the SuperMicro was 0x68450005a8398. This gives a short term TDP of 115W with a duration of 8.8s, with a long term TDP of 138W and a duration of 0.007812s according to the `power_gov' utility from Intel.
I played around a bit and found that a value of 0x685a0006484b0 shows as 150/180 with a short term control time of 281.6s. With that I can run mprime small FFTs at full turbo, with turbostat showing ~142W package power! It does drop back down to 115W in less than `281.6s', so not sure exactly what that value means, but it does run for longer than before, so it's doing something. =)
[update edit] (so as to not fill up the thread with these sorts of things!)
The max `over TDP' turbo time that I seem to be able to get is ~55s (from setting the update on turbostat to 1s and then running a small FFT mprime and counting them). So no running this thing at 142W all day, but still, at least I can get a real all-core turbo on this thing now, so I'm quite happy with that!
I did notice that in the xeon-e5 datasheet that it says that power limit 1 should be in a range of 250mS to 40S, so there probably is an internal register limit somewhere that's not letting it run much longer.
I did also run into some weirdness while testing these settings where my system would `hiccup' (freeze for a second, then come back, then freeze, etc.) under heavy load from mprime. Things would go back to normal once I stopped mprime. There was nothing in the logs either, and it was doing it without my mucking around with the MSRs! I tried all sorts of things (swapped out memory, tweaked voltages, reset the CMOS completely, etc.), but it seems that the culprit was that the EVGA BIOS defaults to Isochronous (ISOC) Mode enabled. Once I turned that off I haven't had the system hiccup again. The description from the BIOS is: ``Enabling the Isochronous Mode option reduces the credits available for memory traffic. For memory requests, this option reduces latency at the expense of throughput under heavy loads.''