my mind is blown... i have no idea what's going on anymore... short story is, as of 30 minute ago, everything, and I mean *everything* seems to be working as expected now.
here's the sequence of events up to this...
after last night's attempt to test the SATA controller coming up with weird x2 and x1 links, this morning i thought i would do some more testing with a different card. i had a Mellanox dual port ConnectX-2 card, which is PCI-E 2.0 x8 laying around so I started testing it in the X9SCM-F:
slot 7: card is present in BIOS, links at 5GT/s @ x8
slot 6: card is present in BIOS, links at 5GT/s @ x8
slot 5: card is present in BIOS, links at 5GT/s @ x4
slot 4: card is present in BIOS, links at 5GT/s @ x4
so, it seems everything works correctly! i then also tried out a Broadcom 1Gbps NIC card that is PCI-E 1.0 x1. A strange "event" happened during this test of the 1Gbps NIC... I plugged it into slot 7 and the system powered on by itself! Not sure what that means... but in any case, the NIC shows up in all slots (4-7) at 2.5GT/s @ x1 just as expected.
one last thing I wanted to try, was the LSI SAS2008 HBA cards i originally tested with, which are actually Dell H310 with LSI IT firmware. These cards occasionally have an issue with the SMBus pins B5/B6. So, i taped those pins over and thought I would try them out again. sure enough, they show as:
slot 7: card is present in BIOS, links at 5GT/s @ x8
slot 6: card is present in BIOS, links at 5GT/s @ x8
slot 5: card is present in BIOS, links at 5GT/s @ x4
slot 4: card is present in BIOS, links at 5GT/s @ x4
wow.. so at this point, i'm thinking maybe all this strangeness with the PCI-E slots was because of this SMBus pin B5/B6 all along! to validate this, I removed the tape on the pins B5/B6 and tried the card in each slot again. Remember, the last time I did this with this card without the tape on B5/B6, it would only show up in slot 4 and 7.... strangely, it shows up on all slots now, at the correct link widths just as above!
So, i grab another H310 card I had around, and tried it in all slots, and sure enough, it works in all slots at the correct link widths. Next, I decide, I'm going to populate all for PCI-E slots with H310 cards and see if maybe too many of them caused the initial issue; but no, all 4 cards show up exactly as expected and at the correct link widths.
i don't know what happened... but everything is working right now? i hope it never regresses because I just don't get it. i'm actually going to try that Marvell chipset SATA controller again later to see if it links at full speed instead of the x1 i saw yesterday. this has been one strange journey...