SMCI X9 Bifurcation I Why some boards but not all ?

Markess

Active Member
May 19, 2018
534
239
43
Interesting thread. My X9DRT-F doesn't have bifurcation, but then again it also doesn't have 3.3 BIOS either.

I wrote to SMC about when/if there would be a 3.3 BIOS for this model (even if its just to address Spectre/Meltdown), and the response I got was "We looked for an updated BIOS for this model and couldn't find one." No really. I had visions of people checking in desk drawers, waste baskets, and the breakroom fridge. "Say, has anybody seen a 3.3 BIOS laying around?" o_O

One thing that may influence the "Does it or doesn't i?" question may be Supermicro's stated use cases? I know my board, at least as far as SMC is concerned, is meant to be used in half-width 1U nodes, and 1U nodes are the only compatible chassis options they list as on their site. As there's no available bifurcation riser that would work in a 1U half-width format there's no incentive for SMC to build in bifurcation (because its half width, the cards plugged into the riser need to be above the I/O ports, and a two slot riser like that simply isn't going to fit in 1U). I know people use Supermicro boards in other chassis (mine is actually in a tower case), but I doubt Supermicro spends much effort supporting them.

I suppose its remotely possible to get bifurcation on a (hypothetical) 3.3.BIOS for this board if its left over from development for a different board. But, I doubt they'd put any overt effort into it if bifurcation isn't physically possible in their stated use case.
 

MBastian

Member
Jul 17, 2016
75
12
8
Düsseldorf, Germany
Just tested an Asus Hyper m.2 X16 V2 with two WD Sn730 512GB drives on my x9dri-ln4f+ (Bios 3.3)
With Arch Linux I see the two drives, regardless if the second drive is in Slot 2 or 4.
With Windows 10 I see only one drive.

On a side note: The X16s onboard fan is rather noisy.
 

frogtech

Well-Known Member
Jan 4, 2016
1,346
209
63
32
Sounds like you hit the nail on that head, they probably didn't develop that board with any intended usage scenario outside of being a 1U half-width node with a single expansion card. \

I see proclaimed bifurcation support more on the boards that use a proprietary on-board connector that adapts say a 32x electrical to dual x16 (such as WIO form factor boards) which can then further be split with BIOS options. And then of course on their newer offerings like X10 and X11 it's more standard due to the adoption of NVMe.
 

gsrcrxsi

Member
Dec 12, 2018
53
3
8
I have bifurcation settings added on my X9DAi with the latest v3.3 BIOS, but I can't make heads or tails of the settings in the BIOS vs. what the hardware actually has.

the board:


the BIOS:


I was trying an AOC-SLG3-2M2 in the top-most x8 slot, which is labelled "CPU2 Slot6" on the board, but the BIOS thinks CPU2 Slot 6 is a x16 slot. I set it to x4x4x4x4 anyway, but it would only recognize a drive in M.2_1 on the AOC, doesnt recognize anything in M.2_2.

what's going on here?
 

itronin

Well-Known Member
Nov 24, 2018
447
278
63
Denver, Colorado
I don't have an X9Dai, but I do have some X9DRD's.

I've found that using a combination of what the BIOS says *and* the block diagram in the motherboard manual you can figure it out with some guided trial and error.

I've found / seen inconsistencies and typos too. And then some things that just don't seem to work. For example Slot 1 on my in production X9DRD won't bifurcate at least not with an AOC-SLG3-2M2. But the other slots will but in some instances I've had to things opposite of what I'd expect based on the block diagram.

On the X9DRD at least mine when I bifurcate phys slot 1 to x4x4 I get subheadings like yours but they are labeled things like slot 1 GEN3, then IOU# 1B GEN3. Your screenshot shows though they aren't making that distinction when you bifurcate. It get's confusing because the the 1B above doesn't really mean slot 1, it refers to the cpu lane grouping # and the second x4 lane assignment in my X9DRD block diagram.

I suspect bifurcation was not really baked / tested well for the X9's and kind of feels like a bit of an after-thought implementation.

The block diagram of your motherboard clearly shows slot 6 as an x8 too ... its not shared with anything else either (unlike your slot 2 which is shared with the PCH). That means you've got a bios typo and/or assignment bug for that slot.

Only real way I've found is to map it out on paper during a bench build with a power cycle in between in each change because I've seen some inconsistencies with just the motherboard doing a reset after the bifurcation change.

EDIT - OH I see on your slot 6... I think as far as the IOU is concerned it is a x16 (40 lanes for the CPU). But they only gave you a physical x8.... so 8 lanes are basically unassigned and hanging out somewhere.

In Slot 6 You might also try some combination of x4x4x8 or x8x4x4 too on slot 6 and see if you can get it to work. The block diagram doesn't show which lane groupings got assigned to slot 6. I'd expect AB or CD but it could be something weird like AD or BC.

Its possible that something gets confused with the x4x4x4x4 because there's x4x4 is not going anywhere? I'm guessing.

You might also try IOU1 #1 / slot 4 with a x4 x4 setting (if it will let you) and see if you can get the SMC m2. dual card to work there.

X9Dai.png
 
Last edited:
  • Like
Reactions: istamov

itronin

Well-Known Member
Nov 24, 2018
447
278
63
Denver, Colorado
oh. On my X9DRD I know if I have the AOC-SLG3-2M2 linked up properly when the LED's blink on and off in TANDEM so ON ON is a good x4x4 link and ON OFF is not sync'ed up for both slots. Might be the same with your combination but YMMV.
 
Last edited:

kagurazakakotori

New Member
Mar 5, 2017
21
15
3
50
The PCIe bifurcation on X9DRi-LN4F+ works fine while X9DR3-F does not work at all. Unlike other boards make the bifurcation hidden in setup, Supermicro X9 boards setup does provide the options and they change the bios variable properly, but it only takes effect on some model.

Update: I emailed Supermicro, they tell me that X9DR3-F 3.3 BIOS actually doesn't support PCIe bifurcation. However, they send me a beta BIOS built on Feb 2020 that supports PCIe bifurcation. I flashed it and it works fine.
 
Last edited:

svtkobra7

Active Member
Jan 2, 2017
348
74
28
The PCIe bifurcation on X9DRi-LN4F+ works fine while X9DR3-F does not work at all. Unlike other boards make the bifurcation hidden in setup, Supermicro X9 boards setup does provide the options and they change the bios variable properly, but it only takes effect on some model.
  • EDIT: Responsive to your EDIT. There goes my theory, but it is still odd that most wins/fails fall down this line:

  • X9DRi-LN4F & X9DRi-F = Intel® C602 chipset = SATA
    • Motherboards with the C602 chipset like bifurcation.
  • X9DR3-LN4F & X9DR3-F = Intel® C606 chipset = SAS
    • Motherboards with the C606 chipset do not seem to like bifurcation.
Could it be related to the uplink port in the C606? as you discuss here: serverfault



My woes:
1. I was running 900p Optane AICs and never had the slightest issue with booting from them without modding the BIOS. So I was hopeful that I would be able to use the AOC-SLG3-2M2 for more than just one M.2 drive. No correlation there o/c.
2. As I go to shut down a server - looking forward to some benchmarking fun of 2 x 905P 380 GB - o/c it won't boot back up. Because I just installed ESXi 7.0. GREAT!

3. And o/c my X9DRH-7TF only sees only one drive on the AOC-SLG3-2M2. So I head to bed, and when I remove it this AM, I found it curious that both controllers felt to be of similar temperature and that they were so hot, not doing anything.
4. Now, I have all this NVMe - that apparently won't bifurcate - and will take up 3 PCIe slots per server. I feel like that doesn't sound like something to "complain" about. The other two will go - well - I not not sure yet - but it seemed reasonable to buy 8 M.2 NVMe drives within a few days time:
  • 2 x P4801x 100 GB
  • 2 x 905P 380 GB
  • 2 x WD SN 750 500 GB
  • 2 x Samsung 970 Evo+ 500 GB
Random thoughts:
EKWB EK-M.2 NVMe Heatsink, Black = crap engineering. Requires a ridiculous amount of force to affix the clips.


WD_Black SN750 500GB = I like this drive generally as it isn't that far off from the Evo+ and for the price I paid new with a heatsink was $20 less.

Micro Connectors, M.2 NVMe PCIe 3.0 x4 Adapter = The engineer must have thought a single green LED to be insufficient and there are six blue ones on it. Also, the bracket is not perforated. It was what the store had.

 

kagurazakakotori

New Member
Mar 5, 2017
21
15
3
50
  • EDIT: Responsive to your EDIT. There goes my theory, but it is still odd that most wins/fails fall down this line:

  • X9DRi-LN4F & X9DRi-F = Intel® C602 chipset = SATA
    • Motherboards with the C602 chipset like bifurcation.
  • X9DR3-LN4F & X9DR3-F = Intel® C606 chipset = SAS
    • Motherboards with the C606 chipset do not seem to like bifurcation.
Could it be related to the uplink port in the C606?
I believe it is not related to the chipset uplink port. Many X9DRi-F (C602) users also say their bifurcation is not working, like this post on STH, that post on STH, another post on STH, and this post on HardForum. There is also a FAQ on this issue.

Supermicro tech support told me that "The motherboard X9DR3-F does not support bifurcation." in his reply, though the manual comes with descriptions about PCIe port bifurcation and the BIOS does have the option.

He also told me they have a beta BIOS to support bifurcation. I updated the beta BIOS he sent to me, the bifurcation works fine.

The BIOS is what makes the difference. I think some X9 motherboards' PCIe bifurcation is simply not implemented.

Almost all AMI Aptio 4 based motherboards have the PCIe bifurcation. If you open an LGA2011 motherboard BIOS file with AMIBCP 4.55.0070, you will most likely find there is a bifurcation option, even if it never shows up in BIOS. Some motherboard implemented this feature but leaves the option hidden, like this tutorial configuring an ASUS X79 motherboard. Supermicro x9 boards all make the option shown, but only a few models implemented the feature behind.

Since the "i" version and "3" version shares the same BIOS file, I think the actual situation is:

  • X9DRi-LN4F+ & X9DR3-LN4F+ with 3.3 stable BIOS: bifurcation works
  • X9DRi-F & X9DR3-F with 3.3 stable BIOS: bifurcation not working
  • X9DRi-F & X9DR3-F with beta BIOS: bifurcation works
I do not have an X9DRi-F or X9DR3-LN4F+. If someone got any of these boards and tried bifurcation, please leave me a reply. If you need the beta BIOS file, you can contact the supermicro support, I would also be glad to send you a copy. The beta BIOS should also work on X9DRi-F.


1. I was running 900p Optane AICs and never had the slightest issue with booting from them without modding the BIOS. So I was hopeful that I would be able to use the AOC-SLG3-2M2 for more than just one M.2 drive. No correlation there o/c.

3. And o/c my X9DRH-7TF only sees only one drive on the AOC-SLG3-2M2. So I head to bed, and when I remove it this AM, I found it curious that both controllers felt to be of similar temperature and that they were so hot, not doing anything.
1. Intel SSD AICs seems to have OpROM onboard, so it can be booted without BIOS modding. But most M.2 SSDs don't come with an OpROM, so a UEFI NVMe driver is required for booting.

3. Though you did nothing, even one disk is not recognized, they are both powered, so they produce heat. It's very similar to just connecting the power cable to HDD, leaving the SATA cable unconnected.

172321jo85m85keqtoqz5t.jpg

The figure above (source link) is the design of a passive x16 to x4x4x4x4 bifurcation card. The power is independent of the PCIe signal.


For the bifurcation card, I have been using the JEYI iHyper M.2 X16 for about 1yr without any problem, and it is the cheapest bifurcation card on the market.
 

olepropell

New Member
Sep 22, 2020
4
3
3
I just tried BIOS 3.4 with my X9DRi-F which didn't accept bifurcation with BIOS 3.3 (non beta). After flashing the BIOS I tried ASUS Hyper M.2 X16 V2 in 4x4x4x4 mode in slot 4 with success. I have only 2 NVMe devices, but I tried them in all 4 slots and all suddenly worked. I also tried Supermicro AOC-SLG3-2M2 which also worked perfectly in 4x4 slot 5.
 

josh

Active Member
Oct 21, 2013
427
101
43
I just tried BIOS 3.4 with my X9DRi-F which didn't accept bifurcation with BIOS 3.3 (non beta). After flashing the BIOS I tried ASUS Hyper M.2 X16 V2 in 4x4x4x4 mode in slot 4 with success. I have only 2 NVMe devices, but I tried them in all 4 slots and all suddenly worked. I also tried Supermicro AOC-SLG3-2M2 which also worked perfectly in 4x4 slot 5.
Just to confirm, this is the 3.4 release Jun 20 on the official site? I returned my Hyper M.2 but I'd get a new one if this is confirmed to work with 2+ drives.
 

olepropell

New Member
Sep 22, 2020
4
3
3
Just to confirm, this is the 3.4 release Jun 20 on the official site? I returned my Hyper M.2 but I'd get a new one if this is confirmed to work with 2+ drives.
Yes, it didn't work for me at all with 3.3 version. And I heard a rumour that a beta version supported it. But when I looked again at the webpage a new official version, 3.4 existed so I just tried that. Then it worked as a charm and correct assignments in bios versus physical slots. I tried with 2 nvme devices in all slots at x4x4x4x4 and all showed up.
 

josh

Active Member
Oct 21, 2013
427
101
43
Yes, it didn't work for me at all with 3.3 version. And I heard a rumour that a beta version supported it. But when I looked again at the webpage a new official version, 3.4 existed so I just tried that. Then it worked as a charm and correct assignments in bios versus physical slots. I tried with 2 nvme devices in all slots at x4x4x4x4 and all showed up.
Interestingly, I tried emailing supermicro for the beta BIOS and they refused to provide it to me. I'm just puzzled why they didn't bother letting me know 3.4 fixed the problem or that there was a new bios dropped even for a supposedly EOL product.

For any custom bios, it would be a special request which is not free for customers. And not all motherboard has bifurcation feature for PCIE because of hardware limitation. Lastly, this product you have already EOL so we no longer have new bios for the motherboard.
 

olepropell

New Member
Sep 22, 2020
4
3
3
Weird ye, but I can just confirm that it works on my X9DRi-F board with one SCU using onboard administration updater.
 

josh

Active Member
Oct 21, 2013
427
101
43
Weird ye, but I can just confirm that it works on my X9DRi-F board with one SCU using onboard administration updater.
Did you have to update firmware as well? It seems like there's a new 3.62 released as well but that was dropped after 3.4 BIOS. I think I'm currently using 3.54
 
Last edited:

josh

Active Member
Oct 21, 2013
427
101
43
Yes, I updated the firmware before the BIOS
I just upgraded to BIOS 3.4 from the official site and can confirm that bifurcation WORKS. Firmware was updated to 3.62 as well. Thanks for the heads up. Now I have to order a Hyper M2 again.
 

StevenDTX

Active Member
Aug 17, 2016
428
145
43
Well, of course, they release a new bios a month after I spend $350 on a card with a PLX chip.
 

josh

Active Member
Oct 21, 2013
427
101
43
Well, of course, they release a new bios a month after I spend $350 on a card with a PLX chip.
Well technically it's been out since June but the 9 users that need bifurcation didn't know to test it out. I've also wasted a bit of money specifically getting the SM dual M2 card just to test if it was a Hyper M2 issue and was just lucky I hadn't sold it off yet. The M2 I got at last year's Black Fri prices is gone though. Will have to hope it goes on sale this year as well.