SMCI X9 Bifurcation I Why some boards but not all ?

svtkobra7 · Apr 17, 2019

I don't mean to beat a dead horse as this topic has been much discussed (190 search results), but it seems as though there is not a definitive answer as to the question: Do SMCI X9 motherboards support bifurcation?

Answers to that question run the gamut: (1) "Hell Yeah," (2) "Only on days that don't end with "day"," (c) "Never tried, but of course, I can see it in the BIOS," (d) "Bi-fur-what?" ...

As a question that I think benefits the community generally: Has anyone given any thought as to what X9 motherboard characteristics establish whether bifurcation works? In cases where it does work, those boards must share some common attribute - it can't just be blind luck. For the sake of discussion, let's assume we are referring to only boards with BIOS updated to rev 3.3 and v2 support from factory (removing those factors from play).

See the "Success Stories" spoiler for a sampling of 4 cases where bifurcation does work for X9 boards

Now, more selfishly, I'd love to know whether my 2 x X9DRH-7TF support bifurcation (I would absolutely love to pick up 2 x AOC-SLG3-2E4R @ almost half the cost of the AOC-SLG3-2E4). But that isn't a particularly common board, so I don't expect anyone to be able to confirm this.

Seeking an answer to my question, I emailed SMCI tech support recently and simply asked the question.
Their initial reply was vague at best and upon prodding for clarity, a more "precise" answer invalidates their initial reply. In summary, no dice.

See the "SMCI Tech Support Replies" spoilers

In addition to the above "success stories," I remain skeptical of their reply:
- I've seen other tech support replies noting bifurcation does work, but doesn't allow you to boot from the device, etc.
- It may be in SMCI's best interest to not advertise full support (so they don't have to support it), etc.

X9DRH-7TF motherboard does not fully support bifurcation.

For X9DRH-7TF motherboard, if you break x16 to x8x8, only the first x8 would work.
If you break x8 to x4x4, only the first x4 would work.
If you use one AOC, place two M.2 NVMe drives on it. Only one NVMe drive would work.
Also X9DRH-7TF does not support M.2 NVMe as bootable device. Unless the M.2 NVMe drive contains embedded bootable firmware.

[ links to actual posts added, '"sth_user" said' = hyperlink ]

X9DAE

muhfugen said:
Some X9 boards do support bifurcation. My X9DAE does, if you're interested in it let me know.

X9DRE-TF+

Dan Langille said:
Yes, yes it did. I am using FreeBSD and it shows up in /var/run/dmesg like this:

nvme0: <Generic NVMe Device> mem 0xdfd00000-0xdfd03fff irq 40 at device 0.0 numa-domain 0 on pci4
nvd0: <BPXP> NVMe namespace
nvd0: 228936MB (468862128 512 byte sectors)

I have bought a second one for mirroring but it is not in use yet.

X9DRD-7LN4F-JBOD

Philmatic said:
I have a Supermicro AOC-SLG3-2M2 on a X9DRi-LN4F-JBOD with two HP x920 NVMe SSDs and Bifurication works great with the latest BIOS, check out this thread.
https://forums.servethehome.com/index.php?threads/aoc-slg3-2m2-to-bifurcate-or-not-bifurcate-on-x9dri-ln4f.19047/

X9DRi-LN4F+

metag said:
Yes, I have it working on my x9dri-ln4f+. Just make sure you set the correct slot up because the sequence number of the bifurcation settings (IOU# as shown in frogtech's screenshot above) is not the same as the PCI slot #, at least on the x9dri IIRC.

I loaded my Asus Hyper M2 x4 card with 4 HP 1TB ex920. They were then setup as a simple stripe disk in Windows Server 2016 storage space. Here are the CrystalDiskMark result:

-----------------------------------------------------------------------
CrystalDiskMark 6.0.1 x64 (C) 2007-2018 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 5018.222 MB/s
Sequential Write (Q= 32,T= 1) : 3461.086 MB/s
Random Read 4KiB (Q= 8,T= 8) : 1055.939 MB/s [ 257797.6 IOPS]
Random Write 4KiB (Q= 8,T= 8) : 800.416 MB/s [ 195414.1 IOPS]
Random Read 4KiB (Q= 32,T= 1) : 190.762 MB/s [ 46572.8 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 151.539 MB/s [ 36996.8 IOPS]
Random Read 4KiB (Q= 1,T= 1) : 57.158 MB/s [ 13954.6 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 110.048 MB/s [ 26867.2 IOPS]
Test : 1024 MiB [H: 0.9% (35.6/3807.9 GiB)] (x5) [Interval=5 sec]
Date : 2018/11/03 9:48:58
OS : Windows Server 2016 Server Standard (full installation) [10.0 Build 14393] (x64)

I noticed the Random R/W 4K Q32T1 numbers are much lower than the numbers published here for the same disk. The number is similar when tested on single disk so it is not storage space issue. My server configuration is dual Xeon E5-2630 with 128GB memory, which I think should be more than enough. So I wonder whether that is a limitation of bifurcation in general or just the ASUS card. For $59, I probably should not expect more.

Rand__ · Apr 17, 2019

svtkobra7 said:
(I would absolutely love to pick up 2 x AOC-SLG3-2E4R @ almost have the cost of the AOC-SLG3-2E4R).

Hmm?

svtkobra7 · Apr 17, 2019

Rand__ said:
Hmm?

Fixed. That is what I get for trying to be thorough with providing links (too many to deal with).

If only I had you around all day to double check my ALL my work!

So can you answer the $1M question? (Answers such as the almighty @Rand__ doesn't know because he doesn't use "old tech" don't count.) For a filer, you still can't beat the performance/price ratio of E5 2600 v2 + DDR3 ... And yes, I would like to "have my cake and eat it too," i.e. X9 + 2.5" NVMe. Optane works great of course in the AIC form factor (even booting from it o/c); however, think it wise to go the 2.5" route for the future.

Hope all is well.

Rand__ · Apr 17, 2019

O/c I can't I am not an expert in ancient history

But from my experiences with bifurcation on x10 boards I would say there is no guarantee that a common rule exists between all x9 boards.

Basically this is made difficult by the problems mentioned in success story #4, which matches my experience depicted in https://forums.servethehome.com/ind...permicro-boards-for-aoc-slg3-2e4t-et-al.17651 - i.e. that slots and bios do not necessarily match in labeling which makes it counter intuitive.
Additionally, not all slots are bifurcateable all the time, eg a (16|8 8|0) combo pair of ports will not work), and sometimes also regular X8 ports don't, depending on where they get the PCIe lanes from ( I think)

Also, there are 3 kinds of cards in total available
-with PLX chip (expensive, eg 2E4)
-with Retimer capability (cheap, eg 2E4R)
-with ? (also Retimer?) (cheap, eg 2e4T)

I don't know wht the official difference between R and T cards is (never investigated) but from my experience the T is way easier to get to run than the R was. I used the R on a X11SDV with lots of issues (in the early days of the R's), i.e. PSODs - never had any trouble with my T's when they started working.

Good luck with the endeavour

svtkobra7 · Apr 17, 2019

Rand__ said:
O/c I can't I am not an expert in ancient history
But from my experiences with bifurcation on x10 boards I would say there is no guarantee that a common rule exists between all x9 boards.

And I even tried to prempt the old tech comment (because I saw it coming 4,635 miles away!)

Rand__ said:
Basically this is made difficult by the problems mentioned in success story #4, which matches my experience depicted in https://forums.servethehome.com/ind...permicro-boards-for-aoc-slg3-2e4t-et-al.17651 - i.e. that slots and bios do not necessarily match in labeling which makes it counter intuitive.
Additionally, not all slots are bifurcateable all the time, eg a (16|8 8|0) combo pair of ports will not work), and sometimes also regular X8 ports don't, depending on where they get the PCIe lanes from ( I think)

Sorry ... to ensure clarity, are you suggesting that these points make bifurcating difficult, or they make answering the question difficult, i.e. what attributes might exist that allow an X9 to bifurcate or not?
My assumption is that for a given offer, X9, in this case the R&D is done to create a common "baseline template" and then components and/or functionality is added or subtracted from that baseline to create your unique SKU. I'm sure it is much, much more complex than that o/c. And stated differently, they aren't engineering every SKU from scratch. Thus, considering their are some ridiculously intelligent and knowledgeable individuals on this board, I'm inclined to think that there is something that differentiates non-bifurcating boards from bifucating boards.

Rand__ said:
Also, there are 3 kinds of cards in total available
-with PLX chip (expensive, eg 2E4)
-with Retimer capability (cheap, eg 2E4R)
-with ? (also Retimer?) (cheap, eg 2e4T)

I don't know wht the official difference between R and T cards is (never investigated) but from my experience the T is way easier to get to run than the R was. I used the R on a X11SDV with lots of issues (in the early days of the R's), i.e. PSODs - never had any trouble with my T's when they started working.

Is a retimer the same as redriver? If not, then + PLX = your 3.

Code:

MPN            Description                   Switch      Connectors
AOC-SLG3-2E4   6.4GB/s Gen-3 Int NVMe HBA    PLX PE8718  2 x SFF 8643
AOC-SLG3-2E4R  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x SFF 8643
AOC-SLG3-2E4T  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x OCuLink
AOC-SLG3-4E2P  6.4GB/s Gen-3 Int NVMe HBA    PLX PE9733  4 x OCuLink
AOC-SLG3-4E4R  12.8GB/s Gen-3 Int NVMe HBA   No Switch   4 x SFF 8643
AOC-SLG3-4E4T  4 Port NVMe PCIE X16 Retimer  No Switch   4 x OCuLink

Rand__ said:
Good luck with the endeavour

You mean attempting to help the community? It is my #1 goal in life!

brinox · Apr 22, 2019

svtkobra7 said:

Code:

MPN            Description                   Switch      Connectors
AOC-SLG3-4E2P  6.4GB/s Gen-3 Int NVMe HBA    PLX PE9733  4 x OCuLink
AOC-SLG3-4E4R  12.8GB/s Gen-3 Int NVMe HBA   No Switch   4 x SFF 8643
AOC-SLG3-4E4T  4 Port NVMe PCIE X16 Retimer  No Switch   4 x OCuLink

Does anyone know if any of these 3 cards will work on the X10SDV-TLN4F to add 4x NVMe storage targets for file serving? I don't necessarily need VROC, but merely functional PCI Express-based NVMe storage. In fact, I'm pretty sure the Xeon-D 1541 doesn't support VROC in any capacity...

Philmatic · Apr 23, 2019

At the very least, the AOC-SLG3-4E2P will work just fine becuase of the onboard PLX chip. NVMe is not a magic technology, it works on any PCIe based motherboard, what this thread is discussing is specifically related to bifurication and whether you can run more than 1 NVMe in a single slot, which the PLX switch allows on any motherboard.

VROC is only necessary if you intent to run RAID, I think it's only available on X11+ though.

copcopcopcop · Apr 25, 2019

I had been waiting for the v3.3 BIOS to release for the X9DR7-LN4F/JBOD for this very same reason. Well, it finally happened this week, and boy was I was disappointed to find out that bifurcation support was NOT included in the BIOS update.

Interestingly, I compared the 3.3 release notes for a handful of other x9 boards of the same category, and many of them list bug fixes and enhancements in addition to the spectre patches (spectre patches seem to be the one feature that every x9 3.3 BIOS includes).

Additionally, I found that many of the boards that DO NOT include v3.3 bifurcation support are listed as "ME" releases.

Not sure what that means...

X9SRL-F
Release Version: 3.3
New features + enhancements listed: YES
https://www.supermicro.com/Bios/softfiles/6863/X9SRL-F_BIOS_3_3_release_notes.pdf

X9DRW-7TPF/iTPF
Release Version: 3.3
New features + enhancements listed: YES
https://www.supermicro.com/Bios/softfiles/6917/P-X9DRW-7TPF-iTPF_BIOS_3_3_release_notes.pdf

X9DRFF-7(T)+/i(T)+
Release Version: 3.3
New features + enhancements listed: YES
https://www.supermicro.com/Bios/softfiles/7031/P-X9DRFF-7(T)_-i(T)__BIOS_3_3_release_notes.pdf

X9DRG-H(T)F+II
Release Version: 3.3
New features + enhancements listed: YES
https://www.supermicro.com/Bios/softfiles/7028/P-X9DRG-H(T)F_II_BIOS_3_3_release_notes.pdf

X9DAL-I/3
Release Version: 3.3 ME: 2.1.5.73
New features + enhancements listed: NO
https://www.supermicro.com/Bios/softfiles/7412/P-X9DAL-I-3_BIOS_3_3_release_notes.pdf

X9DR7/E-LN4F
Release Version: 3.3 ME: 2.1.7.231
New features + enhancements listed: NO
https://www.supermicro.com/Bios/softfiles/7547/P-X9DR7-E-LN4F_BIOS_3_3_release_notes.pdf

svtkobra7 · Apr 25, 2019

Thanks for throwing this together ...

copcopcopcop said:
I had been waiting for the v3.3 BIOS to release for the X9DR7-LN4F/JBOD for this very same reason.

Thats interesting as that board was one of the sampling I listed in the OP - "success stories" ...

copcopcopcop said:
Well, it finally happened this week, and boy was I was disappointed to find out that bifurcation support was NOT included in the BIOS update.

Interestingly, I compared the 3.3 release notes for a handful of other x9 boards of the same category, and many of them list bug fixes and enhancements in addition to the spectre patches (spectre patches seem to be the one feature that every x9 3.3 BIOS includes).

Additionally, I found that many of the boards that DO NOT include v3.3 bifurcation support are listed as "ME" releases.

Also interesting, I'm curious to check whether any of the release notes advertise bifurcation support (can't now due to a meeting, but def will later). I don't think I've ever seen it "advertised" though ...

copcopcopcop · Apr 25, 2019

svtkobra7 said:
Thats interesting as that board was one of the sampling I listed in the OP - "success stories" ...

Your listed success story is the X9DRD-7LN4F-JBOD. My board is the X9DR7-7LN4F-JBOD. Nearly identical, except mine has the onboard SAS controller. Perhaps it has something to do with the onboard SAS controller?

I am going to need more time to do some comparative analysis of all the x9 boards, but that might not be until the weekend.

Edit: This link has been very helpful in comparing release notes.
BIOS & IPMI Download | Intel | Support - Super Micro Computer, Inc.

svtkobra7 · Apr 25, 2019

My haste ... apologies ... (look forward to reviewing in more detail later)

copcopcopcop · Apr 25, 2019

svtkobra7 said:
My haste ... apologies ... (look forward to reviewing in more detail later)

No worries. Let me know what you find out.

Just out of curiosity, which PLX enabled card are you leaning towards? I was between the AOC-SLG3-2E4R and the AOC-SLG3-2E4 depending on bifurcation support on my mobo, but the AOC-SLG3-4E2P also looks interesting. ~~At first glance, I could fit a couple M.2 P4801x Optane sticks in there as opposed to going the 2.5 inch U.2 to miniSAS route with that bulky wiring.~~

Edit: Nevermind, those are OCuLink connectors on the AOC-SLG3-4E2P. Still an interesting option.

Philmatic · Apr 25, 2019

Correct, the success story is mine. I have the X9DRD-7LN4F-JBOD.

I finally rebooted my VM host, so I took some screenshots, here is what my bifcurication settings look like:

svtkobra7 · Apr 25, 2019

copcopcopcop said:
No worries. Let me know what you find out.

As a tangent, I'd be quite curious to see SMCI tech support's reply (to the same question I asked) for a board such as @Philmatic 's X9DRD-7LN4F-JBOD.

copcopcopcop said:
Just out of curiosity, which PLX enabled card are you leaning towards?

Correct me if I'm wrong, but there are only two, right (AOCs w/ PLX)? AOC-SLG3-2E4 and AOC-SLG3-4E2P
So the question becomes connectivity, i.e. 2 x SFF 8643 or 4 x OCuLink ...
But to your question, since it seems the AOC-SLG3-4E2P is a good bit more pricey than the AOC-SLG3-2E4, prolly the later (as I have to do this twice over).

Code:

MPN            Description                   Switch      Connectors
AOC-SLG3-2E4   6.4GB/s Gen-3 Int NVMe HBA    PLX PE8718  2 x SFF 8643
AOC-SLG3-2E4R  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x SFF 8643
AOC-SLG3-2E4T  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x OCuLink
AOC-SLG3-4E2P  6.4GB/s Gen-3 Int NVMe HBA    PLX PE9733  4 x OCuLink
AOC-SLG3-4E4R  12.8GB/s Gen-3 Int NVMe HBA   No Switch   4 x SFF 8643
AOC-SLG3-4E4T  4 Port NVMe PCIE X16 Retimer  No Switch   4 x OCuLink

copcopcopcop said:
I was between the AOC-SLG3-2E4R and the AOC-SLG3-2E4 depending on bifurcation support on my mobo, but the AOC-SLG3-4E2P also looks interesting. ~~At first glance, I could fit a couple M.2 P4801x Optane sticks in there as opposed to going the 2.5 inch U.2 to miniSAS route with that bulky wiring.~~

Edit: Nevermind, those are OCuLink connectors on the AOC-SLG3-4E2P. Still an interesting option.

I think you are referencing these two AOCs (neither which have PLX). I see the redaction and favor this approach generally, but then there is the whole bifurcation support (or not), thing.
AOC-SLG3-2M2 - Low Profile PCIe Riser Card supports 2 M.2 Module
AOC-SHG3-4M2P - Full height, Quad NVMe M.2 SSD PCI-E 3.0 add-on card

What are you trying to integrate, if you don't mind me asking?

copcopcopcop · Apr 25, 2019

Philmatic said:
here is what my bifcurication settings look like:

heh. mine look almost identical, except the bifurcation settings don't change (see pics).

svtkobra7 said:
Correct me if I'm wrong, but there are only two, right (AOCs w/ PLX)? AOC-SLG3-2E4 and AOC-SLG3-4E2P

you are right. that leaves only the two. I wonder at which point the PLX chip begins to bottleneck with a 4 slot card like the AOC-SLG3-4E2P? I assume rather quickly in a x8 slot.

svtkobra7 said:
What are you trying to integrate, if you don't mind me asking?

I'm rebuilding my ZFS file server from scratch and want to add a SLOG device using a mirrored pair of optane drives. The AOC-SLG3-2E4 should do fine. Pcie ports are sacred and I like the idea of future proofing this build to the best of my ability, leaving room for future expansion when needed. Otherwise i'd just grab two 900p IC cards and call it a day.

svtkobra7 · Apr 25, 2019

copcopcopcop said:
I wonder at which point the PLX chip begins to bottleneck with a 4 slot card like the AOC-SLG3-4E2P? I assume rather quickly in a x8 slot.

The specs indicate the 9733 on the -4E2P has quite a bit of bandwidth capability (512GT = >DMI?) and the latency rather low (150 ns):
- PEX 8718
- PEX 9733
Assuming that translates to an immeasurable impact (latency = * 66 of 900p), but I've read on numerous occasions that the switch introduces latency.

copcopcopcop said:
I'm rebuilding my ZFS file server from scratch and want to add a SLOG device using a mirrored pair of optane drives. The AOC-SLG3-2E4 should do fine. Pcie ports are sacred and I like the idea of future proofing this build to the best of my ability, leaving room for future expansion when needed. Otherwise i'd just grab two 900p IC cards and call it a day.

Not sure if you will virtualize or not, and you didn't mention whether consumer or enterprise Optane (other than the "otherwise"), but just to ensure you are aware: Bug #26508: Intel Optane 900p will not work in ESX passthrough - FreeNAS - iXsystems & FreeNAS Redmine
I was never able to get the workaround to work, unsure if due to two 900ps or not.
+1 on your approach

copcopcopcop · Apr 25, 2019

svtkobra7 said:
I've read on numerous occasions that the switch introduces latency.

Likewise, which is why I was holding out for bifurcation support. I suspect i'd never notice, but still... seems silly my board won't bifurcate, especially after comparing my screenshots with the ones a few posts up.

I'll reach out to SM support early next week to see what they have to say about the whole x9 bifurcation thing. I'm in a back and forth with them over an unrelated hardware issue / potential RMA, and i'd rather get this sorted before annoying them with BIOS questions.

svtkobra7 said:
Not sure if you will virtualize or not, and you didn't mention whether consumer or enterprise Optane (other than the "otherwise"), but just to ensure you are aware: Bug #26508: Intel Optane 900p will not work in ESX passthrough - FreeNAS - iXsystems & FreeNAS Redmine

I'm running proxmox, so no passthrough necessary. The host will manage the ZFS pool.

copcopcopcop · Apr 26, 2019

My RMA request for that memory issue I was having was approved by SMCI today. The request also included information about bifurcation not working despite being in the BIOS.

Curious to see what happens with that.

svtkobra7 · Apr 26, 2019

copcopcopcop said:
My RMA request for that memory issue I was having was approved by SMCI today. The request also included information about bifurcation not working despite being in the BIOS.

Curious to see what happens with that.

And you had to bring back the bad memories of "new" Ivy Bridge processors arriving to go in a X9DRi-LN4F+ rev 1.01 board (i.e. not v2 compatible), despite the website confirming support (and finding out the hard way that what SMCI states is not always representative of reality), i.e. the manual says it supports bifurcation and I see it in the BIOS too ...

of course ...

Glad to hear your RMA was approved / keep us posted with their response, please!

fan · Sep 6, 2019

Just test v3.3 BIOS on X9DR7-LN4F, failed: I can't split x16 to x4x4x4x4. Only one option.

SMCI X9 Bifurcation I Why some boards but not all ?

Active Member

Well-Known Member

Active Member

Well-Known Member

Active Member

Member

Active Member

Member

Active Member

Member

Active Member

Member

Active Member

Active Member

Member

Attachments

Active Member

Member

Member

Active Member

New Member

Attachments