SMCI X9 Bifurcation I Why some boards but not all ?

Discussion in 'Hard Drives and Solid State Drives' started by svtkobra7, Apr 17, 2019.

  1. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    I don't mean to beat a dead horse as this topic has been much discussed (190 search results), but it seems as though there is not a definitive answer as to the question: Do SMCI X9 motherboards support bifurcation?
    Answers to that question run the gamut: (1) "Hell Yeah," (2) "Only on days that don't end with "day"," (c) "Never tried, but of course, I can see it in the BIOS," (d) "Bi-fur-what?" ...

    As a question that I think benefits the community generally: Has anyone given any thought as to what X9 motherboard characteristics establish whether bifurcation works? In cases where it does work, those boards must share some common attribute - it can't just be blind luck. For the sake of discussion, let's assume we are referring to only boards with BIOS updated to rev 3.3 and v2 support from factory (removing those factors from play).
    See the "Success Stories" spoiler for a sampling of 4 cases where bifurcation does work for X9 boards​

    Now, more selfishly, I'd love to know whether my 2 x X9DRH-7TF support bifurcation (I would absolutely love to pick up 2 x AOC-SLG3-2E4R @ almost half the cost of the AOC-SLG3-2E4). But that isn't a particularly common board, so I don't expect anyone to be able to confirm this.
    • Seeking an answer to my question, I emailed SMCI tech support recently and simply asked the question.
    • Their initial reply was vague at best and upon prodding for clarity, a more "precise" answer invalidates their initial reply. In summary, no dice.
      See the "SMCI Tech Support Replies" spoilers​
    • In addition to the above "success stories," I remain skeptical of their reply:
      • I've seen other tech support replies noting bifurcation does work, but doesn't allow you to boot from the device, etc.
      • It may be in SMCI's best interest to not advertise full support (so they don't have to support it), etc.
    X9DRH-7TF motherboard does not fully support bifurcation.

    For X9DRH-7TF motherboard, if you break x16 to x8x8, only the first x8 would work.
    If you break x8 to x4x4, only the first x4 would work.
    If you use one AOC, place two M.2 NVMe drives on it. Only one NVMe drive would work.
    Also X9DRH-7TF does not support M.2 NVMe as bootable device. Unless the M.2 NVMe drive contains embedded bootable firmware.

    [ links to actual posts added, '"sth_user" said' = hyperlink ]

    X9DAE


    X9DRE-TF+

    X9DRD-7LN4F-JBOD

    X9DRi-LN4F+
     
    #1
    Markess likes this.
  2. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,752
    Likes Received:
    615
    Hmm?
     
    #2
  3. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    Fixed. That is what I get for trying to be thorough with providing links (too many to deal with). ;) If only I had you around all day to double check my ALL my work! :)

    So can you answer the $1M question? (Answers such as the almighty @Rand__ doesn't know because he doesn't use "old tech" don't count.) For a filer, you still can't beat the performance/price ratio of E5 2600 v2 + DDR3 ... And yes, I would like to "have my cake and eat it too," i.e. X9 + 2.5" NVMe. Optane works great of course in the AIC form factor (even booting from it o/c); however, think it wise to go the 2.5" route for the future.

    Hope all is well.
     
    #3
  4. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,752
    Likes Received:
    615
    O/c I can't I am not an expert in ancient history ;)
    But from my experiences with bifurcation on x10 boards I would say there is no guarantee that a common rule exists between all x9 boards.

    Basically this is made difficult by the problems mentioned in success story #4, which matches my experience depicted in https://forums.servethehome.com/ind...permicro-boards-for-aoc-slg3-2e4t-et-al.17651 - i.e. that slots and bios do not necessarily match in labeling which makes it counter intuitive.
    Additionally, not all slots are bifurcateable all the time, eg a (16|8 8|0) combo pair of ports will not work), and sometimes also regular X8 ports don't, depending on where they get the PCIe lanes from ( I think)

    Also, there are 3 kinds of cards in total available
    -with PLX chip (expensive, eg 2E4)
    -with Retimer capability (cheap, eg 2E4R)
    -with ? (also Retimer?) (cheap, eg 2e4T)

    I don't know wht the official difference between R and T cards is (never investigated) but from my experience the T is way easier to get to run than the R was. I used the R on a X11SDV with lots of issues (in the early days of the R's), i.e. PSODs - never had any trouble with my T's when they started working.

    Good luck with the endeavour:)
     
    #4
  5. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    • And I even tried to prempt the old tech comment (because I saw it coming 4,635 miles away!) ;)
    • Sorry ... to ensure clarity, are you suggesting that these points make bifurcating difficult, or they make answering the question difficult, i.e. what attributes might exist that allow an X9 to bifurcate or not?
    • My assumption is that for a given offer, X9, in this case the R&D is done to create a common "baseline template" and then components and/or functionality is added or subtracted from that baseline to create your unique SKU. I'm sure it is much, much more complex than that o/c. And stated differently, they aren't engineering every SKU from scratch. Thus, considering their are some ridiculously intelligent and knowledgeable individuals on this board, I'm inclined to think that there is something that differentiates non-bifurcating boards from bifucating boards.
    • Is a retimer the same as redriver? If not, then + PLX = your 3.
    Code:
    MPN            Description                   Switch      Connectors
    AOC-SLG3-2E4   6.4GB/s Gen-3 Int NVMe HBA    PLX PE8718  2 x SFF 8643
    AOC-SLG3-2E4R  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x SFF 8643
    AOC-SLG3-2E4T  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x OCuLink
    AOC-SLG3-4E2P  6.4GB/s Gen-3 Int NVMe HBA    PLX PE9733  4 x OCuLink
    AOC-SLG3-4E4R  12.8GB/s Gen-3 Int NVMe HBA   No Switch   4 x SFF 8643
    AOC-SLG3-4E4T  4 Port NVMe PCIE X16 Retimer  No Switch   4 x OCuLink
    
    AOC-SLG3-2E4 | AOC-SLG3-2E4R | AOC-SLG3-2E4T | AOC-SLG3-4E2P | AOC-SLG3-4E4R
    | AOC-SLG3-4E4T

    • You mean attempting to help the community? It is my #1 goal in life!
     
    #5
  6. brinox

    brinox Member

    Joined:
    May 7, 2013
    Messages:
    42
    Likes Received:
    8
    Does anyone know if any of these 3 cards will work on the X10SDV-TLN4F to add 4x NVMe storage targets for file serving? I don't necessarily need VROC, but merely functional PCI Express-based NVMe storage. In fact, I'm pretty sure the Xeon-D 1541 doesn't support VROC in any capacity...
     
    #6
  7. Philmatic

    Philmatic Active Member

    Joined:
    Sep 15, 2011
    Messages:
    109
    Likes Received:
    73
    At the very least, the AOC-SLG3-4E2P will work just fine becuase of the onboard PLX chip. NVMe is not a magic technology, it works on any PCIe based motherboard, what this thread is discussing is specifically related to bifurication and whether you can run more than 1 NVMe in a single slot, which the PLX switch allows on any motherboard.

    VROC is only necessary if you intent to run RAID, I think it's only available on X11+ though.
     
    #7
    svtkobra7 likes this.
  8. copcopcopcop

    copcopcopcop Member

    Joined:
    Feb 2, 2017
    Messages:
    30
    Likes Received:
    6
    I had been waiting for the v3.3 BIOS to release for the X9DR7-LN4F/JBOD for this very same reason. Well, it finally happened this week, and boy was I was disappointed to find out that bifurcation support was NOT included in the BIOS update.

    Interestingly, I compared the 3.3 release notes for a handful of other x9 boards of the same category, and many of them list bug fixes and enhancements in addition to the spectre patches (spectre patches seem to be the one feature that every x9 3.3 BIOS includes).

    Additionally, I found that many of the boards that DO NOT include v3.3 bifurcation support are listed as "ME" releases.

    Not sure what that means...

    X9SRL-F
    Release Version: 3.3
    New features + enhancements listed: YES
    https://www.supermicro.com/Bios/softfiles/6863/X9SRL-F_BIOS_3_3_release_notes.pdf

    X9DRW-7TPF/iTPF
    Release Version: 3.3
    New features + enhancements listed: YES
    https://www.supermicro.com/Bios/softfiles/6917/P-X9DRW-7TPF-iTPF_BIOS_3_3_release_notes.pdf

    X9DRFF-7(T)+/i(T)+
    Release Version: 3.3
    New features + enhancements listed: YES
    https://www.supermicro.com/Bios/softfiles/7031/P-X9DRFF-7(T)_-i(T)__BIOS_3_3_release_notes.pdf

    X9DRG-H(T)F+II
    Release Version: 3.3
    New features + enhancements listed: YES
    https://www.supermicro.com/Bios/softfiles/7028/P-X9DRG-H(T)F_II_BIOS_3_3_release_notes.pdf

    X9DAL-I/3
    Release Version: 3.3 ME: 2.1.5.73
    New features + enhancements listed: NO
    https://www.supermicro.com/Bios/softfiles/7412/P-X9DAL-I-3_BIOS_3_3_release_notes.pdf

    X9DR7/E-LN4F
    Release Version: 3.3 ME: 2.1.7.231
    New features + enhancements listed: NO
    https://www.supermicro.com/Bios/softfiles/7547/P-X9DR7-E-LN4F_BIOS_3_3_release_notes.pdf
     
    #8
    Jax_the_Gnome likes this.
  9. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    Thanks for throwing this together ... :)

    • Thats interesting as that board was one of the sampling I listed in the OP - "success stories" ...
    • Also interesting, I'm curious to check whether any of the release notes advertise bifurcation support (can't now due to a meeting, but def will later). I don't think I've ever seen it "advertised" though ...
     
    #9
  10. copcopcopcop

    copcopcopcop Member

    Joined:
    Feb 2, 2017
    Messages:
    30
    Likes Received:
    6
    Your listed success story is the X9DRD-7LN4F-JBOD. My board is the X9DR7-7LN4F-JBOD. Nearly identical, except mine has the onboard SAS controller. Perhaps it has something to do with the onboard SAS controller?

    I am going to need more time to do some comparative analysis of all the x9 boards, but that might not be until the weekend.

    Edit: This link has been very helpful in comparing release notes.
    BIOS & IPMI Download | Intel | Support - Super Micro Computer, Inc.
     
    #10
    svtkobra7 likes this.
  11. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    My haste ... apologies ... (look forward to reviewing in more detail later)
     
    #11
  12. copcopcopcop

    copcopcopcop Member

    Joined:
    Feb 2, 2017
    Messages:
    30
    Likes Received:
    6
    No worries. Let me know what you find out.

    Just out of curiosity, which PLX enabled card are you leaning towards? I was between the AOC-SLG3-2E4R and the AOC-SLG3-2E4 depending on bifurcation support on my mobo, but the AOC-SLG3-4E2P also looks interesting. At first glance, I could fit a couple M.2 P4801x Optane sticks in there as opposed to going the 2.5 inch U.2 to miniSAS route with that bulky wiring.

    Edit: Nevermind, those are OCuLink connectors on the AOC-SLG3-4E2P. Still an interesting option.
     
    #12
    Last edited: Apr 25, 2019
    svtkobra7 likes this.
  13. Philmatic

    Philmatic Active Member

    Joined:
    Sep 15, 2011
    Messages:
    109
    Likes Received:
    73
    Correct, the success story is mine. I have the X9DRD-7LN4F-JBOD.

    I finally rebooted my VM host, so I took some screenshots, here is what my bifcurication settings look like:
    X9DRD-7LN4F-JBOD_bifcurication_1.png X9DRD-7LN4F-JBOD_bifcurication_2.png X9DRD-7LN4F-JBOD_bifcurication_3.png X9DRD-7LN4F-JBOD_bifcurication_4.png X9DRD-7LN4F-JBOD_bifcurication_5.png
     
    #13
    copcopcopcop and svtkobra7 like this.
  14. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    • As a tangent, I'd be quite curious to see SMCI tech support's reply (to the same question I asked) for a board such as @Philmatic 's X9DRD-7LN4F-JBOD.
    • Correct me if I'm wrong, but there are only two, right (AOCs w/ PLX)? AOC-SLG3-2E4 and AOC-SLG3-4E2P
    • So the question becomes connectivity, i.e. 2 x SFF 8643 or 4 x OCuLink ...
    • But to your question, since it seems the AOC-SLG3-4E2P is a good bit more pricey than the AOC-SLG3-2E4, prolly the later (as I have to do this twice over).
    Code:
    MPN            Description                   Switch      Connectors
    AOC-SLG3-2E4   6.4GB/s Gen-3 Int NVMe HBA    PLX PE8718  2 x SFF 8643
    AOC-SLG3-2E4R  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x SFF 8643
    AOC-SLG3-2E4T  6.4GB/s Gen-3 Int NVMe HBA    No Switch   2 x OCuLink
    AOC-SLG3-4E2P  6.4GB/s Gen-3 Int NVMe HBA    PLX PE9733  4 x OCuLink
    AOC-SLG3-4E4R  12.8GB/s Gen-3 Int NVMe HBA   No Switch   4 x SFF 8643
    AOC-SLG3-4E4T  4 Port NVMe PCIE X16 Retimer  No Switch   4 x OCuLink
    
    I think you are referencing these two AOCs (neither which have PLX). I see the redaction and favor this approach generally, but then there is the whole bifurcation support (or not), thing.
    AOC-SLG3-2M2 - Low Profile PCIe Riser Card supports 2 M.2 Module
    AOC-SHG3-4M2P - Full height, Quad NVMe M.2 SSD PCI-E 3.0 add-on card

    What are you trying to integrate, if you don't mind me asking? ;)
     
    #14
  15. copcopcopcop

    copcopcopcop Member

    Joined:
    Feb 2, 2017
    Messages:
    30
    Likes Received:
    6
    heh. mine look almost identical, except the bifurcation settings don't change (see pics).

    you are right. that leaves only the two. I wonder at which point the PLX chip begins to bottleneck with a 4 slot card like the AOC-SLG3-4E2P? I assume rather quickly in a x8 slot.

    I'm rebuilding my ZFS file server from scratch and want to add a SLOG device using a mirrored pair of optane drives. The AOC-SLG3-2E4 should do fine. Pcie ports are sacred and I like the idea of future proofing this build to the best of my ability, leaving room for future expansion when needed. Otherwise i'd just grab two 900p IC cards and call it a day.
     

    Attached Files:

    #15
  16. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    • The specs indicate the 9733 on the -4E2P has quite a bit of bandwidth capability (512GT = >DMI?) and the latency rather low (150 ns):
    • Assuming that translates to an immeasurable impact (latency = * 66 of 900p), but I've read on numerous occasions that the switch introduces latency.
     
    #16
  17. copcopcopcop

    copcopcopcop Member

    Joined:
    Feb 2, 2017
    Messages:
    30
    Likes Received:
    6
    Likewise, which is why I was holding out for bifurcation support. I suspect i'd never notice, but still... seems silly my board won't bifurcate, especially after comparing my screenshots with the ones a few posts up.

    I'll reach out to SM support early next week to see what they have to say about the whole x9 bifurcation thing. I'm in a back and forth with them over an unrelated hardware issue / potential RMA, and i'd rather get this sorted before annoying them with BIOS questions.

    I'm running proxmox, so no passthrough necessary. The host will manage the ZFS pool.
     
    #17
    Last edited: Apr 25, 2019
  18. copcopcopcop

    copcopcopcop Member

    Joined:
    Feb 2, 2017
    Messages:
    30
    Likes Received:
    6
    My RMA request for that memory issue I was having was approved by SMCI today. The request also included information about bifurcation not working despite being in the BIOS.

    Curious to see what happens with that.
     
    #18
  19. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    324
    Likes Received:
    65
    And you had to bring back the bad memories of "new" Ivy Bridge processors arriving to go in a X9DRi-LN4F+ rev 1.01 board (i.e. not v2 compatible), despite the website confirming support (and finding out the hard way that what SMCI states is not always representative of reality), i.e. the manual says it supports bifurcation and I see it in the BIOS too ...

    ;) of course ...

    Glad to hear your RMA was approved / keep us posted with their response, please!
     
    #19
    copcopcopcop likes this.
  20. fan

    fan New Member

    Joined:
    Mar 11, 2017
    Messages:
    4
    Likes Received:
    0
    Just test v3.3 BIOS on X9DR7-LN4F, failed: I can't split x16 to x4x4x4x4. Only one option.
     

    Attached Files:

    • a.jpg
      a.jpg
      File size:
      70.4 KB
      Views:
      32
    #20
Similar Threads: SMCI Bifurcation
Forum Title Date
Hard Drives and Solid State Drives SMCI NVMe Rear Hot Swap Drive Kit - Integration Options? Apr 18, 2019
Hard Drives and Solid State Drives NVMe Bifurcation x16 PCIe Lane to 4x x4 M.2 Slots - Adapter Wednesday at 6:10 PM

Share This Page