F2uSoe: 12 Hot swap NVME Drives in a 2U, the Intel NVMe Cage works with SuperMicro Card.

Discussion in 'Hard Drives and Solid State Drives' started by Naeblis, Nov 28, 2015.

  1. Naeblis

    Naeblis Active Member

    Joined:
    Oct 22, 2015
    Messages:
    168
    Likes Received:
    123
    After overcoming my disappointment of not getting 25GBs I decided to tinker around with the SuperMicro AOC-SLG3-2E4 card. I found a solution to the hard to find NVMe cables for those people who did not purchase a intel 750


    What would happen if I connected the SuperMicro card using a standard SAS3 cable directly to the Intel NVMe expansion cage? Turns out it works perfectly

    upload_2015-11-28_15-30-54.png


    • No special Cable needed
    • I would assume that you can put this cage in other servers
    • Hot Swappable drive
    • No trying to figure out where to mount all of these drives on the inside of a server
    upload_2015-11-28_15-33-24.png

    Yes I do get that spending 500 on a cage is expensive. However, if we figure out a way to just source the cage and not the riser and the 4 port card it could be much cheaper. On the other hand for I don't think that for Enterprise use the expense for 4 drives using 2 PCI slots would be a show stopper. ($1000.00)

    follow the yellow brick road from the SuperMicro card to the Intel cage.

    upload_2015-11-28_15-33-47.png
     
    #1
    Last edited: Nov 28, 2015
  2. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,832
    Likes Received:
    1,489
    I was curious about this.

    Time to measure :)
     
    #2
  3. zer0gravity

    zer0gravity Active Member

    Joined:
    Feb 15, 2013
    Messages:
    340
    Likes Received:
    44
    I need to find a 4 port card! Any suggestions?
     
    #3
  4. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,561
    Likes Received:
    4,490
    The Intel one pictured requires a special PCIe riser with proper PCIe lane bifurcation. I now have three Intel servers with these and did try using the card in a normal PCIe slot to suboptimal results.
     
    #4
  5. zer0gravity

    zer0gravity Active Member

    Joined:
    Feb 15, 2013
    Messages:
    340
    Likes Received:
    44
    Yea currently, I have 2x of the supermicro cards AOC-SLG2-2E4. Just wish there was a 4 or 8 port card, I thought Dell or supermicro would make one by now.
     
    #5
  6. Naeblis

    Naeblis Active Member

    Joined:
    Oct 22, 2015
    Messages:
    168
    Likes Received:
    123

    4 port cards requires special bios support to divide the 16 into 4x4. the supermacro card "R" has a chip built in. I would guess that if they could have gone 4x they would have.

    with the backplane you can use 2 cards and 4 normal SASHD cables and get 4 hotswapple drives. Tomorrow I will know if i have a reliable source for just the backplane from the intel NVMe kit.
     
    #6
  7. zer0gravity

    zer0gravity Active Member

    Joined:
    Feb 15, 2013
    Messages:
    340
    Likes Received:
    44
    Cool. Let me know how it goes.
     
    #7
  8. Naeblis

    Naeblis Active Member

    Joined:
    Oct 22, 2015
    Messages:
    168
    Likes Received:
    123
    Today the 3rd Intel cage showed up. @T_Minus will be happy that I can finally deliver the last 2 NVMe cables.

    Using 2 SuperMicro Cards and 4 normal Mini SAS HD cables I was able to put the last 4 NVMe Drives in the server.
    upload_2015-12-8_20-27-45.png

    Top down we can see that, on the outer drive bays, 8 NVMe drives are connected using the 2 Intel Risers + their special cards. The middle 4 are using the 2 SuperMicro cards. 10 500GB SATA drives + 1 hot spare make up my tier 2 storage.

    upload_2015-12-8_20-29-22.png

    Routing 15 cables, for 23 drives :D, to the front of the server was a bit of a challenge. However I do think the build turned out relatively clean.

    upload_2015-12-8_20-30-9.png
     
    #8
    Last edited: Dec 8, 2015
    tare55, Chuntzu, Jon Massey and 2 others like this.
  9. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,832
    Likes Received:
    1,489
    Gotta love that capacity :)
     
    #9
  10. mixtecinc

    mixtecinc New Member

    Joined:
    Feb 18, 2013
    Messages:
    27
    Likes Received:
    0
    Does this setup leave you any free
    pcie slots?
     
    #10
  11. Naeblis

    Naeblis Active Member

    Joined:
    Oct 22, 2015
    Messages:
    168
    Likes Received:
    123
    1 v2 x4. so no.. only thing i can think that could go there is a 4 port 1gb NIC, and i don't need that
     
    #11
  12. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,764
    Likes Received:
    858
    Tier 2 storage...LOL you kid I hope, what's your tier1 or 0 stg platform look like?
     
    #12
  13. Dsigma

    Dsigma New Member

    Joined:
    Aug 15, 2015
    Messages:
    10
    Likes Received:
    17
    Awesome.
    How did you get two A2U44X25NVMEDK working in one R2208WT?
    According to Intel R2000WT TPS, the A2U44X25NVMEDK 4 port add-in card can only work in Riser slot#2, how did you make it work in Riser slot#1 ?
     
    #13
  14. Naeblis

    Naeblis Active Member

    Joined:
    Oct 22, 2015
    Messages:
    168
    Likes Received:
    123
    You can set the bifurcation to x4 x4 x4 x4 for riser one in the BIOS
     
    #14
  15. Delo123

    Delo123 New Member

    Joined:
    May 12, 2016
    Messages:
    3
    Likes Received:
    1
    Hi, i have just build the A2U44X25NVMEDK nvme bay into an Intel R2216GZ4 and updated bios etc. However only 1 nvme is detected, always the one in the first slot. Cables should be ok since slot 1 and 2 are shared by the same cable. I have read the riser2 (which i replaced with the included 2 slot 16x/8x riser) should be setup in the bios as x4 x4 x4 x4 slot but i cannot find these options. Only thing i can find is Socket1 and 2 PCIe ports speed links, but they can only be setup to Gen2/Gen3 5 or 8GT/s), could someone point me in the right direction?

    Thx
     
    #15
  16. Delo123

    Delo123 New Member

    Joined:
    May 12, 2016
    Messages:
    3
    Likes Received:
    1
    Answered my own question. I didn't realize S2600GZ (Intel V2) isn't supported, is has to be a WT (V3 or up) to be usable. I kind of missed that "little detail"... anyway, now to get some WT's
     
    #16
    Patrick likes this.
  17. Alan Robertson

    Alan Robertson New Member

    Joined:
    Aug 10, 2016
    Messages:
    21
    Likes Received:
    2
Similar Threads: F2uSoe swap
Forum Title Date
Hard Drives and Solid State Drives F2uSoe : Waiting sucks Dec 7, 2015
Hard Drives and Solid State Drives F2uSoe: 88GB/s will have to wait. She just can take Anymore Nov 28, 2015
Hard Drives and Solid State Drives F2uSoe: Is the performance impacted by any of the 3 different ways of connecting NVMe drives. Nov 28, 2015
Hard Drives and Solid State Drives F2uSoe: She is Hungry for 20 GB/S -- 22.1 GBs Achieved!! Nov 25, 2015
Hard Drives and Solid State Drives F2uSoe: 9 disk NVMe results 19GB/S - IOPS --- 1.5mil (only) Nov 24, 2015

Share This Page