SM x11sca-f + icydock dual U2 enclosure

[denywinarto]

Hi guys, about to buy this dual U2 dock but i cant seem to find reliable info whether its compatible or not
Have emailed icydock support but they havent replied yet...

MB105VP-B_2 Bay 2.5" U.2/U.3 NVMe SSD PCIe 4.0 Mobile Rack Enclosure for External 5.25" Drive Bay (1 x SlimSAS SFF-8654 8i)

global.icydock.com

the cable

Slim SAS SFF-8654 4i Straight to HD Mini SAS 4i SFF-8643 Cable 50CM

One of the benefits of this SlimSAS SFF-8654 4i Straight to HD Mini SAS 4i SFF-8643 Cable is that its smaller size connector and cable saves device space.

www.microsatacables.com

X11SCA-F does have U2 port which in theory should work with the cable.
But the enclosure says it will work with 8654-8i and it doesn't mention 8645-4i

Planning to use it for dual P4510 8 tb, currently running 1 but yesterday its dead, so i wanna try dual setup for a backup.

X11SCA-F | Motherboards | Products | Supermicro

Any thoughts?

 

[BLinux]

I have a feeling that's not the cable. You're going to need the 8i (8 lane) SFF-8654 cable because most NVMe have 4x PCIe lanes and I suspect that cable needs 8 lanes for the 2 U.2 bays. The "4i" part in the description makes me nervous. I would confirm that the cable is wired for all 8 lanes.

 

[denywinarto]

I have a feeling that's not the cable. You're going to need the 8i (8 lane) SFF-8654 cable because most NVMe have 4x PCIe lanes and I suspect that cable needs 8 lanes for the 2 U.2 bays. The "4i" part in the description makes me nervous. I would confirm that the cable is wired for all 8 lanes.

Well if its true I'm out of luck, X11SCA-F only has 1 u2 port and 8i needs 2 u2... Need pcie device then but not sure if i have enough lanes for another one,
I'm using 2278G with the following :
- lsi 3008 QSFP for jbod rack
- dual 10g intel
- m2 nvme
- 6 sata ssd's

 

[jei]

If you have PCIe Gen3 8x slot free, consider NVMe switch eg. Supermicro AOC-SLG3-4E2P

 

[denywinarto]

If you have PCIe Gen3 8x slot free, consider NVMe switch eg. Supermicro AOC-SLG3-4E2P

Well I have this setup

CPU SLOT 4 PCIE 3.0 x8 (in x16) : Intel X520 -DA2
CPU SLOT 8 PCIE 3.0 x16 : LSI SAS 3008 AVAGO (x8 link when SLOT4 is in use)
PCIE M2-M1 and M2-M2 (NVME slot) : Intel SSDPELKX010T8 Hard Drive 1T NVMe PCIe3.1 x4 M.2 22x1
U2: P4510 8tb > can be replaced
Sata slot : 7 ssd's

So that leaves me with
CPU SLOT 2 PCIE 3.0 x4 (shared with m-2.m1)
CPU SLOT 7 PCIE 3.0 x1
PCI SLOT1 33MHz > not usable

Don't think it would be possible to add 2 x4 lanes no?
Unless if there's a way to utilize CPU Slot 2 with PCIE device x4 for U2..
that way at least i could use 2 U2 drives together with the U2 slot. But does such thing exist?

Nevermind, CPU slot 2 is shared with NVME drive, guess i'm stuck

 

[Tech Junky]

NVME drive, guess i'm stuck

Not stuck just not full speed. There are m2 adapters you can use. There also direct mount PCB adapters for single/double/quad drives as well.

 

[denywinarto]

Not stuck just not full speed. There are m2 adapters you can use. There also direct mount PCB adapters for single/double/quad drives as well.

By m2 adapter you mean i should be using the intel nvme drive on sata slot correct?
Well its for image booting so it needs to be above sata speed so im afraid its not an option.
Its mean to be a drive with as few write as possible to preserve the OS.

While the U2 drives are drives that still get updates occasionally and also need higher speed. So i need to separate them, too risky if the main OS fails.

A shared speed is still better i guess,

So you mean something like this right:

1. U2 port = U2 drive 1
2. PCIe 3.0 x4 = HBA x4 lanes card for U2 drive 2 (?)

That will work too if the such card exists, any suggestions? I will try to google a bit

 

[Tech Junky]

OK... so, there are U2 models that will run just fine in X2 IIRC there's Samsung drives that have dual X2 ports on them for some reason. One reason I avoided them when looking to U drives after realizing they're cheaper in higher capacities and go beyond 8TB unlike M2 drives.

So, since you're system is a bit older w/ PCIE 3 being the top end it's a bit of a crunch and/or if you're planning to rebuild to get more bandwidth in the near future it makes a difference in which direction you want to go in.

https://www.amazon.com/dp/B0C6B2PGMC - this 8TB was selling for $400 not too long ago and should come back down
https://www.amazon.com/dp/B0BYD15JVH - M2 adapter for $20
https://www.amazon.com/dp/B094J4QZT9 - M2 cable $30

With the M2 setup and a different Gen4 drive I'm getting the full 6.5GB/s out of the drive. Mind you this is on a G4 system though. On G3 though you should be able to get it up to about 3.5GB/s.

There are x8 Oculink cards though as well if you have the slot open or want to use it for U drives instead of your spinners. There's also x16 cards for $50 that can get you 4 U drives off one slot and the cables are dual drive @ $50/ea. Also, there's an x4 card you could slot as well for a single drive.

https://www.amazon.com/dp/B0BZJQ2V8W

https://www.amazon.com/NFHK-SFF-8639-Express-Raid0-Adapter/dp/B097SJHLQQ - Dual x8
https://www.amazon.com/PCIe-Adapter-Card-SFF-8639-NVMe/dp/B0BX4DXSWV - same but cheaper @ $33

Another option could be go external....

https://www.amazon.com/dp/B0CFKK1CM7

https://www.amazon.com/dp/B09DKLM82S

One of the quad U cards... https://www.amazon.com/WANGFUFU-PCI-Adapter-SFF-8639-NVMe/dp/B0CLXSSMWT

https://www.amazon.com/Extension-Install-SFF%E2%80%918639-Interface-SSD%E2%80%91SFF%E2%80%918639/dp/B09NCFQ6SX

I might suggest the x16 card for $50 and then get a SAS cable for one of the ports for your spinners and get the best of both worlds off a single slot. Might have to get creative and find a smaller 10GE card though like this - https://www.bhphotovideo.com/c/product/1709735-REG/owc_owcpcie10gb_10g_ethernet_pcie_network.html

The other consideration is if the U2 port disables anything when in use vs unused. Seems to me it would trigger some lanes being removed from somewhere else.

So, here's what I would end up with...
CPU SLOT 4 PCIE 3.0 x16) :x16 oculink card split between 2 x U and 2-4 SAS drives
PCIE M2-M1 and M2-M2 (NVME slot) : Intel SSDPELKX010T8 Hard Drive 1T NVMe PCIe3.1 x4 M.2 22x1
Sata slot : 7 ssd's

So that leaves me with
CPU SLOT 2 PCIE 3.0 x4 10GE NIC as linked above for single port
CPU SLOT 7 PCIE 3.0 x1

I feel like maybe I don't have the full picture here though as you listed 7 SATA SSD's at the bottom. What's hanging off the LSI card then?

 

[denywinarto]

The other consideration is if the U2 port disables anything when in use vs unused. Seems to me it would trigger some lanes being removed from somewhere else.

Thats the thing, the manual doesn't mention anything about this

Might have to get creative and find a smaller 10GE card though like this - https://www.bhphotovideo.com/c/product/1709735-REG/owc_owcpcie10gb_10g_ethernet_pcie_network.html

I feel like maybe I don't have the full picture here though as you listed 7 SATA SSD's at the bottom. What's hanging off the LSI card then?

Sorry should have explained it,

I have 2 10G switches, so I need dual 10G NIC.
And the LSI card is for HGST 4u60 which is a 60 bay HDD rack.
I'm not sure if there are x4 card alternatives for these..

So yeah the lanes usage are a bit tight, I thought it was barely enough for this cpu gen to not go xeon variants.
But then i realized i needed a backup for the U2 drive..

 

[Tech Junky]

Even my new AMD build feels constrained. I wish mobo makers allowed for more slots and lane control.

I specifically targeted a board option that didn't auto splice lanes when slotting things with the idea of putting one of those x16 cards into play later in but the media conversion on the CPU just wasn't good enough in terms of speed. Good thing is I got the board for $160 and didn't pay the retail sticker on things. Modern boards seem to have a top end of four slots which makes things tricky when you want to max things out. However modernizing the pcie gen affords more options. At this point I'm waiting for the new boards to surface to see if things improve. I might even cough up some more cash and look into ASRock rack boards as they're a middle ground of server/home use. I just got fed up with Intel and wanted more options to play around with.

 

[denywinarto]

Even my new AMD build feels constrained. I wish mobo makers allowed for more slots and lane control.

I specifically targeted a board option that didn't auto splice lanes when slotting things with the idea of putting one of those x16 cards into play later in but the media conversion on the CPU just wasn't good enough in terms of speed. Good thing is I got the board for $160 and didn't pay the retail sticker on things. Modern boards seem to have a top end of four slots which makes things tricky when you want to max things out. However modernizing the pcie gen affords more options. At this point I'm waiting for the new boards to surface to see if things improve. I might even cough up some more cash and look into ASRock rack boards as they're a middle ground of server/home use. I just got fed up with Intel and wanted more options to play around with.

Indeed, intel still the king of transcoding unfortunately, last time i checked emby transcoding was still better using intel than amd,

Anyways I might find the solution to my problem,
So just found out my HGST 4u60 jbod doesnt actually need both x4 lanes, So something like LSI 9300 4e4i might work,
4e to hgst 4u60
4i to the second U2 drive.
Will try getting one and test this..

 

[jei]

OK... so, there are U2 models that will run just fine in X2 IIRC there's Samsung drives that have dual X2 ports on them for some reason. One reason I avoided them when looking to U drives after realizing they're cheaper in higher capacities and go beyond 8TB unlike M2 drives.

So, since you're system is a bit older w/ PCIE 3 being the top end it's a bit of a crunch and/or if you're planning to rebuild to get more bandwidth in the near future it makes a difference in which direction you want to go in.

https://www.amazon.com/dp/B0C6B2PGMC - this 8TB was selling for $400 not too long ago and should come back down
https://www.amazon.com/dp/B0BYD15JVH - M2 adapter for $20
https://www.amazon.com/dp/B094J4QZT9 - M2 cable $30

With the M2 setup and a different Gen4 drive I'm getting the full 6.5GB/s out of the drive. Mind you this is on a G4 system though. On G3 though you should be able to get it up to about 3.5GB/s.

There are x8 Oculink cards though as well if you have the slot open or want to use it for U drives instead of your spinners. There's also x16 cards for $50 that can get you 4 U drives off one slot and the cables are dual drive @ $50/ea. Also, there's an x4 card you could slot as well for a single drive.

https://www.amazon.com/dp/B0BZJQ2V8W

https://www.amazon.com/NFHK-SFF-8639-Express-Raid0-Adapter/dp/B097SJHLQQ - Dual x8
https://www.amazon.com/PCIe-Adapter-Card-SFF-8639-NVMe/dp/B0BX4DXSWV - same but cheaper @ $33

Another option could be go external....

https://www.amazon.com/dp/B0CFKK1CM7

https://www.amazon.com/dp/B09DKLM82S

One of the quad U cards... https://www.amazon.com/WANGFUFU-PCI-Adapter-SFF-8639-NVMe/dp/B0CLXSSMWT

https://www.amazon.com/Extension-Install-SFF%E2%80%918639-Interface-SSD%E2%80%91SFF%E2%80%918639/dp/B09NCFQ6SX

I might suggest the x16 card for $50 and then get a SAS cable for one of the ports for your spinners and get the best of both worlds off a single slot. Might have to get creative and find a smaller 10GE card though like this - https://www.bhphotovideo.com/c/product/1709735-REG/owc_owcpcie10gb_10g_ethernet_pcie_network.html

The other consideration is if the U2 port disables anything when in use vs unused. Seems to me it would trigger some lanes being removed from somewhere else.

So, here's what I would end up with...
CPU SLOT 4 PCIE 3.0 x16) :x16 oculink card split between 2 x U and 2-4 SAS drives
PCIE M2-M1 and M2-M2 (NVME slot) : Intel SSDPELKX010T8 Hard Drive 1T NVMe PCIe3.1 x4 M.2 22x1
Sata slot : 7 ssd's

So that leaves me with
CPU SLOT 2 PCIE 3.0 x4 10GE NIC as linked above for single port
CPU SLOT 7 PCIE 3.0 x1

I feel like maybe I don't have the full picture here though as you listed 7 SATA SSD's at the bottom. What's hanging off the LSI card then?

Some of these options require bifurcation and X11SCA-F doesn't seem to support it.

 

[jei]

Indeed, intel still the king of transcoding unfortunately, last time i checked emby transcoding was still better using intel than amd,

Anyways I might find the solution to my problem,
So just found out my HGST 4u60 jbod doesnt actually need both x4 lanes, So something like LSI 9300 4e4i might work,
4e to hgst 4u60
4i to the second U2 drive.
Will try getting one and test this..

LSI 9300 4e4i does not support NVMe so it definetely will not work. SAS/NVMe can share same connectors but that does not mean they are compatible. You'd need Tri-Mode HBA for that.

 

[denywinarto]

LSI 9300 4e4i does not support NVMe so it definetely will not work. SAS/NVMe can share same connectors but that does not mean they are compatible. You'd need Tri-Mode HBA for that.

So i guess i need at least something like 9400 8i8e then? it could work partially right meaning i could plug just one each outside and inside?
just googled it a bit,
apparently it needs PCIE 3.1 while x11sca-f is still PCIE 3.0, googled again and somebody says it's backward compatible but it might have stability issues... might not be worth it then for that price,
I was planning to have a mirror backup so i wont have to deal with downtime while copying.
But i think i'll just plug the other one to my mikrotik + HA machine.

 

[mattventura]

Those are expensive and can be a lot slower than native NVMe. They do not pass through the NVMe drives into the host's PCIe topology - they translate and convert them to look like SAS drives.

You also have the option of getting an NVMe switch as well as some adapters to convert your M.2 drives to U.2. That would open up the X4 slot, since 2x10GbE only requires Gen3x4. Can't tell what your JBOD requires without knowing what drives are in it.

 

[denywinarto]

It's HGST 4u60 JBOD right now it's working with LSI 3008 8e, with 2 QSFP+ cable, but I pulled one of them awhile ago and it's still working so I'd assume it'd work with 4x lanes.. drives are 3.5 inch
Got any specific model for this NVME switch?
But actually only 10gbase-t could work with Gen3 x4 (Intel x550)
or you're saying i should just put the x520-DA2 to X4 slot?

 

[mattventura]

It's HGST 4u60 JBOD right now it's working with LSI 3008 8e, with 2 QSFP+ cable, but I pulled one of them awhile ago and it's still working so I'd assume it'd work with 4x lanes..
Got any specific model for this NVME switch?
But actually only 10gbase-t could work with Gen3 x4 (Intel x550)
or you're saying i should just put the x520-DA2 to X4 slot?

For the JBOD, I'm referring more to how much bandwidth you need to not bottleneck. Depends on the performance of drives, how many drives, access patterns, etc.

For the NVMe switch, AOC-SLG3-4E2P would be the official Supermicro switch, but you can certainly find cheaper ones on AliExpress.

Come to think of it, an X520 would bottleneck in an x4 slot, because it's only PCIe 2.0. You would need something PCIe 3.0 if you want it to not bottleneck on PCIe bandwidth.

 

[denywinarto]

For the JBOD, I'm referring more to how much bandwidth you need to not bottleneck. Depends on the performance of drives, how many drives, access patterns, etc.

It's low priority drives and it's not in raid mode or anything like that, just drivepool, max capacity is 60,
but it's half full and probably gonna keep it that way
So assuming max speed of HDD is 550 Mbps, even if all 30 were pulled together it'd need 16.5 Gbps (most likely wont happen)
A single QSPF+ supports 40Gbps, which is why i believe the LSI 3008 is overkill for this thing.

Come to think of it, an X520 would bottleneck in an x4 slot, because it's only PCIe 2.0. You would need something PCIe 3.0 if you want it to not bottleneck on PCIe bandwidth.

Yeah i was going to replace it awhile ago but I couldn't find any PCIE 3.0 dual 10G that works with 4 lanes
But just googled and i found this :
https://www.reddit.com/r/homelab/comments/y4kbsh He said any dual 10G card should work, so i'm considering melanox connect x3 or intel X7xx card

Edit : just remembered, the X4 slot is shared with the NVME slot, so wouldn't it affect the NVME speed then? Thats probably i left it open 2 years ago

 

[jei]

Got any specific model for this NVME switch?

For the NVMe switch, AOC-SLG3-4E2P would be the official Supermicro switch, but you can certainly find cheaper ones on AliExpress.

Yep.. when I recommended AOC-SLG3-4E2P 24 hours ago in this same thread I did first some digging around Aliexpress and compared to Supermicro prices. From EU standpoint atleast the prices for 4x switches are the same for SM and for example Ceasent from Alix. So I wouldn't even bother with non-SM.

https://www.aliexpress.com/item/4000756587057.html

Ceasent products have somewhat good reputation here and I've tested some myself. I wouldn't even recommend trying marginally cheaper random alix/amazon brands.

 

[jei]

Here's a thought. TBH the expenses being in triple digits I would look into new MB.

OLD: CPU SLOT 4 PCIE 3.0 x8 (in x16) : Intel X520 -DA2
NEW: AOC-SLG3-4E2P

OLD: CPU SLOT 8 PCIE 3.0 x16 : LSI SAS 3008 AVAGO (x8 link when SLOT4 is in use)
NEW: KEEP AS IS

OLD: PCIE M2-M1 and M2-M2 (NVME slot) : Intel SSDPELKX010T8 Hard Drive 1T NVMe PCIe3.1 x4 M.2 22x1
NEW: (1x or 2x) Oculink adapted connected to AOC-SLG3-4E2P

OLD: U2: P4510 8tb > can be replaced
NEW: 2x U2 drives in IcyDock cage connected to AOC-SLG3-4E2P

OLD: Sata slot : 7 ssd's
NEW: KEEP, I guess

OLD: CPU SLOT 2 PCIE 3.0 x4 (shared with m-2.m1)
NEW: Chelsio ConnectX-3 2 port model

LEFTOVERS:
CPU SLOT 7 PCIE 3.0 x1
PCI SLOT1 33MHz > not usable

edit: Not sure if M.2 NVMe would be bootable in this setup.