Ideas for SFP+ low power PCIe x 4 slot

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
C

cookiemonsteruk

Member
Mar 4, 2020
32
4
8
Hello community.
I am very unknowledgeable about fiber networking. I have been reading threads in various places but I need now to start getting suggestions from people who know their stuff.
My goal is to identify a suitable solution for me to connect one NAS to a managed switch via SFP+. Suitable is low ongoing costs due to high electricity costs in the UK, and low cost to purchase.
I have a Mikrotik CSS326-24G-2S+ as the managed switch. I wish to connect it via SFP+ to a NAS running on an Gigabyte MC12-LE0-rev-1x. Distance between them is around 4 mtrs in a straight line, so to account for space for curves and some slack I think I need to target a 5 to 10 mtr cable run. My biggest constraint seems to be the NIC for the server; the motherboard has one PCIe 4 X 16 and one PCIe 4 X 4 slots. The x 16 is in use by the storage HBA, an IBM M1015. Therefore I think that can only use a SFP+ NIC that uses a PCIe X 4 slot. The slot is not open to accept larger.

So after reading high and low, I think I want to go with:
1) Modules :
Pack of two 10Gb SFP+ SR Multimode Module 300-meter like link which is £27.99

2) Fibre cable:
I don't know which cable to use.

3) NIC for the server:
Here is where I am really stuck. I want to be able to use higher C states than C2 and was reading that for that reason I would need Connecx-4 or Intel X710 or X720 but they seem all PCIe X 8.

Can someone please gently help me, with both suggestions for NIC and cable? I'll be grateful.
EDIT: NAS OS is freeBSD-based, realtek is not a chipset I can use, as in unwilling to deal with their problems.
Edit 2: This server has ESXi on it. I know, strange choice for such a constrained system but there was a good explanation at the time, and is still valid. The HBA is passed through of course to the NAS VM.
 
Last edited:
B

boerni666

Member
Sep 8, 2023
61
67
18
2) You can use a 10m DAC cable, then you dont need to worry about transceivers.

3) i can really recommend the Realtek RTL8127 card. It works well with Linux 7.0 and ASPM states, uses <2W power. Works flawlessy for me so far. You'll find a PCIe 3.0 x2 SFP+ Version on Aliexpress
 
C

cookiemonsteruk

Member
Mar 4, 2020
32
4
8
Thank you.
2) Power draw at idle and in-use the same ? Something I need to re-read.
3) I should have included this as requirement. The OS of the NAS is freeBSD, therefore I am unwilling to deal with the problems of the realtek drivers.
 
TrevorH

TrevorH

Active Member
Oct 25, 2024
264
112
43
To literally expand your options, I've just ordered something like Kdouehg Dual NVMe PCIe Adapter, PCIe 4.0 X16 to Dual NVMe & 1x More PCIe X16 (X8 Signal) Adapter for M.2 NVMe SSD Adapter: Amazon.co.uk: Computers & Accessories which could sit between your M1015 and the PCIe x16 slot and give you 2 x M.2 x4 sockets to play with. Amazon lists several M.2 to 10GbE devices, single port only AFAICS. Your M1015 is a PCIe 2.0 x8 that looks like it's half height with a full height bracket and a quick Google says your motherboard supports PCIe bifurcation.
 
C

cookiemonsteruk

Member
Mar 4, 2020
32
4
8
TrevorH

TrevorH

Active Member
Oct 25, 2024
264
112
43
I looked at the specs for that mobo and the onboard M.2 is only x1 which I find really quite weird.
 
C

cookiemonsteruk

Member
Mar 4, 2020
32
4
8
yes it is strange, maybe they ran out of lanes.
One very important part of the setup that I inadvertently omitted is that this server has ESXi on it. I know, strange choice for such a constrained system but there was a good explanation at the time, and is still valid. The HBA is passed through of course to the VM. I'll update the OP.
 
S

SnJ9MX

Active Member
Jul 18, 2019
209
155
43
x1 is still good for 500-600MB/s. I have some HGST U2 drives in a x1 electrical, x16 physical slot and it works fine, just at 500-600 MB/s
 
TrevorH

TrevorH

Active Member
Oct 25, 2024
264
112
43
Sure a single lane works but it's far from optimal. This is 1 PCIe 3.0 lane on a board that otherwise has PCIe 4.0 so 1 lane should give more like 880MB/s but if you have a PCIe 4.0 SSD that's capable of 7GB/s that seems like quite a waste. It's quite a waste even if it's a PCIe 3.0 drive - I'd want the other 3GB/s back.

Edit: and to get back OT, an M.2 10GbE adapter in that socket wouldn't have enough b/w to run at full speed.
 
S

SlowmoDK

Active Member
Oct 4, 2023
308
160
43
Intel X710-DA2 or DA4 runs just fine in the 2nd slot (x4) on your Gigabyte MC12-LE0 (slot is open-ended)
I have this precise setup, Linux will complain about not having the full x8 as a warning, but it's just that a warning

Unless you plan to saturate all 4 SFP+ at the same time, x4 is a non issue even though card is only PCIE Gen 3

This solution is battle tested and tried and has by far the best support for ASPM in your usecase
 
Last edited:
C

cookiemonsteruk

Member
Mar 4, 2020
32
4
8
SlowmoDK said:
Intel X710-DA2 or DA4 runs just fine in the 2nd slot (x4) on your Gigabyte MC12-LE0 (slot is open-ended)
I have this precise setup, Linux will complain about not having the full x8 as a warning, but it's just that a warning

Unless you plan to saturate all 4 SFP+ at the same time, x4 is a non issue even though card is only PCIE Gen 3

This solution is battle tested and tried and has by far the best support for ASPM in your usecase
Click to expand...
Do you have this same motherboard? Mine is hard to reach, in a 4U case and I have a "frozen shoulder" making it hard to get to it to check if indeed the second slot is open. If yes then then that opens the possibilities greatly. I am putting the suggestions of bifurcating as a less desirable option for now.
And the disks are a combination of SAS 7.2 KRPM in zfs only 3 or 4 wide vdevs and some SATA SSDs in mirrors. Certainly won't be saturating any links with this. The reason for going 10 Gbps network is I want to have enough separate bandwith for backups and since the SFP+ ports are unused on the switch, I might as well use them.
 
You must log in or register to reply here.
Top Bottom