10gbe Setup Sanity Check?

[jbrukardt]

Environment:

UPLINK1: 5Gbit Uplink, Input is is a 10GBASE-T copper link from the Fiber drop, into SFP+ Port1 on router below

Router1: Opnsense w/ Mellanox ConnectX-4 (Dual SFP+)
-SFP+ Port1: 10GBASE-T Copper SFP+ : https://www.amazon.com/10Gtek-SFP-10G-T-S-Compatible-10GBase-T-Transceiver/dp/B01KFBFL16
-SFP+ Port2: 10 : 10G SFP+ TwinAX DAC: https://www.amazon.com/10G-SFP-DAC-Cable-SFP-H10GB-CU2M/dp/B00WHS3NCA?th=1

Switch1: Brocade 6450-48P
-48 POE+ 1GBE (Rest of Network)
-4 SFP+ 10GBE
-10G Twinax DAC from Router1: SFP+ Port2
-10G Link to NAS1 (undetermined)
-10G Link to PROC1 (undetermined)
-10G Link to DESK1 (undetermined)

NAS1: Unraid with Mellanox ConnectX-4 (Dual SFP+)

PROC1: Debian with Mellanox ConnectX-4 (Dual SFP+)

DESK1: Windows with Mellanox ConnectX-4 (Dual SFP+)


Quick sanity check on my NIC card choices, etc would be welcomed. Obviously im trying to limit 10GBASE-T as much as possible, but it is required from the ISP.

 

[bitbckt]

I don't see any reason why a CX3 wouldn't work here, and would save you some money over the CX4. Is there a particular reason you want/need the newer NIC?

 

[jbrukardt]

I don't see any reason why a CX3 wouldn't work here, and would save you some money over the CX4. Is there a particular reason you want/need the newer NIC?

lower power, pci-e3, future proofing was my rationale. Looking to build a relatively future proof setup.

 

[bitbckt]

So, you're planning to use 25Gb NICs then? At 10Gb, there isn't really any future proofing to be done. It already is what it will always be.

ETA: CX3 is PCI-e 3.0.

 

[mach3.2]

The CX3 drivers should still in the mainline linux kernel so no reason to pay 2x for a CX4 unless you're running ESXi on your computer.


But I mean if you have the budget, why not right?

 

[jbrukardt]

So, you're planning to use 25Gb NICs then? At 10Gb, there isn't really any future proofing to be done. It already is what it will always be.

ETA: CX3 is PCI-e 3.0.

didnt realize that! good news, might be some money saved. And you're right, due to the switch im capped out at 10GBE, and limiting myself to SFP28 isnt particularly useful, as the next likely jump would be 40gbe anyway