Looking for more than GB Ethernet, need advice on faster network setup

Pete L.

Member
Nov 8, 2015
133
23
18
53
Beantown, MA
OK, so what kind of cable would I use to connect the ConnectX2 in my PC or server to the switch's SFP+ port?
I'm thinking of getting the Trendnet TEG-30284 since I like the price and I don't need anything fancy.

Also, I may need to buy a new mobo for my main PC, since it only has 1 PCIE x16 3.0 slot & 3 PCIE x1 slots. Will the connextX2 run in a PCI 3.0 x4 slot?
As mentioned if the systems are close the cheapest way to go will be with a DAC Cable, Fiber Store Sells them, I don't know if the switch is picky about what kind of SFP / SFP+ / DAC Cables it will take (AKA Vendor Locked). Check with Fiber Store, they might sell modules specifically qualifies for that switch. I don't think you will have any issues with the Mellanox Cards, I've never had them complain about any of the SFP Modules I've used and I've used all kinds of no-name ones. The only thing that complained was my Intel Cards, I had to get Intel Specific / Compatible modules.

Here are the DACs you can see why they are the cheapest option
Custom & OEM 10G SFP+ to SFP+ DAC Cables | FS.COM

If you are more than say 7 Meters then you will need to go with Fiber and you have to decide if you want to go with Multimode (10G SFP+ Module is $16 you will need two) or Single Mode (10G SFP+ Module is $35 each x2) plus the fuber of course.

I can't comment on the card working in the slot or not, if it physically fits it will work, you just might not get full bandwidth especially with a dual port card.
 
Last edited:
  • Like
Reactions: Jerry Renwick

aero

Active Member
Apr 27, 2016
312
54
28
50
Connectx2 is a PCIe 2.0 card. So it will run at 2.0 speeds in your 3.0 slots. 500MB/s x 4 lanes = 2000MB/s in a single direction. 10Gbe is 1.25GB/s, so a single port would get full bandwidth, but a dual port would be a little short.
 

fractal

Active Member
Jun 7, 2016
312
69
28
29
It sure is nice to try to future proof your installations. Sometimes you even get it right. Usually not. And when you do, you spent premium dollar for something you could have paid commodity prices for when you actually needed it.

I am going through the same though process myself. It does not take much analysis to realize that you are limited to the slowest link in the chain. 10000 TB networking in the house won't play youtube any faster than your 1.5 megabit DSL modem can deliver it. I use this whimsical and obvious example to highlight the importance of knowing your source. Clearly, few people would be looking to invest in 10GBE to surf the internet.

Connecting your desktop PC to your home switch at 10GBE is an interesting thought. It is like the boss at work who "MUST BE CONNECTED TO THE CORE SWITCH!" Not that being connected to the core switch makes it any faster than being connected to an edge switch. It is just that he IS THE BOSS! and ports on the core switch are a scarce resource, and HE IS WORTH IT! You don't argue, you just do it. Likewise, connecting your PC to a precious 10GBE port is an interesting thought, but without any source to produce content faster than 1 GB or a way to process the data faster than 1GB is kinda like the BOSS!

Yeah, I know... I am being silly ;)

The point I am trying to make is you need to understand where the bottleneck really is. Copying a file from a hard drive on one PC (or server) to a file on a hard drive on another PC will not even get the attention of a gigabit network, much less exceed it. You should probably consider adding faster links to devices that can source or sink data at speeds faster than a gigabit. A SSD can source and sink data faster than gigabit. A small RAID stripe of hard drives like you will find in many homes can source data faster than gigabit speeds but many will struggle to sink it. Again, as I said in my opening remarks. Know where your bottleneck is. Spending money on your network won't improve your file copy times if there is consumer grade spinning media involved.

For me, I put an Intel 10GBE NIC in my latest build. I haven't connected it yet. I did buy a multi-speed optic for it so I can hook it up to one of the precious gigabit SFP's on my switch because it is my network and I AM THE BOSS HERE! And, all the cool kids are running fiber. That and the SFP ports are just sitting there empty ;) The Intel X520-DA1 I bought will run at either 1GB or 10GB with the proper optic, or so the release notes say. Not all 10GBE gear will from what I have read.

I also bought a multi-port 10GBE NIC to put in my file server. I'll connect it to the backup server using 10GBE. A full backup is currently taking days so the extra speed will help. I will connect the file server to an edge switch with a few 10GBE SFP+'s when I get one. My desktop probably won't get one of the limited SFP+ ports even though I am the boss here because I am not the STUPID BOSS! My nightly workstation backups won't notice the difference. Nor will netflix. But, everyone connected to the network at gigabit speeds will get as much as they can consume because the server can serve it fast enough and the link from the server to the switch will be able to handle it.

So, for advice. Since you asked ... analyze your workflows and optimize the bottlenecks. Consider point to point links to improve bottlenecks if you need that speed now. Otherwise, wait for consumer 10GBE to stabilize. Writing on the wall is it will happen "real soon now." If I were to bet, I would bet it will be 10GB-T. But, I am a crappy gambler. I usually loose :(
 
  • Like
Reactions: ridney

ridney

Member
Dec 8, 2015
77
33
18
Singapore
I agree wih fractal but why not? we can do it now right? haha don't limit yourself. Buy a 10g switch and be happy rather than using your old trusty 1gb switch but you're still grumpy :D
 

NetWise

Active Member
Jun 29, 2012
599
131
43
Edmonton, AB, Canada
As many have noted, if the issue is actually a disk bottleneck in your server, going to 10gbit won't make file copies any faster, nor will LinkAgg. Even at 10Gbit, if you're pulling some 2GB file off the server from one client, that's going to be single threaded, and likely not even get up to 10Gbit speeds before the file is done, then ramp down.

Don't get me wrong - go 10GbE if you can for the cool factor. But assess the needs first :)
 

Ancients

New Member
Jun 17, 2016
6
0
1
31
Netgears 10gigE equipment (XS708T/XS716T/XS728T/XS748T) is pretty quiet. Besides a random incompatibility I ran into, it has been easy to configure (gui) and has worked for all my uses.

You end up paying $100+ a port though which is gross. The S3300 switches are more reasonable at ~$20 a port and they get you 2x10GigE and 2x10G SFP+. Those also are pretty easy to use and I haven't had an issue with them yet.
 

Pete L.

Member
Nov 8, 2015
133
23
18
53
Beantown, MA
This will really depend on so many things but most people seem to be able to get over 9Gps out of even lower end / el-cheapo Mellanox ConnectX-2 Cards. Of course doing a speed test and pushing actual file server or whatever other type of data through the connections are very different things.

SMB3 in the right environment works awesome but it doesn't really offer any level of redundancy and you need a Windows 8 or Windows 10 Client or Server 2008 / 2012 / 2016 to utilize it and it seems like switch settings can also cause some issues as well. I've seen it work / work very well in my lab but getting it to work in my house even has been difficult and sadly my Synology Doesn't support it just yet.