Looking for more than GB Ethernet, need advice on faster network setup

[Vlad]

Please help me re-make my home network (I feel the need, the need for speed)

My networking ability/knowledge: slightly better than noob. PC/Hackintosh savvy, built my own PCs for about 20 years now, can set up basic ethernet network (cable modem/router), but networking knowledge is minimal, very good at following directions.

My current setup: 1 file/media server (OpenMediaVault), 1 Hackintosh, 1 HTPC (Win10), 1 Backup PC (Win10), 1 Backup Server (Win10)

Internet: Comcast 75Mbit service

Budget: No more, hopefully, than $500 total

I currently have a wireless 802.11AC setup that's pretty decent for speed (65MB to 85MB/sec), but I'm thinking about going back to Gigabit Ethernet mostly for the speed, but also for the security. I thought to myself, "self, as long as you're looking to go back into wired networking, why not go for something faster than GB Ethernet?"

So many options when it comes to networking. This site/forum BTW is a great source of info & knowledge. So, do I go for 10GB speeds, or try to do link aggregation with GB Ethernet? Not at all concerned with internet download speeds, since I believe any option I consider will give me excellent performance there. My main concern is speed with transferring files internally: PC to PC (I download lots of large files, sometimes as large as 10-15GB, but usually around 5GB, and need to move them from PC to PC).

I'm currently considering 10GB networking, but the switches are expensive, LOUD, & hard to manage for a slightly-better-than-noob like me. What I'd need in a 10GB switch:

- All ports to be 10GB, either RJ45 (?) or SFP+
- Don't need more than 8 ports
- NEED A GUI (did I mention I'm a slightly-better-than-noob with networking?)
- Preferably fanless or at least really quiet
- Solid, reliable, not too concerned with little to no warranty as long as it's reliable

Is a home 10GB setup as simple as:

Cable Modem > 10GB Switch > each PC ???

Other option: Link Aggregation of 2 or more GB Ethernet in each PC/Server + Managed switch ?

Simple as: Cable Modem > GB Managed Switch > each PC with 2 or more GB NICs?

Many suggestions are welcome.

 

[aero]

Perhaps the first thing to note is that link aggregation works by splitting up traffic flows based on a hash calculated from source MAC + destination MAC, src mac + dst mac + src IP + dst IP, or a variety of configurable parameters depending on what switch you have.

What's important about that is it means a single flow (e.g. file copy) will always be limited to a single link (1Gbps). Furthermore, even with multiple flows you may not see a very good distribution of traffic across the links...it depends heavily on the hashing algorithm you have chosen (or what hashing algos the switch supports).

This ultimately means that the best way to increase network bandwidth is higher capacity links, like 10Gb.

Where are your devices going to reside, in relation to each other, that need 10G connectivity? That will dictate what kind of network media you need, and consequently what kind of switch you would need.

You can run 10Gb over cat6 up to roughly 100 feet, but really you should use cat6a for 10Gbase-t which is rated for 100 meters. I really dislike, and would advocate against 10Gbase-t. It takes significantly more power per port on both the switch side, and the NIC side. The NICs and switches will thus run much hotter, and probably need cooling. Also, there are fewer options for switches and NICs that support 10Gbase-t.

If your runs are all in the same room, then passive twinax DACs are a good option. They are limited to ~5 - 7 meters.

Finally, fiber (single mode, or OM3 multimode) is great for longer runs.

So that comes back to the question of where are your devices going to be located?

 

[Vlad]

Thanks aero. I guess link ag will be out of the question then.
4 of 5 of my PC's, including my server, are in my upstairs "Computer room".
The 5th, my HTPC is downstairs in my living room, roughly 50-75' away. Looks like I'll need to consider 10GB or else stay with plain old GB Ethernet.

Perhaps the first thing to note is that link aggregation works by splitting up traffic flows based on a hash calculated from source MAC + destination MAC, src mac + dst mac + src IP + dst IP, or a variety of configurable parameters depending on what switch you have.

What's important about that is it means a single flow (e.g. file copy) will always be limited to a single link (1Gbps). Furthermore, even with multiple flows you may not see a very good distribution of traffic across the links...it depends heavily on the hashing algorithm you have chosen (or what hashing algos the switch supports).

This ultimately means that the best way to increase network bandwidth is higher capacity links, like 10Gb.

Where are your devices going to reside, in relation to each other, that need 10G connectivity? That will dictate what kind of network media you need, and consequently what kind of switch you would need.

You can run 10Gb over cat6 up to roughly 100 feet, but really you should use cat6a for 10Gbase-t which is rated for 100 meters. I really dislike, and would advocate against 10Gbase-t. It takes significantly more power per port on both the switch side, and the NIC side. The NICs and switches will thus run much hotter, and probably need cooling. Also, there are fewer options for switches and NICs that support 10Gbase-t.

If your runs are all in the same room, then passive twinax DACs are a good option. They are limited to ~5 - 7 meters.

Finally, fiber (single mode, or OM3 multimode) is great for longer runs.

So that comes back to the question of where are your devices going to be located?

 

[pyro_]

You could always look at putting 4 of the 5 pcs on 10gig and leave the fifth on gig. Getting a gig switch that has 4x10gb ports would deff be in the budget you listed

 

[Vlad]

How about 2 of these, since I'd like to have as many of my PCs connect at 10GB as possible:

https://www.amazon.com/MikroTik-CRS...1442964255&ref_=sr_1_1&sr=8-1&tag=servecom-20

How would I even set this up, using my cable modem into one of the GB ports in either switch, then the SFP+ port to 4 of my PC's, which would also have a 10GB card (Mellanox probably)?

You could always look at putting 4 of the 5 pcs on 10gig and leave the fifth on gig. Getting a gig switch that has 4x10gb ports would deff be in the budget you listed

 

[pyro_]

You would end up with only two PCs on 10gb with that setup as you would use one of the two ports of 10gb on each switch to link the switches together. You would be better off to have one switch with 4x10gb ports rather than two switches with 2x10gb ports so that all of the 10gb ports could be used on PCs

 

[aero]

I'm assuming your network currently is cable modem -> router -> switch (or switch ports built into the router). It would instead be cable modem -> router -> switch (with 1Gb and 10Gb ports).

I wouldn't recommend those Mikrotiks because they only have 2 10Gb ports each, and you'd probably want to link them together at 10Gb, which would eat up 1 port on each...leaving you with only 2 usable 10Gb ports for hosts.

If you don't expect you'll have more than 4 devices operating at 10G, then a gigabit switch with 4-10Gb ports is probably the most cost effective way to go.

edit: if you're planning for > 4 10Gb capable devices, then one of those Mikrotiks or something like it + higher port count 10Gb switch makes the most sense.

 

[ttabbal]

An HTPC probably won't have a lot of use for >1gb anyway... My TV boxes are Raspberry Pi units that only do 100Mb and they don't have issues.

I'm not aware of much in the 10Gb world that anyone would call quiet though. Are you willing to hardware hack a bit to quiet them down? Replacing fans and such can go a long ways.

My first thought is an LB6M. Nobody would call it quiet, but ~$300 for 24 SFP+ ports and 4 1Gb ports, not a terrible way to go. Would need to add some cards and DACs, but would still come in under the budget. The LB4M thread has some tips for quieting things down, I imagine the LB6M one does as well, and most sound mods should be about the same between them. They are managed switches, but even CLI is pretty easy to make a "dumb" switch out it. And I think they generally have a basic web UI on them.

 

[Vlad]

I've seen a lot of the info posted here on the forums about the LB6M and while hardware-wise it would seem to have what I want, and I'm not opposed to swapping fans, I've pretty much ruled it out since it doesn't seem to have GUI, only CLI. Too bad, since I really like the prices I've seem on eBay. Plus there also seem to be questions about whether some of the commands/parameters actually work. Anything similar hardware-wise to the LB6M but with a web GUI?

An HTPC probably won't have a lot of use for >1gb anyway... My TV boxes are Raspberry Pi units that only do 100Mb and they don't have issues.

I'm not aware of much in the 10Gb world that anyone would call quiet though. Are you willing to hardware hack a bit to quiet them down? Replacing fans and such can go a long ways.

My first thought is an LB6M. Nobody would call it quiet, but ~$300 for 24 SFP+ ports and 4 1Gb ports, not a terrible way to go. Would need to add some cards and DACs, but would still come in under the budget. The LB4M thread has some tips for quieting things down, I imagine the LB6M one does as well, and most sound mods should be about the same between them. They are managed switches, but even CLI is pretty easy to make a "dumb" switch out it. And I think they generally have a basic web UI on them.

 

[j_h_o]

A few random thoughts:

1. SMB 3.0 supports multichannel. Multiple NICs per machine attached to 1Gbps switch will still result in some link aggregation from your Windows 10 boxes.
The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0
So what was discussed above isn't ENTIRELY true, certainly based on your mix of devices.

2. Are you able to (re-)run cables between all of these devices?

3. Do you really need more throughput between your boxes? Could you "just" place your file server on 10Gbps, and leave other nodes on 1Gbps? Then the Mikrotik CRS226 might be a good option. And/or other switches with 4 SFP+ ports might also work.

4. CAT5e will handle 10GBase-T at shorter lengths. How long are the runs of CAT cables right now?

I personally have my servers (some 2012 R2) hanging off at 10Gbps SFP+/DACs, then my workstations either aggregated or using single Gbps links. Looking at 10GBase-T switches now, because I can't easily rewire all parts of my house, so I can't easily "upgrade" to fiber for cheaper 10Gbps connectivity.

 

[Vlad]

1 - Assuming I could get link ag. to work correctly, still not as fast as 10GB, most likely & not a huge price diff assuming I can find the right 10GB switch.
2 - Yes.
3 - Need, no. Want, yes. I hate waiting even 3 minutes to transfer a large file. I can handle 30 seconds
4 - Current CAT5e cables are 15', I could go as short as 10' since the 4 PCs in my computer room are pretty close to each other. The downstairs HTPC though is at least 50' away so that might need to stay on GB.

The D-Link DGS-1510-28X looks interesting. Wish it was at least $100 cheaper ($500+ on Amazon).

A few random thoughts:

1. SMB 3.0 supports multichannel. Multiple NICs per machine attached to 1Gbps switch will still result in some link aggregation from your Windows 10 boxes.
The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0
So what was discussed above isn't ENTIRELY true, certainly based on your mix of devices.

2. Are you able to (re-)run cables between all of these devices?

3. Do you really need more throughput between your boxes? Could you "just" place your file server on 10Gbps, and leave other nodes on 1Gbps? Then the Mikrotik CRS226 might be a good option. And/or other switches with 4 SFP+ ports might also work.

4. CAT5e will handle 10GBase-T at shorter lengths. How long are the runs of CAT cables right now?

I personally have my servers (some 2012 R2) hanging off at 10Gbps SFP+/DACs, then my workstations either aggregated or using single Gbps links. Looking at 10GBase-T switches now, because I can't easily rewire all parts of my house, so I can't easily "upgrade" to fiber for cheaper 10Gbps connectivity.

 

[pyro_]

Also look at the tp-link t1700g-28tq switch

 

[Vlad]

Or what about this puppy:

TRENDnet | Products | TEG-30284 | 28-Port Web Smart Switch with 24 x Gigabit ports and 4 x 10G SFP+ slots

NewEgg has these for $340

Are they any good? Looking for reviews . . .

Can someone look at the emulator to see if there are any glaring missing options? I don't need anything fancy, I mainly want speed & reliability.

 

[Drewy]

Don't be put off by the lb6m's lack of a web gui. If you aren't planning to do anything complicated with it I.e. You are just using it as a straight 10gb switch, it's just plug and play. No need to mess with it via the Cli.

 

[Pete L.]

Sorry if I missed this but I didn't see a whole lot of talk about the Network Cards Needed for 10G if you go with Copper Based 10G Base-T those cards are quite pricey, yes you can absolutely get Fiber Based Cards like the Mellanox ConnectX-2 Cards for very short money BUT they do not support the latest / greatest so getting more than say 4gb/sec transfer might be a little difficult. So no you'd be looking at something like the Intel Cards or the Mellanox ConnectX-3 cards which can be upwards of $100+ each. Then you also have to consider the Fiber / SFP+ Modules that are compatible with the cards / switches (can be an issue for Vendor Locked Cards like Intel / Cisco) or DAC Cables.

Either way if you are going to go with Fiber / SFP+ or DAC I would strongly suggest checking out Fiber Store www.fs.com they have great pricing, ship quick and gaurantee their compatibility (just make sure you let them know the device(s) you will be connecting).

That said the new TP-Link 10G Switch does seem very promising especially for the price point that it is at. Ubiquiti is also going to be coming out with some new 10G Switches that might be worth taking a look at as well.

 

[aero]

Natex has Intel X520-DA1 cards for $28. They can operate at full line rate 10Gbps no problem.

 

[ttabbal]

Not sure why anyone would say that a connectX2 can't get >4Gb. I have a pair here that get 9.8Gb with default settings. I plugged them in and ran iperf. Random ebay optics and OM3 fiber.

The Intel cards from Natex are great as well, but are picky about modules. The Linux drivers let you override that, not sure about others.

 

[aero]

Agreed, I've never used connectx2, but from looking around at various benchmarks they are also fine to hit full 10Gbps line rate.

 

[wildchild]

Have both intel (omnios) and connectx-2 en (esx)...
Directly connected using a cisco dac..
All from natex, running at 9.8 - 9.9 no problem at all

 

[Vlad]

OK, so what kind of cable would I use to connect the ConnectX2 in my PC or server to the switch's SFP+ port?
I'm thinking of getting the Trendnet TEG-30284 since I like the price and I don't need anything fancy.

Also, I may need to buy a new mobo for my main PC, since it only has 1 PCIE x16 3.0 slot & 3 PCIE x1 slots. Will the connextX2 run in a PCI 3.0 x4 slot?