Need Home 10 GigE Switch Recommendation Please

[svtkobra7]

I need your assistance on which 10GigE switch to purchase, please. A certain constraint makes this more complicated than usual (I believe). Objective, Constraint, Requirements outlined below.

Objective

• Provide 10GigE connectivity between (a) virtualization server, (b) FreeNAS server, and (c) 1 workstation, where (a) and (b) are colocated in a home server closet and (c) is in my home office wired via Cat5E.

Constraint

• I would opt for a 1000BASE-T Fiber solution, if not for that single workstation in my office wired via Cat5e. There is no way to run fiber there as it is a condo.

Requirements

• Note the below requirements are representative of my knowledge at the moment.
  
• 2 1000BASE-T Fiber ports (preferable to Copper as I believe they are more cost effective) to link switch to (a) virtualization server and (b) FreeNAS server. Note: No network interface purchased yet for either so I can go either fiber or copper route.
• 1 10GBASE-T Copper port to link to (c) 1 workstation. I should note that I do have 4 Cat5e ports available to that workstation. Could they be aggregated somehow to provide 4 Gbps? If an option I don't know how to do so as Win10 doesn't support NIC teaming. Ideally I end up with 10GigE over Cat5e though. Note: New 10GigE network interface will replace the Gigabit NIC currently being used.
  
• 1 1000BASE-T Copper port to connect the 10GigE switch to existing Cisco SG-300-20T switch (which connects to all Ethernet ports in my condo). Note: If a solution is proposed that allows me to remove this switch from play, that would be fine. While Layer3 capable, it is currently being used in Layer2 mode.

Thank you in advance for your help.

 

[pyro_]

Netgear S3300, Netgear 708E or T, ubiquiti 16xg either Unifi or Edgeswitch version

 

[ServerSemi]

I got a hp 2920 24 port with 2x 10G copper expansion cards. You could also get SFP+ expansion cards for the same switch if you like or one of each (total of 2) Couldn't be happier about my purchase I got enterprise grade switch for about $1,000 including the expansion cards.

 

[svtkobra7]

Is this about the most cost effective way to deploy this? And what would you recommend, going with #1 or #2 below?

#1 $560 Netgear ProSafe S3300-28X http://r.ebay.com/xOl5Xm (allows me to take the Cisco SG-300-20T out of play)
#2 $550 or Ubiquiti ES-16-XG EdgeSwitch Ubiquiti ES-16-XG EdgeSwitch 10Gbps 16 Ports Managed Aggregation Switch | eBay
$108 Intel X540-T2 http://r.ebay.com/m4jsvp (for workstation)
$60 2 x Mellanox NEW MNPA19-XTR 10GB MELLANOX CONNECTX-2 PCIe X8 10Gbe SFP+ NETWORK CARD - PULLS (1 for FreeNAS server + 1 for virtualization server)

Those Mellanox cards need transceivers right? Also, what do you recommend for fiber cable?

 

[pyro_]

Either 1 or 2 will work equally well, will depend on how much addition 10gb expantion you want to have available and how badly you want to take the cisco out of play. Yes the mellanox cards will need either DAC cables or transceivers, which one is best used will depend on the distance between the switch and the servers and also what switch you pick up

 

[svtkobra7]

Either 1 or 2 will work equally well, will depend on how much addition 10gb expantion you want to have available and how badly you want to take the cisco out of play.

• I'm somewhat surprised to hear you say they will work equally well as I hear a lot of praise sent Ubiquiti's way and I've never really been impressed by Netgear. I don't doubt your word though.
  
• What would your recommendation be?
• To the extent that I want to do this as cost effectively as possible, and I do, I can sell the Cisco switch for ~$100 on eBay, but almost hate to go that route and then determine I need more 10GigE switching ability in the future.

Yes the mellanox cards will need either DAC cables or transceivers, which one is best used will depend on the distance between the switch and the servers and also what switch you pick up

• Distance to switch = approximately 1 meter
• Can you provide guidance on how the DAC cables / transceivers differ between #1 and #2?
  
• Is that Mellanox card OK? It isn't highly recommended by the FreeNAS forum (like the Chelsio) as it hasn't been in use for as long. The proposed items are a result of a very, very brief search on ebay, etc. Until a few moments ago I didn't even realize there was a difference between SFP and SFP+
• If you have any better ideas, please let me know and thanks for your help.

 

[pyro_]

I have both the unifi 16XG and the netgear 708T and have found both of them work equaly well. Feel free to do a search on the forums for both of the switches as there is a ton of info for both of them on here to base your decision on. Main reason to go with the S3300 would be to have only one switch and get rid of the cisco. It will mean that you only have one 10gbe port left however so it will depend on what you want do in the future. Granted 10gb switches will most likely get cheaper in the future as well so that is something to consider.

One thing to note with the ubiquiti switches is that they are very picky about DAC cables where the netgear will probably not be. As DAC cables are generally cheaper than fiber for short distances they are preferred. There is a thread on here however with what cables have been confirmed as working with the Ubiquiti switches which can help if you go that route.

I have not had any issues with the Mellanox cards which I have used on my windows desktops and servers however I dont have any experience with FreeNAS so would have to defer to others on that one.

 

[svtkobra7]

It will mean that you only have one 10gbe port left however so it will depend on what you want do in the future.

Thanks for the thorough reply. Your comment implies that you are counting using 1 of 2 ports on the Intel card and 1 port each on the 2 Mellanox cards. To that end, is there any advantage to:

• Using both ports on the Intel card?
  • I don't think these can be teamed at the moment, if ever (at least in Win10).
  • There isn't any real redundancy provided right?
  • I suppose using both ports might allow for connections to different vlans or subnets or something (not really of any use to me).
• Go with dual port Fiber cards?
  • It may not even be a relevant point to make, but there would already be GigE redundancy for both servers.

Thanks again. Your replies have been most helpful in serving as a jumping off point for further research, but I'm really surprised and thankful that you were able to assist with architecting a networking hardware schema so quickly.

 

[pyro_]

Honestly teaming in most cases will not gain you much especially on a destop machine as you will have to have connections to multiple machines at the same time to take advantage of it. the max speed of a connection from one machine to another will still only be 10gb even if you have a team of 2+ ports setup

One other thing to consider is the speed of the disk system that you will need to have to drive a 10gb network and it would need to be even faster if you wanted to have multiple 10gb streams going at once.

depending on what you are trying to do with the servers you could also consider a direct connection between the two servers with the additional 10gb port on the nics, this could work well if you are doing things like live VM transfers between servers or HA file systems

 

[svtkobra7]

depending on what you are trying to do with the servers you could also consider a direct connection between the two servers with the additional 10gb port on the nics, this could work well if you are doing things like live VM transfers between servers or HA file systems

I thought this was possible ... and makes sense, but is the switch going to be a bottleneck with a 128Gbps backplane (Netgear S3300-28X)? Or do you suggest that this be done as that second port (direct connection) wouldn't allow traffic coming through the switch to slow down that direct connection? If the later consideration, wouldn't it only be relevant if you are saturating the first port, which with 10GigE I imagine would take a bit of effort. I don't know how much effort, but I"m really, really looking forward to seeing transfer speeds above 113 MB/s!!!

Single Port Scenario:

• Port 1/1: FreeNAS Server Card <=> DAC <=> Switch <> DAC <=> Virtualization Server Card

Dual Port Scenario:

• Port 1/2: FreeNAS Server Card <=> DAC <=> Switch <> DAC <=> Virtualization Server Card
• Port 2/2: FreeNAS Server Card <=> DAC <=> Virtualization Server Card

 

[pyro_]

with any proper 10gb switch you will not have issues with the backplane being fast enough.

Mainly how you connect them will depend on the ports you have available on the switch. if you only have 4 10gb ports on the switch then you are limited to one connection for each of your 3 machines with room for one more and you would then need to direct the two servers if you had a need to have a second 10gb connection between them. If you went with the Ubiquiti switch then i would connect everything through the switch and use vlans to split up the connections

 

[fractal]

If you are willing to take baby steps, you could do what I did. I bought a pair of mellanox 10G cards with a DAC cable for under 50 USD. I put one in my NAS and the other in my VM server. And voila, instant data network at 10G separate from everything else for a small investment.

Using text-art, you get

NAS 10G -> DAC -> Virtualization server 10G

NAS 1G -> cat5 -> switch
Virtualization server 1G -> cat5 -> switch
Workstation 1G -> cat5 -> switch

I have since replaced the 20 dollar single port mellanox cards with 50 dollar dual port cards so I can do things like connect my workstation directly to the NAS but those ports are still unpopulated.

Four x 7200 RPM drives in the NAS peak between 2 and 3 gbps to vmware so it was worth it for me. It will take a beefier nas to saturate 10G