Questions About Moving to 10gbe

Roman2179

Member
Sep 23, 2013
48
14
8
As the title states, I am looking into moving some and maybe all of my servers over to 10gbe. Partially to help speed up file transfers and partially to simplify some of the cabling since I have a few LAGG groups that I would like to get rid of. Since I am complete newbie when it comes to anything 10gbe/fiber, I am hoping that you guys can help me and save me some money on trial and error.

To start things off, here are the servers:
FreeNAS
Supermicro X9SCM-F
Intel e3-1230 v2
32gb RAM
12x3TB WD RED
6x2TB WD RED
Will be upgrading to 9.3

ESXi 5.5 Server #1
Supermicro X9SCM-F
Intel e3-1230 v2
32gb RAM
Quad Port 1gbe NIC

ESXi 5.5 Server #2
Supermicro A1SAM-2750F
32gb RAM
Quad Port 1gbe NIC

So the two servers I would really like to get on 10gbe are the FreeNAS server and the first ESXi server since I move the most files between those two. I would really like to be able to speed up the archive process. Ideally, if I can get a switch that has four 10gbe ports for a reasonable price, I would like to move the second ESXi server and my workstation to 10gbe as well.

From my research so far, it seems like the Intel x520 is an ideal choice for NICs as far as driver support goes but those get expensive fast if I were to get four of them. It seems like the Intel X520 is a good choice for FreeNAS but seems to have driver issues in some instances. Chelsio t3/t4/t5 cards work with FreeNAS as well but they don't seem to be any cheaper than the Intel cards but look like they are the preferred card by iXsystems. As far as the ESXi server go, can I use something like Connectx-2 EN cards since they go for about 35 on ebay or Brocade 1010/1020 cards?

Next decision is the switch. I was looking at the MicroTik CRS226-24G-2S+RM since it was in the price range I am looking for and satisfies the criteria of switching two servers to 10gbe but leaves zero room to expand. With some reading on the forums here, the Netgear ProSafe GSM7328Sv2 seems like a good choice and gives me the option of adding the other servers later on. Are there any switches that have four 10gbe SFP+ ports that are near close the price range of the two switches I have listed?

Finally, the transcievers, I have no idea which ones work with what switches and cards. Do I need to buy the 60+ dollar ones for the Intel cards or can I use the $18 ones from the the fiberstore?

Thanks for the help in advance!

EDIT:
Thinking of picking up a couple of these Connectx-2 en cards on Ebay. Thoughts? Also what transceivers work with these?
 
Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,046
1,583
113
CA
NetGear has a 4 port 10gb SFP switch with iirc 24 1gigE ports ~600$ which I think is a great deal for all you get, and would have gone that way myself had I not got their 14 Port 10gige switch and a LB4M (that im selling to get the netgear to make life easier).

I was going with Mellanox, direct connect cables, cat6, and fiber and decided it just got too messy for home/lab/etc use, and am ditching most of my non Cat6 stuff, fiber staying for switch connections.

I'm also using the X520 Intel cards. Those + NetGear make life about 10x easier to setup and get going.

If you want to do more then maybe the MEllanox, etc is for you, but I decdied if I'm going that way I'm going 40gig eventually :D LOL
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,971
4,932
113
Check FreeNAS and those Mellanox cards for compatibility.
 

Roman2179

Member
Sep 23, 2013
48
14
8
I was thinking of using the Mellanox cards for the ESXi servers and maybe the workstation. I doesn't look like FreeNAS will support Mellanox cards until 10.x. I think the safe bet for FreeNAS is an Intel x520, pricier but the seems to have the least issues.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,046
1,583
113
CA
Yes, the Intel is by far the easiest and fastest to setup solution :)
 

neo

Well-Known Member
Mar 18, 2015
672
362
63
For 10 GbE - Chelsio cards seem to get the most attention from FreeBSD developers. I've used almost every brand card, and do prefer the Chelsio.

As a cheap plug, I do have a T420 for sale on my other thread.
 

Hank C

Active Member
Jun 16, 2014
644
66
28
this will be my thread to see. I am also in the same situation. Looking to get either all 10gb on storage traffic or getting just 10gb on just the storage server and 1gb for the storage clients
 

Roman2179

Member
Sep 23, 2013
48
14
8
Any thoughts on the the Connectx-2 cards for the ESXi servers? Looking to go as cheap as possible for least the ESXi server #2 since it has a PCIe 8x slot that is being used by the RAID card and a PCIe 4x slot available for the network card which means I would have to cut the card down to 4x. Would rather cut a ~$35 dollar card vs modifying a $300+ motherboard.
 

Entz

Active Member
Apr 25, 2013
269
62
28
Canada Eh?
The Connectx-2 EN cards work great in ESXi . That is what I am using (5.5) and they are dirt cheap. Not sure how well they do cut though..

With the move to 10GBe is the raid card even still required (i.e. All storage on the NAS/Fileserver)?
 
  • Like
Reactions: T_Minus

Hank C

Active Member
Jun 16, 2014
644
66
28
but the sfp+ switch are still on the high side though...even if the nic are dirt cheap
 
  • Like
Reactions: T_Minus

seang86s

Member
Feb 19, 2013
147
12
18
I picked up the D-Link SmartPro DGS-1510-28X which is a 24 port 1GBe switch with 4 SFP+ ports. I'm using it for a 3 node ESXi cluster and 1GBe ports for a few desktops and miscellaneous home stuff. Got it for $550 from Provantage in March, but I see the price is $30 higher now, after a $20 instant rebate. I'm using 1.5 meter twinax cables to connect the machines I got off eBay for cheap.

Here's the link: PROVANTAGE: D-Link Systems DGS-1510-28X SmartPro 24-Port Gigabit Switch with 4 10GbE SFP+
 
Last edited:

Entz

Active Member
Apr 25, 2013
269
62
28
Canada Eh?
Very true, switches are the problem (and optics) but it is a pretty critical part and something you build your network around and don't change very often. That being said, you can get away with a single CRS226 for 3-4 servers at the expense of some bandwidth provided you have the PCI-e slots for extra cards in the file server..

Good price on the DGS-1510-28X!
 

Roman2179

Member
Sep 23, 2013
48
14
8
I ended up getting a couple Connectx-2 EN cards for 33 a piece. Now the question is, which transceivers work with these?

At this point, could I got without a switch if I got a 2 port card for the FreeNAS server and just connected the two ESXi server to it directly? I think I want to hold out and pick up the D-Link switch. Has all of the features I would want and the price seems pretty reasonable compared to everything else I've seen. Power usage for it seems pretty good as well.

EDIT:
Looks like a need a couple of transceivers for a Chelsio T420 as well. Thanks neo!
 
Last edited:
  • Like
Reactions: neo

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,046
1,583
113
CA
The seller took $15 for the Arista optics in case anyone is curious.
He countered my offer with $15 t00, btw. Declined my low-ball for fun offer on 25 units, countered for 15 for 10 units. So I don't think he's going under $15 unless you're going to get a lot. My last offer, 13$ well see :) LOL!! I love negotiating on ebay, its fast and simple LOL.

And they rapidly declined the $13, and didn't even re-counter with $15. Some sellers, I wonder if they get how ebay works. I'm not going to BIN For $25, and the $15 is off the table, and I'm out of offers... LOL Ah-well

$12-15 is going rate what I paid for past units, so I'd say that's the best deal they're gonna do :)
 

Roman2179

Member
Sep 23, 2013
48
14
8
Chelsio SM10G-SR transceivers seem to be going for about $35 each on ebay, probably a little less with best offers. Has anyone used any other transceivers for T420-CR cards?
 
Last edited:

Entz

Active Member
Apr 25, 2013
269
62
28
Canada Eh?
I have used Mikrotik, Brocade (Fiberstore) and some generic DAC (not sure what it is lol). From what I have seen/read the ConnectX-2 cards do not really seem to care.
 
  • Like
Reactions: T_Minus