Looking to for advice on "cheap" 10Gbe cards

[tandrews_ny]

I am building two bigger NAS boxes (ECC, ZFS, etc) and I'd like to link them at a higher speed. I don't need to connect anything but the two servers together.

I found these cards:
Supermicro AOC-STGN-I1S Single Port 10GbE Ethernet Card Adapter X520 672042137480 | eBay
Mellanox Mcx311a-xcat CX311A Connectx-3 Network Card 10gbe SFP Low Profile | eBay

I've also found the ASUS 10GBe ethernet cards which use RJ45. What I'm trying to determine is, are those card cheap because they are just that...cheap or are they reliable useful cards? I just need the speed, nothing else. I'll be transferring a lot of data occasionally and for the price, this seemed reasonable.

Any recommendations would be great, also any buyer bewares would be great.

 

[Mishka]

The ASUS are designed to be cheap, if you want copper I would try to keep an eye out for Dell Intel T540-X2's they have a small 40mm fan on the heatsink which helps a fair bit and more likely to be legit.

 

[StammesOpfer]

Nothing wrong with either of those.... What OS (and version) are you running? Heck you can get some of the OEM branded Mellanox ConnectX-2 10Gbe cards for $10. Plenty of us have these cheap cards. FreeNAS/FreeBSD has better support for Chelsio and Intel cards but the most recent version should have Mellanox drivers as well.

 

[tandrews_ny]

Nothing wrong with either of those.... What OS (and version) are you running? Heck you can get some of the OEM branded Mellanox ConnectX-2 10Gbe cards for $10. Plenty of us have these cheap cards. FreeNAS/FreeBSD has better support for Chelsio and Intel cards but the most recent version should have Mellanox drivers as well.

Am I going nuts, did you say I get them for $10? Why aren't more people going 10GBe? I guess the routers are what limit the appeal? I'll be using freenas, so I should be good then.

Any special cable I need beyond the stand SPF?
I also want to confirm, I can just direct connect the cards right? I don't NEED a router do I?

 

[StammesOpfer]

You do not need a router or switch between 2 computers. You will need SFP+ (not SFP) if the boxes are near each other then the smart/cheap option is SFP+ DAC cable otherwise 2 SFP+ 10G SR optics and some multimode fiber works.

 

[EffrafaxOfWug]

IIRC the Asus cards are using the (comparatively) new Aquantia chips which, last time I read anyway, had windows and linux drivers but support for BSD was lacking, so if you're planning on using freeNAS those are out of the question for the time being.

Yes, you can do direct NIC to NIC connection with just 10Gb cards, but you will likely need to register different hostnames for the 10GbE interfaces/IPs and/or have some fun with routing tables - depends on how you want them talking to each other really.

 

[tandrews_ny]

You do not need a router or switch between 2 computers. You will need SFP+ (not SFP) if the boxes are near each other then the smart/cheap option is SFP+ DAC cable otherwise 2 SFP+ 10G SR optics and some multimode fiber works.

They are going to be literally on top of each other. So SFP + DAC it is.

IIRC the Asus cards are using the (comparatively) new Aquantia chips which, last time I read anyway, had windows and linux drivers but support for BSD was lacking, so if you're planning on using freeNAS those are out of the question for the time being.

Yes, you can do direct NIC to NIC connection with just 10Gb cards, but you will likely need to register different hostnames for the 10GbE interfaces/IPs and/or have some fun with routing tables - depends on how you want them talking to each other really.

GREAT!!! I guess I better go hunt down my cards.

 

[i386]

Why aren't more people going 10GBe?

They (probably) don't know about it and 1gbe works for most use cases (unless they want to play uncompressed 4k+ videos)

 

[tandrews_ny]

Before I commit, any issues with these:

1m SFP+ 10G cable for MNPA19-XTR Mellanox ConnectX-2 10Gbe RT8N1 Network Card | eBay

HP 671798-001 10GB PCI-E Mellanox ConnectX-2 Card High Profile AS# 666172-001 | eBay

I'm having issues finding anything for BSD? Seems like only the ConnectX-3/4 are supported?
- Are Mellanox ConnectX-2 cards supported natively in FreeNAS 11? • r/freenas - Looks like it works out of the box.

My only concern is I need LP but I guess I can just remove the bracket as I don't see any low profile for cheap.

You guys are as awesome as my friend said. Thanks for making me feel like I could do this. I'm a big tech guy but hate wasting money. I'm building a 40TB NAS and need to transfer 26TB to the new build.

 

[StammesOpfer]

Mellanox has not been well supported in FreeNAS historically. As I said before they have added drivers in FreeNAS 11 but I have not verified that as working. If you are willing to take a risk that they won't work with FreeNAS then those should be fine. Also make sure they have some airflow they do run a little hot.

 

[BackupProphet]

I have used Mellanox with FreeBSD for many years, rock stable stuff. The source code for the driver is even included in the FreeBSD source.

 

[Barry]

All good ones. But you should also use good dac cables or transceivers for best performance.