Best (most stable, enterprise proven) 10GB NIC model for passthrough to Linux VMs?

[Dreece]

Just wondering if there is a general consensus on this...

I'm finding Chelsio cards to be ridiculously poorly supported every corner I turn.

I'm thinking those 7 series Intel cards are the ticket, but have no idea nor any experience with them...

any tips gents?

 

[Evan]

Intel 7 series are great cards and lower power but at a cost.

The good old intel X520/X540 is the go for available, priced ok, supported by just about anything you can imagine 10G NIC.
Pick the one you want depending on SFP+ or RJ45

 

[arglebargle]

I'd look at Mellanox cards as well, they seem to be the go-to for a lot of high performance enterprise offerings. I've done some (decidedly non-enterprise) SR-IOV passthrough at home with the 40GbE Connect-X3 and it worked out quite well. They're also cheap as chips, the Oracle part (7046442) sells for $25/ea now and the HP OEM boards can be had for <$35. You'll probably want to cross flash to standard Mellanox firmware when you get them, but that should take all of about 5-10 minutes to figure out.

 

[Dreece]

Thanks gents. I'm juggling Intel x550 and Mellanox X3... preferably after 10GB nics which are currently supported under the Linux-Azure kernel stack, I've noticed they've dropped Chelsio in the latest version.

I've got my eyes on a tasty x710-DA4 too, 40GB of tasty bandwidth, but as you already know, they come at a premium.

If anyone is feeling charitable and has a 'spare' x722 floating around destined for a hobbyist who will put it to awesome use, please do consider PM

 

[TRACKER]

Hello guys,
what do you think about:
Dell Broadcom 57810S Dual Port 10GBE Converged Network Adapter W1GCR
Is there anything specific why these adapters are so cheap (~45-50$)?

 

[Anathematician]

Just wondering if there is a general consensus on this...

I'm finding Chelsio cards to be ridiculously poorly supported every corner I turn.

I'm thinking those 7 series Intel cards are the ticket, but have no idea nor any experience with them...

any tips gents?

I’m probably the odd one out here since most everyone uses Mellanox. I’ve got chelsio T6100 100GBE cards, T625 25GBE cards, and a smattering of older 5/4th gen 10gbe cards in production and haven’t found a system/OS yet without support. I think some of the older cards are no longer supported in the current ESXI but they work fine in a windows server 2016 hyper-v environment.

I think I did have an instance of Ubuntu that complained about old firmware, but that’s my fault for not keeping the firmware up to date.

 

[arglebargle]

I’m probably the odd one out here since most everyone uses Mellanox. I’ve got chelsio T6100 100GBE cards, T625 25GBE cards, and a smattering of older 5/4th gen 10gbe cards in production and haven’t found a system/OS yet without support. I think some of the older cards are no longer supported in the current ESXI but they work fine in a windows server 2016 hyper-v environment.

I think I did have an instance of Ubuntu that complained about old firmware, but that’s my fault for not keeping the firmware up to date.

I think it's mostly cost keeping a lot of us away from the newer Chelsio cards. For $25 I can pick up a dual port 40Gb Mellanox card that's supported by every major OS vendor, that's really hard to beat for home network tinkering.

 

[Dreece]

I’m probably the odd one out here since most everyone uses Mellanox. I’ve got chelsio T6100 100GBE cards, T625 25GBE cards, and a smattering of older 5/4th gen 10gbe cards in production and haven’t found a system/OS yet without support. I think some of the older cards are no longer supported in the current ESXI but they work fine in a windows server 2016 hyper-v environment.

I think I did have an instance of Ubuntu that complained about old firmware, but that’s my fault for not keeping the firmware up to date.

I have a T520CR and a T520-LL-CR... both cards are NOT supported as Virtual Functions in the Linux Azure stack, and both cards also do NOT function as SRIOV VFs under Hyper-V linux guests.

But they work absolutely fine on Windows Hyperviser to Windows VMs as VFs. Chelsio just didn't bother to sign-off on driver development for those who run Linux on Hyper-V hosts. From my discussions with my mate Dave, the T6 series are in the same boat.

The Linux Azure kernel support is what I'm really after.

Mellanox and Intel are kings when it comes to support and general public support. Ever tried sending Chelsio an email? good luck.

 

[Dreece]

The big bug with the Chelsio VF driver is that it attempts to change the MAC, the thing with Hyper-V is that IT wants to be in charge of controlling the MAC, so the Chelsio VF bombs out.... I even went through the driver code and commented out the bit that attempts to set the MAC, unfortunately instead of an error message I ended up with a hard kernel panic hang

I think Chelsio are well aware of it, but there may be something more low-level at play which they are unable to 'fix' at the driver level (just a hunch, because any other manufacturer would have jumped on this years ago, but Chelsio haven't since T4 all the way through to T6)... it is a shame, because I actually do love my T4s and T5s, but I definitely won't be going T6s.

 

[arglebargle]

Mellanox and Intel are kings when it comes to support and general public support. Ever tried sending Chelsio an email? good luck.

I got Chelsio to write back to me exactly once telling me I was SoL when a NIC I'd been using for a couple of days decided on its own to attempt to update its firmware when I inserted the module on a newer linux host. It bricked itself with zero user intervention, it was absolutely ridiculous. I keep one Chelsio T420 around in case I ever need a known supported 10Gb card for pfSense but that's it.

 

[Dreece]

Apparently Mellanox X4 are supported by the Azure stack... http://events17.linuxfoundation.org/sites/events/files/slides/linuxCon_Toronto_Aug21_2016.pdf

So if I want RDMA in a Linux guest running under a Hyper-V host, then I need to buy a Mellanox X4.

and the shopping for a deal begins...

(not only are they costly, they also only do SFP28 (backwards compatible with SFP+ kit) and QSFP28 port connections, so I'm guessing it's time to bring my Brocade into service as core switch to help suck up that bandwidth juice, uffff, so now I have to learn Brocade's CLI too LOL... too much fun)

ps. apparently X3 will also work, but I am unable to find any info on this, anyone else have Mellanox X3 VF'ing from Hyper-V to Linux guests?

 

[Dreece]

Good news, the open-source coders along with Mellanox support have already got Connect-X3s working in the Linux-Azure kernel, step-by-step instructions here: Microsoft/azure-linux-kernel

I'll grab a Sun Oracle X3 then and flash it accordingly. @arglebargle - these are easily flashable correct?

 

[arglebargle]

Good news, the open-source coders along with Mellanox support have already got Connect-X3s working in the Linux-Azure kernel, step-by-step instructions here: Microsoft/azure-linux-kernel

I'll grab a Sun Oracle X3 then and flash it accordingly. @arglebargle - these are easily flashable correct?

Trivially, I've done two of the X3's myself. Recently reposted instructions here: https://forums.servethehome.com/ind...x-oem-firmware-40-usd-each.23947/#post-222930