Aside from ATTO, any PCIe SFP28 or QSFP+ (40 or 56) PCIe NICs for MacOS ...?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

TrumanHW

Active Member
Sep 16, 2018
253
34
28
I've seen SolarFlare, but they're limited to 10GbE, as is literally EVERYONE except ATTO if I'm not mistaken.

The ATTO stuff is so expensive, that it's cheaper to use a PCIe to TB3 enclosure than ATTO if one exists.

Better still? Someone shows up with competition to ATTOs. $2,000 for a TB3 to SFP28 is nuts.

Thanks.
 

wifiholic

Member
Mar 27, 2018
45
51
18
37
I was going to suggest Chelsio, as that's the kind of NIC I used back when I was daily driving a heavily upgraded Mac Pro 2010. Unfortunately, it appears that they haven't updated their macOS drivers since Catalina, so I'm afraid that's probably not a viable option.
 
  • Like
Reactions: TrumanHW

TrumanHW

Active Member
Sep 16, 2018
253
34
28
Thanks buddy ... I have a few Chelsios, but, I'll double check and see which speeds they support.
Sometimes their older drivers still work (or aren't even required) on later OS.
 

reasonsandreasons

Active Member
May 16, 2022
130
87
28
Atto's the only game in town from a driver perspective (even supporting Apple Silicon), but the FastFrame NQ41 and NQ42 are just rebranded Mellanox ConnectX-3 cards. I think the same is true for the FastFrame N311 and the ConnectX-4. There was a thread on MacRumors where a group of folks were trying to flash generic ConnectX-3 cards with Atto's firmware; unfortunately they stalled out late last year as they needed some firmware from the Atto cards, which are obviously rare.
 

reasonsandreasons

Active Member
May 16, 2022
130
87
28
Apologies for bringing this back from the dead, but I've found something potentially exciting on this front. Sonnet has a new 25Gbe card with DriverKit support, the Twin25G. I was looking through the manual and saw that Windows reports the card as a standard Mellanox ConnectX4-LX, specifically the MCS4121A-ACAT. From what I can see they look basically identical besides the Sonnet branding. Provided they're not enforcing any lock to their specific cards, this might be a really good option for greater-than-10G networking--MCS4121A-ACAT's run for $70 on eBay. If anyone has a Mac with PCIe slots kicking around it's worth a shot.

Update: On some more digging they explicitly tell you to download the Mellanox OFED drivers for Windows. I think a PCIe Thunderbolt enclosure is in my future.
 
Last edited:
  • Like
Reactions: blunden

blunden

Active Member
Nov 29, 2019
440
136
43
Apologies for bringing this back from the dead, but I've found something potentially exciting on this front. Sonnet has a new 25Gbe card with DriverKit support, the Twin25G. I was looking through the manual and saw that Windows reports the card as a standard Mellanox ConnectX4-LX, specifically the MCS4121A-ACAT. From what I can see they look basically identical besides the Sonnet branding. Provided they're not enforcing any lock to their specific cards, this might be a really good option for greater-than-10G networking--MCS4121A-ACAT's run for $70 on eBay. If anyone has a Mac with PCIe slots kicking around it's worth a shot.

Update: On some more digging they explicitly tell you to download the Mellanox OFED drivers for Windows. I think a PCIe Thunderbolt enclosure is in my future.
Good find! Be sure to let us know if you end up trying it out. :)
 

unphased

Active Member
Jun 9, 2022
148
26
28
Provided they're not enforcing any lock to their specific cards, this might be a really good option for greater-than-10G networking--MCS4121A-ACAT's run for $70 on eBay. If anyone has a Mac with PCIe slots kicking around it's worth a shot.

Update: On some more digging they explicitly tell you to download the Mellanox OFED drivers for Windows. I think a PCIe Thunderbolt enclosure is in my future.
So I've been running MCX3 hardware on my windows/linux setups, there are some quirks with it, but I can work with them. High speed ethernet has still proven elusive for me on macOS, because every Aquantia AQC107 based 10G card i've tried in my thunderbolt enclosure fails to work on apple silicon, I know there may be some newer aquantia based ones that have drivers and supposedly work, but I'm not interested in any further investment into 10Gbit connectivity... gotta go faster to make it worthwhile.

Hopefully it will be possible to use a thunderbolt enclosure and one of these CX4 cards with this driver. *however* I note from your manual link it shows only Mac Pro 7,1 (2019) compatibility, which is not an apple silicon part. So this gives me pause.
 

reasonsandreasons

Active Member
May 16, 2022
130
87
28
I think it’s just that the Mac Pro is the only Mac natively compatible with it as it’s the only one with internal PCIe slots—they have a Thunderbolt 25Gbe solution that I think is just that card in an enclosure.

I’m on vacation right now but I’m hoping to have a CX4 and enclosure waiting when I get home. Happy to be the guenia pig here as I’ve got an alternative use for the enclosure and CX4 if this goes south.
 
  • Like
Reactions: blunden

unphased

Active Member
Jun 9, 2022
148
26
28
Hey I got a MCX455A-ECAT and after reconfiguring it for Ethernet from IB it works out of the box in Ventura on my M1 Max Macbook. Needs a thunderbolt card dock contraption (eGPU unit), of course. But it's great since the driver comes with macOS now.

The problem is these native Apple drivers, the MLX5 driver (but also the other ones, there seems to be chelsio and intel support too!) has a MTU max of 2034. It seems very arbitrary. It looks like my setup (at least iperf3...) can receive 9000 byte jumbo packets at 20Gbit/s but I can only push 16.4 or so Gbit/s due to the 2034 max MTU.

Doubt anyone here is knowledgeable in macOS kernel driver hacking so I'll pursue this in other venues.
 
Last edited:

reasonsandreasons

Active Member
May 16, 2022
130
87
28
Okay, that's fascinating. These are Apple-developed kernel drivers? It's wild that they're in there since they've never shipped anything above 10Gbe in a product--I wonder why they developed those. I'll test the MCX4121A-ACAT with those when I get it in addition to the Sonnet drivers, which will (hopefully, at least) be less limited in that respect. I'd be really interested to hear if you're able to lift the MTU cap on the native drivers, as well as anything else they're able to find when they're in there. What kext are these under?
 

unphased

Active Member
Jun 9, 2022
148
26
28
Yeah they're drivers written by Apple apparently. Most of the knowledge on this stuff can be found in the MacRumors forums. You can just search some of these keywords and you will find the topics.

AFAIK the first party Apple 10GbE devices are Aquantia (now Marvell?) AQC107S or some such. I have actually two similar ones (old asus areion 10G card, and a M.2 form factor adapter (Innodisk EGPL-T101, oh maybe this one is the less well-supported AQC113, not sure)), but have not had luck getting them to work in my testing. Though, things may be different with the newer operating system.

Anyway one of the bigger threads is here: Mellanox ConnectX-3 40 GbE using ATTO FastFrame macOS driver
 
  • Love
Reactions: reasonsandreasons

reasonsandreasons

Active Member
May 16, 2022
130
87
28
Just got home and unpacked the card and the Thunderbolt chassis! Bad news is that the Sonnet drivers don't work with the stock MCX4121A-ACAT. I imagine an NVRAM flash would take care of that, but I haven't heard of anyone with a Twin25G kicking around.

Good news is that it appears to be working with the built-in macOS drivers. I need to update to Sonoma which should hopefully iron out some issues (reporting is weird right now--the ethernet interfaces appear in settings but not system profiler). More soon once I actually get it wired into the network.
 

reasonsandreasons

Active Member
May 16, 2022
130
87
28
Confirming that Sonoma ironed out a meaningful number of issues in terms of reporting--both ethernet interfaces appear correctly in system profiler and system settings. I'm also able to edit the interface type and MTU, though it's still capped at 2034. Hopefully there will be some progress on lifting that.
 

unphased

Active Member
Jun 9, 2022
148
26
28
i find it hilarious that a SCP file transfer bottlenecked to around 250MB/s and rsync was even worse, not sure what rsync was doing. But i have samba already configured for multipath which helped with speed from macOS a lot and I got what appeared to be line speed roughly 2GB/s from macOS to NVMe on my Linux box. Really stoked about this.
 
  • Like
Reactions: reasonsandreasons

reasonsandreasons

Active Member
May 16, 2022
130
87
28
That's great to hear--realistically all I really want is to be able to make good use of the NVMe mirror in my TrueNAS Core box. I finally wired things up to take advantage of the 10G port on my Mac Studio and I'm discovering that my HDD pool can hit 700 MB/s with three mirrors. I'd like to put some distance between the pools.
 

unphased

Active Member
Jun 9, 2022
148
26
28
One issue I'm running into with testing the fiber prior to running my fiber across the house is that at least with one of my CX3 cards, it works great off the DAC, but connecting the fiber transceiver causes it to drop off the network adapters list in Ubuntu. Pretty certain this problem is on the CX3/linux side of things. currently troubleshooting.
 

unphased

Active Member
Jun 9, 2022
148
26
28
Aha. Perfect. So yeah on my Mac, I had to select Manual configuration in order to push the MTU from 1500 default to 2304. So I had that set during my DAC testing. Which meant it was manually set to 40GBase-CR4, for the DAC. Then this didn't work on fiber! So, once I switched it back to auto, it connected at 40GBase-SR4 on my fiber transceivers, and I could then set it manually again to get the extra speed from 2304 MTU. Brilliant... now i can confidently run my fiber!
 

unphased

Active Member
Jun 9, 2022
148
26
28
Right ok so it's back to the same CX3 issue where after having it disconnected for a while, on the linux side it will go into some kind of state where I need to physically unplug and replug the transceiver or reboot the machine before it will connect again. This is not going to be good enough, so I will be evaluating

- flashing ethernet-only firmware to the cx3
- upgrade that side from cx3 to cx4 as well

my cx3's can still be potentially useful for connecting any always-on systems, I suppose...

update: sorry I can't keep track of all the threads in which I'm posting about this setup but I did more or less end up with a nice working Mac setup.

after some fiddling, the link was mostly reliably coming up. So I can just plug in the usbc going to my egpu enclosure hosting the cx4 connected via fiber now to my sx6036 and I get 87W PD and up to 20 or so gbit on this 40Gbit equipment. There are some aforementioned MTU shenanigans. Make sure that config is matched up with the other side. I found a capacitor broken off on this cx4 card. I got a lesser (not 100G and I believe it can't be flashed to... will fetch the model number later) cx4 card in from eBay but haven't tested that out. Anyway it proves affordable 20G on usbc is possible with Apple silicon MacBook, and that's pretty great.
 
Last edited: