ATTO 3102- Thunderbolt Mellanox ConnectX-4 Lx (Dual Port 25GbE) - $189 or less

[b0bb]

Wanted to echo the "mlx5 is terrible" sentiment. I have a ConnectX-4 in an external enclosure and in my testing it performs worse than my built-in 10 Gigabit port; the ConnectX-4 reads at 641.9 MB/s from a NVMe mirror in my main server where the built-in NIC does 820.1 MB/s. Speeds don't indicate that I'd benefit from a 25Gbe connection regardless, but still a disappointing result.

Stock Connectx-4 does not play well with external enclosure's 4-lane PCIe.
I get about 15MBit/s with a Connectx4-LX in an OWC enclosure.

Mellanox's linux driver does complain the 4-lane setup is suboptimal.

Atto's TLN3 is a different matter entirely, with the same mlx5 driver on the mac I am getting just under 20Mbit/s with iperf3.
On Win11 I get 22Gbit/s which is near the limit due to Thunderbolt overhead.
On linux is shows up as a PCIe x4 device, no complaints about suboptimal performance


Samba on NVME 2 device mirror, Intel P4500. Blackmagic disk test.
Read speed is about 16Gbps/s

 

[b0bb]

@nimik I did the flint on this

flint -d mt4117 --image fw_TLNS3252_32.1010.bin -allow_psid_change b

And it kept failing. Is there a jumper or something I need to do?

The flash has to unlocked before PSID change is allowed.

DEVICE=/dev/mst/mt4117_pciconf0
FWFILE=fw_TLNS3252_32.1010.bin
sudo /usr/bin/flint -d $DEVICE -override_cache_replacement hw set Flash0.WriteProtected=Disabled
sudo /usr/bin/flint -allow_psid_change -d $DEVICE -i $FWFILE  burn

 

[Civiloid]

The flash has to unlocked before PSID change is allowed.

That heavily depends on a card actually. Most of them doesn't lock anything.

sudo /usr/bin/flint -d $DEVICE -override_cache_replacement hw set Flash0.WriteProtected=Disabled

Don't forget that you need flint_oem for that to work, normal mstflint from the distro or normal flint from OFED/DOCA won't work.

 

[Civiloid]

Stock Connectx-4 does not play well with external enclosure's 4-lane PCIe.

I haven't noticed that. That is old Echo Express SEL enclosuer, so I can't get full 25G bandwidth, but I casually get more than 10 Gbps. Endpoint is Linux machine with ConnectX-6 on a 100G link.

 

[b0bb]

View attachment 45144

I haven't noticed that. That is old Echo Express SEL enclosuer, so I can't get full 25G bandwidth, but I casually get more than 10 Gbps. Endpoint is Linux machine with ConnectX-6 on a 100G link.

Your Sonnet enclosure is one of the best Thunderbolt3 enclosures out there with a price to match (370-420)
The lower cost OWC I tried only managed 15Mbits/s

The DIY route with an external enclosure is a non-starter for me given the prices on the ATTO TLN3 is now less than what the OWC costs (230) and has better thruput.

The latest TLN3 I got off ebay is brand new with the warning stickers still stuck over the thunderbolt ports at the price mentioned here (189)
This is my 3rd unit.

 

[b0bb]

That heavily depends on a card actually. Most of them doesn't lock anything.

First 2 units I had will not flash unless unlocked
The latest one with newer firmware allowed a flash without unlock

 

[b0bb]

A couple of adjustments resulted in a 500Mbps bump in the TLN3 thruput (19.9Gbps to 20.4 Gbps)
- M4 Max instead of M1 Max
- 9000 MTU
- Optimized iperf3-darwin instead of vanilla iperf


Windows 11 iperf numbers, same unit moved over to the PC.
Average thruput 21.2Gbps, slight drop from my initial tests showing 22Gbps, win11 updated itself in between
This shows what the hardware is capable of.
- Ryzen 7950X3D
- 9000 MTU


ATTO TLN3 is 34.6% faster than the OWC Mercury Helios 3S enclosure + Connectx4-LX I was using previously.
This is on the Mac, the thruput is higher still on Windows, I have one of the units on the Win11 box as well

The major down side is Connectx4-LX end-of-life was 2022, it is an open question how long software and OS support will be around.
I think this is one reason the cost in the used market has fallen from the 1000+ MSRP

 

[Civiloid]

The major down side is Connectx4-LX end-of-life was 2022, it is an open question how long software and OS support will be around.

Latest firmware updated happened in 2024 (some CVE fix, AFAIR). But mlx5 driver on Linux/FreeBSD and osx to extent supports everything from CX4 EN (slighly older than Lx) until ConnectX-8, so unless there will be a driver change I don't think that support would go away, but we'll have to live with the unfixed bugs and some smaller compatibility problems that Cx4 still have.

I think this is one reason the cost in the used market has fallen from the 1000+ MSRP

CX6 DX on used market can be spotted for less than 300$ rather casually (still not typical price, but if you observe the market for a month you'll likely see one or two listings). So far I've seen few occasions when even ConnectX-7 was sold for less than 600$.

 

[Civiloid]

Your Sonnet enclosure is one of the best Thunderbolt3 enclosures out there with a price to match (370-420)

I don't think that matters that much - that is the aliexpress single port 25G one... Well, maybe there are some enclosures that are just bad, but it is based on cheap Intel TB3 controller with hacked firmware (to unlock more bandwidth):

 

[reasonsandreasons]

I think all of the Thunderbolt 3 enclosures out there are using the same handful of Intel chipsets, mostly based on the reference design. I'd be pretty surprised if there was that much of a difference beyond quirks of each generation.

 

[Civiloid]

Depends. For example here is the screenshot from old TB4 eGFX that uses some other chipset (vendor ID is not intel's). Also working good enough, slightly worse than the aliexpress card (aliexpress card actually was doing around line rate 25G of 1500b packets)

 

[jmsq]

For those of you who jumped on this hoping to get an easy TB3 25g adapter, you may want to take a closer look at the model numbers. I ended up with a TLNS-3102-D00, which I can't really find references to on their website. But as soon as you plug it in on Windows or Linux, there's a pretty obvious difference:



AFAICT this version is actually a ConnectX-3 Pro instead of the ConnectX-4 Lx, and as a result has no known firmware updates. I did try in case I was crazy and this cross flash might actually work (and the model number discrepancy didn't matter) but things still don't look promising there either, with either the 3102 or 3252 image:



If anyone has successfully flashed when their device presented as CX3-Pro, please let me know, but there might be older variants roaming around on ebay that aren't gonna do anything better than 10g SFP+ (maybe 40g QSFP+ if I'm lucky).

 

[epycmaster]

For those of you who jumped on this hoping to get an easy TB3 25g adapter, you may want to take a closer look at the model numbers. I ended up with a TLNS-3102-D00, which I can't really find references to on their website. But as soon as you plug it in on Windows or Linux, there's a pretty obvious difference:

View attachment 45249

AFAICT this version is actually a ConnectX-3 Pro instead of the ConnectX-4 Lx, and as a result has no known firmware updates. I did try in case I was crazy and this cross flash might actually work (and the model number discrepancy didn't matter) but things still don't look promising there either, with either the 3102 or 3252 image:

View attachment 45250

If anyone has successfully flashed when their device presented as CX3-Pro, please let me know, but there might be older variants roaming around on ebay that aren't gonna do anything better than 10g SFP+ (maybe 40g QSFP+ if I'm lucky).

Oh wow, wouldn't have known this issue. Did you buy it from a seller that listed TLN3-3102 as the model number? I guess you could ask for a return as the model wouldn't have matched.

 

[jmsq]

Oh wow, wouldn't have known this issue. Did you buy it from a seller that listed TLN3-3102 as the model number? I guess you could ask for a return as the model wouldn't have matched.

This is a case where I bought it months ago and sat on it until today, so any return wouldn't be possible regardless. That being said, the title did not explicitly have the model number listed, and the listing has aged out so I can't double check the details to see if it did. Considering there's zero mention of TLNS-3102 anywhere on Atto's website, I wouldn't even have known to look for that discrepancy. Live and learn

 

[nimik]

This is a case where I bought it months ago and sat on it until today, so any return wouldn't be possible regardless. That being said, the title did not explicitly have the model number listed, and the listing has aged out so I can't double check the details to see if it did. Considering there's zero mention of TLNS-3102 anywhere on Atto's website, I wouldn't even have known to look for that discrepancy. Live and learn

Interesting. If you flip yours over, does the part number match this?



These are the different model numbers on ATTO's page:

 

[jmsq]

Interesting. If you flip yours over, does the part number match this?

View attachment 45252

These are the different model numbers on ATTO's page:

View attachment 45253

Unfortunately not, and my model number doesn’t match any in the list on their website:

 

[nimik]

Wild that they made the model numbers so similar

 

[MortenB]

Would this adapter work as intended in newest MacOS version without lots of tinkering?

 

[kapone]

Would this adapter work as intended in newest MacOS version without lots of tinkering?

If your Mac has pcie slots (i.e. not a Mac mini etc) then a regular Connectx-4 works out of the box. No tinkering.

 

[MortenB]

Just bought MacBook Pro 14" M4 24GB RAM/ 1TB SSD. Was thinking of utilizing the 25G ports on my Ubiquiti Enterprise xg24