ATTO 3102- Thunderbolt Mellanox ConnectX-4 Lx (Dual Port 25GbE) - $189 or less

[nimik]

The ATTO ThunderLink TLN3-3102-D00 is actually a Thunderbolt 3 ConnectX-4 with two independent SFP28 ports.




They're available on eBay from various sellers- here's a listing with multiple available:

Pardon Our Interruption...

Another, need to source a power adapter though:

Pardon Our Interruption...

There are some cheaper one-off listings if you look around (sold after this post, naturally):

Pardon Our Interruption...

Out of the box, the 3102 is capped at 10GbE and won't link at 25GbE. That's easy enough to fix. You can go to ATTO's download page, flash the TLNS-3252 firmware (fw_TLNS3252_32.1010.bin in the same zip file), and 1/10/25GbE will work.



Note that even new, these don't come with Thunderbolt cables. You have to supply your own. They also require PCIe tunnelling to work (TB3, TB4, USB4- not a normal USB-C port).

Here's what one looks like outside the all-metal case. The fan is not bad.



Now my Mac mini has 25 gig!

 

[luckylinux]

Prices are HUGE in Europe for something like that (> 1000 EUR).

Isn't it possible to build something like this using something like a eGPU Enclosure (ideally ATX, but I guess it could also be done with a 12V PicoPSU) ?

Something like:
- Case: https://vi.aliexpress.com/item/1005005500276507.html (55 EUR for SFX / 80 EUR for ATX)
- Board: https://vi.aliexpress.com/item/1005006121040283.html
- PSU: any SFX/ATX or possibly PicoPSU + Blanking Plate ?
- NIC: probably around 25-50 EUR (I already have it)

Just for Case and Board it's around 200-250 EUR so complete most likely around 300 EUR (and could be repurposed).

I just wonder if we can "DIY" it using some components that are more available than this ATTO Solution (particularly in Europe) .

Not sure if there are much smaller & cheaper Cases & Boards, maybe my Aliexpress-Fu is NOT that good .

 

[nexox]

Isn't it possible to build something like this using something like a eGPU Enclosure (ideally ATX, but I guess it could also be done with a 12V PicoPSU) ?

I just (mostly) finished building exactly that, same dock but with a Supermicro riser to turn the NIC 90 degrees and make it a lot smaller, PicoPSU, printed case that I'll probably upload at some point, and a blower fan from a NUC, cost a reasonable amount more than the ATTO posted here and the dock has gotten $25 more expensive since then.

 

[mattventura]

You can absolutely DIY it, but unless it *must* be thunderbolt due to limitations of the host system, you're generally better off using an OCuLink dock for lower cost and latency. Or any plain PCIe connector.

 

[nexox]

You can absolutely DIY it, but unless it *must* be thunderbolt due to limitations of the host system, you're generally better off using an OCuLink dock for lower cost and latency. Or any plain PCIe connector.

Any real PCIe is of course better than Thunderbolt, but my work laptop doesn't have anything else, and I suspect that's the situation where people look for this kind of option. Still beats 1G and I don't have any 2.5G ports at my desk, plus I'm totally going to get NFS over RDMA working between my laptop and desktop to offload builds... some day.

Edit: note also when I was searching there were some cheaper docks, but they wouldn't charge my laptop, so I didn't investigate them much, though some had older Thunderbolt chipsets which may be lower performance.

 

[Prophes0r]

...between my laptop and desktop to offload builds...

This feels like the exact use-case for a point-to-point TB/USB4 network link.
Yeah you will have a bunch of overhead that stop you from actually getting the full 40Gbit link speed, but it should still beat the pants off a 10Gbit link and require no hardware other than a cable.

 

[luckylinux]

I just (mostly) finished building exactly that, same dock but with a Supermicro riser to turn the NIC 90 degrees and make it a lot smaller, PicoPSU, printed case that I'll probably upload at some point, and a blower fan from a NUC, cost a reasonable amount more than the ATTO posted here and the dock has gotten $25 more expensive since then.

Could you share your Design and BOM when you finalized it ? I'm mainly curious about it and my Laptop for now is only using WiFi. I'm not even sure I will need it, but it's nice to know that there are Options in Case / when I do need it. No Time Pressure on my End though .

EDIT 1: I don't need Power Delivery to charge my Laptop I believe, it would actually be better to decouple the Laptop and external GPU Card, if possible at all.

 

[Civiloid]

Isn't it possible to build something like this using something like a eGPU Enclosure (ideally ATX, but I guess it could also be done with a 12V PicoPSU) ?

I have TB4 Enclosure where I have ConnectX-4 Lx installed. You can cross-flash it with ATTO's firmware (for different card, which is identical to normal 4 Lxs).

You can get something like th3p4 lite, 3d print (or find) a case for it, only problem is that you need to mod it to get a fan header installed.

Board: https://vi.aliexpress.com/item/1005006121040283.html

Going for TH3P4 Lite would be slightly better, as I think TH3P4G3 has older firmware and can't utilize the full 32 Gbps bandwidth that should be allocated to PCIe by the dock, unless there is some other revision that fixed that problem

 

[luckylinux]

I have TB4 Enclosure where I have ConnectX-4 Lx installed. You can cross-flash it with ATTO's firmware (for different card, which is identical to normal 4 Lxs).

You can get something like th3p4 lite, 3d print (or find) a case for it, only problem is that you need to mod it to get a fan header installed.



Going for TH3P4 Lite would be slightly better, as I think TH3P4G3 has older firmware and can't utilize the full 32 Gbps bandwidth that should be allocated to PCIe by the dock, unless there is some other revision that fixed that problem

So something like this:

https://www.aliexpress.com/item/1005006057316982.html

https://www.aliexpress.com/item/1005008419281716.html

The nice Thing seems to be much more Compact and you install the NIC horizontally without need for PCIe Riser.

But yeah, unsure about the Case.

 

[nexox]

This feels like the exact use-case for a point-to-point TB/USB4 network link.
Yeah you will have a bunch of overhead that stop you from actually getting the full 40Gbit link speed, but it should still beat the pants off a 10Gbit link and require no hardware other than a cable.

I looked into that, but there's no good way to get Thunderbolt on my desktop since the motherboard doesn't support it, also don't think it supports RDMA.

 

[pimposh]

Just curious. At the moment macOS still lacks RDMA. Given that guess SMB caps at 15Gbps (without going multichannel). And what about NFS ?

 

[luckylinux]

Just curious. At the moment macOS still lacks RDMA. Given that guess SMB caps at 15Gbps (without going multichannel). And what about NFS ?

Don't both SMB and NFS support multi-channel ?

 

[pimposh]

This was more macOS oriented question.
Without going multichannel/multipath for NFS (if it is supported on macOS?) this seems to be bit capped no matter what speed ports are linking at.

Due to TB limitations anyway its capped to 32Gbps in total, no matter of os, or smth changed in here?

 

[cy384]

Just got the TLN3-3102-D00, bumped it up to the 25Gbe firmware, and it works great! Came right up on my ubuntu laptop.

 

[Greg1234]

Hey there - ended up buying one of these on this thread but cant seem to flash the 25Gbe firmware on my mac os device. The mac software only seems to work with the bundle and I cant select the specific 3252 firmware. I tried to recreate the bundle by copying the 3252 files into the 3102 folder to "trick" it but couldnt get it to use the bundle. Any hints getting this going on mac os?

 

[Civiloid]

Any hints getting this going on mac os?

I'm not sure that it would work on MacOS. For Mellanox cards (and that is essentially Mellanox) you needed to use opensource version of mstflint to cross-flash it, and it doesn't support macos at all.

You might have some luck if you'll use a virtual machine with Ubuntu though and can pass-through the device (though I never tried to do that with thunderbolt so I don't know if its even possible on macos)

 

[nimik]

The answer might be in the Compatibility List folder in the macOS firmware bundle. I don't know if their atflash utility lets you change the PSID, though.

I tried to passthrough Thunderbolt on macOS and it didn't work in the hypervisors I tried. I flashed my 3102 on a PC then moved it back to my Mac.

 

[Greg1234]

Thanks for the response. I ended up stealing a friend's windows computer and solved it

 

[nexox]

Going for TH3P4 Lite would be slightly better, as I think TH3P4G3 has older firmware and can't utilize the full 32 Gbps bandwidth that should be allocated to PCIe by the dock, unless there is some other revision that fixed that problem

I finally got some 25G transceivers to test my TH3P4G3V3, I don't know about the full 32Gbps bandwidth, but it'll max out a 25G port pretty well (iperf3, single thread, two Linux systems directly connected, 1500 MTU):

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.67 GBytes  22.9 Gbits/sec    0   2.23 MBytes      
[  5]   1.00-2.00   sec  2.68 GBytes  23.0 Gbits/sec    0   2.23 MBytes      
[  5]   2.00-3.00   sec  2.68 GBytes  23.0 Gbits/sec  620   1.96 MBytes      
[  5]   3.00-4.00   sec  2.68 GBytes  23.0 Gbits/sec    0   2.02 MBytes      
[  5]   4.00-5.00   sec  2.68 GBytes  23.0 Gbits/sec    0   2.05 MBytes      
[  5]   5.00-6.00   sec  2.68 GBytes  23.0 Gbits/sec    0   2.15 MBytes      
[  5]   6.00-7.00   sec  2.68 GBytes  23.0 Gbits/sec  543   1.84 MBytes      
[  5]   7.00-8.00   sec  2.68 GBytes  23.0 Gbits/sec    0   1.87 MBytes      
[  5]   8.00-9.00   sec  2.68 GBytes  23.0 Gbits/sec    0   2.00 MBytes      
[  5]   9.00-10.00  sec  2.68 GBytes  23.0 Gbits/sec    0   2.05 MBytes      
[  5]  10.00-10.00  sec  1.25 MBytes  20.4 Gbits/sec    0   2.05 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  26.8 GBytes  23.0 Gbits/sec  1163             sender

 

[kathampy]

There are generic 25G USB4 NICs on Amazon for $200 that don't seem to require external power. Presumably, they use the same chipset.