I have the following equipment:
So far I have had great success with connectivity and speed, with this very cheap set of fiber transceivers and fiber. I am glad to report great success both last year with Windows (I could pack the MCX353A card, an ultra low profile card, with its bracket removed, into the SFF enclosure. When not using that, the x4 pcie, itself adapted from an m.2 slot, adapts back into an m.2 slot.) and more recently with macOS on my AppleSilicon macbook, achieving 26 and 20Gbps respectively, it will be good enough for me to be satisfied for the better part of a decade, a performance capability which will not be threatened by other technologies until Wifi 8 and likely beyond.
However! One very significant hurdle remains. When the connection from windows or macOS goes down for a long enough time, and I do not have a handle on what this amount of time is, but I feel like it is between 30 minutes and 5 hours, the fiber link cannot come back up again until I do the very specific thing of replugging the transceiver from the linux side of the connection. Yes. Replug the transceiver. The following do NOT restore the connection:
- rebooting the other computer
- replugging transceiver in the other computer
- replugging the fiber into the transceiver in the other computer
- replugging the fiber into the transceiver in the linux computer
- rebooting the linux computer
- disabling the pcie device and bringing it back with rescan via sysfs (
As you might imagine I cannot justify spending all of this effort to set all this stuff up if i have to repeatedly every day go back downstairs and reach down under the desk to unplug and replug the transceiver. Neither will it make sense to try to make a solenoid or servo contraption to automate that, although that would work, would be doable (since there is that nice pull mechanism; doing it with RJ-45 would be impractically difficult by comparison...) and be really frickin' cool, it would certainly wear out the mechanism after some time. Although I might be able to convince myself that with the transceiver and the CX3 card both in the neighborhood of $30 I actually could afford such wear and tear, I will reserve this solution as a last resort approach. But I WILL GO TO THOSE LENGTHS IF NECESSARY.
Does anyone know how to make progress into this issue?
- Are the mlx4 drivers closed source?
- Do we think the transceiver might be to blame for this weird "power saving" behavior? How might I confirm this?
I will try to do some further testing to compare behavior between the fiber connection with which I reliably encounter this problem, and doing the same with a DAC. In such a test the transceiver and optical fiber part is the only part that changes. I think if I do not experience this behavior (e.g. with a DAC the connection spins right back up after leaving it offline overnight) then it would give me more faith to explore other fiber transceiver combinations.
- If i place the CX4 card into linux and use that instead, I think behavior would likely be different, it seems like mlx5 driver would be used instead of mlx4 (for cx3) and I'm not sure that this would test anything other than possibly confirm that i could get more CX4 cards to use on the linux side. But I'd rather not, if CX3 cards suffice as I have so many already. It won't hurt but I don't need to spend $150 on dual port CX4 cards yet. I will consider it if it will be the only way to resolve this problem though, and it seems like it will still be cheaper than switching fiber tech to LC from MPO.
So far it would appear that the most appealing avenues to explore are:
- other MPO transceivers???
- switch to CX4 on linux side. Can get by with a single CX4 dual port card to wire up my mac and windows connections. Other linux to linux server connections I can get by fine on CX3's if they run 24/7.j
- possibly give up; run the windows box 24/7 and use workaround of plugging in CX4 USB-C connection to iMac at same desk when not in use with macbook. I am testing this now too and it does work but causes the iMac to disable all USB devices because of "drawing too much power". But nothing is failing or shut down yet. But it does indicate something is not great about power delivery to connect it in this way. I could also use my iPad to keep the fiber connection alive. But again that's just not going to work if i want to bring both ipad and macbook around the house... Oh nice. that brought me to one more idea. I can also bring my Steam Deck into this rotation and use it to keep the connection alive. haha. (Edit: Tested and yeah no, Steam Deck has no thunderbolt support. It just charges from that cable.)
- Ubuntu workstation (5950X, 128GB RAM, 2xRTX 3090 NVLink, 6x14TB Seagate Exos (ZFS), etc.)
- M1 Max 64GB Macbook Pro
- Windows SFF desktop (5800X3D, RTX 3080Ti, etc.)
- 3x MCX354A-QCBT/FCBT (they should be interchangeable and most are flashed to FCBT already)
- 1x MCX353A-QCBT
- 1x MCX455A-ECAT
- 10Gtek MPO QSFP+ 40Gb transceivers https://www.amazon.com/gp/product/B01JM73WHO/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1
- 50ft MPO fiber patch cable https://www.amazon.com/gp/product/B08Q327KBN/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
- 2x 1.5m QSFP+ DACs from FS.comj
- Sonnet Breakaway Box 550W (eGPU enclosure)
So far I have had great success with connectivity and speed, with this very cheap set of fiber transceivers and fiber. I am glad to report great success both last year with Windows (I could pack the MCX353A card, an ultra low profile card, with its bracket removed, into the SFF enclosure. When not using that, the x4 pcie, itself adapted from an m.2 slot, adapts back into an m.2 slot.) and more recently with macOS on my AppleSilicon macbook, achieving 26 and 20Gbps respectively, it will be good enough for me to be satisfied for the better part of a decade, a performance capability which will not be threatened by other technologies until Wifi 8 and likely beyond.
However! One very significant hurdle remains. When the connection from windows or macOS goes down for a long enough time, and I do not have a handle on what this amount of time is, but I feel like it is between 30 minutes and 5 hours, the fiber link cannot come back up again until I do the very specific thing of replugging the transceiver from the linux side of the connection. Yes. Replug the transceiver. The following do NOT restore the connection:
- rebooting the other computer
- replugging transceiver in the other computer
- replugging the fiber into the transceiver in the other computer
- replugging the fiber into the transceiver in the linux computer
- rebooting the linux computer
- disabling the pcie device and bringing it back with rescan via sysfs (
echo 1 > /sys/bus/pci/device/0000:0e:00.0/remove; sleep 1; echo 1 > /sys/bus/pci/rescan as root)As you might imagine I cannot justify spending all of this effort to set all this stuff up if i have to repeatedly every day go back downstairs and reach down under the desk to unplug and replug the transceiver. Neither will it make sense to try to make a solenoid or servo contraption to automate that, although that would work, would be doable (since there is that nice pull mechanism; doing it with RJ-45 would be impractically difficult by comparison...) and be really frickin' cool, it would certainly wear out the mechanism after some time. Although I might be able to convince myself that with the transceiver and the CX3 card both in the neighborhood of $30 I actually could afford such wear and tear, I will reserve this solution as a last resort approach. But I WILL GO TO THOSE LENGTHS IF NECESSARY.
Does anyone know how to make progress into this issue?
- Are the mlx4 drivers closed source?
- Do we think the transceiver might be to blame for this weird "power saving" behavior? How might I confirm this?
I will try to do some further testing to compare behavior between the fiber connection with which I reliably encounter this problem, and doing the same with a DAC. In such a test the transceiver and optical fiber part is the only part that changes. I think if I do not experience this behavior (e.g. with a DAC the connection spins right back up after leaving it offline overnight) then it would give me more faith to explore other fiber transceiver combinations.
- If i place the CX4 card into linux and use that instead, I think behavior would likely be different, it seems like mlx5 driver would be used instead of mlx4 (for cx3) and I'm not sure that this would test anything other than possibly confirm that i could get more CX4 cards to use on the linux side. But I'd rather not, if CX3 cards suffice as I have so many already. It won't hurt but I don't need to spend $150 on dual port CX4 cards yet. I will consider it if it will be the only way to resolve this problem though, and it seems like it will still be cheaper than switching fiber tech to LC from MPO.
So far it would appear that the most appealing avenues to explore are:
- other MPO transceivers???
- switch to CX4 on linux side. Can get by with a single CX4 dual port card to wire up my mac and windows connections. Other linux to linux server connections I can get by fine on CX3's if they run 24/7.j
- possibly give up; run the windows box 24/7 and use workaround of plugging in CX4 USB-C connection to iMac at same desk when not in use with macbook. I am testing this now too and it does work but causes the iMac to disable all USB devices because of "drawing too much power". But nothing is failing or shut down yet. But it does indicate something is not great about power delivery to connect it in this way. I could also use my iPad to keep the fiber connection alive. But again that's just not going to work if i want to bring both ipad and macbook around the house... Oh nice. that brought me to one more idea. I can also bring my Steam Deck into this rotation and use it to keep the connection alive. haha. (Edit: Tested and yeah no, Steam Deck has no thunderbolt support. It just charges from that cable.)
Last edited:
