I've been running the DX010 for a few years now, I'm the originator of this post. I'm happy to help if you come across any problems.
I am still using the old build I posted about, where breakout still works. The link and password in my original post are still active. I have not tried the new builds again in a long time. As long as you install "my" build, you will be OK - But the DX010 (and SONiC for that matter) isn't for everyone:
I used to have a Dell Z9100, which I sold on this forum, it is a very clean bug-free prod-ready switch. If that's what you're looking for, you may not like a DX010. SONiC on whitebox hardware has a lot of advantages (like unlocked SFP compatibility), but it takes some degree of messing around, be ready.
There are a couple other annoyances you may run into with the older build (or possibly the new ones too) that I have discovered, I'll share them below for any DX010 users out there, I suspect we will see an influx because they are available cheaply again.
- This post and its comments has a lot of useful getting started info:
After starting with that post, also be aware of the following:
- You need to add a MGMT_PORT entry manually into the /etc/sonic/config_db.json to avoid errors. Here is where you put it:
The text is:
- The static route add command doesn't always put the correct entries in the config file. See my own config file at the bottom for examples.
- If you enable DCB (Data Center Bridging) for RoCE PFC/QoS, future breakouts fail after configuring DCB. I run RoCE without DCB and it works perfectly fine. I would recommend just not enabling it. If you must, you will need to remove all DCB config to do a breakout, then do the breakout, and config DCB again after.
- If you create multiple VRFs, and create routes that cross VRF boundaries, that traffic needs to be routed by the slow Atom CPU, not the ASIC. This isn't unique to the DX010. In my case, I have one VRF with all VLANs in there and it works perfectly.
- There is a strange issue that happens with very little network load: The ARP entries for the upstream gateways can age-off in a strange way that prevents them from coming back. Not everybody seems to hit this issue, but if you do, see my ridiculous workaround in the references links below.
- If using DHCP helper service, sometimes it just stops working for no reason, and starts putting errors in the syslog. I have never really tried to fix it because it happens so infrequently. I usually reboot the switch frequently enough to never hit it. I'm open to creating a workaround if other people have this.
- For some SFPs and devices at the far-end, a specific set of commands is needed to get the link to come up after the breakout, especially for 1G and some 10G. This is also necessary for a specific model of 4x10G AOC breakout I use; in this case put 10000 instead of 1000. I also have a 4x25G AOC that needs speed 25000 and fec mode fc. For 1x100G and 2x50G usually none of the below is needed, all you have to do is set fec mode rs and it comes right up.
Links:
My DX010 config file: Online Notepad with Privacy and Publishing
My Github post about ARP age-off and workaround: Aging-off of ARP entries for nexthop gateways can break static routes · Issue #1210 · sonic-net/SONiC
I am still using the old build I posted about, where breakout still works. The link and password in my original post are still active. I have not tried the new builds again in a long time. As long as you install "my" build, you will be OK - But the DX010 (and SONiC for that matter) isn't for everyone:
I used to have a Dell Z9100, which I sold on this forum, it is a very clean bug-free prod-ready switch. If that's what you're looking for, you may not like a DX010. SONiC on whitebox hardware has a lot of advantages (like unlocked SFP compatibility), but it takes some degree of messing around, be ready.
There are a couple other annoyances you may run into with the older build (or possibly the new ones too) that I have discovered, I'll share them below for any DX010 users out there, I suspect we will see an influx because they are available cheaply again.
- This post and its comments has a lot of useful getting started info:
After starting with that post, also be aware of the following:
- You need to add a MGMT_PORT entry manually into the /etc/sonic/config_db.json to avoid errors. Here is where you put it:
The text is:
Code:
"MGMT_PORT": {
"eth0": {
"admin_status": "up",
"alias": "eth0"- If you enable DCB (Data Center Bridging) for RoCE PFC/QoS, future breakouts fail after configuring DCB. I run RoCE without DCB and it works perfectly fine. I would recommend just not enabling it. If you must, you will need to remove all DCB config to do a breakout, then do the breakout, and config DCB again after.
- If you create multiple VRFs, and create routes that cross VRF boundaries, that traffic needs to be routed by the slow Atom CPU, not the ASIC. This isn't unique to the DX010. In my case, I have one VRF with all VLANs in there and it works perfectly.
- There is a strange issue that happens with very little network load: The ARP entries for the upstream gateways can age-off in a strange way that prevents them from coming back. Not everybody seems to hit this issue, but if you do, see my ridiculous workaround in the references links below.
- If using DHCP helper service, sometimes it just stops working for no reason, and starts putting errors in the syslog. I have never really tried to fix it because it happens so infrequently. I usually reboot the switch frequently enough to never hit it. I'm open to creating a workaround if other people have this.
- For some SFPs and devices at the far-end, a specific set of commands is needed to get the link to come up after the breakout, especially for 1G and some 10G. This is also necessary for a specific model of 4x10G AOC breakout I use; in this case put 10000 instead of 1000. I also have a 4x25G AOC that needs speed 25000 and fec mode fc. For 1x100G and 2x50G usually none of the below is needed, all you have to do is set fec mode rs and it comes right up.
Code:
sudo config int fec Ethernet32 none
sudo config int speed Ethernet32 1000
sudo config int advertised-speeds Ethernet32 1000
sudo config int autoneg Ethernet32 disabled
sudo config int transceiver lpmode Ethernet32 disable
sudo config int shut Ethernet32
sudo config int start Ethernet32My DX010 config file: Online Notepad with Privacy and Publishing
My Github post about ARP age-off and workaround: Aging-off of ARP entries for nexthop gateways can break static routes · Issue #1210 · sonic-net/SONiC
