Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[explosivelobster]

Firstly a big thanks to fohdeesha for this cache of info, came at exactly the right time for me upgrading my homelab networking!

Is there any more info on breakout compatibility with the rear QSFP ports? I have an ICX-6610-24 en route and as I don't think it's got any licences for the SFP+ ports I was hoping to use a breakout to get 10GBe, but hearing that some people are having problems means I've held off on buying anything yet. Hosts will be using X520-DA2 cards (no transceivers yet)...

Would a ConnectX-3 CX353A-FCBT be suitable to connect hosts at 40Gbe to the pair of non breakout QSFP ports?

 

[PGlover]

hmm, if it is indeed a BPDU issue, the brocade might have a bug where it simply will not flood BPDU packets unless you turn STP on, but that would be a pretty serious bug

You can just enable STP or RSTP on the brocade, at the vlan level of the vlans the sonos talk on. It shouldn't be necessary, but it seems due to a bug or otherwise you might not have a choice.

There's a chance this has absolutely nothing to do with STP and is a misconfiguration somewhere else, but I dunno. turning on STP on the brocade will confirm that. if it STILL doesn't work, then it's not stp related. You need to be sure to turn stp on for the vlans, and remove all the no spanning-tree commands you added

I turned STP on Vlan 1 and it did not work. Vlan 1 is where the Glover Wi-Fi network is on. For testing, I pointed Sonos to use the Guest network (Vlan 30) where pfFsense is the DHCP server for Vlan 30. While at least it gave the SonosZP device valid IP address.

Once again, I was wondering if there is something about my DHCP server setup on the ICX 6610.

configure terminal
ip dhcp-server enable
ip dhcp-server pool 192.168.1.0/24
network 192.168.1.0 255.255.255.0
excluded-address 192.168.1.1 192.168.1.99
dhcp-default-router 192.168.1.2
dns-server 75.75.75.75 75.75.76.76 8.8.8.8
exit
ip dhcp-server pool 192.168.1.0/24
deploy

 

[PGlover]

After removing all the STP, I still have the command "global-stp" in my running config file. The "no global-stp" command does not remove the command from my config file.

 

[PGlover]

This is show stopper for me.. If I can't get Sonos to work with the ICX 6610 switch, I will need to revert back to my old setup. The wife use Sonos on a daily basis. This is really sad. I had high hopes on the ICX 6610...

 

[PGlover]

I was able to go the status mode of the Sonos device. Lots of information.

http://192.168.30.114:1400/status/ifconfig

Here is some information on the various interface on the Sonos device...

'eth0' and 'eth1' correspond to the 2 wired ports
'lo' and 'br0' interfaces are virtual networking devices used internally by the Linux kernel
'ath0' is the Atheros is the WiFi interface



STP information on the Sonos device....

 

[fohdeesha]

This is show stopper for me.. If I can't get Sonos to work with the ICX 6610 switch, I will need to revert back to my old setup. The wife use Sonos on a daily basis. This is really sad. I had high hopes on the ICX 6610...

The fact that it worked on the guest VLAN (which is still going through the brocade) 100% rules out STP, and means it's most likely a DHCP issue. The DHCP implementation on the 6610 is 100% RFC 2131 compliant (and I've used it with hundreds of devices), so either the sonos DHCP implementation is broken, or you have a misconfig somewhere

My first guess would be it doesn't like getting handed a gateway address that isn't the first usable IP in the subnet (your DHCP config has it handing out 192.168.1.2 for a gateway instead of 192.168.1.1). A compliant ethernet device should have no issue with this, but again we're talking IoT stuff

does the VE interface in that vlan actually have an IP of 192.168.1.2? because that's the gateway your DHCP config is giving out to clients. In the config you sent me in pm a few days ago the VE had an address of 192.168.1.1, so if you're telling devices to use 192.168.1.2, that's going to fail. Try changing it to 192.168.1.1, and change your DHCP config to hand out 192.168.1.1 for the default gateway.

The global STP config line is just because STP was previously enabled in a vlan, it doesn't turn STP on witohut a vlan config as well. if you reboot it'll go away from the config, but I wouldn't worry about it

After making these changes, turn all of the sonos off, and turn them on one at a time, starting with just the wired controller, and see if the wired controller gets an address

 

[fohdeesha]

If that still doesn't work, watch the log on the 6610 when turning a sonos on (show log), it should show DHCP server activity. You can get more verbose output (literally print every DHCP packet to the console) with debug commands, but that's harder to explain. I can take a look at it for you if you can provide remote access, but I have a feeling this is a simple misconfig

 

[fohdeesha]

Firstly a big thanks to fohdeesha for this cache of info, came at exactly the right time for me upgrading my homelab networking!

Is there any more info on breakout compatibility with the rear QSFP ports? I have an ICX-6610-24 en route and as I don't think it's got any licences for the SFP+ ports I was hoping to use a breakout to get 10GBe, but hearing that some people are having problems means I've held off on buying anything yet. Hosts will be using X520-DA2 cards (no transceivers yet)...

Would a ConnectX-3 CX353A-FCBT be suitable to connect hosts at 40Gbe to the pair of non breakout QSFP ports?

Licenses are free for STH members, just shoot me a PM. As far as qsfp breakout goes, I have verified that optical breakout works 100% with or without a stacking config

That would be a transceiver like this: Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

with a breakout fiber cable plugged into it: MPO Fan-out Patch Cable, Multimode Fiber, MPO to LC Uniboot, 3-meter 10FT 603149398665 | eBay

then you have 4x LC 10gbE fiber connectors, that would plug into SFP+ optics in the intel cards. I ordered a monoprice DAC breakout, and it's also working. A member in PM's have reported a fiberstore DAC breakout works as well. glover is having some issues with his passive breakout DAC, but he seems to be the only one at this point


the CX353A-FCBT will work, but you can save a lot of money by buying the QCBT and flashing it with the FCBT firmware as outlined here - https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

examples of cheap QCBT mellanox cards are HP 649281-B21 Infiniband FDR/EN 10/40GB 2P 544QSFP ADAPTER 656089-001 661685-001 | eBay

if you need the full height bracket: Mellanox MTM000574 FRU Tall Bracket for 2-Port QSFP Adapter with Gasket | eBay

PS the mellanox Connectx3 series is much cheaper and has better compatibility (in regards to optics/transceivers etc) and is much cheaper versus the intels, if you're interested: Mellanox MCX311A-XCAT CX311A ConnectX-3 Network 10GbE SFP+ Both Low+Full Profile | eBay


dual port version: CX312A MELLANOX CONNECTX-3 DUAL PORT GIGABIT ETHERNET CARD MCX312A-XCBT | eBay

 

[PGlover]

The fact that it worked on the guest VLAN (which is still going through the brocade) 100% rules out STP, and means it's most likely a DHCP issue. The DHCP implementation on the 6610 is 100% RFC 2131 compliant (and I've used it with hundreds of devices), so either the sonos DHCP implementation is broken, or you have a misconfig somewhere

My first guess would be it doesn't like getting handed a gateway address that isn't the first usable IP in the subnet (your DHCP config has it handing out 192.168.1.2 for a gateway instead of 192.168.1.1). A compliant ethernet device should have no issue with this, but again we're talking IoT stuff

does the VE interface in that vlan actually have an IP of 192.168.1.2? because that's the gateway your DHCP config is giving out to clients. In the config you sent me in pm a few days ago the VE had an address of 192.168.1.1, so if you're telling devices to use 192.168.1.2, that's going to fail. Try changing it to 192.168.1.1, and change your DHCP config to hand out 192.168.1.1 for the default gateway.

The global STP config line is just because STP was previously enabled in a vlan, it doesn't turn STP on witohut a vlan config as well. if you reboot it'll go away from the config, but I wouldn't worry about it

After making these changes, turn all of the sonos off, and turn them on one at a time, starting with just the wired controller, and see if the wired controller gets an address

I have been working on this all day. Here is the temporary fix to get Sonos working. I decided to take 1 of the 3 UniFi APs I have and dedicate it for Sonos wireless. The AP has a SSID for Sonos wi-fi traffic. I don't need 2 APs for upstairs anyway. I have bypassed the ICX 6610 and connected the AP directly into my upstream router that is on a 192.168.50.0 network. The router setup for DHCP on the 192.168.50.0 network. I have connected 3 Sonos devices and it is working fine..

I still would like to troubleshoot this issue further on the ICX6610.

 

[fohdeesha]

Did you check the VE IP and that the DHCP server is handing out the first IP in the subnet (eg 192.168.1.1)? putting it on its own vlan to bypass the switch should absolutely not be necessary, you have something really strange misconfigged somewhere

also, are the sonos completely up to date firmware/software wise?

 

[i386]

examples of cheap QCBT mellanox cards are HP 649281-B21 Infiniband FDR/EN 10/40GB 2P 544QSFP ADAPTER 656089-001 661685-001 | eBay

These are fcbt nics with "custom" HP firmware, f = fdr, q = qdr

 

[fohdeesha]

the ones I ordered came with the QCBT firmware (limited to 10gbE over ethernet). they are identical hardware though and can be cleanly flashed with the stock FCBT mellanox firmware for 40gbE ethernet, using 8 of them at home. With the A2 and up revisions, the FCBT and QCBT cards are the same hardware running different firmware - the big difference being "en_427_mhz = true" in the board definition file

 

[arglebargle]

Does anyone know if the dual port VPI cards can simultaneously run infiniband on one port and ethernet on another? I'd like to try NFS over RDMA between two hosts directly and plug the other port into an ethernet switch. No huge problem if not, I just wanted to bench test RDMA vs TCP/IP.

 

[fohdeesha]

99% sure, but I can test real quick

 

[fohdeesha]

Look like yup, you just need to use mlxconfig to set the ports accordingly. quick mlxconfig tutorial in the second part of my post here - https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

To set to infiniband on one and ethernet on another you would just run:

#Set port 1 to Infiniband
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=1
#set port 2 to Ethernet
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P2=2
reboot
#for the end number, 1=infiniband, 2=ethernet, 3=auto

After a reboot it shows 1 port ethernet, 1 port IB:

root@debianlab4:~# mlxconfig -d /dev/mst/mt4099_pci_cr0 query

Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        IB(1)
         LINK_TYPE_P2                        ETH(2)

 

[arglebargle]

Man, STH is going to be the death of my budget.

I've got a pair of HP branded dual 40Gb cards on the way along with a 40Gb DAC, QSFP+ to SFP+ adapters, and more SFP+ DACs to get everything connected to the 6450 under my desk.

Looking forward to testing NFS over RDMA between the VMs hosted on my workstation and the backing storage pool of SSDs in my Centos box.

Thanks @fohdeesha!

 

[arglebargle]

Also, this is a shot in the dark, but for $7.99 I thought I'd just buy it and test.

NetApp X6559-R6 SAS expansion Cable 5m 5 meter QSFP-QSFP 112-00178 15ft 11200178 | eBay

Anyone know if these are capable of 40Gb between two C-X3s?

 

[i386]

Anyone know if these are capable of 40Gb between two C-X3s?

I have a few broadcom branded "external sas" cables with qsfp+ connectors that worked without problems with the mellanox nics and arista & mellanox switches.
I would say they should work for 40gbe.

 

[arglebargle]

I have a few broadcom branded "external sas" cables with qsfp+ connectors that worked without problems with the mellanox nics and arista & mellanox switches.
I would say they should work for 40gbe.

Awesome, thank you! $8 for a 40Gb DAC is about my speed right now.

 

[PGlover]

Did you check the VE IP and that the DHCP server is handing out the first IP in the subnet (eg 192.168.1.1)? putting it on its own vlan to bypass the switch should absolutely not be necessary, you have something really strange misconfigged somewhere

also, are the sonos completely up to date firmware/software wise?

The VE IP and the DHCP server router IP are the same.. I have PM you my configuration file. Maybe you can find something in the config file that is not correct.

Over the next week or so, I plan to test this thoroughly using my spare ICX 6610 I have. I don't want to continue making changes in my production setup.