Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[PGlover]

wait, those two links aren't going to two different pfsense boxes are they? that's not how LACP works

No... They are going to the same pfSense box...

 

[PGlover]

Do you think I need to set the speed and duplex on both interfaces? Currently they are set to auto...

 

[fohdeesha]

why do they both have the same MAC address? that doesn't seem right. and one of them seems to be linking at 100m, you got something going on with the pfsense box it looks like. you shouldn't have to manually configure the speeds

 

[fohdeesha]

just ordered a passive breakout DAC to test with to see if I can reproduce your breakout issue - it could be that DAC breakout is not supported on the rear ports, only optical

2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable 889028000663 | eBay

 

[PGlover]

why do they both have the same MAC address? that doesn't seem right. and one of them seems to be linking at 100m, you got something going on with the pfsense box it looks like. you shouldn't have to manually configure the speeds

When I do a show interfaces brief on the ICX6610 active controller, all the interfaces Mac addresses are the same.. Check it out... Weird...

ICX6610-48P Router>show interface brief

Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c To pfSense Mast
1/1/2 Up Blocked Full 100M 2 Yes N/A 0 cc4e.24b8.5c4c To pfSense Back
1/1/3 Down None None None None Yes 1 0 cc4e.24b8.5c4c
1/1/4 Down None None None None Yes 1 0 cc4e.24b8.5c4c
1/1/5 Down None None None None Yes 1 0 cc4e.24b8.5c4c
1/1/6 Down None None None None Yes 1 0 cc4e.24b8.5c4c
1/1/7 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/8 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/9 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/10 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/11 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/12 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/13 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/14 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/15 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/16 Down None None None None No 99 0 cc4e.24b8.5c4c
1/1/17 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/18 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/19 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/20 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/21 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/22 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/23 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/24 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/25 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/26 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/27 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/28 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/29 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/30 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/31 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/32 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/33 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/34 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/35 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/36 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/37 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/38 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/39 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/40 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/41 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/42 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/43 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/44 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/45 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/46 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/47 Down None None None None No 1 0 cc4e.24b8.5c4c
1/1/48 Down None None None None No 1 0 cc4e.24b8.5c4c
1/2/1 Up Forward Full 40G None No N/A 0 cc4e.24b8.5c7d
1/2/2 Down None None None None No 10 0 cc4e.24b8.5c4c
1/2/3 Down None None None None No 10 0 cc4e.24b8.5c4c
1/2/4 Down None None None None No 1 0 cc4e.24b8.5c4c
1/2/5 Down None None None None No 1 0 cc4e.24b8.5c4c
1/2/6 Up Forward Full 40G None No N/A 0 cc4e.24b8.5c82
1/2/7 Down None None None None No 1 0 cc4e.24b8.5c4c
1/2/8 Down None None None None No 1 0 cc4e.24b8.5c4c
1/2/9 Down None None None None No 1 0 cc4e.24b8.5c4c
1/2/10 Down None None None None No 1 0 cc4e.24b8.5c4c
1/3/1 Down None None None 3 No 20 0 cc4e.24b8.5c4c To SAN Server 1
1/3/2 Down None None None 4 No 20 0 cc4e.24b8.5c4c To SAN Server 2
1/3/3 Down None None None None No 20 0 cc4e.24b8.5c4c
1/3/4 Down None None None None No 20 0 cc4e.24b8.5c4c
1/3/5 Down None None None None No 20 0 cc4e.24b8.5c4c
1/3/6 Down None None None None Yes 1 0 cc4e.24b8.5c4c
1/3/7 Down None None None None Yes 1 0 cc4e.24b8.5c4c
1/3/8 Down None None None None Yes 1 0 cc4e.24b8.5c4c
2/1/1 Down None None None 1 Yes N/A 0 cc4e.24b8.5c4c
2/1/2 Up Blocked Full 1G 2 Yes N/A 0 cc4e.24b8.5c4c
2/1/3 Down None None None None Yes 1 0 cc4e.24b8.5c4c
2/1/4 Down None None None None Yes 1 0 cc4e.24b8.5c4c
2/1/5 Down None None None None Yes 1 0 cc4e.24b8.5c4c
2/1/6 Down None None None None Yes 1 0 cc4e.24b8.5c4c
2/1/7 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/8 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/9 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/10 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/11 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/12 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/13 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/14 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/15 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/16 Down None None None None No 99 0 cc4e.24b8.5c4c
2/1/17 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/18 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/19 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/20 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/21 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/22 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/23 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/24 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/25 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/26 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/27 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/28 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/29 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/30 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/31 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/32 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/33 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/34 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/35 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/36 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/37 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/38 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/39 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/40 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/41 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/42 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/43 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/44 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/45 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/46 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/47 Down None None None None No 1 0 cc4e.24b8.5c4c
2/1/48 Down None None None None No 1 0 cc4e.24b8.5c4c
2/2/1 Up Forward Full 40G None No N/A 0 cc4e.24bc.a089
2/2/2 Down None None None None No 10 0 cc4e.24b8.5c4c
2/2/3 Down None None None None No 10 0 cc4e.24b8.5c4c
2/2/4 Down None None None None No 1 0 cc4e.24b8.5c4c
2/2/5 Down None None None None No 1 0 cc4e.24b8.5c4c
2/2/6 Up Forward Full 40G None No N/A 0 cc4e.24bc.a08e
2/2/7 Down None None None None No 1 0 cc4e.24b8.5c4c
2/2/8 Down None None None None No 1 0 cc4e.24b8.5c4c
2/2/9 Down None None None None No 1 0 cc4e.24b8.5c4c
2/2/10 Down None None None None No 1 0 cc4e.24b8.5c4c
2/3/1 Down None None None 3 No 20 0 cc4e.24b8.5c4c
2/3/2 Down None None None 4 No 20 0 cc4e.24b8.5c4c
2/3/3 Down None None None None No 20 0 cc4e.24b8.5c4c
2/3/4 Down None None None None No 20 0 cc4e.24b8.5c4c
2/3/5 Down None None None None No 20 0 cc4e.24b8.5c4c
2/3/6 Down None None None None Yes 1 0 cc4e.24b8.5c4c
2/3/7 Down None None None None Yes 1 0 cc4e.24b8.5c4c
2/3/8 Down None None None None Yes 1 0 cc4e.24b8.5c4c
mgmt1 Down None None None None No None 0 cc4e.24b8.5c4c

Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
ve1 Down N/A N/A N/A None N/A N/A N/A cc4e.24b8.5c4c
ve10 Down N/A N/A N/A None N/A N/A N/A cc4e.24b8.5c4c
ve20 Down N/A N/A N/A None N/A N/A N/A cc4e.24b8.5c4c
ve99 Down N/A N/A N/A None N/A N/A N/A cc4e.24b8.5c4c
ve2000 Down N/A N/A N/A None N/A N/A N/A cc4e.24b8.5c4c
ICX6610-48P Router>

 

[fohdeesha]

that's normal on the switch side, every port in a VLAN will have the mac of the VE interface

However I've never seen duplicate MACs like that in pfsense, are you sure you didn't accidentally set the MAC of one of the interfaces?

 

[PGlover]

just ordered a passive breakout DAC to test with to see if I can reproduce your breakout issue - it could be that DAC breakout is not supported on the rear ports, only optical

2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable 889028000663 | eBay

I purchased 2 of them.. I can ship you the unused one to try....

 

[PGlover]

that's normal on the switch side, every port in a VLAN will have the mac of the VE interface

However I've never seen duplicate MACs like that in pfsense, are you sure you didn't accidentally set the MAC of one of the interfaces?

no...

 

[fohdeesha]

I purchased 2 of them.. I can ship you the unused one to try....

too late just paid 30 seconds ago

 

[fohdeesha]

can you screenshot or copy/pastebin the entirety of the status > interfaces page in pfsense

 

[PGlover]

can you screenshot or copy/pastebin the entirety of the status > interfaces page in pfsense

Will do.. Even though 1 interface is LACP BLOCK, I am able to ping devices on my network..

 

[PGlover]

can you screenshot or copy/pastebin the entirety of the status > interfaces page in pfsense

I have the SYNC interface in pfSense unplugged right now...

 

[kapone]

just ordered a passive breakout DAC to test with to see if I can reproduce your breakout issue - it could be that DAC breakout is not supported on the rear ports, only optical

2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable 889028000663 | eBay

Incorrect. DAC breakout works perfectly on the rear ports.

Edit: This is the cable I'm using - DELL 27GG5 Force 10 QSFP+ 4X SFP+ 3M Copper Breakout Cable

 

[fohdeesha]

Hmmm, but do you also have stacking enabled (on just the 40gbE ports)? PGlover and now someone else in PM's cannot get the breakout ports to work with a passive DAC, they both can only seem to get one or two of the breakout channels to come up. PGlover has the cisco breakout cable and the guy in PM's has one from fiberstore

Really curious to figure this one out, I wonder what's going non

 

[PGlover]

Incorrect. DAC breakout works perfectly on the rear ports.

Edit: This is the cable I'm using - DELL 27GG5 Force 10 QSFP+ 4X SFP+ 3M Copper Breakout Cable

Do you have a stack configuration?

 

[fohdeesha]

@PGlover your LAN and guest interfaces both have the same MAC address, not sure how that happened. And also LACP does not support links with different speeds in the same group, so the switch is probably blocking the 100mbps interface for it's own good. Do you have a bad interface or cable on that link? Are you sure it's plugged into the right ports on the pfsense box (eg the same ports you created the lag with)

 

[PGlover]

Hmmm, but do you also have stacking enabled (on just the 40gbE ports)? PGlover and now someone else in PM's cannot get the breakout ports to work with a passive DAC, they both can only seem to get one or two of the breakout channels to come up. PGlover has the cisco breakout cable and the guy in PM's has one from fiberstore

Really curious to figure this one out, I wonder what's going non

The cable is Cisco compatible. It is a Monoprice cable. I was really upset because I thought the cable was from Cisco.

 

[fohdeesha]

The cable is Cisco compatible. It is a Monoprice cable. I was really upset because I thought the cable was from Cisco.

It really shouldn't matter, they're all the same electrically and brocade is vendor agnostic when it comes to optics and DACs. I'm starting to think you have something else going on

 

[PGlover]

@PGlover your LAN and guest interfaces both have the same MAC address, not sure how that happened. And also LACP does not support links with different speeds in the same group, so the switch is probably blocking the 100mbps interface for it's own good. Do you have a bad interface or cable on that link? Are you sure it's plugged into the right ports on the pfsense box (eg the same ports you created the lag with)

Yes.. It's a subinterface in pfSsense over LAGG1. LAGG1 carries the LAN and GUEST vlans (Vlan 2000 and Vlan 30).

 

[PGlover]

@PGlover your LAN and guest interfaces both have the same MAC address, not sure how that happened. And also LACP does not support links with different speeds in the same group, so the switch is probably blocking the 100mbps interface for it's own good. Do you have a bad interface or cable on that link? Are you sure it's plugged into the right ports on the pfsense box (eg the same ports you created the lag with)

Yes.. But let me double-check....