Brocade ICX-6610 with Bell Canada Fibe FTTH

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
I have no idea what I'm doing with this Brocade switch. (My "Cisco" SG500X refuses to handshake with the ISP provided ONT.)

With FTTH, Bell provides an SFP ONT and requires PPPoE to connect, on VLAN 35.
Yes, you CAN bypass the HomeHub 3000!! - Bell Canada | DSLReports Forums

On the Brocade, I have 1/3/8 configured to tag vlan 35, then I've tried:
a) an untagged/access vlan35 to my pfsense 2.4.3-p1
and
b) a tagged vlan35 with pfsense tagging vlan35
And the PPPoE won't connect either way. It just cycles endlessly, not receiving any response.

When I move the SFP into the Bell provided HH3000, it connects, so I know the ONT is connected/working, and I know my PPPoE login/password works (I'm not getting any auth failure anyway, just no response.)

PORT-VLAN 35, Name Fibe, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 3 4
Tagged Ports: (U1/M1) 2
Tagged Ports: (U1/M3) 1 8
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

icx6610(config)#show interfaces ethernet 1/3/8
10GigabitEthernet1/3/8 is up, line protocol is up
Port up for 18 hour(s) 59 minute(s) 25 second(s)
Hardware is 10GigabitEthernet, address is (removed)
Interface type is unknown
Configured speed 1Gbit, actual 1Gbit, configured duplex fdx, actual fdx
Member of 3 L2 VLANs, port is tagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level1, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 1024 bits/sec, 2 packets/sec, 0.00% utilization
300 second output rate: 1048 bits/sec, 1 packets/sec, 0.00% utilization
29522 packets input, 10640590 bytes, 0 no buffer
Received 0 broadcasts, 5 multicasts, 29517 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
56876 packets output, 6627605 bytes, 0 underruns
Transmitted 356 broadcasts, 28525 multicasts, 27995 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Egress queues:
Queue counters Queued packets Dropped Packets
0 56876 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0

So, what should I be doing on my Brocade to get it to take all VLAN 35 traffic, tag it, and egress? And what happens to untagged traffic that happens to come back in from the SFP?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
can you post your actual config? (output of show run)

At first glance your vlan config looks correct (tagged 35 on the WAN/SFP port, untagged 35 on the pfsense WAN connected port).

I see you have 5 ports in vlan 35, I would cut that down to JUST the pfsense port and the SFP port for now - some ISP's do not like seeing more than one MAC address/device on the end of their fiber. Also make sure you don't have any virtual interfaces defined for vlan 35 - you just want it to be an empty layer2 vlan.

I would also set the MTU back to 1500 for everything and retry to narrow it down, make sure the pfsense wan interface (in the pfsense appliance config) is also set to 1500

Also disable flow control on all ports involved with this vlan

Could also very well be a pfsense config issue, is the WAN interface a plain untagged interface with WAN type set to PPPoE? have you checked pfsense logs?

I've reverse engineered this ICX model to hell and back, if you catch me online sometime and can hop on teamviewer/google hangouts or something I can do some debug and see what's going on. Also let me know if you want/need licenses for it, to unlock the 10gbE ports and rear 40GbE ports and advanced routing features and all that

the port shows it's receiving and sending plenty of unicast traffic so I'd rule that side of it out
 

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
I was hoping you'd respond :)

The ISP requires PPPoE to connect, so I don't think the MACs are an issue, but I can disconnect the currently configured ports.

  1. I've tweaked MTU values on my devices, no avail.
  2. How do I set the flow control on the switch?
  3. pfSense ppp logs is full of retries.

Current configuration:
!
ver 08.0.30sT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-trunk 1/2/1 to 1/2/2
stack-trunk 1/2/6 to 1/2/7
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 30 name DSL by port
tagged ethe 1/1/2 ethe 1/3/1
!
vlan 34 name Bell by port
tagged ethe 1/3/8
!
vlan 35 name Fibe by port
tagged ethe 1/1/2 ethe 1/3/1 ethe 1/3/8
untagged ethe 1/1/3 to 1/1/4
!
vlan 36 name Bell36 by port
tagged ethe 1/3/8
!
!
!
!
!
qos tagged-priority 1 qosp0
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
jumbo
hostname icx6610
ip dhcp-client disable
ip dns server-address (removed)
ip route 0.0.0.0/0 (removed)
!
no telnet server
username root password .....
password-change any
!
!
web-management https
web-management list-menu
!
!
!
!
!
!
!
interface ethernet 1/1/2
port-name Trunk to SG500x
dual-mode
!
interface ethernet 1/3/1
dual-mode
!
interface ethernet 1/3/8
priority 1
!
interface ve 1
ip address (removed)
!
!
!
!
!
!
!
!
!
end
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
Did you mean to have multiple VLANs on the SFP port? For now to narrow it down I would make sure it's ONLY a member of vlan 35

also go into interface ethernet 1/3/8 and remove "priority 1" (just run "no priority 1" at the "interface ethernet 1/3/8" level). (can I ask why this was there?)

to turn off flow control just run "no flow-control" under each interface you want to turn off (just the sfp port and router copper port for now)

after making this many changes I would unplug the fiber from the SFP for a couple seconds and plug it back in just in case, just to reset any funky shit/tracking going on with their FTTH headend
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
I would also remove "qos tagged-priority 1 qosp0" from the global config. This switches backplane isn't oversubscribed, it'll do full duplex full throughput on all ports at the same time (and on the sfp ports, that's 10gbps), so when only pushing 1gbps internet service through it there's no need to start playing around with moving traffic to different hardware queues - that's better left for when you're going from traffic > 1gbps from a 10gbps port going down to a 1gbps port, or when you're riding right on the backplane and/or port line rate limit and want to limit jitter/buffer behavior etc
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
That post/thread seems to reference the dsl service, i don't remember seeing anyone needing to set priority in the fiber thread. have you tried it without that?
 

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
Yeah. My internet won't connect! :) pfSense has been trying repeatedly, and failing.

Bell uses the "same" infrastructure for the fiber and DSL services -- both use PPPoE over vlan 35, AFAICT.

Since I've been going CRAZY with this the last few days, I've found a magic incantation that allows my pfSense to connect:
  1. Based on the configuration above, I have the Bell provided router's LAN port plugged into 1/1/3, powered off.
  2. I set the pfSense to dial the PPPoE, which it will do happily, repeatedly, and fail to connect for 45 minutes+ (I just left it)
  3. Then I power on the Bell router, and while it's booting up, with 1 of it's LAN port connected to 1/1/3, the PPPoE session connects on pfSense (!!)
  4. Then I power off the Bell router and my PPPoE session still works, routing internet happily.
  5. Then I hang up the PPPoE, and it won't reconnect (back to step 1, above).
I have no idea what's going on. I probably need to start mirroring the traffic and inspecting the traffic frames by hand to see what the heck is going on.

My random guesses:
  1. Somehow during boot, it comes up like a dumb switch briefly, and just passes packets across all the VLANs, and somehow that's required. But no one else on the forums with the $20 TP-Link SFP to Ethernet media converter runs into this problem. They just connect the sfp into the converter, then Ethernet into pfSense and away they go with PPPoE, tagged vlan 35.
  2. My current pfSense installation is fubared, or there's a bug with pfSense PPPoE implementation. So I'm going to fire up a VM and try with a fresh installation, or try another Tomato router and see if it can connect when it's connected thru the Brocade.
  3. There's some kind of authentication that's new/being rolled out, where the access concentrators don't respond without some other handshake with the Bell router.
I'd love to hear your thoughts -- but most importantly: assuming none of these conspiracy theories are valid, do you see anything else I should do on the Brocade so that it is functionally equivalent to SFP -> sfp to ethernet media converter -> pfsense that is tagging vlan 35? Because that seems to work for everyone else in the world.

I ordered one of the media converters (sigh) so I can plug directly into pfSense and see if that still repros. Then I can know for sure that it's not a pfSense issue and/or a Brocade config issue.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
wow, that is really odd, having the routers LAN port plugged into that vlan makes it work?? is the router in BRIDGE mode when you do this? This makes me think their headend is looking for the MAC address of that router/bridge. If that's the case you can spoof it by just punching it into the interface config for the PFSENSE WAN interface.

Can you post your most recent switch config after the recommended changes? Assuming flow control, priority and others have been removed, then it is indeed acting as a dumb sfp (tagged vlan 35) >copper (untagged) adapter. There's some fastiron debug commands that will allow you print traffic from specific interfaces/ports/protocols to the console that could be helpful for this, that way you don't have to resort to adding a mirror port in that vlan and sending it to wireshark (however that would be very helpful)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
but again, if the (ISP) router is in BRIDGE mode, and plugging it into the vlan 35 (but untagged) makes it work, that tells me their headend is looking for something on that router box, hopefully just the mac address, because you can easily punch that into pfsense
 

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
And say I read the WAN MAC address off the sticker on the ISP provided router -- do I override MAC on the pfSense PPPoE adapter or the WAN adapter?

And how do I get a dump of connected MACs on which iface on the Brocade so I can confirm what the ISP router is, and confirmation that my spoofing is working?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
Does pfsense have separate WAN and PPPOE interfaces? that doesn't sound right, but it's been awhile since i've used pppoe on pfsense. I would start by spoofing it on the WAN interface, if that doesn't work set it back to default (just clear the mac box) then try the pppoe interface.

Under the interfaces dropdown in pfsense though I would think you're only supposed to have one WAN interface, and when you click it, the "type" dropdown should be ppppoe
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
to show all macs globally, just run "show mac-address", to show macs seen on a specific port, just do "show mac-address ethernet 1/1/1" (substitute your own port)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
I also see you're trunking vlan35 over to a cisco switch, in case you haven't already, make sure the ONLY two devices/ports under vlan 35 are the SFP port, and your pfsense copper port
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
OK, searching that thread, I did find a couple people who had to spoof the WAN MAC from their HH3000, so I'd be willing to bet that's what is going on here