Brocade ICX-6610 with Bell Canada Fibe FTTH

Discussion in 'Networking' started by j_h_o, Jul 1, 2018.

  1. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    I have no idea what I'm doing with this Brocade switch. (My "Cisco" SG500X refuses to handshake with the ISP provided ONT.)

    With FTTH, Bell provides an SFP ONT and requires PPPoE to connect, on VLAN 35.
    Yes, you CAN bypass the HomeHub 3000!! - Bell Canada | DSLReports Forums

    On the Brocade, I have 1/3/8 configured to tag vlan 35, then I've tried:
    a) an untagged/access vlan35 to my pfsense 2.4.3-p1
    and
    b) a tagged vlan35 with pfsense tagging vlan35
    And the PPPoE won't connect either way. It just cycles endlessly, not receiving any response.

    When I move the SFP into the Bell provided HH3000, it connects, so I know the ONT is connected/working, and I know my PPPoE login/password works (I'm not getting any auth failure anyway, just no response.)

    PORT-VLAN 35, Name Fibe, Priority level0, Spanning tree Off
    Untagged Ports: (U1/M1) 3 4
    Tagged Ports: (U1/M1) 2
    Tagged Ports: (U1/M3) 1 8
    Uplink Ports: None
    DualMode Ports: None
    Mac-Vlan Ports: None
    Monitoring: Disabled

    icx6610(config)#show interfaces ethernet 1/3/8
    10GigabitEthernet1/3/8 is up, line protocol is up
    Port up for 18 hour(s) 59 minute(s) 25 second(s)
    Hardware is 10GigabitEthernet, address is (removed)
    Interface type is unknown
    Configured speed 1Gbit, actual 1Gbit, configured duplex fdx, actual fdx
    Member of 3 L2 VLANs, port is tagged, port state is FORWARDING
    BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
    Link Error Dampening is Disabled
    STP configured to ON, priority is level1, mac-learning is enabled
    Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is enabled
    Mirror disabled, Monitor disabled
    Mac-notification is disabled
    Not member of any active trunks
    Not member of any configured trunks
    No port name
    MTU 10200 bytes, encapsulation ethernet
    300 second input rate: 1024 bits/sec, 2 packets/sec, 0.00% utilization
    300 second output rate: 1048 bits/sec, 1 packets/sec, 0.00% utilization
    29522 packets input, 10640590 bytes, 0 no buffer
    Received 0 broadcasts, 5 multicasts, 29517 unicasts
    0 input errors, 0 CRC, 0 frame, 0 ignored
    0 runts, 0 giants
    56876 packets output, 6627605 bytes, 0 underruns
    Transmitted 356 broadcasts, 28525 multicasts, 27995 unicasts
    0 output errors, 0 collisions
    Relay Agent Information option: Disabled
    Egress queues:
    Queue counters Queued packets Dropped Packets
    0 56876 0
    1 0 0
    2 0 0
    3 0 0
    4 0 0
    5 0 0
    6 0 0
    7 0 0

    So, what should I be doing on my Brocade to get it to take all VLAN 35 traffic, tag it, and egress? And what happens to untagged traffic that happens to come back in from the SFP?
     
    #1
  2. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    can you post your actual config? (output of show run)

    At first glance your vlan config looks correct (tagged 35 on the WAN/SFP port, untagged 35 on the pfsense WAN connected port).

    I see you have 5 ports in vlan 35, I would cut that down to JUST the pfsense port and the SFP port for now - some ISP's do not like seeing more than one MAC address/device on the end of their fiber. Also make sure you don't have any virtual interfaces defined for vlan 35 - you just want it to be an empty layer2 vlan.

    I would also set the MTU back to 1500 for everything and retry to narrow it down, make sure the pfsense wan interface (in the pfsense appliance config) is also set to 1500

    Also disable flow control on all ports involved with this vlan

    Could also very well be a pfsense config issue, is the WAN interface a plain untagged interface with WAN type set to PPPoE? have you checked pfsense logs?

    I've reverse engineered this ICX model to hell and back, if you catch me online sometime and can hop on teamviewer/google hangouts or something I can do some debug and see what's going on. Also let me know if you want/need licenses for it, to unlock the 10gbE ports and rear 40GbE ports and advanced routing features and all that

    the port shows it's receiving and sending plenty of unicast traffic so I'd rule that side of it out
     
    #2
  3. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    I was hoping you'd respond :)

    The ISP requires PPPoE to connect, so I don't think the MACs are an issue, but I can disconnect the currently configured ports.

    1. I've tweaked MTU values on my devices, no avail.
    2. How do I set the flow control on the switch?
    3. pfSense ppp logs is full of retries.

    Current configuration:
    !
    ver 08.0.30sT7f3
    !
    stack unit 1
    module 1 icx6610-48p-poe-port-management-module
    module 2 icx6610-qsfp-10-port-160g-module
    module 3 icx6610-8-port-10g-dual-mode-module
    stack-trunk 1/2/1 to 1/2/2
    stack-trunk 1/2/6 to 1/2/7
    !
    !
    !
    !
    vlan 1 name DEFAULT-VLAN by port
    router-interface ve 1
    !
    vlan 30 name DSL by port
    tagged ethe 1/1/2 ethe 1/3/1
    !
    vlan 34 name Bell by port
    tagged ethe 1/3/8
    !
    vlan 35 name Fibe by port
    tagged ethe 1/1/2 ethe 1/3/1 ethe 1/3/8
    untagged ethe 1/1/3 to 1/1/4
    !
    vlan 36 name Bell36 by port
    tagged ethe 1/3/8
    !
    !
    !
    !
    !
    qos tagged-priority 1 qosp0
    aaa authentication web-server default local
    aaa authentication enable default local
    aaa authentication login default local
    jumbo
    hostname icx6610
    ip dhcp-client disable
    ip dns server-address (removed)
    ip route 0.0.0.0/0 (removed)
    !
    no telnet server
    username root password .....
    password-change any
    !
    !
    web-management https
    web-management list-menu
    !
    !
    !
    !
    !
    !
    !
    interface ethernet 1/1/2
    port-name Trunk to SG500x
    dual-mode
    !
    interface ethernet 1/3/1
    dual-mode
    !
    interface ethernet 1/3/8
    priority 1
    !
    interface ve 1
    ip address (removed)
    !
    !
    !
    !
    !
    !
    !
    !
    !
    end
     
    #3
  4. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    Did you mean to have multiple VLANs on the SFP port? For now to narrow it down I would make sure it's ONLY a member of vlan 35

    also go into interface ethernet 1/3/8 and remove "priority 1" (just run "no priority 1" at the "interface ethernet 1/3/8" level). (can I ask why this was there?)

    to turn off flow control just run "no flow-control" under each interface you want to turn off (just the sfp port and router copper port for now)

    after making this many changes I would unplug the fiber from the SFP for a couple seconds and plug it back in just in case, just to reset any funky shit/tracking going on with their FTTH headend
     
    #4
    Last edited: Jul 1, 2018
  5. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    I would also remove "qos tagged-priority 1 qosp0" from the global config. This switches backplane isn't oversubscribed, it'll do full duplex full throughput on all ports at the same time (and on the sfp ports, that's 10gbps), so when only pushing 1gbps internet service through it there's no need to start playing around with moving traffic to different hardware queues - that's better left for when you're going from traffic > 1gbps from a 10gbps port going down to a 1gbps port, or when you're riding right on the backplane and/or port line rate limit and want to limit jitter/buffer behavior etc
     
    #5
    Last edited: Jul 1, 2018
  6. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    Because I don't know what I'm doing with the switch :)

    How do I set VLAN 35 (tagged) and 802.1q priority = 1 on 1/3/8?
     
    #6
  7. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    why do you need to set the priority?
     
    #7
  8. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    #8
  9. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    That post/thread seems to reference the dsl service, i don't remember seeing anyone needing to set priority in the fiber thread. have you tried it without that?
     
    #9
  10. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    Yeah. My internet won't connect! :) pfSense has been trying repeatedly, and failing.

    Bell uses the "same" infrastructure for the fiber and DSL services -- both use PPPoE over vlan 35, AFAICT.

    Since I've been going CRAZY with this the last few days, I've found a magic incantation that allows my pfSense to connect:
    1. Based on the configuration above, I have the Bell provided router's LAN port plugged into 1/1/3, powered off.
    2. I set the pfSense to dial the PPPoE, which it will do happily, repeatedly, and fail to connect for 45 minutes+ (I just left it)
    3. Then I power on the Bell router, and while it's booting up, with 1 of it's LAN port connected to 1/1/3, the PPPoE session connects on pfSense (!!)
    4. Then I power off the Bell router and my PPPoE session still works, routing internet happily.
    5. Then I hang up the PPPoE, and it won't reconnect (back to step 1, above).
    I have no idea what's going on. I probably need to start mirroring the traffic and inspecting the traffic frames by hand to see what the heck is going on.

    My random guesses:
    1. Somehow during boot, it comes up like a dumb switch briefly, and just passes packets across all the VLANs, and somehow that's required. But no one else on the forums with the $20 TP-Link SFP to Ethernet media converter runs into this problem. They just connect the sfp into the converter, then Ethernet into pfSense and away they go with PPPoE, tagged vlan 35.
    2. My current pfSense installation is fubared, or there's a bug with pfSense PPPoE implementation. So I'm going to fire up a VM and try with a fresh installation, or try another Tomato router and see if it can connect when it's connected thru the Brocade.
    3. There's some kind of authentication that's new/being rolled out, where the access concentrators don't respond without some other handshake with the Bell router.
    I'd love to hear your thoughts -- but most importantly: assuming none of these conspiracy theories are valid, do you see anything else I should do on the Brocade so that it is functionally equivalent to SFP -> sfp to ethernet media converter -> pfsense that is tagging vlan 35? Because that seems to work for everyone else in the world.

    I ordered one of the media converters (sigh) so I can plug directly into pfSense and see if that still repros. Then I can know for sure that it's not a pfSense issue and/or a Brocade config issue.
     
    #10
    Last edited: Jul 1, 2018
  11. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    wow, that is really odd, having the routers LAN port plugged into that vlan makes it work?? is the router in BRIDGE mode when you do this? This makes me think their headend is looking for the MAC address of that router/bridge. If that's the case you can spoof it by just punching it into the interface config for the PFSENSE WAN interface.

    Can you post your most recent switch config after the recommended changes? Assuming flow control, priority and others have been removed, then it is indeed acting as a dumb sfp (tagged vlan 35) >copper (untagged) adapter. There's some fastiron debug commands that will allow you print traffic from specific interfaces/ports/protocols to the console that could be helpful for this, that way you don't have to resort to adding a mirror port in that vlan and sending it to wireshark (however that would be very helpful)
     
    #11
  12. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    Sorry, can you clarify: what changes are you asking me to do on pfSense?
     
    #12
  13. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    sorry, check my post again, edited some bits - meant to say switch config
     
    #13
  14. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    but again, if the (ISP) router is in BRIDGE mode, and plugging it into the vlan 35 (but untagged) makes it work, that tells me their headend is looking for something on that router box, hopefully just the mac address, because you can easily punch that into pfsense
     
    #14
  15. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    And say I read the WAN MAC address off the sticker on the ISP provided router -- do I override MAC on the pfSense PPPoE adapter or the WAN adapter?

    And how do I get a dump of connected MACs on which iface on the Brocade so I can confirm what the ISP router is, and confirmation that my spoofing is working?
     
    #15
  16. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    Does pfsense have separate WAN and PPPOE interfaces? that doesn't sound right, but it's been awhile since i've used pppoe on pfsense. I would start by spoofing it on the WAN interface, if that doesn't work set it back to default (just clear the mac box) then try the pppoe interface.

    Under the interfaces dropdown in pfsense though I would think you're only supposed to have one WAN interface, and when you click it, the "type" dropdown should be ppppoe
     
    #16
  17. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    to show all macs globally, just run "show mac-address", to show macs seen on a specific port, just do "show mac-address ethernet 1/1/1" (substitute your own port)
     
    #17
  18. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    I also see you're trunking vlan35 over to a cisco switch, in case you haven't already, make sure the ONLY two devices/ports under vlan 35 are the SFP port, and your pfsense copper port
     
    #18
  19. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    670
    Likes Received:
    460
    OK, searching that thread, I did find a couple people who had to spoof the WAN MAC from their HH3000, so I'd be willing to bet that's what is going on here
     
    #19
  20. j_h_o

    j_h_o Active Member

    Joined:
    Apr 21, 2015
    Messages:
    312
    Likes Received:
    60
    How can I flush the list of discovered MACs on the switch?
     
    #20
Similar Threads: Brocade ICX-6610
Forum Title Date
Networking LB6M (brocade firmware) Trouble Routing Vlans Jul 13, 2018
Networking Brocade ICX6450 / ICX6610 / ETC Jul 12, 2018
Networking Having Issues With Brocade Switch - Not Booting May 5, 2018
Networking Running out of 10g ports on Brocade ICX 6610, what to do? Jan 12, 2018
Networking SFP+ DAC showing as 1000base-CX (Aruba S2500, Brocade BR1020) Dec 27, 2017

Share This Page