1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Ubuntu 16.04 (Dell C6220) not playing together with Quanta LB6M..

Discussion in 'Networking' started by vrod, Jan 9, 2018.

  1. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    192
    Likes Received:
    27
    Hi guys,

    I'm trying to set up the networking on a ubuntu server, running off a dell c6220. On the c6220, there are 2x 10gb intel cards, both ports connected to port 7 and 8 in my quanta lb6m switch. For whatever reason, only the second interface, plugged into port 8 can manage to get network working... The first interface does not even have the ability to grab a DHCP address.

    The VLAN # is 1000 and I am using tagging. I have tried to configure the ports as switchports but same behaviour.

    Here's the LB6M config:
    interface 0/7
    description 'NODE02-NIC0'
    spanning-tree edgeport
    vlan participation include 1000
    vlan tagging 1000
    exit
    interface 0/8
    description 'NODE02-NIC1'
    spanning-tree edgeport
    vlan participation include 1000
    vlan tagging 1000
    exit
    "show vlan 1000" gives the following output:
    0/7 Include Include Tagged
    0/8 Include Include Tagged

    Ubuntu /etc/network/interfaces config
    auto lo
    iface lo inet loopback

    auto enp130s0f0
    iface enp130s0f0 inet manual

    auto enp130s0f1
    iface enp130s0f1 inet manual

    auto enp130s0f0.1000
    iface enp130s0f0.1000 inet dhcp
    vlan-raw-device enp130s0f0

    auto enp130s0f1.1000
    iface enp130s0f1.1000 inet dhcp
    vlan-raw-device enp130s0f1

    enp130s0f0.1000 cannot get a DHCP address, but enp130s0f1.1000 has no problems getting one.

    The ports are connected as such

    LB6M Port 7 -> enp130s0f0
    LB6M Port 8 -> enp130s0f1

    When I do a tcpdump on the enp130s0f0.1000 interface, I see traffic flowing without issues. I can see traffic going between other devices.

    The weird thing is also: IF I configure a static adddress on enp130s0f0.1000 and ping another server = no response. However, pinging the LB6M switch is successful, both ways.

    Does anyone have an idea of what the heck is going on here? I have spent the entire last night trying to figure this out and it's starting to drive me insane. Would really be happy if someone could give some advice on this.

    Best regards,
    Chris
     
    #1
  2. AT S37=0

    AT S37=0 Member

    Joined:
    Jul 12, 2016
    Messages:
    56
    Likes Received:
    22
    Vrod, it looks like a layer 3 problem, the routes between the hosts have multiple paths, so only one is used.

    What happens if you bring down port8 and just use port7?

    You may need a LAG/portchannel between the two. The Linux side looks like this:

    Code:
    auto eth0
        iface eth0 inet manual
        bond-master bond0
    
    auto eth1
         iface eth1 inet manual
         bond-master bond0
    
    auto bond0
         iface bond0 inet manual
    
    auto bond0.1000
         iface bond0.1000 inet dhcp
         vlan-raw-device bond0
    
    The LB6M side looks like this:

    Code:
    interface 1/1
    vlan participation include 1000
    vlan tagging 1000
    !
    interface 0/7
    no auto-negotiate
    addport 1/1
    exit
    !
    interface 0/8
    no auto-negotiate
    addport 1/1
    exit
    
    I'm not running these at the moment, so I didn't test the above. Hopefully it helps.
     
    #2
  3. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    192
    Likes Received:
    27
    Hello,

    Looks like disabling port 8 made the primary interface work somewhat again. I then configured LACP on the switch and in ubuntu, but are getting sporadic timeouts...

    Ping to server from client
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Request timed out.
    Request timed out.
    Request timed out.
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Request timed out.
    Request timed out.
    Request timed out.
    Request timed out.
    Request timed out.
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
    Reply from 1.2.3.4: bytes=32 time<1ms TTL=64

    Here's the linux config

    auto enp130s0f0
    iface enp130s0f0 inet manual
    bond-master bond0

    auto enp130s0f1
    iface enp130s0f1 inet manual
    bond-master bond0

    auto bond0
    iface bond0 inet manual
    bond-slaves enp130s0f0 enp130s0f1
    bond-mode 4
    bond-miimon 100
    bond-lacp-rate 1

    auto bond0.1000
    iface bond0.1000 inet static
    address 1.2.3.4
    netmask 255.255.255.0

    And the switch config.....

    interface 0/7
    no auto-negotiate
    addport 1/5
    exit
    interface 0/8
    no auto-negotiate
    addport 1/5
    exit

    - - -

    interface 1/5
    description 'STORAGE02-LACP'
    no port-channel static
    vlan acceptframe vlanonly
    vlan participation include 1000
    vlan tagging 1000
    exit

    Do i need to add something else to the configuration?
     
    #3
  4. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    152
    Likes Received:
    99
    brocade firmware :p
     
    #4
    mangodoc likes this.
  5. AT S37=0

    AT S37=0 Member

    Joined:
    Jul 12, 2016
    Messages:
    56
    Likes Received:
    22
    Somewhat? It should work :)

    In the interface config, these may be useful:

    Code:
    interface 0/7
    port lacptimeout actor short
    port lacptimeout partner short
    no lacp actor admin state longtimeout
    no lacp partner admin state longtimeout
    
    ! same for interface 0/8
    
    Do a 'show port-channel' on the switch, see if the links are disabled when the ping is timing out.

    There is a similar status in linux 'cat /proc/net/bond0/info'

    It looks like the LB6M is defaulting to long timeouts and your linux config is set for fast, according to this: https://www.kernel.org/doc/Documentation/networking/bonding.txt

    I thought it would negotiate this if both sides are active, which it looks like they are.
     
    #5
  6. AT S37=0

    AT S37=0 Member

    Joined:
    Jul 12, 2016
    Messages:
    56
    Likes Received:
    22
    Is it possible to extract the userland from the brocade? They are probably using the same Linux kernel and the same Broadcom SDK underneath. That might make it reversable.
     
    #6
  7. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    192
    Likes Received:
    27
    I think I might have gotten it to work now.. i’ll still set the options you suggested. :) thank you!
    And yes you can flash the brocade firmware onto the lb6m. He made a topic about it here
     
    #7
  8. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    192
    Likes Received:
    27
    It’s in the making, just gotta find my damn console cable! :D
     
    #8
    fohdeesha likes this.
  9. AT S37=0

    AT S37=0 Member

    Joined:
    Jul 12, 2016
    Messages:
    56
    Likes Received:
    22
    Cool.

    The way these switches OSes work is sort of like a Live CD. The filesystem is loaded into a ramdisk and persistent storage is mounted for only for config files. The brocade firmware requires a special version of 'grub' that can't be removed -- the bootloader. If you can extract the files from the brocade ramdisk, you can create your own ramdisk using the bootloader that is installed on the quanta and get rid of the evil brocade bootloader that can't be removed.

    There are a lot of LB6M out there, because it is meant to be upgraded. The googles and amazons created their own OSes through their own skunkworks. Us poor plebes are stuck with an OS that nobody uses. There are some LB6Mes on ebay with the flashes drilled out because the cloud skunkworks don't want their trade secrets to be revealed. I think the skunkworks OS will eventually leak out somewhere.
     
    #9
  10. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    152
    Likes Received:
    99
    It can be removed (and updated) no problem, I've already written over the brocade bootloader contents :) . Brocade's bootloader is just rebranded uboot, some of the flash copy commands are still hidden under there. I just need to get it in an easy to do step and it'll be added to the guide
     
    #10
  11. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    192
    Likes Received:
    27
    So an update... i went to the datacenter after work and flashed the switch. It all works perfectly now! Might look into utilizing the L3 routing feature.
     
    #11
    fohdeesha likes this.
  12. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    152
    Likes Received:
    99
    ha! I was mostly kidding but that's pretty funny. I wonder what it didn't like in the fastpath config
     
    #12
  13. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    192
    Likes Received:
    27
    I do think it had something to do with the LACP settings on the switch... But the fact that the fastpath system seemed rather buggy and that I'm now onto an actual "supported" system with proper documentation... Not to say that I can finally throw the 8 port 1gbit hub away that I have needed for my management port interfaces on the servers because the brocade firmware supports SFP -> RJ45 adapters... I also did see an option to set the port to 100tx, so looks promising. :)
     
    #13
    fohdeesha likes this.

Share This Page