Ubuntu 16.04 (Dell C6220) not playing together with Quanta LB6M..

vrod

Active Member
Jan 18, 2015
233
35
28
29
Hi guys,

I'm trying to set up the networking on a ubuntu server, running off a dell c6220. On the c6220, there are 2x 10gb intel cards, both ports connected to port 7 and 8 in my quanta lb6m switch. For whatever reason, only the second interface, plugged into port 8 can manage to get network working... The first interface does not even have the ability to grab a DHCP address.

The VLAN # is 1000 and I am using tagging. I have tried to configure the ports as switchports but same behaviour.

Here's the LB6M config:
interface 0/7
description 'NODE02-NIC0'
spanning-tree edgeport
vlan participation include 1000
vlan tagging 1000
exit
interface 0/8
description 'NODE02-NIC1'
spanning-tree edgeport
vlan participation include 1000
vlan tagging 1000
exit
"show vlan 1000" gives the following output:
0/7 Include Include Tagged
0/8 Include Include Tagged

Ubuntu /etc/network/interfaces config
auto lo
iface lo inet loopback

auto enp130s0f0
iface enp130s0f0 inet manual

auto enp130s0f1
iface enp130s0f1 inet manual

auto enp130s0f0.1000
iface enp130s0f0.1000 inet dhcp
vlan-raw-device enp130s0f0

auto enp130s0f1.1000
iface enp130s0f1.1000 inet dhcp
vlan-raw-device enp130s0f1

enp130s0f0.1000 cannot get a DHCP address, but enp130s0f1.1000 has no problems getting one.

The ports are connected as such

LB6M Port 7 -> enp130s0f0
LB6M Port 8 -> enp130s0f1

When I do a tcpdump on the enp130s0f0.1000 interface, I see traffic flowing without issues. I can see traffic going between other devices.

The weird thing is also: IF I configure a static adddress on enp130s0f0.1000 and ping another server = no response. However, pinging the LB6M switch is successful, both ways.

Does anyone have an idea of what the heck is going on here? I have spent the entire last night trying to figure this out and it's starting to drive me insane. Would really be happy if someone could give some advice on this.

Best regards,
Chris
 
Hi guys,

When I do a tcpdump on the enp130s0f0.1000 interface, I see traffic flowing without issues. I can see traffic going between other devices.

The weird thing is also: IF I configure a static adddress on enp130s0f0.1000 and ping another server = no response. However, pinging the LB6M switch is successful, both ways.
Vrod, it looks like a layer 3 problem, the routes between the hosts have multiple paths, so only one is used.

What happens if you bring down port8 and just use port7?

You may need a LAG/portchannel between the two. The Linux side looks like this:

Code:
auto eth0
    iface eth0 inet manual
    bond-master bond0

auto eth1
     iface eth1 inet manual
     bond-master bond0

auto bond0
     iface bond0 inet manual

auto bond0.1000
     iface bond0.1000 inet dhcp
     vlan-raw-device bond0
The LB6M side looks like this:

Code:
interface 1/1
vlan participation include 1000
vlan tagging 1000
!
interface 0/7
no auto-negotiate
addport 1/1
exit
!
interface 0/8
no auto-negotiate
addport 1/1
exit
I'm not running these at the moment, so I didn't test the above. Hopefully it helps.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
Hello,

Looks like disabling port 8 made the primary interface work somewhat again. I then configured LACP on the switch and in ubuntu, but are getting sporadic timeouts...

Ping to server from client
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64
Reply from 1.2.3.4: bytes=32 time<1ms TTL=64

Here's the linux config

auto enp130s0f0
iface enp130s0f0 inet manual
bond-master bond0

auto enp130s0f1
iface enp130s0f1 inet manual
bond-master bond0

auto bond0
iface bond0 inet manual
bond-slaves enp130s0f0 enp130s0f1
bond-mode 4
bond-miimon 100
bond-lacp-rate 1

auto bond0.1000
iface bond0.1000 inet static
address 1.2.3.4
netmask 255.255.255.0

And the switch config.....

interface 0/7
no auto-negotiate
addport 1/5
exit
interface 0/8
no auto-negotiate
addport 1/5
exit

- - -

interface 1/5
description 'STORAGE02-LACP'
no port-channel static
vlan acceptframe vlanonly
vlan participation include 1000
vlan tagging 1000
exit

Do i need to add something else to the configuration?
 
Hello,

Looks like disabling port 8 made the primary interface work somewhat again. I then configured LACP on the switch and in ubuntu, but are getting sporadic timeouts...

Do i need to add something else to the configuration?
Somewhat? It should work :)

In the interface config, these may be useful:

Code:
interface 0/7
port lacptimeout actor short
port lacptimeout partner short
no lacp actor admin state longtimeout
no lacp partner admin state longtimeout

! same for interface 0/8
Do a 'show port-channel' on the switch, see if the links are disabled when the ping is timing out.

There is a similar status in linux 'cat /proc/net/bond0/info'

It looks like the LB6M is defaulting to long timeouts and your linux config is set for fast, according to this: https://www.kernel.org/doc/Documentation/networking/bonding.txt

I thought it would negotiate this if both sides are active, which it looks like they are.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
Is it possible to extract the userland from the brocade? They are probably using the same Linux kernel and the same Broadcom SDK underneath. That might make it reversable.
I think I might have gotten it to work now.. i’ll still set the options you suggested. :) thank you!
And yes you can flash the brocade firmware onto the lb6m. He made a topic about it here
 
I think I might have gotten it to work now.. i’ll still set the options you suggested. :) thank you!
And yes you can flash the brocade firmware onto the lb6m. He made a topic about it here
Cool.

The way these switches OSes work is sort of like a Live CD. The filesystem is loaded into a ramdisk and persistent storage is mounted for only for config files. The brocade firmware requires a special version of 'grub' that can't be removed -- the bootloader. If you can extract the files from the brocade ramdisk, you can create your own ramdisk using the bootloader that is installed on the quanta and get rid of the evil brocade bootloader that can't be removed.

There are a lot of LB6M out there, because it is meant to be upgraded. The googles and amazons created their own OSes through their own skunkworks. Us poor plebes are stuck with an OS that nobody uses. There are some LB6Mes on ebay with the flashes drilled out because the cloud skunkworks don't want their trade secrets to be revealed. I think the skunkworks OS will eventually leak out somewhere.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,674
113
31
fohdeesha.com
It can be removed (and updated) no problem, I've already written over the brocade bootloader contents :) . Brocade's bootloader is just rebranded uboot, some of the flash copy commands are still hidden under there. I just need to get it in an easy to do step and it'll be added to the guide
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
So an update... i went to the datacenter after work and flashed the switch. It all works perfectly now! Might look into utilizing the L3 routing feature.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,674
113
31
fohdeesha.com
So an update... i went to the datacenter after work and flashed the switch. It all works perfectly now! Might look into utilizing the L3 routing feature.
ha! I was mostly kidding but that's pretty funny. I wonder what it didn't like in the fastpath config
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
ha! I was mostly kidding but that's pretty funny. I wonder what it didn't like in the fastpath config
I do think it had something to do with the LACP settings on the switch... But the fact that the fastpath system seemed rather buggy and that I'm now onto an actual "supported" system with proper documentation... Not to say that I can finally throw the 8 port 1gbit hub away that I have needed for my management port interfaces on the servers because the brocade firmware supports SFP -> RJ45 adapters... I also did see an option to set the port to 100tx, so looks promising. :)
 
  • Like
Reactions: fohdeesha