Quanta LB6M (10GbE) -- Discussion

CKat

New Member
May 20, 2017
19
1
3
42
My experience with VLANs has been similar to yours. When you set it up the first time it just seems to work, but when re-configuring something goes haywire. I think that something in the processing of tagged frames hangs around in the volatile regions of the switch when you unset some options related to VLAN. I have found cutting power to the device and letting it load back up will restore it to sanity.
I restarted the switch ("reload" command), it did not fix anything for me.

My issue currently is that the VLAN traffic is not passing across to switch before this one. I can ping gateway on the VLAN, but cannot ping any servers from the switch or anything else connected to it.

I want to start over -- if I do "clear config", will it also delete the ip settings?

I ask because if it defaults to 192.168.2.1 instead of my network, I will lose access to everything...
Will telnet continue to work on the existing IP (192.168.5.100) after clear config? I cannot seem to find what the clear config will actually clear.
 

J Hart

Active Member
Apr 23, 2015
145
100
43
42
I restarted the switch ("reload" command), it did not fix anything for me.

My issue currently is that the VLAN traffic is not passing across to switch before this one. I can ping gateway on the VLAN, but cannot ping any servers from the switch or anything else connected to it.

I want to start over -- if I do "clear config", will it also delete the ip settings?

I ask because if it defaults to 192.168.2.1 instead of my network, I will lose access to everything...
Will telnet continue to work on the existing IP (192.168.5.100) after clear config? I cannot seem to find what the clear config will actually clear.
My problems sounded a lot like yours. I reconfigured some VLANs and ended up where packets would go through the switch in one direction, but not the other. I ended up removing the switch and replacing it with a different LB6M which was wiped and transferring the config back via TFTP. Worked fine. The original one when plugged back in and config reloaded also worked fine. I think that some of the configutation details are present in the Broadcom chips and are not cleared out properly unless the switch is hard power cycled. At least that seems to be the case with the firmware they came with. I guess that the Brocade one is probably doing it correctly.
 

CKat

New Member
May 20, 2017
19
1
3
42
My problems sounded a lot like yours. I reconfigured some VLANs and ended up where packets would go through the switch in one direction, but not the other. I ended up removing the switch and replacing it with a different LB6M which was wiped and transferring the config back via TFTP. Worked fine. The original one when plugged back in and config reloaded also worked fine. I think that some of the configutation details are present in the Broadcom chips and are not cleared out properly unless the switch is hard power cycled. At least that seems to be the case with the firmware they came with. I guess that the Brocade one is probably doing it correctly.
I only have 1 LB6M, so cannot replicate this. However, I pulled the cord on the LB6m to power off completely, and then let it come back up.. It did not fix my issues.

This is really frustrating... I have been facing this issue for over 2 weeks now and cannot make this thing work.

If I do a "clear config", would it reset the IP address too? Because this means I may have to travel to the data center which is 2 hours flight away and needs lots of planning...
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,470
2,628
113
31
fohdeesha.com
clear config would clear everything so yes i'd imagine that would include the IP. Does your DC not provide remote hands? They should have employees on hand who can serial console into your switch for you
 

CKat

New Member
May 20, 2017
19
1
3
42
They do provide remote hands. However I have never used that for actual configuration changes - mostly been for power cycling etc.
I will give this a try...
 

CKat

New Member
May 20, 2017
19
1
3
42
shouldn't be a problem, it's pretty much their job to walk around with a laptop and serial cable :p

OK, did a "Clear config", which wiped all settings. Reconfigured the switch.
Still cannot ping servers on same VLAN.

My setup is below.
upload_2018-6-30_10-54-9.png

After clear config, in the Quanta switch, my running-config is below. Super Simple - just want to be able to ping between 2 servers across switch (xenserver1 and NFS Server 1):

Is the below setting enough to pass the traffic through interface 0/25 to 0/3?

My expectation is that the path taken would be the one highlighted in red.


Code:
(FASTPATH Routing) #show running-config
!Current Configuration:
!
!System Description "Quanta LB6M, 1.2.0.14, Linux 2.6.21.7"
!System Software Version "1.2.0.14"
!System Up Time          "0 days 21 hrs 37 mins 29 secs"
!Additional Packages     FASTPATH QOS
!Current SNTP Synchronized Time: Not Synchronized
!
vlan database
vlan 10-11
vlan name 10 "vlan10"
vlan name 11 "vlan11"
exit
configure
aaa authentication enable "enableList" enable
line console
exit
line telnet
exit
line ssh
exit
spanning-tree configuration name "C8-0A-A9-03-B6-76"
!
interface 0/3

--More-- or (q)uit

vlan participation include 10-11
vlan tagging 10-11
exit
interface 0/25
vlan participation include 10-11
vlan tagging 10-11
exit
router rip
exit
router ospf
exit
exit

NOW _ I did some digging from xenserver 1 (NOTE - this is the XCP-ng one)
I tried to do a arp-scan.

10.50.11.35 is the NFS Server 1 IP address
However, note that the arp-scan returns 2 mac addresses. The one ending in :ec is the correct mac address. The one ending in :0e is the mac address of PORT 1 of the Dell switch

I am thinking that could be the cause of he failed pings. Because if you look at the tcpdump, it is responding with that mac address, not the correct one.

I tried restarting the switches, restarting the servers as well. I also did an arp cache clear, mac address table clearing...
What am I missing?

Code:
[root@nexusvm3 ~]# arp-scan -I xapi1 -l | grep 10.50.11.35
10.50.11.35     24:5e:be:04:f0:ec       (Unknown)
10.50.11.35     1c:74:0d:fa:9c:0e       (Unknown) (DUP: 2)


[root@nexusvm3 ~]# tcpdump -i xapi1 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xapi1, link-type EN10MB (Ethernet), capture size 65535 bytes
10:37:10.970251 ARP, Reply 10.50.11.105 is-at 1c:74:0d:fa:9c:0e (oui Unknown), length 42
10:37:11.010253 ARP, Reply 10.50.11.35 is-at 1c:74:0d:fa:9c:0e (oui Unknown), length 42
10:37:11.050257 ARP, Reply 10.50.11.20 is-at 1c:74:0d:fa:9c:0e (oui Unknown), length 42
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,470
2,628
113
31
fohdeesha.com
It's hard to understand what's going on without having a better view of your setup, but if the dell switch is answering for the IP of 10.50.11.35 when it shouldn't be (it's supposed to belong to NFS server), that pretty much narrows it down to an issue or misconfiguration of the dell switch. has it been accidentally configured with an IP, either static or dhcp?

it seems like the dell switch is trying to do some type of layer3 op/routing or proxy arp on vlan 11, if it's answering ARPs in that vlan

Out of curiosity, what ethernet NIC do you have in the xenserver machine?
 
Last edited:

BlueTip

New Member
Feb 4, 2017
5
6
3
Hi, TheBloke,
To answer your question, I haven't hooked up the tach for the left fan (opposite the power supply), but I'm getting correct PWM control, reporting of fan status, and temperature with the Arctic F8 fans (33C intake, 36C idle exhaust at fan speed 3 - see below).

SSH@10g-sw1(config)#show chassis
Power supply 1 not present
Power supply 2 (NA - NA - Regular) present, status ok

fan 1 failed
Fan 2 ok, speed (manual): 1<->2<->[[3]]
Fan 3 ok, speed (manual): 1<->2<->[[3]]

Fan controlled temperature: 36.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->30 deg-C
Speed 2: 25<----->40 deg-C
Speed 3: 35<----->90 deg-C (shutdown)


Exhaust Side Temperature Readings:
Current temperature : 36.0 deg-C
Warning level.......: 80.0 deg-C
Shutdown level......: 90.0 deg-C
Intake Side Temperature Readings:
Current temperature : 33.0 deg-C
Boot Prom MAC: 00e0.52c1.ec8b
SSH@10g-sw1(config)#


Here's some ballpark measurements after 30 minutes idle at each speed 1-3 (updated after repeating testing).

Fan Speed 1: 36C intake, 40C exhaust
Fan Speed 2: 36C intake, 38C exhaust
Fan Speed 3: 33C intake, 36C exhaust

I've got most of the design worked out, and I'm 3D printing the fan assembly in PETG via 3DHubs for about $30 - should be here by the weekend to test. I ended up making the fan shroud reversible (vertically) to allow more flexibility in my rack space.
Some pics of the fan shrouds printed in PETG and used on both my switches. They are attached to the existing tray and completely removable, they can also be flipped. I'm pretty happy with the results.
 

Attachments

Kaytro

Currently breaking things...
Feb 24, 2015
10
9
3
32
Just want to let everyone know that after you flash the brocade os you can use the super cheap Cisco FET-10G transceiver. They can be had for under $3 each.

I use emulex Oce10100,11100,and 14100 cards and they also support the Cisco FET-10G optics.
 

Bowski

New Member
Aug 14, 2018
1
0
1
OK, did a "Clear config", which wiped all settings. Reconfigured the switch.
Still cannot ping servers on same VLAN.
Hi, I see that you turned on tagging on 10-11. Maybe it is silly question but did you enable tagging on servers nics? If so ok, if not turn off tagging on 10-11 to untagged ports. It should work. I have router on stick plug to one port @LB6M and doing packet filtering/routing between vlans. This is the only tagged port (in both vlans). The rest is untagged but ports participate in vlans.
 

epicurean

Active Member
Sep 29, 2014
721
57
28
Hi,
I need to change the IP address of the MGMT 1, but every time I do so it says I can only put in 1 ip address per subnet?
Can I set it such that it pulls and ip address via DHCP?
 

fullstackinfo

New Member
Aug 25, 2018
1
0
1
I just pickup myself up a sister switch to the Quanta. It's a Delta Broadcom Anatel ET-DT7040. Same os and feature set. Came with the x.x.x.18 FW. This one has hot swap fans in the back and is layed out a bit different but is based on the same chipset and software. I have the same modules as most here. Fastpath Routing and Fastpath QOS. Works perfect with the Finisar sfp+ modules and also the 3m cisco DAC cables that came free with my Mellanox Connect 2 cards. $37 for 2 cards and 2 dacs on flea bay. These switches have rear to front airflow so when mounted as TOR switches, the airflow matches the rest of the equipment in the rack. The switch was $300 delivered. UnixPlusCom is the seller if anyone is interested.

I have two LACP groups configured and working well, one is a trunk group to my Ubiquiti Unifi 48 switch the other is to a Poweredge R510 with 2 mellanox cards running Windows server 2016 datacenter. I have plans to setup a few vlans and vlan routing with LACP groups inside the vlans. I ordered 4 of these to quiet the beast down. (1 spare) Top Motor 40x28mm 12V 9000 rpm PWM fan with connector #DF124028BL-PWMG | Coolerguys

Love the forum, lots of good info.

Thanks,
Kirk
Hey Kirk, just wondering, how are those fans working out for you? I have some Noctua's in mine, but they barely move enough air and I'm afraid to leave it on during the day in the corner of my closet lol. Current low-load temp is about 65C. I know the shutdown on the TurboIron has it set at 90C, but that seems really high to me. Thanks in advance!
 

narapon

New Member
Mar 31, 2017
14
1
3
32
Currently looking around for a good heatsink candidate for cooling the Broadcom chip. First gotta figure a way to mount something like an Thermalright AXP-200 on it.. does anyone know what kind of threading are the 4 screw mounts around the chip?
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
42
Brighton, UK
Hey Kirk, just wondering, how are those fans working out for you? I have some Noctua's in mine, but they barely move enough air and I'm afraid to leave it on during the day in the corner of my closet lol. Current low-load temp is about 65C. I know the shutdown on the TurboIron has it set at 90C, but that seems really high to me. Thanks in advance!
FWIW my LB6M (running Brocade firmware) has been running at 55C (night) to 70C (daytime) for the past 6 months, including throughout the whole (British) summer.

I went with a single Noctua 12mm fan option, mounted in the top panel, right above the main CPU. Dead silent at all times. Pictured here a few months ago, when I'd first done the mod:



The switch does feel rather hot to the touch, eg when I pull an SFP+ it's pretty warm. I wouldn't do this in production.. but it's going fine at home :) That said, my load could be called "extra low", as I only have 4 x 10G ports populated + 4 x 1G, and I'm barely pushing anything through it.

Regardless, I took it at its word when it said 80C was warning and 90C was shutdown, and figured that as long as I kept it below 75C it'd be fine. And it has been.
 
Last edited:
  • Like
Reactions: Tha_14

Ulli

New Member
Sep 6, 2018
4
0
1
in the manual of the LB6M i see that it has a web-management Interface.
does anybody know how to enable it?
 

Corsaire

New Member
Feb 3, 2018
21
0
1
in the manual of the LB6M i see that it has a web-management Interface.
does anybody know how to enable it?
This was already answered somewhere in the thread but I'll tell you again.

The Quanta was sold as a generic "brandless" (not entirely true, but whatever) 10G switch to very big and well known companies like Amazon and Google for their datacenters.
They would buy tons of them and asked for modifications in the firmware (or did it themselves) to remove part they didn't intent to use anyway, like the GUI.

So yes, the GUI indeed exist, but as an option. And this option was not very popular. That's why you can see it in the Quanta manuals, but isn't installed in the models we can have for a cheap price on ebay or other places.
 

Ulli

New Member
Sep 6, 2018
4
0
1
Thanks for your reply.
Is there perhaps an application to manage the quanta if i put the brocade-image on it?
 

CKat

New Member
May 20, 2017
19
1
3
42
Hello all,

I have an LB6M thats working great. However I have a new requirement, I need to block traffic within VLAN -
for example if my VLAN 100 has IP range = 10.10.10.0/24, none of the IP within that should be able to communicate with each other (looking at multi-tenant environment).

Is there a way to implement this? I applied the rules on firewall but seems that the traffic never hits the firewall since it could be on the same Virtual environment (different VMs).
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
42
Brighton, UK
Hello all,

I have an LB6M thats working great. However I have a new requirement, I need to block traffic within VLAN -
for example if my VLAN 100 has IP range = 10.10.10.0/24, none of the IP within that should be able to communicate with each other (looking at multi-tenant environment).

Is there a way to implement this? I applied the rules on firewall but seems that the traffic never hits the firewall since it could be on the same Virtual environment (different VMs).
I am by no means experienced in this sort of advanced config, or indeed advanced switch config in general. But your question interested me so I went through the Brocade documentation.

I believe what you want should be achievable using rule-based ACLs. This is definitely supported on a Brocade-flashed LB6M, and all of the following info is taken from Brocade documentation and tested on my own Brocade-flashed LB6M.

I can't say for absolute certain that the same is available on the default FastPath LB6M firmware, however I checked my Quanta LB4M running FastPath routing FW (version 5.13.12.14) and found the same ACL commands were supported there (access-list and ip access-group), albeit with slightly different syntax.

Reading the Brocade documentation file TurboIron24X_08001_ConfigGuide.pdf, starting page 897:
  1. Types of IP ACLs
    You can configure the following types of IP ACLs:
    • Standard - Permits or denies packets based on source IP address. Valid standard ACL IDs are 1 - 99 or a character string.

    • Extended - Permits or denies packets based on source and destination IP address and also based on IP protocol information. Valid extended ACL IDs are a number from 100 - 199 or a character string

The latter, Extended ACLs, supporting denying packets based on both source and destination IP, sounds like what you want? Specifically you'd block packets from 10.10.10.0/24 to 10.10.10.0/24 - though presumably also with a rule specifically allowing access to any router/gateway on that subnet.

The full Brocade docs are provided on the web page that details the Brocade flash, which is documented here in this thread. The best doc to read is the TurboIron ConfigGuide I just mentioned, as the other files are general docs for all FastIron switches, rather than specifically listing everything supported (or not) on the TurboIron - and therefore also supported on a Brocade-flashed LB6M.

I tested this myself on my Brocade-flashed LB6M, and it seems to work:
Code:
vlan 100 name vlan100 by port
 untagged ethe 2 ethe 24
 spanning-tree
!
access-list 100 bridged-routed
access-list 100 permit ip 192.168.200.0 0.0.0.255 host 192.168.200.10
access-list 100 deny ip 192.168.200.0 0.0.0.255 192.168.200.0 0.0.0.255
access-list 100 permit ip any any
!
interface ethernet 24
 port-name 10Gdesktop2
 ip access-group 100 in
This achieved the following result:
  • Port 24, configured with 192.168.200.20, could ping 192.168.200.10 (which is connected to port 2)
  • But it could not access any other IP on 192.168.200.0/24
  • It could access any other subnets it was configured with, eg port 24 configured with 192.168.210.20 could ping 192.168.210.12 on port 2.
The command access-list 100 bridged-routed is required to enable ACL use in a L2 config - you may not need this if you have set the switch up for L3.

I was only able to apply the ip access-group 100 to port 24 once port 24 was in a VLAN. I wasn't previously using VLANs on this switch so I created one to test it out. That might be because I'm missing some other config that would enable per-port access-group without a VLAN. Although in your case you want it on a VLAN anyway. (Note that this did not apply on the LB4M, where ip access-group was available on any port, regardless of it already being in a VLAN.)

One final point: according to the Brocade docs, on our switch ACLs can only be applied "inbound" on a port, not outbound. Inbound means "from the NIC connected to the given port", ie you can ACL restrict traffic that comes from a NIC to a specified port, but not traffic from elsewhere that will exit out of that port (and end up at the NIC.) When I checked my LB4M, I found the same limitation there - it will only apply an access-group for "in".

I don't believe that affects what you want to do, given you want to block all traffic between 10.10.10.0/24, so it doesn't matter whether you block it at source or destination. Just bear in mind you need to write the ACL such that it blocks traffic from each NIC (in to its switch port), not traffic destined for a given NIC (which will exit out from its switch port.)

So from what I've read and can understand, I believe rule-based ACLs can achieve what you want? If you're running Brocade FW - as is highly recommend - then the above should work for you directly. If you're still on FastPath LB6M FW, I expect the same basic commands will exist (access-list and ip access-group) but might use slightly different syntax. FastPath might also have minor differences in implementation compared to Brocade, h0wever it seems my LB4M supports the same basic concepts as the Brocade-flashed LB6M, so I'd be quite surprised if a FastPath LB6M was much different.

That said, if you are on the FastPath FW, maybe this would make a good opportunity to flash to Brocade FW :) It seems to be superior in most every way, and is also far better documented - a really big benefi in my view.
 
Last edited: