Quanta LB6M (10GbE) -- Discussion

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

keoki

New Member
Jun 2, 2016
21
19
3
61
So I hit a big snag... Heat! Everything but a laptop, and the parts needed to get Internet running are off right now, as I let the room cool back down. My office doesn't have enough HVAC for the equipment in here. On first glance, it was easy to point the finger at the two new switches that I was bringing on-line, but once I started doing measurements, I realized the largest issue was the second FreeNAS system, the new switches just pushed me over the tipping point. I can't run this much heat load in this room. I can probably get away with moving the new FreeNAS system into my wife's office, as her office is an open-air room with plenty of open ceiling space to flow the heat out, and there is a much larger cooling register in there... My wife just complained that the house is too warm, and I see the house is 3 degrees warmer that it should be, based on the thermostat setting. So clearly my over-heat problem is more complex than turning on two large switches, but I'm not going to be able to go much father for the moment.

After having the freenas systems off for a few hours, the room is cooled off again...

But with regard to bringing up the network itself, the management port ever came up, it is set to use DHCP by default, and that failed to do anything.

(FASTPATH Routing) #show serviceport

Interface Status............................... Up
IP Address..................................... 0.0.0.0
Subnet Mask.................................... 0.0.0.0
Default Gateway................................ 0.0.0.0
Configured IPv4 Protocol....................... DHCP
Burned In MAC Address.......................... 08:9E:01:17:96:8C

I think I saw some talk about that earlier in the thread...

But the trunking from my netgear switch to the Cisco and to the 6M is all functional, and the "home" vlan IP on the 6m is pingable, so in general I had no problems getting the 6m running. I can telnet into the IP on the vlan port, so clearly I don't /need/ the management port. My telnet is passing through every bit of my network stack... Wireless router, to the firewall switch, to the netgear switch to the Cisco to the 6m...

I have not configured snmp yet, nor ssh, so I can't say the management configuration is done by a long shot, but that is all secondary. I also need to go tweak the idle timeouts on my terminal sessions, it is constantly logging me off... but that is pretty normal.

But the bottom line is I hit no real snags. everything "just worked". When things "just work" I feel like I didn't learn anything. It was a bit more simple than configuring a Cisco, but not by much. The command line is different from the Cisco, but close enough that it all felt pretty easy. The context help was more useful than the manuals. The manuals mostly didn't have the right commands for the things I was looking up, so the context help was the main source of documentation I found to be useful.

I'm not doing any multicast here at home, so the temptation to dig into that isn't very strong. At the office I have multicast configured for IPTV head-end simulation, but I have no nead for streaming multicast at home... All of my home IPTV is from Plex, Directv, or o0ne of the many OTT video sources, and all of that is unicast.
 
  • Like
Reactions: PGlover and Chuntzu

keoki

New Member
Jun 2, 2016
21
19
3
61
actually just found stuff under configure# set igmp
Happy day!
So I can't resist asking... What are you using multicast for? Do you have a uverse connection, or doing security video feeds or something? I'm intrigued by what you might be doing. At the office I have a bank of video streams that feed several TV headend simulations in the lab, but I don't hear a lot of people talking about multicast so I'm curious...
 
  • Like
Reactions: Chuntzu

Chuntzu

Active Member
Jun 30, 2013
383
98
28
So I can't resist asking... What are you using multicast for? Do you have a uverse connection, or doing security video feeds or something? I'm intrigued by what you might be doing. At the office I have a bank of video streams that feed several TV headend simulations in the lab, but I don't hear a lot of people talking about multicast so I'm curious...
I was curious as well.

Sent from my SM-N920T using Tapatalk
 

josh

Active Member
Oct 21, 2013
614
190
43
But the trunking from my netgear switch to the Cisco and to the 6M is all functional, and the "home" vlan IP on the 6m is pingable, so in general I had no problems getting the 6m running. I can telnet into the IP on the vlan port, so clearly I don't /need/ the management port. My telnet is passing through every bit of my network stack... Wireless router, to the firewall switch, to the netgear switch to the Cisco to the 6m...
How did you manage to trunk to the 6m? Can't find the switchport mode trunk option.

Also, if anyone would like to help with some weird routing issues over the 6m it would be great. I'm having problems accessing the internet from machines on the LAN after placing a pfsense box in between the 6m and the edgerouter. I've disabled the firewall on the pfsense just to make sure it wasn't the firewall blocking stuff.

Routes have been set up:
ER:
1. 192.168.5.0/24 -> x.x.x.14 (pfsense WAN ip)
pfsense:
1. 192.168.5.0/24 -> 172.16.1.2 (6m LAN ip)
2. 0.0.0.0 -> x.x.x.1 (gateway of ER, same subnet as pfsense WAN ip)
6m:
1. 0.0.0.0 -> 172.16.1.1 (pfsense LAN ip)
2. 192.168.5.0/24 is a VLAN with routing enabled

Running traceroute to a 192.168.5.x machine from the switch turns up 0.0.0.0 as the first hop. But pinging the same machine from the switch turns up successful.
Pinging from the 192.168.5.x machine is only successful up to 172.16.1.2 (6m LAN ip). Seems like the packet is getting lost between the 6m and the pfsense box.
Traceroute works fine from 6m to 192.168.2.x machine. Which doesn't really make sense as the only difference is 192.168.2.0/24 is the default VLAN. Why is the 6m routing 192.168.5.0/24 through the default gateway when there's a clear route set up as seen in the routing table?

Route table from 6m:
Code:
Route Codes: R - RIP Derived, O - OSPF Derived, C - Connected, S - Static
       B - BGP Derived, IA - OSPF Inter Area
       E1 - OSPF External Type 1, E2 - OSPF External Type 2
       N1 - OSPF NSSA External Type 1, N2 - OSPF NSSA External Type 2

S      0.0.0.0/0 [1/0] via 172.16.1.1,   13h:27m:59s,  0/28
C      172.16.1.0/24 [0/1] directly connected,   0/28
C      192.168.2.0/24 [0/1] directly connected,   2/1
C      192.168.5.0/24 [0/1] directly connected,   2/2
 

keoki

New Member
Jun 2, 2016
21
19
3
61
Basically what I discovered is that all ports are trunk ports. If a port is unconfigured, it appears to be a trunk port still. any thing I did, seemed to leave the port as a trunk port. But I could, in theory, at least control the VLANs by either including, or excluding specific vlans or ranges of vlans... except that doesn't seem to work.

So I have vlan1 by default (one of the reasons most network people never use vlan 1) and I added vlan 100 and 200. I put an IP on VLAN 100.

I started trying different configuration statements, while I was running a ping "from afar", and nothing would stop the pings, including "vlan participation exclude 100" on the port I was using for trunking.

I did notice it adds config lines for you sometimes, for example if I exclude 1,100, it added the line "vlan tagging 200" to the interface, but my pings to vlan 100 continue to run on that port. I deleted all config, causing the port to disappear from the config, and my pings continued...

If you are a network specialist, you realize this is crazy insecure, and on a big network would probably take the network down with broadcast storms. But it will run some fast ethernet to my san and other servers.

I tried a command to set vlan tagging for all ports but one, the 1-4093 range, and it complained once for every valid vlan.

So early on I thought it assumed vlan tagging by simply using "vlan participation exclude/include vlan#" but when I started testing the exclude/include function it fell short of showing that they did anything.

I did not test every possible configuration, but as far as I can tell, it will trunk on every port for every vlan that is defined, unless the port is in the shutdown state. The only real control was setting the PVID, which sets the untagged vlan for the port.

This switch with the software that is on it, it for a trusted network. A trusted network that has some smart devices in places to prevent one big broadcast storm. This is the trunking version of a really dumb switch... at least until we find some proper software to install.

So to sum it up, to trunk a set of vlans, the vlans must be defined in the switch, and the port must not be shut down. If those are true, it is a trunk port.

Now I didn't test ALL possible commands. I tried several, and they appeared to just simply work, but when I tried to un-do the commands, trunking continued to work, no matter what. The only thing I appeared to be able to steer is what vlan it would trip for access mode traffic.

The software is crap, as far as I can tell. Pretty buttons that mostly don't do anything. And I tried rebooting a few times, just to be sure I was really running the new config, including the blank config.

The software I have, anyway, appears to be demo level software that only lets the broadcom chips do default things. As if it is a framework of commandline stubs, with none of them actually implemented. It is like a baby's toy car console, it has a steering wheel, a horn, some buttons that go click, but none of it does much.

I think the command line is mostly stubs, with very little code behind it. Some input validation, and a whole bunch of things on the to-do list for a programmer. I think we know why Amazon is dumping these... It is a 10g busybox (not to be confused with the software package with the same name).

So if you are looking for a software feature, don't get excited just because you found the commands you think you need, there is a good chance the commands don't do anything. Vlan tag stripping for the PVID seems to be the only feature implemented besides shutdown. The chips just do whatever they do by default.

Now I need to find some real software to load on this thing.

FYI I'm running this:
System Description............................. Quanta LB6M, 1.2.0.14, Linux
2.6.21.7


"Look at me mommy!!! I'm configuring a big switch all by myself, and it actually works!!! I'm a big admin now!"
 
Last edited:
  • Like
Reactions: qamaro and Fritz

keoki

New Member
Jun 2, 2016
21
19
3
61
And by the way... The lack of real features has some potential implications on performance under layer 3. It may be best to try to use this in layer 2 only if you have a performance application in mind... I mean you did go out and buy a 10g switch, so you might plan on using it. I'm seeing a lot of slowness in my pings... on an otherwise idle switch. I know pings to a switch or router are rarely answered by the fastest path in the switch with great priority, but I doubt there IS a fast path in this switch... let alone QOS, spanning tree, routing protocols, channelgroups, or much of anything else behind all of that fancy command line UI.
 

keoki

New Member
Jun 2, 2016
21
19
3
61
Time for everyone to scream like godzilla! ;)

And don't buy one of these for use on a production network!

There are some really good deals out there that actually work... The 6509's are selling cheap, and with a 720 gig processor card, and 40gig per slot of bandwidth, the 10g is not exciting, but it is cheap and the commands on the switch are real...
 

keoki

New Member
Jun 2, 2016
21
19
3
61
Sorry for posting 4 posts in a row, I'm ready to stfu about the discovery of how half the config commands don't do anything.

How is it on the firmware front? I'm guessing that every firmware version out there has some missing feature implemented. So later firmware versions could have more features, so I'm willing to investigate that. I work on a team that develops similar hardware, but on completely different chips. I have experience watching a team of people go from reference broadcom hardware/software to a working, tested and secure system. This is a 300,000 man-hour task. Could be less, likely more. And that assumes the hardware is right, that we don't need to tweak the board a dozen times to fix flaws, or add fast path support. But based on where the software I have sits, this is less functional than most broadcom reference systems I have seen. To me that means it is probably on first rev level hardware design, but I can hold out hope that more functional software is out there.

You can't just pick software from another product and install it on the lb6m, the hardware differences for something like that are likely to make that a failure, even if you had the tools to force the firmware to load. If someone out there has inside knowledge on this product could whisper in my ear what was going on, that would be helpful to all of us that have these. Especially those buying lots of them for production use. I see other threads on this board of people buying them for their colo facility, and frankly that is a scary thing... people might be buying hundreds of these thinking they work and hoping to put them in production. Clearly Amazon did that, but with the elastic cloud services they sell, any hacker can come along for almost free, and own a vm in that cloud where you could exploit what you know about these switches, and essentially own whatever datacenter they were installed in. The only way to prevent that sort of abuse is to carefully deploy these for hidden internal use only, like between a database server and a SAN that is several layers deep, and completely isolated from where users' packets live.

So everyone chime in here, what firmware version do you have? Do you think it works?

For the $300 price I see these selling for, it is a functional device good for home use, if it has any layer 2 thruput at all. Somewhere I saw a number on it's internal thruput capability, but at this stage I really question where that number came from, as thruput is not just adding up the bancwidth of all of the ports, or taking a number from a chip specification. Bad software can hurt a good chip, a programmer can fix a bug, by replacing the bug with a work-around that moves packets from a fast path to a glacially slow path. Feature implementations and bug fixes alike have to be tested over and over, as each one could destroy performance. Just developing the test cases for this would take a team of people many months.
 
  • Like
Reactions: TheBloke

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,126
584
113
New York City
www.glaver.org
The software I have, anyway, appears to be demo level software that only lets the broadcom chips do default things. As if it is a framework of commandline stubs, with none of them actually implemented. It is like a baby's toy car console, it has a steering wheel, a horn, some buttons that go click, but none of it does much.

I think the command line is mostly stubs, with very little code behind it. Some input validation, and a whole bunch of things on the to-do list for a programmer. I think we know why Amazon is dumping these... It is a 10g busybox (not to be confused with the software package with the same name).
I have been using this software (on other switches, of course) since it was created by LVL7 Systems (which was eventually purchased by Broadcom). It was pretty complete even back then, and my complaints were with implementation errors (or differences from the way Cisco IOS did it), not with missing features.

Having said that, it seems that these Quanta switches have some features disabled unless a license key is provided, and do not implement some features found in the reference implementation.

Here's some commands for a newer version of the software on a Dell switch:
Code:
show tech
...
System Description............................. Powerconnect 8024, 5.1.10.1, VxWorks 6.6
Machine Description............................ Dell Ethernet Switch
Machine Type................................... Powerconnect 8024
Machine Model.................................. PC8024
Serial Number.................................. xxxxxxxx
FRU Number.....................................
Part Number.................................... BCM56820
Maintenance Level.............................. A
Manufacturer................................... 0xbc00
Burned In MAC Address.......................... D067.E59F.xxxx
Software Version............................... 5.1.10.1
Operating System............................... VxWorks 6.6
Network Processing Device...................... BCM56820_B0
Additional Packages............................ FASTPATH QOS
  FASTPATH Multicast
  FASTPATH Stacking
  FASTPATH Data center
...
Force a port to non-trunk mode instead of auto:
Code:
switchport mode access
switchport access vlan N
Discard untagged frames on a trunk port:
Code:
switchport general acceptable-frame-type tagged-only
The same command set works pretty much across all platforms supported. The Netgear GS110TP even supports the same command set, although it doesn't normally provide access to the command line (the console port hardware is missing and the Telnet server runs on an unusual port number).
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,126
584
113
New York City
www.glaver.org
I'm seeing a lot of slowness in my pings... on an otherwise idle switch. I know pings to a switch or router are rarely answered by the fastest path in the switch with great priority, but I doubt there IS a fast path in this switch... let alone QOS, spanning tree, routing protocols, channelgroups, or much of anything else behind all of that fancy command line UI.
You mention that you work on similar hardware. So you should know that the CPU executing the management functions is almost always not involved in the packet forwarding path, at least for switched (as opposed to routed) packets. And since you mentioned Cisco gear, you probably have experienced this. Take a look at pretty much any fixed-config Cisco IOS switch, and you'll see things like "last output never" in "show interface" for ports that are obviously up and passing packets. That's because the management CPU is not involved in simple packet forwarding. Going back further, the Catalyst 2900XL spent about 60% of its management CPU time in "LED Process" - simply blinking the front panel LEDs.

This isn't just switches, either. Emulex used to make aftermarket controllers for the computer systems DEC used to make. The SC31 was a SMD controller with a bit-slice CPU taking up a good portion of the real estate on the board. All operations passed through the CPU. The Emulex UD33 talked to the same SMD drives, but used an Intel 8039 microprocessor. To give you an idea of the speed of that processor, an enhanced model was used as the keyboard controller in an IBM PC. o_O But all it did on the UD33 was download code into some FPGAs and do some high-level monitoring.
 

Toby

New Member
May 6, 2016
17
4
3
49
Keoki, I think you are doing something wrong.

I'm pinging from my desktop on VLAN 255, with an IP of 10.255.200.6/255.255.0.0
My default gateway is 10.255.255.254 which is the IP of the VLAN 255 interface on my LB6M.

Meanwhile, I have a VMware host with multiple VLAN's on a trunk port. One of those VLAN's is VLAN 150.
On that VLAN I have a machine with an IP address of 192.168.150.10/255.255.255.0. The default gateway for that machine is 192.168.150.1 which is the IP of the VLAN 150 interface on my LB6M.

Here is the configuration of the port my machine is on:
Code:
interface 0/24
vlan pvid 255
vlan participation include 255
exit

Here is the configuration of the port my ESXi host is on:
Code:
interface 0/23
vlan pvid 255
vlan participation include 50,150,198-200,255,400,500,600
vlan tagging 50,150,198-200,400,500,600
exit
When I take away the VLAN participation of 150 on my trunk to my ESXi host, I lost my pings. When I include it again, my pings work.

Something tells me you aren't setting up your VLAN's correctly or something.

Please post your entire configuration.

VLAN's do work fine from everything I've setup and tested.

See attached image also. It shows a constant ping, shows me doing an 'exclude', pings stop, then an 'include' and my pings come back.

You can also 'take away' vlan participation by doing "vlan tagging auto ###". This is for GVRP.
 

Attachments

dwright1542

Active Member
Dec 26, 2015
372
73
28
50
And by the way... The lack of real features has some potential implications on performance under layer 3. It may be best to try to use this in layer 2 only if you have a performance application in mind... I mean you did go out and buy a 10g switch, so you might plan on using it. I'm seeing a lot of slowness in my pings... on an otherwise idle switch. I know pings to a switch or router are rarely answered by the fastest path in the switch with great priority, but I doubt there IS a fast path in this switch... let alone QOS, spanning tree, routing protocols, channelgroups, or much of anything else behind all of that fancy command line UI.
I'm not using L3 much, but as backend SAN switches, these things rock. I'm consistently maxing out port bandwidth on iSCSI.
 

asiaserverhost

New Member
Jun 17, 2016
1
0
1
45
hi friends,

I am looking to buy few of this LB6M.. may I ask some quick questions..

1) these SFP ports can also support 1G sfp fiber/copper modules?

2) can I use different types of sfp at different slots at same time?

3) can do trunk with cisco switch port ?

4) 802.1Q tunneling (aka Q-in-Q) is possible ?

thanks so much in advance for information.. ^_^
 

LeeSter

New Member
Jun 25, 2016
6
0
1
53
I'm new to fiber and would like to understand what transceiver type and cables I will need. I don't want get burned shopping on EBAY. Any assistance, even a link to reference would be appreciated.
 

LeeSter

New Member
Jun 25, 2016
6
0
1
53
Hope this info helps you out my friend, and if you have any other questions, feel free to ask anytime!
Thanks so much for the assistance Sleyk. I had no idea there were copper interconnects. I'm going to use those since I have mostly short runs.
Thanks again!
 

segfault

New Member
Apr 5, 2016
29
13
3
42
hasno.info
I picked one of these up a short while ago and have been slowly migrating over to it. I have vlan routing, vrrp, sflow, lag, and ospf running on it. So far so good. Performance-wise it's quite fast and with cut-through enabled the latency is minuscule compared to the 3750x I'd been using as a core.

From a ping between servers using mellanox connectx-2's
rtt min/avg/max/mdev = 0.052/0.066/0.077/0.011 ms

Right now my only gripe is that the sflow config disappears. I'm guessing that's likely the behavior when it times out.
The information in this forum has been invaluable in setting all of this up. Thanks everyone for documenting all of your experiences and experiments.
 
Last edited:

acmcool

Banned
Jun 23, 2015
610
76
28
39
Woodbury,MN
How do we configure trunk port on this switch? I don't want layer 3 routing on switch..
I configured a port as vlan tunnel with participating vlan's..But that does not seem to work.
Anyone willing to share working config that I can look at?
 

segfault

New Member
Apr 5, 2016
29
13
3
42
hasno.info
How do we configure trunk port on this switch? I don't want layer 3 routing on switch..
I configured a port as vlan tunnel with participating vlan's..But that does not seem to work.
Anyone willing to share working config that I can look at?
Here's some relevant snippets from mine:

vlan database
vlan 10,20,30,40,50

interface 0/25
addport 1/1
port lacptimeout actor short
port lacptimeout partner short​

interface 1/1
description 'a lag interface w/ 802.1q vlan trunking'
no port-channel static
port-channel load-balance 6
ip dhcp snooping trust
ip arp inspection trust
mtu 9000
vlan participation include 10,20,30,40,50
exit​
 
Last edited:

acmcool

Banned
Jun 23, 2015
610
76
28
39
Woodbury,MN
Here's some relevant snippets from mine:

vlan database
vlan 10,20,30,40,50

interface 0/25
addport 1/1
port lacptimeout actor short
port lacptimeout partner short​

interface 1/1
description 'a lag interface w/ 802.1q vlan trunking'
no port-channel static
port-channel load-balance 6
ip dhcp snooping trust
ip arp inspection trust
mtu 9000
vlan participation include 10,20,30,40,50
exit​
Thanks...Looks I was missing the add port...Thnks