And by the way... The lack of real features has some potential implications on performance under layer 3. It may be best to try to use this in layer 2 only if you have a performance application in mind... I mean you did go out and buy a 10g switch, so you might plan on using it. I'm seeing a lot of slowness in my pings... on an otherwise idle switch. I know pings to a switch or router are rarely answered by the fastest path in the switch with great priority, but I doubt there IS a fast path in this switch... let alone QOS, spanning tree, routing protocols, channelgroups, or much of anything else behind all of that fancy command line UI.
Quanta LB6M (10GbE) -- Discussion
- Thread starter T_Minus
- Start date
Time for everyone to scream like godzilla! 
And don't buy one of these for use on a production network!
There are some really good deals out there that actually work... The 6509's are selling cheap, and with a 720 gig processor card, and 40gig per slot of bandwidth, the 10g is not exciting, but it is cheap and the commands on the switch are real...
And don't buy one of these for use on a production network!
There are some really good deals out there that actually work... The 6509's are selling cheap, and with a 720 gig processor card, and 40gig per slot of bandwidth, the 10g is not exciting, but it is cheap and the commands on the switch are real...
Sorry for posting 4 posts in a row, I'm ready to stfu about the discovery of how half the config commands don't do anything.
How is it on the firmware front? I'm guessing that every firmware version out there has some missing feature implemented. So later firmware versions could have more features, so I'm willing to investigate that. I work on a team that develops similar hardware, but on completely different chips. I have experience watching a team of people go from reference broadcom hardware/software to a working, tested and secure system. This is a 300,000 man-hour task. Could be less, likely more. And that assumes the hardware is right, that we don't need to tweak the board a dozen times to fix flaws, or add fast path support. But based on where the software I have sits, this is less functional than most broadcom reference systems I have seen. To me that means it is probably on first rev level hardware design, but I can hold out hope that more functional software is out there.
You can't just pick software from another product and install it on the lb6m, the hardware differences for something like that are likely to make that a failure, even if you had the tools to force the firmware to load. If someone out there has inside knowledge on this product could whisper in my ear what was going on, that would be helpful to all of us that have these. Especially those buying lots of them for production use. I see other threads on this board of people buying them for their colo facility, and frankly that is a scary thing... people might be buying hundreds of these thinking they work and hoping to put them in production. Clearly Amazon did that, but with the elastic cloud services they sell, any hacker can come along for almost free, and own a vm in that cloud where you could exploit what you know about these switches, and essentially own whatever datacenter they were installed in. The only way to prevent that sort of abuse is to carefully deploy these for hidden internal use only, like between a database server and a SAN that is several layers deep, and completely isolated from where users' packets live.
So everyone chime in here, what firmware version do you have? Do you think it works?
For the $300 price I see these selling for, it is a functional device good for home use, if it has any layer 2 thruput at all. Somewhere I saw a number on it's internal thruput capability, but at this stage I really question where that number came from, as thruput is not just adding up the bancwidth of all of the ports, or taking a number from a chip specification. Bad software can hurt a good chip, a programmer can fix a bug, by replacing the bug with a work-around that moves packets from a fast path to a glacially slow path. Feature implementations and bug fixes alike have to be tested over and over, as each one could destroy performance. Just developing the test cases for this would take a team of people many months.
How is it on the firmware front? I'm guessing that every firmware version out there has some missing feature implemented. So later firmware versions could have more features, so I'm willing to investigate that. I work on a team that develops similar hardware, but on completely different chips. I have experience watching a team of people go from reference broadcom hardware/software to a working, tested and secure system. This is a 300,000 man-hour task. Could be less, likely more. And that assumes the hardware is right, that we don't need to tweak the board a dozen times to fix flaws, or add fast path support. But based on where the software I have sits, this is less functional than most broadcom reference systems I have seen. To me that means it is probably on first rev level hardware design, but I can hold out hope that more functional software is out there.
You can't just pick software from another product and install it on the lb6m, the hardware differences for something like that are likely to make that a failure, even if you had the tools to force the firmware to load. If someone out there has inside knowledge on this product could whisper in my ear what was going on, that would be helpful to all of us that have these. Especially those buying lots of them for production use. I see other threads on this board of people buying them for their colo facility, and frankly that is a scary thing... people might be buying hundreds of these thinking they work and hoping to put them in production. Clearly Amazon did that, but with the elastic cloud services they sell, any hacker can come along for almost free, and own a vm in that cloud where you could exploit what you know about these switches, and essentially own whatever datacenter they were installed in. The only way to prevent that sort of abuse is to carefully deploy these for hidden internal use only, like between a database server and a SAN that is several layers deep, and completely isolated from where users' packets live.
So everyone chime in here, what firmware version do you have? Do you think it works?
For the $300 price I see these selling for, it is a functional device good for home use, if it has any layer 2 thruput at all. Somewhere I saw a number on it's internal thruput capability, but at this stage I really question where that number came from, as thruput is not just adding up the bancwidth of all of the ports, or taking a number from a chip specification. Bad software can hurt a good chip, a programmer can fix a bug, by replacing the bug with a work-around that moves packets from a fast path to a glacially slow path. Feature implementations and bug fixes alike have to be tested over and over, as each one could destroy performance. Just developing the test cases for this would take a team of people many months.
I have been using this software (on other switches, of course) since it was created by LVL7 Systems (which was eventually purchased by Broadcom). It was pretty complete even back then, and my complaints were with implementation errors (or differences from the way Cisco IOS did it), not with missing features.
Having said that, it seems that these Quanta switches have some features disabled unless a license key is provided, and do not implement some features found in the reference implementation.
Here's some commands for a newer version of the software on a Dell switch:
Code:
show tech
...
System Description............................. Powerconnect 8024, 5.1.10.1, VxWorks 6.6
Machine Description............................ Dell Ethernet Switch
Machine Type................................... Powerconnect 8024
Machine Model.................................. PC8024
Serial Number.................................. xxxxxxxx
FRU Number.....................................
Part Number.................................... BCM56820
Maintenance Level.............................. A
Manufacturer................................... 0xbc00
Burned In MAC Address.......................... D067.E59F.xxxx
Software Version............................... 5.1.10.1
Operating System............................... VxWorks 6.6
Network Processing Device...................... BCM56820_B0
Additional Packages............................ FASTPATH QOS
FASTPATH Multicast
FASTPATH Stacking
FASTPATH Data center
...
Code:
switchport mode access
switchport access vlan N
Code:
switchport general acceptable-frame-type tagged-onlyYou mention that you work on similar hardware. So you should know that the CPU executing the management functions is almost always not involved in the packet forwarding path, at least for switched (as opposed to routed) packets. And since you mentioned Cisco gear, you probably have experienced this. Take a look at pretty much any fixed-config Cisco IOS switch, and you'll see things like "last output never" in "show interface" for ports that are obviously up and passing packets. That's because the management CPU is not involved in simple packet forwarding. Going back further, the Catalyst 2900XL spent about 60% of its management CPU time in "LED Process" - simply blinking the front panel LEDs.
This isn't just switches, either. Emulex used to make aftermarket controllers for the computer systems DEC used to make. The SC31 was a SMD controller with a bit-slice CPU taking up a good portion of the real estate on the board. All operations passed through the CPU. The Emulex UD33 talked to the same SMD drives, but used an Intel 8039 microprocessor. To give you an idea of the speed of that processor, an enhanced model was used as the keyboard controller in an IBM PC.
But all it did on the UD33 was download code into some FPGAs and do some high-level monitoring.Keoki, I think you are doing something wrong.
I'm pinging from my desktop on VLAN 255, with an IP of 10.255.200.6/255.255.0.0
My default gateway is 10.255.255.254 which is the IP of the VLAN 255 interface on my LB6M.
Meanwhile, I have a VMware host with multiple VLAN's on a trunk port. One of those VLAN's is VLAN 150.
On that VLAN I have a machine with an IP address of 192.168.150.10/255.255.255.0. The default gateway for that machine is 192.168.150.1 which is the IP of the VLAN 150 interface on my LB6M.
Here is the configuration of the port my machine is on:
Here is the configuration of the port my ESXi host is on:
When I take away the VLAN participation of 150 on my trunk to my ESXi host, I lost my pings. When I include it again, my pings work.
Something tells me you aren't setting up your VLAN's correctly or something.
Please post your entire configuration.
VLAN's do work fine from everything I've setup and tested.
See attached image also. It shows a constant ping, shows me doing an 'exclude', pings stop, then an 'include' and my pings come back.
You can also 'take away' vlan participation by doing "vlan tagging auto ###". This is for GVRP.
I'm pinging from my desktop on VLAN 255, with an IP of 10.255.200.6/255.255.0.0
My default gateway is 10.255.255.254 which is the IP of the VLAN 255 interface on my LB6M.
Meanwhile, I have a VMware host with multiple VLAN's on a trunk port. One of those VLAN's is VLAN 150.
On that VLAN I have a machine with an IP address of 192.168.150.10/255.255.255.0. The default gateway for that machine is 192.168.150.1 which is the IP of the VLAN 150 interface on my LB6M.
Here is the configuration of the port my machine is on:
Code:
interface 0/24
vlan pvid 255
vlan participation include 255
exitHere is the configuration of the port my ESXi host is on:
Code:
interface 0/23
vlan pvid 255
vlan participation include 50,150,198-200,255,400,500,600
vlan tagging 50,150,198-200,400,500,600
exitSomething tells me you aren't setting up your VLAN's correctly or something.
Please post your entire configuration.
VLAN's do work fine from everything I've setup and tested.
See attached image also. It shows a constant ping, shows me doing an 'exclude', pings stop, then an 'include' and my pings come back.
You can also 'take away' vlan participation by doing "vlan tagging auto ###". This is for GVRP.
Attachments
I'm not using L3 much, but as backend SAN switches, these things rock. I'm consistently maxing out port bandwidth on iSCSI.
hi friends,
I am looking to buy few of this LB6M.. may I ask some quick questions..
1) these SFP ports can also support 1G sfp fiber/copper modules?
2) can I use different types of sfp at different slots at same time?
3) can do trunk with cisco switch port ?
4) 802.1Q tunneling (aka Q-in-Q) is possible ?
thanks so much in advance for information.. ^_^
I am looking to buy few of this LB6M.. may I ask some quick questions..
1) these SFP ports can also support 1G sfp fiber/copper modules?
2) can I use different types of sfp at different slots at same time?
3) can do trunk with cisco switch port ?
4) 802.1Q tunneling (aka Q-in-Q) is possible ?
thanks so much in advance for information.. ^_^
Hello new friend! Welcome to STH!
So, to answer your questions:
1) So far, we have been discussing trying to get 1GB sfp+ transceivers to work. Someone has the magic module, but, the latest is that we gotta wait for a bit to get the model number. I think they are waiting for downtime on their servers. They said they would post it as soon as they get a chance to see the modules. As of right now, no one else has reported any other success or joy. So it is still abit up in the air.
2) Yup, you can use 10GB Avago, Cisco, heck even cheap Chinese modules work quite well in this bad boy. The switch recognizes most of 'em.
3) Yessir, If I am not mistaken, I believe you can. I don't have a Cisco switchport, but I think it is very doable.
4) Yes my good sir, I think Q tunneling/Vlan translation can be done from what I read, but if asking me how to do it? That, my friend might be difficult!
Sorry I couldn't help further! I am still reading up on Vlans and other more complicated network characteristics and protocols. Please see posts #220 and #295 for easy startup info on the switch if you do decide to get it! It's a compilation of all the info we gathered from the manual and people's knowledge of the switch so far.
Ask us any questions you have! Alot of the guys here are very knowledgeable and will definitely be able to help.
So, to answer your questions:
1) So far, we have been discussing trying to get 1GB sfp+ transceivers to work. Someone has the magic module, but, the latest is that we gotta wait for a bit to get the model number. I think they are waiting for downtime on their servers. They said they would post it as soon as they get a chance to see the modules. As of right now, no one else has reported any other success or joy. So it is still abit up in the air.
2) Yup, you can use 10GB Avago, Cisco, heck even cheap Chinese modules work quite well in this bad boy. The switch recognizes most of 'em.
3) Yessir, If I am not mistaken, I believe you can. I don't have a Cisco switchport, but I think it is very doable.
4) Yes my good sir, I think Q tunneling/Vlan translation can be done from what I read, but if asking me how to do it? That, my friend might be difficult!
Sorry I couldn't help further! I am still reading up on Vlans and other more complicated network characteristics and protocols. Please see posts #220 and #295 for easy startup info on the switch if you do decide to get it! It's a compilation of all the info we gathered from the manual and people's knowledge of the switch so far.
Ask us any questions you have! Alot of the guys here are very knowledgeable and will definitely be able to help.
Last edited:
Hey LeeSter! Welcome to STH! We are glad to have you! So you want to get into Fiber? Great!
So first things first. Are you looking at the LB6M? Depending on the switch you get, you will have to get cables and transceivers for it. If you are looking to purchase the LB6M, there is good news in that you can actually start your 10Gb network without having to buy fiber cables and separate transceivers. There has been a great listing on eBay that has a single port Mellanox 10Gb card bundled with a 3 meter (10ft) DAC (direct attached cable). See this link/thread my friend:
Mellanox Single Port with 3m DAC
The cable itself is made of copper, but works the same as Fiber for 10Gb. The ends of the cable have a transceiver built onto the cable, so no need to get separate transceivers, which is good. If you need cables lengths longer than around 20ft, then you will probably have to get a fiber optic cable and buy separate transceivers and "make" your cable.
For fiber cables, I believe the type you are looking for is LC-LC multimode 125/50 fiber optic cables. This type is the best type of cable in use now for fiber and 10Gb, although they do have other types that are still in use.
As for transceivers, most transceivers work well for the LB6M 10Gb switch. Avago, Cisco, Juniper, Solarflare and even cheapo generic ones from China. If you are gonna grab the LB6M, but you have need for long cable lengths, and you wanna make your own cables, then you can pretty much search for the cheapest branded transceivers you can find on eBay. If you can run your network with 10ft cables or less, then its probably better to grab a DAC cable instead. It's less of a headache.
Also, as a quick note my friend, I believe DAC's have lower latency than optical fiber cables. So you can't go wrong with DAC's for cables if you need less than 10ft or so. For Direct attached Cables, I believe the limit for length of the cable runs to about 7m or 23ft. After that, latency and other naughty things happen to the signal with a DAC cable. So if you will need long runs greater than 20ft or so, then you must make a fiber cable. If short cables can be used, then a DAC is excellent.
Here is a link to a listing with LC-LC multimode fiber cables: (if you search around, you can definitely snag them cheaper)
10m LC-LC 10Gb 50/125 OM3 Duplex Multimode PVC Fiber Optic Cable - Aqua NEW
Here is a link to the cheapest 10Gb transceivers I can currently find:
Juniper 10GB SFP+ SR Transceiver Gbic 740-021308
Here is another direct link to the listing on eBay for the 10Gb card with the 3m (10ft) DAC bundled with it:
671798-001 666172-001 MNPA19-XTR HP 10GB ETHERNET NETWORK INTERFACE CARD W/CABLE
Remember, seller will take $18 bucks best offer for the listing, so don't give 'em $20. Save the $2 bucks! Heheh!
Hope this info helps you out my friend, and if you have any other questions, feel free to ask anytime!
So first things first. Are you looking at the LB6M? Depending on the switch you get, you will have to get cables and transceivers for it. If you are looking to purchase the LB6M, there is good news in that you can actually start your 10Gb network without having to buy fiber cables and separate transceivers. There has been a great listing on eBay that has a single port Mellanox 10Gb card bundled with a 3 meter (10ft) DAC (direct attached cable). See this link/thread my friend:
Mellanox Single Port with 3m DAC
The cable itself is made of copper, but works the same as Fiber for 10Gb. The ends of the cable have a transceiver built onto the cable, so no need to get separate transceivers, which is good. If you need cables lengths longer than around 20ft, then you will probably have to get a fiber optic cable and buy separate transceivers and "make" your cable.
For fiber cables, I believe the type you are looking for is LC-LC multimode 125/50 fiber optic cables. This type is the best type of cable in use now for fiber and 10Gb, although they do have other types that are still in use.
As for transceivers, most transceivers work well for the LB6M 10Gb switch. Avago, Cisco, Juniper, Solarflare and even cheapo generic ones from China. If you are gonna grab the LB6M, but you have need for long cable lengths, and you wanna make your own cables, then you can pretty much search for the cheapest branded transceivers you can find on eBay. If you can run your network with 10ft cables or less, then its probably better to grab a DAC cable instead. It's less of a headache.
Also, as a quick note my friend, I believe DAC's have lower latency than optical fiber cables. So you can't go wrong with DAC's for cables if you need less than 10ft or so. For Direct attached Cables, I believe the limit for length of the cable runs to about 7m or 23ft. After that, latency and other naughty things happen to the signal with a DAC cable. So if you will need long runs greater than 20ft or so, then you must make a fiber cable. If short cables can be used, then a DAC is excellent.
Here is a link to a listing with LC-LC multimode fiber cables: (if you search around, you can definitely snag them cheaper)
10m LC-LC 10Gb 50/125 OM3 Duplex Multimode PVC Fiber Optic Cable - Aqua NEW
Here is a link to the cheapest 10Gb transceivers I can currently find:
Juniper 10GB SFP+ SR Transceiver Gbic 740-021308
Here is another direct link to the listing on eBay for the 10Gb card with the 3m (10ft) DAC bundled with it:
671798-001 666172-001 MNPA19-XTR HP 10GB ETHERNET NETWORK INTERFACE CARD W/CABLE
Remember, seller will take $18 bucks best offer for the listing, so don't give 'em $20. Save the $2 bucks! Heheh!
Hope this info helps you out my friend, and if you have any other questions, feel free to ask anytime!
Last edited:
Thanks so much for the assistance Sleyk. I had no idea there were copper interconnects. I'm going to use those since I have mostly short runs.
Thanks again!
I picked one of these up a short while ago and have been slowly migrating over to it. I have vlan routing, vrrp, sflow, lag, and ospf running on it. So far so good. Performance-wise it's quite fast and with cut-through enabled the latency is minuscule compared to the 3750x I'd been using as a core.
From a ping between servers using mellanox connectx-2's
rtt min/avg/max/mdev = 0.052/0.066/0.077/0.011 ms
Right now my only gripe is that the sflow config disappears. I'm guessing that's likely the behavior when it times out.
The information in this forum has been invaluable in setting all of this up. Thanks everyone for documenting all of your experiences and experiments.
From a ping between servers using mellanox connectx-2's
rtt min/avg/max/mdev = 0.052/0.066/0.077/0.011 ms
Right now my only gripe is that the sflow config disappears. I'm guessing that's likely the behavior when it times out.
The information in this forum has been invaluable in setting all of this up. Thanks everyone for documenting all of your experiences and experiments.
Last edited:
How do we configure trunk port on this switch? I don't want layer 3 routing on switch..
I configured a port as vlan tunnel with participating vlan's..But that does not seem to work.
Anyone willing to share working config that I can look at?
I configured a port as vlan tunnel with participating vlan's..But that does not seem to work.
Anyone willing to share working config that I can look at?
Here's some relevant snippets from mine:
vlan database
vlan 10,20,30,40,50
interface 0/25
addport 1/1
port lacptimeout actor short
port lacptimeout partner short
vlan 10,20,30,40,50
interface 0/25
addport 1/1
port lacptimeout actor short
port lacptimeout partner short
interface 1/1
description 'a lag interface w/ 802.1q vlan trunking'
no port-channel static
port-channel load-balance 6
ip dhcp snooping trust
ip arp inspection trust
mtu 9000
vlan participation include 10,20,30,40,50
exit
description 'a lag interface w/ 802.1q vlan trunking'
no port-channel static
port-channel load-balance 6
ip dhcp snooping trust
ip arp inspection trust
mtu 9000
vlan participation include 10,20,30,40,50
exit
Last edited:
Thanks for posting that bit of info to help acmcool! You guys are awesome, and thanks for contributing to the thread!
Thanks much for the post Sleyk. I followed your recommendations and did both the 3x case fan swap, as well as the 2x powersupply fan swap. I ended up using the higher RPM fans for the powersupply to get a bit more air flow (they're still almost completely silent).
The powersupply fans did indeed have the pins completely jumbled up from standard 3-pin fan headers. My fix to this was to take a very small precision flathead screwdriver and pop the pins out from the header and order them correctly for the Quanta LB6M.
Finally both my 24-bay 4U server as well as the Quanta LB6M switch are quiet!
Thanks again!
--LS
The powersupply fans did indeed have the pins completely jumbled up from standard 3-pin fan headers. My fix to this was to take a very small precision flathead screwdriver and pop the pins out from the header and order them correctly for the Quanta LB6M.
Finally both my 24-bay 4U server as well as the Quanta LB6M switch are quiet!
Thanks again!
--LS