Quanta LB6M (10GbE) -- Discussion

[charlie]

Hi,

How can i setup trunk port on one port? In the documentation i see some VTP, but these options are missod from my switches.

The goal is to connect to LB6M, if all vlan created on both switch, these vlans connectiong via the trunk port.

 

[c6100]

Just a little more detail on the VSAN performance I'm seeing, I just modified my storage policy and am now resyncing ~.5TB. on each node, i'm getting between 300MB/s and 550MB/s through the VSAN VMK on each node(this is pretty much the limits of the disks i'm using, 1 cache drive and 1 capacity SSD per node), and between 15,000 and 30,000 IOPS on each node (4 nodes). So this switch, I can safely say is NOT a bottleneck for a VSAN setup.

Thanks for sharing! Are you using just the vCenter stats to determine your IOPs and network usage for vSAN? My IO Controller is on the HCL and the stats in vCenter are not anywhere near that in a 1 cache and 1 capacity drive AF setup. This is why I wanted to move to this switch.

Are you using 3 GBps or 6 GBps SATA? Which IO controller are you currently on?

Sent from my Nexus 6P using Tapatalk

 

[Evan Battle]

Thanks for sharing! Are you using just the vCenter stats to determine your IOPs and network usage for vSAN? My IO Controller is on the HCL and the stats in vCenter are not anywhere near that in a 1 cache and 1 capacity drive AF setup. This is why I wanted to move to this switch.

Are you using 3 GBps or 6 GBps SATA? Which IO controller are you currently on?

Sent from my Nexus 6P using Tapatalk

I am using esxtop on each node to see the per-node stats in real time. The stats in vCenter or vROps show averages, so you'll always miss the peaks.

I am using a 631xESB SAS/SATA controller which is 3Gb, so the 500MB/s numbers are bursts from the NIC, but I do see sustained BW of 300MB/s. These numbers were internal VSAN resync traffic. I've only seen ~10,000 IOPS on any single VM, although I havent tried running anything like IOmeter against it.

Before moving to this switch I was using a quad port 1Gb adapter LAG'd via LACP to my vDS, performance was passable, but slow enough to piss me off. I am so far very happy with this switch, next weekend I plan to swap out the single port NICs for dual port and create LACP LAG's to see if I can reduce the latency I occasionally see on VSAN. DAVG/sec in esxtop is generally below 1ms (usually around 350usec) on each host, but I do see much higher latency from VM's, and in the VSAN stats, occasionally.

 

[c6100]

I have VSAN up and running with this switch, both VSAN 6.2 and VSAN 6.5. My setup appears to have the same feature set yours does:

Additional Packages............................ FASTPATH QOS
FASTPATH Routing

I enabled IGMP querier on my VSAN VLAN, and confirmed multicast functioned as expected using tcpdump-uw (vSphere 6.5 has a nifty VSAN tester in the GUI that will test multicast too)

I was looking through the command manual, there were a lot of IGMP options. Which did you end up using for this to finally work?

e.g.: ip igmp snooping mrouter <vlanid>

 

[Evan Battle]

Hey, so has anyone had any issues with setting the serviceport IP? From what I can read, this is the MGMT ports on the switch. I keep getting "IP Address/Netmask entered conflicts with the configured IP Address/Netmask of the switch". Network parms are working for right now, but I fear that is not using the management ports. Or am I misunderstanding? Thanks!

Are you trying to set the Network parms or the serviceport? The IP address for the management ports are set using the serviceport command from the enable prompt:

(switchname) # serviceport protocol none
(switchname) # serviceport ip {IPAddress} {Netmask} {Gateway}

 

[Evan Battle]

I was looking through the command manual, there were a lot of IGMP options. Which did you end up using for this to finally work?

e.g.: ip igmp snooping mrouter <vlanid>

So i have the following:

(switchname) (Config)# set igmp 10
(switchname) (Config)# set igmp querier 10
(switchname) (Config)# set igmp querier 10 address 10.1.0.100 <<-this is an address on my VSAN VLAN

Then "set igmp" on each interface

(switchname) #show igmpsnooping

Admin Mode..................................... Enable
Multicast Control Frame Count.................. 296952
IGMP Router-Alert check........................ Disabled
Interfaces Enabled for IGMP Snooping........... 0/1
0/2
0/3
0/4
0/5

VLANs enabled for IGMP snooping................ 10

 

[c6100]

So i have the following:

(switchname) (Config)# set igmp 10
(switchname) (Config)# set igmp querier 10
(switchname) (Config)# set igmp querier 10 address 10.1.0.100 <<-this is an address on my VSAN VLAN

Then "set igmp" on each interface

(switchname) #show igmpsnooping

Admin Mode..................................... Enable
Multicast Control Frame Count.................. 296952
IGMP Router-Alert check........................ Disabled
Interfaces Enabled for IGMP Snooping........... 0/1
0/2
0/3
0/4
0/5

VLANs enabled for IGMP snooping................ 10

Did you use an IP address from when you setup the vlan 10 interface? Or did you just pick any IP from that subnet?

 

[Evan Battle]

Did you use an IP address from when you setup the vlan 10 interface? Or did you just pick any IP from that subnet?

I just picked an IP from that Subnet that could communicate with my VSAN vmkernel ports.

 

[c6100]

I just picked an IP from that Subnet that could communicate with my VSAN vmkernel ports.

Thank you! Btw, do you know if mgmt 1 and mgmt 2 ports can be used just like ethernet ports (like 25-28)?

Sent from my Nexus 6P using Tapatalk

 

[Evan Battle]

Thank you! Btw, do you know if mgmt 1 and mgmt 2 ports can be used just like ethernet ports (like 25-28)?

Sent from my Nexus 6P using Tapatalk

I don't believe they can be. They don't show up in the list of available ports.

 

[PigLover]

Thank you! Btw, do you know if mgmt 1 and mgmt 2 ports can be used just like ethernet ports (like 25-28)?

Sent from my Nexus 6P using Tapatalk

No. They cannot.

 

[gr8ape]

Are you trying to set the Network parms or the serviceport? The IP address for the management ports are set using the serviceport command from the enable prompt:

(switchname) # serviceport protocol none
(switchname) # serviceport ip {IPAddress} {Netmask} {Gateway}

Actually the issue was I was tyring to use an IP address that was already on a routable connection. Apparently the switch will not let you set it up once you have it as an ip network. I had to keep it disconnected and set the serviceport through the console, then it worked. Thanks for the info.

 

[Martin Kiefer]

Hi.

I just received my LB6M switch and what a cheap way to get 10G here at my home. Got it off Ebay for just around 250 USD

Now starting to configure the switch a few questions are coming up:
Is there a CLI command to verify the LACP channel is up and running?

I come from a Cisco background where I would use "show etherchannel" to verify the operating status of the channel.

Another funny thing. I tried to use a GLC-T RJ45 in the LB6M to connect a Cisco 3560E and the LB6M. Both ends come up with a link, and the Cisco switch will send out packets. But the LB6M does not receive any nor does it send out any packets.

I am using the Cisco TWINAX cables and SFP-10G-SR SFP+ modules to connect a couple of Cisco UCS C220 M3 servers with Fujitsu based Intel 520 cards inside the servers. That seem to work just fine.

and I have VMware NSX running between them in Hybrid mode for the VxLAN traffic

/Martin

 

[Drewy]

Show port-channel all

 

[c6100]

Hi.

I just received my LB6M switch and what a cheap way to get 10G here at my home. Got it off Ebay for just around 250 USD

Now starting to configure the switch a few questions are coming up:
Is there a CLI command to verify the LACP channel is up and running?

I come from a Cisco background where I would use "show etherchannel" to verify the operating status of the channel.

Another funny thing. I tried to use a GLC-T RJ45 in the LB6M to connect a Cisco 3560E and the LB6M. Both ends come up with a link, and the Cisco switch will send out packets. But the LB6M does not receive any nor does it send out any packets.

I am using the Cisco TWINAX cables and SFP-10G-SR SFP+ modules to connect a couple of Cisco UCS C220 M3 servers with Fujitsu based Intel 520 cards inside the servers. That seem to work just fine.

and I have VMware NSX running between them in Hybrid mode for the VxLAN traffic

/Martin

That is a bummer to hear about the transceiver. My switch arrives tomorrow and I was really hoping this would work, I need the Ethernet ports.

Keep me posted if you figure out how to get it to work!

Sent from my Nexus 6P using Tapatalk

 

[Terry Kennedy]

Another funny thing. I tried to use a GLC-T RJ45 in the LB6M to connect a Cisco 3560E and the LB6M. Both ends come up with a link, and the Cisco switch will send out packets. But the LB6M does not receive any nor does it send out any packets.

I've gone into this in detail a few times previously, even in this thread. To recap - many switches with SFP+ ports (particularly ports that aren't dual SFP+/RJ45 selectable) only have hardware for port operation at ~10Gbit/sec. That hardware doesn't know what to do with data that arrives at other clock rates. Communication with the SFP/SFP+ for things like link status, etc. happens in the I²C channel, which is separate from the path used for user data, so a switch might report a link as "up" even though it won't pass data.

There are 2 flavors of Cisco(ish) SFP/RJ45 adapters - ones that do speed conversion internally (the SFP side always sends to the switch at GigE speeds) and ones that don't (the SFP side operates at 10/100/1000, depending on what is on the other end of the RJ45). The device needs to have explicit support for the type of adapter - the switch software needs to know how to set the speed for a speed-converting one (so you can say "speed 100; duplex half") and the hardware needs to support multiple speeds for the non-speed-converting one. Some Cisco gear insists on one type (normally speed converting, like the Catalyst 3750G-12S), some insists on the other (the ASR1001-X, though it is fibbing - it works with both), and some insists on non speed converting).

There's no reason a manufacturer couldn't create a speed converting SFP+ that operates at 10/100/1000, although (per the above) if the switch software doesn't know about multiple speeds, you'd be stuck with forced auto-negotiation. That would take less power than the previously-thought-to-be-impossible SFP+/RJ45 10GBASE-T module, which is now available (at a high price) from a number of manufacturers.

 

[c6100]

I've gone into this in detail a few times previously, even in this thread. To recap - many switches with SFP+ ports (particularly ports that aren't dual SFP+/RJ45 selectable) only have hardware for port operation at ~10Gbit/sec. That hardware doesn't know what to do with data that arrives at other clock rates. Communication with the SFP/SFP+ for things like link status, etc. happens in the I²C channel, which is separate from the path used for user data, so a switch might report a link as "up" even though it won't pass data.

There are 2 flavors of Cisco(ish) SFP/RJ45 adapters - ones that do speed conversion internally (the SFP side always sends to the switch at GigE speeds) and ones that don't (the SFP side operates at 10/100/1000, depending on what is on the other end of the RJ45). The device needs to have explicit support for the type of adapter - the switch software needs to know how to set the speed for a speed-converting one (so you can say "speed 100; duplex half") and the hardware needs to support multiple speeds for the non-speed-converting one. Some Cisco gear insists on one type (normally speed converting, like the Catalyst 3750G-12S), some insists on the other (the ASR1001-X, though it is fibbing - it works with both), and some insists on non speed converting).

There's no reason a manufacturer couldn't create a speed converting SFP+ that operates at 10/100/1000, although (per the above) if the switch software doesn't know about multiple speeds, you'd be stuck with forced auto-negotiation. That would take less power than the previously-thought-to-be-impossible SFP+/RJ45 10GBASE-T module, which is now available (at a high price) from a number of manufacturers.

Thanks for the insight Terry. Simply put, is it your thoughts that there just isn't a way to use 1 Gbe with this switch?

Sent from my Nexus 6P using Tapatalk

 

[Terry Kennedy]

Thanks for the insight Terry. Simply put, is it your thoughts that there just isn't a way to use 1 Gbe with this switch?

I don't have one, so I can't say for sure. But if there's no mention of speeds other than 10GbE in the documentation and the CLI doesn't let you say "speed 1000" (or the equivalent configuration command on this switch), the hardware probably doesn't support it. As far as I remember (I'm not going to re-read all 30 pages of this thread ) there was only 1 user who reported success with a GigE SFP, and the exact brand / model was never disclosed. I may be mis-remembering, though.

 

[c6100]

I don't have one, so I can't say for sure. But if there's no mention of speeds other than 10GbE in the documentation and the CLI doesn't let you say "speed 1000" (or the equivalent configuration command on this switch), the hardware probably doesn't support it. As far as I remember (I'm not going to re-read all 30 pages of this thread ) there was only 1 user who reported success with a GigE SFP, and the exact brand / model was never disclosed. I may be mis-remembering, though.

In the manual I see

6.2.1.6 speed-duplex
Note:The 10-Giga Interface will not provide the following command
This command is used to set the speed and duplex mode for the interface.

It would appear to me that one can not set the speed on the 10 gb interfaces :-(

Sent from my Nexus 6P using Tapatalk

 

[Terry Kennedy]

It would appear to me that one can not set the speed on the 10 gb interfaces :-(

These switches are probably old enough that people weren't buying them for speeds other than 10GbE. The first switches with 10GbE were top-of-rack consolidation switches that had some number (12/24/48) of 10/100/1000 ports and a much smaller number* of 10GbE ports used to link that rack to a larger aggregation switch that was 10GbE-only or sometimes mostly 10GbE with some faster (40GbE) ports to uplink to the next level.

So there was no point in the silicon manufacturers adding support for multiple speeds on aggregation switches - 10GbE was expensive and if the customers needed 10/100/1000 ports in the aggregation rack they'd simply use another top-of-rack switch and uplink it to the aggregation switch.

As process technology improved and the silicon manufacturers had time to refine their chips, we start to see ports that support multiple speeds. As I mentioned, switches that have both a SFP+ and an RJ45 connector on a single port (although you can only use one or the other at any given time) normally have chips that support multiple speeds, at least on the RJ45 half of the port. If they don't support that on the SFP+ half, it is either a hardware limitation, the vendor enforcing SFP+ "lock-in" (if they don't sell a GigE SFP for that switch, the software will just say "unrecognized transceiver" or similar), or it is a simple oversight due to the manufacturer assuming that "anybody buying 10GbE switches wants to use the ports for 10GbE", as I mentioned in my first paragraph.

* History lesson - The very first switches that were "10GbE-capable" used XENPAK transceivers. The XENPAK-to-switch interface is basically a parallel port - all of the clocking, SERDES, etc. functions were done in the XENPAK. This made it very inexpensive for the manufacturer to claim they "supported" 10GbE - all they had to do was put one or two parallel ports with the right connector in the switch, which didn't cost much at all. But the XENPAK transceivers were very expensive, both because it was a new technology but also because of the complexity of the XENPAK. Most customers didn't buy those switches with a plan to use the 10GbE port(s) immediately - it was more of a "pay as you grow", and you paid a lot for the XENPAK if you decided you needed 10GbE uplink port(s). The next form factor was X2, which had the same electrical interface as XENPAK but traded an obnoxious, space-wasting mounting method (two "elephant ears" with thumbscrews) for a slightly smaller but far more bizarre mounting method (you had to pull on the fiber connector, which would extend from the X2 and release some internal catches). SFP+ was a (relatively) simple extension of the existing GigE SFP design, which is simply an electrical-to-optical (and back the other way) device - serial data on the fiber comes out the other end of the SFP / SFP+ as serial electrical signals. All of the clocking, SERDES, etc. is done in the chips soldered to the switch mainboard. That means that the SFP / SFP+ is much less expensive, but pushes the cost into the switch chassis. That reinforced the "per-port price is high, so we assume our customers want to use those ports only for 10GbE" thinking I mentioned above.