Quanta LB6M (10GbE) -- Discussion

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

graanco

New Member
Nov 15, 2016
3
0
1
52
Hi, I just received one this past week it seems to work good. I am running it in a VMWARE VSAN env and intergraded it into the network at the same time. My vmotions are screamingly fast but for datastorage transfers; they are very slower than a snail. Could the network traffic be creating the slow down. I had tested it with jumbo frames enable, and just came across with more VSAN issues. Does anyone have any thoughts in how to improve the bandwidth for the backside of the vm env datastorage transfers?
 

Drewy

Active Member
Apr 23, 2016
208
56
28
54
If you are doing vmotions of vm's on vsan storage to another compute node that is also running vsan then it's likely that the "vmotion" is only actually transferring memory contents from 1 node to the next (if the vm is up). The 2nd node already has the vm on it's vsan storage.
When you move a vm from one data store to another you are moving the whole shebang.
 

graanco

New Member
Nov 15, 2016
3
0
1
52
I wish that was it... but no one thing I found was the ups was being over loaded and making the switch reboot very 3 hours... well since that been resolved, I decide to trunk my vm hosts to a vSwitch. and vlan throttle the networks. I gave vSan 5gs to max 8gd with a burst for the whole switch. One things I learned for there this provisioning setting for cold storage moves (not vmotion) and I turned that on. Well with all that speed from a 10g switch.. and basically I am still not seeing any improvements at all on the back plane for the systems. I was thinking moving away from FC and using iSCSI for the backup NAS but I am not sure at this time I could recommend. Does VMWARE off any tools to diagnose the bottlenecks?
 

gr8ape

Member
Jun 15, 2015
42
4
8
Have you tried using iperf from one ESXi to the other to determine if the switch is the bottleneck?
 

khrystoph

New Member
Nov 16, 2016
2
1
3
38
So, I've been digging through threads and I was able to figure out and set up ssh via DHCP on the LB6M (which I got off ebay for $260 all costs) on the management port.

What I'm currently trying to figure out is how I can set an ssh public key to login via ssh leveraging RSA key exchange. I've seen a few mentions here and there as well as on the cli that show that you can do something like this:
copy scp://user@host/path/to/key/file.pem nvram:sshkey-rsa2

However, what I'm curious about is if this is actually the ssh HOST key or the ssh User key. Is it possible to set up passwordless login to this switch?

Also, I haven't had the opportunity to check out a lot of features, but will share some of my automation scripts once I get smaller things (like the ssh conundrum) ironed out.

Also, all my efforts in turning up a way to update the firmware has basically gone the way of nothing. Has anyone tried to load an OpenSource 3rd party software and been successful?
 
  • Like
Reactions: fvanlint

c6100

Member
Oct 22, 2013
163
1
18
USA
I wish that was it... but no one thing I found was the ups was being over loaded and making the switch reboot very 3 hours... well since that been resolved, I decide to trunk my vm hosts to a vSwitch. and vlan throttle the networks. I gave vSan 5gs to max 8gd with a burst for the whole switch. One things I learned for there this provisioning setting for cold storage moves (not vmotion) and I turned that on. Well with all that speed from a 10g switch.. and basically I am still not seeing any improvements at all on the back plane for the systems. I was thinking moving away from FC and using iSCSI for the backup NAS but I am not sure at this time I could recommend. Does VMWARE off any tools to diagnose the bottlenecks?
I was considering the same switch for vSAN as well. Were you able to overcome the latency issues with it?
 

c6100

Member
Oct 22, 2013
163
1
18
USA
I have learned that Broadcom offered two different versions of the FASTPATH software, FASTPATH-SMB and FASTPATH-ENT. Some documents also refer to the "Advanced" feature set. I am not sure if that means FASTPATH-ENT or not. So far, I have not seen any keys or information on how to upgrade from -SMB to -ENT, or to add individual features. It seems like you get one or the other. Did all LB6M switches come with the same FASTPATH feature set, or are there different switch versions?

For vSAN I really just need Multicast support. Has anyone besides me tried to set up vSAN on one of these switches? Could upgrading the firmware help my cause?
Were you ever able to get vSAN working with this switch?
 

c6100

Member
Oct 22, 2013
163
1
18
USA
Has anyone had any luck with 1G copper SFPs in the SFP+ slots? I have tried Finisar FCLF-8521-3, eNet GLC-T-ENC, and Mikrotik S-RJ01, but none of them are working.


Admin Physical Physical Link Link LACP Actor
Intf Type Mode Mode Status Status Trap Mode Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
0/8 Enable 10G Full Down Enable Enable long

Interestingly I never see any Phy mode other than 10G Full. And the usual Brocade "speed-duplex 1000-full" command is not allowed in interface.

"interface speed" only lists 10 and 100, and those are not valid modes for the 10G ports.

I have some 10G DAC cables that are working fine. But I have some older gear as well I'd like to connect at 1G.

Edit: Went back and re-read the entire thread. Looks like nobody else has gotten this working either.

What happens if you set the physical interface to 100?
 

Evan Battle

New Member
Nov 22, 2016
12
9
3
47
I have learned that Broadcom offered two different versions of the FASTPATH software, FASTPATH-SMB and FASTPATH-ENT. Some documents also refer to the "Advanced" feature set. I am not sure if that means FASTPATH-ENT or not. So far, I have not seen any keys or information on how to upgrade from -SMB to -ENT, or to add individual features. It seems like you get one or the other. Did all LB6M switches come with the same FASTPATH feature set, or are there different switch versions?

For vSAN I really just need Multicast support. Has anyone besides me tried to set up vSAN on one of these switches? Could upgrading the firmware help my cause?
 

Evan Battle

New Member
Nov 22, 2016
12
9
3
47
I have learned that Broadcom offered two different versions of the FASTPATH software, FASTPATH-SMB and FASTPATH-ENT. Some documents also refer to the "Advanced" feature set. I am not sure if that means FASTPATH-ENT or not. So far, I have not seen any keys or information on how to upgrade from -SMB to -ENT, or to add individual features. It seems like you get one or the other. Did all LB6M switches come with the same FASTPATH feature set, or are there different switch versions?

For vSAN I really just need Multicast support. Has anyone besides me tried to set up vSAN on one of these switches? Could upgrading the firmware help my cause?
I have VSAN up and running with this switch, both VSAN 6.2 and VSAN 6.5. My setup appears to have the same feature set yours does:

Additional Packages............................ FASTPATH QOS
FASTPATH Routing

I enabled IGMP querier on my VSAN VLAN, and confirmed multicast functioned as expected using tcpdump-uw (vSphere 6.5 has a nifty VSAN tester in the GUI that will test multicast too)
 
  • Like
Reactions: alex1002 and c6100

c6100

Member
Oct 22, 2013
163
1
18
USA
I have VSAN up and running with this switch, both VSAN 6.2 and VSAN 6.5. My setup appears to have the same feature set yours does:

Additional Packages............................ FASTPATH QOS
FASTPATH Routing

I enabled IGMP querier on my VSAN VLAN, and confirmed multicast functioned as expected using tcpdump-uw (vSphere 6.5 has a nifty VSAN tester in the GUI that will test multicast too)
That's awesome to hear it indeed works! If you don't mine, what type of performance are you seeing from vSAN in your proactive tests in vCenter?

Sent from my Nexus 6P using Tapatalk
 

Evan Battle

New Member
Nov 22, 2016
12
9
3
47
I have VSAN up and running with this switch, both VSAN 6.2 and VSAN 6.5. My setup appears to have the same feature set yours does:

Additional Packages............................ FASTPATH QOS
FASTPATH Routing

I enabled IGMP querier on my VSAN VLAN, and confirmed multicast functioned as expected using tcpdump-uw (vSphere 6.5 has a nifty VSAN tester in the GUI that will test multicast too)
[ESXHOST:~] tcpdump-uw -i vmk3 -n -s0 -t -c 20 udp port 12345 <<- Command to verify the vmKernel port
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk3, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 10.1.0.34.17089 > 224.1.2.3.12345: UDP, length 200 <<-- responses from each f the hosts
IP 10.1.0.32.55200 > 224.1.2.3.12345: UDP, length 200

[ESXHOST:~] nc -uz 224.1.2.3 12345 <<--Command to generate some multicast traffic
Connection to 224.1.2.3 12345 port [udp/*] succeeded!
 

Evan Battle

New Member
Nov 22, 2016
12
9
3
47
That's awesome to hear it indeed works! If you don't mine, what type of performance are you seeing from vSAN in your proactive tests in vCenter?

Sent from my Nexus 6P using Tapatalk
My issue currently is that I'm using an unsupported SAS controller, and only a single 10Gb NIC in each host, So when I apply a load, with other VLAN's on the same port i notice a spike in write latency, read latency is almost always in the 1ms or less range (I'm running an all flash setup), but i've seen write latency hit 100ms. The only place i really feel it is in a couple of VMs running docker containers that consume storage via NFS across that same port. I have 7 active VLAN's going in and out of the single port.

I have ordered additional multiport 10GbE adapters, but have not installed yet to see if the issue is resolved. Local disk latency isn't a problem, thats why I've focused on the single NIC port as the culprit. I've been able to measure up to 1GB/s (~8Gb/s) read and write into and out of the VSAN datastore, so sequential access (storage vMotion) can easily saturate the link.

BTW, this is just a lab setup.
 
  • Like
Reactions: c6100

Evan Battle

New Member
Nov 22, 2016
12
9
3
47
My issue currently is that I'm using an unsupported SAS controller, and only a single 10Gb NIC in each host, So when I apply a load, with other VLAN's on the same port i notice a spike in write latency, read latency is almost always in the 1ms or less range (I'm running an all flash setup), but i've seen write latency hit 100ms. The only place i really feel it is in a couple of VMs running docker containers that consume storage via NFS across that same port. I have 7 active VLAN's going in and out of the single port.

I have ordered additional multiport 10GbE adapters, but have not installed yet to see if the issue is resolved. Local disk latency isn't a problem, thats why I've focused on the single NIC port as the culprit. I've been able to measure up to 1GB/s (~8Gb/s) read and write into and out of the VSAN datastore, so sequential access (storage vMotion) can easily saturate the link.

BTW, this is just a lab setup.

Just a little more detail on the VSAN performance I'm seeing, I just modified my storage policy and am now resyncing ~.5TB. on each node, i'm getting between 300MB/s and 550MB/s through the VSAN VMK on each node(this is pretty much the limits of the disks i'm using, 1 cache drive and 1 capacity SSD per node), and between 15,000 and 30,000 IOPS on each node (4 nodes). So this switch, I can safely say is NOT a bottleneck for a VSAN setup.