Quanta LB6M (10GbE) -- Discussion

cypher526

New Member
Mar 1, 2017
2
1
3
41
Just got a LB6M recently off ebay and running great (cool and quiet) thanks to all the help from this thread. Found some good fans that are fairly quiet (not louder than your average 80mm or 120mm fan and way better than the stock whiny "jet engines") yet still offer more than ample cooling: Top Motor 40x28mm from coolerguys.com. (oh also, these are 4-pin PWM fans so you can still use show environment to keep an eye on fans/temps)

Here's my # show environment output after installing these (note: I have these configured for reverse air flow):

Temp (Celsius)................................. 37
Fan Speed, RPM................................. 4157
Fan Duty Level................................. 50%
Temperature traps range: 0 to 45 degrees (Celsius)

Temperature Sensors:
Unit Sensor Description Temp (Celsius) State
---- ------ ---------------- -------------- --------------
1 1 37 Normal

Fans:
Unit Fan Type Speed Duty level State
---- ------ ---------- -------- ---------- --------------
1 1 Fixed 4157 50% Operational
1 2 Fixed 4275 50% Operational
1 3 Fixed 4103 50% Operational

Power supplies:
Unit Power supply Type State
---- ------------ ---------- --------------
1 1 Removable Operational
1 2 Removable Not powered​

For comparison sake, with the stock fans (normal, exhaust air flow) temp was, I believe, around 41C (with around 50% duty cycle).

As for the the ps fans, I soldered up some (removable) extension wires with inline resistors (can't remember exact values, derp) to bring the 12V down to about 5.3V-ish. So now those guys are quiet too. Enough to knock down the "whine"-factor but still move some air.

I've heard mention that this switch is supposedly a Broadcom reference design (BCM56820, perhaps). Anyone have a schematic? Probably unlikely. I'm only using a handful of ports and would like to modify the switch (if possible, but probably not) to reduce overall power consumption (i.e. physically turn off unused ports / driver ICs). Not sure if it's even possible just an idea/theory. Wouldn't be surprised if the massive BCM56820 ASIC is what's chewing up all the power (~128W with virtually no load).
 
  • Like
Reactions: BackupProphet

Tester_02

New Member
Feb 26, 2017
10
2
3
51
Just got a LB6M recently off ebay and running great (cool and quiet) thanks to all the help from this thread. Found some good fans that are fairly quiet (not louder than your average 80mm or 120mm fan and way better than the stock whiny "jet engines") yet still offer more than ample cooling: Top Motor 40x28mm from coolerguys.com. (oh also, these are 4-pin PWM fans so you can still use show environment to keep an eye on fans/temps)
Nice writeup! I agree the resistor on the power supply bring it down to acceptable levels.
Nice to see that the replacement fans make the noise acceptable. I checked those fans out, and they are 9000r pm, so I would have thought they would be too noisy. I am guessing it's the design of the stock fans that is the issue, not the speed (I think stock showed 9000 rpm also). Thanks for the link, I actually ordered those fans, as I am still leary about the other modders doing the cutouts and adding in the 120mm fans (I was about to do this until your post).
Just to confirm, you reversed the flow of those 3 rear fans and are pushing air into the case, not out (stock).

Thanks!
 

Dravor

New Member
Aug 17, 2015
20
1
3
45
So I just ordered the fans that Cypher listed. Do we think we will see much of a different if we use the fans in the standard direction?

On a separate note, I have a computer on my 2nd floor that I want to connect, and server room is in the basement. It's way too long for a Twinax cable. So I want to run fiber, for about 50 ft. The question is what kind of fiber cable am I looking for? Would I buy a mellanox connectx-2 card, and then the fiber module for that card an the switch?

Was hoping to have a dual 10Gbe connection. That way I can move VM's between the basement and the upstairs without too much of an issue.

Thanks!
 

RADCOM

New Member
Aug 27, 2012
11
4
3
So I just ordered the fans that Cypher listed. Do we think we will see much of a different if we use the fans in the standard direction?

On a separate note, I have a computer on my 2nd floor that I want to connect, and server room is in the basement. It's way too long for a Twinax cable. So I want to run fiber, for about 50 ft. The question is what kind of fiber cable am I looking for? Would I buy a mellanox connectx-2 card, and then the fiber module for that card an the switch?

Was hoping to have a dual 10Gbe connection. That way I can move VM's between the basement and the upstairs without too much of an issue.

Thanks!
Yep that would work just make sure you get the right connection on the fibre (normally an LC connection). Finistar are a fair price and work very well.

Sent from my SM-N9005 using Tapatalk
 

thisn00bagain

New Member
Mar 2, 2017
2
0
1
42
Need some help here. running the lb6m in default mode (no configs made out side the factory defaults).

Tried testing the server and here is my setup:

Server A
- connected to port 1 via DAC

Server B
- connected to port 2 via DAC

Uplink to distribution switch 2960G on port 28 of LB6M. Servers should be talking to each other since they are local to the switch and uplink shouldn't matter.

I've verified that both ports are running in 10000 mbps on the servers themselves and the switches.

Code:
Settings for eth2:
    Supported ports: [ FIBRE ]
    Supported link modes:   10000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: No
    Advertised link modes:  10000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Speed: 10000Mb/s
    Duplex: Full
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: off
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes
Code:
Settings for enp3s0f0:
    Supported ports: [ FIBRE ]
    Supported link modes:   10000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: No
    Advertised link modes:  10000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Speed: 10000Mb/s
    Duplex: Full
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: off
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

Code:
(FASTPATH Routing) #show ip interface 0/1

Routing Interface Status....................... Down
Method......................................... None
Routing Mode................................... Disable
Administrative Mode............................ Enable
Forward Net Directed Broadcasts................ Disable
Proxy ARP...................................... Enable
Local Proxy ARP................................ Disable
Active State................................... Active
Link Speed Data Rate........................... 10000 Full
MAC address.................................... 08:9E:01:17:93:4A
Encapsulation Type............................. Ethernet
IP MTU......................................... 1500
Bandwidth...................................... 10000000 kbps
Destination Unreachables....................... Enabled
ICMP Redirects................................. Enabled

(FASTPATH Routing) #show ip interface 0/2

Routing Interface Status....................... Down
Method......................................... None
Routing Mode................................... Disable
Administrative Mode............................ Enable
Forward Net Directed Broadcasts................ Disable
Proxy ARP...................................... Enable
Local Proxy ARP................................ Disable
Active State................................... Active
Link Speed Data Rate........................... 10000 Full
MAC address.................................... 08:9E:01:17:93:4A
Encapsulation Type............................. Ethernet
IP MTU......................................... 1500
Bandwidth...................................... 10000000 kbps
Destination Unreachables....................... Enabled
ICMP Redirects................................. Enabled

Did some tests for transfer one large zip file over scp and nfs and cannot get any speeds over 140 MB/s .

Code:
scp -r logs.gz user@192.168.2.69:/mnt/ramdisk/
testfile.gz                                                                                                                                                                 100%  
14GB 143.9MB/s   01:39
thought it could be SSD limitation so made large RAMDISK partitions on each server and tried reading/writing to RAMDISKs. Still getting same results.



looking through the thread seems there is mixed results of changing MTU. Could this be the problem ? Should I change the MTU ?
 
Last edited:

Dravor

New Member
Aug 17, 2015
20
1
3
45
Need some help here. running the lb6m in default mode (no configs made out side the factory defaults).


looking through the thread seems there is mixed results of changing MTU. Could this be the problem ? Should I change the MTU ?
Have you tried using iPerf? Setup iPerf on your client, and then run the test from the other host. That is how we are all checking for max throughput.
 

dwright1542

Active Member
Dec 26, 2015
370
73
28
49
MTU's not going to help unless your processors are pegged, and most 10G NIC's have TCP offload anyway. SCP's got all kinds of overhead. Also, what kinds of NIC's are you using?

Here's what I'd do:

1. Test speed locally on both machines read and write.
2. connect the 2 machines locally via DAC and run an iperf
3. connect the 2 machines locally via DAC and run your SCP test
4. connect the 2 machines thru switch and run iperf

report back and let use know the findings.
 

thisn00bagain

New Member
Mar 2, 2017
2
0
1
42
Yeah Thanks,

The iperf confirmed link speed was about 9.46 gbps.

Not sure why the servers are unable to transfer between each other at any speed over 1 gbps.
I get the overhead part, but it shouldn't be about 9 gbps worth of it.
Same results over nfs, rsync, scp.
 

Dravor

New Member
Aug 17, 2015
20
1
3
45
Yeah Thanks,

The iperf confirmed link speed was about 9.46 gbps.

Not sure why the servers are unable to transfer between each other at any speed over 1 gbps.
I get the overhead part, but it shouldn't be about 9 gbps worth of it.
Same results over nfs, rsync, scp.
What make/model disks are you using on both ends?

Using RAID at all?
 

Tester_02

New Member
Feb 26, 2017
10
2
3
51
As an FYI, adding in resistors on the rear fans is not good. Besides very low airflow, then end up hunting in speed and sound terrible.
I was trying it as a temporary solution until the fans suggested by chpher256 show up. I had high hopes that since the resistors worked for the power supply fans, it could work for the rear fans.
And does that CPU (what looks like a CPU with a big heatsink) in the middle ever get hot with low airflow! The temperature sensor can't be for the CPU, as even with the stock jet engines, it's really Hot.
I also tried some 1GB ethernet to SPF adapter and it does not work. I had read these work on almost all sfp+ ports in most routers. it does seem to confirm that this LB6M does not negotiate down.
 

Marek J

New Member
Nov 24, 2016
16
2
3
48
Hello guys,

I am relative fresh with that switch, but I can say, it's quite challenging.
I am trying to get VLAN to work.
I have two Dell Poweredge servers, with two hosts and two test guests running on each host.

I use two ports 0/1 for dell1 (as bellow on pictures) and 0/7 for dell2 (as bellow on pictures)
I build network:test_vlan (as bellow on pictures)
Guest test1: IP:10.4.0.10
Guest test2: IP:10.4.0.11
There are pingable each other through normal vSwitches (as bellow on pictures)
001.PNG
002.PNG
003.PNG

004.PNG
005.PNG
006.PNG
007.PNG

My initial config:

!Current Configuration:
!
!System Description "Quanta LB6M, 1.2.0.18, Linux 2.6.21.7"
!System Software Version "1.2.0.18"
!System Up Time "0 days 1 hrs 47 mins 9 secs"
!Additional Packages FASTPATH QOS
!Current SNTP Synchronized Time: Not Synchronized
!
serviceport protocol none
serviceport ip 10.3.0.200 255.255.255.0 0.0.0.0
vlan database
exit
ip ssh server enable
no ip telnet server enable
configure
aaa authentication enable "enableList" enable
line console
exit
line telnet
exit
line ssh
exit
spanning-tree configuration name "08-9E-01-13-A6-96"
!

--More-- or (q)uit

router rip
exit
router ospf
exit
exit

OK. Then I try to setup VLAN ID=10 on interfaces 0/1 and 0/7.
I also went for the firmware upgrade to 1.2.0.18 just in case.


The config looks like bellow:

!Current Configuration:
!
!System Description "Quanta LB6M, 1.2.0.18, Linux 2.6.21.7"
!System Software Version "1.2.0.18"
!System Up Time "0 days 1 hrs 52 mins 44 secs"
!Additional Packages FASTPATH QOS
!Current SNTP Synchronized Time: Not Synchronized
!
serviceport protocol none
serviceport ip 10.3.0.200 255.255.255.0 0.0.0.0
vlan database
vlan 10
vlan routing 10 1
exit
ip ssh server enable
no ip telnet server enable
configure
ip routing
aaa authentication enable "enableList" enable
line console
exit
line telnet
exit
line ssh

--More-- or (q)uit

exit
spanning-tree configuration name "08-9E-01-13-A6-96"
!
interface 0/1
vlan pvid 10
vlan participation include 10
exit
interface 0/7
vlan pvid 10
vlan participation include 10
exit
interface 2/1
routing
exit
router rip
exit
router ospf
exit
exit

I setup VLAN ID=10 on both vSS and VLAN ID on both interfaces inside the guests:
008.PNG
009.PNG
010.PNG
011.PNG

Sorry guys to ask, but am I missing something please?
I went through couple of config examples and I end up with destination host unreachable :(
I really appreciate your help.

Cheers,
Marek
 

sthsep

Member
Mar 7, 2016
72
10
8
I'm not a expert nor it's my daily job just some experience from my lab.
I dont get why you set a fixed vlan and then give to the vswitch also a vlan.
You fixed your vlan to 10 so you dont need to set it at the vswitch and also not on the guest.
So if you have only one vlan at a port just add a normal vswitch port (vlan empty or 0) and dont set a vlan inside vm.
If you have more vlans you need VLAN tagging. On the switch it looks something like this:

mtu 9216
vlan pvid 999
vlan participation include 5,10,20,30,50,100,999
vlan tagging 5,10,20,30,50,100,999

then you can set the vlans in the vswitch BUT you dont need to set the vlans at guest level!!!
If you want to set vlans at guest level you need also to set the vswitch to trunking (Just set the vlan of the vswitch port to 4095) then you need at guest to set the vlan as you have done on the lastest picture.

So remove the vlan at guest and also at the port and it should work.
 
  • Like
Reactions: Dravor and Marek J

Marek J

New Member
Nov 24, 2016
16
2
3
48
Hi sthsep,

Thanks man :)
You are hero.
Your solution works.
I just had to remove VLAN IDs from the guest interfaces and added tagging to each port participated and everything works.
I made the extra ip on the interface 2/1 to be sure, it's pingable both sides.
I need to keep VLAN IDs inside the hypervisors, due I go for the quite advanced config, juggling with Starwind, Video monitoring (I'm big fan of Unifi last days, because it just works ;) ) etc.

Only one thing makes me insane: the lack of GUI (sometimes, I like to quick have a look what happens over there, due I work remotely on this system mostly).

My start config is just bellow:

!Current Configuration:
!
!System Description "Quanta LB6M, 1.2.0.18, Linux 2.6.21.7"
!System Software Version "1.2.0.18"
!System Up Time "0 days 12 hrs 39 mins 39 secs"
!Additional Packages FASTPATH QOS
!Current SNTP Synchronized Time: Not Synchronized
!
serviceport protocol none
serviceport ip 10.3.0.200 255.255.255.0 0.0.0.0
vlan database
vlan 10
vlan routing 10 1
exit
ip ssh server enable
no ip telnet server enable
configure
ip routing
aaa authentication enable "enableList" enable
line console
exit
line telnet
exit
line ssh

--More-- or (q)uit

exit
spanning-tree configuration name "08-9E-01-13-A6-96"
!
interface 0/1
vlan pvid 10
vlan participation include 10
vlan tagging 10
exit
interface 0/7
vlan pvid 10
vlan participation include 10
vlan tagging 10
exit
interface 2/1
routing
ip address 10.4.0.1 255.255.255.0
exit
router rip
exit
router ospf
exit
exit

012.PNG 013.PNG
 
  • Like
Reactions: Fritz

Marek J

New Member
Nov 24, 2016
16
2
3
48
Hi,

I removed vlan pvid 10 from interfaces, even ip routing from configure and everything still works, but of course, due some laziness ;) , I have the next question:
do I have any chance to aggregate couple of vlan parameters, then "link" them to the ports specific please?
It's fine to setup two ports, but what about 20 :) ?

My current test config...:

!Current Configuration:
!
!System Description "Quanta LB6M, 1.2.0.18, Linux 2.6.21.7"
!System Software Version "1.2.0.18"
!System Up Time "0 days 14 hrs 32 mins 50 secs"
!Additional Packages FASTPATH QOS
!Current SNTP Synchronized Time: Not Synchronized
!
serviceport protocol none
serviceport ip 10.3.0.200 255.255.255.0 10.3.0.250
vlan database
vlan 10
vlan name 10 "VLAN 10"
vlan routing 10 1
exit
ip ssh server enable
no ip telnet server enable
configure
aaa authentication enable "enableList" enable
line console
exit
line telnet
exit
line ssh

--More-- or (q)uit

exit
spanning-tree configuration name "08-9E-01-13-A6-96"
!
interface 0/1
vlan participation include 10
vlan tagging 10
exit
interface 0/7
vlan participation include 10
vlan tagging 10
exit
interface 2/1
routing
ip address 10.4.0.1 255.255.255.0
exit
router rip
exit
router ospf
exit
exit
 

Oxzhor

Member
Jan 11, 2017
30
3
8
38
The Netherlands, Amsterdam
I install TFTPD32 : an opensource IPv6 ready TFTP server/service for windows : TFTP server on my pc and start it, then i ssh to the switch and follow the comands


login as: admin
admin@192.168.1.100's password:

(FASTPATH Routing) >enable
Password:

(FASTPATH Routing) #show bootvar

Image Descriptions

active :
backup :


Images currently available on Flash

--------------------------------------------------------------------
unit active backup current-active next-active
--------------------------------------------------------------------

1 1.2.0.14 1.2.0.14 1.2.0.14 1.2.0.14

(FASTPATH Routing) #copy active backup
Copying active to backup
Copy operation successful

(FASTPATH Routing) #copy tftp://192.168.1.106/1.2.0.14

Command not found / Incomplete command. Use ? to list commands.

What do i wrong here?
Somebody know how to solved this?
 

Marek J

New Member
Nov 24, 2016
16
2
3
48
Somebody know how to solved this?
copy active tftp://<ip-of-pc-running-tftp-server>/active
copy backup tftp://<ip-of-pc-running-tftp-server>/backup
if you copy your active or backup to tftp server

or
copy tftp://192.168.255.111/newimage active
copy tftp://192.168.255.111/newimage backup
if you update your newimage from tftp server
 
Last edited:
  • Like
Reactions: Oxzhor

sthsep

Member
Mar 7, 2016
72
10
8
Marek J I dont quite understand what you want to achieve? Can you make a small picture/plan what you want or rephrase?
Now I understand that you want set your vlans at guest level because you have multiple vlans per VM. If that was the question you have to options. 1) Create a portgroup for each vlan and then just add that much virtual nics to your VM.
The second methode that I already mentioned before is that you create a "tagging" portgroup then you can set the vlan at nic level.

Offtopic: Also you mentioned that you use starwind right? Do you use Starwind Virtual San (Free?) and if so do you recommend it? Was looking quite often to setup a Starwind Testplattform but I have still not done it yet.