Quanta LB6M (10GbE) -- Discussion

[LeeSter]

Hi everybody. I really like messing around with this thing. I have a networking newbie question. (a couple)
Do I have to set an IP for an interface if I'm never going to route? Specifically on this switch, but switches in general.
My thoughts are, all I have to do is add the VLAN and associate the VLAN with the port. Speaking of... how to I change the port from trunk to access?
The command set on this thing is certainly odd. I find myself going back and forth between the quanta mesh and FASTPATH guides. Thankfully, I'm making full use of "?".

 

[Sleyk]

Thanks much for the post Sleyk. I followed your recommendations and did both the 3x case fan swap, as well as the 2x powersupply fan swap. I ended up using the higher RPM fans for the powersupply to get a bit more air flow (they're still almost completely silent).

The powersupply fans did indeed have the pins completely jumbled up from standard 3-pin fan headers. My fix to this was to take a very small precision flathead screwdriver and pop the pins out from the header and order them correctly for the Quanta LB6M.

Finally both my 24-bay 4U server as well as the Quanta LB6M switch are quiet!

Thanks again!

You are welcome my friend! I'm so glad this small mod was able to help! I think the switch performs well with slower fans, and it can take a little increase in temps with no problems. Should remain stable and good.

Hey Sleyk, Just curious, what are your temps now, with your mods?

Hey LeeSter, I am at work tonight, but as soon as I get home in the morning, I will take a small laser temp gauge and measure temps. As of right now, it has been wonderfully quiet since the mods, and it runs perfectly since day one. I actually haven't had to restart or reboot the switch at all since I modded the switch. Switch has been super stable even at the higher temps. I am also using the slowest scythe kaze 40mm fans. Will post temps for ya soon.

 

[Sleyk]

Hi everybody. I really like messing around with this thing. I have a networking newbie question. (a couple)
Do I have to set an IP for an interface if I'm never going to route? Specifically on this switch, but switches in general.
My thoughts are, all I have to do is add the VLAN and associate the VLAN with the port. Speaking of... how to I change the port from trunk to access?
The command set on this thing is certainly odd. I find myself going back and forth between the quanta mesh and FASTPATH guides. Thankfully, I'm making full use of "?".

As far as setting an ip address for one of the interfaces, do you mean setting a static ip for a management interface? I think it is good if you wanna ssh into the switch. I think i would set one for any switch that can be managed. Should come in handy when say, your network goes down or you change routers or install pfsense. That way, you don't have to look up your mac address table or something like that to see what ip address was given to the switch through dhcp.

Yeah, I find myself going back and forth between those papers too, seeking out the exact syntax commands. Sometimes, i see a command through the "?" command, but oftentimes it doesnt work, and I gotta refer to the docs for the exact method of input. It really is fun working with the switch though. I have mine setup all nice for me. I must admit though, I still haven't mastered truck ports and vlans as yet! I know it is so simple though. I think it is the only aspect of networking I need further reading up on. I not to long ago purchased a book on networking and will be reading up on this soon.

 

[josh]

Does anyone know how to set an access port on this switch? There's no switchport mode access command.

 

[Ub3r]

Has anyone here successfully configured any sort of VLAN routing? I'm having a hell of a time getting it to work.

Edit/Update: It seems like a lot of the commands simply don't do what they're supposed to. I've got a ton of experience with Cisco and the syntax differences aren't a big deal, however the switch seems to just not want to route. There's a pretty big chance I'm just stupid and forgetful so I figured I'd ask if anyone else has the same issue.

 

[acmcool]

I am still struggling to get trunk port working...Can anyone post their config that is working?Any help is appreciated

 

[segfault]

I am still struggling to get trunk port working...Can anyone post their config that is working?Any help is appreciated

It might be better to post your non-functional config. Some of us can have a look and something may be apparent when looking it.

 

[dswartz]

Mine just came from natex. Worked like a champ out of the box, and a LOT quieter than I had feared...

 

[PGlover]

Need help...

I am redesigning my home network with the approach of a small (100 or less people) size business network design in mind.

I have posted a preliminary network design for reconfiguring my network in the link below.
Router on a Stick -vs- Layer 3 Routing with Switch

I have a number of switches already:
1. Two Quanta LB4M
2. One Quanta LB6M
3. Three Juniper EX3300-24T

The goal is to have a DMZ zone and an internal network zone with VLANs. In the DMZ zone and internal network zone, there will be virtual machines. I only have 1 VM host and 1 SAN Server that needs to be shared with the DMZ zone and my internal network zone.

Additionally I want computers/servers in the DMZ zone and in the internal network zone to have direct access to file shares located on the SAN server.

Should I have the Quanta LB6M as my core switch and 2 of the Juniper EX3300-24T hanging off it. One to serve the DMZ zone and the other to server the internal network zone. Then my VM host and Storage server would be directly connected to the LB6M (core switch).

With this approach what are some of the security risks? Is there a chance that someone would be able to hack into the VM host and SAN server?

 

[wildchild]

What i would do :
Quanta seperated as 10g storage, internal lan and dmz as vlan;l3 lan routing done by junipers, dmz vlanned, but routed by fw

 

[PGlover]

What i would do :
Quanta seperated as 10g storage, internal lan and dmz as vlan;l3 lan routing done by junipers, dmz vlanned, but routed by fw

Wildchild... Can you provide a quick drawing on you proposal. A visual for me would make things easier as this network stuff is all new to me...

 

[whmcr]

What i would do :
Quanta seperated as 10g storage, internal lan and dmz as vlan;l3 lan routing done by junipers, dmz vlanned, but routed by fw

This is almost exactly what I've got being done, except with a number more vlans.

The L2 Layout is the below:

I've got a number of devices off the LB6M

NAS1 - LACP with 2x DACs (FS.com 30awg DAC - Intel vedor code) to the 10GB SFP Nic on a X10SDV-4C-7TP4F
Hyp1 - LACP with 2x DACs (FS.com 30awg DAC - Intel vendor code) to a Intel x520-DA2 on a X10SDV-TLN4F
CCR - LACP with 2x DACs (FS.com 30awg DAC - Generic Vendor code) to a Mikrotik CCR1036-8g-2s+ <-- this is routing between the VLANs
CRSCAB - Static LAG with 2x DAC (FS.com 30awg DAC - Generic Vednor code) to a Mikrotik CRS226-24G-2S+RM
CRSOffice - Static LAG with 2x SFP+ (FS.com 10g 850nm SR (300m) - Generic vendor code) to a Mikrotik CRS226-24G-2S+RM
TPSW1 - Single Ethernet to TPLink SG2210P

My Config on the LB6M is as follows

vlan database
vlan 2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48
vlan 50,52,54,56,58,60,1000-1005,1010,3000-3001,3996-4000
vlan name 2 "Name Removed"
!.....
!..... Vlan names removed
!.....
vlan name 4000 "Name Removed"
exit
configure
interface 0/1
no auto-negotiate
addport 1/1
exit
interface 0/2
no auto-negotiate
addport 1/1
exit
interface 0/3
no auto-negotiate
addport 1/2
exit
interface 0/4
no auto-negotiate
addport 1/2
exit
interface 0/5
no auto-negotiate
addport 1/3
exit
interface 0/6
no auto-negotiate
addport 1/3
exit
interface 0/7
no auto-negotiate
addport 1/4
exit
interface 0/8
no auto-negotiate
addport 1/4
exit
interface 0/9
no auto-negotiate
addport 1/5
exit
interface 0/10
no auto-negotiate
addport 1/5
exit
!
interface 0/1
description 'Hyp1-vmnic0'
snmp-server enable traps violation
exit

interface 0/2
description 'Hyp1-vmnic1'
snmp-server enable traps violation
exit

interface 0/3
description 'NAS-ix0'
snmp-server enable traps violation
exit

interface 0/4
description 'NAS-ix1'
snmp-server enable traps violation
exit

interface 0/5
description 'CCR-SFP1'
snmp-server enable traps violation
exit

interface 0/6
description 'CCR-SFP2'
snmp-server enable traps violation
exit

interface 0/7
description 'CRSOffice-SFP1'
snmp-server enable traps violation
exit

interface 0/8
description 'CRSOffice-SFP2'
snmp-server enable traps violation
exit

interface 0/9
description 'CRSCAB-SFP1'
snmp-server enable traps violation
exit

interface 0/10
description 'CRSCAB-SFP2'
snmp-server enable traps violation
exit

interface 0/28
description 'TPLSW1'
vlan participation include 8,12,18,20,22,24,28
vlan tagging 8,12,18,20,22,24,28
snmp-server enable traps violation
exit

interface 1/1
description 'Hyp1-lag0'
no port-channel static
vlan participation include 14,3999
vlan tagging 14,3999
snmp-server enable traps violation
exit

interface 1/2
description 'NAS-lagg0'
no port-channel static
vlan pvid 3999
vlan acceptframe admituntaggedonly
vlan participation include 3999
snmp-server enable traps violation
exit

interface 1/3
description 'LACP-CCR'
no port-channel static
vlan participation include 2,4,6,8,10,12,14,16,18,20,22,24,26,28,60,1000-1005
vlan participation include 1010,3000-3001,3997-3998
vlan tagging 2,4,6,8,10,12,14,16,18,20,22,24,26,28,60,1000-1005,1010,3000-3001
vlan tagging 3997-3998
snmp-server enable traps violation
exit

interface 1/4
description 'LACP-CRSOffice'
vlan participation include 8,12,18,26,60,3997-3998
vlan tagging 8,12,18,26,60,3997-3998
snmp-server enable traps violation
exit

interface 1/5
description 'LACP-CRSCab'
vlan participation exclude 60,3996-4000
vlan participation include 2,4,6,8,10,12,14,16,18,20,22,24,26,28,1000-1005
vlan participation include 1010,3000-3001
vlan tagging 2,4,6,8,10,12,14,16,18,20,22,24,26,28,1000-1005,1010,3000-3001
snmp-server enable traps violation
exit

My CRSes are similar to below

/interface ethernet
set [ find default-name=sfpplus2 ] master-port=sfp-sfpplus1
/interface vlan
add interface=sfp-sfpplus1 name=MGMT-Networking vlan-id=##############
/interface ethernet
set [ find default-name=ether2 ] master-port=sfp-sfpplus1
set [ find default-name=ether3 ]  disabled=yes master-port=sfp-sfpplus1
set [ find default-name=ether4 ] master-port=sfp-sfpplus1
set [ find default-name=ether5 ] master-port=sfp-sfpplus1
set [ find default-name=ether6 ] master-port=sfp-sfpplus1
set [ find default-name=ether7 ] master-port=sfp-sfpplus1
set [ find default-name=ether22 ] master-port=sfp-sfpplus1
set [ find default-name=ether23 ] master-port=sfp-sfpplus1
set [ find default-name=ether24 ] master-port=sfp-sfpplus1
set [ find default-name=sfpplus2] master-port=sfp-sfplus1
/interface ethernet switch trunk
add member-ports=sfp-sfpplus1,sfpplus2 name=trunk1
/interface ethernet switch egress-vlan-tag
add tagged-ports=trunk1,switch1-cpu vlan-id=######
add tagged-ports=trunk1 vlan-id=#####
add tagged-ports=trunk1 vlan-id=#####
add tagged-ports=trunk1 vlan-id=#####
add tagged-ports=trunk1,ether3 vlan-id=#####
add tagged-ports=trunk1 vlan-id=####
add tagged-ports=trunk1 vlan-id=####
/interface ethernet switch ingress-vlan-translation
add new-customer-vid=#### ports=ether23 sa-learning=no
add new-customer-vid=#### ports=ether22 sa-learning=no
add new-customer-vid=#### ports=ether2 sa-learning=no
add new-customer-vid=##### ports=ether4 sa-learning=no
add new-customer-vid=##### ports=ether7 sa-learning=no
/interface ethernet switch vlan
add ports=trunk1,switch1-cpu vlan-id=#####
add ports=trunk1,ether23 vlan-id=#####
add ports=trunk1,ether22 vlan-id=#####
add ports=trunk1,ether2 vlan-id=#####
add ports=trunk1,ether3 vlan-id=######
add ports=trunk1,ether4,ether7 vlan-id=#####
add ports=trunk1,ether4 vlan-id=#####
/ip dhcp-client
add default-route-distance=0 dhcp-options=hostname,clientid disabled=no interface=MGMT-Networking

The CCR is as follows

/interface bonding
add comment="Uplink to LB6M Switch" lacp-rate=1sec mode=802.3ad name=bonded_lb6m slaves=sfp-sfpplus1,sfp-sfpplus2 transmit-hash-policy=layer-2-and-3

 

[PGlover]

whmcr... Thanks for the drawing. I have a few questions:

1. In your design, which switches are considered the "core" switch and which ones are the "edge/access" switches.
2. What device are you using as your firewall?
3. What is the part numbers of the 10G DAC purchased for FS.com?
4. Do you have a DMZ zone? If yes, how is it incorporated in your setup? If no, how would you incorporate it in your setup?

 

[whmcr]

whmcr... Thanks for the drawing. In your design, which switches are considered the "core" switch and which ones are the "edge/access" switches. What device are you using as your firewall? What is the part numbers of the 10G DAC purchased for FS.com?

The LB6M would be at the core, as all of the other switches that devices are connecting to connect to that. The oddity is the CCR which is the router which has one of the internet connections directly connected, as there is a bit of a firmware bug on the modem (Virgin Media Superhub 3 doesn't work well in Modem mode if you're in a region with an Arris CMTS) and i've got it plumbed in directly to rule out some things.

For the firewall, the CCR1036 is being used with ACLs being applied for inter vlan traffic.

The DACs are #21254 for the generics, for the Intel's I ordered #40144 but I get the same results with the 21254's, so I'm not using the 40144's.

 

[PGlover]

The LB6M would be at the core, as all of the other switches that devices are connecting to connect to that. The oddity is the CCR which is the router which has one of the internet connections directly connected, as there is a bit of a firmware bug on the modem (Virgin Media Superhub 3 doesn't work well in Modem mode if you're in a region with an Arris CMTS) and i've got it plumbed in directly to rule out some things.

For the firewall, the CCR1036 is being used with ACLs being applied for inter vlan traffic.

The DACs are #21254 for the generics, for the Intel's I ordered #40144 but I get the same results with the 21254's, so I'm not using the 40144's.

whmcr... Got it.. As you mentioned before, the CCR is performing all the routing between Vlans. You have no Layer3 switches performing the routing. I noticed that CCR router has 2 SFP+ ports. I am using Pfsense and only have 1GbE ports on the WatchGuard X515 box. If I use your design with Pfsense performing all the routing between my Vlans and my DMZ zone, I would need to build a custom Pfsense box with 10GbE ports on it. I would think that the Pfsense box (1GbE ports) would be a bottleneck in the routing to the 10G core switch.

Additionally, why are you using a static LAG connection rather than a LACP connection to the LB6M from the CRS-CAB and CRS-Office switches?

 

[whmcr]

Got it.. As you mentioned before, the CCR is performing all the routing between Vlans. You have no Layer3 switches performing the routing. I noticed that CCR router has 2 SFP+ ports. I am using Pfsense and only have 1GbE ports on the WatchGuard X515 box. If I use your design with Pfsense performing all the routing between my Vlans and my DMZ zone, I would need to build a custom Pfsense box with 10GbE ports on it. I would think that the Pfsense box (1GbE ports) would be a bottleneck in the routing to the 10G core switch.

It's been a while since I was working with ACLs on L3 switches, but the last time I did, the main vendors were not doing stateful firewalls on L3 switches, as that would eat into their firewall markets

I'm not sending the storage networking via the router, so this cuts down a bit on the traffic.

If you're wanting to have SPI it's either going to be a lot of money on switching, or have a bit of a bottle neck on the router. For the CCR its able to be doing about 20-22 gbit of firewalling which gives a hard line of the maximum intervlan routing, but this is fine for me, as realistically, thats a very high number

 

[PGlover]

Thanks for the information whmcr.. I plan to post another design later today using the LB6M as the core switch. Using the LB6M as the core switch will definitely cut down on the 10G connections I need based on the original design.

Additionally, why are you using a static LAG connection rather than a LACP connection to the LB6M from the CRS-CAB and CRS-Office switches?

 

[whmcr]

Additionally, why are you using a static LAG connection rather than a LACP connection to the LB6M from the CRS-CAB and CRS-Office switches?

The CRS doesn't implement 802.3ad so only static LAGs can be used rather than LACP.

 

[PGlover]

I have a new layout using the Quanta LB6M as the core switch; however, I have a few basic network questions based on the design. Once again, I am not a network guru, so please be patience with me.

1. Do I really need a separate Juniper EX3300-24T switch for the DMZ zone? Can I just use the DMZ vlan on the core switch? The goal is to have virtual servers running in the DMZ and maybe up to 4 physical servers.
2. How will the LAN network talk to vlan 10, 20. and 30 on the core switch? I understand how the LAN network with talk to vlan 1 on the core switch because of the direct 10G connection to the vlan 1 port.
3. How will vlan 1, 10, 20, and 30 talk to each other on the core switch (Quanta LB6M)?

Also I am still fuzzy on how pfsense will do the DMZ traffic routing and filtering.

Thanks for all the comments so far...

 

[Fritz]

Awesome schematic.