Quanta LB6M (10GbE) -- Discussion

TheBloke

Active Member
Feb 23, 2017
200
40
28
41
Brighton, UK
PS. Of course this ACL stuff requires that the traffic actually hits the switch. You said:

I applied the rules on firewall but seems that the traffic never hits the firewall since it could be on the same Virtual environment (different VMs).
You'd need to be sure that the traffic between VMs is actually going out through the physical NIC(s) to the switch, and is not being routed internally inside the VM software. Otherwise no config on the switch could help.

I don't know what virtualisation software you're using, but I'd expect most decent software to also provide virtualised networking. VMWare ESX does for example, where you can create a virtualised switch to which your VMs connect and apply restrictions there like you would on a real switch. That can include advanced ACLs (here's an example, a blog post describing traffic filtering rules in a vSphere Virtual Distributed Switch.)

So if all your multi-tenants are on the same virtualisation platform, it might be you can do what you want entirely with network virtualisation and don't need config on the physical LB6M (except perhaps as a second line of defence.)
 

ReeseRiverson

New Member
Oct 15, 2018
4
1
3
United States
Hello everyone,

I still consider myself relatively new to networking, despite what I've been dabbling in thus far between my home setup and our small business's configuration. Primarily one of my friends helped get me into the used enterprise gear market for various needs, and to be quite honest I really wouldn't want to give that up! :)

So my experience with networking has mostly consisted of running my own networking cable throughout the home and office, and planning out where I need to go with it. Of course setting up the patch panels and connecting everything to the network switch. My first managed network switch is my Cisco units, then the Blade switches.

Since I'm finally getting ready to explore 10gbe networking, I've explored my options and had stumbled upon this thread which interested me into getting the Quanta LB6M, which sounded like the perfect solution.

However, I feel this is where I'm getting myself sent out into the battlefield, so to speak. My whole system is also my learning tool, but there's a lot I'm not quite understanding and/or I may not be configuring correctly.

I have read through quite a few pages, but not all the pages of this thread since there's so much to take in, on top of everyone's needs/uses are going to vary.

So I'm going to start with listing what I have currently both at home and our business.

Home:
Servers:
  • 1u Supermicro Mini ITX Xeon-D Server - PfSense Router (This hardware was originally purchased to be used as a FreeNAS server, but plans changed for this as soon as I got my hands on a nice 2u chassis.)
  • 2u Dell PowerEdge R710 - ESXi
  • 2u Supermicro 12 x 3.5" bay chassis - FreeNAS (Primary)
  • 2u Supermicro 12 x 3.5" bay chassis - Spare / Future Expansion
  • 1u Dell PowerEdge R410 - FreeNAS (Secondary Backup - For more important data that's stored on the primary FreeNAS)
Network Switches:
  • Cisco Catalyst 4948 (Currently the primary switch)
  • Cisco SG300-10PP (PoE switch for the IP cameras and two Ubiquiti APs)
  • Blade G8000 with two SFP+ modules (Not in use right now.)
  • Quanta LB6M (Not in use right now.)

Office:
Servers:
  • 2u Dell PowerEdge R720xd - ESXi
  • 1u Dell PowerEdge R410 - PfSense Router
Network Switches:
  • Blade G8000 with two SFP+ modules (Primary)
  • Quanta LB6M (Not in use right now.)

So at home, I'm utilizing dual WAN on the PfSense Router since we have two ISPs on hand, a faster connection from a local ISP and a DSL connection for backup.

Currently on LAN from there goes to the first interface port on my Cisco Catalyst 4948. Then everything else connects to this same switch. I've even managed to configure my FreeNAS box to utilize LACP with this recently. Though I did that through the GUI that Cisco provides for management.

Otherwise I have little to no experience with CLI for the network switches, though probably only the basics like maybe just assigning VLANs.

Right now I'm not using VLANs to separate anything within the network. Primarily right now I just have the FreeNAS be configured for us to have our own shares, plus a separate one for the media. Which that is accessed by Emby running on Ubuntu under a VM on my Dell R710. I plan to run more on ESXi later.

Though the primary goal with the Quanta LB6M is to upgrade actual servers to utilizing 10GbE. I want to get the ESXi and FreeNAS box on 10GbE. Then possibly have the LB6M provide my computer and one or both of mom's computers 10GbE as well.

Effectively the same/similar idea is to be done at the office as well, where the LB6M will provide the server and some office workstations 10GbE. At the office I originally ran shielded Cat6A cabling for preparing for 10GbE, but after seeing the costs of SFP+ modules that'd provide 10GBASE-T or 10GBASE-T switches I figured the LB6M would provide the best value. I don't mind running OM3 fiber cable.

So that brings me to what I'm working with right now. So far only one OM3 cable arrived for me to try the LB6M out with. I stuck an Intel X520-DA2 into my R710 and connected the OM3 cable between the two. Both ends using the Cisco FET-10G 10-2566-02 transceivers.

Then I plugged one of the 1GbE ports on the Quanta into my Cisco switch and the serial console cable to a laptop. Thus far I've managed to see that the Quanta shows to show a link on both, and identifying the link as up on both interface 0/1 and 0/27. "Show network" even gives me a DHCP given IP address within the console. Though I'm not sure what else to really do here.

ESXi on the R710 recognizes that I've added the Intel X520-DA2, and and with the one transceiver plugged in it also shows up as an interface I can add. Though it shows the link to be down. Despite the LED on the card showing some form of activity along with the indication on the Quanta. Thus no IP received from the DHCP server either.

The other OM3 cables should arrive today so I should be able to expand my test platform to being able to tap off the Blade G8000.

A lot of how-tos I've been reading around keep showing various VLAN configurations on this switch, which I hadn't planned to be using on this switch. Which might be a no-no, I'm not sure as I'm trying to learn how configure my whole network setup. So far with the 1GbE side, it's relatively easy since it's been plug-n-play experience or a few minor tweaks within the Cisco GUI to take care of what I needed it to. Of course, if the Quanta was that easy, it wouldn't be a good learning tool for me, now would it? :)

So what do I really need to do with the Quanta LB6M as far as configuration? I've never configured SFP+ interfaces before, so I don't know if there are any special requirements needed to get it to work with another component in a system.

The idea is to use the Blade G8000's SFP+ interfaces with the Quanta so the 1GbE switch has high speed traffic to the 10GbE switch that will handle the higher speed traffic for the servers and certain machines.

I've read the article that STH has on using the LB6M, but I'm not really understanding what OSPF is, or other options, or certain configuration with what's going on there, since it looks like it's going on with different subnets. :confused::eek:

Anyway, thank you guys in advance for your time, I hope I've provided enough information to at least get the idea out on what I'm wanting to do here. ;)
 

narapon

New Member
Mar 31, 2017
13
1
3
31
If I measured it correctly, the chipset heatsink has 5.5cm by 5.5cm mounting hole distance center to center, would a xeon-d heatsink fit?
 

ReeseRiverson

New Member
Oct 15, 2018
4
1
3
United States
So what do I really need to do with the Quanta LB6M as far as configuration? I've never configured SFP+ interfaces before, so I don't know if there are any special requirements needed to get it to work with another component in a system.
Alright, so to update the situation at hand... this confirms my exact statement. More specifically with the SFP+ interfaces.

It turns out the Quanta LB6M's configuration wasn't the issue, in fact it's because of the transceiver compatibility on the X520-DA2 adapters. The Cisco ones they came with... well just doesn't work. I ended up testing the card in my Windows 7 machine and event viewer said it was an unsupported SFP+ module. So that explains why I'm having issues with my test. :eek:

The good news is, my IBM/Blade and Cisco transceivers all seem to be compatible with the Quanta LB6M just fine. I ended up firing up my Blade G8000 and hooking the Quanta up to it via SFP+, then a laptop up to one of the 1GbE ports. Everything simply just worked. :cool:

So right now I'm going to plan on getting the correct SFP+ modules for the X520-DA2s, until then I can't really move on with anything. :D
 

Sleyk

Well-Known Member
Mar 25, 2016
1,184
559
113
Stamford, CT
Hey Reese! Welcome to STH my brother!

I read your whole first post and I noticed you were using those Cisco FET transcievers, and I think they were your culprits :.)

I do believe that if you flash the LB6M to the Brocade firmware it would work (confirmed earlier in thread I believe) but I haven't actually bought any of the Cisoc FET's to test out meself. There was a time I and a few others were trying to get some 1Gb 'ceivers to work.

Also another question my fam, ever consider using the Mellanox line Connectx-2 adaps work great and they're super cheap. Heck, I'm using the old Chelsio's and they work great, but I know you have to use the hardware you have available to you already.

Lemme know what you think?
 

ReeseRiverson

New Member
Oct 15, 2018
4
1
3
United States
Hey Reese! Welcome to STH my brother!

I read your whole first post and I noticed you were using those Cisco FET transcievers, and I think they were your culprits :.)
Yo Sleyk, thanks for the welcome! :)

They were certainly on the Intel X520-DA2 card, I just didn't catch on until I check it out on my Windows 7 rig.

I do believe that if you flash the LB6M to the Brocade firmware it would work (confirmed earlier in thread I believe) but I haven't actually bought any of the Cisoc FET's to test out meself. There was a time I and a few others were trying to get some 1Gb 'ceivers to work.
Actually the Cisco transceivers work perfectly fine with my LB6M as-is, since I'm getting fully functional networking between it and my Blade G8000 with them. The real issue was just the X520-DA2 cards being picky. ;)

Also another question my fam, ever consider using the Mellanox line Connectx-2 adaps work great and they're super cheap. Heck, I'm using the old Chelsio's and they work great, but I know you have to use the hardware you have available to you already.

Lemme know what you think?
I've read about the Mellanox cards being used, but I wasn't sure what models to look into. I certainly would have no problems considering them, especially in other machines that really wouldn't need dual SFP+, especially considering the price points.

With this being my first giant leap into network advancements, I've got quite a bit to learn from. Step one was figuring out transceiver compatibility with the Intel cards. :D

Which I did get some compatible transceivers ordered. One of which I'm trying from FS. I hope it arrives by time the weekend comes around. I know I'm getting some DACs in for sure. So if anything I can get my Dell R710 and Supermicro 12-bay chassis fully switched over to the LB6M. Especially when I get some final pieces to get everything finally switched over to the rack I got in today.

(It's not very fun working with machines simply stacked on top of each other... The more I keep adding, the more reason for a proper rack to be set in place. :) )
 
  • Like
Reactions: Sleyk

slimshizn

New Member
Oct 26, 2018
3
0
1
Hey guys thanks for all the info on this forum. I have read over and seem to have it working, but would like to have LACP working. My hopes are to have interface 0/1 and 0/2 to my unifi 48 switch. I see that LACP mode is already active on all ports, so that being said if I just plug in the ports on 0/1 and 0/2 will they automatically be ready? Aggregation is setup on the unifi switch already. ( I did read over page 12, but had a hard time understanding it, just looking for guidance). Other than that I have 3 servers and one PC to plug in that I don't need aggregation on at this time. Will LACP mode being enabled on those ports create an issue?


Edit: Think I may have got it, testing it out before I make the full switch to this switch. lol

show port all shows


1/1 Enable Up Disable N/A N/A


shouldn't LACP Mode here be Enable instead of N/A?
 
Last edited:

slimshizn

New Member
Oct 26, 2018
3
0
1
So I have a backup/test server connected, seems that when using jumbo frame on my main pc, I couldn't connect to the server. Disabling it fixed it. (Jumbo frames is on switch and all servers). Not sure why that happened. I'll play around with it for a bit see how it goes.
 

slimshizn

New Member
Oct 26, 2018
3
0
1
My entire network is at 9000 MTU, so I sat the switch at 9000. That didn't work and the highest amount didn't work, so I'll just put my PC back to 1500.
 

fbcadmin

New Member
Nov 18, 2018
19
0
1
Hello tjk - thank you for the fast response

you wrote ' LB8M' we have LB6M . Which I assume is the same - not stackable.
 

tjk

Active Member
Mar 3, 2013
294
55
28
www.servercentral.com
I assume then linux bond is not possible using two non stacked switches. hopefully am I wrong?
Depends on the bond type. I've created a lag between the switches and have used bonding active/backup and RR without problems. I have not done LACP bond across switches, not sure that'll work since that signaling isn't carried across the switches like it would be in a true stacked config.
 
  • Like
Reactions: fbcadmin

fbcadmin

New Member
Nov 18, 2018
19
0
1
Depends on the bond type. I've created a lag between the switches and have used bonding active/backup and RR without problems. I have not done LACP bond across switches, not sure that'll work since that signaling isn't carried across the switches like it would be in a true stacked config.
with active/backup and non stacking switches - does a patch cable need to connect the two switches?

the two switches are for an isolated storage network.
 

fbcadmin

New Member
Nov 18, 2018
19
0
1
searched and found a connections is needed. We've done this before - it was at the end of one of those really long days when i asked.. so for anyone reading my question in the future:


https://www.kernel.org/doc/Documentation/networking/bonding.txt

Code:
11.2 High Availability in a Multiple Switch Topology
----------------------------------------------------

   With multiple switches, the configuration of bonding and the
network changes dramatically.  In multiple switch topologies, there is
a trade off between network availability and usable bandwidth.

   Below is a sample network, configured to maximize the
availability of the network:

               |                                     |
               |port3                           port3|
         +-----+----+                          +-----+----+
         |          |port2       ISL      port2|          |
         | switch A +--------------------------+ switch B |
         |          |                          |          |
         +-----+----+                          +-----++---+
               |port1                           port1|
               |             +-------+               |
               +-------------+ host1 +---------------+
                        eth0 +-------+ eth1

   In this configuration, there is a link between the two
switches (ISL, or inter switch link), and multiple ports connecting to
the outside world ("port3" on each switch).  There is no technical
reason that this could not be extended to a third switch.