Quanta LB6M (10GbE) -- Discussion

[keoki]

I can ping the VLAN "gateway" (192.168.5.1) from the edge router. I can ping a machine on the VLAN itself (192.168.5.2). On the machine I can ping up to the internet gateway (x.x.x.1). Traceroute dies off at x.x.x.1 as well. Something still getting lost between the edge router and the switch.

I guess I don't understand what you think is getting lost.

So a machine at 192.168.5.2 can ping the internet gateway... so your internal routing is working.

What are you doing a traceroute from and to? From 192.168.5.2 to google for example, if it dies at the edge router, then your nat isn't working yet... is that what you mean?

 

[keoki]

Keep in mind that when you set up nat, you have to nat everything, not just the network directly connected to the edge router. So your nat statements need to cover both the directly connected network, and all of your other private networks as well.

Nat statements vary a lot for different kinds of equipment, and not every cheap soho Internet router can nat for networks that are not directly connected.

So assuming your edge router has enough support to do it, you need to either set a netmask on the nat statement large enough to include everything (like 192.168.0.0/16) or you may have to have a separate NAT statement for each subnet.

 

[keoki]

There are lots of classes for networking, courses all over the net. TCP/IP for Dummies is a great book, but unless they revised it a lot since I used it to train my staff, it may not cover VLANs. I used that book to train my Internet support staff, starting back in 1994. but I have been building networks since around the early 1980's.

The TCP/IP for dummies book covers some protocols like FTP, talks about servers, but also takes a deep dive into things like netmasks... Critical knowledge.

 

[josh]

Keep in mind that when you set up nat, you have to nat everything, not just the network directly connected to the edge router. So your nat statements need to cover both the directly connected network, and all of your other private networks as well.

Nat statements vary a lot for different kinds of equipment, and not every cheap soho Internet router can nat for networks that are not directly connected.

So assuming your edge router has enough support to do it, you need to either set a netmask on the nat statement large enough to include everything (like 192.168.0.0/16) or you may have to have a separate NAT statement for each subnet.

I just have separate static routes on the edge router pointing each subnet to the public ip on the LB6M. My edge router is literally EdgeRouter by Ubiquiti

Here's what I'm trying to do. I have two NICs on this machine. One connects to the edge router direct (just for testing), the other through the LB6M on VLAN 192.168.5.0/24. #traceroute 8.8.8.8 -S publicip goes all the way to Google DNS. #traceroute 8.8.8.8 -S 192.168.5.2 has a last hop at the edge gateway. Something is broken but I can't wrap my mind around it. The packet makes it to the edge gateway but doesn't go further than that.

Edit: Nvm I realised it didn't save the NAT rule for 192.168.5.0/24 to the LB6M. It's working now. For anyone else trying to do the same thing in the future, you need both a static route and a NAT rule. Won't work without both.

 

[keoki]

I think it is your nat configuration is failing. Same gateway in both cases, I assume... so we know the Internet side works, directly connected nat works, it is the nat for the routed subnets that is failing. The fact your traceroutes get all the way to the edge router means that path is working. Remember things like ping and traceroute test both directions, so I think your internal routing is sound. what I think is failing is that the default NAT behavior is for the locally connected subnet, so you may have to add more nat statements or make the one that is there more inclusive with a fat netmask. I don't know anything about that router, but that is where the issue is. The problem could be that the right support for routed nat isn't there, or that you need to connect your switch to a port with a private network. From a software point of view, anything should be configurable. But most soho routers have a simplified interface, and simple documentation, so they don't expose the full power of the box unless you can get to the actual command line. I don't mean the configuration command line, I mean like a shell.

But I just looked at the docs and the gui seems to be oriented to things you aren't trying to do. You want the Masquerade function, the src and dest are more for allowing public access to servers, but I don't see what I'm looking for. The Masquerade function should be able to be given a private subnet to translate, but the description of source and destination don't seem to be for Masquerade. You may need to ask this question in the ubiquity forums. What you want is to use Masquerade to nat multiple subnets that are not directly connected.

 

[Chuntzu]

Keep up the good discussion with this vlan routing, I will be going through this shortly after I decommission my z9000 and set up my lb6m. One thing I am curious about is does this switch have dhcp helper address function? It took me two weeks to figure this out along with intervlan routing on a Cisco Nexus switch I was using so I could use 1 dhcp server to give out addresses to each vlan segment.i hope to not spend that much time this go around with the lb6m. Thank you.

Sent from my SM-N920T using Tapatalk

 

[keoki]

I saw DHCP helper functionality in the documentation posted in this thread... Pretty much all VLAN capable enterprise grade switches have to have that. Every enterprise class company I have ever worked in has enough VLANs that it would be stupid not to have this feature. One of the first VM's I plan to implement at home will be a DHCP/DNS server. Routers do DHCP ok, but nothing beats having a real server for that. but yes, at the office I have only ever needed a single DHCP server, while I see people in other offices avoiding VLANs because they think they need to build a DHCP server for every vlan, or have a dozen network ports in the DHCP server... My DHCP server has no layer 2 DHCP services, the subnet it is in is a static server subnet with lots of isolation. All of the DHCP comes in over layer 3 from my DHCP helper statements in every VLAN that requires DHCP. The main exceptions to using the IP helper is when the testbed provides DHCP as part of the protocol test function, and for server vlans where I don't want resource race conditions causing issues after returning from a power issue. I've never been given a budget for UPS power in the lab... so server subnets have static IP allocations so servers can come up before other parts of the network. But VLANs make it easy to have different configuration policies, and to change policies as needed for a single class of systems.

 

[josh]

I think it is your nat configuration is failing. Same gateway in both cases, I assume... so we know the Internet side works, directly connected nat works, it is the nat for the routed subnets that is failing. The fact your traceroutes get all the way to the edge router means that path is working. Remember things like ping and traceroute test both directions, so I think your internal routing is sound. what I think is failing is that the default NAT behavior is for the locally connected subnet, so you may have to add more nat statements or make the one that is there more inclusive with a fat netmask. I don't know anything about that router, but that is where the issue is. The problem could be that the right support for routed nat isn't there, or that you need to connect your switch to a port with a private network. From a software point of view, anything should be configurable. But most soho routers have a simplified interface, and simple documentation, so they don't expose the full power of the box unless you can get to the actual command line. I don't mean the configuration command line, I mean like a shell.

But I just looked at the docs and the gui seems to be oriented to things you aren't trying to do. You want the Masquerade function, the src and dest are more for allowing public access to servers, but I don't see what I'm looking for. The Masquerade function should be able to be given a private subnet to translate, but the description of source and destination don't seem to be for Masquerade. You may need to ask this question in the ubiquity forums. What you want is to use Masquerade to nat multiple subnets that are not directly connected.

I managed to get it to work by fixing my NAT rules after reading your post. I'm trying to figure out another problem. It seems that EQL SANs don't support joining a VLAN. I came up with a theoretical workaround. Is it possible to configure the switch to dynamically add/remove tags from traffic passing through a particular port?

Let's say I have 192.168.10.0/24 on VLAN 3 designated for traffic between hosts and SAN (192.168.10.3). If I hook the SAN to port 0/1, can I route VLAN3 tagged traffic with destination 192.168.10.3 to port 0/1 and add the VLAN3 tag to all traffic coming out of port 0/1? Would that allow machines on VLAN3 to communicate with the SAN that isn't on VLAN3? How would I go about doing this? Forgive me if it sounds stupid

 

[aero]

Keep up the good discussion with this vlan routing, I will be going through this shortly after I decommission my z9000 and set up my lb6m. One thing I am curious about is does this switch have dhcp helper address function? It took me two weeks to figure this out along with intervlan routing on a Cisco Nexus switch I was using so I could use 1 dhcp server to give out addresses to each vlan segment.i hope to not spend that much time this go around with the lb6m. Thank you.

Sent from my SM-N920T using Tapatalk

Why on earth would you replace a z9000 with lb6m? Please message me if you get rid of your z9000.

 

[keoki]

You cannot likely "dynamically" remove tags. However with Cisco equipment I can put one untagged vlan on a trunk port. This is a trick I have used to solve different problems, such as having a management network connection to a device on the same port as a trunk on a device that forces the use of $2500 SFP's just for a 1g connection... so they typically have 1 1g port for management, but if that port is fully configurable, I use it for both.

Another problem I solved this was was when we were using a carrier's managed ethernet service to connect two buildings. We saw weird problems that only affected specific packets. We think that the vendor was basically doing VLAN stacking, and our trunk was inside of a VLAN that the carrier used to seperate customers on their ethernet. Stacked vlans allow you to support 16 million vlans while observing the 4096 vlan limit. Carriers use this a lot, but I digress. One of our authentication protocols blasted an encrypted payload that could not get a critical packet through a stacked vlan. The solution was to map the one vlan carrying that payload to the one untagged vlan on the port.

The untagged vlan on a trunk port I have heard called the "native" vlan, but I don't know if that is an official term, or just local jargon. I think this is also known as the pvid.

Many (most?) devices can't do vlan tagging, so the usual thing to do is simply put them on a port in access mode. Some devices require a special firmware load, or at least a poorly documented configuration. Most enterprise systems support vlans on some level, but may have limitations. Under linux, for example, you use sub-interfaces which usually have a limit of 255 subinterfaces. A limit usually bypassed for linux based systems controlling enterprise network appliances like switches. I think the bypass comes from not using subinterfaces, and just tracking the vlans directly in the network port driver. But now I'm just screaming like godzilla...

I just racked my switches up, I had to pull out two old servers from my home rack, I had not used them in a long time, and it was time for them to GO. Two very capable boat anchors. But I have not started the configuration, so I can't "see" your problem yet.

Ideas to solve a problem are rarely stupid.

To directly answer your question, many devices allow you to do a full trunk that includes ALL of your vlans, and a separate pvid for your vlan bypass purpose.

But if your SAN does not support vlans you should not be connecting it to a trunk port. So I'm not sure why you have this problem, unless all ports must be trunk ports. Most trunk ports support vlan filters, and probably pvid's too. So in that case, I would limit the trunk to one vlan that was either not connected to anything, or even shut down. then I would use the pvid to connect the port to the vlan you have for your SAN.

I'll be figuring out how to do all of this soon, but not today. Right now all of my windows machines in the office don't have 10g yet, and the cables to reach my wife's office has not yet arrived. She has a desktop with big storage, and a video editing system with fast SSD, so two valid machines for 10g connections to the home SAN. The 20 meter cables needed for those runs has not yet arrived, won't be here for a couple weeks, I'm guessing Amazon is shipping them out of China, based on the price I got.

I think the most I will get done this weekend is to get my NAS/SAN machines moved off the toy switches and onto the big iron, and to get one of them on 10g. The other does not have the slots for 10g, so I need to replace the motherboard with something heaftier... it's a shame, it was a sweet board for a low power 36 drive NAS but with the mini itx form factor, only the hard drive controller has a slot. I guess they aren't officially SAN's till I have iscsi mounts feeding vmware... All in good time. Basically I have dozens of systems, most are off... I'm re-spinning the home network to be a smaller stack of vmware systems, and my stand-alone FreeNAS systems.

But that means that today and tonight I will be working on the configuration, hoping I don't hit any snags. There are ALWAYS snags...

 

[keoki]

Think I will go with this one on Amazon instead:

Networking All-in-One For Dummies: Doug Lowe: 9781119154723: Amazon.com: Books

I was reading a few of the reviews and it seems like TCP/IP for dummies seems alittle dated as you mentioned Keoki. This one seems abit more of a recent all-rounder basics book for networking.

I already have alittle knowledge, and I know this book won't make me an expert in networking, but it should get me familiar with some more of the networking concepts I'm still not familiar with yet.

Sounds like a great plan... a lot has changed. And TCP/IP is very dated in the context of IPv6...

 

[keoki]

Let's say I have 192.168.10.0/24 on VLAN 3 designated for traffic between hosts and SAN (192.168.10.3). If I hook the SAN to port 0/1, can I route VLAN3 tagged traffic with destination 192.168.10.3 to port 0/1 and add the VLAN3 tag to all traffic coming out of port 0/1? Would that allow machines on VLAN3 to communicate with the SAN that isn't on VLAN3? How would I go about doing this? Forgive me if it sounds stupid

Yes, basically most systems configured for IP are not configured to handle VLAN tags. So for those systems we set the ports to be in access mode, rather than trunking mode. In this case you want VLAN 3, so you make the access ports members of vlan3 that are connected to the SAN and the machines that use the SAN. In the case of SAN ports, assuming your SAN has a management port on a seperate vlan, and no non-SAN traffic exists on vlan 3, then vlan 3 does not need any IP address in the switches and routers. You don't /have/ to isolate your SAN VLAN, but it is a common practice to keep that vlan free of broadcasts, malware probes, and other traffic not associates with the SAN. My SAN has dual 10G nic ports, so I can set one 10g port for iSCSI mounts only, and the other can be the NFS , CIFS and other services. Of course if you only have the 1 10G nic port, you can do all of these services over the same port. But it is up to you how you do this... it depends on where you have the most traffic and need the most performance, or where you are the most sensitive to latency. iSCSI is a low latency service (usually), while NFS and CIFS have a lot of protocol latency and chatter, so while you can't make NFS much faster, but you can try and keep it from slowing down your iSCSI. But the quality of my advice here really depends on what you are trying to do.

 

[revyve]

So been playing around with this switch but wondering if anyone had any luck configuring anything multicast on this yet snooping or otherwise... thanks,

 

[PigLover]

True multicast or IGMP snooping support for hosts doing multicast?

IGMP snooping works perfectly.

Sent from my SM-G925V using Tapatalk

 

[axemann]

Not to derail the multicast questions, but is there any way to set the time zone on the '6M? I have SNTP configured but didn't find a way to config the time zone...


Sent from my iPad using Tapatalk

 

[revyve]

True multicast or IGMP snooping support for hosts doing multicast?

IGMP snooping works perfectly.

Sent from my SM-G925V using Tapatalk

Yeah just saw the snooping working which makes happy, but hoping for true multicast support? can't find the commands to see

 

[revyve]

Yeah just saw the snooping working which makes happy, but hoping for true multicast support? can't find the commands to see

actually just found stuff under configure# set igmp
Happy day!

 

[Terry Kennedy]

Not to derail the multicast questions, but is there any way to set the time zone on the '6M? I have SNTP configured but didn't find a way to config the time zone...

On a different brand of switch running the same software:

sntp unicast client enable
sntp server aaa.bbb.ccc.ddd
clock summer-time recurring USA zone "EDT"
clock timezone -5 minutes 0 zone "EST"

 

[axemann]

On a different brand of switch running the same software:

sntp unicast client enable
sntp server aaa.bbb.ccc.ddd
clock summer-time recurring USA zone "EDT"
clock timezone -5 minutes 0 zone "EST"

Thanks Terry, I'll give that a shot here shortly...


Sent from my iPad using Tapatalk

 

[axemann]

Thanks @Terry Kennedy for the kick in the right direction.

I was able to set the default timezone to CDT using your last command, and once I fixed an issue with the SNTP server I was trying to contact (I switched servers), my time and timezone are showing correctly using 'show clock detail' (although summer time is still showing disabled).

According to the switch, it wants the command to set DST in this format:

clock summer-time recurring {<week> <day> <month> <hh:mm> <week> <day> <month> <hh:mm>} [offset <offset>] [zone <acronym>]

I was able to get it set by using the following:

clock summer-time recurring 2 Sun Mar 02:00 2 Sun Nov 02:00 zone "CDT"

Notice that the above doesn't include the 'offset <offset>' value, as the switch would not accept anything I fed it for that setting.

I had some initial issues figuring out exactly what it wanted with respect to <day> and <month> (I thought it wanted the actual numeric date and month), but with some fiddling I finally got it to work.

Hope this helps any folks out there with time/timezone issues (or OCD like me).