Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

never seen that before, looks like weird firmware. does it fully boot into the OS? if it does paste the output of "show version" then try and follow the update guide to get the latest on it

 

[LodeRunner]

It did boot to

SW: Version 08.0.30dT211

I had wrong flow control in serial so it was ignoring my keyboard. Every other switch is no flow control, 7450-48p wants it.

Also did not get those messages after issuing the factory reset in uboot. So must have been config related. Switch was previously part of a stack.

It's updating fine it appears, so yay. Got a 4x10g mod coming in, it had both PSUs and 2 40G mods in the back.

 

[fohdeesha]

wow that is oooooold. maybe the first 7 series ICX I've seen come with the 8030 train installed, let alone of the first releases ever on that train

 

[LodeRunner]

Here's the original show flash output:

[MEMBER]local-4@ICX7450-48P Switch#sh flash
Stack unit 4:
  Compressed Pri Code size = 22642415, Version:08.0.40aT211 (SPS08040a.bin)
  Compressed Sec Code size = 28680388, Version:08.0.30dT211 (SPS08030d.bin)
  Compressed Boot-Monitor Image size = 786944, Version:10.1.06T215
  Code Flash Free Space = 1777840128

It was booting from secondary when I got it.

 

[Roelf Zomerman]

Got my eye on a 6610-48p to replace my 6450-48p.
Just to confirm, I can use 2x 40 in the back for my server connections and the other 2 qsfp ports are just 4x 10g breakouts.? Or can I also connect those to a Mellanox X3 card on my 3rd server?

 

[creidhne]

I'm still somewhat confused with this. In theory I do understand how VLAN works and what are tagged and untagged ports, but the entire L3 eludes me. Can someone ELI5 those to me? For example subnet VLANs. So let's say there's vlan 10 by port, and it has 1/1/1 to 1/1/12 assigned to it untagged. There was `ip-subnet 192.168.0.0/24` added to it, and according to manual this creates another vlan inside that by-port one. Do I understand right that it has no ID? What happens to traffic that comes from 192.168.1.1? And now I cannot stick router interface into it because there's ip-subnet? ...and why does even require having a static port, since router-interface is virtual anyway?

Another example, let's say I have 3 VLANS (id 10 and 20 and 30), and tagged 1/1/1 belongs to 10 and 20. There's Access point plugged to it, and management/configuration is untagged, while traffic from the WiFi is tagged. I want to forward untagged traffic to vlan 30. Do I do just `dual mode 30` on this port and it'll work? Or do I have to put that port in vlan 30 as untagged? Or as tagged even though I don't want tagged traffic in vlan 30 from this port and don't expect any?

 

[EngChiSTH]

I've dealt with exactly your issues a few tries now. I finally bit the bullet, set up a VM running BIND/named + ISC dhcpd and it's working so much more streamlined than pfSense - it just takes a few more keystrokes to configure - and watch out for typos.

I have never used the Unifi stuff outside of the Unifi controller software and a single WAP. I'm not sure what the L3 performance of the UDM Pro is but I suspect it is less than that of the ICX6450. I am going to blanket recommend using the ICX6450 as your L3 VLAN router and send any other traffic to the UDM Pro for internet access. You know your network needs more than me, though.

I finally am getting my network cleaned up after doing some research and trial+error. LLDP, FDP, OSPF, Loopback interfaces for management, a proper (non-pfSense) DNS+DHCP server (soon to be 2 for redundancy), a /30 trunk to my OPNSense WAN firewall for NAT duty and native IPv6 filtering with /64 subnetting, and a Lancache server running custom domains for the many installs of Void and Ubuntu Linux I have o the network - oh and Steam, XBox, and Windows updates - it's very fun downloading OS packages or games up to almost 10Gbps from a local cache.

Would you be willing to describe/write a guide on how you did it?

I am running Windows 2016 domain right now which is also my DNS and DHCP and would like to offload those functions (so then Windows updates install everything is not down).

I am considering building a cluster for virtualization (Proxmox or Hyper-V) and moving more towards isolating specific functions (unifi controller that I run on VM, Pi-Hole, DNS, DHCP).

Thank you

 

[fohdeesha]

Got my eye on a 6610-48p to replace my 6450-48p.
Just to confirm, I can use 2x 40 in the back for my server connections and the other 2 qsfp ports are just 4x 10g breakouts.? Or can I also connect those to a Mellanox X3 card on my 3rd server?

correct, two are breakout only and cannot be connected to a 40gbE NIC at 40gbE

 

[nerdalertdk]

Hi All

So bought this switch ICX7150-48ZP, got it for 500$ including shipping to Denmark so you guys in US might be able to get it cheaper.

Ruckus ICX7150-48ZP-E2X10G Z Series ICX 7150-48ZP Switch *Read Description* | eBay

For sale is a Ruckus ICX7150-48ZP-E2X10G. Local pick up is available, and always free. The mode lights are not working but status light still works. Mode/reset buttons do not function either. 1x RJ45 Console Cable.

www.ebay.com

After updating firmware to 8.0.80f the front mode button was working.
it came with 8.0.6 something

/CC @fohdeesha as an answer for the PM I sent you


Also should i run 8.0.9xx or 08.0.8xxx

 

[Fallen Kell]

I'm still somewhat confused with this. In theory I do understand how VLAN works and what are tagged and untagged ports, but the entire L3 eludes me. Can someone ELI5 those to me? For example subnet VLANs. So let's say there's vlan 10 by port, and it has 1/1/1 to 1/1/12 assigned to it untagged. There was `ip-subnet 192.168.0.0/24` added to it, and according to manual this creates another vlan inside that by-port one. Do I understand right that it has no ID? What happens to traffic that comes from 192.168.1.1? And now I cannot stick router interface into it because there's ip-subnet? ...and why does even require having a static port, since router-interface is virtual anyway?

Another example, let's say I have 3 VLANS (id 10 and 20 and 30), and tagged 1/1/1 belongs to 10 and 20. There's Access point plugged to it, and management/configuration is untagged, while traffic from the WiFi is tagged. I want to forward untagged traffic to vlan 30. Do I do just `dual mode 30` on this port and it'll work? Or do I have to put that port in vlan 30 as untagged? Or as tagged even though I don't want tagged traffic in vlan 30 from this port and don't expect any?

Yes, you seem to be a little confused. In your first example where you have ports 1/1/1, and 1/1/12 assigned to a untagged VLAN 10, with the switch running in layer 3, the IP assignment isn't really creating a new VLAN, but defining some of the address space being used on VLAN 10 (so that inter VLAN routing can be configured). Without defining the address space (and without definine unique address space for each VLAN) the switch will not be able to perform inter-vlan routing as it won't know that you want to talk to 192.168.1.10 on VLAN 1 or VLAN 10 if they share the same address space.

Also, in terms of tagged vs untagged, it simply means that the packet itself already contains a VLAN tag for that packet. If the switch receives a packet that is untagged, it will treat the packet as though it is from the VLAN defined for the untagged vlan for that port which the packet arrived to the switch. It may tag the packet itself at that time before sending it elsewhere).

In your later example of VLAN 30 for untagged management on certain ports, yes, you want to place the ports into dual mode if you want to treat untagged packets as a specific VLAN, but also have tagged packets from other VLANs available on those port, and you need to set the untagged VLAN to 30 in your case.

 

[ArmedAviator]

@EngChiSTH , sure thing. It won't be a full guide at present as time doesn't permit but I'll get what I can on this post.

Here's the important bits from my ICX6610 which is on L3 router duty.

Configuring the L3 switch

Global config section:

Note 1: The default route for IPv4 (0.0.0.0/0) and IPv6 :/0) point to the LAN interface IP of OPSense (or pfSense, or EdgeRouter or whatever).

Note 2: The dns-domain-list entries are for each of my local VLANs, resolved by my local DNS.

hostname ks-icx-01
ip dhcp-client disable
ip dns domain-list mgmt.rgn
ip dns domain-list app.rgn
ip dns domain-list iot.rgn
ip dns domain-list cli.rgn
ip dns domain-list nas.rgn
ip dns domain-list pve.rgn
ip dns domain-list voip.rgn
ip dns server-address 10.1.26.5 10.1.26.3
ip route 0.0.0.0/0 10.1.99.2
!
ipv6 dns server-address 2605:aaaa:bbbb:7a99:1c3f:83ff:feef:411d
ipv6 unicast-routing
ipv6 route ::/0 2605:aaaa:bbbb:7a99:1c3f:83ff:feef:411d

Here's my router interfaces for each of my VLANs.

Note 1: Each VLAN interface (ve) has the same ip helper-address which is my DHCP/DNS server, discussed later.

Note 2: If you're new to IPv6 (as I am), each ve that I want IPv6 on is getting it's own globally routable /64. That's basically the second two digits in the 4th section (i.e. 7a01, 7a02, 7a03, etc. in my case). I chose to set the /24 in the IPv4 subnets and the /64 in the IPv6 subnets to the VLAN ID just for simplicity's sake. The next few sections (not sure what they're called....they're not octets anymore) I kept as zeroes so they can be shortened to :: and the trailing 1 is equivalent to the host address in IPv4 such as 10.1.1.1.

interface ve 2
 port-name VLAN-VOIP
acl-logging
ip access-group VOIP in
ip address 10.1.2.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 3
port-name VLAN-IOT
acl-logging
ip access-group IOT in
ip address 10.1.3.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 4
port-name VLAN-SAN
ip address 10.1.4.1 255.255.255.0
ip mtu 9000
ip ospf area 0
ipv6 mtu 9000
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 5
port-name VLAN-MGMT
ip address 10.1.1.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a01::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 10
port-name VLAN-CLI
ip address 10.1.10.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a10::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 26
port-name VLAN-APP
ip address 10.1.26.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a26::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 99
port-name VLAN-OPNS
ip address 10.1.99.1 255.255.255.252
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a99::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface loopback 1
port-name Management
ip address 10.0.0.1 255.255.255.255
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a00::1/128
ipv6 ospf area 0
!

So that's the switch section doing majority of routing. To set up your WAN gateway router (pfSense, OPNSense, Edgerouter, etc.) you should set it up with a single LAN interface, no VLANs (that's a whole different setup and what I wanted to get away from).

Configuring the WAN router

IPv6 step only:

If you are setting up for IPv6 with multiple subnets as above you will want to request a /56 subnet from your ISP. I believe most residential IPSs give out a /56 when requested, but your router must specifically request it in most if not all cases. In OPNSense and pfSense, there's an option under the WAN configuration under DHCP6. Select "Prefix delegation size" to /56 and check the box that says "Send IPv6 prefix hint." Release and renew your DHCP/DHCP6 and see what you get - if it worked, your WAN interface will have an IPv6 address of it's own, in my case it gets a /128 address and a completely seperate /56 for my LAN devices.

Here are the IPv6 addresses I get from DHCP6 on my WAN interface:

IPv6 address  2605:aaaa:ffff:10:5047:ae63:29ef:d367 / 128
Delegated prefix  2605:a000:d401:7a00::/56

The delegated prefix is what we used in the above switch configuration section.


Now, whether or not you use IPv6, you need to set up your LAN interface with the correct addresses.

In my case, I chose to use a /30 IPv4 subnet which allows for 2 hosts. I know a /31 works on most devices, but it's a bit of a "well, I suppose we'll allow it" approach and not guaranteed to work on all devices. In my case I used 10.1.99.0/30 which means I can set the switch to 10.1.99.1 and the LAN interface to 10.1.99.2. For IPv6, I just gave it a /64 for the hell of it.

At this point, your WAN router does not know about the rest of your LAN.

In pfSense and OPNSense, you need to add a gateway before you can add a route. I added the gateways 10.1.99.1 and 2605:aaaa:bbbb:7a99::1/64 which are the addresses of the L3 switch virtual interface trunked to OPNsense.


Add a route or routes appropriate to your network. In my case, everything IPv4 is contained within a 10.1.0.0/16 subnet, so I added the route 10.1.0.0/16 via 10.1.99.1 (the gateway we added above). For IPv6, we route the entire /56 to the switch so it'll be 2605:a000:d401:7a00::/56 via 2605:a000:d401:7a99::1 (the gateway we added above).

You should now have working native IPv6, but NAT will not work yet for IPv4.

In pfSense or OPNSense, go to Firewall > NAT > Outbound. Change the Mode to Manual. Change or copy each of the rules from the original LAN network (10.1.99.0/30) to 10.1.0.0/16. This allows the firewall to NAT outbound traffic from all of your LAN.

Add appropritate firewall rules on your LAN interface to allow IPv4 and IPv6 outbound traffic.

You now should have working IPv4 internet.


DHCP and DNS configuration

Here's just a dump of my /etc/dhcpd.conf file (less the secret):

Note 1: I did not setup reverse DNS yet.

Note 2: Some hosts do not have IP addresses configured in DHCP because they are configured manually on the device.

Note 3: This configuration will update the BIND named DNS server in real-time.

ddns-updates on;
ddns-update-style interim;
update-static-leases on;
authoritative;
key "rndc-key" {
algorithm hmac-sha256;
secret "nope";
};
allow unknown-clients;
use-host-decl-names on;
log-facility local7;


zone mgmt.rgn. {
primary localhost;
key rndc-key;
}

zone cli.rgn. {
primary localhost;
key rndc-key;
}

zone app.rgn. {
primary localhost;
key rndc-key;
}

zone voip.rgn. {
primary localhost;
key rndc-key;
}

zone iot.rgn. {
primary localhost;
key rndc-key;
}



subnet 10.1.1.0 netmask 255.255.255.0 {
range 10.1.1.150 10.1.1.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "mgmt.rgn";
ddns-domainname "mgmt.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.1.1;
option broadcast-address 10.1.1.255;
default-lease-time 600;
max-lease-time 7200;
}

subnet 10.1.2.0 netmask 255.255.255.0 {
range 10.1.2.150 10.1.2.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "voip.rgn";
ddns-domainname "voip.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.2.1;
option broadcast-address 10.1.2.255;
default-lease-time 600;
max-lease-time 7200;
}

subnet 10.1.3.0 netmask 255.255.255.0 {
range 10.1.3.150 10.1.3.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "iot.rgn";
ddns-domainname "iot.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.3.1;
option broadcast-address 10.1.3.255;
default-lease-time 600;
max-lease-time 7200;
}

subnet 10.1.10.0 netmask 255.255.255.0 {
range 10.1.10.150 10.1.10.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "cli.rgn";
ddns-domainname "cli.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.10.1;
option broadcast-address 10.1.10.255;
default-lease-time 600;
max-lease-time 7200;
}

subnet 10.1.26.0 netmask 255.255.255.0 {
range 10.1.26.150 10.1.26.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "app.rgn";
ddns-domainname "app.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.26.1;
option broadcast-address 10.1.26.255;
default-lease-time 600;
max-lease-time 7200;
}


##
## VLAN 2
###
host pbx01.voip.rgn {
hardware ethernet 36:22:12:27:7e:b0;
fixed-address 10.1.2.5;
ddns-hostname "pbx01";
}


##
### VLAN 3
##
host ks-prnt-01.iot.rgn {
hardware ethernet 74:40:bb:aa:6d:a5;
fixed-address 10.1.3.15;
ddns-hostname "ks-prnt-01";
}

host ks-nvr-01.iot.rgn {
hardware ethernet ec:71:db:d3:12:bf;
fixed-address 10.1.3.20;
ddns-hostname "ks-nvr-01";
}

host ks-cam-01.iot.rgn {
hardware ethernet ec:71:db:ac:87:ec;
fixed-address 10.1.3.21;
ddns-hostname "ks-cam-01";
}

host ks-cam-03.iot.rgn {
hardware ethernet b0:41:1d:25:fd:79;
fixed-address 10.1.3.23;
ddns-hostname "ks-cam-03";
}

host ecobee.iot.rgn {
hardware ethernet 44:61:32:64:f0:05;
fixed-address 10.1.3.50;
ddns-hostname "ecobee";
}


##
## VLAN 5
##
host ks-icx-01 {
hardware ethernet 74:8e:f8:e7:b4:b0;
fixed-address ks-icx-01.mgmt.rgn;
}

host ls-icx-02 {
hardware ethernet 74:8e:f8:82:e8:60;
fixed-address ks-icx-02.mgmt.rgn;
}

host arbiter {
hardware ethernet a2:7d:20:dc:94:01;
fixed-address arbiter.mgmt.rgn;
}

host neutron {
hardware ethernet 00:02:c9:3b:a0:40;
fixed-address neutron.mgmt.rgn;
}

host proton {
hardware ethernet 00:02:c9:3b:61:30;
fixed-address proton.mgmt.rgn;
}

host ks-bmc-sm1u-01 {
hardware ethernet 02:25:90:24:7e:86;
fixed-address ks-bmc-sm1u-01.mgmt.rgn;
}

host ks-bmc-sm2u-01 {
hardware ethernet 02:30:48:ca:e1:b0;
fixed-address ks-bmc-sm2u-01.mgmt.rgn;
}

host ks-bmc-sm2u-02 {
hardware ethernet 02:25:90:18:9a:c0;
fixed-address ks-bmc-sm2u-02.mgmt.rgn;
}

host ks-bmc-r710-01 {
hardware ethernet 84:2b:2b:71:7d:17;
fixed-address ks-bmc-r710-01.mgmt.rgn;
}

host ks-bmc-r710-02 {
hardware ethernet 78:2b:cb:23:85:84;
fixed-address ks-bmc-r710-02.mgmt.rgn;
}

host unifictl {
hardware ethernet 8A:CA:FD:F4:24:25;
fixed-address unifictl.mgmt.rgn;
}

host ks-uap-01 {
hardware ethernet 78:8a:20:86:5d:93;
fixed-address ks-uap-01.mgmt.rgn;
}

host ks-pve-01 {
hardware ethernet f2:6f:9b:2c:8b:73;
fixed-address ks-pve-01.mgmt.rgn;
}

host ks-pve-02 {
hardware ethernet 62:7e:20:4d:1f:82;
fixed-address ks-pve-02.mgmt.rgn;
}

host ks-pve-03 {
hardware ethernet ba:48:7e:02:81:6c;
fixed-address ks-pve-03.mgmt.rgn;
}


##
## VLAN 10
##
host area51.cli.rgn {
hardware ethernet 00:02:c9:1b:fe:10;
fixed-address 10.1.10.10;
ddns-hostname "area51";
}

host zenith.cli.rgn {
hardware ethernet fc:f8:ae:7b:c1:13;
fixed-address 10.1.10.11;
ddns-hostname "zenith";
}

host zenith-eth.cli.rgn {
hardware ethernet bc:ee:7b:17:d8:ec;
fixed-address 10.1.10.12;
ddns-hostname "zenith-eth";
}

host htpc01-eth.cli.rgn {
hardware ethernet 84:39:be:68:42:81;
fixed-address 10.1.10.13;
ddns-hostname "htpc01-eth";
}

#wifi
host htpc01.cli.rgn {
hardware ethernet 10:d0:7a:87:68:e7;
fixed-address 10.1.10.14;
ddns-hostname "htpc01";
}

host htpc02.cli.rgn {
hardware ethernet f8:b1:56:df:17:2e;
fixed-address 10.1.10.15;
ddns-hostname "htpc02";
}

host elana.cli.rgn {
hardware ethernet a8:a1:59:2e:53:a9;
fixed-address 10.1.10.16;
ddns-hostname "elana";
}

host xboxones.cli.rgn {
hardware ethernet b8:31:b5:ef:c4:48;
fixed-address 10.1.10.31;
ddns-hostname "xboxones";
}

host xboxones-wifi.cli.rgn {
hardware ethernet b8:31:b5:ef:c4:4a;
fixed-address 10.1.10.32;
ddns-hostname "xboxones-wifi";
}

host andromeda.cli.rgn {
hardware ethernet c2:fb:42:27:d8:66;
fixed-address 10.1.10.61;
ddns-hostname "andromeda";
}

host galaxys5.cli.rgn {
hardware ethernet fc:c2:de:83:9b:70;
fixed-address 10.1.10.62;
ddns-hostname "galaxys5";
}

host cassiesgalaxy.cli.rgn {
hardware ethernet 24:18:1d:60:89:41;
fixed-address 10.1.10.63;
ddns-hostname "cassiegalaxy";
}


##
## VLAN 26
##
host dhcp01 {
hardware ethernet f2:08:5d:b9:8d:a6;
fixed-address dhcp01.app.rgn;
}

host cache01 {
hardware ethernet 96:47:84:a0:2b:48;
fixed-address cache01.app.rgn;
}

host slb01 {
hardware ethernet 52:44:eb:1a:d1:73;
fixed-address slb01.app.rgn;
}

host slb02 {
hardware ethernet 4a:ff:4a:54:8a:c6;
fixed-address slb02.app.rgn;
}

host slb03 {
hardware ethernet fe:fe:2b:95:5c:27;
fixed-address slb03.app.rgn;
}

host web01 {
hardware ethernet a2:90:4f:c1:2c:e8;
fixed-address web01.app.rgn;
}

host web02 {
hardware ethernet 22:11:ad:8a:7d:73;
fixed-address web02.app.rgn;
}

host web03 {
hardware ethernet 92:01:b6:bb:8a:41;
fixed-address web03.app.rgn;
}

host redis01 {
hardware ethernet ca:80:2d:7f:c8:b4;
fixed-address redi01.app.rgn;
}

host mrdb01 {
hardware ethernet fa:18:03:ab:c2:3a;
fixed-address mrdb01.app.rgn;
}

host sync01 {
hardware ethernet 1a:3d:7c:c7:f3:56;
fixed-address sync01.app.rgn;
}

host mon01 {
hardware ethernet 62:55:04:ac:c1:93;
fixed-address mon01.app.rgn;
}

host torr01 {
hardware ethernet 82:a2:71:74:d4:bf;
fixed-address torr01.app.rgn;
}

And here's a dump of my /etc/named/named.conf:

options {
directory "/var/named";
pid-file "/var/run/named/named.pid";
auth-nxdomain yes;
datasize default;
statistics-file "/var/cache/bind/stats";
zone-statistics yes;
listen-on-v6 { any; };

allow-recursion {
10.1.0.0/16;
127.0.0.1;
};
forwarders {
1.1.1.1;
1.0.0.1;
9.9.9.9;
};
allow-query {
10.1.0.0/16;
127.0.0.1;
};
allow-query-cache {
10.1.0.0/16;
127.0.0.1;
};
allow-transfer {
10.1.0.0/16;
127.0.0.1;
};
allow-update {
10.1.0.0/16;
127.0.0.1;
};
version none;
hostname none;
server-id none;
};

key "rndc-key" {
algorithm hmac-sha256;
secret "nope";
};

zone "mgmt.rgn" {
type master;
file "mgmt.rgn.zone";
allow-update { key rndc-key; };
};

zone "iot.rgn" {
type master;
file "iot.rgn.zone";
allow-update { key rndc-key; };
};

zone "cli.rgn" {
type master;
file "cli.rgn.zone";
allow-update { key rndc-key; };
};

zone "voip.rgn" {
type master;
file "voip.rgn.zone";
allow-update { key rndc-key; };
};

zone "app.rgn" {
type master;
file "app.rgn.zone";
allow-update { key rndc-key; };
};

zone "localhost" IN {
type master;
file "localhost.zone";
allow-transfer { any; };
};

zone "0.0.127.in-addr.arpa" IN {
type master;
file "127.0.0.zone";
allow-transfer { any; };
};

zone "." IN {
type hint;
file "root.hint";
};

logging {
channel xfer-log {
file "/var/log/named.log";
print-category yes;
print-severity yes;
print-time yes;
severity info;
};
category xfer-in { xfer-log; };
category xfer-out { xfer-log; };
category notify { xfer-log; };
};

As far as the zone files go, you can do some web searches how to make those. I just copied and pasted from some website, changed the domains, and listed all of the A records I needed to. Note: only put in A records for devices that are configured manually. DHCP assigned addresses will be added/amended automatically.


Cache Server

I run the LanCache.Net monolithic docker setup with the exception of I set up a few extra distro repository domains to the list (Debian, Ubunutu, Void Linux) so my updates are way quicker. I gave it 600GB and 8GB of RAM as a VM on Proxmox. The majority of the space is used by Steam and Xbox games.

To utilize the cache server, it must be the first DNS nameserver to be queried (see dhcpd.conf).


OPSense/pfSense as VM

I'm using Proxmox VE in a 3-node cluster. It is NOT configured HA due to storage differences between them (2x R710 with 2TB of H700 RAID5 SAS and a 1U whitebox with 120GB MD-RAID1 SSD). The hypervisors are using OpenVswitch with an LACP bond on 2x 10G ports configured with mtu 9000 and another LACP bond on 2x 1G ports configured with mtu 1500 (primarily for management of Proxmox and VoIP traffic).

Setting these up as a VM is pretty straightforward. Install as normal. Connect your home modem to whatever port you desire on your switch and configure as an untagged port in a VLAN. The L3 switch should not have a virtual interface on this VLAN - it's purely L2. Add one vNIC on the virtual machine to the VLAN your modem is on and the other vNIC to the VLAN that is configured with the virtual interface IPs configured in the switch and router configs above.

To increase traffic throughput on the vNICs, set the vNIC queues to the number of vCPUs you gave the VM. You must elect "Advanced" in the Network Device Configuration popup.

To get AES-NI for VPN servers/clients, be sure to select an appropriate CPU architecture to emulate. In my case, I went with Westmere as all of my servers are Westmere or newer architecture so this will advertise and pass-thru the AES-NI CPU instructions.


LLDP / CDP / FDP

On the switch, in gloval config, just do:

lldp run
cdp run
fdp run

You can view detected devices using these protocols with

show lldp neighbors
show cdp neighbors
show fdp neighbors

LLDP is useful to configure the voice-vlan automatically on LLDP-enabled VoIP phones (see interfaces section of Switch Configuration above).

LLDP can be installed on most Linux distros easily. Install it, run it, you can view connected devices (probably just the switch), but it makes figuring out what's connected to what port very easily on the switch using the commands above.

SSH@ks-icx-01>show lldp neighbors
Lcl Port Chassis ID      Port ID         Port Description            System Name    
1/1/23 842b.2b71.7d0f 842b.2b71.7d0f eno1 ks-pve-01.vmh~
1/1/25 842b.2b71.7d0f 842b.2b71.7d11 eno2 ks-pve-01.vmh~
1/1/35 10.1.2.155 0008.5d1b.472a port 0 Mitel IP Phon~
1/1/37 788a.2086.5d93 788a.2086.5d93 br0 ks-uap-01
1/1/45 10.1.2.156 0008.5d2a.5e60 port 0 Mitel IP Phon~
1/2/2 842b.2b71.7d0f 0002.c91a.faa1 enp6s0d1 ks-pve-01.vmh~
1/2/7 842b.2b71.7d0f 0002.c91a.faa0 enp6s0 ks-pve-01.vmh~
1/3/1 748e.f882.e860 748e.f882.e87c 10GigabitEthernet1/2/4 ks-icx-02
1/3/2 748e.f882.e860 748e.f882.e87b 10GigabitEthernet1/2/3 ks-icx-02
1/3/3 748e.f882.e860 748e.f882.e87a 10GigabitEthernet1/2/2 ks-icx-02
1/3/4 748e.f882.e860 748e.f882.e879 10GigabitEthernet1/2/1 ks-icx-02

 

[dswartz]

Always an adventure googling for info on how to do something. I want to set up a VLAN to restrict access to various IPMI ports. My workstation has two nics. One is on 10.0.0.0/24 network (default vlan 1). My workstation currently is plugged into a shitty netgear switch, with an uplink cable which travels above the acoustic tiles to my work area. I have a 7150 12 port I am about to replace the netgear with. Plan to use one of the ports to plug the existing workstation nic into (will be in default vlan 1). I want to create vlan 4 on both switches, and have the 2nd nic on my workstation plug into a port on the 7150 that will carry accept untagged traffic from the 2nd nic and tag it with vlan 4. On the 7250, I want the port that currently has the cable from my work area to accept input for untagged vlan 1 and tagged vlan 4. Various ipmi ports will then be changed from no vlan to untagged vlan 4. This should all work, no? The confusion was that I am running 08.0.92bT211, and the various google hits all referred to using the 'dual-mode' command, which doesn't seem to exist on my 7250 stacked switch. Wasn't until I putzed around a bit that I found out that dual-mode is now implicit. Always fun...

 

[creidhne]

Yes, you seem to be a little confused. In your first example where you have ports 1/1/1, and 1/1/12 assigned to a untagged VLAN 10, with the switch running in layer 3, the IP assignment isn't really creating a new VLAN, but defining some of the address space being used on VLAN 10 (so that inter VLAN routing can be configured). Without defining the address space (and without definine unique address space for each VLAN) the switch will not be able to perform inter-vlan routing as it won't know that you want to talk to 192.168.1.10 on VLAN 1 or VLAN 10 if they share the same address space.

Also, in terms of tagged vs untagged, it simply means that the packet itself already contains a VLAN tag for that packet. If the switch receives a packet that is untagged, it will treat the packet as though it is from the VLAN defined for the untagged vlan for that port which the packet arrived to the switch. It may tag the packet itself at that time before sending it elsewhere).

In your later example of VLAN 30 for untagged management on certain ports, yes, you want to place the ports into dual mode if you want to treat untagged packets as a specific VLAN, but also have tagged packets from other VLANs available on those port, and you need to set the untagged VLAN to 30 in your case.

Still a bit confused, but it's a bit better than before. Thanks! Now on to read some more...

 

[losx]

Have a question and wonder if I am screwing things up. Got it all to work as a layer 2 switch but now I am trying to use the icx6450 to do intervlan routing.

I setup 2 vlans 20 (10.0.20.0) Port 1 and 30(10.0.30.0) Port 24 on the switch. I can ping between the different devices on separate vlans. I then connected to my firewall. I setup a transit gateway vlan 2.

On vlan 2 router is 192.168.1.1 and switch 192.168.1.2.
On router I also created route of 10.0.0.0/8 to route to 192.168.1.2 as next hop
On switch i setup ip route 0.0.0.0/0 to 192.168.1.1

I could not get my switch to hit the internet until I added the port in my transit vlan as a tagged port on port 1 and port 24... does this seem correct or am I missing something?

Also to verify it is all working correctly I assume you just run trace route and make sure not to see the router for intervlan routing?

 

[itronin]

Have a question and wonder if I am screwing things up. Got it all to work as a layer 2 switch but now I am trying to use the icx6450 to do intervlan routing.

You did not mention which switch port your firewall/router is connected to nor whether that port is a member of vlan 2.
Posting your switch config would help.
Also what are you using as your firewall/router? Do you have the transit vlan configured as a tagged interface on your firewall/router?

 

[losx]

Sorry I figured I should have put config in.

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 2 name Transit by port
tagged ethe 1/2/1
router-interface ve 2
!
vlan 20 name Cameras by port
tagged ethe 1/2/1
untagged ethe 1/1/1
router-interface ve 20
!                                                                
vlan 30 name IoT by port
tagged ethe 1/2/1
untagged ethe 1/1/24
router-interface ve 30
!
vlan 40 name TestVlan40Network by port
tagged ethe 1/2/1
untagged ethe 1/1/25
router-interface ve 40
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
hostname honstnamehere
ip dhcp-client disable
ip route 0.0.0.0/0 192.168.1.1
!
no telnet server                                                
username root password .....
!
!
!
!
!
interface ethernet 1/2/1
dual-mode  2
!
interface ve 1
!
interface ve 2
ip address 192.168.1.2 255.255.255.0
!
interface ve 20
ip address 10.0.20.2 255.255.255.0
ip helper-address 1 192.168.1.1
!
interface ve 30
ip address 10.0.30.2 255.255.255.0
ip helper-address 1 192.168.1.1
!
interface ve 40
ip address 10.0.40.2 255.255.255.0
ip helper-address 1 192.168.1.1
!
!                                                                
!
!
!
!
!
!
!
end

Firewall/Router: Unifi UDMP
- IP 192.168.1.1
- Port 9 (SFP+ port)
- Static Route 10.0.0.0/8 Next hop: 192.168.1.2

Switch: ICX6450-48P
- IP 192.168.1.2
- Port 1/2/1 (SFP+ port)

Also the last question is something I missed... On the router I have the Switch on the 192.168 network BUT it is not tagged and part of the default network. I probably need to create VLAN 2 on the router and put it on say 172.16.0.1 and the switch at 172.16.0.2, that is likely an issue! I half made a transit vlan and somehow forgot to add it to the router... and update switch to have an IP within that... probably shouldn't be configuring things at 3am....

 

[ArmedAviator]

No need to use tagged VLAN between router and switch. Use untagged VLAN 2 on the switch and no VLANs on the router.

Also be sure that the DHCP server is sending the devices the correct gateway! The IP address of each respective VLAN interface should be used as the gateway on every device.

 

[losx]

ah yeah I just noticed I need to change those to 10.0.20.1 and 10.0.30.1 etc

 

[neb50]

The router just needs to be connected to one of the VLAN's and your setup looks like it should be VLAN 2. The other VLAN's don't need to go to the router unless you are wanting it to do the routing. You can either do that by setting up the UDMP to use port 9 on VLAN2 and keep it tagged on the switch or untag it on switch and have the UDMP use the untagged port 9.

The DHCP server in your UDMP can have the gateway for each subnet point to the ip of each of the ve interfaces (10.0.x.2) and the switch will figure the rest out. In the Edgerouter it is listed as Router in the DHCP web setup to configure the gateway.

 

[losx]

So the problem I am running into is if I take vlan 40 for example and remove the tag to e 1/2/1 which is on vlan 2 I can no longer ping anything outside the switch. Within the switch I can access other vlans but I lose access to the anything outside the switch and outside the switch I can't get back in either.

Is something being done incorrectly?

vlan 40 name TestVlan40Network by port
untagged ethe 1/1/25
router-interface ve 40

interface ve 40
ip address 10.0.40.2 255.255.255.0
ip helper-address 1 10.0.40.1