@EngChiSTH , sure thing. It won't be a full guide at present as time doesn't permit but I'll get what I can on this post.
Here's the important bits from my ICX6610 which is on L3 router duty.
Configuring the L3 switch
Global config section:
Note 1: The default route for IPv4 (0.0.0.0/0) and IPv6 :/0) point to the LAN interface IP of OPSense (or pfSense, or EdgeRouter or whatever).
Note 2: The dns-domain-list entries are for each of my local VLANs, resolved by my local DNS.
Code:
hostname ks-icx-01
ip dhcp-client disable
ip dns domain-list mgmt.rgn
ip dns domain-list app.rgn
ip dns domain-list iot.rgn
ip dns domain-list cli.rgn
ip dns domain-list nas.rgn
ip dns domain-list pve.rgn
ip dns domain-list voip.rgn
ip dns server-address 10.1.26.5 10.1.26.3
ip route 0.0.0.0/0 10.1.99.2
!
ipv6 dns server-address 2605:aaaa:bbbb:7a99:1c3f:83ff:feef:411d
ipv6 unicast-routing
ipv6 route ::/0 2605:aaaa:bbbb:7a99:1c3f:83ff:feef:411d
Here's my router interfaces for each of my VLANs.
Note 1: Each VLAN interface (ve) has the same ip helper-address which is my DHCP/DNS server, discussed later.
Note 2: If you're new to IPv6 (as I am), each ve that I want IPv6 on is getting it's own globally routable /64. That's basically the second two digits in the 4th section (i.e. 7a01, 7a02, 7a03, etc. in my case). I chose to set the /24 in the IPv4 subnets and the /64 in the IPv6 subnets to the VLAN ID just for simplicity's sake. The next few sections (not sure what they're called....they're not octets anymore) I kept as zeroes so they can be shortened to :: and the trailing 1 is equivalent to the host address in IPv4 such as 10.1.1.1.
Code:
interface ve 2
port-name VLAN-VOIP
acl-logging
ip access-group VOIP in
ip address 10.1.2.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 3
port-name VLAN-IOT
acl-logging
ip access-group IOT in
ip address 10.1.3.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 4
port-name VLAN-SAN
ip address 10.1.4.1 255.255.255.0
ip mtu 9000
ip ospf area 0
ipv6 mtu 9000
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 5
port-name VLAN-MGMT
ip address 10.1.1.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a01::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 10
port-name VLAN-CLI
ip address 10.1.10.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a10::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 26
port-name VLAN-APP
ip address 10.1.26.1 255.255.255.0
ip helper-address 1 10.1.26.3
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a26::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface ve 99
port-name VLAN-OPNS
ip address 10.1.99.1 255.255.255.252
ip mtu 1500
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a99::1/64
ipv6 enable
ipv6 mtu 1500
ipv6 ospf area 0
ipv6 nd router-preference high
!
interface loopback 1
port-name Management
ip address 10.0.0.1 255.255.255.255
ip ospf area 0
ipv6 address 2605:aaaa:bbbb:7a00::1/128
ipv6 ospf area 0
!
So that's the switch section doing majority of routing. To set up your WAN gateway router (pfSense, OPNSense, Edgerouter, etc.) you should set it up with a single LAN interface, no VLANs (that's a whole different setup and what I wanted to get away from).
Configuring the WAN router
IPv6 step only:
If you are setting up for IPv6 with multiple subnets as above you will want to request a /56 subnet from your ISP. I believe most residential IPSs give out a /56 when requested, but your router must specifically request it in most if not all cases. In OPNSense and pfSense, there's an option under the WAN configuration under DHCP6. Select "Prefix delegation size" to /56 and check the box that says "Send IPv6 prefix hint." Release and renew your DHCP/DHCP6 and see what you get - if it worked, your WAN interface will have an IPv6 address of it's own, in my case it gets a /128 address and a completely seperate /56 for my LAN devices.
Here are the IPv6 addresses I get from DHCP6 on my WAN interface:
Code:
IPv6 address 2605:aaaa:ffff:10:5047:ae63:29ef:d367 / 128
Delegated prefix 2605:a000:d401:7a00::/56
The delegated prefix is what we used in the above switch configuration section.
Now, whether or not you use IPv6, you need to set up your LAN interface with the correct addresses.
In my case, I chose to use a /30 IPv4 subnet which allows for 2 hosts. I know a /31 works on most devices, but it's a bit of a "well, I suppose we'll allow it" approach and not guaranteed to work on all devices. In my case I used 10.1.99.0/30 which means I can set the switch to 10.1.99.1 and the LAN interface to 10.1.99.2. For IPv6, I just gave it a /64 for the hell of it.
At this point, your WAN router does not know about the rest of your LAN.
In pfSense and OPNSense, you need to add a gateway before you can add a route. I added the gateways 10.1.99.1 and 2605:aaaa:bbbb:7a99::1/64 which are the addresses of the L3 switch virtual interface trunked to OPNsense.
Add a route or routes appropriate to your network. In my case, everything IPv4 is contained within a 10.1.0.0/16 subnet, so I added the route 10.1.0.0/16 via 10.1.99.1 (the gateway we added above). For IPv6, we route the entire /56 to the switch so it'll be 2605:a000:d401:7a00::/56 via 2605:a000:d401:7a99::1 (the gateway we added above).
You should now have working native IPv6, but NAT will not work yet for IPv4.
In pfSense or OPNSense, go to Firewall > NAT > Outbound. Change the Mode to Manual. Change or copy each of the rules from the original LAN network (10.1.99.0/30) to 10.1.0.0/16. This allows the firewall to NAT outbound traffic from all of your LAN.
Add appropritate firewall rules on your LAN interface to allow IPv4 and IPv6 outbound traffic.
You now should have working IPv4 internet.
DHCP and DNS configuration
Here's just a dump of my /etc/dhcpd.conf file (less the secret):
Note 1: I did not setup reverse DNS yet.
Note 2: Some hosts do not have IP addresses configured in DHCP because they are configured manually on the device.
Note 3: This configuration will update the BIND named DNS server in real-time.
Code:
ddns-updates on;
ddns-update-style interim;
update-static-leases on;
authoritative;
key "rndc-key" {
algorithm hmac-sha256;
secret "nope";
};
allow unknown-clients;
use-host-decl-names on;
log-facility local7;
zone mgmt.rgn. {
primary localhost;
key rndc-key;
}
zone cli.rgn. {
primary localhost;
key rndc-key;
}
zone app.rgn. {
primary localhost;
key rndc-key;
}
zone voip.rgn. {
primary localhost;
key rndc-key;
}
zone iot.rgn. {
primary localhost;
key rndc-key;
}
subnet 10.1.1.0 netmask 255.255.255.0 {
range 10.1.1.150 10.1.1.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "mgmt.rgn";
ddns-domainname "mgmt.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.1.1;
option broadcast-address 10.1.1.255;
default-lease-time 600;
max-lease-time 7200;
}
subnet 10.1.2.0 netmask 255.255.255.0 {
range 10.1.2.150 10.1.2.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "voip.rgn";
ddns-domainname "voip.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.2.1;
option broadcast-address 10.1.2.255;
default-lease-time 600;
max-lease-time 7200;
}
subnet 10.1.3.0 netmask 255.255.255.0 {
range 10.1.3.150 10.1.3.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "iot.rgn";
ddns-domainname "iot.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.3.1;
option broadcast-address 10.1.3.255;
default-lease-time 600;
max-lease-time 7200;
}
subnet 10.1.10.0 netmask 255.255.255.0 {
range 10.1.10.150 10.1.10.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "cli.rgn";
ddns-domainname "cli.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.10.1;
option broadcast-address 10.1.10.255;
default-lease-time 600;
max-lease-time 7200;
}
subnet 10.1.26.0 netmask 255.255.255.0 {
range 10.1.26.150 10.1.26.250;
option domain-name-servers 10.1.26.5, 10.1.26.3;
option domain-name "app.rgn";
ddns-domainname "app.rgn";
option domain-search "mgmt.rgn", "app.rgn", "cli.rgn", "san.rgn", "iot.rgn", "voip.rgn", "glust
er.rgnet";
option routers 10.1.26.1;
option broadcast-address 10.1.26.255;
default-lease-time 600;
max-lease-time 7200;
}
##
## VLAN 2
###
host pbx01.voip.rgn {
hardware ethernet 36:22:12:27:7e:b0;
fixed-address 10.1.2.5;
ddns-hostname "pbx01";
}
##
### VLAN 3
##
host ks-prnt-01.iot.rgn {
hardware ethernet 74:40:bb:aa:6d:a5;
fixed-address 10.1.3.15;
ddns-hostname "ks-prnt-01";
}
host ks-nvr-01.iot.rgn {
hardware ethernet ec:71:db:d3:12:bf;
fixed-address 10.1.3.20;
ddns-hostname "ks-nvr-01";
}
host ks-cam-01.iot.rgn {
hardware ethernet ec:71:db:ac:87:ec;
fixed-address 10.1.3.21;
ddns-hostname "ks-cam-01";
}
host ks-cam-03.iot.rgn {
hardware ethernet b0:41:1d:25:fd:79;
fixed-address 10.1.3.23;
ddns-hostname "ks-cam-03";
}
host ecobee.iot.rgn {
hardware ethernet 44:61:32:64:f0:05;
fixed-address 10.1.3.50;
ddns-hostname "ecobee";
}
##
## VLAN 5
##
host ks-icx-01 {
hardware ethernet 74:8e:f8:e7:b4:b0;
fixed-address ks-icx-01.mgmt.rgn;
}
host ls-icx-02 {
hardware ethernet 74:8e:f8:82:e8:60;
fixed-address ks-icx-02.mgmt.rgn;
}
host arbiter {
hardware ethernet a2:7d:20:dc:94:01;
fixed-address arbiter.mgmt.rgn;
}
host neutron {
hardware ethernet 00:02:c9:3b:a0:40;
fixed-address neutron.mgmt.rgn;
}
host proton {
hardware ethernet 00:02:c9:3b:61:30;
fixed-address proton.mgmt.rgn;
}
host ks-bmc-sm1u-01 {
hardware ethernet 02:25:90:24:7e:86;
fixed-address ks-bmc-sm1u-01.mgmt.rgn;
}
host ks-bmc-sm2u-01 {
hardware ethernet 02:30:48:ca:e1:b0;
fixed-address ks-bmc-sm2u-01.mgmt.rgn;
}
host ks-bmc-sm2u-02 {
hardware ethernet 02:25:90:18:9a:c0;
fixed-address ks-bmc-sm2u-02.mgmt.rgn;
}
host ks-bmc-r710-01 {
hardware ethernet 84:2b:2b:71:7d:17;
fixed-address ks-bmc-r710-01.mgmt.rgn;
}
host ks-bmc-r710-02 {
hardware ethernet 78:2b:cb:23:85:84;
fixed-address ks-bmc-r710-02.mgmt.rgn;
}
host unifictl {
hardware ethernet 8A:CA:FD:F4:24:25;
fixed-address unifictl.mgmt.rgn;
}
host ks-uap-01 {
hardware ethernet 78:8a:20:86:5d:93;
fixed-address ks-uap-01.mgmt.rgn;
}
host ks-pve-01 {
hardware ethernet f2:6f:9b:2c:8b:73;
fixed-address ks-pve-01.mgmt.rgn;
}
host ks-pve-02 {
hardware ethernet 62:7e:20:4d:1f:82;
fixed-address ks-pve-02.mgmt.rgn;
}
host ks-pve-03 {
hardware ethernet ba:48:7e:02:81:6c;
fixed-address ks-pve-03.mgmt.rgn;
}
##
## VLAN 10
##
host area51.cli.rgn {
hardware ethernet 00:02:c9:1b:fe:10;
fixed-address 10.1.10.10;
ddns-hostname "area51";
}
host zenith.cli.rgn {
hardware ethernet fc:f8:ae:7b:c1:13;
fixed-address 10.1.10.11;
ddns-hostname "zenith";
}
host zenith-eth.cli.rgn {
hardware ethernet bc:ee:7b:17:d8:ec;
fixed-address 10.1.10.12;
ddns-hostname "zenith-eth";
}
host htpc01-eth.cli.rgn {
hardware ethernet 84:39:be:68:42:81;
fixed-address 10.1.10.13;
ddns-hostname "htpc01-eth";
}
#wifi
host htpc01.cli.rgn {
hardware ethernet 10:d0:7a:87:68:e7;
fixed-address 10.1.10.14;
ddns-hostname "htpc01";
}
host htpc02.cli.rgn {
hardware ethernet f8:b1:56:df:17:2e;
fixed-address 10.1.10.15;
ddns-hostname "htpc02";
}
host elana.cli.rgn {
hardware ethernet a8:a1:59:2e:53:a9;
fixed-address 10.1.10.16;
ddns-hostname "elana";
}
host xboxones.cli.rgn {
hardware ethernet b8:31:b5:ef:c4:48;
fixed-address 10.1.10.31;
ddns-hostname "xboxones";
}
host xboxones-wifi.cli.rgn {
hardware ethernet b8:31:b5:ef:c4:4a;
fixed-address 10.1.10.32;
ddns-hostname "xboxones-wifi";
}
host andromeda.cli.rgn {
hardware ethernet c2:fb:42:27:d8:66;
fixed-address 10.1.10.61;
ddns-hostname "andromeda";
}
host galaxys5.cli.rgn {
hardware ethernet fc:c2:de:83:9b:70;
fixed-address 10.1.10.62;
ddns-hostname "galaxys5";
}
host cassiesgalaxy.cli.rgn {
hardware ethernet 24:18:1d:60:89:41;
fixed-address 10.1.10.63;
ddns-hostname "cassiegalaxy";
}
##
## VLAN 26
##
host dhcp01 {
hardware ethernet f2:08:5d:b9:8d:a6;
fixed-address dhcp01.app.rgn;
}
host cache01 {
hardware ethernet 96:47:84:a0:2b:48;
fixed-address cache01.app.rgn;
}
host slb01 {
hardware ethernet 52:44:eb:1a:d1:73;
fixed-address slb01.app.rgn;
}
host slb02 {
hardware ethernet 4a:ff:4a:54:8a:c6;
fixed-address slb02.app.rgn;
}
host slb03 {
hardware ethernet fe:fe:2b:95:5c:27;
fixed-address slb03.app.rgn;
}
host web01 {
hardware ethernet a2:90:4f:c1:2c:e8;
fixed-address web01.app.rgn;
}
host web02 {
hardware ethernet 22:11:ad:8a:7d:73;
fixed-address web02.app.rgn;
}
host web03 {
hardware ethernet 92:01:b6:bb:8a:41;
fixed-address web03.app.rgn;
}
host redis01 {
hardware ethernet ca:80:2d:7f:c8:b4;
fixed-address redi01.app.rgn;
}
host mrdb01 {
hardware ethernet fa:18:03:ab:c2:3a;
fixed-address mrdb01.app.rgn;
}
host sync01 {
hardware ethernet 1a:3d:7c:c7:f3:56;
fixed-address sync01.app.rgn;
}
host mon01 {
hardware ethernet 62:55:04:ac:c1:93;
fixed-address mon01.app.rgn;
}
host torr01 {
hardware ethernet 82:a2:71:74:d4:bf;
fixed-address torr01.app.rgn;
}
And here's a dump of my /etc/named/named.conf:
Code:
options {
directory "/var/named";
pid-file "/var/run/named/named.pid";
auth-nxdomain yes;
datasize default;
statistics-file "/var/cache/bind/stats";
zone-statistics yes;
listen-on-v6 { any; };
allow-recursion {
10.1.0.0/16;
127.0.0.1;
};
forwarders {
1.1.1.1;
1.0.0.1;
9.9.9.9;
};
allow-query {
10.1.0.0/16;
127.0.0.1;
};
allow-query-cache {
10.1.0.0/16;
127.0.0.1;
};
allow-transfer {
10.1.0.0/16;
127.0.0.1;
};
allow-update {
10.1.0.0/16;
127.0.0.1;
};
version none;
hostname none;
server-id none;
};
key "rndc-key" {
algorithm hmac-sha256;
secret "nope";
};
zone "mgmt.rgn" {
type master;
file "mgmt.rgn.zone";
allow-update { key rndc-key; };
};
zone "iot.rgn" {
type master;
file "iot.rgn.zone";
allow-update { key rndc-key; };
};
zone "cli.rgn" {
type master;
file "cli.rgn.zone";
allow-update { key rndc-key; };
};
zone "voip.rgn" {
type master;
file "voip.rgn.zone";
allow-update { key rndc-key; };
};
zone "app.rgn" {
type master;
file "app.rgn.zone";
allow-update { key rndc-key; };
};
zone "localhost" IN {
type master;
file "localhost.zone";
allow-transfer { any; };
};
zone "0.0.127.in-addr.arpa" IN {
type master;
file "127.0.0.zone";
allow-transfer { any; };
};
zone "." IN {
type hint;
file "root.hint";
};
logging {
channel xfer-log {
file "/var/log/named.log";
print-category yes;
print-severity yes;
print-time yes;
severity info;
};
category xfer-in { xfer-log; };
category xfer-out { xfer-log; };
category notify { xfer-log; };
};
As far as the zone files go, you can do some web searches how to make those. I just copied and pasted from some website, changed the domains, and listed all of the A records I needed to. Note: only put in A records for devices that are configured manually. DHCP assigned addresses will be added/amended automatically.
Cache Server
I run the LanCache.Net monolithic docker setup with the exception of I set up a few extra distro repository domains to the list (Debian, Ubunutu, Void Linux) so my updates are way quicker. I gave it 600GB and 8GB of RAM as a VM on Proxmox. The majority of the space is used by Steam and Xbox games.
To utilize the cache server, it must be the first DNS nameserver to be queried (see dhcpd.conf).
OPSense/pfSense as VM
I'm using Proxmox VE in a 3-node cluster. It is NOT configured HA due to storage differences between them (2x R710 with 2TB of H700 RAID5 SAS and a 1U whitebox with 120GB MD-RAID1 SSD). The hypervisors are using OpenVswitch with an LACP bond on 2x 10G ports configured with mtu 9000 and another LACP bond on 2x 1G ports configured with mtu 1500 (primarily for management of Proxmox and VoIP traffic).
Setting these up as a VM is pretty straightforward. Install as normal. Connect your home modem to whatever port you desire on your switch and configure as an untagged port in a VLAN. The L3 switch should not have a virtual interface on this VLAN - it's purely L2. Add one vNIC on the virtual machine to the VLAN your modem is on and the other vNIC to the VLAN that is configured with the virtual interface IPs configured in the switch and router configs above.
To increase traffic throughput on the vNICs, set the vNIC queues to the number of vCPUs you gave the VM. You must elect "Advanced" in the Network Device Configuration popup.
To get AES-NI for VPN servers/clients, be sure to select an appropriate CPU architecture to emulate. In my case, I went with Westmere as all of my servers are Westmere or newer architecture so this will advertise and pass-thru the AES-NI CPU instructions.
LLDP / CDP / FDP
On the switch, in gloval config, just do:
You can view detected devices using these protocols with
Code:
show lldp neighbors
show cdp neighbors
show fdp neighbors
LLDP is useful to configure the voice-vlan automatically on LLDP-enabled VoIP phones (see interfaces section of Switch Configuration above).
LLDP can be installed on most Linux distros easily. Install it, run it, you can view connected devices (probably just the switch), but it makes figuring out what's connected to what port very easily on the switch using the commands above.
Code:
SSH@ks-icx-01>show lldp neighbors
Lcl Port Chassis ID Port ID Port Description System Name
1/1/23 842b.2b71.7d0f 842b.2b71.7d0f eno1 ks-pve-01.vmh~
1/1/25 842b.2b71.7d0f 842b.2b71.7d11 eno2 ks-pve-01.vmh~
1/1/35 10.1.2.155 0008.5d1b.472a port 0 Mitel IP Phon~
1/1/37 788a.2086.5d93 788a.2086.5d93 br0 ks-uap-01
1/1/45 10.1.2.156 0008.5d2a.5e60 port 0 Mitel IP Phon~
1/2/2 842b.2b71.7d0f 0002.c91a.faa1 enp6s0d1 ks-pve-01.vmh~
1/2/7 842b.2b71.7d0f 0002.c91a.faa0 enp6s0 ks-pve-01.vmh~
1/3/1 748e.f882.e860 748e.f882.e87c 10GigabitEthernet1/2/4 ks-icx-02
1/3/2 748e.f882.e860 748e.f882.e87b 10GigabitEthernet1/2/3 ks-icx-02
1/3/3 748e.f882.e860 748e.f882.e87a 10GigabitEthernet1/2/2 ks-icx-02
1/3/4 748e.f882.e860 748e.f882.e879 10GigabitEthernet1/2/1 ks-icx-02