Help with Switch & possibly Firewall

[losx]

I am working on my new hardware for my home rack and was hoping I could get some help/recommendations/tell me I am being dumb.

Currently I am running:
Unifi UDM Pro
non managed switch (for the moment)
Unifi AP's
Unifi Camera's
FreeNas server with storage, plex and a few other goodies and VM's

I picked up the UDM-Pro very recently (few days ago) and was trying to figure out what switch to go for. The problem I am running into is that Unifi only offers L3 functionality on their Pro line switches and that is "coming soon". While I guess I could wait I was hoping to use something that is available soon. I know the UDMP has its issues but for home use having an integrated controller, protect, access and router is very easy. I can also enable IPS/IDS on the 1GB symmetrical connection and have VPN setup as well.

I amd thinking my future setup will be UDMP SFP+ 10gb to switch and switch 10gb to freenas server.

I want to setup separate vlans for:
IOT, Cameras, Guest Network, Main Network and management network with AP's hosting a few different ones and other elements wired up directly. Switch will need to provide POE.

Issues I am currently running into:
Unifi doesn't appear to support L3 on the switches yet (when will it come?) but I will be honest their UI makes management extremely easy (when things work that is) and its hard to not see that as a big selling point! They pro switches are also fairly expensive for seeming to not yet offer anything

I expect to have around 10 - 1080p cameras when finished with half running constantly and the others on motion with UDMP running protect application. IOT devices will be streaming video's etc from the Freenas server/internet (kids) and home machines will be interacting with freenas server as well with weekly backups to cloud. I can probably get away not needing an L3 switch if the UDMP is routing at 10GB but I would prefer not to have to go back to the router for inter vlan traffic and honestly its a new toy. After much reading here I was interested in the brocade 6450 but was wondering how well that will play with the UDMP (should be just fine). In theory as long as the UDMP follows tagging and 802.1q I should be fine but after watching a lot of videos and reading setting up and maintaining the brocade seems to be a bit of a pain especially compared to unifi which has a great management UI and maps/shows the ports on the UI.

Thing is a 6450 can be had for about 120 which gives POE, L3 switching capabilities, is a true enterprise switch and 4 SFP+ ports vs a unifi pro switch which is 699 and doesn't seem to yet have l3 routing functionality from what I can see. I may be mistaken.

I guess do others have experience with using unifi as their router and a different switch? I know many may recommend getting PFsense but I prefer a device that is rack mountable and has a low power footprint. Running a xeon processor will burn more power than I really want and I prefer my router to not be a power hog.

Also for people who understand basic routing conceptually but don't do it as a day job how hard is it to maintain a brocade switch through console? I have seen a ton of excel sheets around to help manage the ports and honestly this is one of the really nice unifi features. Yes I am somewhat lazy ill be honest but how rough a learning curve will the brocade switch be and what would you all recommend given everything above?

 

[gregsachs]

FWIW, I've got a similar setup albeit a USG-3p and ICX6450-48p. I purely use the ICX as L2, and trunk between the USG and the ICX. That means I'm limited to 1gb between vlans (or less, depending on how much downlink I'm using), but the firewall/acl is easier to do on the USG than the ICX. RIght now I have 1GB/35MB service, and don't care about IPS/DPI, so the USG works fine for me.... I moved from an aruba to the ICX, the console isn't terrible. I find the ICX web ui has more features than the aruba, but it is harder to do something than it was in the aruba. The aruba had a nicer dashboard, too.
Once you get things setup, there shouldn't be much maintenance. Once in a while change a port vlan or similar.

 

[losx]

That is helpful and I do agree it is much easier managing in unifi after seeing videos and reading tutorials on the icx switch. If you use it as just a L2 switch then why not get a unifi switch except those wont have SFP+?

 

[CLos]

I have a UDM Pro attached to a 6450-24P and don't regret the decision of going with Brocade. I too was looking at the Unifi pro series as I needed SFP+ ports for a couple of servers. At the end of the day I couldn't justify the higher cost and only getting 2 SFP+ ports. My 6450 powers several G3 flexes, voip phones and a U6-Mesh. Initially I was going to discount the 6450 since only two ports are enabled for 10Gb and the license to enable the additional ports plus L3 routing is prohibitively expensive. Fortunately I read fohdeesha's thread and learned about the license trick.

 

[gregsachs]

That is helpful and I do agree it is much easier managing in unifi after seeing videos and reading tutorials on the icx switch. If you use it as just a L2 switch then why not get a unifi switch except those wont have SFP+?

Because the unifi switches are all stupid expensive vs the $120 or so i paid for the icx6450. (except the USW-FLEX-MINI, at $29 those are pretty nice).

 

[losx]

I hear that greg!

CLos: what DAC or SFP+ module did you leverage since I will know that works. I know the brocade isn't picky but it seems the udmp might be. Since it seems like you didnt have any issues I guess its time to start learning the brocade cli a little better...

 

[CLos]

This one. Have had no issues with it: https://www.amazon.com/dp/B06XHF7JSG/ref=dp_iou_view_product?ie=UTF8&th=1

 

[gregsachs]

I hear that greg!

CLos: what DAC or SFP+ module did you leverage since I will know that works. I know the brocade isn't picky but it seems the udmp might be. Since it seems like you didnt have any issues I guess its time to start learning the brocade cli a little better...

currently using cisco 3m dac cables;
Port 1/2/1: Type : 10GE Passive Twinax 3m (SFP +) (Not supported)
Vendor: CISCO-MOLEX Version: 09
Part# : 74752-9520 Serial#: MOC174102HJ
prior i think i used a brocade sfp+ that was $6 on ebay, an aruba sfp+, and probably an avago sfp+

 

[rdillingham]

I'm working with a similar setup and need some help.. I'm using a UDM Pro uplinked to a Brocade ICX 6450-48P using the 10G SFP+
Currently running the L2 code on the brocade and I'm looking to setup some VLANs on my network, but I'm struggling to get traffic moving between the VLANs. Would anyone be willing to share with me what their config looks like on the brocade so I can compare?

 

[gregsachs]

This is probably most of what you need:
Uplink from ICX to USG-3 is e 1/1/1

icx6450(config)#show vlan

Total PORT-VLAN entries: 4
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 2 5 6 7 8 9 17 18 19 20 21 22
Untagged Ports: (U1/M1) 23 25 26 27 28 29 30 31 32 33 34 35
Untagged Ports: (U1/M1) 36 37 38 39 40 41 42 43 44 45 46
Untagged Ports: (U1/M2) 2 4
Tagged Ports: None
Uplink Ports: None
DualMode Ports: (U1/M1) 1 10 11 12 13 14 15 16 48
DualMode Ports: (U1/M2) 1 3
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 10, Name mgmt, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 3 4 24
Tagged Ports: (U1/M1) 1 10 11 12 13 14 15 16 48
Tagged Ports: (U1/M2) 1 3
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 98, Name IOT, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 47
Tagged Ports: (U1/M1) 1 10 11 12 13 14 15 16 48
Tagged Ports: (U1/M2) 1 3
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 99, Name GUEST, Priority level0, Spanning tree Off
Untagged Ports: None
Tagged Ports: (U1/M1) 1 10 11 12 13 14 15 16 48
Tagged Ports: (U1/M2) 1 3
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

 

[rdillingham]

Thanks Greg.. Maybe my issue is actually elsewhere as my config looks similar.

Port 1/2/1 is my uplink from ICX to UDM
1/1/47 and 1/1/48 are Unifi APs

I have those ports in dual mode because they have a management IP in VLAN 10 (192.168.10.0), but also tagged in VLAN20.

Total PORT-VLAN entries: 3
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M2) 2 3 4
Tagged Ports: None
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 10, Name MGMT_VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 1 2 3 4 5 6 7 8 9 10 11 12
Untagged Ports: (U1/M1) 13 14 15 16 17 18 19 20 21 22 23 24
Untagged Ports: (U1/M1) 25 26 27 28 29 30 31 32 33 34 35 36
Untagged Ports: (U1/M1) 37 38 39 40 41 42 43 44 45
Tagged Ports: None
Uplink Ports: None
DualMode Ports: (U1/M1) 47 48
DualMode Ports: (U1/M2) 1
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 20, Name DATA_VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 46
Tagged Ports: (U1/M1) 47 48
Tagged Ports: (U1/M2) 1
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled



I have my networks created on my UDM and I can ping 192.168.20.1 from my laptop that is currently sitting in the 192.168.10.0/24 network, but I can't ping any devices that are on VLAN20 and plugged in to the ICX (Currently have a PC plugged in to 1/1/46 for testing). Any advice?

 

[gregsachs]

Does the UDM have an IP address in each vlan?
You need to either have the UDM in each vlan, and it handles the routing between vlans and to outside world, or have the brocade in each vlan handling routing between vlans and have a dummy vlan with only the brocade and udm that all traffic routes out through.
So my USG has an Ip at .1 in each vlan, which is assigned as the gateway in each vlan.

 

[rdillingham]

Yes the UDM has an IP in each VLAN VLAN10 (192.168.10.1) and VLAN20 (192.168.20.1)
The Brocade is configured with a management address (192.168.10.2).

From my laptop which is currently in VLAN10 and I can ping the UDM gateway in each VLAN, but if I put my laptop in VLAN20 on the brocade switch I can't ping the UDM at all.

 

[gregsachs]

Hmm. If I click on the USG, then interfaces, i see the 4 IP addresses listed. Under network, make sure each has the vlan assigned. (Unifi settings, networks, edit, vlan id under advanced; that wasn't set once on a vlan for me and it was fubar...). I do not have any static routes assigned in my usg.
What is doing DHCP for the vlans, and what gateway is getting assigned? My USG does the dhcp, and the usg address in each vlan is the provided gateway.
What does show interfaces on the udm give?

I get:
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 - A/D
eth1 192.168.15.1/24 u/u LAN
eth1.10 192.168.10.1/24 u/u
eth1.98 192.168.98.1/24 u/u
eth1.99 192.168.99.1/24 u/u
eth2 x.x.x.x/22 u/u WAN

(note that eth0 on my usg is dead)

 

[rdillingham]

Unfortunately the UDM doesn't have the same CLI as USG so I can't run the Show Interfaces command. However when I click on settings then Networks from the GUI this is what I currently have.



UDM Pro is running DHCP for both DATA_VLAN and MGMT_VLAN and the UDM Pro interface in that VLAN is the default gateway for each.

 

[rdillingham]

I did some more testing today and I plugged a machine directly in to one of the switch ports on the UDM and set that specific port to VLAN20 and it works great. I can ping from my laptop in VLAN10 on the brocade to the machine in VLAN20 on the UDM. However if I try and put a machine in VLAN20 on the Brocade I still can't ping it. It must be something with my uplink ports from the UDM to brocade right?

 

[gregsachs]

I did some more testing today and I plugged a machine directly in to one of the switch ports on the UDM and set that specific port to VLAN20 and it works great. I can ping from my laptop in VLAN10 on the brocade to the machine in VLAN20 on the UDM. However if I try and put a machine in VLAN20 on the Brocade I still can't ping it. It must be something with my uplink ports from the UDM to brocade right?

One thing I see is that MGMT is not called out explicitly as vlan10, which makes me think that it is treating mgmt as untagged/vlan if it originates on the UDM. The brocade, meanwhile, will tag it as vlan 10. This might be the issue, but it may also be a unifi showing things funky issue.

 

[rdillingham]

One thing I see is that MGMT is not called out explicitly as vlan10, which makes me think that it is treating mgmt as untagged/vlan if it originates on the UDM. The brocade, meanwhile, will tag it as vlan 10. This might be the issue, but it may also be a unifi showing things funky issue.

I'm thinking you may definitely be on to something here. I repurposed the default "LAN" Network on the UDM Pro which doesn't seem to allow you to tag a specific VLAN ID. I'm going to try and make some changes this week and will post an update once completed. Thanks for all your help!

 

[rdillingham]

Sorry for the delay, but I finally had a chance to mess with this again this last weekend and sure enough that was my issue! I changed the name of my MGMT_VLAN back to the default LAN name then created a new tagged VLAN 10 with the 192.168.10.0/24 subnet and I'm in business! Thanks for the help!

 

[gregsachs]

Cool, glad i was able to help out.