pfSense / 10Gbe Networking Help

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

tsteine

Active Member
May 15, 2019
167
83
28
@IamSpartacus I was a bit quick with the copy paste when I initially made the diagram. the POE edgeswitch only needs vlans 1,5,10, 20, and 90.

edit: as for DMZ services, that would be reverse proxy, nextcloud, game servers, etc. Essentially anything that is connectable remotely resides in the DMZ to be firewalled from the rest of the network
 

blinkenlights

Active Member
May 24, 2019
157
65
28
What throughput can you sustain with smaller packets (1500 MTU or smaller packets)? Imix would be interesting but there is no easy way to do it with iPerf.
Agreed. The best way to prove it out would be a GRE/VPN tunnel out a 10 Gbps WAN with a minimum of 10 Gbps spare capacity all the way to the far end. Smaller packets will result in lower throughput, but I suspect you already know that ;) The key metric here would be millions of packets per second (Mpps) anyhow.

IamSpartacus said:
I'm not clear is your pfsense box doing your inter-vlan routing on the Brocade ICX7450-48?
Yes, my ICX7450-48 runs in L2 mode and the interfaces are configured as untagged VLANs - in other words, the pfSense box is doing all of the routing. I do have some non-standard tweaks but I doubt they make a big difference:

Code:
buffer-sharing-full
symmetrical-flow-control enable
qos mechanism strict
Nothing too fancy, in my opinion.
 
  • Like
Reactions: martini dry

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
@IamSpartacus I was a bit quick with the copy paste when I initially made the diagram. the POE edgeswitch only needs vlans 1,5,10, 20, and 90.

edit: as for DMZ services, that would be reverse proxy, nextcloud, game servers, etc. Essentially anything that is connectable remotely resides in the DMZ to be firewalled from the rest of the network
Do you place the entire server(s) running those services in the DMZ or do you just trunk to those servers and then tag speciifc VM's/containers with the DMZ vlan? I only ask because I've been considering something similar for some time now.


Agreed. The best way to prove it out would be a GRE/VPN tunnel out a 10 Gbps WAN with a minimum of 10 Gbps spare capacity all the way to the far end. Smaller packets will result in lower throughput, but I suspect you already know that ;) The key metric here would be millions of packets per second (Mpps) anyhow.



Yes, my ICX7450-48 runs in L2 mode and the interfaces are configured as untagged VLANs - in other words, the pfSense box is doing all of the routing. I do have some non-standard tweaks but I doubt they make a big difference:

Code:
buffer-sharing-full
symmetrical-flow-control enable
qos mechanism strict
Nothing too fancy, in my opinion.
Very impressive. My pfsense box runs on a c3758, wonder what kind of performance I'd be limited to compared to your E5 v4.
 

tsteine

Active Member
May 15, 2019
167
83
28
@IamSpartacus I place the entire server running the service in the DMZ. If someone were to get access to the server running the service if the entire server is placed in the DMZ; then they will be firewalled from the rest of the network/hypervisors. If I place only the service on the DMZ, but someone gains access to the underlying server through that service, they'll have access outside the DMZ and not be firewalled off from the rest of the network, which would bypass the point of the DMZ in the first place.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
@IamSpartacus I place the entire server running the service in the DMZ. If someone were to get access to the server running the service if the entire server is placed in the DMZ; then they will be firewalled from the rest of the network/hypervisors. If I place only the service on the DMZ, but someone gains access to the underlying server through that service, they'll have access outside the DMZ and not be firewalled off from the rest of the network, which would bypass the point of the DMZ in the first place.
Are those risks mitigated at all by putting the RP on a small lightweight host in a DMZ but many of the services the RP is pointing to on a server on the LAN?
 

tsteine

Active Member
May 15, 2019
167
83
28
@IamSpartacus Yes, it does add an extra layer of security.
If you firewall the DMZ from the rest of your network then place a reverse proxy in the DMZ, that server will only have access to the rest of your network on the ports you have allowed through your firewall.

The point of a DMZ and putting internet-facing services while firewalling it from the rest of your network is essentially just that if you have a breach, they will only gain access to the servers in your DMZ and need to work on penetrating the rest of your network from there.

If you have intrusion detection, you will have time to throw out whomever is breaching your DMZ before they gain access to the rest of your network.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
@IamSpartacus Yes, it does add an extra layer of security.
If you firewall the DMZ from the rest of your network then place a reverse proxy in the DMZ, that server will only have access to the rest of your network on the ports you have allowed through your firewall.

The point of a DMZ and putting internet-facing services while firewalling it from the rest of your network is essentially just that if you have a breach, they will only gain access to the servers in your DMZ and need to work on penetrating the rest of your network from there.

If you have intrusion detection, you will have time to throw out whomever is breaching your DMZ before they gain access to the rest of your network.
Thank you for clarifying that as that's what I suspected.

I don't think I've seen you mention what hardware your TNSR box is running off of? I just upgraded my pfsense box to this unit. I imagine it could route at 10Gbps but not sure beyond that.
 

tsteine

Active Member
May 15, 2019
167
83
28
Thank you for clarifying that as that's what I suspected.

I don't think I've seen you mention what hardware your TNSR box is running off of? I just upgraded my pfsense box to this unit. I imagine it could route at 10Gbps but not sure beyond that.
My box is reasonably powerful, with high clock speeds, although only 4 core/8threads.
Machine specs:
Asus WS C246 Pro
WS C246 PRO | Servere og arbeidsstasjoner | ASUS Norge

Intel Xeon E-2134 (4-core 3.5 base, 4.5 turbo)
Intel® Xeon® E-2134 Processor (8M Cache, up to 4.50 GHz) Product Specifications

32gb 2400mhz ddr4 ECC memory
https://www.kingston.com/datasheets/KVR24E17S8_8.pdf

1x Intel X710-DA2
Intel® Ethernet Converged Network Adapter X710-DA2 Product Specifications


1x Intel XL710-QDA2

1x Intel X550-T1
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
That is one beast of a unit @tsteine. No way my unit will hold up to that kind of performance but I guess I don't really need it when I think about it. I'm thinking I won't need to put any servers in my DMZ network that will need high speed file transfers.

I also just found out that TNSR can not do IDS/IPS, Web Filtering and Malware. Do you use another appliance for IDS/IPS?
 

tsteine

Active Member
May 15, 2019
167
83
28
@IamSpartacus

I've not currently set up any IDS/IPS, I have been looking into options for this and considering setting up something like Suricata as a VM to handle IDS/IPS, and just forward copies of all packets on the external interface to Suricata.

If you want active intrusion protection though, you'd have to apply some coding elbow grease and leverage the TNSR api to stop the traffic.

It's also possible to do this internally, but once we start forwarding (at least in my network) up to 40gbit and 100gbit to suricata, it's going to have some trouble keeping up.
 

blinkenlights

Active Member
May 24, 2019
157
65
28
It's also possible to do this internally, but once we start forwarding (at least in my network) up to 40gbit and 100gbit to suricata, it's going to have some trouble keeping up.
Agreed, but even on sub-Gigabit interfaces you have to be selective with the rules. I know of several people who enabled the entire ET or Sourcefire set and were shocked, shocked I say, that they were dropping packets on flows in the 100-200 Mbps range. My humble opinion, certain functions are better dealt with using an inline filtering proxy or DNS-based block list, not an IDS/IPS.

Appreciate the positive feedback on the firewall, @IamSpartacus. I should point out that I recycled the guts of my file server into the firewall (replacing C2758 servers) when I upgraded... that was not some brilliant insight into what I would need down the road to support 10 GigE ;)
 

Don.key

Member
Apr 10, 2020
48
22
8
I have setup a cluster of two opnSense firewalls recently, based on SM X11SCM-LN8F Mobo's + I3 9100f CPU and T520-CR 10GbE card in each. Switch is Mikrotik CRS317-1G-16S+RM with Chinese Finisar FTLX8571D3BCL Gbics (probably fakes).

I cannot get more then 3gbps with one or multiple iperf3 sessions with default MTU of 1500 between the firewalls,

If I test between one firewall and DL385 Gen10 FreeBSD based Server with T540-CR connected to same switch then I get ~4.5gbps.

I did not spend any time troubleshooting this because it's enough for me now and I have no time but it is not very good. Might be opnsense issue, I remember reading on opnsense forums that opnsense chokes at about this speed.
 
Last edited:

blinkenlights

Active Member
May 24, 2019
157
65
28
Very impressive. My pfsense box runs on a c3758, wonder what kind of performance I'd be limited to compared to your E5 v4.
@IamSpartacus you might be interested in this.. I swapped in one of the E5-1650 v4 processors mentioned here: https://forums.servethehome.com/ind...3-6ghz-base-4-0ghz-turbo-for-under-200.29031/ and ran some additional tests.

Relative to my E5-2667v4 sample (2.9GHz base, 3.2GHz boost I believe) there was virtually no discernible difference in performance. In fact, the biggest difference was Chelsio driver warnings related to receive queue (nrxq) distribution:

Code:
cxl2: nrxq (6), hw RSS table size (16); expect uneven traffic distribution.
cxl0: nrxq (6), hw RSS table size (16); expect uneven traffic distribution.
cxl3: nrxq (6), hw RSS table size (16); expect uneven traffic distribution.
cxl1: nrxq (6), hw RSS table size (16); expect uneven traffic distribution.
With only four interfaces (two physical cards) in the box, I think sticking with the E5-1650v4 is the best plan. The next logical step up would be the E5-1680v4 (8 cores) or E5-2697Av4 (16 cores), both of which are still too expensive for my taste and likely not going to improve performance.