Pfsense multi-lan config.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
So I have a Pfsense router setup with dual wan which has been working fine for the last 6 months.

I have now decided to add a 10GbE network card to it and move my two other machines that are direct connected over 10GbE on to the router.

I am a little at a loss on how to configure the router to manage the connectivity and routing.

I have a Chelsio T420 card which I have now put in the Pfsense router. The nic is dual port but is showing 4 interfaces;
cxgbe0
cxgbe1
ncxgbe0
ncxgbe1

Adding the cx interfaces brings the links up. Using the ncx interfaces seems to do nothing (links stay down).

I am also unclear whether I should bridge the two 10GbE interfaces or have them as separate networks and whichever way the interfaces are configured, how to make sure the two machines on 10GbE connections route to each other via 10GbE and not the 1GbE links they also have.

Current interfaces are;
em0 - Wan --> ISP1
em1 - Wan2 --> ISP2
igb0 - Lan (1GbE) --> HP1810-48g switch --> home network ports.
cxgbe0 - 10GbE1 (10GbE) --> Media Server (FreeNas)
cxgbe1 - 10GbE2 (10GbE) --> My main PC

I have been playing aorund but cannot get the bridging or even just having them on different subnets working.

Any ideas or direction greatly appreciated.
 

acquacow

Well-Known Member
Feb 15, 2017
784
439
63
42
I tried to do similar and use a pfsense VM to route between my 10Gig, and gigE networks and handle jumbo frame issues/etc... I never did get it working, so I'm subscribing in case you do =)

I ended up just keeping with direct cabling between my media VM and desktop box and used some firewalling and interface metrics to put the services I need on the correct links.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Hmm, I have connectivity working and can get 100% 1GbE bandwifth usage conisitantly on a 14GB file. I have got to around 230Gb/s for a 3rd of the file but I suspect caching.

I am going from an 8 drive Raid 10 (4 pairs) HGST Ultrastar 7.2k drives to a Samsung 940 SSD.

One problem may be that I have both 1GbE routes and 10GbE routes to both machines. I will try to disable to 1GbE route on one machines and see what completely breaks ;).
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Umm, ok. I am now getting 280Gb/s consistantly with large movie files.

Both my machine and the server have their 1GbE interfaces disabled so 10GbE is the only way to go.

The CPU spikes to 38% then down to 24% during the 8GB transfer (its an E3-1220L) but ram usage stays the same.

What I did ??

I deleted all the crap I had put in with bridges etc, I then added the two 10GbE interfaces with static IPs on different subnets (192.168.3 & 192.168.4). I then setup DHCP servers on each interface and mapped a SMB drive from teh Freenas server on to my qWin10 server and copied the file.

I also have a few firewall rules allowing all access (any protocal, v4&v6 IP, *,*,* etc) on both 10Gbe interfaces and thats it. No bridging or interface groups at all.