PFSense to build switches with 10GbE

[Keljian]

I think what @Patrick is saying is that you need to know:
1. Which optical transmitters to buy (sfp+) if going fibre
2. Which cable/fibre to use(om3/om4, lc-lc)
3. Whether direct attach (twinax) is an option as it is sometimes cheaper
4. The drawbacks of using twinax vs fibre. (Twinax is limited to 10 metres, where fibre can do 300+ depending on the fibre)

Once you have all that it is reasonably straightforward

 

[Pri]

I see, thanks guys that's good information.

 

[gigatexal]

I have plans to build a homemade 10G switch with PFsense next month. I've bought a few dual port Mellanox 10G SFP+ cards which will hopefully arrive within a few weeks(cheap shipping to europe takes time). I'm gonna write a small article about it, so stay tuned!

did you ever get around to this?

 

[BackupProphet]

I will start in two weeks.

 

[BackupProphet]

Sitting here now, PFSense cannot recognize my Mellanox MNPH29D-XTR ConnectX-2 EN. Couldn't install the drivers either as PFsense has removed build tools from the base system. Trying FreeBSD 10.2 now, looks I will have to compile the drivers first and we will see. They work in Linux though.

 

[BackupProphet]

Works great with FreeBSD 10.2
Drivers are not enabled by default.
To enable the driver simply do this (compile, install and load the new kernel module)

cd /usr/src/sys/modules/mlxen && make && make install && kldload mlxen

Then you need to create a bridge in FreeBSD

# show all devices
ifconfig -l
mlxen1 mlxen2 mlxen3 mlxen4

ifconfig bridge create
ifconfig bridge0 addm mlxen1 addm mlxen2 addm mlxen3 addm mlxen4

# make sure they are up
ifconfig mlxen1 up
ifconifg mlxen2 up
ifconfig mlxen3 up
ifconfig mlxen4 up

# add ipp adress
ifconfig bridge0 inet 192.168.0.1/24

And that should be it, you should now have a switch.

I just tested the performance with iperf. 4 servers with Linux(Ubuntu 14.04 server). Two servers connected to one dual port NIC (mlxen1 & mlxen2). The two clients connected to the other NIC(mlxen3 & mlxen4). Managed to get total 17GB/s, 8.5 GB/s each. If a server and client used the same NIC, 18.2GB/s

This is the results without any tuning and default 1500 MTU. All systems are older AMD Athlon 2 X2 servers, the "switch" is the exception with an Phenom2 X4. I wonder if it is possible to compile the driver on FreeBSD 10.1 and copy the /boot/kernel/mlxen.ko file to PFSense.

 

[gigatexal]

wow, good speeds there!

 

[alex1002]

Does the switch depend at all on the speed of cpu,ram,hdd. In other words will the box that you use to create a switch be possible to create a bottle neck?

 

[pyro_]

Hmmm interesting. Could definitely have some fun with a mix of 10 and 40gbe nics depeding on how well this worked

 

[gigatexal]

Does the switch depend at all on the speed of cpu,ram,hdd. In other words will the box that you use to create a switch be possible to create a bottle neck?

I don't know how overhead plays into it all but I do know that modern switches have dedicated asic's that do all the switching and just that so they're quick and low power, I'd imagine you'd need a fast cpu for this (lot's of hertz vs. more cores) though ram and hdd speeds would really only be needed for logging me thinks.

 

[BackupProphet]

I didn't properly benchmark this. But FreeBSD is heavily multithreaded, so more cores are better than more hertz. Keep in mind that hyperthreading may have a negative effect. The best way is to try it out for yourself, and tune the system for your satisfaction. I wouldn't be afraid to hit a bottleneck with 4-6 10G ports.

Recommended reading:
FreeBSD Network Performance Tuning @ Calomel.org
FreeBSD forwarding Performance [BSD Router Project]