Questions for expanding my network

[CobaltFire]

So I just registered, but I've been lurking for a while. Before I begin with the questions I'd like to thank all those who have contributed in the past. I've gotten quite a bit of good info!

A bit over a year ago I built a FreeNAS box based on advice I read here. Here's the build:

Supermicro A1SRM-C2550F-O
8GB Crucial ECC DDR3
Seasonic SSR-360GP (360 watt 80+ Gold)
3 Seagate 2TB NAS HDDs

This has been running perfect for a year now (it's a straight FreeNAS box, no jails). We have frequent power outages and it always starts right back up with no issues. I haven't had a single issue with data corruption, which was the reason I built it. I know I should get a battery backup, and that's in the cards soon. However, we are moving and I am going to be building out my network a little. To that end here are my questions:

I'm looking at building out a pfsense machine with a Supermicro C2X58 board (likely the A1SRM-LN7F-2358F-O, though the 2758 version for $140 more is interesting if I can USE the cores), how do I go about getting the 2.5GbE speeds out of the i354 adapter? Is it automatic? I'd like to at least have my FreeNAS machine running that speed to the pfsense machine (which will handle all traffic; no other switches). What other quality (intel?) adapters are advised (for the PCIe slot) to build out the rest of my network connections?

I've been looking at offsite backup options, and with the upcoming move one has presented itself. Would building a twin of my current box and putting it in place at another home (my parents) with something like BTSync running be a reasonable plan? It gives them a local backup that is also off-site'd automatically for the cost of hardware, bandwidth, and power. Seems to knock a few issues out at once (they could use the backup as well) but I may be missing something.

I've heard quite a bit about pfsense machines being unsuited (for security reasons) to running other applications. I want to run a few minor servers that I cannot justify the hardware for separately (gaming VOIP, webserver, etc.), so want to see about running them either on the FreeNAS or pfsense box. Looking for advice and options here. It just seems that to properly build out the pfsense machine I'm leaving a lot of capability sitting idle. I'm probably overspeccing the system, but I'd rather have idle capability and overspend than have issues, etc. and regret building it in the first place. I've been down that road in the past.

I appreciate the assistance!

 

[MiniKnight]

2.5gb is really just there for KR networking. That's used in the HP Moonshot, Supermicro microblade and others where you have a specific PCB network into a chassis specific switch.

How fast is your WAN?

That motherboard is nice but low CPU and 7x LAN is a lot.

If you did want to run pfsense + jails: *NEW*SuperMicro A1SRM-2758F uATX Motherboard -PCIe x8, PCIe x4, USB2 x7,Quad LAN would be a better option.

 

[CobaltFire]

Forgive my ignorance, but what is "KR networking"? I can make a guess (backplane connections) but am not sure. That said, does that mean I cannot utilize it?

The board you recommended was on my short list, but I will be building out at least 5 ports in the house, preferably 10. I use wired connections where possible rather than wifi for everything. That is why the LN7F was interesting to me. Unless you are recommending that board so that I can add adapters?

If 2.5GbE won't be possible between the FreeNAS and pfsense machine what are my options for building out a backbone there? I'm adding drives to it and the three in it can already saturate GbE.

I suppose one option I hadn't considered was going ahead and virtualizing both machines on something like the X10SDV-TLN4F-O. I know it's not best practices, but if I used VT-d I could hand the i350 to pfsense and a seperate HBA to FreeNAS and use a virtual switch. Would I get better results this way? I know I'd have to run an external switch, which isn't ideal, but if the benefits outweigh the the downsides...

 

[Lost-Benji]

Forgive me if I seem lost but all that comes to mind is to repeat the KISS principle: Keep It Simple, Stupid.

It sounds like you have your networking confused. PFSense is just what its name means, Packet Filter. It is a Firewall and yes, it can be used as a router. the following flow map is how to keep your network simple, fast and stable.

WAN > PFSense (On its own box with two NICs) > LAN (Gigabit Switch, dumb or managed for LACP/LAG)

Now I know some will go all "KB Warrior" on me but I always keep my firewalls (also known as network edge or border machines) on their own hardware NOT shared with other VM's meaning that they would be inside the private network.
The hardware you are looking at I feel is not good use of the money, using an old machine (dual core Pentium D or a Core-2 Duo is great start) and a PCI NIC to supliment the onboard NIC. One will be RED, other is GREEN. If you want to go multi-WAN and run multiple networks (separated networks) then more NIC ports is good but you can always use VLANs as well.

Put a dumb or managed gigabit switch after the PFSense machine and let it do the traffic flow control like it supposed to.

Keep the FreeNAS the way it is, Get a UPS sooner than later as by your own admissions, you are flirting with danger. The money saved on not using an expensive board can go to a NAS.

Offsite backups: Depends on what data amount and how good your WAN uplink is.


P.S I forgot, PFSense sucks at multi-core/multi-threaded. Fast dual-cored CPU will will win everytime.

 

[CobaltFire]

Thanks for the input!

So would it make sense if I went with a small appliance for the pfsense box and something like a ZyXEL XGS1910-24 (24x GbE + 2x 10GbE for ~$475) to build out the internal backbone of the network? I hadn't realized I could get a halfway decent switch with 10GbE ports for under $500.

Is there a reason not to go with something like the C2358, seeing as I don't own any older hardware currently (moving overseas caused me to give away all my older stuff)? It's not the fastest processor, but it seems to be acceptable for most people. I'll be running pfsense, likely some packages as I get to know it, and that's it. My internet will be 150/20 so the throughput isn't too high. Note that power efficiency is very high priority for me, as this will be in an un-airconditioned condo (though the weather is mild) and I'd rather not heat the place up too much.

On another note, are there any recommended UPS's out there? I'm not averse to a DIY solution; I work on more complex stuff for my dayjob.

For the offsite DIY solution, my uplink would be 20, and the other end would be somewhere from 5-20 (have to find out what they have, but it's reasonably fast). Data cap on my side will be ~2TB/month, I'll have to check on the other side (but they don't use much).

 

[MiniKnight]

So would it make sense if I went with a small appliance for the pfsense box and something like a ZyXEL XGS1910-24 (24x GbE + 2x 10GbE for ~$475) to build out the internal backbone of the network? I hadn't realized I could get a halfway decent switch with 10GbE ports for under $500.

Heh - or take the plunge fully: Mikrotik CRS226-24G-2S+IN Cloud Router 10Gbps Layer 3 Switch 24xGbit LAN 2xSFP+

STH review - http://www.servethehome.com/mikrotik-crs226-24g-2s-in-review-you-want-one/

Oh and it's fanless.

 

[CobaltFire]

That's a heck of an option there. I don't know that it would replace a pfsense box, but it makes for a perfect router choice for my needs.

 

[CobaltFire]

Ok, so doing more research I'm really liking that switch. That said I'm looking at my options for the 10GbE SFP+ PCIe card for the FreeNAS box. Are there any recommendations? The intel X520 cards look good and well supported, but they are a bit pricey and require the specific intel transceiver for a fiber link. That's not a HUGE deal right now, but would add a bit of flexibility in placement. Then again, I'm only doing one machine with 10GbE for now (maybe a second down the road a bit).

As far as WiFi AP's I'm hearing good things about Ubiquiti (Arstechnica's review was informative). Are there any other options out there I should look at?

You mentioned that the C2358 would be limited for pfsense with jails. I don't mind spending a bit more for some headroom, but how likely am I to actually use it? I'd like to run a VPN, SNORT, and I'm not sure what else yet.

Also, I want to thank those who gave input so far again. I used to do this for a living, but that was over 10 years ago (and it was Federal Gov, so limited to big name/big budget stuff). Nowadays I'm pretty ignorant and need all the help I can get!

 

[Cole]

Plug and Play in my NAS4Free 10 box 671798-001 HP 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC

 

[Lost-Benji]

Thanks for the input!

1. So would it make sense if I went with a small appliance for the pfsense box and something like a ZyXEL XGS1910-24 (24x GbE + 2x 10GbE for ~$475) to build out the internal backbone of the network? I hadn't realized I could get a halfway decent switch with 10GbE ports for under $500.
2. Is there a reason not to go with something like the C2358, seeing as I don't own any older hardware currently (moving overseas caused me to give away all my older stuff)? It's not the fastest processor, but it seems to be acceptable for most people. I'll be running pfsense, likely some packages as I get to know it, and that's it. My internet will be 150/20 so the throughput isn't too high. Note that power efficiency is very high priority for me, as this will be in an un-airconditioned condo (though the weather is mild) and I'd rather not heat the place up too much.
3. On another note, are there any recommended UPS's out there? I'm not averse to a DIY solution; I work on more complex stuff for my dayjob.
4. For the offsite DIY solution, my uplink would be 20, and the other end would be somewhere from 5-20 (have to find out what they have, but it's reasonably fast). Data cap on my side will be ~2TB/month, I'll have to check on the other side (but they don't use much).

1. This would see a lott more control on the network but freedom to do what you want. There plenty of switches that will do the job fine or run a mixture, smart/managed for core and dumb for bulk ports.
2. I have always found PFSense very touchy when used on slower CPU's when used with things like IPS/IDS, filtering and other ad-ons that need CPU. As I mentioned, PF has never been a real performer on high core-counts. I ran PF on a Del 1950 III (2x 2.4GHz quads & 32GB RAM) for a month and compared to an old Dell PE 850 (Pentium D 3.0GHz & 2GB RAM). The 850 performed just as well and sucks a lot less juice. A fast dual-core will still do best for your speed but quads are more common now. Low on heat/power to keep cool.
   
3. Eaton, APC etc etc. Stay with known brands, avoid cheapies. I love good old APC's as they are built strong, easy to replace batteries and the maggots just keep on going.
   
4. Look at Arcserve, local de-dupe and off-site backup. It will cut down traffic to bare minimum and the backups are continuous incrementals.

 

[Lost-Benji]

1. Ok, so doing more research I'm really liking that switch. That said I'm looking at my options for the 10GbE SFP+ PCIe card for the FreeNAS box. Are there any recommendations? The intel X520 cards look good and well supported, but they are a bit pricey and require the specific intel transceiver for a fiber link. That's not a HUGE deal right now, but would add a bit of flexibility in placement. Then again, I'm only doing one machine with 10GbE for now (maybe a second down the road a bit).
2. As far as WiFi AP's I'm hearing good things about Ubiquiti (Arstechnica's review was informative). Are there any other options out there I should look at?
3. You mentioned that the C2358 would be limited for pfsense with jails. I don't mind spending a bit more for some headroom, but how likely am I to actually use it? I'd like to run a VPN, SNORT, and I'm not sure what else yet.
4. Also, I want to thank those who gave input so far again. I used to do this for a living, but that was over 10 years ago (and it was Federal Gov, so limited to big name/big budget stuff). Nowadays I'm pretty ignorant and need all the help I can get!

1. A word of caution, MikroTiks are cheap and have huge amounts of options.....but, they a very low on grunt. I have used several combos with them and keep hitting same issues of bogging down with traffic flows. Simple 2-4 port LAG's are enough to see big hits in speeds. I certainly don't like them for actuall routing and as border devices.
2. I have used Ubiquiti for a long time, great stuff and does the job well. If you can run up a small linux/Windows VM, run the the UniFi controller on it and then go to town. Lots of options for more than just Wi-Fi. Also pay attention to the new UniFi switches that can do manage the gear as well as power it and kill a few more birds.
3. SNORT is hungry and single-threaded.
4. No worries, we are happy to help.

 

[CobaltFire]

So here's an odd question. I'm preparing for my move and found my old Macbook Pro (3,1 model; 2.2 GHz C2D with 2GB RAM in a single DIMM). It works aside from one of the DIMM slots, and currently has a fresh OS X install on it, but it's so slow we don't use it. Would it function as a decent pfsense entry machine with an expresscard or USB NIC?

Cost isn't the issue, I just don't want to toss the machine if it might do ok as a machine to start with.

 

[CobaltFire]

So here's an odd question. I'm preparing for my move and found my old Macbook Pro (3,1 model; 2.2 GHz C2D with 2GB RAM in a single DIMM). It works aside from one of the DIMM slots, and currently has a fresh OS X install on it, but it's so slow we don't use it. Would it function as a decent pfsense entry machine with an expresscard or USB NIC?

Cost isn't the issue, I just don't want to toss the machine if it might do ok as a machine to start with.

Well, to at least partially answer my own question:

Systems of this age run PCIe 1.0A, which has a bandwidth of 250Mbps for each lane (and only has one lane available via ExpressCard/34), and USB 2.0 which has a theoretical peak of 480Mbps. Given a connection of >100Mbps for your WAN this machine would become a bottleneck simply due to bus throughput. I'm looking at a 300Mbps rated connection so this machine isn't suitable.

Hope that helps someone in the future!

 

[Mikey0843]

Plug and Play in my NAS4Free 10 box 671798-001 HP 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC

The connectx-2 won't work in the current version of freenas because freenas is still stuck on freebsd 9.x which doesn't have driver support for the mellanox cards.
freebsd has included drivers for chelsio 10gbe cards. I recently bought one of these which worked instantly in my freenas box, and it also came with a 3m twinax cable: Chelsio S310E-CR 10Gig Ethernet SFP+ PCIe NIC

 

[PnoT]

The connectx-2 won't work in the current version of freenas because freenas is still stuck on freebsd 9.x which doesn't have driver support for the mellanox cards.
freebsd has included drivers for chelsio 10gbe cards. I recently bought one of these which worked instantly in my freenas box, and it also came with a 3m twinax cable: Chelsio S310E-CR 10Gig Ethernet SFP+ PCIe NIC

I was hoping to get InfiniBand support for FreeNAS working but all the forums posts I see say it hasn't been implemented and devs are not going to waste any time on it. Talk about a buzz kill..