Building 3 machines on the cheap

[HorizonXP]

Hi folks,

So I'm gearing up to build a home office lab in my new house. I'm looking to add the following rack-mountable machines:

1) pfSense router
2) NAS
3) VMware ESXi box

I'm running 10GbE through these, via SFP+ connectors.

I think I've settled on a SUPERMICRO SYS-5018A-MLTN4 1U box to run pfSense on. It shouldn't have any trouble keeping up with 10GbE inter-VLAN routing, right?

I'd like to run dual Xeon E5-2670s in the NAS and VMware boxes, but I can't seem to figure out which chassis and motherboard to use. I'm trying to find stuff on eBay at a decent price, but everything seems to run into the $1000-range.

Any suggestions on what I should be looking out for? I will have plenty of rack space, so they don't need to be 1U or 2U units.

Thanks!

 

[cheezehead]

5018 should work fine

There is a lot of used Supermicro gear on eBay that could work (and some on the forums). Really depends on how many drive bays you need.

In general 1U boxes tend to be on the noisy side (the 5018 isn't bad with the blower and fan control though). There are always a large quantity of 2U boxes available, really depends on what your looking for. The 2670's are pretty much a known price, the unknown is the really the motherboard cost, E5 motherboards are still holding their value pretty well.

Generally it's cheaper to build out from ebay than to purchase whole systems. Where you can pickup deals from time to time is if the seller isn't fully aware of what they are selling. I've been able to pickup some older Westmere servers for no more than the price of a chassis (free memory at the very least). Originally I was looking into both boxes running E5's but the demands of my NAS box didn't really justify the cost, so I ended up sticking with the Westmere mobo/cpu and saved a bit there.

 

[Patrick]

How many drives in the NAS?

 

[HorizonXP]

Maybe 6-8? 3TB WD Reds seem to be the cheapest per TB, but I'm not hard set on that. Plan is to set them up in RAID-Z2 via FreeNAS.

If I can find 1TB or 2TB drives on the cheap, I'd happily go for a 12-bay or 24-bay monstrosity.

How many drives in the NAS?

 

[PigLover]

Not sure why you want/need a dual-CPU system for the NAS. If you are trying to stay "on the cheap" that seems overkill. A low-power Skylake E3 build should save you some money and serve the NAS role perfectly. A D-1508 2-core would do the trick too, but might cost a bit more (X10SDV-2C-7TP4F might not even cost more since it includes MB/CPU/HBA/10Gbe all on one board).

I've never understood the desire for big-iron dual-CPU systems for a NAS...

 

[Deslok]

Maybe 6-8? 3TB WD Reds seem to be the cheapest per TB, but I'm not hard set on that. Plan is to set them up in RAID-Z2 via FreeNAS.

If I can find 1TB or 2TB drives on the cheap, I'd happily go for a 12-bay or 24-bay monstrosity.

4tb seagate 2.5 drives can be found around 110 and 8tb wd's are around 250 depending on how much capacity you need.

 

[Patrick]

Not sure why you want/need a dual-CPU system for the NAS. If you are trying to stay "on the cheap" that seems overkill. A low-power Skylake E3 build should save you some money and serve the NAS role perfectly. A D-1508 2-core would do the trick too, but might cost a bit more (X10SDV-2C-7TP4F might not even cost more since it includes MB/CPU/HBA/10Gbe all on one board).

I've never understood the desire for big-iron dual-CPU systems for a NAS...

Two reasons why I use dual CPUs for storage:

• When I run storage + compute on the same box. E.g. Proxmox + ZFS and Ceph or virtualized NAS on a compute node
• When I need PCIe lanes. For example, the 24x NVMe drive storage node. Then again, that is a $50k+ box and you actually need high numbers of CPU cores/ clocks just to push the storage at that point.

My home, cold storage NAS is still an Atom C2550 with 32GB.

 

[pricklypunter]

I've never understood the desire for big-iron dual-CPU systems for a NAS...

Me either. My little single E3-1245v2 does an amazing job for me on the storage front. I can only imagine how well one of the modern Xeon's would perform in the right setup

 

[PigLover]

@Patrick, I get it when you are doing AIO or using the extra compute for non-storage tasks. But as a pure NAS there is little reason for it.

As for your 24x NVMe box...that's not a NAS...that's a monster. It's a different class of problem.

Sent from my SM-G925V using Tapatalk

 

[Patrick]

As for your 24x NVMe box...that's not a NAS...that's a monster. It's a different class of problem.

Well, it has 4x X550 NICs built-in. I was considering adding a dual XL710 40GBE card and making a post called: Fastest FreeNAS ever?

Even with dual E5-2698 V4's and spreading threads across cores I can push well over 80% CPU utilization without much effort.

 

[Patrick]

On topic (apologies):

1) pfSense - Atom C2550 is good. C2750 marginal cost increase, power you will likely not need. C2558 might be a better option in the event pfSense supports OpenSSL 1.1 with QAT.
2) NAS/ VMware box - if you want two Dual E5-2670 machines, then I would think about virtualizing FreeNAS. At some point, you are going to want to learn clustering. If this completely disinterests you, then the advice of getting a lower power box is a good one.

If you were looking for really cool systems, @rockitlikeithott has $1000 range Cisco UCS systems in his FS forum thread. Those are beastly systems and well beyond what you will find on ebay for that price.

 

[HorizonXP]

Thanks guys,

So for the NAS box, I agree, I'm probably going overboard.

It looks like I can pick up a LGA1155 Xeon on eBay for about $300 total (CPU + MB + 32GB ECC RAM). I would probably need to pick up an IBM M1015 SAS card to add some SATA3 ports. Any suggestions on a chassis to look out for?

 

[HorizonXP]

I saw his post. We use a couple of Cisco C260s at work. I'd love to get my hands on the C240s, but I probably can't make the purchase until late July, and I'm in Canada. I doubt @rockitlikeithott would still have them by then, and be willing to ship it.

On topic (apologies):

1) pfSense - Atom C2550 is good. C2750 marginal cost increase, power you will likely not need. C2558 might be a better option in the event pfSense supports OpenSSL 1.1 with QAT.
2) NAS/ VMware box - if you want two Dual E5-2670 machines, then I would think about virtualizing FreeNAS. At some point, you are going to want to learn clustering. If this completely disinterests you, then the advice of getting a lower power box is a good one.

If you were looking for really cool systems, @rockitlikeithott has $1000 range Cisco UCS systems in his FS forum thread. Those are beastly systems and well beyond what you will find on ebay for that price.

EDIT: Just realized that we could use the server for work purposes on another site we're working on. At $3k for the lot, we could handle taking them all, and then I get one for "free"...

 

[PigLover]

On the chassis for your NAS. You've already chosen 1u rack mount for the other two nodes so you probably prefer rack mount for your NAS?

You need to explore a couple of things to get a good answer. Requirements, if you will. Do you want 2.5 or 3.5 inch drives? How many? How much does noise matter? Do you mind deep chasssis or do you want to atay short depth? Are you willing to buy used/fleabay?

Sent from my SM-G925V using Tapatalk

 

[HorizonXP]

Yup, looking for rackmount on the NAS. Probably 3.5" drives since they're more economical, up to 8. Noise doesn't matter since it will be locked away in the furnace room. Deep chassis is perfectly fine. And yes, willing to go used/fleabay.

On the chassis for your NAS. You've already chosen 1u rack mount for the other two nodes so you probably prefer rack mount for your NAS?

You need to explore a couple of things to get a good answer. Requirements, if you will. Do you want 2.5 or 3.5 inch drives? How many? How much does noise matter? Do you mind deep chasssis or do you want to atay short depth? Are you willing to buy used/fleabay?

Sent from my SM-G925V using Tapatalk

 

[PigLover]

Then look for deals on sc825 (8 drive bays) or 826 (12 drive bays). Prefer 'A' or 'TQ' backplanes (direct attach) to avoid using expanders.

Sent from my SM-G925V using Tapatalk

 

[pricklypunter]

The SuperMicro chassis are really well built and I second the TQ choice on backplane if you decide to go that route. If you ever decide that you need to move to 12Gbps, having the TQ backplane will make it dead easy for ya

 

[pricklypunter]

Thanks guys,

So for the NAS box, I agree, I'm probably going overboard.

It looks like I can pick up a LGA1155 Xeon on eBay for about $300 total (CPU + MB + 32GB ECC RAM). I would probably need to pick up an IBM M1015 SAS card to add some SATA3 ports. Any suggestions on a chassis to look out for?

If you decide to go the 1155 route, @T_Minus has a nice SM board for sale for $60+Shipping in the trade forum, can't go wrong at that price!

 

[_alex]

I would go for cse-826 or 216 for the Nas, depends on what type of drives you want to use.

For the backplane, i think it's hard to saturate an SAS2 with 2x28 as long as it's not all-flash. Also, switching a backplane is a 10-minute task, did it recently replacing an 'A' with 'SAS2' for less cabling mess and the ability to go with a single adapter on all bays.

So you could upgrade at a later point if necessary. Or just add a JBOD via external SAS.

 

[cheezehead]

I would go for cse-826 or 216 for the Nas, depends on what type of drives you want to use.

For the backplane, i think it's hard to saturate an SAS2 with 2x28 as long as it's not all-flash. Also, switching a backplane is a 10-minute task, did it recently replacing an 'A' with 'SAS2' for less cabling mess and the ability to go with a single adapter on all bays.

I just went the opposite on my cse-216, went from SAS2 with single-port expander to 'A'. Beyond the SSD bottleneck scenario, it allows you to go with SAS3.