New to Enterprise Gear - Looking for a Starting Point

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

MazerRackham101

New Member
Jul 31, 2019
12
0
1
I'm not sure if this is the right place to post this, but: I just ordered some old server gear (SuperMicro X9DRi-F and CSE-846 with 2xE5-2630L v2 and 64GB ECC RAM), and am looking into HDDs. My NEED is to expand my current storage solution, serve media to my house (currently happily using Plex), maintain a bittorrent client, and do it through my VPN.

My WANT is to experiment and play around with enterprise server gear and just generally do more, like having my own cloud storage solution. I'd like to play with some sandboxy-Windows VMs, an OSX VM, and some linux distros (not all at the same time). I'd also like to add whatever network security functionality I can, plus things like PiHole. The biggest issue is I'm struggling with the leap from consumer-grade solutions to enterprise-grade solutions.

I am currently leaning towards ESXi with FreeNAS and some utilities (like pihole, maybe a PXE server?), but there are just SO many options I don't really know where to go. I understand ESXi can do ZFS natively, is that better? Should I just use Ubuntu Server? The last thing I want to do is have to wipe the storage pool and start from scratch with re-transferring all the files over.
 
Last edited:

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
A highly subjective first post, welcome to the madhouse :)

That server will make a really good base to get you going, and should have plenty of tweakability going forward, but I warn you, it's addictive :p

My own personal feelings are that I like to keep it simple, and as modular an approach as possible. Every functional block I have, is in it's own VM using ESXi. For example, I run a Debian VM with ZoL to provide my storage solution. I then hand out ZVoL's over iSCSI to anything I need to from there. I have several other VM's all doing various things, from firewalling to media serving, all on one box. There are lots and lots of options open to you. A lot of folks use a pre-built NAS, like FreeNAS for example, and use NFS to met out space to ESXi to use for datastores etc.

On the disk front, best bang for buck right now is probably still the 3TB HGST's, those can be had really cheap now, but are getting long in the tooth. Good ones are probably fine for a year or so to get you going though. A real close second now would be the 8TB HGST's, if you have the budget available. The price on them is tumbling quickly, now that the 10TB and above is gaining traction. If performance is what you need from your disks, the newer the better and you'll want to mirror them. If on the other hand, you need lots of space for media/ file storage, you might prefer to use something like RaidZ2, with the caveat that any rebuild times for bad disks will be appreciably longer. Don't forget something in the way of the all important back-up strategy/ disaster planning. You never know when it's going to strike, but I assure you, it will, and when you least want or expect it. So plan for it from the get go :D

Overall, I would say, look closely at each Hypervisor you plan on using and see what is going to be the least painful way of achieving your goals in terms of VM's Management and Administration etc. Don't be afraid to test out several methods of getting going before finally committing to what you feel comfortable with using. You might not need VM's at all and may be able to achieve your goals using containers, for example. It's a fun journey. Also, don't be afraid of the scary command line or new to you OS's, some of the fun is in learning things outside your comfort zone :)
 
  • Like
Reactions: nthu9280

MazerRackham101

New Member
Jul 31, 2019
12
0
1
A highly subjective first post, welcome to the madhouse :)

That server will make a really good base to get you going, and should have plenty of tweakability going forward, but I warn you, it's addictive :p
Thanks for the welcome and the warning. I have no doubt.

My own personal feelings are that I like to keep it simple, and as modular an approach as possible. Every functional block I have, is in it's own VM using ESXi. For example, I run a Debian VM with ZoL to provide my storage solution. I then hand out ZVoL's over iSCSI to anything I need to from there. I have several other VM's all doing various things, from firewalling to media serving, all on one box. There are lots and lots of options open to you. A lot of folks use a pre-built NAS, like FreeNAS for example, and use NFS to met out space to ESXi to use for datastores etc.
My hope is to play with a bunch of stuff without borking my media/storage server side of things. That's why I'm really trying to do homework on the foundation right now. ESXi seems like a good solution (although there are great FOSS alternatives that might be better). The actual ZFS pool implementation I'm leaning to FreeNAS but I don't want to have to redo this once I get it set up and working. I currently have ~8TB usable that's nearly full and I want to move to something that's got ~32TB usable. Redoing that simply won't be possible for me once I unwind my current "server" as I won't have anywhere to put that much data.

ZOL (via Ubuntu Server, maybe?) is my second option, and I think that may be the most "normal" for me in that I'd have a "normal" OS handling things for me.

On the disk front, best bang for buck right now is probably still the 3TB HGST's, those can be had really cheap now, but are getting long in the tooth. Good ones are probably fine for a year or so to get you going though. A real close second now would be the 8TB HGST's, if you have the budget available. The price on them is tumbling quickly, now that the 10TB and above is gaining traction. If performance is what you need from your disks, the newer the better and you'll want to mirror them. If on the other hand, you need lots of space for media/ file storage, you might prefer to use something like RaidZ2, with the caveat that any rebuild times for bad disks will be appreciably longer. Don't forget something in the way of the all important back-up strategy/ disaster planning. You never know when it's going to strike, but I assure you, it will, and when you least want or expect it. So plan for it from the get go :D
I'm working on proper backup solutions right now, but I've got everything duplicated and air-gapped that REALLY matters to me. My plan is to either suck it up and pay the monthly for an online backup solution or set up a remote server at my parents' house a few hundred miles away. Either way, that's going to be further down my list of priorities.

As for the 3/8TB HGST drives: Are you talking about these? I'm still trying to finish out my HDD selection, and I've been leaning towards shuccing 5x 8TB WD Elements (BHPhoto $120) or 8TB Seagate Backup Plus Hub (Costco $120). I've seen some 3TB drives (I can't remember what they are now) that were 6GB/s SAS and had the same cost/TB as the 8TB listed above. What I'm torn on is whether or not I'd rather 12x3TB drives with two parity drives (Z2) or 5x8TB drives with only one parity drive (Z1). Also: Is it better to get a bunch of the exact same drive, or would it be better to vary it a bit? The crazy in me wants all identical drives for more uniformity, but pragmatically it seems like there'd be an increased chance of total data loss if all the drives were Mfg'd in the same batch and had the same defect.

Overall, I would say, look closely at each Hypervisor you plan on using and see what is going to be the least painful way of achieving your goals in terms of VM's Management and Administration etc. Don't be afraid to test out several methods of getting going before finally committing to what you feel comfortable with using. You might not need VM's at all and may be able to achieve your goals using containers, for example. It's a fun journey. Also, don't be afraid of the scary command line or new to you OS's, some of the fun is in learning things outside your comfort zone :)
Thanks, and that's definitely my plan. Like I said above: I'm looking forward to playing with options, but I'm dreading the thought of having to start again after transferring all my data over the first time. I'm likely going to set up my storage pool and get it working right with a dramatically reduced subset of what I want to stream (ie: Just a couple of shows instead of everything all at once) so I can still fall back to my current solution. However, I don't want that to take too long. Once the Hypervisor and Storage Pool solutions are settled, I look forward to virtualizing everything I can think of.
 

zack$

Well-Known Member
Aug 16, 2018
701
315
63
If you insist on a FreeNAS VM, ESXI is the only Hypervisor, that I am aware of, that is approved by ixsystems. Others do not properly pass through the HBA and disk info, which is crucial for SMART.

If you have an HBA, then you can pass thru the HBA to FreeNAS (for bulk storage) and use the Sata and SCU for local storage. With the X9DRi-F, you should be able to pass thru the sata ports (via pch) differently from the SCU, which gives you more options.

If you are going to use NFS for VM storage, you will want to consider the usual sync tweaks and mirrored vdevs is a must for performance.

There is a lot of talk about the pros and cons of multiple servers vs one server setups on the forums. Depends on what you want really in terms of risk, ease of management, power, noise etc.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Oh, I forgot to mention, if you are going the ESXi route, the VMUG deal is pretty sweet considering what you're getting to play with :)

The best "free" alternative that comes to mind would be Xen, which is also excellent. The nice thing about Xen, is that it's built on Linux, so is highly agile.

Don't, I repeat don't put your data on anything that even remotely looks like a raid5 array (raidz1). Disks are not that expensive nowadays, especially good used enterprise disks from reputable IT re-sellers/ recycler's, which if I were you, I would seriously consider. Instead, you should be adding redundancy, not risking losing your data in a resilver, if another disk fails. The WD Red's are great disks, and there's plenty of folks here can testify to that fact. I must confess to being a bit of an HGST fanboi in recent years, after having more than my share of Sickgate failures, but everyone has their favourite manufacturer, based on life experience :)

The disks you linked to are great disks, but they are both SAS and 4Kn. I personally use disks that are 512e just for the broadest compatibility and the ability to re-purpose them if and when the need arises. If array performance is your goal, go multi-path SAS, but if a lighter usage or being able to re-purpose the disks later is envisaged, go with SATA disks for the compatibility. Obviously, you don't want to mix them in the same array. To touch on your other point, it's just as likely for more than one disk to fail from disks manufactured 10 years apart as it is with those 10 minutes apart. It's a myth that all the disks from the same batch suffer the same fate. Disks are pretty much unique now, each has it's own identity and lifespan. Obviously there are other influences that play a part in determining a disks useful life, like vibration, thermal shock, humidity and etc, but you get the idea :)
 
  • Like
Reactions: dswartz

MazerRackham101

New Member
Jul 31, 2019
12
0
1
If you insist on a FreeNAS VM, ESXI is the only Hypervisor, that I am aware of, that is approved by ixsystems. Others do not properly pass through the HBA and disk info, which is crucial for SMART.
I'm definitely not stuck on FreeNAS. I'm also heavily considering ZOL on Ubuntu or Debian. ZFS is really what I'm after, FreeNAS is just the way to achieve that that I've done before.

If you have an HBA, then you can pass thru the HBA to FreeNAS (for bulk storage) and use the Sata and SCU for local storage. With the X9DRi-F, you should be able to pass thru the sata ports (via pch) differently from the SCU, which gives you more options.
I had to google all of those acronyms, but: Yeah, that would be the plan. Unfortunately, I'll have to look more closely at getting the Hypervisor/VM drives plugged into something other than the HBA.

If you are going to use NFS for VM storage, you will want to consider the usual sync tweaks and mirrored vdevs is a must for performance.
I'll have to look into what any of that means. I'm hoping to keep my VM storage off of my primary storage, and I've got some small-ish drives around the house I can dedicate to those.

There is a lot of talk about the pros and cons of multiple servers vs one server setups on the forums. Depends on what you want really in terms of risk, ease of management, power, noise etc.
...and I'm doing a lot of reading on it, it's just a lot to take in. Part of the process!
 

Marsh

Moderator
May 12, 2013
2,644
1,496
113
What I'm torn on is whether or not I'd rather 12x3TB drives with two parity drives (Z2) or 5x8TB drives with only one parity drive (Z1)
There are some good deal on 4TB SAS or SATA drives.
I managed to purchased over 50 x 4TB drives for $27.5 to $30 each drive shipped.

I built few 16 x 4 TB drives storage server as backup server for not much money.
 

MazerRackham101

New Member
Jul 31, 2019
12
0
1
There are some good deal on 4TB SAS or SATA drives.
I managed to purchased over 50 x 4TB drives for $27.5 to $30 each drive shipped.

I built few 16 x 4 TB drives storage server as backup server for not much money.
$30/drive for 4TB drives sounds great. Any way of pointing me towards where I can start looking for those kinds of deals?

At those prices, I'm much much more tempted to go that way despite the increased power consumption. Something like 6x4TB in Z2 would double my current capacity, give me two parity drives, and is still cheaper per usable gig than 5x8TBs in Z1.
 

MazerRackham101

New Member
Jul 31, 2019
12
0
1
Oh, I forgot to mention, if you are going the ESXi route, the VMUG deal is pretty sweet considering what you're getting to play with :)

The best "free" alternative that comes to mind would be Xen, which is also excellent. The nice thing about Xen, is that it's built on Linux, so is highly agile.
I've got Proxmox and Xen images to try tonight. I'm looking for ESXi (the same thing as vSphere, right?) but I'm struggling. I signed up for a VMWare account and keep trying to download a trial license for it. I've read that the "free license" is more than enough for what I need (the limits are no cloud admin and no more than 10 virtual sockets per VM) but I can't figure out how to get an image under the free license or a reference to it on the VMWare site.
 

Marsh

Moderator
May 12, 2013
2,644
1,496
113
Any way of pointing me towards where I can start looking for those kinds of deals?
Search Ebay with these search keyword in computer category not just hard drive sub-category

4TB SAS
lot 4TB SAS

4TB SATA
lot 4TB SATA

May last batch of 8x 4TB SAS , I hit the jack-pot, $28.5 each.
All 8 drives are almost new, only have 1-2 days power on time.
 
  • Like
Reactions: MazerRackham101

zack$

Well-Known Member
Aug 16, 2018
701
315
63
I've got Proxmox and Xen images to try tonight. I'm looking for ESXi (the same thing as vSphere, right?) but I'm struggling. I signed up for a VMWare account and keep trying to download a trial license for it. I've read that the "free license" is more than enough for what I need (the limits are no cloud admin and no more than 10 virtual sockets per VM) but I can't figure out how to get an image under the free license or a reference to it on the VMWare site.
Download VMware vSphere Hypervisor for Free
 

MazerRackham101

New Member
Jul 31, 2019
12
0
1
Yeah, that's right where I've been trying. I follow your link, log in, and then click "Register" at the bottom to gain access to the download screen. Then I get a grey screen that's infinitely stuck loading.

Update: I got it quickly on Chrome but Firefox is still loading. It's all good now. I've got it added to the list of Hypervisors to try, along with XCP-ng, Proxmox, and Xenserver.
 
Last edited:

nk215

Active Member
Oct 6, 2015
412
143
43
49
To a home user with a single system, I wouldn’t recommend FreeNAS. Snapshots are nice etc. but the ability to expand the storage pool is very restricted.

Serving NFS back to ESXi as the data store is also a poor choice for home use. The speed is much reduced unless sync is disabled or an Optane cache drive is in the equation.

For a simple setup, I recommend Xpenology running as a VM on an ESXi as a NAS. I still use Xpenology 5.2 on my ESXi and it has been stable as rock for many years. RAID6 is more than enough for Flex.

VM can be stored locally on the host SSD/NVMe (Intel P36xx is my recommended drive for home use. Very stable, good speed and inexpensive). Back up the VM to the NAS or to another local SSD storage.

Xpenology comes with its own cloud service to backup mobile devices to. For remote backup, just buy a Synology and left it as a remote location, and use rsync.

More servers are nice to play around with but look out for utility cost. Your setup may run you 200 watts with all the hard drives. That’s 20$/month for utility.

And No, ESXi does not have ZFS natively.

I stay with ESXi mostly because Grid K2 can be had for cheap to play around with vGPU.
 

zack$

Well-Known Member
Aug 16, 2018
701
315
63
Just my 2cents: Xpenology is nice as a VM but requires some tweaking to work on ESXI (not natively supported).

Also, upgrading your Xpenology can be another issue.
 

MazerRackham101

New Member
Jul 31, 2019
12
0
1
So, I've tried installing ESXi a couple of times with no joy, and no hassles installing XenServer, XCP-ng, and Proxmox. I've been playing with those three and am still not entirely decided on any of them, but XCP-ng is in the lead. I've installed each a few times and installed a VM or two on each, just to see how painful the process was going to be and they all seem very similar in most ways....I've just found clear documentation of passing an LSI card through directly to FreeNAS on XCP-ng.

I've decided on 5x8TB drives and am still targeting a ZFS1 pool, and I'm thinking about reusing my current 5x2TB set of drives in ZFS2 (probably) for backup of my sensitive info (pictures, tax documents, etc). As for expandability down the road, by chassis has 24xLFF drive bays. I'll be using ~12 of them (5x8TB main pool, 5x2TB backup pool, 1x Caching SSD, 1x VM-storage SSD). It'll be limiting in that I'll need to add a full 5-drive vDev, but adding five drives at a time should be okay for when upgrades become necessary. Even at that, I don't see 32TB getting full for a long while.

The reason I'm so interested in ZFS is the protections from bitrot and copy-on-write protection.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
No, not an idiot, I suspect you are just not doing something quite right during your install ;)

Your hardware is fully compatible with ESXi, assuming you are using the X9 Mainboard, Xeon's and RAM you originally listed in your post.
How are you installing ESXi? Onto what media, USB, CF/SD Card, or straight onto a boot/ datstore SSD?
 

MazerRackham101

New Member
Jul 31, 2019
12
0
1
No, not an idiot, I suspect you are just not doing something quite right during your install ;)

Your hardware is fully compatible with ESXi, assuming you are using the X9 Mainboard, Xeon's and RAM you originally listed in your post.
How are you installing ESXi? Onto what media, USB, CF/SD Card, or straight onto a boot/ datstore SSD?
Oh, I'm definitely an idiot...and I'm under-informed on the enterprise world still....but I'm getting there!

Yeah, I'm running the hardware originally posted. X9+2xE5-2630L+64GB ECC RAM. I've got two small HDDs in there and am installing to one of them.

I'm installing ESXi via SuperMicro's IPMI "virtual media/virtual CD Rom" thing, pulling from a networked SMB share. It seems to load fine and start fine, but then hangs with the same error screen:

upload_2019-8-22_14-17-47.png
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Why make things so complicated for yourself...install it onto a wee USB stick on your PC and then boot the server from that on the internal USB port :)

Now while I agree that there are a gazillion ways to go about this, most of which work out fine, until you work out the kinks, keep it simple ;)
 
  • Like
Reactions: MazerRackham101

MazerRackham101

New Member
Jul 31, 2019
12
0
1
Why make things so complicated for yourself...install it onto a wee USB stick on your PC and then boot the server from that on the internal USB port :)

Now while I agree that there are a gazillion ways to go about this, most of which work out fine, until you work out the kinks, keep it simple ;)
Do you mean install from a plugged-in USB Drive and install onto one of the target HDDs? If so, that's easy to answer: I can install hypervisors via IPMI while I'm physically seated at work. Proxmox, Xenserver, and XCP-ng both work fine that way. I'll try ESXi tonight via flash drive with Rufus.