Finalizing new ESXi build - looking for last touches/suggestions

[thecoffeeguy]

Hey all.

As the title states, looking for a few suggestions to finish out my build for a new ESXi home lab.
Here is what I have so far (purchased):

The chassis: Norco 3U fits up to 12x10 mobo (purposely bought a big chassis for noise and cooling

Here is the mobo:
SuperMicro:

http://www.newegg.com/Product/Produc...82E16813182927

Here is the CPU:

Intel Xeon E5-2603 v3 Haswell 1.6 GHz 6 x 256KB L2 Cache 15MB L3 Cache LGA 2011-3 85W BX80644E52603V3 Server Processor - Newegg.com

Going to add 32-64gb of memory to begin with.

Originally, I was going to go with maybe one or two SSD's but I started to think. "You know, I have a pretty sweet mobo, maybe I should explore other options and ask others for recommendations."

That in mind, I wanted to ask:

-storage. I was going to grab one or two SATA or SSD drives for local storage. Then I started to think it would be nice to have some fast storage with IO so my VM's dont lag. I started to think about RAID, maybe even a RAID card. Perhaps even convert one or two of the external 5.25" outer bays to hot-swap bays.

-NIC's. Comes with 2 by default, but maybe add another dual NIC or quad NIC card. Gives me more options to play and setup with in ESXi (vlans, storage networks, DMZ etc.).


Storage is the one i keep coming back to.
My question to friends here is, anything i should consider or keep in mind? I can put a lot of storage in this case (dont have to, but it can hold a lot.) What about RAID?

Lastly, PSU. Probably my weakest area is picking out a proper PSU unit.
What should I be looking for? I know some brands by name, but for a server build like this, this is where i would ask the expertise of others to steer me in the right direction.

Pretty excited for the build. My mobo and CPU arrived today. Case is next week. Hoping to get the memory, PSU and drives ordered this week so i can build it all next week.

I appreciate all the feedback everyone.
looking forward to it

cheers,

coffee

 

[Keljian]

What storage (amount) are you looking to fit it out with?

Your power supply will be dictated by that. My server has a Coolermaster V450 in it, I have 7 drives + an SSD. Factor into your equations that each drive will eat 10W (being conservative) under full load, so 7 drives = 70W.

Add the CPU (your link is dead, so I don't know which you're talking about) which when pressed will probably hit about 100W, a raid card (probably an M1015 or equivalent) @ 10W and a network card at 10W (again conservatively). Add some ram, call it 10W, and other misc 20W.

So total you're looking at about 220W under full load (with 7 drives). I don't know how often the machine is going to be under this load, but if you are looking for a PSU, and this is going to be running 24/7, then you should be looking for something that is most efficient around the power figure you most often use. Mine mostly sits at idle (40-70W) so I picked a power supply which has the highest possible efficiency around that range.

Re Nic - if you're going to do storage, then I'd suggest feeding the rest of the network with 10gig, just because it can simplify things. No need for LACP or anything, one cable, everything runs across it. For home use the Chelsio T420-SO-CR was my choice. You can get a Mellanox Connectx-3 for similar pricing. You can always use virtual functions and vlans to experiment with different configurations.

 

[thecoffeeguy]

What storage (amount) are you looking to fit it out with?

Your power supply will be dictated by that. My server has a Coolermaster V450 in it, I have 7 drives + an SSD. Factor into your equations that each drive will eat 10W (being conservative) under full load, so 7 drives = 70W.

Add the CPU (your link is dead, so I don't know which you're talking about) which when pressed will probably hit about 100W, a raid card (probably an M1015 or equivalent) @ 10W and a network card at 10W (again conservatively). Add some ram, call it 10W, and other misc 20W.

So total you're looking at about 220W under full load (with 7 drives). I don't know how often the machine is going to be under this load, but if you are looking for a PSU, and this is going to be running 24/7, then you should be looking for something that is most efficient around the power figure you most often use. Mine mostly sits at idle (40-70W) so I picked a power supply which has the highest possible efficiency around that range.

Re Nic - if you're going to do storage, then I'd suggest feeding the rest of the network with 10gig, just because it can simplify things. No need for LACP or anything, one cable, everything runs across it. For home use the Chelsio T420-SO-CR was my choice. You can get a Mellanox Connectx-3 for similar pricing. You can always use virtual functions and vlans to experiment with different configurations.

Thanks Keljian.
CPU link is fixed.

Storage wise, say a 6-12 terabyte range. Fast is what I want for the VM's. Will have a couple of databases running that is why i was thinking of RAID setup.

The server load will range, but will not be under stress for a better part of the day. Mostly during work hours when i work and test from home. At night, everything should run idel (maybe some batch jobs at night).

I was looking at some Seasonic PSU's. Any thing I should look for in a PSU (server type vs desktop, number of connectors etc.)

And as far as HD: does it come down to price in how much do i want to spend (SATA vs SSD drives).

I appreciate it.

Coffee

 

[Keljian]

Briefly as I'm about to start getting my dinner ready:

If you're running 6-12 terabytes - is most of that going to be databases or just generic storage?

Desktop power supplies will typically be quieter than server, but rated to less working hours - that said, I've run desktop supplies in my previous home servers and they seem to last perfectly fine. My last one was a bronze Seasonic, I only changed up because I was looking for more efficiency (breakeven was about 1 year).

Being that your idle will probably be in the 70W-100W range and the box will spend most of its life in that state, you will probably want to look at power supplies that are most efficient through that range, and peak(efficiency) around 200W. re connectors? well that depends on how many drives you are going to run, I would suggest fewer larger drives (4x4 TB in Raid 10?) if you are going spinners vs ssds. - Your IO depends on your budget

There are quite a few options for more throughput using second hand SSDs (a lot of deals are coming up if you keep an eye out on the deals forum here)

 

[thecoffeeguy]

Briefly as I'm about to start getting my dinner ready:

If you're running 6-12 terabytes - is most of that going to be databases or just generic storage?

Generic storage mostly, with some databases (Oracle, Mongo, SQL Server)

Desktop power supplies will typically be quieter than server, but rated to less working hours - that said, I've run desktop supplies in my previous home servers and they seem to last perfectly fine. My last one was a bronze Seasonic, I only changed up because I was looking for more efficiency (breakeven was about 1 year).

Being that your idle will probably be in the 70W-100W range and the box will spend most of its life in that state, you will probably want to look at power supplies that are most efficient through that range, and peak(efficiency) around 200W. re connectors? well that depends on how many drives you are going to run, I would suggest fewer larger drives (4x4 TB in Raid 10?) if you are going spinners vs ssds. - Your IO depends on your budget

Ya, budget will be key.
I guess I could mix and match if i want, depending on price.

There are quite a few options for more throughput using second hand SSDs (a lot of deals are coming up if you keep an eye out on the deals forum here)

I will do that. THanks for the tip.

Last note.
Could I convert one or two o those 5.25 bay to a hot swap bay?
might be nice to have that capability for this setup.

Thanks again!

COffeeguy

 

[T_Minus]

I would convert 1x5.25" to a 2.5" hot swap with icy dock!
Icy Dock 8x2.5" in one 5.25" is the best density you can get, and it fits only the standard/slim SSD none of the older thicker or HGST thick cased SSD.

That way you can run 1 HBA all 8. That gives you the option of adding a lot of SSD capacity or 2.5" HDD capacity.

If you only need 12TB and want performance I would consider doing 8x3TB and running them in a pool of mirrors in ZFS, and then adding a 200GB S3700/S3710 for SLOG device. That would be some great performance at a great price.

 

[thecoffeeguy]

I would convert 1x5.25" to a 2.5" hot swap with icy dock!
Icy Dock 8x2.5" in one 5.25" is the best density you can get, and it fits only the standard/slim SSD none of the older thicker or HGST thick cased SSD.

That way you can run 1 HBA all 8. That gives you the option of adding a lot of SSD capacity or 2.5" HDD capacity.

If you only need 12TB and want performance I would consider doing 8x3TB and running them in a pool of mirrors in ZFS, and then adding a 200GB S3700/S3710 for SLOG device. That would be some great performance at a great price.

I was looking at that exact ICYDock, so I will go ahead and go with that. Nice density and i like its slim.

Performance will be a big piece for me. Just out of curiosity, since this server will host ESXi, can i setup those drives for ZFS? Just trying to walk through the logic on how to set that up.

I agree with you though. Going with what you said as well as others, 8x3 setup in a pool running RAID 10?

On a quick side note, any good places to get deals on hard drives these days?

Going to order my PSU and drives ASAP so i can get start building this out, ASAP!

Thanks again!

COffee

 

[whitey]

Performance will be a big piece for me. Just out of curiosity, since this server will host ESXi, can i setup those drives for ZFS? Just trying to walk through the logic on how to set that up.

HBA in vt-D (passthru mode) on the ESXi host in an all-in-one (AIO) config w/ that HBA connected up to the icydock, however that mess hooks up :-D (hopefully two 8087 conn's and not 8 sata conn's, have not looked at picture...ouch it's 8 sata's).

 

[T_Minus]

I would go with something like an E5-1650 v3 instead of the CPU you selected...
Way more snappy, and overall much more powerful. If you want lower budget E5-1620 V3 is a great CPU too.

The popular All-in-one setup around here is with @gea `s NAPP-IT setup for esxi guest os, storage in general, etc... Napp-IT website has an "AIO" step-by-step guide that tells you how to install esxi to usb, install napp-it, pass-through controller, setup storage, share back to esxi, etc...

@whitey they use fan-out def. a lot of cables!

 

[Keljian]

Unfortunately it does use sata, but that in one of the aforementioned cases would probably be your smallest footprint

 

[Keljian]

Re raid: I prefer mdadm raid 6 with lots of drives due to ease of resizing the array, but many here prefer ZFS.

 

[thecoffeeguy]

I would go with something like an E5-1650 v3 instead of the CPU you selected...
Way more snappy, and overall much more powerful. If you want lower budget E5-1620 V3 is a great CPU too.

The popular All-in-one setup around here is with @gea `s NAPP-IT setup for esxi guest os, storage in general, etc... Napp-IT website has an "AIO" step-by-step guide that tells you how to install esxi to usb, install napp-it, pass-through controller, setup storage, share back to esxi, etc...

The plan was to eventually upgrade my CPU to a better CPU due to budget items. That is my long term goal.

I will look for that guide from gea. Sounds very interesting.

The more I think about this, the more ideas I am having about how to make this better and a lot of fun to setup and test.

That said, what would be the recommendation to get my started for now? Meaning, i have a single 512gb SSD at home that I could use now, but in the future, adding more storage, including a RAID setup for better IO, I was curious about a planning perspective. I know that ties into my budget, but I wanted to plan this out now so I can now what my growth plan will be like.

Should I opt to start with the SSD for now? add the ICY dock? Maybe just go for a (drivesize)x3 for RAID?

I appreciate the feedback and suggestions. Really cant thank everyone enough.

Cheers,

COffeee

 

[thecoffeeguy]

The popular All-in-one setup around here is with @gea `s NAPP-IT setup for esxi guest os, storage in general, etc... Napp-IT website has an "AIO" step-by-step guide that tells you how to install esxi to usb, install napp-it, pass-through controller, setup storage, share back to esxi, etc...

I found the guide from Gea and read up on the NAPP-IT website.

Sounds very cool. I have always been a fan of ZFS (simple, fast and great). Initially, i was planning on hooking up some NFS or iSCSI storage from a NAS box I was going to create (which I still probably do eventually).

But, the more I read, the more options I would like to add.

The board itself supports up to 10 SATA connections out of the box.
My case has (9) 3.5" bays, (3) 5.25" bays which I could easily convert to 3.5" or use a ICY dock for density. Room for growth.

To get myself started, I thought about using a couple of 2 TB drives I have laying around and maybe picking up 2 more drives. A thought: can you mix and match different drive types (speed mostly)? Or is that not recommended?

I think what i will do is get myself started with the SSD I have currently (51gb) but plan on buying more disks and using RAID, maybe a 4x4 setup (depends on cost).

Side note. If i were to use something like NAPP-IT, does that negate the need to use a hardware RAID card?

Just thought i would ask.

THanks,

 

[Deci]

I found the guide from Gea and read up on the NAPP-IT website.
To get myself started, I thought about using a couple of 2 TB drives I have laying around and maybe picking up 2 more drives. A thought: can you mix and match different drive types (speed mostly)? Or is that not recommended?

Side note. If i were to use something like NAPP-IT, does that negate the need to use a hardware RAID card?

first part, yes you can mix and match speeds, but the slowest speed drives dictate the speed of the whole array.

expanding ZFS storage isnt great, better off starting with the number and size drives you want or making a second array/pool and migrating the data into that then retire the old array/pool. it does not self balance so any drives added after the pools started being used wont balance to the same fill level as the others, the less free space your pool has the more this hurts performance.

second part, you dont need raid cards, but you do need HBA cards if you do not have enough sata onboard, running an AIO nappit/esx box requires that you can do VT-d and pass the HBA to the nappit VM but you might get away with passing through the onboard sata, you just have to test it and see.

 

[thecoffeeguy]

hey everyone. Appreciate the help on this thread. Incredibly helpful.
I finally got around to getting all my parts and getting ready to build.

To my embarrassment, as i was looking for my parts, I forgot to grab a heatsink for my CPU.
That said, looking for a good, cooling CPU, but nothing over the top. I do not expect to be pushing the CPU that hard, but something to keep it happy.

One more question as well.
Are there molex to sata converters? I have my PSU (seasonic) which came with 4 SATA connectors. I have about 5 or 6 molex connectors that would be nice to convert those into SATA so I can add more drives. Any brand or items I should be looking for?

Thanks again!

Coffee

 

[Keljian]

There are plenty of cheap molex to sata connectors on eBay which are fine.

As for the heatsink, I (being a little elitist) tend only to buy noctua now. The products are well developed and come with everything you could need. Just get one that fits

 

[thecoffeeguy]

There are plenty of cheap molex to sata connectors on eBay which are fine.

As for the heatsink, I (being a little elitist) tend only to buy noctua now. The products are well developed and come with everything you could need. Just get one that fits

cool.
how are they on noise? Looking to make this pretty quiet server build.
I'll check ebay for the molex to sata connectors.

Thanks!

 

[Keljian]

Noctua typically are very quiet, you would set up fan control though.

 

[T_Minus]

I have had great luck with the "StarTech.com" branded power adapters.
You can get them on Amazon too.

 

[thecoffeeguy]

I have had great luck with the "StarTech.com" branded power adapters.
You can get them on Amazon too.

Great. I just ordered 6 of them.

Very cool.

now for the heatsink and fan...heheh