Building my first server, need some guidance

[ridney]

Hi,

I am looking into building an all-in-one server that will be an ESXI host and NAS at the same time. My current setup is a 2x Dell R210 (ESXI hosts) and a QNAP TS-412 NAS though one of my Dell R210 just died after a power trip (not running a UPS) and the reason for a new build. I am planning to consolidate all my servers into one box as I only have a limited budget for one server and retire my old servers.

So I just purchased from ebay a used Supermicro X9SRH-7TF board to match an E5-2670 and will be with me in the next 2 weeks or so. This will be in a 3U short depth case and placed in my wall mounted rack.

Proposed build setup

1. Supermicro X9SRH-7TF (bought)

2. 4 x 850 EVO 256gb ssd as datastores (from existing setup)

3. 4 x 4tb 2.5” drives for NAS storage (planned)

I am also running around 8 VMs at the moment (firewall, windows, mail, wordpress, reverse proxy, voip etc..)

So my few questions:

1. I am currently running all my ssds as single disk datastores. Is it alright to continue with this setup for my new build? The reason why I am using single disks is I have exactly no idea how to configure a raid setup for my datastores. If you can advise me the proper setup and lead me to some guides I would highly appreciate it.


2. I want to make this new box to replace my NAS and I’m looking for something that would about as close to what I experience with QNAP? I really like QNAP and its web based interface but my old cheap NAS is really slow and wanted to upgrade to something faster while maintaining the same or close to its web interface. What are my options for this? I have read about FreeNAS, Nass4Free, OpenMedia Volt, Xpenology but having no experience with one of these so I am having a hard time choosing which one to go for. I am looking for lightweight OS where I don’t need to run a lot of memory just to get up and running. Would appreciate if you can give some advice. Also, what are the consequences of running all my drives for the NAS under the onboard LSI 2308 SAS controller? Is this going to be a problem?

Thanks in advance

 

[Keljian]

If you are looking for a low resources nas distribution then open media vault and xpenology both fit the bill.

I like OMV because it is based on Debian, I am most familiar with that and can fix stuff if things go wrong - it is very lightweight (2 gig install, 2gig memory recommended). Xpenology I have no experience with.

Nas4free and freenas use ZFS, and this will use more memory, with the benefit of having error checking and fixing. ZFS is an interesting beast.

 

[ridney]

Hi Keljian, thanks for the recommendations. I did a bit of reading about openmediavault and the web interface looks just like Qnap's and I'm sure I'll be at home with it right away. It looks like everything I need is there except perhaps a file browser and a torrent downloader. I guess you can add these through add-ons? I am downloading the 2.1 iso right now and will try to play around with it. It's good to know that it requires minimal hardware just what I am looking for.

So yesterday I downloaded Xpenology's latest build and installed it as a VM with a virtual disk and I must say I am impressed with the web interface and I think all the features are working very well plus the add-on packages. Plex and Media sharing was also working smoothly. I was able to stream to my TV and PS4 without any problems. Downloader was fast, I was able to download torrent around 30Mb/s where I could only do with 2Mb/s in my ARM based QNAP. What a huge difference.

I am already leaning into Xpenology as my NAS OS as it looks and functions just like a Synology but will still give OpenMediVault a try this weekend.

 

[ridney]

Nothing much is happening but I wanted to list down the items I am using with this build.

Build’s Name: FileServer
Operating System/ Storage Platform: vSphere Hypervisor 5.5
CPU: Intel Xeon E5 2670
Motherboard: Supermicro X9SRH-7TF
Chassis: PlinkUSA IPC-G3370S
Drives: 4 x Samsung 850 Evo 256gb (datastores)

4 x Seagate 4TB 2.5" HDD (storage)​

RAM: 2 x Samsung 16GB DDR3 ECC

4 x Samsung 8GB DDR3 ECC
2 x Hynix 4GB DDR3 ECC
(Total memory 72 GB)​

Add-in Cards: None
Power Supply: Seagate S12II 520W
Other Bits: PlinkUSA Sata Backplane 8 x 2.5" Bay in 2 x 5.25"

Noctua NH D9DX i4 3U CPU Cooler​

Usage Profile: All-in-one build XPEnology with some VMs

All items have been purchased mostly from ebay and 2.5" drives from Amazon. It will be a bit of wait for these items to arrive from US to Singapore but so far I have received the board and got the PSU which I bought locally.

The PSU is the cheapest Seasonic I could find. I think the 520w 80 Plus Bronze should be fine for my setup.

The motherboard was packed really nice and with so much protection which I am happy about but I also paid quite bit for shipping though ($60) sigh.

It looks a little used though (bought used) hopefully it is just fine and will be working smoothly. First I did was check the jumpers were on their default positions then I clear CMOS using a screw driver and removed the battery while waiting for my processor, heatsink fan to arrive so I can do some testing.

 

[Patrick]

That is certainly better than most used motherboards come packaged. If you do go ZFS and want peace of mind on a ZIL/ SLOG device this is a great one: Dell R730XD Intel DC S3710 MLC 400GB SSD 2.5 6Gb/s SATA III Wrnty AUG 2020

 

[ridney]

I would love to have that ssd but I'm a little over my budget right now though that will be an option I'll be looking forward to in the near future. At least with this build it is good to know that it allows me to have an option to upgrade compared to my current Dell R210 which is very limited.

Alright, Saturday morning here in Singapore and have nothing much to do so I thought I would share a little bit of my decision why I went with a 3U short depth case.

Most people here in Singapore live in a flat and I'm one of them so I don't have the option to have a basement or a garage to dump all my gears to so I had no choice but to wall mount my rack in the living room. During the 2 plus years that I have installed this I have been having a hard time managing fan noise especially with 2 x 1U servers with fans running at 9000 rpm constant. Although I have dampened the noise quite a bit using some acoustic foam but still some faint high pitch noise is still bothering me and sometimes really getting on my nerves. So this time I took the chance to choose a bigger case to avoid having small noisy fans and instead install some bigger quiet ones. Lessons learned.

Here is a photo of my rack. I think it is a 15U 500mm depth rack if I recall correctly.

Because the depth of rack is only 500mm, I'm limited with case choices but luckily saw some short depth ones on ebay. I was going back and forth in choosing between a 2U and 3U and all of it depends on the size of the board that I am going with. ATX pairs with 3U and mATX with 2U. However, there were no mATX mobos out there for LGA2011 and i was already set on getting E5 2670 as it's the cheaper way so the 2U was out and got the 3U instead. If I went with a 2U then I would then have to get an X10 mobo, E5 v3 and could not use my spare DDR3 dimms which would have gotten the total cost a bit higher.

PlinkUSA IPC-G3370S
3U (LCD Display Fan Control) (3x5.25"+7x3.5" HDD Bay) (ATX / Micro-ATX / Mini ITX) (14.96" Deep) Rackmount Chassis IPC-G3370S (PLinkUSA.com / RackBuy.com)

The case was designed for mATX but could accept ATX with some compromises. The third 5.25" bay at the bottom couldn't be used for ATX mobos so I only have 2 x 5.25" bays left. Initially I was fine with having a 3 x 3.5" bay backplane but a further research around the forum, I have read some people using 2.5" drives by shucking them out of their portable cases so it gave me an idea that I could at least have the option of getting 8 x 2.5" bay shoved into the case! So happy with the idea! Now I can get my all-in-one box inside a short depth 3U case that can fit in my rack which would hopefully be so much better at the noise department. It got me excited

So I bought the mobo and cpu and then quickly realized there might be a problem of heatsink fan clashing with internal 3.5" hdds. Did further checking and found it to be just clear off the hdds, what a relief.

Heatsink fan will be over hanging on top of two dimm slots but if I can install the dimms first I think it wouldn't be a problem.

Here are a few photos of the 8 x 2.5" bay

 

[ridney]

Why do I need to flash my onboard lsi 2308 to IT mode? Will it not work with xpenology if I just leave it at IR mode?

E5 2670 just arrived! Well packed by the seller. Now just waiting for heatsink to fire this up.

she's getting ready

Also got the 2 x 2.5 drives and another 2 coming in a few days. Getting them in pairs to avoid import tax.

Update: Benchmark results

Seagate 2.5" 4tb vs WD Green 3.5" 4TB

 

[ridney]

Got the last pair of 2.5" drives today and after registering and testing, they're all going under the knife

Used a sharp thin knife to pry the edge and used a chisel to finish the rest. Just a note of caution to those doing the same, be careful not to put too much force on the knife when prying as the hdd pcb is directly exposed behind the cover and might accidentally scratch it.

It looks like the case can't be used back almost all the clips are broken.

And now the rest of the hdd's.

These drives are so tiny in comparison to 3.5"

That's it for today. Schedule for the case to arrive is tomorrow. Will see

 

[ridney]

Case is here

Accessories

Front panel

Bottom

Back of case

Fan controller and 120mm Fan, it's a Yate Loon DSM so hopefully its whisper quiet

Fan filter

Slight damage to the case but not necessarily a concern

Cover has two screws on each side

Front panel

Bottom of 3 x 5.25" bay has a gap for ATX motherboards so only 2 bays are available

PS2/ATX Power Supply

Motherboard

It looks like it would be really tight

In comparison with my 1U Dell R210

 

[ridney]

Sorry for posting so much pictures, is it too much? hahaha

Motherboard and psu installed but sadly 3.5" drives can't be installed on the 1st HDD cage, there's just no more room for power connectors and sata cables. The heatsink fan will be so close to the drives there's no more space in between. So I will have to use 2.5" drives here

But good news! 2nd HDD cage is just fine! The power cable sits right on top of one of the dimms. Also no room for any GPU cards there but I don't need one anyway.

The case came with no fans at the back. It's a 2 x 60mm fan mounting.

But luckily, I have an extra ram cooler that I no longer use. This was a spare from my desktop that I have kept for roughly 8 years! Yes, that's how old my desktop is and it's still working fine.

It's fits perfectly.

So far this is what it looks like.

Now back to waiting. How I wish refreshing the tracking web page speeds up the process haha

Alright, now for some info of my rack.
Top to bottom
1. Patch panel
2. Cisco 2504 Wireless Controller
3. ISP ONT - currently have 2 x 1gbps/500mbps down/up connection
4. Cisco SG200 26 Port Switch
5. Cisco UC520 - It is my POE switch for phones and configured as SIP gateway
5a. Power Injectors for Access Points. I'm running 4 x Cisco 3602 with AC modules.
6. Cisco 2509 Access Server (for troubleshooting only)
7. Dell R210 - Currently dead so the reason for this new build. Removed from rack.
8. Dell R210 - Will be used as a backup server but powered down
9. Cisco 3825 Router - It is my gateway to the internet. The other is from my Dell R210 running Sophos. This is also where I host my PVDMs for the Call Manager.
10. QNAP TS-412 - To be removed from rack and will be used to backup my personal data.

And this is my planned arrangement when this server is done.

Looking to add a 2U rack fan at the bottom but couldn't find one that can support 230v.
If I can add a 10GB switch in the future then the 2509 needs to be out. Also the SG200 switch needs to be replaced with a stackable one.

 

[Taco]

Sorry for posting so much pictures, is it too much? hahaha

Pictures are great, don't worry.
I quite like your hanging rack as a space saving solution by the way.

 

[ridney]

Thanks Taco. I like it too but I wish it was a little bit more deeper though

 

[Patrick]

Great pictures and write up.

 

[ridney]

Thanks Patrick. Great site btw

 

[ridney]

I was thinking of getting an Intel X540-T2 or T1 card and running a direct cable to this server. My cables are only Cat6 so not sure if it would work though distance from Desktop to Server is around 30m or less.

This is the card
Intel X540-T2 10G Dual RJ45 Ports PCI-Express Ethernet Converged Network Adapter

Any advise?

 

[Keljian]

Don’t! Get a mellanox connectx-2 or 3 - intel gigabit needs intel sfp+ lamps

 

[ridney]

Thanks. My board is already 10GBase-T. Any reason why you suggested SFP+?

 

[ttabbal]

Thanks. My board is already 10GBase-T. Any reason why you suggested SFP+?

Probably thought you were using SFP+/Fiber as that's pretty common around here. If you already have a 10GBase-T port, give it a go. Can't promise anything over <cat6A, but with normal cat6 on a short run, it might work. If not it will probably fall back to 1Gb.

 

[Keljian]

Probably thought you were using SFP+/Fiber as that's pretty common around here. If you already have a 10GBase-T port, give it a go. Can't promise anything over <cat6A, but with normal cat6 on a short run, it might work. If not it will probably fall back to 1Gb.

Yep! Fibre is typically cheaper, which is why I assumed. ( it is also lower latency)

 

[ridney]

Okay thanks for the recommendations, it's a go. Just wondering what's the best to set this up on my desktop?

1. Dual nics on my desktop, one connected to switch and the other directly connected to the server? Desktop is Windows 10 so I guess putting the server ip in the host file would do the trick? So every time I want to access files on the server it would use the direct connection and when accessing the rest of the network and internet it will use the other nic?

2. Single 10GB nic to the server, and use a VM (Sophos) to route me to the internet and the rest of the network?

With option 1, I will have to run a new cable as I have exhausted all the ports I had near my desktop.