Hardware for Proxmox Node/File Server

[skelleton]

I am upgrading my server and my current File Server (Debian 9, 20 Disks all luks encrypted and zfs on top)
Will be replaced by something new. However im not quite sure what to buy without losing performance.

I currently have a Xeon E3-1235 in the file server. I am thinking about replacing it with a Xeon-D.
But I have not seen Benchmarks, that compare luks performance of a sandy bridge e3 with with a Xeon-D 1518.
Would the E3 be faster in encryption?

If I need one of the 8 core Xeon-D to not lose performance, its probably smarter to buy a dual socket. From what I see the price for a dual socket 2011-3 Board and a E5 2609 or 2620v4 I figure i get something more future proof for the same price there.

Any suggestions what if I which platform would be better to go with.

The primary function of the server would be file server. For now it will only be on Proxmox as a Quorum Node.

 

[ullbeking]

Hey skelleton, did you make any progress? I'm trying to do pretty much the exact the same thing, ie, build a single server that serves files and is a virt host, on Debian with ZoL. The X10SDV boards look great, however I'm tempted to go for something bigger, with expansion possibilities, like an X11 board with E5-1200 v3 in ATX, and put it in a roomy but nice case. Easier to keep cool that way too

 

[skelleton]

I am going Sandy Bridge-E for now. Found a pretty decent offer where I pay around 800-900 for everything after import fees(2x e5-2620 128gb RAM 4x100gb ssd and 2x lsi2008, and a bunch of parts I need to change on the case.).

I hope that all the parts will have arrived by the end of the month. So I will know more then. But given the amount of RAM I will probably run a few machines on it as well.

I will use a whole lot more power, but it was also a lot cheaper to build, I have more expansion options and it should not matter to much that its an older chip. It's still going to be mostly a file server after all.

 

[Patrick]

Pictures!

 

[skelleton]

So it took a while but between gamescom and waiting for all the parts to arrive I only got to assembling everything today.

Currently I just updated the Bios/and IPMI firmware and installed Proxmox. I may have already messed up the Proxmox install though. Won't boot atm. I suspect, it may not have liked that IT overwrote the partition table of the 2 ssds that were not part of the boot zpool. But we will see. in doubt its no great loss, since I didn't actually do anything within Proxmox yet.

Hardware so far:
- SuperMicro X9DRI-F
- 2x Xeon E5-2620
- 4x HGST 100GB SSD
- 2x LSI 2008
- 1x Mellanox Connect-x 2

I am still waiting for:
- 2x Xeon E5 2696v2
- 2x SM Active 2u cooler

And the following disks will go into the chassis once everything is racked up and prepared:
- 11x Seagate Archive 8TB
- 10x Seagate IronWolf
- 2x Sandisk UltraII 512GB

I noticed one minor issue:
Does anyone have the pinout for the 16 pin front panel connector for the SM SC847.
I have a 20 pin front panel connector on the board and i am wondering if they are compatible.

Edit: I am aware that I sill need to do some cable management.
And does anone know how many amperes I can safely put on the fan connectors of the motherboard. I can't find that in the manual.

 

[ullbeking]

Currently I just updated the Bios/and IPMI firmware and installed Proxmox. I may have already messed up the Proxmox install though. Won't boot atm. I suspect, it may not have liked that IT overwrote the partition table of the 2 ssds that were not part of the boot zpool. But we will see. in doubt its no great loss, since I didn't actually do anything within Proxmox yet.

Hardware so far:
- SuperMicro X9DRI-F
- 2x Xeon E5-2620
- 4x HGST 100GB SSD
- 2x LSI 2008
- 1x Mellanox Connect-x 2

I am still waiting for:
- 2x Xeon E5 2696v2
- 2x SM Active 2u cooler

And the following disks will go into the chassis once everything is racked up and prepared:
- 11x Seagate Archive 8TB
- 10x Seagate IronWolf
- 2x Sandisk UltraII 512GB

I noticed one minor issue:
Does anyone have the pinout for the 16 pin front panel connector for the SM SC847.
I have a 20 pin front panel connector on the board and i am wondering if they are compatible.

Edit: I am aware that I sill need to do some cable management.
And does anone know how many amperes I can safely put on the fan connectors of the motherboard. I can't find that in the manual.

Hey @skelleton, I'm really curious as to how you got on with this..? How have you configured everything in order to get both NAS/file serving and virtualization in the one box?

 

[K D]

Match pin1 of your chassis cable (pink stripe)with pin 1 of the header on the motherboard (white marking). The extra pin 19 and 20 are not needed.

 

[skelleton]

@K D Thank you figured that out last Sunday when I found the pinout of the SM control panel on some obscure blog.
Though SM should put that into the manual.

@ullbeking Everything is mostly configured, but I had a crazy week at work so not much has happened with it.
Here is the status:
- It is running Proxmox
- It is running samba and serving my files from the drives(encrypted) and Zpools that were already in my previous file server.
- I seem to have gotten a slight performance boost in some situations I suspect that is due to 128GB RAM vs 16 before.
- NFS is also running fine.
- for some reason I will have to redo my group permissions on the Linux side. It does not resolve the group ID properly anymore (even though this Proxmox node is a member server in my ad and group ids are stored in the AD)
- I have not had any problems when running Samba + NFS + Proxmox together on bare metal
- I have not done much with containers and VMs, but there were no issues on a quick test
- For the Samba setup I pretty much follow my blog post from a few years ago, except the installing Debian part(let me know if you are interested in my shameless self promotion)

I hope that I will get to the following this weekend:
- getting openvswitch running (never used it so far)
- clustering all 3 Proxmox nodes that I have now.
- fixing some of the file server scripts/ cron jobs that I ported over (I got them into salt stack), I seem to have messed up some of them.

 

[skelleton]

@ullbeking
So I have Proxmox as part of a 3 node cluster now and Samba running directly from the host. The CPUs are upgraded to 2x Xeon E5 2696v2.

I can't see any down sides so far. File Server performance seems to be slightly higher than with Xeon E3 1235 under certain conditions. I am guessing due to more RAM for ZFS. I have not run many machines from this machine, but I am seeing no negative impact from doing so.

I ran a Windows VM with all cores assigned and did a few benchmarks on it and there was no impact on the video streams the the host was serving via Samba.