Low-power Ceph/ProxMox node

[niekbergboer]

Build’s Name: Fessup *)
Operating System/ Storage Platform: ProxMox 4.2 (Linux 4.4.6, Debian Jessie derived)
CPU: Core i3-6100T
Motherboard: SuperMicro X11SSL-F
Chassis: SuperMicro SuperChassis 510T-203B (2 x 2.5" SATA3 hot-swap)
Drives: 2 x Intel DC3610 (200 GB)
RAM: 2 x 16GB DDR4 ECC UDIMM.
Add-in Cards: Intel X520-DA1 (10 Gbit SFP+)
Power Supply: (SuperMicro, comes with the chassis) 200W.
Other Bits: SuperMicro-specific 1U passive cooler (on MoBo), 3x40mm PWM fans (in case).

Usage Profile: This system is one of my three converged ProxMox / Ceph nodes.

Other information: Intel is not marketing the Core i3s as such, so let me say this very quietly: A Core i3 is pretty much a dual-core Xeon. It has ECC support ,VT-d support. The only two things, compared the E3v5s, that is doesn't have, it TSX and Turbo Boost. This little system runs at 22W idle, which is amazing, given that there is quite a power-hungry X520 in there.

*) Fessup: This is straight from the Helliconia trilogy by Brian Aldiss. Fessups are one of the two kinds of "spirits of the earth" that give living beings important information. Accordingly, my next node will be named after the other kind: Gossie.

 

[Patrick]

Did you install Proxmox VE 4.2 directly or did you install it atop Debian Jessie?

What are the other nodes?

 

[PigLover]

Very nice. 22W idle for a capable Ceph/Proxmox hyperconverved node w/10Gbe is impressive. Using the I3 "T" chip gets you more performance than an equiv. Xeon-D build (2C/4T D1508) and should come in more than $100 less for MB/CPU/10Gbe than X10SDV-2C-TP4F / X10SDV-2C-TLN2F. Assuming you need 3-5 of these babies to be useful for Ceph that adds up fast.

Any power measurements under "typical" loads rather than idle?

 

[niekbergboer]

@Patrick I installed Proxmox VE 4.2 directly (I wanted an as much out-of-the-box install experience as possible, since I'm now treating the hardware itself as a commodity). The other nodes are "Batalix" (a Xeon E3-1265Lv3 on a SuperMicro X10SAE which, in addition to being a member of the Proxmox/Ceph crowd, runs my main ZFS storage and does NFS/SMB), and "NUC". "NUC" is a (surprise!) NUC (Skylake i3 6100U) with an internal NVMe drive and an external SATA/USB3 storage.

The NUC is very non-optimal. No ECC, but most importantly, no proper storage housing and no expansion possibilities for 10GBe. I'm looking to replace that NUC by something fairly similar to Fessup.

I tried getting a X10SDV-2C-TP4F, but it's next to impossible to get shipped to Switzerland, and then only at insane prices. The Fessup setup has a clear (and by the time I need it, cheap) upgrade path: I can just plunk in a quad-core E3v5 at that point, and I can still expand RAM to 64GB.

 

[niekbergboer]

@PigLover : Here's about 12 hours in the live of Fessup, through my Ubiquiti mFi setup:

 

[niekbergboer]

Ooooh, Fessup's sibling ("Gossie") has been ordered, so I'm looking forward to having my full-10-gbe Proxmox cluster very soon now :-D

 

[Scott_Paladin]

I'm actually a little curious as to how you provisioned your drives for this node. You've got two 200GB. Is one boot/LVM and the other a Ceph OSD? ZFS Mirror?

 

[niekbergboer]

Proxmox really wants to use full drives for OSDs, and although I used to play complex games with command-line utils to get it to use partitions for OSDs, I ended up just buying a SATA DOM which runs the OS. That SATA DOM is not redundant, but that is fine, since losing a node OS is not problem: I'm now N+1 / N+2 on the node level; if the DOM breaks, I'll get a new one, rsync the OS back onto it from backup, boot from a live USB to reinstall GRUB, and off I go.

 

[PigLover]

@niekbergboer - would you mind giving an update? Are you still happy with the Low Power Ceph node? Anything you'd change or do different?

I'm still impressed by the power consumption, BTW!

 

[niekbergboer]

@PigLover Sure.

I now have my three nodes. Two of them (Fessup and Gossie) have the specifications of the first post, with the difference that both now have a SuperMicro SATA-DOM as OS drive, and two 300GB Intel S3500s as OSDs. They average at 22-26W (with some short peaks at 30-35W), depending on the number of VMs that they host.

The third node, as well as main ZFS storage and NFS host, is Batalix. That's a Xeon E3-1265Lv3 on a SuperMicro X10SAE in a Norco RPC-3216 (16x 3.5"/2.5" SAS2/SATA3 hotswap), using the 8 on-board SATA ports as well as a cross-flashed IBM M1015 (LSI2008 IT mode). The main ZFS pool runs of of the LSI2008 and consists of 8 3GB WD Reds in RAID-Z2. The system itself runs on two 1 TB Kingston KC-400s in RAID1, and those also contain the main SSD ZFS pool in MIRROR mode. It also has two 300GB Intel S3500s as OSDs. It has the same Intel X520-DA1 NIC as the two other nodes. Batalix usually spins down its HDDs, but even then it's quite a bit more power hungry than the two other nodes, running at ~57W idle.

By now, all that runs Proxmox VE 4.4.

All of the above is tied together by a TP-Link T1700G-28TQ, all DACs. This switch runs at ~15W.

I'm generally happy; running a 3-node redundant virtualization solution at ~120W is not bad. Now that I changed the Ethernet frame size to over 4 kb (didn't try 8kb yet), throughput is nice: Big files over NFS from the SSD pool run at SATA speed (~550 MByte/s). Big files off CephFS seem to run at ~670 Mbyte/s. At peak throughput, they hit the 1.1 Gbyte/s (10 Gbe) limit though! :-D.

The main reason for me to have this setup is redundancy and the reliability that comes with that. I work for a large bay-area firm in Zurich, and I became very spoiled with well-working redundant systems. I wanted that at home as well. It's just very nice to migrate all VMs off a host to another with a single click if I want to reboot that host, or work on it for some other reason. The reason for my setup is not processing power: my setup has only 8 real cores between all three servers, but that is fine, since I am not CPU bound at all.

What I'd do different? Not so much on the Proxmox VE side. On the ZFS side, I might have gone for fewer larger drives in order to get rid of the LSI2008, to keep idle power down. OTOH, Power is cheap here: 1 W 24/7 will set me back just ~CHF 2 (also ~USD 2) per year; that's not worth investing in lower-power gear for.

 

[PigLover]

Very nice, very impressive small build. Thanks for sharing.

Sent from my SM-G925V using Tapatalk