FINAL Bachelor Build - Xeon D vSAN Cluster

[IamSpartacus]

So I picked up a couple of cheap plastic ducts to get the front intake air directly to the heatsink. Also installed some Notcua 60mm PWM fans (though I've since moved the fans to behind the heatsink pulling air off of it).

Temps are 51C idle and 77C with all 8 cores 100% though the fans (set to "optimal") only ramp up to about 80-85% at this temp. With the fans at 100% idle temps are 42C with load temps of 73C. Considering how much less noise the fans make with the fans set to "optimal" as opposed to "full speed" I can live with the 51C/77C temps. Also it should be noted that SuperMicro lists the High Critical Threshold at 104C so I think anything under 90C should be fine.

 

[miraculix]

What about the heatsink?

Me personally, I used a very small drop of Loctite liquid high temp epoxy on each of the HS screw heads and mounted a 60mm Noctua on top per this post (you'll also see the original blog post that got me thinking about this). The silicone anchors that come with the fan fit perfectly into the cupped/rimmed screw heads (which likely kept the epoxy in place while setting). I agree in general with the comments about not using epoxy but it worked great nonetheless.

It's hard to tell for sure from the pics but it looks like your heatsinks are taller. Due to that, or if you're mounting the fan to blow through the HS fins from the side instead of the top, it may be better to come up with another way to fasten the fan (or stick with ducting only)

EDIT: your heatsinks are definitely taller and I doubt the method I used will be as simple/straightforward in your situation. If you can get a ducting solution that brings the air *all* the way to the HS securely and through the fins, that would probably fix the remaining excess bit of temperature you're seeing. Maybe someone can suggest good material to use that won't generate ESD. Once I get back home this weekend I can also take pics of the shroud that came with my 1U systems if that helps.

 

[IamSpartacus]

Well it appears the X10SDV-4C-7TP4F I received on Wednesday night is DOA. Luckily WiredZone is awesome and is going to overnight me a new board that I will get by Monday the latest.

 

[canta]

Sit).

....

if you are willing to share, where do you get plastic duck?

 

[IamSpartacus]

if you are willing to share, where do you get plastic duck?

Here you go @canta.

 

[canta]

thanks!!

 

[IamSpartacus]

While I wait on my replacement D-1518 board (SM hasn't had the stock to overnight it until today it appears), I was able to finish the initial wiring for my rack last night.

 

[Evan]

@IamSpartacus thats some nice wiring !

 

[whitey]

OCD much? lol j/k, mine's 'abt' as clean as that these days. Pic of other side please? :-D

 

[IamSpartacus]

OCD much? lol j/k, mine's 'abt' as clean as that these days. Pic of other side please? :-D

Nothing is connected in the back yet. Waiting on the replacement board and then I'll post some pics of the final product .

 

[IamSpartacus]

After finally receiving my replacement motherboard and spending some time configuring (and troubleshooting) my vSAN cluster setup I'm happy to say that I'm up and running. I've yet to move any of my "production" VMs to the vSAN datastore yet as I had some issues with it going down that I believe I've resolved but want to be sure before I migrate things. Some further testing over the next week should yield me more confidence in that respect.

What I'm contemplating now is how best to make use of my 10Gb networking on the vSAN cluster. Currently the single 10Gb NIC in each host is assigned to a vDS called vDS-Storage which has my 'vMotion' and 'vSAN' vmkernels. I then have the two 1Gb NICs assigned to a vDS called vDS-VM_Manage which has my 'Management' and 'VM Network' vmkernels. However I don't foresee myself using vMotion very often so it seems like a bit of a waste to have the 10Gb link only in use by vSAN. Thus I'm considering creating a 'VM Network 10Gb' vmkernel on vDS-Storage so that I can assign it to any VM that might need more bandwidth.

 

[miraculix]

For a home lab where the ESXi hosts only have 2x10GE (and some quantity of 1GE ports like our X10SDVs)... yeah I would simply put all vmks on the 10GE and not even use the 1GEs except for exceptions including backup/backout if you screw up your networking.

EDIT: almost forgot... also be sure to enable NIOC. It's not foolproof but it has good thresholds for different traffic types including VSAN. The VMTN link also mentions decent NIC failover strategies per vmk / traffic type.

 

[msvirtualguy]

Any updates to he build..new pics etc? How are the temps?

Very close to ordering my first node based on the X10SDV-7TP4F.

 

[IamSpartacus]

Any updates to he build..new pics etc? How are the temps?

Very close to ordering my first node based on the X10SDV-7TP4F.

Sorry for the lack of updates. Been doing a lot of testing, configuring of my setup over the past 2 weeks.

vSAN cluster is fully up and operational and running very nicely. Got vCenter configured and running on the vSAN datastore with vDS. That part is all working well after some minor issues I had at the beginning.

The temps have been good (IMO) considering I have no climate control in the room and the rack is fully enclosed with 2 quiet exhaust fans. All 3 Xeon D's are idling in the 55-58C range and load temps are in the mid 70's.

I've been spending the past week testing/configuring my new Media Server setup. I used to run all my media applications (Plex, Sonarr, CP, NZBGet, Hydra, PlexPy, PlexRequests, Muximux, etc.) in Dockers on UnRAID. I've since migrated my dockers over to Ubuntu Server 16.04. I've also configured MergeRFS on the server to pool a set of NFS mounts (from both my local UnRAID array and my remote (VPN connected) UnRAID array) to my bulk media. So now I can take my local bulk media array down and still have Plex be up as it can stream across the VPN connected NFS shares.

I am however dealing with one issue that I'm having trouble isolating. Ever since I moved my Dockers over to Ubuntu Server on the vSAN datastore and offloaded my 'Cache' and 'Media' Plex folders to NFS shares instead of locally in the config folder, I'm getting very slow background transcoding within Plex. My server used to convert files very fast and now it's like a quarter of the speed. I'm thinking it could possibly be a a write speed limitation of MergeRFS. I'm still working on isolating this as it's a major annoyance since both myself and my remote users do a lot of mobile syncing.

 

[IamSpartacus]

Alright so I determined the bottleneck for my transcoding and it is MergeRFS. If I offload my Plex 'Cache' directory to a simple NFS share, transcoding speeds increase 5-6x to what I'm used to. The moment I change the directory to a MergeRFS pool, it slows again.

Back to brainstorming a way to offload my Plex cache to a share but still keep redundancy...

 

[TuxDude]

If its just cache, why have it on NFS at all? Just use a bit of local storage space on the same box running the docker containers for the cache.

 

[IamSpartacus]

If its just cache, why have it on NFS at all? Just use a bit of local storage space on the same box running the docker containers for the cache.

It's not cache in the traditional sense. The Plex 'Cache' directory is where all cached images for Plex apps are stored along with all temporary transcoded video files (on the fly transcoding, optimized versions and mobile sync transcodes). If a user decided to sync an entire season of Game of Thrones at original quality (5GB per file), we're talking about 50GB right there just sitting in the cache directory until those files are fully downloaded to their mobile device.

Suffice to say the cache directory can become several hundred GB at times. I also need this directory to be redundant as Plex can not function without it. So if this directory is down, Plex is unusable.

 

[TuxDude]

So its a large cache - but from a redundancy point of view there's still no reason it can't be local disk on the same server as is running the Plex docker. That machine is still the single point of failure for Plex, whether the cache is there or not. Unless I missed something somewhere and you also have a way to move the docker container to another node - in which case a loss of the contents of the cache directory (assuming the new node hosting the container also has a local space for cache) would mean restarting some transcoding jobs if any happened to be running when the failover took place. A background rsync job to keep the cache directories in sync across both hosting nodes would minimize the amount of content that would need to be re-created too, maybe run by cron every hour or so.

Other options you could consider....
* AUFS - kernel based and would give much better performance than fuse-based MergerFS. But can be a pain to get installed if your distro doesn't include the kernel patches it requires.
* MD-raid - you can software-raid remote block devices eg. iSCSI.
* DRBD - mirroring at the block-layer between two nodes.
* Possibly some distributed filesystems could handle the replication for you eg. GlusterFS.

And on a semi-related note... only 5GB per GoT episode?

 

[Continuum]

Just a shot in the dark, have you tried the direct_io mount option for MergerFS? Linuxserver.io reports that the option increases performance.
Debian, MergerFS, SnapRAID and Docker – The Perfect Home Media Server 2016 – Linuxserver.io

 

[IamSpartacus]

So its a large cache - but from a redundancy point of view there's still no reason it can't be local disk on the same server as is running the Plex docker. That machine is still the single point of failure for Plex, whether the cache is there or not. Unless I missed something somewhere and you also have a way to move the docker container to another node - in which case a loss of the contents of the cache directory (assuming the new node hosting the container also has a local space for cache) would mean restarting some transcoding jobs if any happened to be running when the failover took place. A background rsync job to keep the cache directories in sync across both hosting nodes would minimize the amount of content that would need to be re-created too, maybe run by cron every hour or so.

Other options you could consider....
* AUFS - kernel based and would give much better performance than fuse-based MergerFS. But can be a pain to get installed if your distro doesn't include the kernel patches it requires.
* MD-raid - you can software-raid remote block devices eg. iSCSI.
* DRBD - mirroring at the block-layer between two nodes.
* Possibly some distributed filesystems could handle the replication for you eg. GlusterFS.

And on a semi-related note... only 5GB per GoT episode?

Yes you are right, I can just add a second vmdk file to the Ubuntu Server running my dockers and put the cache there. Just that having to vMotion a potential 400+GB file when a host goes down or when I need to do maintenance is not ideal.

I hadn't considered some of those other options because MergeRFS was just so simple to setup but if it's not meeting my needs I may have to explore these other optoins.

As for GoT, yes I'm one of those people who only stores remuxed rips. So 1080p TV episodes are 4-6GB, and movies are 8-20GB. Gotta be more efficient when you have 2000 movies and 15,000 TV episodes.

Just a shot in the dark, have you tried the direct_io mount option for MergerFS? Linuxserver.io reports that the option increases performance.
Debian, MergerFS, SnapRAID and Docker – The Perfect Home Media Server 2016 – Linuxserver.io

Oh interesting, I haven't tested that option yet. I'll give it a shot and see how much it hurts read speeds.


EDIT: Wow, that makes a HUGE difference.