FINAL Bachelor Build - Xeon D vSAN Cluster

[chief_j]

@IamSpartacus I think your build pushed me over the edge to move to xeon-d finally. I've been teetering on if I should hold out and get something more powerful, or just take the plunge. The biggest thing I was worried about was plex on-the-fly transcoding for internet users.

I think I saw your post on the ES-16XG switch as well, and I have the Unifi 16XG and can say that I've had very similar problems with DAC's. Finally had to go the transceiver route to get everything working into that switch.

 

[IamSpartacus]

@IamSpartacus I think your build pushed me over the edge to move to xeon-d finally. I've been teetering on if I should hold out and get something more powerful, or just take the plunge. The biggest thing I was worried about was plex on-the-fly transcoding for internet users.

I think I saw your post on the ES-16XG switch as well, and I have the Unifi 16XG and can say that I've had very similar problems with DAC's. Finally had to go the transceiver route to get everything working into that switch.

@chief_j You're going to be very happy with the Xeon D platform. I haven't regretted it for a moment since I went that route and I continue to put more and more workload on my servers.

Which Xeon D processor are you looking at?

As for the ES-16-XG, yes my experience with it left a bitter taste in my mouth. Even if i had all optical transceivers, many users still have issues with the 10Gb Copper NICs as well. Networking gear is the last thing i want to spend my time with troubleshooting so I ponied up the dough for a reliable Cisco SG350XG-24F.

 

[PigLover]

...I ponied up the dough for a reliable Cisco SG350XG-24F.

Very good switch, though a bit pricey.

Curious - do you have power measurements on the switch?

 

[SGN]

Do you use DAC or optics? If optics, do you have any thoughts or comments regarding SFP+ compatibility with different Supermicro X10SDV boards? What SFP+ do you use?

 

[IamSpartacus]

Very good switch, though a bit pricey.

Curious - do you have power measurements on the switch?

Definitely pricey but if it lasts me 10 years it'll be worth it for sure.

I don't have any recent power numbers unfortunately. When I first got it I was using 10 DACs and power usage was between 30 - 35w. Now I'm using 8 optical transceivers and 2 DACs.

 

[IamSpartacus]

Do you use DAC or optics? If optics, do you have any thoughts or comments regarding SFP+ compatibility with different Supermicro X10SDV boards? What SFP+ do you use?

I'm using fiber modules from FS.com. I asked for half to be coded for Intel and the other half for Cisco. They work flawlessly.

 

[Patrick]

@IamSpartacus I have to admit, I have not read every reply, but I like that the "Final Bachelor Build" has now lasted almost 13 months

 

[IamSpartacus]

@IamSpartacus I have to admit, I have not read every reply, but I like that the "Final Bachelor Build" has now lasted almost 13 months

@Patrick While I am no longer a bachelor as of December 30th, the build will live on as long as I can hold on to that nostalgia .

 

[K D]

This was the thread that first got me on to these forums when I was researching on Xeon D boards.

 

[acquacow]

The only thing holding me back from doing Xeon D builds is the cost of DDR4. Supposedly on the latest boards you can use DDR3, but I haven't seen anyone actually do it. There's so much cheap used DDR3 on ebay/etc, the price savings there alone easily offsets the extra power used by higher TDP processors.

 

[chief_j]

@chief_j You're going to be very happy with the Xeon D platform. I haven't regretted it for a moment since I went that route and I continue to put more and more workload on my servers.

Which Xeon D processor are you looking at?

As for the ES-16-XG, yes my experience with it left a bitter taste in my mouth. Even if i had all optical transceivers, many users still have issues with the 10Gb Copper NICs as well. Networking gear is the last thing i want to spend my time with troubleshooting so I ponied up the dough for a reliable Cisco SG350XG-24F.

D-1537's on the supermicro x10sdv-7tp4f, and pretty much for the exact same reasons you went with them. Was there that much performance difference between the 1537 & 1541 on the D1541D4U-208R board you purchased on your transcodes?

 

[IamSpartacus]

D-1537's on the supermicro x10sdv-7tp4f, and pretty much for the exact same reasons you went with them. Was there that much performance difference between the 1537 & 1541 on the D1541D4U-208R board you purchased on your transcodes?

Live transcode wise...no, not that I can measure. But I do notice my background transcodes (a lot of my users do mobile syncing and I use optimized versions) go a good deal faster.

 

[Dan K.]

Curious...why do the optics need to be coded? I picked up some f5 optics on eBay for my es-16-xg and haven't had issues...that I know of.

 

[chief_j]

Curious...why do the optics need to be coded? I picked up some f5 optics on eBay for my es-16-xg and haven't had issues...that I know of.

I've had a complete mixed bag with optics & dac's on the us-16-xg, and it has seemed like a mixed bag for others. They just released an alpha firmware that increases dac compatibility as well, so i'll be installing that this weekend.

 

[Dan K.]

What issues would improper coding cause? Just wondering if there may be some unapparent issue with mine that I haven't discovered yet

 

[IamSpartacus]

Curious...why do the optics need to be coded? I picked up some f5 optics on eBay for my es-16-xg and haven't had issues...that I know of.

FS uses generic modules that they code for each purchasers individual hardware needs.

 

[Dan K.]

Ahaa that's what the customized transceivers are!

 

[chief_j]

This might be slightly off topic, but what has you experience with vsan on the xeon-d's been like?

 

[Evan]

This might be slightly off topic, but what has you experience with vsan on the xeon-d's been like?

For play or home use ? Or actual workload ?
Any kind of real workload limited by memory mostly and then network ports (really need 4 x 10g to do it right) and then lack of slots for io adapters.

That's why I have never tried a production system and also we have great E5 prices in 2u that makes lots more sense for vSAN

 

[IamSpartacus]

I guess I may not have mentioned it in this thread and being that the thread title still mentions vSAN I can see the confusion even if I had. But anyhow, I'm no longer running vSAN in this cluster. I enjoyed running it, the performance was solid (nothing amazing though) and I can see how useful it would be in production. But for my home server where I'm constantly tinkering, it just wasn't flexible at all. vSAN REALLY doesn't like when it's nodes are down for long and it just causes headaches. So while I'll be running it in production at work this upcoming summer, it just wasn't ideal for home use.

So since I was using 1 x Hitachi HUSSL 400GB SAS SSD and 1 x Intel S3500 800GB SSD in each vSAN node, I converged them all into a single node (my D-1508 board) and created a FreeNAS SAN that I present to my ESXi cluster via NFS. I created two zpools with the 8 SSDs. The Hitachi HUSSLs are my "tier01" pool for my higher intensive write applications/services and the S3500's are my "tier02" pool for less important and test VMs. I had done this all in Corral but have since migrated my pools back to 9.10.2 because I didn't want to worry about using a platform that would not be supported going forward.