High end cluster

[zunder1990]

Can someone look over this to make sure that I am not on the crazy train. I am looking to spec a cluster for work based on ovirt and gluster. The plan is to use the Gluster auto tiering, the SSD level would be a pair of systems with replicated gluster brick. The HDD level would be 4 systems what would be distributed replicated. For the network each system has a dual port 25gb port and be connected to the 100gb switch with a 4to1 breakout cable.
Storage
Hot
E3-1220 v5 4core
16gb ram
128gb ssd OS
6.5 TB with 8x1.2tb in raid6 SSD(intel S3610)
9361-8i card card
MCX4121A-ACAT NIC

Cold
2x E5-2603 6core
32gb ram
2x 128gb ssd OS
54.6 TB with 12x6tb in raid6
9361-8i card card
MCX4121A-ACAT NIC

Compute
RAX XS8-2260V3-10G
2x 10core E5-2630 v4
512gb ram
2x 500gb boot drives
MCX4121A-ACAT NIC

Networking
Mellanox SN2100
16x 100gb ports

Total cost looks to be about $120k

 

[Patrick]

Zach, on the road right now. I will check this over later this afternoon and provide comments.

 

[TuxDude]

Maybe there's something I'm missing, since my experience with gluster is quite limited, but is there a reason you are putting your gluster bricks on top of raid-6 storage? And is there a reason you are using physically separate nodes for the different tiers of disk? It would seem to me that a better configuration for the storage side would be all identical nodes, each having a mix of SSD and HDDs, and no per-host RAID at all just using physical drives as gluster bricks. So to get approximately the same storage, say 6 nodes total, but each having 3 of the 1.2TB SSDs, and 8 of the HDDs - which would add up to 18x SSDs (2 more than your original config), and 48x HDDs (identical to original config).

Also keep in mind that ovirt is going to require you to use 'replica-3' mode to protect against split-brain - which is part of the reason for my above suggestion about all identical nodes. You can use an arbiter volume so that you still only need 2 copies of the data instead of 3 full copies, but it will still need at least 3 nodes.

 

[zunder1990]

Maybe there's something I'm missing, since my experience with gluster is quite limited, but is there a reason you are putting your gluster bricks on top of raid-6 storage? And is there a reason you are using physically separate nodes for the different tiers of disk? It would seem to me that a better configuration for the storage side would be all identical nodes, each having a mix of SSD and HDDs, and no per-host RAID at all just using physical drives as gluster bricks. So to get approximately the same storage, say 6 nodes total, but each having 3 of the 1.2TB SSDs, and 8 of the HDDs - which would add up to 18x SSDs (2 more than your original config), and 48x HDDs (identical to original config).

The use of the raid is to protect from a single failed drive, gluster is to protect from a whole node offline and spread load over many nodes.

Also keep in mind that ovirt is going to require you to use 'replica-3' mode to protect against split-brain - which is part of the reason for my above suggestion about all identical nodes. You can use an arbiter volume so that you still only need 2 copies of the data instead of 3 full copies, but it will still need at least 3 nodes.

Yes you are right about that, I forgot about that.

 

[Patrick]

Zach,

I sent you a PM. I just helped a not too dissimilar system and have some ideas for you.

Regards,
Patrick

 

[CookiesLikeWhoa]

Why the dual processors for the cold storage? Seems like you could get away with something much more efficient for cold storage like a E3 or even simpler really.

 

[PigLover]

Why the dual processors for the cold storage? Seems like you could get away with something much more efficient for cold storage like a E3 or even simpler really.

I was going to ask this too - but held back since it's pretty low-end dual cpu setup. I've never understood the desire to have dual CPU in a storage server at all - especially one designated for cold storage with a high performance storage server sitting in the same rack.

But...as I was looking at it is realized that a different server would only save a $couple of $hundred on a fairly large build - perhaps the cost of one or two disk drive in the server. In context, not a big deal.

Sent from my SM-G925V using Tapatalk

 

[Patrick]

But...as I was looking at it is realized that a different server would only save a $couple of $hundred on a fairly large build - perhaps the cost of one or two disk drive in the server. In context, not a big deal.

The typical reason you move to dual CPU is to get more RAM or PCIe slots. You also see a lot of storage servers with a single low power CPU using dual socket motherboards.

We should have a cool 100-120TB cold storage platform arriving in the lab later this week. Operationally should be close to 1A lower power consumption than the spec'd dual CPU platform even with CPUs at idle.

@zunder1990 I sent you my contact info. Happy to setup a system or two in our lab. We just helped a company that was building a hyperconverged all-flash cluster. They got way more storage and compute than you spec'd at a lower cost and are actually going to host the cluster in the rack next to ours for their customers. They started with specs similar to what you have above so I think jumping on a machine or two may help you decide what to get.

 

[CookiesLikeWhoa]

I was going to ask this too - but held back since it's pretty low-end dual cpu setup. I've never understood the desire to have dual CPU in a storage server at all - especially one designated for cold storage with a high performance storage server sitting in the same rack.

But...as I was looking at it is realized that a different server would only save a $couple of $hundred on a fairly large build - perhaps the cost of one or two disk drive in the server. In context, not a big deal.

Sent from my SM-G925V using Tapatalk

There's also the energy saved. Dual CPU systems are hot and hungry. Though, that may really be an after thought too.

Perhaps to repurpose it later... My only concern would be those CPU's though; I can't think of many applications that wouldn't be better served by almost any other CPU.

Networking also seems way overkill considering the main storage solution but does provide huge room for growth later.

 

[Patrick]

@CookiesLikeWhoa those are very popular CPUs for this type of application. But two seems like overkill for one SAS controller and NIC. Usually you see only one used.

 

[zunder1990]

I went back and redid the layout of the storage hosts.
Here are the spec
2x E5-2603 6core
32gb ram
2x 128gb ssd OS
54.6 TB with 12x6tb in raid6
2.2TB ssd with 4x1.2tb in raid10
9361-8i card card
3u
MCX4121A-ACAT=$500
The per system cost looks to be about $12k

 

[Patrick]

@zunder1990

On the SSD side, it is likely cheaper to go with 2x NVMe SSDs in RAID 1, plus you will get more performance out of them.

For my money, here's what I would do at ~$100K $110K:

• 5x NVMe hyperconverged nodes 20TB using 40GbE
• 1x backup node(s) either a big 120TB node or cluster the backup as well across 2-3 nodes making each node smaller.
• Either the same switch (looking to 100GbE in the future) or just get a $8K or so 32 port 40GbE switch.

It will likely come in at a few thousand cheaper than you were planning and perform significantly better. The big obstacle that we are seeing with the NVMe scale-out clusters is that bottlenecks elsewhere appear. 40GbE is inexpensive now but at some point they will require 100GbE.

 

[CookiesLikeWhoa]

@CookiesLikeWhoa those are very popular CPUs for this type of application. But two seems like overkill for one SAS controller and NIC. Usually you see only one used.

Seems like a E3 would give better performance/price/energy consumption for the same solution. Though if you need gobs of RAM the E5 2603 would be better. But at that point, why not go with a 16xx series? Would be cheaper and provide access to loads of RAM.

To each their own though.

 

[Patrick]

Seems like a E3 would give better performance/price/energy consumption for the same solution. Though if you need gobs of RAM the E5 2603 would be better. But at that point, why not go with a 16xx series? Would be cheaper and provide access to loads of RAM.

To each their own though.

I think the reason is that the E5-1600 V4 series is 140W while the E5-2603 V4 or E5-2609 V4 chips are 85W and usually run very cool compared to their TDP. You also have the option of going to a second CPU if you use the E5-2600 series.

The major issue with the E3 is that you get stuck with unbuffered DIMMs and really low memory levels. For cold storage, the Xeon D has become popular since you get lower power, lower cost 10GbE and can use 4x 32GB RDIMMs.

 

[TLN]

We should have a cool 100-120TB cold storage platform arriving in the lab later this week. Operationally should be close to 1A lower power consumption than the spec'd dual CPU platform even with CPUs at idle.

Mind you asking, where do you work?
Do you guys need one smart networking guy with programming skills in the team?

 

[Patrick]

Mind you asking, where do you work?
Do you guys need one smart networking guy with programming skills in the team?

I work for a website called ServeTheHome

We have racks in a data center where we do all of our reviews out of: STH Data Center Lab Update May 2016

There are very few sites that use an actual data center. Johan @ Anandtech uses a university lab. StorageReview uses converted office space and they have the best setups aside from us in terms of rack space. Most other review sites, even those larger than STH (e.g. at The Register) will do server tests in a home. That includes large storage systems and blade servers.

We have the capability to host demos for other people including STH members, other review sites, industry analysts, consultants looking to showcase a solution, companies looking to host hardware/ solutions and etc.

I would love to be able to hire STH folks full time, and do need a networking guru, but revenue is (quite a bit) lacking at the moment.

 

[TLN]

I work for a website called ServeTheHome

We have racks in a data center where we do all of our reviews out of: STH Data Center Lab Update May 2016

There are very few sites that use an actual data center. Johan @ Anandtech uses a university lab. StorageReview uses converted office space and they have the best setups aside from us in terms of rack space. Most other review sites, even those larger than STH (e.g. at The Register) will do server tests in a home. That includes large storage systems and blade servers.

We have the capability to host demos for other people including STH members, other review sites, industry analysts, consultants looking to showcase a solution, companies looking to host hardware/ solutions and etc.

I would love to be able to hire STH folks full time, and do need a networking guru, but revenue is (quite a bit) lacking at the moment.

Oh, I got it now. I thought that you have access to the equipment in some enterprise company, and publish tests and labs here.Either way, having latest equipment, always exciting.

I would be happy to participate in network-related project. I have some spare time at the moment, and I want to contribute to community. Either finally complete my own networking tool and publish it to the public, or participate in other networking-related project.