Software Defined Storage

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

macrules34

Active Member
Mar 18, 2016
407
28
28
40
I have 4 dell servers each with 2x 1TB hard drives. I would like to use the extra space from the hard drives to create a storage array like you would with ScaleiO.

I was wondering if anyone has come across a free Software Defined Storage solution like ScaleiO? I inquired with Dell EMC about ScaleIO software licenses but was told that they no longer sell just the software, its know bundled with their hardware.
 

NISMO1968

[ ... ]
Oct 19, 2013
87
13
8
San Antonio, TX
www.vmware.com
You can get the licenses under the table still, but ScaleIO has mediocre performance on few nodes, so anything below 8 simply doesn't make sense. Did you try Ceph?

I have 4 dell servers each with 2x 1TB hard drives. I would like to use the extra space from the hard drives to create a storage array like you would with ScaleiO.

I was wondering if anyone has come across a free Software Defined Storage solution like ScaleiO? I inquired with Dell EMC about ScaleIO software licenses but was told that they no longer sell just the software, its know bundled with their hardware.
 

macrules34

Active Member
Mar 18, 2016
407
28
28
40
I have looked through the Ceph documentation and was wondering if you could use a portion of a disk as storage space? One partition with the OS and the remainder of the disk used as storage. With Scale IO I know you are able to do that.
 

macrules34

Active Member
Mar 18, 2016
407
28
28
40
I was reading the Ceph manual for hardware recommendations and noticed that it said you need 2 Ethernet ports for each of the following:
1. OSD
2. MON
3. MDS

So does that mean that I would need six Ethernet ports per node? Each node will be providing storage. I am using some 1U servers and only have one expansion slot, the servers also have 2 onboard Gigabit NICs, plus an IPMI port.

Will gigabit Ethernet suffice or should I be using 10GB?

FYI, client server only has 1GB Ethernet and not was to add 10GB.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
I wouldn't bother with Ceph in such a small cluster. Almost guaranteed you won't be satisfied with the performance or the resiliency (ability to withstand faults) at such a small scale. Same issue for ScaleIO or really any other fully distributed SDS.

At the small scale of your lab Gluster is likely ideal.
 

hhp

New Member
Aug 3, 2016
20
3
3
55
Just as far as OSS SDS software solutions, I would go with ceph if you have IOP performance needs or gluster for media, file sharing, SMB etc. That said you will need a minimum 2 - 10 gbps network ports per server and really should use a switch with enough ports at minimum. To get performance out of these platforms you need a fast control network, decent/nearly current H/W and as many OSD (data) nodes, mon etc. as perf targets dictate.

If you are planning to reuse 3-4 old servers from the westmere or older era on a 1 gbps network you will be very disappointed (_very_), especially if they are already running something( and will need to continue in this role). The storage nodes should be 100% dedicated to the role and nothing else, they should be reasonably fast with many cores (min 1 osd process per disk, 4 for ssd, nvme - plan for 1 core per disk) and one or more fast flash device for journal, WAL and only enough osd processes/disks assigned that the flash device can service.

If you use consumer SSD's, old servers with too many disk for the available CPU/RAM to handle, especially running on a 1 gb network your cluster will most likely be slower than a single disk during the time it actually runs before crashing. The thing to remember about Ceph and Gluster is they were designed to run on commodity servers etc. yes - but on a scale most never see, multiple racks of servers per pool etc. To get performance at lower node counts you need very fast, expensive H/W. If you take old servers already running something else and then try to layer Ceph on top of it, that will be a disaster.

I just went though this learning experience and thanks to experienced Ceph users feedback and suggestions, saved a fair sum of money I was planning to spend on used westmere era servers with lots of disk and slower procs. From those discussions and my research I decided to go with 1U (2x - E5-26xx v2 10C/20T) servers with a maximum of 8 storage devices (10k SAS 1.2 TB) using smaller NVMe addin's for write ahead log etc and a minium of 64GB RAM per SD node. Total of 6 OSD nodes with mon and admin etc. all running on their own lower spec 1U servers of the same class. This is just for a test environment to verify a Ceph implementation will work for our intended purpose, if we go this route the initial prod deployment will most likely be 2-3 racks of EPYC servers using a large amount of NVMe U.2 SSDs and SAS SSD's in a tiered config fronting a few hundred 10k drives.

My network is 40gbps top of rack so I was already good there, using 1 gbps network is a non-starter for an SDS solution, you would just be wasting your time other than learning a few new things. Also keep in mind, just getting a few servers running in a Ceph cluster is just a start, you will need to master the crushmap and other platform components to really optimize and get the performance out of it. If you do manage to get something running on servers already performing a task , using un-allocated disk space, I doubt the cluster would be able to heal/recover or it would take a few lifetimes of waiting.

One more thing to keep in mind when sizing, you need to target your specs on your performance minimums during a node failure/recovery not just a clean good state, with a minimum of nodes (3-4) you will lose a large percentage of your capacity, recovery & re-balancing is very resource intensive.

I am back in the research stage for my test environment double checking everything before making the spend so take what I say with a grain of salt, I realistically don't think this will work for you or deliver what you want out of it.
 
  • Like
Reactions: jak

macrules34

Active Member
Mar 18, 2016
407
28
28
40
I am using 4 dedicated servers that are Dell CS24-SC with 16GB RAM each server has two 1TB hard drives and I will run the OS off a USB flash drive. There are two free slots for drive expansion. The server has one expansion slot.

The client machine has 1GB Ethernet and not possible to upgrade to 10GB, so the client would have to connect to the storage at 1GB, there's no other option that I'm aware of. I could use 10GB for back end duties.