1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

CEPH write performance pisses me off!

Discussion in 'Linux Admins, Storage and Virtualization' started by whitey, Jan 25, 2017.

  1. grogthegreat

    grogthegreat New Member

    Joined:
    Apr 21, 2016
    Messages:
    23
    Likes Received:
    7
    Ceph is awesome since it does file, block, and object. On the other hand if you only need block and performance is a concern, I've been happy with ScaleIO. Just three SSDs gets me 4.3Gb write and 10.9Gb read.

    [​IMG] [​IMG]
     
    #21
  2. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,217
    Likes Received:
    677
    Impressive, I'm pretty happy w/ 3 OSD's and 3 ssd's behind each OSD delivering 2Gbps read, 1Gbps write.

    What type of ssd's are you using in your ScaleIO config?
     
    #22
  3. marcoi

    marcoi Active Member

    Joined:
    Apr 6, 2013
    Messages:
    369
    Likes Received:
    35
    is scaleIO completely free? I'm assuming both these solutions require three nodes? I'm trying to decide how to do home storage, if i want a single server running freenas giving the other two servers nfs/iscsi storage for both vm datastores usage and archival storage(ie media, pictures, docs, etc. ) or trying something like scaleio, where each box has a few sets of drives and all shared about.

    Maybe im off on my idea what scaleio or ceph is suppose to be used for?

    Right now the two esxi nodes each have their own local storage. I'm building out a third server which is limited to 8 2.5 hdd so I'm trying to decide is it best to pool all my storage under freenas, then pass it out or do something else with it. I may just stick to local to keep it simple.

    any thoughts appreciated.
     
    #23
  4. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,353
    Likes Received:
    829
    @marcoi - no, ScaleIO is not "completely free". It is a commercial product that has a "free option" for small scale deployments. That is sorta the opposite of Ceph, which is FOSS but has a paid "support" option from RedHat/InkTank.

    For the use case you describe Ceph or ScaleIO could work - but they are probably more trouble for you than value. Your initial thought of a storage server serving iSCSI/NFS to two workload platforms is a good one - and will be much easier to manage. You do still have the single point of failure in the storage server. But in practice - unless you really need 3 to 5 "9s" reliability - that is of little consequence (if it stops you just fix it, move on and be happy).

    both Ceph and ScaleIO are "scale out" solutions. They are really designed for use cases where your storage needs are quite large and likely growing. They are designed to "scale horizontally", which means you can add additional disks/servers/racks/etc as you need them without impact to active service and with linear impact to performance (in theory...). In the horizontal scale environment getting consistent and predictable performance as you grow is usually more important than getting absolute maximum performance possible, though ScaleIO does emphasize performance while Ceph tends to emphasize flexibility and consistency of performance.

    Neither solution is for the feint of heart to install/manage. To the extent they exist at all, their installation, operation and monitoring tools are - to be kind - "maturing". There is a fair amount of learning that has to be put place before you really understand how to manage it well.

    At the small end of the reasonable use case, for users like @Patrick, the high-availability features are very important (largely because we all start harassing him if STH stops working :)). And a side effect of the horizontal scalability design allows him to make good use of the mixed bag of disks he's collected over the years. But Ceph is only one part of the whole picture of how he achieves high availability.

    Of course - if you want to learn the "scale out" approaches or test them and understand them and play with them - have at it. Join me, @whitey, @grogthegreat, @Patrick and many others. There is lots to learn and share - and both Ceph and ScaleIO have community involvement so you might even be able to contribute.
     
    #24
    whitey likes this.
  5. marcoi

    marcoi Active Member

    Joined:
    Apr 6, 2013
    Messages:
    369
    Likes Received:
    35
    @PigLover thanks for the clarification. I don't build out datacenters for a living, so my use case for learning all of this new tech and software is for personal pleasure and to keep up with a broad scope of IT knowledge. It tends to help with other work related IT stuff I do.

    With that said, both of these technologies, as interesting as they are, would be way over kill for home use. I might play with them down the road but for now i want to just get home stuff running by having a simple solution that works and is easy to support.
     
    #25
  6. Marsh

    Marsh Well-Known Member

    Joined:
    May 12, 2013
    Messages:
    1,011
    Likes Received:
    360
    @marcoi

    You don't need to wait to play with the new stuff.
    Using your ESXi box , you have opportunity to create dozen of nested virtual lab.
    vsphere vSan cluster , Proxmox cluster , cepth cluster , SaclieIO cluster , S2D cluster , Cisco network lab etc.
    If you want speed , add more ram, PCIe SSD, 10Gbit network and switch is buildin to vSpehre.
     
    #26
    PigLover likes this.
  7. marcoi

    marcoi Active Member

    Joined:
    Apr 6, 2013
    Messages:
    369
    Likes Received:
    35
    @Marsh I do need to wait - else Wife will not like me any more. ;) plus i still need to get the new server squared away first.
     
    #27
  8. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,217
    Likes Received:
    677
    @PigLover hit it on the head, nothing to add here other than saying well done sir! My vote for you and I've seen a few of your requests, stick w/ either a dedicated NAS box (I won't warp your mind here) or a all-in-one if you want a lil' more utility out of the setup.
     
    #28
  9. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,353
    Likes Received:
    829
    OK - so confession time. I have more toys available than I need and run a 7 node (soon growing to 9) Proxmox/Ceph hyperconverged cluster for testing and playing and stuff...but for the things I care about I keep it really simple.

    FreeNAS storage host stuffed with 4TB Seagate portable "cracked" drives for volume, a few SSDs in a pool for speed, and one small Proxmox host for VMs. Another FreeNAS host for local backups. And stuff I really care about backed up to the Cloud. This part of my network I never need to worry about - it just runs and runs and runs.

    And @Marsh is spot-on - you can do all the playing/learning part on VMs built inside the two ESXi nodes. Without spending any more money (to make the wifey happy).
     
    #29
  10. grogthegreat

    grogthegreat New Member

    Joined:
    Apr 21, 2016
    Messages:
    23
    Likes Received:
    7
    @whitey If you didn't see it in the picture, the SSDs are PM853T 960GB SATA drives. Found the amazing ebay deal for them on this site in fact.

    @marcio Since no one directly answered the minimum node question, yes scaleIO does need at least three nodes. I've had both three and four nodes in my cluster and find that 4 nodes is much better since you get a higher usable percent out of your raw total storage. Plus I don't like to be at the limits of anything. That benchmark is with only three nodes, each with only one SSD.
    While PigLover is correct that it has a free option, the great thing is that the free option is identical to the paid version with no limits other than no support and you aren't allowed to use in production.

    I'm not the target audience but I do use ScaleIO at home to run both my home lab as well as storage for movies, tv shows, music etc. I like how I don't need to do anything if a hard drive or an entire server dies. The data gets reduplicated right away instead of waiting for me to replace hardware. I can add hard drives or servers as my needs grow and the data gets re-balanced onto the new hardware. After researching ceph and giving up, I found scaleio easy to install and manage. There are downsides to software that isn't FOSS. Dell (after taking over EMC) could decide to abandon the software or stop releasing the unlimited and free for person use version. That significant risk is worth the benefits to me but might not be for you.
     

    Attached Files:

    #30
    marcoi likes this.
  11. marcoi

    marcoi Active Member

    Joined:
    Apr 6, 2013
    Messages:
    369
    Likes Received:
    35
    I see ScaleIO has a VM version. Anyone try that?

    Here is my Idea dunno if its good.
    Create a Scale IO VM host on three separate esxi nodes. Each will have 1 raid card in pass-through or if it doesn't matter then i will use local data store with either same amount of hdd space assigned or 1 250gb ssd drive if in pass through since i have 4 of the same models. Each will be assigned to a new vm switch which will have vm iscsi kernal and 2 physical 1GB nics (each node has atleast 4x 1gb nics). My idea is have the 2 nics of each host networked to allow the cross host access to duplicate the data while the internal switch provides access to the local ScaleIO vm.

    Not sure if that is possible? would it be worth testing?
     
    #31
  12. grogthegreat

    grogthegreat New Member

    Joined:
    Apr 21, 2016
    Messages:
    23
    Likes Received:
    7
    What you described is called 'hypercovered' if you like tech lingo. ScaleIO is neat in that it can run hyperconvered (storage and hypervisor in the same box), or not.
    Not only does ScaleIO support it, but it even has a vcenter web client add on that will deploy that config for you; even creating the VMs, setting up the networking, and attaching the disks. I had the software deploy the scaleio VMs to 32GB SATADOMs. The disks that you choose get directly attached to the VMs as part of the install process. ScaleIO doesn't like raid cards so use a HBA or just sata drives. Changing scaleio networking after it is setup is a pain so I kept it simple and just had the ScaleIO VMs only have a single IP address and virtual nic that they used for Scaleio VM to VM communication as well as ScaleIO VM to esxi communication. Since you already have SSDs and will probably get more in the future, I strongly recommend getting a 10Gb switch. I'm using a quanta lb6m switch and connectx-2 single port NICs which kept the cost low. Using vlans lets me have as many subnets as I could want all on the single nic.

    The free scaleio download includes getting started, user, and deployment guides. Read through them and you'll get a good understanding of how the software works and if it matches what you are trying to do.
     
    #32
Similar Threads: CEPH write
Forum Title Date
Linux Admins, Storage and Virtualization Ceph is pool never creating? Feb 6, 2017
Linux Admins, Storage and Virtualization Proxmox VE docs to upgrade Ceph to Jewel Jan 5, 2017
Linux Admins, Storage and Virtualization Proxmox 4.4 now supports Ceph "Jewel" release Jan 5, 2017
Linux Admins, Storage and Virtualization Proxmox WordPress Ceph and IOPS Dec 21, 2016
Linux Admins, Storage and Virtualization Whitey's foray into the world of CEPH Dec 12, 2016

Share This Page