Project Proposal: ELK Stack for Monitoring Proxmox, pfSense, FreeNAS

Discussion in 'Software Stuff' started by Biren78, Apr 26, 2017.

  1. Biren78

    Biren78 Active Member

    Joined:
    Jan 16, 2013
    Messages:
    550
    Likes Received:
    94
    Proposed Project Name: ELK-PPF

    Objective: Monitor Home Labs

    Technologies: Elasticsearch, Logstash, Kibana, Docker

    Description
    I want to propose a project. We should have a standard launcher for an ELK stack in Docker. Then, we should work on getting Proxmox, pfSense and FreeNAS logs into the ELK stack.

    I don't have the skills to do this myself. I'm noticing a lot of Promxox pfSense, FreeNAS in everyone's labs. Since we all seem to have 2-3 of those, what if we had an official STH user ELK setup to monitor those, all with guides?

    Open for discussion. From what I've been reading, ELK is the hipster thing to do for monitoring nowadays.
     
    #1
    nitrobass24, wsuff, Kybber and 2 others like this.
  2. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,888
    Likes Received:
    1,511
    Ooooh I like where this is heading :)

    Have you played around with any of it at all yet @Biren78 ? If so what kind of issues.
     
    #2
  3. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,956
    Likes Received:
    863
    So who's got experience leading this kind of project team?
     
    #3
  4. Biren78

    Biren78 Active Member

    Joined:
    Jan 16, 2013
    Messages:
    550
    Likes Received:
    94
    #4
    Last edited: Apr 26, 2017
  5. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,625
    Likes Received:
    4,581
    I like this.evenbetter if we can add nginx monitoring
     
    #5
    Evan and T_Minus like this.
  6. wsuff

    wsuff Member

    Joined:
    Aug 16, 2015
    Messages:
    72
    Likes Received:
    11
    Looking forward to seeing how this evolves. Haven't looked at ELK in depth. Trying to dig into Prometheus/Grafana in the short term since immediate use case is more blackbox stuff with some pretty graphs.
     
    #6
  7. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,082
    Likes Received:
    126
  8. wsuff

    wsuff Member

    Joined:
    Aug 16, 2015
    Messages:
    72
    Likes Received:
    11
    Is there a preferred place to put doc efforts for things like this? I like seeing things suggested like this since collectively there is a lot of knowledge/skill here & the varying environments/needs should produce some interesting ideas/use cases. Need to tackle Nagios/Incigia as a container as well for myself so be happy to contribute that.
     
    #8
  9. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,082
    Likes Received:
    126
  10. wsuff

    wsuff Member

    Joined:
    Aug 16, 2015
    Messages:
    72
    Likes Received:
    11
    The Guide section here would likely be a good fit once it's been completed & reviewed but until then. Git likely would be the best option to store docker-compose/docs. I guess it could be started as personal projects on GitHub or similar until STH adopts it as a community project.
     
    #10
  11. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,625
    Likes Received:
    4,581
    #11
    T_Minus likes this.
  12. wsuff

    wsuff Member

    Joined:
    Aug 16, 2015
    Messages:
    72
    Likes Received:
    11
    Yup just saw it and forked it.
     
    #12
  13. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,082
    Likes Received:
    126
    Yea it seems we might need to do both at first.

    1. I like that someone else has already made an ELK Docker, in fact there are quite a few out there! But I wonder if we can do it better ourselves?
    2. Need to come up with a list of use-cases. Its nice to say FreeNAS, nginx...but what exactly do you want to monitor in FreeNAS, nginx, etc. I think once that is defined we could have someone focus on building out that functionality. Almost in a plug-in like fashion so that it could be its own code-base and is a bolt on to the platform.
     
    #13
    wsuff likes this.
  14. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,625
    Likes Received:
    4,581
    @nitrobass24 - here is my suggestion on that:

    We need a base ELK install. From there, we can get plugins developed for folks.

    If we can get the basic stack working, I am happy to reach out to the various teams and see if they would provide support.
     
    #14
    T_Minus likes this.
  15. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,082
    Likes Received:
    126
    Yep completely agreed. Will spend some time on this once I get my storage setup this weekend.


    Sent from my iPhone using Tapatalk
     
    #15
  16. wsuff

    wsuff Member

    Joined:
    Aug 16, 2015
    Messages:
    72
    Likes Received:
    11
    It appears GitHub - deviantony/docker-elk: The ELK stack powered by Docker and Compose. does similar using official images for elastic search logstash and kibana. I think we likely will want to do docker-compose slightly different to make it easier for uses to adapt to their lab. Perhaps include a sample .env file and have as part of the docs copy/modify that to fit your environment.
     
    #16
  17. RobertFontaine

    RobertFontaine Active Member

    Joined:
    Dec 17, 2015
    Messages:
    666
    Likes Received:
    148
    Hmm, and I was just looking at cacti and snmp this morning.
     
    #17
  18. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,888
    Likes Received:
    1,511
    As a docker n00b are we going to run into any limitations by utilizing Docker for this instead of doing the same thing and making a generic VM images? Other than needing docker that is...
     
    #18
  19. Monoman

    Monoman Active Member

    Joined:
    Oct 16, 2013
    Messages:
    327
    Likes Received:
    90
    I'm interested, but I'd prefer to see a traditional VM approach instead of docker honestly.

    I should be fairly simple to spin up a Turnkey Linux VM, write a script that modifies everything as want installed. I've done it many times for software. TKL is pretty amazing if you've not used it yet.
     
    #19
  20. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,625
    Likes Received:
    4,581
    @Monoman - easier to run Docker in a VM than a VM in Docker.

    @T_Minus just need to ensure we have mount points for persistent data versus VM where we need the virtual disk size to be large enough and the correct provisioning of resources.

    Also, doing Docker v. VM is going to be easier to maintain with newer versions.
     
    #20
    MiniKnight and T_Minus like this.

Share This Page