Project Proposal: ELK Stack for Monitoring Proxmox, pfSense, FreeNAS

Biren78

Active Member
Jan 16, 2013
550
94
28
Proposed Project Name: ELK-PPF

Objective: Monitor Home Labs

Technologies: Elasticsearch, Logstash, Kibana, Docker

Description
I want to propose a project. We should have a standard launcher for an ELK stack in Docker. Then, we should work on getting Proxmox, pfSense and FreeNAS logs into the ELK stack.

I don't have the skills to do this myself. I'm noticing a lot of Promxox pfSense, FreeNAS in everyone's labs. Since we all seem to have 2-3 of those, what if we had an official STH user ELK setup to monitor those, all with guides?

Open for discussion. From what I've been reading, ELK is the hipster thing to do for monitoring nowadays.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,007
1,569
113
CA
Ooooh I like where this is heading :)

Have you played around with any of it at all yet @Biren78 ? If so what kind of issues.
 

Biren78

Active Member
Jan 16, 2013
550
94
28
Last edited:

wsuff

Member
Aug 16, 2015
72
11
8
Looking forward to seeing how this evolves. Haven't looked at ELK in depth. Trying to dig into Prometheus/Grafana in the short term since immediate use case is more blackbox stuff with some pretty graphs.
 

wsuff

Member
Aug 16, 2015
72
11
8
Is there a preferred place to put doc efforts for things like this? I like seeing things suggested like this since collectively there is a lot of knowledge/skill here & the varying environments/needs should produce some interesting ideas/use cases. Need to tackle Nagios/Incigia as a container as well for myself so be happy to contribute that.
 

wsuff

Member
Aug 16, 2015
72
11
8
The Guide section here would likely be a good fit once it's been completed & reviewed but until then. Git likely would be the best option to store docker-compose/docs. I guess it could be started as personal projects on GitHub or similar until STH adopts it as a community project.
 

nitrobass24

Moderator
Dec 26, 2010
1,083
127
63
TX
Yea it seems we might need to do both at first.

1. I like that someone else has already made an ELK Docker, in fact there are quite a few out there! But I wonder if we can do it better ourselves?
2. Need to come up with a list of use-cases. Its nice to say FreeNAS, nginx...but what exactly do you want to monitor in FreeNAS, nginx, etc. I think once that is defined we could have someone focus on building out that functionality. Almost in a plug-in like fashion so that it could be its own code-base and is a bolt on to the platform.
 
  • Like
Reactions: wsuff

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
@nitrobass24 - here is my suggestion on that:

We need a base ELK install. From there, we can get plugins developed for folks.

If we can get the basic stack working, I am happy to reach out to the various teams and see if they would provide support.
 
  • Like
Reactions: T_Minus

nitrobass24

Moderator
Dec 26, 2010
1,083
127
63
TX
Yep completely agreed. Will spend some time on this once I get my storage setup this weekend.


Sent from my iPhone using Tapatalk
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,007
1,569
113
CA
As a docker n00b are we going to run into any limitations by utilizing Docker for this instead of doing the same thing and making a generic VM images? Other than needing docker that is...
 

Monoman

Active Member
Oct 16, 2013
335
96
28
I'm interested, but I'd prefer to see a traditional VM approach instead of docker honestly.

I should be fairly simple to spin up a Turnkey Linux VM, write a script that modifies everything as want installed. I've done it many times for software. TKL is pretty amazing if you've not used it yet.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
@Monoman - easier to run Docker in a VM than a VM in Docker.

@T_Minus just need to ensure we have mount points for persistent data versus VM where we need the virtual disk size to be large enough and the correct provisioning of resources.

Also, doing Docker v. VM is going to be easier to maintain with newer versions.