Homelab Monitoring Platform?

Eric Faden

Member
Dec 5, 2016
98
6
8
38
Hey All,

Sorry in advance for the length of this post….

So I have a new computer I'm setting up to use as a monitoring platform. It is an AMD based machine (GX-415GA + 8GB RAM + 1 TB Seagate FireCuda). Right now I have Ubuntu 17.04 installed on it. Right now I have 2 sites I want to monitor. I want to monitor:


Site A
- MikroTik RouterOS
- MikroTik RB3011
- MikroTik NetMetal 5 x 3
- MikroTik CRS125
- MikroTik SWOS
- MikroTik RB260GSP
- QNAP TVS-682
- QNAP TS-439 Pro
- Proxmox Servers x 2 (Supermicro Based, 1 x Xeon D and 1 x Xeon E5)
- UPS (USB Connected to QNAP TVS-682)


Site B
- MikroTik RouterOS
- MikroTik RB3011
- MikroTik CRS112
- MikroTik hEX POE x 2
- MikroTik wAP ac x 3
- MikroTik SWOS
- MikroTik RB260GSP
- NetGear GS752TP
- QNAP TVS-682
- QNAP TS-439 Pro
- UPS (USB Connected to QNAP TVS-682)


I see the following needs….


1) Uptime + Alerting (via Ping? SNMP? ???) => I need to monitor to make sure the Network is alive and that nothing is down.

2) Latency => Site to Internet, Site To Site (VPN), etc

3) CPU/Memory/Disk/etc from 3 Computers

4) SNMP from QNAP, Network Equipment (MikroTik + NetGear), Web Power Switch, UPS, etc

5) Network Map/Weathermap Style (Network Weathermap)

6) Netflow (IPFIX) Data from RB3011s


My thoughts so far….


For #1 & #2 I was thinking Zabbix, but it seems like overkill (complex, graphs I don’t need). I had/have a lot of it setup, but it is having some issues with high disk I/O and so I started rethinking if there was a simpler system to use for just the Uptime/Alerting/Latency. Is there a simpler system for this? (Can I use InfluxDB for this?)

For #3 I was thinking of Netdata (client/server model) with the output being stored in InfluxDB (or similar) and then using Grafana for visualization.

For #4 I was thinking of using Telegraf with the SNMP Plugin to pull the data into InfluxDB and then using Grafana for visualization.

For #5 …. Unsure…. Maybe PHP Weathermap w/ php-weathermap-influxdb/WeatherMapDataSource_influxdb.php at master · guequierre/php-weathermap-influxdb · GitHub ?? … or rebuild that with GitHub - influxdata/influxdb-php: influxdb-php: A PHP Client for InfluxDB, a time series database …. or Grafana-Diagram … or GitHub - otm/networkmap.js: A network weathermap written in Javascript and SVG with editing capabilities. or … mermaid - Generation of diagrams and flowcharts from text in a similar manner as markdown. … or jQuery Mapael - Ease the build of pretty data visualizations on dynamic vector maps

For #6 …. Thinking of NTOPNG or PMACCT w/ InfluxDB …. Thoughts here? Goal would be to get the IPFix data to be able to get an idea of which protocols, clients, etc are using bandwidth?

Mostly I’m just doing this for fun….. Thoughts?

Eventually once I get this all setup I’m planning to post instructions for what I did….
 

Eric Faden

Member
Dec 5, 2016
98
6
8
38
So Far...

1) Fresh Installed Ubuntu 17.04
2) Installed NetData (Instructions from NetData Site)
3) Installed Docker (Edge)
 

Rand__

Well-Known Member
Mar 6, 2014
4,576
911
113
What did you end up with?
Using librenms for similar (smaller scale) topics and am currently looking into netflow data which led me here...
not too happy with nfsen but not sure whether thats the tool, the raw data (dvSwitch) or my crappy setup :p
 

Eric Faden

Member
Dec 5, 2016
98
6
8
38
Honestly haven't found anything good and open source for NetFlow. I used NTOP for a bit until the trial expired.... but otherwise.... nothing great.
 

dandanio

Active Member
Oct 10, 2017
147
50
28
As a legacy I have:
1) munin + smokeping (which I am migrating away from, and keep it JIC)
2) InfluxDB + Telegraf + Grafana (this is the monitorin-ng, what I am migrating to). It includes many plugins like SNMP and many custom ones.

At work we use:
1) Nagios + Thruk and Check_MK.

I used SevOne, zabbix, icinga (well, still nagios), cacti and I liked cacti the most. But after migrating, I do not look back.

How about PRTG (free)?
 

Rand__

Well-Known Member
Mar 6, 2014
4,576
911
113
Honestly haven't found anything good and open source for NetFlow. I used NTOP for a bit until the trial expired.... but otherwise.... nothing great.
Is there meaningful data in ntop from netflow?
Wikipedia lists a bunch of interesting data points but all I get on nfsen is protocol statistics

upload_2017-12-24_10-48-1.png

Edit: Looks like I should follow sth like
Nfsen: Traffic Classification


2) InfluxDB + Telegraf + Grafana (this is the monitorin-ng, what I am migrating to). It includes many plugins like SNMP and many custom ones.
How about PRTG (free)?
The TICK stack looks interesting but seems fairly limited in automatic setup (if I look at the plugins documentation on git).
Neither any company I know (nor I personally) are willing to spend days/weeks in setting up basic monitoring any more. Maybe there are some advanced detection/setup features in the enterprise version, have not looked at that.

PRTG free is way too limited. 100 Sensors are like 4 boxes. O/c i one limits to key stats then one can cover more but where's the fun in that;)
 
Last edited:

voxadam

Member
Apr 21, 2016
107
14
18
Portland, Oregon
If you're looking for an open source tool to generate/export network flow data you might want to check out ipt-netflow. It's a Linux kernel module that creates an iptables target for generating NetFlow v5, v9, and IPFIX (aka NetFlow v10) flow data. I've been meaning to try it but haven't quite gotten around to it.

Code:
=========================
= Detailed Feature List =
=========================

   * High performance and scalability. For highest performance module could be
    run without conntrack being enabled in kernel. Reported to be able to
    handle 10Gbit traffic with more than 1500000 pps with negligible server
    load (on S5500BC).

   * NetFlow v5, v9, and IPFIX are fully supported.

    Support of v9/IPFIX is adding flexibility to exporting of flow data
    plus greater visibility of traffic, letting export many additional fields
    besides what was possible in v5 era. Such as
   
   * IPv6 option headers, IPv4 options, TCP options, ethernet type, dot1q
    service and customer VLAN ids, MAC addresses, and

   * Full IPv6 support,

   * NAT translations events (from conntrack) using NetFlow Event Logging (NEL).
    This is standardized way for v9/IPFIXr, but module export such events even
    for v5 collectors via specially crafted pseudo-records.

   * Deterministic (systematic count-based), random and hash Flow Sampling.
    With appropriate differences in support of v5, v9, and IPFIX.

   * SNMP agent (for net-snmp) for remote management and monitoring.

   * Options Templates (v9/IPFIX) let export useful statistical,
    configurational, and informational records to collector.
    Such as metering, exporting, sampling stat and reliability stat, sampling
    configuration, network devices ifName, ifDescr list.

   * Tested to compile and work out of the box on Centos 5, 6, 7, Debian and
   * Ubuntu. Many vanilla Linux kernels since 2.6.18 up to the latest (as of
   * writing is 3.19) are supported and tested.

   * Module load time and run-time (via sysctl) configuration.

   * Flexibility in enabling features via ./configure script. This will let you
    disable features you don't need, which increase compatibility with custom
    kernels and performance.

   * SNMP-index translation rules, let convert meaningless and unstable
    interface indexes (ifIndex) to more meaningful numbering scheme.

   * Easy support for catching mirrored traffic with promisc option. Which is
    also supporting optional MPLS decapsulation and MPLS-aware NetFlow.
 

Cmdrd

New Member
Jun 23, 2016
21
2
3
29
Just to throw my $0.02 in here for my homelab setup that works well. It's a bit of work to implement but allows for a lot of customization and is quite extensible.

For monitoring and infrastructure stats collection I use Prometheus for the back-end with a Grafana front-end. Prometheus is super flexible, has a lot of power in it's querying language PromQL, and you can build exporters for pretty much everything which was a fun project to work on.

In terms of *flow information, I have an ELK stack (Elasticsearch, Logstash, Kibana) for ingesting NetFlow and SFlow along with doing packet capture analysis. It scales out really well and blows tools like Wireshark out of the water for packet capture analysis after building out a good group of filters.
 
  • Like
Reactions: Tha_14

hhp

New Member
Aug 3, 2016
18
3
3
52
I would suggest looking at LibreNMS, I monitor 10's of thousands of devices with it (entire ISP network), from the juniper core routers, cisco and every tik device made or very close to it to the CPE gear. Also: vmware from hypervisors to vcenter, many linux and win servers, some LTE backend, pretty much anything smnp capable. It is highly configurable, you can add MIB's for devices that are not in the release etc. It already uses RRD, weathermap etc and has many integrations. The UI for making custom network maps is still clunky but you can do it, I think its easier to just create and edit the files. I only skimmed the thread but it should cover most of what I saw mentioned.
 

Robert Fontaine

Active Member
Jan 9, 2018
113
28
28
53
I am likely going to go down the microsloth service center route as I need to spend some time learning the tool but I doubt I would pick it otherwise.
 

Rand__

Well-Known Member
Mar 6, 2014
4,576
911
113
you can add MIB's for devices that are not in the release etc.
Any good pointers on how to do that?
The official docs are not compatible with my way of understanding in this part somehow;
eg I wanted to add netgear mibs that would be capable to get the internal temperature but the system never picked them up...

And network maps - the editor is very very basic but if you get into manually working on the files the result is really awesome
 

hhp

New Member
Aug 3, 2016
18
3
3
52
Check the doc's on Observium, LibreNMS was forked from it after the lead had a tantrum. There are write ups etc on adding the MIBs, I havent had to add any in quite a while but its not difficult. From memory you need the MIB's from the manufacturer and need to put them in the MIB's folder.. past that its to fuzzy without reading up again.