Homelab Monitoring Platform?

Discussion in 'Software Stuff' started by Eric Faden, May 14, 2017.

  1. Eric Faden

    Eric Faden Member

    Joined:
    Dec 5, 2016
    Messages:
    97
    Likes Received:
    6
    Hey All,

    Sorry in advance for the length of this post….

    So I have a new computer I'm setting up to use as a monitoring platform. It is an AMD based machine (GX-415GA + 8GB RAM + 1 TB Seagate FireCuda). Right now I have Ubuntu 17.04 installed on it. Right now I have 2 sites I want to monitor. I want to monitor:


    Site A
    - MikroTik RouterOS
    - MikroTik RB3011
    - MikroTik NetMetal 5 x 3
    - MikroTik CRS125
    - MikroTik SWOS
    - MikroTik RB260GSP
    - QNAP TVS-682
    - QNAP TS-439 Pro
    - Proxmox Servers x 2 (Supermicro Based, 1 x Xeon D and 1 x Xeon E5)
    - UPS (USB Connected to QNAP TVS-682)


    Site B
    - MikroTik RouterOS
    - MikroTik RB3011
    - MikroTik CRS112
    - MikroTik hEX POE x 2
    - MikroTik wAP ac x 3
    - MikroTik SWOS
    - MikroTik RB260GSP
    - NetGear GS752TP
    - QNAP TVS-682
    - QNAP TS-439 Pro
    - UPS (USB Connected to QNAP TVS-682)


    I see the following needs….


    1) Uptime + Alerting (via Ping? SNMP? ???) => I need to monitor to make sure the Network is alive and that nothing is down.

    2) Latency => Site to Internet, Site To Site (VPN), etc

    3) CPU/Memory/Disk/etc from 3 Computers

    4) SNMP from QNAP, Network Equipment (MikroTik + NetGear), Web Power Switch, UPS, etc

    5) Network Map/Weathermap Style (Network Weathermap)

    6) Netflow (IPFIX) Data from RB3011s


    My thoughts so far….


    For #1 & #2 I was thinking Zabbix, but it seems like overkill (complex, graphs I don’t need). I had/have a lot of it setup, but it is having some issues with high disk I/O and so I started rethinking if there was a simpler system to use for just the Uptime/Alerting/Latency. Is there a simpler system for this? (Can I use InfluxDB for this?)

    For #3 I was thinking of Netdata (client/server model) with the output being stored in InfluxDB (or similar) and then using Grafana for visualization.

    For #4 I was thinking of using Telegraf with the SNMP Plugin to pull the data into InfluxDB and then using Grafana for visualization.

    For #5 …. Unsure…. Maybe PHP Weathermap w/ php-weathermap-influxdb/WeatherMapDataSource_influxdb.php at master · guequierre/php-weathermap-influxdb · GitHub ?? … or rebuild that with GitHub - influxdata/influxdb-php: influxdb-php: A PHP Client for InfluxDB, a time series database …. or Grafana-Diagram … or GitHub - otm/networkmap.js: A network weathermap written in Javascript and SVG with editing capabilities. or … mermaid - Generation of diagrams and flowcharts from text in a similar manner as markdown. … or jQuery Mapael - Ease the build of pretty data visualizations on dynamic vector maps

    For #6 …. Thinking of NTOPNG or PMACCT w/ InfluxDB …. Thoughts here? Goal would be to get the IPFix data to be able to get an idea of which protocols, clients, etc are using bandwidth?

    Mostly I’m just doing this for fun….. Thoughts?

    Eventually once I get this all setup I’m planning to post instructions for what I did….
     
    #1
  2. RobertFontaine

    RobertFontaine Active Member

    Joined:
    Dec 17, 2015
    Messages:
    666
    Likes Received:
    148
    following...
     
    #2
  3. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,363
    Likes Received:
    286
    Want to see where this goes.
     
    #3
  4. Eric Faden

    Eric Faden Member

    Joined:
    Dec 5, 2016
    Messages:
    97
    Likes Received:
    6
    You and me both.... So far I have netdata setup....

    Sent from my Pixel XL using Tapatalk
     
    #4
  5. Eric Faden

    Eric Faden Member

    Joined:
    Dec 5, 2016
    Messages:
    97
    Likes Received:
    6
    So Far...

    1) Fresh Installed Ubuntu 17.04
    2) Installed NetData (Instructions from NetData Site)
    3) Installed Docker (Edge)
     
    #5
  6. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,490
    Likes Received:
    325
    What did you end up with?
    Using librenms for similar (smaller scale) topics and am currently looking into netflow data which led me here...
    not too happy with nfsen but not sure whether thats the tool, the raw data (dvSwitch) or my crappy setup :p
     
    #6
  7. Eric Faden

    Eric Faden Member

    Joined:
    Dec 5, 2016
    Messages:
    97
    Likes Received:
    6
    Honestly haven't found anything good and open source for NetFlow. I used NTOP for a bit until the trial expired.... but otherwise.... nothing great.
     
    #7
  8. Jannis Jacobsen

    Jannis Jacobsen Active Member

    Joined:
    Mar 19, 2016
    Messages:
    231
    Likes Received:
    36
    why not use mikrotik the dude?
     
    #8
  9. Eric Faden

    Eric Faden Member

    Joined:
    Dec 5, 2016
    Messages:
    97
    Likes Received:
    6
    Because I'd rather have something RRD and web based....
     
    #9
  10. dandanio

    dandanio Member

    Joined:
    Oct 10, 2017
    Messages:
    30
    Likes Received:
    8
    As a legacy I have:
    1) munin + smokeping (which I am migrating away from, and keep it JIC)
    2) InfluxDB + Telegraf + Grafana (this is the monitorin-ng, what I am migrating to). It includes many plugins like SNMP and many custom ones.

    At work we use:
    1) Nagios + Thruk and Check_MK.

    I used SevOne, zabbix, icinga (well, still nagios), cacti and I liked cacti the most. But after migrating, I do not look back.

    How about PRTG (free)?
     
    #10
  11. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,490
    Likes Received:
    325
    Is there meaningful data in ntop from netflow?
    Wikipedia lists a bunch of interesting data points but all I get on nfsen is protocol statistics

    upload_2017-12-24_10-48-1.png

    Edit: Looks like I should follow sth like
    Nfsen: Traffic Classification


    The TICK stack looks interesting but seems fairly limited in automatic setup (if I look at the plugins documentation on git).
    Neither any company I know (nor I personally) are willing to spend days/weeks in setting up basic monitoring any more. Maybe there are some advanced detection/setup features in the enterprise version, have not looked at that.

    PRTG free is way too limited. 100 Sensors are like 4 boxes. O/c i one limits to key stats then one can cover more but where's the fun in that;)
     
    #11
    Last edited: Dec 24, 2017
  12. Eric Faden

    Eric Faden Member

    Joined:
    Dec 5, 2016
    Messages:
    97
    Likes Received:
    6
    #12
  13. voxadam

    voxadam Member

    Joined:
    Apr 21, 2016
    Messages:
    105
    Likes Received:
    11
    If you're looking for an open source tool to generate/export network flow data you might want to check out ipt-netflow. It's a Linux kernel module that creates an iptables target for generating NetFlow v5, v9, and IPFIX (aka NetFlow v10) flow data. I've been meaning to try it but haven't quite gotten around to it.

    Code:
    =========================
    = Detailed Feature List =
    =========================
    
       * High performance and scalability. For highest performance module could be
        run without conntrack being enabled in kernel. Reported to be able to
        handle 10Gbit traffic with more than 1500000 pps with negligible server
        load (on S5500BC).
    
       * NetFlow v5, v9, and IPFIX are fully supported.
    
        Support of v9/IPFIX is adding flexibility to exporting of flow data
        plus greater visibility of traffic, letting export many additional fields
        besides what was possible in v5 era. Such as
       
       * IPv6 option headers, IPv4 options, TCP options, ethernet type, dot1q
        service and customer VLAN ids, MAC addresses, and
    
       * Full IPv6 support,
    
       * NAT translations events (from conntrack) using NetFlow Event Logging (NEL).
        This is standardized way for v9/IPFIXr, but module export such events even
        for v5 collectors via specially crafted pseudo-records.
    
       * Deterministic (systematic count-based), random and hash Flow Sampling.
        With appropriate differences in support of v5, v9, and IPFIX.
    
       * SNMP agent (for net-snmp) for remote management and monitoring.
    
       * Options Templates (v9/IPFIX) let export useful statistical,
        configurational, and informational records to collector.
        Such as metering, exporting, sampling stat and reliability stat, sampling
        configuration, network devices ifName, ifDescr list.
    
       * Tested to compile and work out of the box on Centos 5, 6, 7, Debian and
       * Ubuntu. Many vanilla Linux kernels since 2.6.18 up to the latest (as of
       * writing is 3.19) are supported and tested.
    
       * Module load time and run-time (via sysctl) configuration.
    
       * Flexibility in enabling features via ./configure script. This will let you
        disable features you don't need, which increase compatibility with custom
        kernels and performance.
    
       * SNMP-index translation rules, let convert meaningless and unstable
        interface indexes (ifIndex) to more meaningful numbering scheme.
    
       * Easy support for catching mirrored traffic with promisc option. Which is
        also supporting optional MPLS decapsulation and MPLS-aware NetFlow.
    
     
    #13
  14. gary miller

    gary miller New Member

    Joined:
    Jan 4, 2018
    Messages:
    2
    Likes Received:
    0
    Following, and I would like to know more about it.
     
    #14
  15. Gio

    Gio New Member

    Joined:
    Apr 8, 2017
    Messages:
    16
    Likes Received:
    1
    what kind of setup did you end up with?
     
    #15
  16. Cmdrd

    Cmdrd New Member

    Joined:
    Jun 23, 2016
    Messages:
    19
    Likes Received:
    2
    Just to throw my $0.02 in here for my homelab setup that works well. It's a bit of work to implement but allows for a lot of customization and is quite extensible.

    For monitoring and infrastructure stats collection I use Prometheus for the back-end with a Grafana front-end. Prometheus is super flexible, has a lot of power in it's querying language PromQL, and you can build exporters for pretty much everything which was a fun project to work on.

    In terms of *flow information, I have an ELK stack (Elasticsearch, Logstash, Kibana) for ingesting NetFlow and SFlow along with doing packet capture analysis. It scales out really well and blows tools like Wireshark out of the water for packet capture analysis after building out a good group of filters.
     
    #16
    Tha_14 likes this.
  17. hhp

    hhp New Member

    Joined:
    Aug 3, 2016
    Messages:
    18
    Likes Received:
    3
    I would suggest looking at LibreNMS, I monitor 10's of thousands of devices with it (entire ISP network), from the juniper core routers, cisco and every tik device made or very close to it to the CPE gear. Also: vmware from hypervisors to vcenter, many linux and win servers, some LTE backend, pretty much anything smnp capable. It is highly configurable, you can add MIB's for devices that are not in the release etc. It already uses RRD, weathermap etc and has many integrations. The UI for making custom network maps is still clunky but you can do it, I think its easier to just create and edit the files. I only skimmed the thread but it should cover most of what I saw mentioned.
     
    #17
  18. Robert Fontaine

    Robert Fontaine Active Member

    Joined:
    Jan 9, 2018
    Messages:
    108
    Likes Received:
    25
    I am likely going to go down the microsloth service center route as I need to spend some time learning the tool but I doubt I would pick it otherwise.
     
    #18
  19. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,490
    Likes Received:
    325
    Any good pointers on how to do that?
    The official docs are not compatible with my way of understanding in this part somehow;
    eg I wanted to add netgear mibs that would be capable to get the internal temperature but the system never picked them up...

    And network maps - the editor is very very basic but if you get into manually working on the files the result is really awesome
     
    #19
  20. hhp

    hhp New Member

    Joined:
    Aug 3, 2016
    Messages:
    18
    Likes Received:
    3
    Check the doc's on Observium, LibreNMS was forked from it after the lead had a tantrum. There are write ups etc on adding the MIBs, I havent had to add any in quite a while but its not difficult. From memory you need the MIB's from the manufacturer and need to put them in the MIB's folder.. past that its to fuzzy without reading up again.
     
    #20
Similar Threads: Homelab Monitoring
Forum Title Date
Software Stuff Homelab VMware backup options May 29, 2018
Software Stuff Homelab SIEM recommendations? Apr 10, 2017
Software Stuff Network Monitoring Advice Dec 17, 2017
Software Stuff pretty dashboard monitoring home lab vitals? Sep 10, 2017
Software Stuff Parental Control/Monitoring Solutions? Jun 30, 2017

Share This Page