System Monitoring

Lately I’ve been experimenting with various different system monitoring, charting, alarming, etc solutions.

Some of the systems I’ve experimented with include:

  • Graphdat – Great upcoming product that provides an experience like no other.
  • Cacti – My current solution for monitoring internal SNMP systems.
  • Munin – Looked nice, but out-of-the box SNMP wasn’t so successful.
  • Graphene ontop of Graphite – Looks absolutely beautiful, but you essentially have to write (or find) your own data collectors for everything.
  • MRTG – my previous solution, and great for monitoring basic SNMP counters. Could no longer handle the size of my network storage – hence the search for a replacement.
  • New Relic – a little slow, and had some issues with their dashboards appearing / disappearing. No way to customize hostnames currently, which is annoying for anyone with even mild OCD

So far, I must say that the offering from GraphDat is simply amazing. Check it out for yourself below.

Adding SELinux policies for your apps

I recently had all sorts of nightmares trying to configure cacti to talk to haproxy, via an snmp perl-script. It turns out, the problem wasn’t the normal chmod fun, wrong paths, or anything like that. Instead it was a feature of some distros of Linux known as SELinux (Security Enhanced Linux).

The most confusing part, was that all the errors I was getting, were directing me at commands, and configs, that I’d already checked.

e.g.

snmpbulkwalk -c public -v2c 127.0.0.1 1.3.6.1.4.1.29385
# SNMPv2-SMI::enterprises.29385 = No Such Object available on this agent at this OID
 
perl /etc/snmp/haproxy.pl
# Warning: no access control information configured.
# It's unlikely this agent can serve any useful purpose in this state.
# Run "snmpconf -g basic_setup" to help you configure the Haproxy.conf file for this agent.

Eventually, after 8 hours of scouring the interwebs, I sent an email off to the developer asking for help. Of course, I managed to discover the problem less than an hour after sending said email.

Regardless, at the end of the day, the following was the magic bullet I used to fix my problem:

setenforce Permissive
rm /var/log/audit/audit.log
service auditd restart
[yourCommand]
cat /var/log/audit/audit.log | audit2allow -M [filename]
semodule -i [filename].pp
setenforce Enforcing