One of the best parts of an Elasticsearch, Logstash and Kibana (ELK) deployment is the ability the visualise data parsed from logfiles and metrics gathered on hosts. Through the use of the filebeat plugin we can slurp log files on a CloudForms host and push them straight to logstash for ingest and eventual searching through Kibana.
Except we don’t want to just naively gather up evm.log and push it to logstash. Sure, we can search for key phrases that we’re interested in, but the evm.log contains a wealth of useful data regarding the health of the cluster.
To extract this information, I wrote a few Grok filters for logstash that pluck key metrics out of evm.log and ingest it into Elasticsearch.