Kibana and LogStash — Visualizing Application Behavior

The analyzing of log files remains an unheralded, yet important, part of the software development and QA process. Bugs, performance problems, and other related issues are typically found by parsing the information contained within an application or server log. Unfortunately, this task can be tedious and time consuming, as developers and QA engineers struggle with finding something actionable out of a mass of textual data.

Two applications used together make dealing with log files an easier task. Kibana is a visualization plugin used with the Elasticsearch open source search engine. LogStash is another Elasticsearch plugin designed to process a wide range of log data formats. Together, they facilitate the visualization of any application’s performance data.

Let’s see if the two make sense for your software development team.

The Kibana Visualization Tool

Kibana provides a fully-featured set of graphing and other visualization functionality inside of Elasticsearch. It helps analysts — including developers — make sense of any kind of structured or unstructured data. A whole host of graph formats are possible depending on the scenario, including bar graphs, pie charts, line and scatter plots, maps, and more.

The tool makes it possible to publish meaningful reports on application performance able to be read and understood by everyone from a QA engineer to a business stakeholder.

Make Sense of your Log Data with LogStash

The folks at Elastic call LogStash a “data pipeline” used for handling log files and other application event data sources. No matter your specific app, the underlying platform, and its event log data format, LogStash converts that data into something able to be easily analyzed. It even provides a mechanism for using programmatic logic to parse unique log formats; greatly adding to the tool’s extensibility.

Of course, as an open source tool, LogStash boasts a robust community of users who are constantly creating new plugins to handle different event data formats. This is helpful if your team doesn’t want to spend the time building its own plugins. LogStash is also highly scalable, which is vital in today’s Cloud-based computing world.

Kibana and LogStash — a Match made in Heaven

After Elastic developed LogStash, the need for a way to visualize and report on its data quickly became apparent. This use-case drove the creation of Kibana as the tool for this purpose. The two plugins are meant to be used together as part of Elasticsearch.

Kibana provides a browser-based interface offering ease of use when searching for event data, visualizing the found information, and also creating new graphs and reports as necessary. The tool also makes the sharing of visualizations simple — something necessary in a collaborative DevOps environment. Elasticsearch’s powerful analytics engine is also accessible from Kibana, allowing the transformation of data to fit any reporting need.

Needless to say, if your shop is responsible for the development and maintenance of a variety of public-facing applications, you understand the need for the speedy analysis of event data. Kibana, LogStash, and Elasticsearch need to be considered as an important addition to your team’s QA toolbox

Stay tuned to the Betica Blog for more dispatches from the worlds of QA and software development.