What Is Log Analytics? Find Important Patterns in Your Logs

If your work in IT, especially in the software field, then you’re almost certainly aware of logs. An application that doesn’t generate any type of logging would be unheard of, to say the least. What about “log analytics?” Do you know this term? Depending on your specific role, your knowledge of the subject could be at any point on the spectrum, from “passing familiarity” to “intimate knowledge.” This post targets the former end of said spectrum—we’ll explain what log analytics is and why your organization should care about it.

We start the post by offering a quick overview of logging and its importance. Feel free to skip this section if you’re well-acquainted with logging.

After that, we get into the log analytics part. We start by defining it and talking about some of its applications. Then, we proceed to explain how log analytics works and how your organization can profit from it. Let’s begin.

The Importance of Logging

Before we get into the nuts and bolts of log analytics, it’s important to take a step back and examine logging first. What is logging, and why should you care about it?

In short, logging consists of recording information about what your application does while it’s running. The idea is to have this documentation about your app’s behavior written to a persistent medium, so you can access it later. Why would anyone want to do that in the first place, though?

There are several reasons for that. The primary use for logging is to be able to perform what people sometimes call a “post-mortem debug.” That is, when—not if—your software fails in production, you want to be able to read through your log files in order to understand what went wrong. By carefully analyzing the log files, you should be able to reconstruct the scenario, “replaying” step after step until you hit the point when things went sour.

This is essentially a reactive use of logging, though. Using log entries to reconstruct failure scenarios, while useful, is no more than putting out fires. You’re reacting to something that happened. What if you could be one step ahead and avoid the fires altogether?

Stepping Up Your Game: Log Analytics

As it turns out, there are ways in which you can turn logging into a more proactive approach, so to speak. It’s possible to analyze your log entries, obtaining valuable insights from them. How do your users really use your application? What are the features they use the most? Which ones do they rarely touch? What times of day see more and less activity?

Log analytics can be of great helping answering these and more questions.

Defining Log Analytics

Even though we haven’t formally defined log analytics, it should be somewhat clear by this point what it’s about—analyzing log entries to find insights. It’s a short and precise definition. It’s also boring and unremarkable at that. Let me propose another definition then:

Log analytics means taking an active approaching to logging, instead of a passive/reactive one. It means analyzing log entries to better understand our applications, to be one step ahead and solve problems as quickly as possible—or prevent them before they happen.

So, log analytics manage to turn your logging approach into an active process, using the log entries to obtain useful insights for the organization. What are those insights? How can we apply log analytics in practice, and what do we get from it?

Applications of Log Analytics

We keep saying that log analytics can help you get insights from your log entries. But what kind of insights are those? What are the true benefits you can get by adopting log analytics? That’s what you’re going to see in this section—the application of log analytics.

The potential applications for log analytics are many. Determining trends is something that quickly comes to mind, but the technique can benefit other areas. Take security, for instance. Log analytics might help you detect protocol failure, network outages, and increase awareness about security as a whole.

But the business as a whole benefits from log analytics and the useful insights it can bring. What follows is a non-exhaustive list of the types of insights you can obtain with log analytics:

• Determine page popularity. Log analytics can help you determine the most visited pages on your web sites, as well as the time of day with the most visits. You can use data like this to measure the variation of the public over time.
• Detect suspicious occurrences. Log analytics could also help you detect unusual or suspicious events. For instance, a user of a payment service making a purchase with a value way above its average.
• Personalize the user’s experience. You could use data from logs to perform business analysis on a user’s behavior, and then use the results of such analysis to personalize their experience on your app.
• Improve security. Through a comprehensive analysis of the security logs, you can learn about invasion attempts and other malicious activities.
• Improve troubleshooting of issues, by helping you localize the root cause of different types of errors.

How Does Log Analytics Work: A Brief Overview

You’ll now see a quick overview of how log analytic works in practice. Keep in mind that this will be a high-level, general view—different tools can employ different approaches, and the details may vary. After this little disclaimer, we’re ready to go.

So, in order to perform your log analytics process, you first need to feed it with log files. The log analytics tool might start by collecting log files from a wide variety of sources and events. Security breaches, installation, user sessions—these events are just a fraction of the types of occurrences that usually leave logs behind.

After collecting log files, the tool should you start parsing/analyzing them. How it does that will depend on the information included in each log entry. At the bare minimum, this is what a log entry should record for the average application:

• A timestamp. There’s no use in recording an event without the date and time in which it happened.
• A logging level. Important to understand the severity of the event, for later filtering and searching.
• The log message. A description of the event of interest.
• The identification of the logged user.

The log analytics process consists of organizing the data found on the log entries. To do that, it employs techniques like classification and tagging, pattern recognition, artificial ignorance, and correlation analysis.

Classification and tagging are exactly what it sounds like—it classifies events into categories and assesses keywords to them. Pattern recognition consists of comparing new events to old ones and, that way, determining which events will be relevant. Artificial ignorance is a strategy that makes use of machine learning to decide which parts of a log file are noisy data and should be ignored. In other words, it’s a type of filter that helps eliminate useless data, making relevant events more noticeable.

There’s More to Logging Than Putting Out Fires

Log analytics is a technique that enables you to evolve your logging approach, putting it into use in an active (rather than reactive) way. If your organization just collects logs and then do nothing with them, you’re wasting a valuable resource. Failing to extract the insights hidden in your logs is like having a treasure buried in your property and not profiting from it.

Don’t waste your log efforts. Profit from them. Put log analytics to work for you ASAP. In this post, we’ve offered you an introduction, but what are the next steps?

Make an effort to have log entries that are not only parseable by machines, but human-readable as well. Learn about tools at your disposal that can make your life easier. Also, continue checking out this blog, which will continue to feature posts where you can learn a lot about everything log-related.

This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.