What Is Log Collection? The Value of Unifying Your Logging

If you work in IT, especially on the decision-making side, then you should be aware of how vital logging is. When things go wrong with software in production, a solid logging approach often is the difference between hours of fruitless troubleshooting and an easy fix. Today, we’re here to talk about logging by covering a specific angle. Namely, we’ll answer the question, “What is log collection?”

Why is this an important question to answer? As it turns out, a typical, modern tech organization will have many types of logs, written to an equally large number of targets, or destinations. We’re here to argue that, although there’s certainly a reason and a place for each of these log types, you can also find benefits in unifying them. That’s what this post is about.

We’ll start with the basics. Before getting into log collection, we need to be on the same page regarding logging itself, so that’s where we’ll begin: by defining logging, explaining why this is such a big deal, and covering the benefits it provides. Then, we’ll proceed to cover the variety one can find in the logging world. We’ll cover the types of logs, explaining the reasons for each one.

Then, at last, we’ll get to the meat of the article. By answering the title question, we’ll explain why there is value to be found in unifying your various log sources. We part ways with some final considerations and tips on tools you can use to unify your logging approach.

Let’s get started.

Log Collection Basics

As promised, we’ll start with some definitions. The first one of them is a definition of logging itself. Feel free to skip ahead if you’re already experienced or are at least familiar with logging.

What Is Logging?

So, what is logging? In short, logging is the practice of creating and maintaining logs. A log is a recording of an event of interest that happened in a computer system or application.

It’s common to write log entries to text files on the disk, but that’s not the only available option. It just happens to be one of the most popular ones. As you’ll see later on this post, you can write logs to a variety of destinations, such as database tables, email, cloud services, and many more.

Since we’ve just covered the “what,” it’s time to get to the “why.” Why do people bother logging? What do they expect to get from it?

We Use Logging Because Tech Is Complex

Why bother logging? The TL;DR version is tech is complicated and things break often.

Now, for the slightly longer answer.

IT is complex. Let’s take software development as an example.

Developers write code, along with unit tests, to verify its behavior. When they’re done, they check in their changes to a CI server. The CI server will then build the application, resolving whatever dependencies that code requires, besides running the unit tests and maybe additional types of automated tests.

Depending on the type and size of the organization, the code might be handed to a QA department, which will then submit the application to myriad types of tests, both manual and automated. When they’re satisfied, the application might still need to go through an acceptance phase, in which a stakeholder gives their sign-off.

Even with all those verifications and tests done at each stage, there’s still no guarantee the application will work as intended when it gets to the production environment. When things do go wrong, the programmers assigned to fix the problem will need to know, among other things,

• The exact date and time the problem occurred.
• In which part of the application the problem happened.
• Details about the user session in the application.
• The stack trace of the error or exception that occurred.

And what would be a sure way to get these pieces of information where they need to be? You’ve got it: logging.

Logs in All Shapes and Sizes

In order to understand why it might make sense to unify your logs from various sources, you must first understand what all those sources we keep telling you about are. That’s what this section covers.

We’ll start by covering some of the different types of logs an IT organization might employ. After that, you’ll see some of the most often used destinations (or targets) for logs.

Types of Logs

What follows is a list of existing types of logs. It’s not an exhaustive list, but it features the most common types of logs you’re likely to find at your company.

Application Logs

Application logs are simply logs that are generated by the various applications you happen to use. Their main purpose is to allow maintenance programmers to perform an after-the-fact debugging session of the application. It’s sort of like a time machine: by reading through the log entries generated by the application, the developers can understand how it behaved in production and, hopefully, detect and fix whatever is causing it to malfunction.

Detecting errors isn’t the only thing application logs are good for, however. Logging can also help you answer questions that are of interest to the business, such as:

• What are the most and least used features?
• How many visitors does each page get per day (and other time periods)?
• How much time do visitors spend on each page or feature?

In short: application logs are useful to detect and fix problems and also to help the organization gather data and insights that can be useful to improve the application.

System Logs

System logs are another important type of log. Unlike application logs, system logs are generated by the operating system itself—hence the name.

Most entries you’ll find in the system log are written by the operating system itself, but applications can also add log entries there.

On Windows, the system logs are called the event log, which you can view through the Event Viewer application. If you’re using Linux, the system log consists of the Syslog service and its successors.

Finally, if you’re a macOSX user, you can access the system logs by using the Console app.

Web and Application Server Logs

The final log type we’re covering today is the log generated by an application or web server.

Web servers such as Apache, IIS, and NGINX generate log files that are essential to troubleshoot problems with websites or web applications, and the same is true for application servers.

Common Logging Destinations

You’ve just read about three of the most common types of logs. Now you’ll learn about some of the most common destinations or targets for logs.

Text Files

It’s very common to write logs to plain text files. Text files are very approachable. They’re cheap, easy to use, and ubiquitous. You don’t need fancy software to read them; a common text editor will do. Except for nuisances such as differences in end-of-line characters between Windows and *nix systems, text files are intrinsically cross-platform.

Database

Database tables are another common destination for logs. The main justification for picking a database as a target for logging would be the ease of querying and the other relational facilities, such as relational integrity.

Email

Emails are yet another interesting target for a log. By redirecting the log to a specialized email address, you could, for instance, automate the opening of tickets when a problem happens in production.

Cloud Services

An increasingly popular alternative is to integrate logging targets with cloud solutions. For instance, you could have a logging target that writes log messages to providers such as Amazon CloudWatch Logs, Azure Storage, or Google Logging.

Log Collection; AKA Why Unifying Your Logging Is Worth It

As you’ve seen so far, logging can take many forms. Logs vary in their types, formats, and destinations. They vary in their severity levels, intended target audience, and in how often they get written.

When you take all of that variety into account, it doesn’t take long until you realize that a comprehensive logging approach can be an incredible asset for the business. Analyzing a large number of log entries can be a powerful way of obtaining insights about pretty much all facets of an IT operation.

However, all of those precious insights might be out of your reach if your logs are all scattered through a thousand different places.

That’s when log collection comes in handy. By having an automated process that collects logs from varied sources and aggregates them into one place, you have the best of both worlds, so to speak. For starters, you don’t need to change anything to the way you currently do logging. In other words, it’s the log collection process that will adapt to your already existing logging approach and not the other way around.

And then you get all of the benefits of having your logs concentrated in a single place. Having all logs from the organization at your fingertips will allow you to obtain insights that wouldn’t be possible otherwise.

You could, for instance, analyze web server logs along with application logging, to determine whether there’s a correlation between the server’s performance and the time users spend on the application. That’s just an example, but the potential is limitless.

What Now?

Let’s do a quick summary of what we’ve seen today.

We’ve learned what logging is, why it’s important, and the benefits it brings to an organization. You’ve learned that logs come in many shapes and sizes, different by their types and destinations, among other properties.

Getting all these myriad types of logs together in one place gives you the ability to obtain insights that wouldn’t be possible to get otherwise. That process is what we call “log collection,” and today you’ve learned why it is valuable. Where to go now?

First, don’t stop learning. Continue reading articles, on this blog and elsewhere, to learn about logging and related topics. Also, research and learn about tools at your disposal. For instance, take a look at XpoLog’s offering, which is a full-fledged log management tool, powered by AI and completely automated.

Thanks for reading, and see you next time.