Preparing XpoLog Log Management Center 7 for Data Load

CHAPTER 5

With XpoLog Center 7 installed and configured, users can start collecting logs from different systems.

Before doing this, there are some important points to remember.

* If you haven’t downloaded XpoLog log analysis tool download free here.

XpoLog can ingest, process, search and display:

  • Any plain-text log files such as syslog and other application log files. These files can be in any format such as custom-delimited, CSV, JSON or XML
  • Windows event logs (XpoLog Center 7 needs to be running on a Windows Server)
  • Data from database tables

For collecting logs and data, XpoLog can use one of the following “pull methods”:

  • Direct access to the local server (where XpoLog is running) log directories
  • Direct access to remote file shares using UNC path such as \\server_name\shared_folder\. This is can be used by XpoLog Center 7 running in a Windows machine
  • Direct access to mounted directories
  • SSH access to remote Linux or Unix machine log directories (agent-less collection)
  • JDBC connection to database instances
  • Integration with HDFS file system
  • Integration with AWS S3 buckets
  • Integration with Google App Engine
  • Access to remote XpoLog servers (when the remote XpoLog acts as a syslog server)

Log sources can also send data to (“push”) XpoLog Center 7 in real-time. XpoLog “listeners” are configured to capture such oncoming log data.

It is possible to ingest log files from a single log directory or multiple log directories to XpoLog. The methods for both are different.

This chapter will show a series of preparatory steps before adding source data.

These steps can be also done at the end of each data source configuration.

Creating Log Folders

Logs ingested into XpoLog Center 7 are logically arranged in an internal structure called folders which are like regular folders in a computer file system.

Folders can be used to logically arrange log files into different namespaces.

When adding logs, XpoLog gives the option to choose a logs folder.

If none is chosen, the logs directory is added under the XpoLog folders root.

Log folders can be created when adding a data source.

To pre-create folders, follow these steps:

  • Open a browser window and browse to the XpoLog Center 7 home page:

Nevigate to this address to access XpoLog

  • Once the XpoLog Center 7 home page comes up, click on the gear icon from the top right corner of the web page:

XpoLog Manager interface

  • This is will show the XpoLog Manager interface. From the left navigation panel, choose “Folders and Logs”:

XpoLog log analysis tool - adding data sources

  • This opens the “Folders and Logs” screen:

Creating Log Folders

  • Click “ADD” and then from the submenu choose “Folder”.

Adding data to XpoLog

  • In the “CREATE NEW FOLDER” screen, create a folder for MySQL as shown below. Make sure the folder is created under the root (“Folders and Logs”). Click “SAVE”.

create a folder for MySQL logs

  • Once created, the folder list should look like this:

 Once created, the folder list should look like this

  • Similarly, create three other folders:
  • one for remote server syslogs
  • one for database data
  • one for S3 log data

create the following folders to add data to XpoLog log analysis tool

Creating AppTags

AppTags allow users to attach metadata information to logs and other XpoLog components such as dashboards, folders or user accounts.

It helps to classify and group logs that belong to a source or application, which in turn can streamline search queries.

AppTags can be also used as a means of implementing security.

User accounts that have an AppTag attached to it will only have access to the components that belong to the same AppTag.

To create an AppTag:

  • Bring up the XpoLog Manager interface. From the left navigation panel, choose “TAGS”, and then “AppTags” from the submenu:

Bring up the XpoLog Manager interface. From the left navigation panel, choose “TAGS”, and then “AppTags” from the submenu

  • This shows the AppTags already created. The image below shows the AppTags XpoLog Center 7 ships with:

Create AppTags to classify and group logs that belong to a source or application, which in turn can streamline search queries

  • Click on the “ADD” button, this opens the “Create tag” screen.
  • From here, give a name and short description of the tag.
  • Note how the tag can be attached to different XpoLog components like folders, logs, environment tables, user accounts or saved searches.
  • For a new AppTag, provide only the name and description:

How to create an AppTag for MySQL logs?

  • Click “SAVE”. This will add the AppTag and it will be visible from the list.

Creating Log Collection Policy

An XpoLog log collection policy defines:

  • How often XpoLog will communicate with a data source and collect logs from it
  • Where will it save the log files
  • How long will it keep the log files online
  • Where will it archive the log files
  • When will it delete archived log files

XpoLog Center 7 comes with a default log collection policy, but users can create multiple collection policies and attach them to different data sources.

To create a log collection policy:

Bring up the XpoLog Manager interface. From the left navigation panel, choose “DATA”, then “Collection Policies” under “SETTINGS” from the submenu:

 Bring up the XpoLog Manager interface. From the left navigation panel, choose “DATA”, then “Collection Policies” under “SETTINGS” from the submenu

This opens the “Collection Policies” screen. By default, only the “Default Collection Policy” is listed:

opens the “Collection Policies” screen

Click on the “NEW COLLECTION POLICY” button. This opens the “Create Collection Policy” screen. From here, provide a name for the collection policy:

Click on the “NEW COLLECTION POLICY” button. This opens the “Create Collection Policy” screen

Select the XpoLog folder that will contain the logs:

selecting folder that will contain_the_log

From the “Storage Repository” section, specify where the logs would be saved in the XpoLog server and how long would they be kept there:

adding data to XpoLog 7 center

Define a schedule XpoLog will use to collect logs from the source:

defining log collection policy

To enable archiving, provide the following options from the “ARCHIVE LOCATION” section of the collection policy:

  • Enable archiving
  • Archive in a local folder or an S3 bucket
  • Archive path
  • How long to keep archive files before deleting
  • Checksum algorithm

Configuration of log archive in XpoLog log management

TIP: Log archiving is not a mandatory step for creating a collection policy.

However, this can help in more efficient storage of older, unnecessary logs from a busy network.

Unlike searchable, “current” data, archived logs are not indexed.

Non-indexed, archived data requires 9% of the disk space consumed by the raw ingested data, compared to 35% disk space required by searchable, indexed data.

Archived logs are stored in a compressed format by XpoLog, which also runs regular checksums on the data to make sure it has not been tampered with.

Attach an AppTag with the collection policy. This can be set later when creating a new AppTag:

log collection policy creation - XpoLog

Finally, select any other log collection endpoints to send logs to. Logs can be forwarded to:

  • A syslog server – In this case, XpoLog works as a log forwarder for syslog server
  • An HTTP endpoint: XpoLog sends logs to another application’s REST API endpoint
  • XpoLog forwarder: XpoLog sends data to another XpoLog instance or other SIEM / log management tools. Forwarded data can be filtered so not everything is sent across the network.

forward log data to XpoLog

  • Once all sections are configured, click “SAVE” on the top right corner. The new collection policy is then created.

Adding Listeners

Occasionally, it is neither possible nor desired for XpoLog to periodically connect to a remote data source or device and pull data across.

Instead, it may be easier for the target systems to “push” their data to XpoLog at regular intervals.

With this approach, you can create one or more listeners in XpoLog.

Listeners are port interfaces where applications and devices can send their payload using a protocol.

Once a listener is running, XpoLog ensures it’s “listening” on that port for any newly arrived data.

Any data captured on the port is then decoded and saved.

Currently, XpoLog listeners support the following typ3es of traffic types in its listeners:

  • Syslog over TCP (any port)
  • Syslog over UDP (any port)
  • HTTP or HTTPS (the default system HTTP/S ports)
  • XpoLog transport protocol
  • Apache Kafka
  • Cisco routers and switch using Netflow protocol

Creating a Listener

To add a listener:

From XpoLog Manager navigation menu, select “DATA”, then “Listen To Data”:

Listen to logs, use log data listeners for log data forwarding

From the Listeners Configuration Management screen, select “HTTP Listener”:

forward and collect log data using log data listeners

From the top right corner, click the “Add Account” button.

In the pop-up window, give a name to the new listener, and accept the default values for the token and the URL.

This is the URL the applications or devices will send their data to.

give a name to the new log listener

Click on “Advanced Settings

From the advanced settings fields:

  • Accept the parent folder name (it is the same as the new listener name) or select/create a folder where the received data will be stored
  • Accept the default collection policy or select a policy to be automatically applied to the data received on this listener
  • Accept the log name prefix or change it to a custom prefix
  • Specify if data received on this listener will be part of one single log, or if it will be split across multiple log files based on the source device
  • Accept all other values and click “Save

A newly created listener is displayed

The newly created listener will be displayed

log listeners creation

All chapters: 

Read more about:

How to secure Amazon S3 buckets
View how raw data from the Champions League easily turn into insights
Troubleshoot your Windows task scheduler in 5 minutes! XPLG automated log analyzer and problem detection