Adding Log Data to XpoLog Center 7

CHAPTER 6

XpoLog can collect log files and log data from many different sources.

These include:

  • Local log files from the XpoLog machine (both Linux and Windows servers)
  • Log files from remote Linux machines using SSH protocol (agent-less collection)
  • Syslog messages sent by different systems
  • Log files from remote Windows machines or Windows file shares
  • Windows Event Logs from remote Windows servers
  • Log files from Amazon S3 buckets
  • Data from Hadoop HDFS
  • Logs from third-party platforms like Google App Engine, Palo Alto or VMWare
  • Databases table data
  • Logstash messages sent to XpoLog HTTP listener

This chapter will show how to add a few different types of data to XpoLog.

Skip to chapter:

  1. Adding Log Data – Local Windows Event Files.
  2. Adding Remote Linux Log Files Over SSH.
  3. Adding MySQL Database Table
  4. Adding Files from S3 Bucket.

Adding Log Data – Local Windows Event Files

To collect Windows Event Logs from the same server where XpoLog Center 7 is running.

  • Open a browser window and navigate to the XpoLog Center 7 home page:
    Open a browser window and browse to the XpoLog Center 7 home page
  • Once the XpoLog Center 7 home page comes up, click on the link “ADD DATA” from the top right corner of the page and choose “ADD WINDOWS EVENT LOG” from the dropdown menu:

Adding Windows Event Logs to XpoLog

  • This starts the SOURCE AND COLLECTION SETTINGS From the “CONNECTION DETAILS” section, ensure Localhost is chosen.
  • Please note that for remote Windows hosts, you can specify IP address or DNS name.

Add Windows Event Logs Connection Details

  • In the “SELECT HOST” section, ensure “localhost” is selected in the “COLLECT THE FOLLOWING TYPES” text box, and select the checkboxes for “Application”, “System” and “Security”:

Select Host

  • Click “DONE”. This opens the “Log Collection Settings” window:

Log data collection settings

  • In the “SELECT PARENT LOG FOLDER” field, click “BROWSE” and then choose the “Windows Event Logs” folder from the pop-up window. After that, click “SELECT”:

Settings XpoLog to add log data

  • For the “COLLECTION POLICY”, keep “Default Collection Policy” selected.
  • Keep the App Tag field blank.
  • Click “SAVE AND CLOSE”.

Verify Added Log Data

To verify the remote server’s log data has been added:

  • Go to Folders and Logs screen, expand the “Windows Event Logs” folder:

verify that remote servers log data has been added

  • Double click on the Security Log to open it in the log viewer (log viewer will be covered in more detail in the next chapter).
  • Filter for “Audit Failure”:

Double click on the Security Log to open it in the Log Viewer

Adding Remote Linux Log Files Over SSH

XpoLog can collect log files from remote Linux servers using the SSH protocol.

These log files can be from the operating system itself or any other application running in the server.

XpoLog initiates the log collection process (as opposed to using a listener) and runs it in an agent-less manner.

In this setup, no application or process needs to be installed in the target server.

Let’s consider an Amazon EC2 Linux machine running an Nginx web server.

By default, Nginx creates two log files:

  • Access Log
  • Error Log

By default, these files are located under /var/log/nginx directory.

In this section, you will configure XpoLog to collect the Nginx log files from the remote machine

Prerequisite 1: Configure SSH Connection

To make agentless log collection work,

  • The remote Linux server needs to be accessible to XpoLog over a TCP/IP network
  • The XpoLog server should not have to use SSH port forwarding to connect to the remote server
  • SSH keys for the account running the XpoLog process should be configured to access the remote server

Prerequisite 2: Copy LogAway for Java v4.5

When XpoLog collects logs over SSH, it does not require any tools to be installed or running in the target machine.

This is the default behavior and works for most Linux systems.

Among other things, XpoLog uses the Unix “less” command on the remote server to collect the logs and sends over SSH.

Sometimes, however, the less command may not be available for remote SSH connections.

To address this issue, XpoLog uses a JAR file that can be placed in the SSH user’s home directory.

This JAR file is a passive agent, which means it does not run as a process or service in the target machine.

It runs only when requested to do so by the XpoLog server.

The agent is called LogAway for Java and it can be downloaded from XpoLog.

The uncompressed files of Log Away for Java should be placed under the remote user’s home directory in the target machine.

The code snippet below shows one such setup:

# ls -l /root

drwxrwxrwx 2 root root    70 Apr 22  2014 xpologAgent

-rw-r–r– 1 root root 92626 Mar 19 12:07 xpologAgent.tar.gz

# ls -l /root/xpologAgent

-rwxrwxrwx 1 root root   444 Apr 22  2014 runAgent.sh

-rwxrwxrwx 1 root root 76161 Mar 18  2013 xpologagent.jar

-rwxrwxrwx 1 root root 30272 Mar  7  2011 xpologbase.jar

Configure SSH Log Collection

Once the prerequisites have been met:

  • Open a browser window and navigate to the XpoLog Center 7 home page:
    Open a browser window and browse to the XpoLog Center 7 home page
  • Once the XpoLog Center 7 home page comes up, click on the link “ADD DATA” from the top right corner of the page and choose “ADD LOG” from the dropdown menu:

Configure SSH log collection

  • This starts the SOURCE AND COLLECTION SETTINGS From the data source types, click on the SSH icon:

adding log data - NGINX logs over SSH

  • In the next screen, click on the “NEW” button to start creating an SSH connection:

creating an SSH connection

  • In the “Add [Over SSH] account” section under “CONNECTION DETAILS”, provide the details for the SSH connection already created.
  • Note that there are no fields for SSH port forwarding:

adding NGINX logs to XpoLog for log analysis

  • Click the “VERIFY” button to check XpoLog can connect to the remote Nginx server. Click “SAVE” after the verification succeeds.
  • Click the “BROWSE” button under “LOG PATH OR DIRECTORY” section, then drill through the Nginx server’s file system until the log files are visible. Click on the access log file to select it:

Adding NGINX log data - Add_Log_Path

  • It’s also possible to define advanced log collection parameters in the “COLLECTION SETTINGS” section:

adding logs to XpoLog log analyzer

  • Leave this section and click the “ADD LOG” button. XpoLog will collect the Nginx access log and display it in the “ROW LOG SAMPLE” box:

Raw log sample

  • XpoLog will display the log pattern it thinks best represents the data. It will also show other patterns the user can choose:

active log pattern list

  • From here, users can write and add their own custom patterns, or choose one of the system-recommended patterns:

write and add own custom patterns, or choose one of the system-recommended patterns

At the bottom of the page, XpoLog will show the parsed data based on the pattern selected:

XpoLog shows the parsed data based on the pattern selected

  • In the following image, one of the suggested patterns has been dragged to the top of the active log pattern list:

 In the following image, one of the suggested patterns has been dragged to the top of the active log pattern list

  • The log file is now parsed in a much granular way:

 The log file is now parsed in a much granular way:

 The log file is now parsed in a much granular way

  • Click the “SAVE” button. This will open the “LOG COLLECTION SETTINGS” dialog box:

adding log data to XpoLog - Linux files, NGINX logs

  • Provide a descriptive name for the Nginx access log file.
  • For the parent log folder, click “BROWSE” and then create a new folder under the root folder. You can also create a new folder before adding the log:

create a new folder before adding the log

  • For the collection policy, choose the “Default Collection Policy”. You can also create a new collection policy before adding the log
  • Type relevant app tags for the log (e.g. “nginx”, “web server” etc.). You can also create new app tags here. App tags are separated by commas
  • Provide relevant log type tags. You can use system-recommended tags (XpoLog will display available log type tags once you start typing) or add own custom tags

adding NGINX log data to XpoLog

  • Click “SAVE & CLOSE” to complete the wizard.

Verify Added Log Data

To verify that Nginx log data has been added to XpoLog:

  • Double click on the log file name under the folder in the “Folders and Log Files” page:

adding Linux log data over SSH

view the added log data in the log viewer

Adding MySQL Database Table

XpoLog allows you to add data from relational database tables.

The added data is treated like structured log files with fixed-length fields.

The data can then be searched the same way as any other log file.

There are many systems which store important log events in structured data stores.

The ability to access and add data from such data sources is one of the core strengths of XpoLog as a log management solution.

In this section, you will see how XpoLog can be configured to access a MySQL database and add data from a table.

This demonstration will use a freely available MySQL sample database called “sakila” running in an AWS RDS MySQL instance. The instance name is rds-dev-mysql:

Adding MySQL Database Table

The database has a table called “customers”:

adding database log data for investigation, monitoring and analysis

To add the table’s data to XpoLog:

  • Click “ADD LOG” from the “ADD DATA” menu:

 Click “ADD LOG” from the “ADD DATA” menu

  • From the Add Data screen, click on Database:

 From the Add Data screen, click on Database

  • In the “CONNECTION DETAILS” section, click on “NEW

adding log data from database table

  • In the “Add [Database] account” section, choose MySQL:

	In the “Add [Database] account” section, choose MySQL

  • Specify the account details for the database instance:

	Specify the account details for the database instance

TIP: If the MySQL JDBC driver is not installed in the XpoLogs server, you will receive a prompt to install the driver. You can download the platform-independent MySQL Connector/J, uncompress it, and upload the binary from the XpoLog interface.

Once the connection is verified, click on “SAVE”.

The wizard goes back to the “Add Data” screen with the newly created connection selected.

In the “DB QUERY DEFINITION” box, type the SQL query to extract the data:

 Once the connection is verified, click on “SAVE”. The wizard goes back to the “Add Data” screen with the newly created connection selected. In the “DB QUERY DEFINITION” box, type the SQL query to extract the data

  • Click “VERIFY QUERY

The selected columns will be shown, and you will be asked for an ordering column.

Choose the appropriate field name from “Available Columns” and click on the arrow beside it to add it under “ORDER BY THIS COLUMNS” column:

add MySQL log data for log analysis and investigation

  • Click “DONE”.

XpoLog shows the raw data it has read from the table and displays the pattern it has used to parse the data:

View logs after adding them to the log analysis tool

  • Unless there is anything to correct, keep the pattern XpoLog has chosen and then click “SAVE” from the top right corner of the screen
  • The “Log Collection Settings” pop-up window is displayed.
  • From here, provide a name for the log (XpoLog considers the table data as a log), choose a parent log, accept the default log collection policy or create a new one and add app tags and log type tags:

choose a parent log, accept the default log collection policy or create a new one and add app tags and log type tags

  • Click “SAVE AND CLOSE”.

From XpoLog Manager, choose the “Folders and Logs” menu and expand the folder containing the MySQL data:

adding Linux log data using SSH connection

  • Double-clicking on the log will open it in the Log Viewer:

View the log data in XpoLog log viewer

The customer table data is now available in XpoLog just like any log data.

Adding Files from S3 Bucket

XpoLog can also import files from Amazon S3 buckets.

For this to work, the bucket needs to have an appropriate policy in place so the XpoLog node can access it.

To add data from S3:

Select “ADD LOG” from the “ADD DATA” menu:

How to add log data from AWS S3 buckets?

Click on “AWS S3 Bucket” icon:

how to add S3 log data to XpoLog

Select the S3 bucket account from the list:

Select the S3 bucket account from the list
This assumes you have already created an S3 account in XpoLog.  If you have not created one such account, you can click “NEW”, and then provide the AWS API credentials to create the account.

XpoLog will use the API keys to impersonate an IAM user to access the bucket.

The IAM user, in this case, needs to have at least read and list privileges on S3.

click “NEW”, and then provide the AWS API credentials to create the account.

Once you have created and/or selected the S3 account, click “BROWSE” in the “LOG PATH OR DIRECTORY” field and select the target S3 bucket: 

● Once you have created and/or selected the S3 account, click “BROWSE” in the “LOG PATH OR DIRECTORY” field and select the target S3 bucket

In the images below, we have chosen the XpoLog memory log file by double-clicking on a bucket called “xplg.lab.demo”, and then browsing through a folder by double-clicking on it:

In the images below, we have chosen the XpoLog memory log file by double-clicking on a bucket called “xplg.lab.demo”, and then browsing through a folder by double-clicking on it

step 1 - add s3 logs for log analysis

add AWS S3 log data for log analysis - step 2

Select the target file once you have traversed to it and click “ADD LOG”.

This will show a portion of the file’s contents as XpoLog reads it.

Also, the “ACTIVE LOG PATTERN LIST” field will show a log pattern  XpoLog has found by parsing the file.

Although we could add a custom pattern to tell XpoLog how to parse the file, in this particular case, we are accepting the default pattern:

Select the target file once you have traversed to it and click “ADD LOG”

 Click “SAVE” from the top left corner of the screen:

save

This will open the “Log Collection Settings” dialogue box.

Here, we can give the log file a meaningful name, specify a folder for it, create or choose a collection policy and also add one or more app tags and logtype tags:

Here, we can give the log file a meaningful name, specify a folder for it, create or choose a collection policy and also add one or more app tags and logtype tags

Click “SAVE & CLOSE” in the dialog box. This will create the log file under the folder:

SAVE and close in the dialog box. This will create the log file under the folder

Double-clicking the log file will now display the file’s contents in the XpoLog Log Viewer.

Conclusions 

Once logs are ingested into XpoLog, users can not only search them, they can also use XpoLogs’ log analysis apps (if one is available for the type of log) to get a better visualization of the data.  

All chapters: 

Read more about:

How to secure Amazon S3 buckets
View how raw data from the Champions League easily turn into insights
Troubleshoot your Windows task scheduler in 5 minutes! XPLG automated log analyzer and problem detection