GUIDE 6

Adding Log Data to PortX/XpoLog

XPLG Log Management Product Suite

This Guide Covers Adding Data (Data Collection)  to XPLG log Management Product Suite, PortX/XpoLog from different sources, and the collection of various data types.
Installing PortX/XpoLog is simple as shown in the Installation QuickStart Guides
(Find the latest product version  XPLG Free Download.)

PorX / XpoLog can collect log files and log data from many different sources.

These include:

  • Local log files from the PortX/XpoLog machine (both Linux and Windows servers)
  • Log files from remote Linux machines using SSH protocol (agent-less collection)
  • Syslog messages sent by different systems
  • Log files from remote Windows machines or Windows file shares
  • Windows Event Logs from remote Windows servers
  • Log files from Amazon S3 buckets
  • Data from Hadoop HDFS
  • Logs from third-party platforms like Google App Engine, Palo Alto or VMWare
  • Databases table data
  • Logstash messages sent to PortX/XpoLog HTTP listener

This chapter will show how to add or collect different types of data to PortX/XpoLog.

Adding Log Data
Local Windows Event Files

When PortX/XpoLog is deployed on a windows server, it can collect Windows Event Logs from the same server where PortX/XpoLog is running.

  • Open a browser window and navigate to the XPLG Product Suite home page:
    Open a browser window and browse to the XpoLog Center 7 home page
  • Click on the “ADD DATA” button from the top right corner of the page and choose “ADD WINDOWS EVENT LOG” from the dropdown menu:

Adding Windows Event Logs to XpoLog

  • Ensure “Localhost” is chosen under the “CONNECTION DETAILS” section.
  • Please note that for remote Windows hosts, you can specify IP address or DNS name.

Add Windows Event Logs Connection Details

  • In the “SELECT HOST” section, ensure “localhost” is selected in the “COLLECT THE FOLLOWING TYPES” text box, and select the checkboxes for “Application”, “System” and “Security”:

Select Host

  • Click “DONE”.
  • This will open the “Log Collection Settings” window:

Log data collection settings

  • In the “SELECT PARENT LOG FOLDER” field, click “BROWSE” and then choose the “Windows Event Logs” folder from the pop-up window. After that, click “SELECT”:

Settings XpoLog to add log data

  • For the “COLLECTION POLICY”, keep “Default Collection Policy” selected.
  • Keep the App Tag field blank.
  • Click “SAVE AND CLOSE

Verify Added Log Data

To verify the remote server’s log data has been added:

  • Go to the Folders and Logs screen, expand the “Windows Event Logs” folder:

  • Double click on the Security Log to open it in the log viewer (log viewer will be covered in more detail in the next chapter).
  • Filter for “Audit Failure”:

Double click on the Security Log to open it in the Log Viewer

Adding Remote Linux Log Files Over SSH

PortX/XpoLog can collect log files from remote Linux servers using the SSH protocol.

These log files can be from the operating system itself or any other application running in the server.

PortX/XpoLog initiates the log collection process (as opposed to using a listener) and runs it in an agent-less manner.

In this setup, no application or process needs to be installed in the target server.

Let’s consider an Amazon EC2 Linux machine running an Nginx web server.

By default, Nginx creates two log files:

    • Access Log
    • Error Log

By default, these files are located under /var/log/nginx directory.

In this section, you will configure PortX/XpoLog to collect the Nginx log files from the remote machine

Prerequisite 1: Configure SSH Connection

To make agentless log collection work,

    • The remote Linux server needs to be accessible to PortX/XpoLog over a TCP/IP network
    • The PortX/XpoLog server should not have to use SSH port forwarding to connect to the remote server
    • SSH keys for the account running the PortX/XpoLog process should be configured to access the remote server

Prerequisite 2: Copy LogAway for Java v4.5

When PortX/XpoLog collects logs over SSH, it does not require any tools to be installed or running in the target machine.

This is the default behavior and works for most Linux systems.

Among other things, PortX/XpoLog uses the Unix “less” command on the remote server to collect the logs and sends over SSH.

Sometimes, however, the less command may not be available for remote SSH connections.

To address this issue, PortX/XpoLog uses a JAR file that can be placed in the SSH user’s home directory.

This JAR file is a passive agent, which means it does not run as a process or service in the target machine.

It runs only when requested to do so by the XpoLog server.

The agent is called LogAway for Java and it can be downloaded from PortX / XpoLog.

The uncompressed files of Log Away for Java should be placed under the remote user’s home directory in the target machine.

The code snippet below shows one such setup:

# ls -l /root

drwxrwxrwx 2 root root    70 Apr 22  2014 xpologAgent

-rw-r–r– 1 root root 92626 Mar 19 12:07 xpologAgent.tar.gz

# ls -l /root/xpologAgent

-rwxrwxrwx 1 root root   444 Apr 22  2014 runAgent.sh

-rwxrwxrwx 1 root root 76161 Mar 18  2013 xpologagent.jar

-rwxrwxrwx 1 root root 30272 Mar  7  2011 xpologbase.jar

Configure SSH Log Collection

Once the prerequisites have been met:

  • Browser to the XPLG Product Suite home page:
    Open a browser window and browse to the XpoLog Center 7 home page
  • Click on the link “ADD DATA” from the top right corner of the page and choose “ADD LOG” from the dropdown menu:

Configure SSH log collection

  • This starts the SOURCE AND COLLECTION SETTINGS From the data source types, click on the SSH icon:

adding log data - NGINX logs over SSH

  • In the next screen, click on the “NEW” button to start creating an SSH connection:

creating an SSH connection

  • In the “Add [Over SSH] account” section under “CONNECTION DETAILS”, provide the details for the SSH connection already created.
  • Note that there are no fields for SSH port forwarding:

adding NGINX logs to XpoLog for log analysis

  • Click the “VERIFY” button to check PortX/XpoLog can connect to the remote Nginx server.
    Click “SAVE” after the verification succeeds.
  • Click the “BROWSE” button under “LOG PATH OR DIRECTORY” section, then drill through the Nginx server’s file system until the log files are visible. Click on the access log file to select it:

Adding NGINX log data - Add_Log_Path

  • It’s also possible to define advanced log collection parameters in the “COLLECTION SETTINGS” section:

adding logs to XpoLog log analyzer

  • Leave this section and click the “ADD LOG” button. PortX/XpoLog will collect the Nginx access log and display it in the “ROW LOG SAMPLE” box:

Raw log sample

  • PortX/XpoLog will display the log pattern it thinks best represents the data. It will also show other patterns the user can choose:

active log pattern list

  • From here, users can write and add their own custom patterns, or choose one of the system-recommended patterns:

write and add own custom patterns, or choose one of the system-recommended patterns

At the bottom of the page, PortX/XpoLog will show the parsed data based on the pattern selected:

XpoLog shows the parsed data based on the pattern selected

  • In the following image, one of the suggested patterns has been dragged to the top of the active log pattern list:

 In the following image, one of the suggested patterns has been dragged to the top of the active log pattern list

  • The log file is now parsed in a much granular way:

 The log file is now parsed in a much granular way:

 The log file is now parsed in a much granular way

  • Click the “SAVE” button. This will open the “LOG COLLECTION SETTINGS” dialog box:

adding log data to XpoLog - Linux files, NGINX logs

  • Provide a descriptive name for the Nginx access log file.
  • For the parent log folder, click “BROWSE” and then create a new folder under the root folder. You can also create a new folder before adding the log:

create a new folder before adding the log

  • For the collection policy, choose the “Default Collection Policy”. You can also create a new collection policy before adding the log
  • Type relevant app tags for the log (e.g. “nginx”, “web server” etc.). You can also create new app tags here. App tags are separated by commas
  • Provide relevant log type tags. You can use system-recommended tags (PortX/XpoLog will display available log type tags once you start typing) or add own custom tags

adding NGINX log data to XpoLog

  • Click “SAVE & CLOSE” to complete the wizard.

Verify Added Log Data

To verify that Nginx log data has been added to PortX/XpoLog:

  • Double click on the log file name under the folder in the “Folders and Log Files” page:

adding Linux log data over SSH

view the added log data in the log viewer

Adding MySQL Database Table

PortX/XpoLog allows you to add data from relational database tables.

The added data is treated like structured log files with fixed-length fields.

The data can then be searched the same way as any other log file.

There are many systems which store important log events in structured data stores.

The ability to access and add data from such data sources is one of the core strengths of PortX/XpoLog as a log management solution.

In this section, you will see how PortX/XpoLog can be configured to access a MySQL database and add data from a table.

This demonstration will use a freely available MySQL sample database called “sakila” running in an AWS RDS MySQL instance. The instance name is rds-dev-mysql:

Adding MySQL Database Table

The database has a table called “customers”:

adding database log data for investigation, monitoring and analysis

To add the table’s data to PortX/XpoLog:

  • Click “ADD LOG” from the “ADD DATA” menu:

 Click “ADD LOG” from the “ADD DATA” menu

  • From the Add Data screen, click on Database:

 From the Add Data screen, click on Database

  • In the “CONNECTION DETAILS” section, click on “NEW

adding log data from database table

  • In the “Add [Database] account” section, choose MySQL:

 In the “Add [Database] account” section, choose MySQL

  • Specify the account details for the database instance:

 Specify the account details for the database instance

TIP: If the MySQL JDBC driver is not installed in the PortX/XpoLogs server, you will receive a prompt to install the driver. You can download the platform-independent MySQL Connector/J, uncompress it, and upload the binary from the PortX/XpoLog interface.

Once the connection is verified, click on “SAVE”.

The wizard goes back to the “Add Data” screen with the newly created connection selected.

In the “DB QUERY DEFINITION” box, type the SQL query to extract the data:

 Once the connection is verified, click on “SAVE”. The wizard goes back to the “Add Data” screen with the newly created connection selected. In the “DB QUERY DEFINITION” box, type the SQL query to extract the data

  • Click “VERIFY QUERY

The selected columns will be shown, and you will be asked for an ordering column.

Choose the appropriate field name from “Available Columns” and click on the arrow beside it to add it under “ORDER BY THIS COLUMNS” column:

add MySQL log data for log analysis and investigation

  • Click “DONE”.

PortX/XpoLog shows the raw data it has read from the table and displays the pattern it has used to parse the data:

View logs after adding them to the log analysis tool

  • Unless there is anything to correct, keep the pattern PortX/XpoLog has chosen and then click “SAVE” from the top right corner of the screen
  • The “Log Collection Settings” pop-up window is displayed.
  • From here, provide a name for the log (PortX/XpoLog considers the table data as a log), choose a parent log, accept the default log collection policy or create a new one and add app tags and log type tags:

choose a parent log, accept the default log collection policy or create a new one and add app tags and log type tags

  • Click “SAVE AND CLOSE”.

From PortX/XpoLog Manager, choose the “Folders and Logs” menu and expand the folder containing the MySQL data:

  • Double-clicking on the log will open it in the Log Viewer:

View the log data in XpoLog log viewer

The customer table data is now available in PortX/XpoLog just like any log data.

Adding Files from S3 Bucket

PortX/XpoLog can also import files from Amazon S3 buckets.

For this to work, the bucket needs to have an appropriate policy in place so the PortX/XpoLog node can access it.

To add data from S3:

Select “ADD LOG” from the “ADD DATA” menu:

How to add log data from AWS S3 buckets?

Click on “AWS S3 Bucket” icon:

how to add S3 log data to XpoLog

Select the S3 bucket account from the list:

Select the S3 bucket account from the list
This assumes you have already created an S3 account in PortX/XpoLog.  If you have not created one such account, you can click “NEW”, and then provide the AWS API credentials to create the account.

PortX/XpoLog will use the API keys to impersonate an IAM user to access the bucket.

The IAM user, in this case, needs to have at least read and list privileges on S3.

click “NEW”, and then provide the AWS API credentials to create the account.

Once you have created and/or selected the S3 account, click “BROWSE” in the “LOG PATH OR DIRECTORY” field and select the target S3 bucket: 

● Once you have created and/or selected the S3 account, click “BROWSE” in the “LOG PATH OR DIRECTORY” field and select the target S3 bucket

In the images below, we have chosen the PortX/XpoLog memory log file by double-clicking on a bucket called “xplg.lab.demo”, and then browsing through a folder by double-clicking on it:

In the images below, we have chosen the XpoLog memory log file by double-clicking on a bucket called “xplg.lab.demo”, and then browsing through a folder by double-clicking on it

step 1 - add s3 logs for log analysis

add AWS S3 log data for log analysis - step 2

Select the target file once you have traversed to it and click “ADD LOG”.

This will show a portion of the file’s contents as PortX/XpoLog reads it.

Also, the “ACTIVE LOG PATTERN LIST” field will show a log pattern  PortX/XpoLog has found by parsing the file.

Although we could add a custom pattern to tell PortX/XpoLog how to parse the file, in this particular case, we are accepting the default pattern:

Select the target file once you have traversed to it and click “ADD LOG”

 Click “SAVE” from the top left corner of the screen:

save

This will open the “Log Collection Settings” dialogue box.

Here, we can give the log file a meaningful name, specify a folder for it, create or choose a collection policy and also add one or more app tags and logtype tags:

Here, we can give the log file a meaningful name, specify a folder for it, create or choose a collection policy and also add one or more app tags and logtype tags

Click “SAVE & CLOSE” in the dialog box. This will create the log file under the folder:

SAVE and close in the dialog box. This will create the log file under the folder

Double-clicking the log file will now display the file’s contents in the PortX/XpoLog Log Viewer.

Conclusions 

Once logs are ingested into PortX/XpoLog, users can not only search them, they can also use PortX/XpoLog XPLG Products Suites” log analysis apps (if one is available for the type of log) to get a better visualization of the data.  

XPLG Products Suite QuickStart Guides: 

Read more about:

Aquick introduction, logstash plugins, filters, logstash inputs, output, logstash alternatives