PorX / XpoLog can collect log files and log data from many different sources.

These include:

• Local log files from the PortX/XpoLog machine (both Linux and Windows servers)
• Log files from remote Linux machines using SSH protocol (agent-less collection)
• Syslog messages sent by different systems
• Log files from remote Windows machines or Windows file shares
• Windows Event Logs from remote Windows servers
• Log files from Amazon S3 buckets
• Data from Hadoop HDFS
• Logs from third-party platforms like Google App Engine, Palo Alto or VMWare
• Databases table data
• Logstash messages sent to PortX/XpoLog HTTP listener

This chapter will show how to add or collect different types of data to PortX/XpoLog.

Adding Log Data
Local Windows Event Files

When PortX/XpoLog is deployed on a windows server, it can collect Windows Event Logs from the same server where PortX/XpoLog is running.

• Open a browser window and navigate to the XPLG Product Suite home page:
  
• Click on the “ADD DATA” button from the top right corner of the page and choose “ADD WINDOWS EVENT LOG” from the dropdown menu:

• Ensure “Localhost” is chosen under the “CONNECTION DETAILS” section.
• Please note that for remote Windows hosts, you can specify IP address or DNS name.

• In the “SELECT HOST” section, ensure “localhost” is selected in the “COLLECT THE FOLLOWING TYPES” text box, and select the checkboxes for “Application”, “System” and “Security”:

• Click “DONE”.
• This will open the “Log Collection Settings” window:

• In the “SELECT PARENT LOG FOLDER” field, click “BROWSE” and then choose the “Windows Event Logs” folder from the pop-up window. After that, click “SELECT”:

• For the “COLLECTION POLICY”, keep “Default Collection Policy” selected.
• Keep the App Tag field blank.
• Click “SAVE AND CLOSE”

Verify Added Log Data

To verify the remote server’s log data has been added:

• Go to the Folders and Logs screen, expand the “Windows Event Logs” folder:

• Double click on the Security Log to open it in the log viewer (log viewer will be covered in more detail in the next chapter).
• Filter for “Audit Failure”:

Adding Remote Linux Log Files Over SSH

PortX/XpoLog can collect log files from remote Linux servers using the SSH protocol.

These log files can be from the operating system itself or any other application running in the server.

PortX/XpoLog initiates the log collection process (as opposed to using a listener) and runs it in an agent-less manner.

In this setup, no application or process needs to be installed in the target server.

Let’s consider an Amazon EC2 Linux machine running an Nginx web server.

By default, Nginx creates two log files:

• • Access Log
  • Error Log

By default, these files are located under /var/log/nginx directory.

In this section, you will configure PortX/XpoLog to collect the Nginx log files from the remote machine

Prerequisite 1: Configure SSH Connection

To make agentless log collection work,

• • The remote Linux server needs to be accessible to PortX/XpoLog over a TCP/IP network
  • The PortX/XpoLog server should not have to use SSH port forwarding to connect to the remote server
  • SSH keys for the account running the PortX/XpoLog process should be configured to access the remote server

Prerequisite 2: Copy LogAway for Java v4.5

When PortX/XpoLog collects logs over SSH, it does not require any tools to be installed or running in the target machine.

This is the default behavior and works for most Linux systems.

Among other things, PortX/XpoLog uses the Unix “less” command on the remote server to collect the logs and sends over SSH.

Sometimes, however, the less command may not be available for remote SSH connections.

To address this issue, PortX/XpoLog uses a JAR file that can be placed in the SSH user’s home directory.

This JAR file is a passive agent, which means it does not run as a process or service in the target machine.

It runs only when requested to do so by the XpoLog server.

The agent is called LogAway for Java and it can be downloaded from PortX / XpoLog.

The uncompressed files of Log Away for Java should be placed under the remote user’s home directory in the target machine.

The code snippet below shows one such setup:

Configure SSH Log Collection

Once the prerequisites have been met:

• Browser to the XPLG Product Suite home page:
  
• Click on the link “ADD DATA” from the top right corner of the page and choose “ADD LOG” from the dropdown menu:

• This starts the SOURCE AND COLLECTION SETTINGS From the data source types, click on the SSH icon:

• In the next screen, click on the “NEW” button to start creating an SSH connection:

• In the “Add [Over SSH] account” section under “CONNECTION DETAILS”, provide the details for the SSH connection already created.
• Note that there are no fields for SSH port forwarding:

• Click the “VERIFY” button to check PortX/XpoLog can connect to the remote Nginx server.
  Click “SAVE” after the verification succeeds.
• Click the “BROWSE” button under “LOG PATH OR DIRECTORY” section, then drill through the Nginx server’s file system until the log files are visible. Click on the access log file to select it:

• It’s also possible to define advanced log collection parameters in the “COLLECTION SETTINGS” section:

• Leave this section and click the “ADD LOG” button. PortX/XpoLog will collect the Nginx access log and display it in the “ROW LOG SAMPLE” box:

• PortX/XpoLog will display the log pattern it thinks best represents the data. It will also show other patterns the user can choose:

• From here, users can write and add their own custom patterns, or choose one of the system-recommended patterns:

At the bottom of the page, PortX/XpoLog will show the parsed data based on the pattern selected:

• In the following image, one of the suggested patterns has been dragged to the top of the active log pattern list:

• The log file is now parsed in a much granular way:

• Click the “SAVE” button. This will open the “LOG COLLECTION SETTINGS” dialog box:

• Provide a descriptive name for the Nginx access log file.
• For the parent log folder, click “BROWSE” and then create a new folder under the root folder. You can also create a new folder before adding the log:

• For the collection policy, choose the “Default Collection Policy”. You can also create a new collection policy before adding the log
• Type relevant app tags for the log (e.g. “nginx”, “web server” etc.). You can also create new app tags here. App tags are separated by commas
• Provide relevant log type tags. You can use system-recommended tags (PortX/XpoLog will display available log type tags once you start typing) or add own custom tags

• Click “SAVE & CLOSE” to complete the wizard.

Verify Added Log Data

To verify that Nginx log data has been added to PortX/XpoLog:

• Double click on the log file name under the folder in the “Folders and Log Files” page:

• This will show the log file in PortX/XpoLog Log Viewer:

Adding MySQL Database Table

PortX/XpoLog allows you to add data from relational database tables.

The added data is treated like structured log files with fixed-length fields.

The data can then be searched the same way as any other log file.

There are many systems which store important log events in structured data stores.

The ability to access and add data from such data sources is one of the core strengths of PortX/XpoLog as a log management solution.

In this section, you will see how PortX/XpoLog can be configured to access a MySQL database and add data from a table.

This demonstration will use a freely available MySQL sample database called “sakila” running in an AWS RDS MySQL instance. The instance name is rds-dev-mysql:

The database has a table called “customers”:

To add the table’s data to PortX/XpoLog:

• Click “ADD LOG” from the “ADD DATA” menu:

• From the Add Data screen, click on Database:

• In the “CONNECTION DETAILS” section, click on “NEW”

• In the “Add [Database] account” section, choose MySQL:

• Specify the account details for the database instance:

TIP: If the MySQL JDBC driver is not installed in the PortX/XpoLogs server, you will receive a prompt to install the driver. You can download the platform-independent MySQL Connector/J, uncompress it, and upload the binary from the PortX/XpoLog interface.

Once the connection is verified, click on “SAVE”.

The wizard goes back to the “Add Data” screen with the newly created connection selected.

In the “DB QUERY DEFINITION” box, type the SQL query to extract the data:

• Click “VERIFY QUERY”

The selected columns will be shown, and you will be asked for an ordering column.

Choose the appropriate field name from “Available Columns” and click on the arrow beside it to add it under “ORDER BY THIS COLUMNS” column:

• Click “DONE”.

PortX/XpoLog shows the raw data it has read from the table and displays the pattern it has used to parse the data:

• Unless there is anything to correct, keep the pattern PortX/XpoLog has chosen and then click “SAVE” from the top right corner of the screen
• The “Log Collection Settings” pop-up window is displayed.
• From here, provide a name for the log (PortX/XpoLog considers the table data as a log), choose a parent log, accept the default log collection policy or create a new one and add app tags and log type tags:

• Click “SAVE AND CLOSE”.

From PortX/XpoLog Manager, choose the “Folders and Logs” menu and expand the folder containing the MySQL data:

• Double-clicking on the log will open it in the Log Viewer:

The customer table data is now available in PortX/XpoLog just like any log data.

Adding Files from S3 Bucket

PortX/XpoLog can also import files from Amazon S3 buckets.

For this to work, the bucket needs to have an appropriate policy in place so the PortX/XpoLog node can access it.

To add data from S3:

Select “ADD LOG” from the “ADD DATA” menu:

Click on “AWS S3 Bucket” icon:

Select the S3 bucket account from the list:

This assumes you have already created an S3 account in PortX/XpoLog.  If you have not created one such account, you can click “NEW”, and then provide the AWS API credentials to create the account.

PortX/XpoLog will use the API keys to impersonate an IAM user to access the bucket.

The IAM user, in this case, needs to have at least read and list privileges on S3.

Once you have created and/or selected the S3 account, click “BROWSE” in the “LOG PATH OR DIRECTORY” field and select the target S3 bucket: 

In the images below, we have chosen the PortX/XpoLog memory log file by double-clicking on a bucket called “xplg.lab.demo”, and then browsing through a folder by double-clicking on it:

Select the target file once you have traversed to it and click “ADD LOG”.

This will show a portion of the file’s contents as PortX/XpoLog reads it.

Also, the “ACTIVE LOG PATTERN LIST” field will show a log pattern  PortX/XpoLog has found by parsing the file.

Although we could add a custom pattern to tell PortX/XpoLog how to parse the file, in this particular case, we are accepting the default pattern:

 Click “SAVE” from the top left corner of the screen:

This will open the “Log Collection Settings” dialogue box.

Here, we can give the log file a meaningful name, specify a folder for it, create or choose a collection policy and also add one or more app tags and logtype tags:

Click “SAVE & CLOSE” in the dialog box. This will create the log file under the folder:

Double-clicking the log file will now display the file’s contents in the PortX/XpoLog Log Viewer.