Adding Log Data to XpoLog Center 7
XpoLog can collect log files and log data from many different sources.
- Local log files from the XpoLog machine (both Linux and Windows servers)
- Log files from remote Linux machines using SSH protocol (agent-less collection)
- Syslog messages sent by different systems
- Log files from remote Windows machines or Windows file shares
- Windows Event Logs from remote Windows servers
- Log files from Amazon S3 buckets
- Data from Hadoop HDFS
- Logs from third-party platforms like Google App Engine, Palo Alto or VMWare
- Databases table data
- Logstash messages sent to XpoLog HTTP listener
This chapter will show how to add a few different types of data to XpoLog.
Skip to chapter:
Adding Log Data – Local Windows Event Files
To collect Windows Event Logs from the same server where XpoLog Center 7 is running.
- Open a browser window and navigate to the XpoLog Center 7 home page:
- Once the XpoLog Center 7 home page comes up, click on the link “ADD DATA” from the top right corner of the page and choose “ADD WINDOWS EVENT LOG” from the dropdown menu:
- This starts the SOURCE AND COLLECTION SETTINGS From the “CONNECTION DETAILS” section, ensure Localhost is chosen.
- Please note that for remote Windows hosts, you can specify IP address or DNS name.
- In the “SELECT HOST” section, ensure “localhost” is selected in the “COLLECT THE FOLLOWING TYPES” text box, and select the checkboxes for “Application”, “System” and “Security”:
- Click “DONE”. This opens the “Log Collection Settings” window:
- In the “SELECT PARENT LOG FOLDER” field, click “BROWSE” and then choose the “Windows Event Logs” folder from the pop-up window. After that, click “SELECT”:
- For the “COLLECTION POLICY”, keep “Default Collection Policy” selected.
- Keep the App Tag field blank.
- Click “SAVE AND CLOSE”.
Adding Remote Linux Log Files Over SSH
XpoLog can collect log files from remote Linux servers using the SSH protocol.
These log files can be from the operating system itself or any other application running in the server.
XpoLog initiates the log collection process (as opposed to using a listener) and runs it in an agent-less manner.
In this setup, no application or process needs to be installed in the target server.
Let’s consider an Amazon EC2 Linux machine running an Nginx web server.
By default, Nginx creates two log files:
- Access Log
- Error Log
By default, these files are located under /var/log/nginx directory.
In this section, you will configure XpoLog to collect the Nginx log files from the remote machine
Prerequisite 1: Configure SSH Connection
To make agentless log collection work,
- The remote Linux server needs to be accessible to XpoLog over a TCP/IP network
- The XpoLog server should not have to use SSH port forwarding to connect to the remote server
- SSH keys for the account running the XpoLog process should be configured to access the remote server
Prerequisite 2: Copy LogAway for Java v4.5
When XpoLog collects logs over SSH, it does not require any tools to be installed or running in the target machine.
This is the default behavior and works for most Linux systems.
Among other things, XpoLog uses the Unix “less” command on the remote server to collect the logs and sends over SSH.
Sometimes, however, the less command may not be available for remote SSH connections.
To address this issue, XpoLog uses a JAR file that can be placed in the SSH user’s home directory.
This JAR file is a passive agent, which means it does not run as a process or service in the target machine.
It runs only when requested to do so by the XpoLog server.
The agent is called LogAway for Java and it can be downloaded from XpoLog.
The uncompressed files of Log Away for Java should be placed under the remote user’s home directory in the target machine.
The code snippet below shows one such setup:
|# ls -l /root|
drwxrwxrwx 2 root root 70 Apr 22 2014 xpologAgent
-rw-r–r– 1 root root 92626 Mar 19 12:07 xpologAgent.tar.gz
# ls -l /root/xpologAgent
-rwxrwxrwx 1 root root 444 Apr 22 2014 runAgent.sh
-rwxrwxrwx 1 root root 76161 Mar 18 2013 xpologagent.jar
-rwxrwxrwx 1 root root 30272 Mar 7 2011 xpologbase.jar
Configure SSH Log Collection
Once the prerequisites have been met:
- Open a browser window and navigate to the XpoLog Center 7 home page:
- Once the XpoLog Center 7 home page comes up, click on the link “ADD DATA” from the top right corner of the page and choose “ADD LOG” from the dropdown menu:
- This starts the SOURCE AND COLLECTION SETTINGS From the data source types, click on the SSH icon:
- In the next screen, click on the “NEW” button to start creating an SSH connection:
- In the “Add [Over SSH] account” section under “CONNECTION DETAILS”, provide the details for the SSH connection already created.
- Note that there are no fields for SSH port forwarding:
- Click the “VERIFY” button to check XpoLog can connect to the remote Nginx server. Click “SAVE” after the verification succeeds.
- Click the “BROWSE” button under “LOG PATH OR DIRECTORY” section, then drill through the Nginx server’s file system until the log files are visible. Click on the access log file to select it:
- It’s also possible to define advanced log collection parameters in the “COLLECTION SETTINGS” section:
- Leave this section and click the “ADD LOG” button. XpoLog will collect the Nginx access log and display it in the “ROW LOG SAMPLE” box:
- XpoLog will display the log pattern it thinks best represents the data. It will also show other patterns the user can choose:
- From here, users can write and add their own custom patterns, or choose one of the system-recommended patterns:
At the bottom of the page, XpoLog will show the parsed data based on the pattern selected:
- In the following image, one of the suggested patterns has been dragged to the top of the active log pattern list:
- The log file is now parsed in a much granular way:
- Click the “SAVE” button. This will open the “LOG COLLECTION SETTINGS” dialog box:
- Provide a descriptive name for the Nginx access log file.
- For the parent log folder, click “BROWSE” and then create a new folder under the root folder. You can also create a new folder before adding the log:
- For the collection policy, choose the “Default Collection Policy”. You can also create a new collection policy before adding the log
- Type relevant app tags for the log (e.g. “nginx”, “web server” etc.). You can also create new app tags here. App tags are separated by commas
- Provide relevant log type tags. You can use system-recommended tags (XpoLog will display available log type tags once you start typing) or add own custom tags
- Click “SAVE & CLOSE” to complete the wizard.
Verify Added Log Data
To verify that Nginx log data has been added to XpoLog:
- Double click on the log file name under the folder in the “Folders and Log Files” page:
- This will show the log file in XpoLog Log Viewer:
Adding MySQL Database Table
XpoLog allows you to add data from relational database tables.
The added data is treated like structured log files with fixed-length fields.
The data can then be searched the same way as any other log file.
There are many systems which store important log events in structured data stores.
The ability to access and add data from such data sources is one of the core strengths of XpoLog as a log management solution.
In this section, you will see how XpoLog can be configured to access a MySQL database and add data from a table.
This demonstration will use a freely available MySQL sample database called “sakila” running in an AWS RDS MySQL instance. The instance name is rds-dev-mysql:
The database has a table called “customers”:
To add the table’s data to XpoLog:
- Click “ADD LOG” from the “ADD DATA” menu:
- From the Add Data screen, click on Database:
- In the “CONNECTION DETAILS” section, click on “NEW”
- In the “Add [Database] account” section, choose MySQL:
- Specify the account details for the database instance:
TIP: If the MySQL JDBC driver is not installed in the XpoLogs server, you will receive a prompt to install the driver. You can download the platform-independent MySQL Connector/J, uncompress it, and upload the binary from the XpoLog interface.
Once the connection is verified, click on “SAVE”.
The wizard goes back to the “Add Data” screen with the newly created connection selected.
In the “DB QUERY DEFINITION” box, type the SQL query to extract the data:
- Click “VERIFY QUERY”
The selected columns will be shown, and you will be asked for an ordering column.
Choose the appropriate field name from “Available Columns” and click on the arrow beside it to add it under “ORDER BY THIS COLUMNS” column:
- Click “DONE”.
XpoLog shows the raw data it has read from the table and displays the pattern it has used to parse the data:
- Unless there is anything to correct, keep the pattern XpoLog has chosen and then click “SAVE” from the top right corner of the screen
- The “Log Collection Settings” pop-up window is displayed.
- From here, provide a name for the log (XpoLog considers the table data as a log), choose a parent log, accept the default log collection policy or create a new one and add app tags and log type tags:
- Click “SAVE AND CLOSE”.
From XpoLog Manager, choose the “Folders and Logs” menu and expand the folder containing the MySQL data:
- Double-clicking on the log will open it in the Log Viewer:
The customer table data is now available in XpoLog just like any log data.
Adding Files from S3 Bucket
XpoLog can also import files from Amazon S3 buckets.
For this to work, the bucket needs to have an appropriate policy in place so the XpoLog node can access it.
To add data from S3:
Select “ADD LOG” from the “ADD DATA” menu:
Click on “AWS S3 Bucket” icon:
Select the S3 bucket account from the list:
XpoLog will use the API keys to impersonate an IAM user to access the bucket.
The IAM user, in this case, needs to have at least read and list privileges on S3.
Once you have created and/or selected the S3 account, click “BROWSE” in the “LOG PATH OR DIRECTORY” field and select the target S3 bucket:
In the images below, we have chosen the XpoLog memory log file by double-clicking on a bucket called “xplg.lab.demo”, and then browsing through a folder by double-clicking on it:
Select the target file once you have traversed to it and click “ADD LOG”.
This will show a portion of the file’s contents as XpoLog reads it.
Also, the “ACTIVE LOG PATTERN LIST” field will show a log pattern XpoLog has found by parsing the file.
Although we could add a custom pattern to tell XpoLog how to parse the file, in this particular case, we are accepting the default pattern:
Click “SAVE” from the top left corner of the screen:
This will open the “Log Collection Settings” dialogue box.
Here, we can give the log file a meaningful name, specify a folder for it, create or choose a collection policy and also add one or more app tags and logtype tags:
Click “SAVE & CLOSE” in the dialog box. This will create the log file under the folder:
Double-clicking the log file will now display the file’s contents in the XpoLog Log Viewer.
Once logs are ingested into XpoLog, users can not only search them, they can also use XpoLogs’ log analysis apps (if one is available for the type of log) to get a better visualization of the data.
Read more about: