This Guide Covers Adding Data (Data Collection) to XPLG log Management Product Suite, PortX/XpoLog from different sources, and the collection of various data types. Installing PortX/XpoLog is simple as shown in the Installation QuickStart Guides (Find the latest product version XPLG Free Download.)
PortX/XpoLog allows you to add data from relational database tables.
The added data is treated like structured log files with fixed-length fields.
The data can then be searched the same way as any other log file.
There are many systems which store important log events in structured data stores.
The ability to access and add data from such data sources is one of the core strengths of PortX/XpoLog as a log management solution.
In this section, you will see how PortX/XpoLog can be configured to access a MySQL database and add data from a table.
This demonstration will use a freely available MySQL sample database called “sakila” running in an AWS RDS MySQL instance. The instance name is rds-dev-mysql:
The database has a table called “customers”:
To add the table’s data to PortX/XpoLog:
Click “ADD LOG” from the “ADD DATA” menu:
From the Add Data screen, click on Database:
In the “CONNECTION DETAILS” section, click on “NEW”
In the “Add [Database] account” section, choose MySQL:
Specify the account details for the database instance:
TIP: If the MySQL JDBC driver is not installed in the PortX/XpoLogs server, you will receive a prompt to install the driver. You can download the platform-independent MySQL Connector/J, uncompress it, and upload the binary from the PortX/XpoLog interface.
Once the connection is verified, click on “SAVE”.
The wizard goes back to the “Add Data” screen with the newly created connection selected.
In the “DB QUERY DEFINITION” box, type the SQL query to extract the data:
Click “VERIFY QUERY”
The selected columns will be shown, and you will be asked for an ordering column.
Choose the appropriate field name from “Available Columns” and click on the arrow beside it to add it under “ORDER BY THIS COLUMNS” column:
PortX/XpoLog shows the raw data it has read from the table and displays the pattern it has used to parse the data:
Unless there is anything to correct, keep the pattern PortX/XpoLog has chosen and then click “SAVE” from the top right corner of the screen
The “Log Collection Settings” pop-up window is displayed.
From here, provide a name for the log (PortX/XpoLog considers the table data as a log), choose a parent log, accept the default log collection policy or create a new one and add app tags and log type tags:
Click “SAVE AND CLOSE”.
From PortX/XpoLog Manager, choose the “Folders and Logs” menu and expand the folder containing the MySQL data:
Double-clicking on the log will open it in the Log Viewer:
The customer table data is now available in PortX/XpoLog just like any log data.
PortX/XpoLog can also import files from Amazon S3 buckets.
For this to work, the bucket needs to have an appropriate policy in place so the PortX/XpoLog node can access it.
To add data from S3:
Select “ADD LOG” from the “ADDDATA” menu:
Click on “AWS S3 Bucket” icon:
Select the S3 bucket account from the list:
This assumes you have already created an S3 account in PortX/XpoLog. If you have not created one such account, you can click “NEW”, and then provide the AWS API credentials to create the account.
PortX/XpoLog will use the API keys to impersonate an IAM user to access the bucket.
The IAM user, in this case, needs to have at least read and list privileges on S3.
Once you have created and/or selected the S3 account, click “BROWSE” in the “LOG PATH OR DIRECTORY” field and select the target S3 bucket:
In the images below, we have chosen the PortX/XpoLog memory log file by double-clicking on a bucket called “xplg.lab.demo”, and then browsing through a folder by double-clicking on it:
Select the target file once you have traversed to it and click “ADD LOG”.
This will show a portion of the file’s contents as PortX/XpoLog reads it.
Also, the “ACTIVE LOG PATTERN LIST” field will show a log pattern PortX/XpoLog has found by parsing the file.
Although we could add a custom pattern to tell PortX/XpoLog how to parse the file, in this particular case, we are accepting the default pattern:
Click “SAVE” from the top left corner of the screen:
This will open the “Log Collection Settings” dialogue box.
Here, we can give the log file a meaningful name, specify a folder for it, create or choose a collection policy and also add one or more app tags and logtype tags:
Click “SAVE & CLOSE” in the dialog box. This will create the log file under the folder:
Double-clicking the log file will now display the file’s contents in the PortX/XpoLog Log Viewer.
Once logs are ingested into PortX/XpoLog, users can not only search them, they can also use PortX/XpoLog XPLG Products Suites” log analysis apps (if one is available for the type of log) to get a better visualization of the data.
https://www.xplg.com/wp-content/uploads/2020/03/main-image-tal.png200381Jane Kernelhttps://www.xplg.com/wp-content/uploads/2022/03/LogoDark1color.pngJane Kernel2020-03-15 08:10:372020-07-14 09:34:17Hybrid Cloud Challenges: How to optimize monitoring and analytics for ITOps
https://www.xplg.com/wp-content/uploads/2019/08/nginx-Blog-image-2.png200381Jane Kernelhttps://www.xplg.com/wp-content/uploads/2022/03/LogoDark1color.pngJane Kernel2019-08-20 07:14:332020-03-17 06:10:07NGINX Logs: The Full Guide to Monitoring and Analyzing Your NGINX Logs
https://www.xplg.com/wp-content/uploads/2019/06/cyber-security-cybersecurity.jpg200381Jane Kernelhttps://www.xplg.com/wp-content/uploads/2022/03/LogoDark1color.pngJane Kernel2019-06-30 14:30:092020-06-07 06:27:23What Should You Learn from the Massive Data Breach “Operation Soft Cell”?
https://www.xplg.com/wp-content/uploads/2019/06/S3-SECURITY.png200381Jane Kernelhttps://www.xplg.com/wp-content/uploads/2022/03/LogoDark1color.pngJane Kernel2019-06-12 09:23:292020-04-30 07:13:41S3 Security: How to Easily Secure and Audit AWS S3 Buckets?
https://www.xplg.com/wp-content/uploads/2016/02/security.jpg200381Jane Kernelhttps://www.xplg.com/wp-content/uploads/2022/03/LogoDark1color.pngJane Kernel2019-04-07 08:37:362020-05-05 08:21:05Linux Security Guide 2020: How to Investigate Suspected Break-in Attempts in Linux