GUIDE 5 | XPLG Products Suite | PortX / XpoLog
Prepare / Log Management
Log data collection policies
After you have installed and configured (PortX/XpoLog), your next step is preparing Log Management and Data Collection policies of your data (for all types or from any data source).
Whether you want to use PortX to collect, manage & optimize your log-data streams and route data/ship the data forward to any log analysis platforms (ELK or others), or whether you want to analyze your log data using XpoLog automatic Log analysis, search, and augmented log insights.
This guide will take you through the simple steps you need to take for preparing your Log Management Data Collection.
(If you haven’t installed XPLGs’ Log Management & Analysis Products suite, download the latest Vr. XPLG Free Download.)
Overview
Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive
Adding Listeners
for Data shipped to PortX/XpoLog
Collecting Log/IT Data with PortX/XpoLog is Simple.
You can be sure that PortX/XpoLog Log management supports a large variety of data types and collection methods. (Collect, Parses, Process, Search, and Display)
Supported Log Data Types
- Plain-text log file collection such as Syslog collection and other application log files.
- Various formats such as Custom-delimited, CSV, JSON, or XML Collection.
- Windows Event Logs Collection(When Running on a Windows Server)
- Database tables data collection
and more…
Dynamic Log Data Collection Methods
Shipping Data to PortX/XpoLog Listeners
(“Push Collection” method)
Log sources can send data (“push”) to PortX/XpoLog in real-time. PortX “listeners” are configured to capture such incoming log data.
Setting up Data Collection with PortX/XpoLog
(“pull Collection” methods)
- Local server log directories data collection (direct access where PortX/XpoLog is running).
- Direct access to Remote file shares using a UNC path such as \server_nameshared_folder.
This is can be used by PortX/XpoLog running on a Windows machine. - Mounted directories direct access.
- SSH access to remote Linux or Unix machine log directories (agent-less collection).
- JDBC connection to database instances.
- Integration with HDFS file system.
- Integration with AWS S3 buckets.
- Integration with Google App Engine.
- Access to remote PortX/XpoLog servers (when the remote PortX/XpoLog acts as a Syslog server)
Single log directory or Multiple log directories.
PortX/XpoLog can ingest log files from a single log directory or multiple log directories.
The methods for both are different and this guide will show a series of preparatory steps before adding source data.
These steps can be also done at the end of each data source configuration.
Pre-Create Folders
Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive
Adding Listeners
for Data shipped to PortX/XpoLog
Log Management Data Collection
Creating Log Folders
Logs ingested into PortX/XpoLog are logically arranged in the internal structure of folders (similar to the folders in most computer file systems).
Folders are used to organize and arrange log files into different namespaces.
When adding new logs, the option to choose a target logs folder appears (if left blank the log is added under the root folder.)
Log folders can also be created when adding a new data source.
Pre-Create Folders
It is simple to pre-create folders, simply follow these steps:
- Open a browser window to the PortX/XpoLog home page:
- Click the PortX icon on the top left (or the Gear icon from the top right) corner of the web page:
- Select “Folders and Logs” from the navigation panel:
- On the “Folders and Logs” page
- Click “ADD” and choose “Folder”.
- On the “CREATE NEW FOLDER” Panel,
- Create a folder named for example “MySQL Logs” as shown below.
Make sure the folder is created under the root (“Folders and Logs”). - When done click “SAVE”.
- Create a folder named for example “MySQL Logs” as shown below.
- Similarly, create three other folders:
- Remote Syslog folder
- Database Data folder
- S3 Logs folder
- Once created, the folder list should look like this:
Create App-Tags
Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive
Adding Listeners
for Data shipped to PortX/XpoLog
Log Management Data Collection
Creating App-Tags
PortX App-Tags allow users to attach metadata information to logs and other PortX/XpoLog components such as dashboards, folders, or user accounts.
It helps to classify and group logs that belong to a source or application, which in turn can streamline search queries.
App-Tags can be also used as a means of implementing security.
User accounts that have an App-Tag attached to it will only have access to the components that belong to the same App-Tag.
Create an AppTag:
- Choose “TAGS”, and then “AppTags” from the Portx/XpoLog Manager navigation panel
(PortX/XpoLog have Predefined AppTags installed)
- Click on the “ADD” button.
- Name and add a short description to the AppTag on the “Create tag” screen.
- Note! AppTags can be attached to different components like:
folders, logs, environment tables, user accounts, or saved searches. - For a new AppTag, provide only the name and description:
- Note! AppTags can be attached to different components like:
- When done, click “SAVE”.
The AppTag will be added to the App-Tags list.
Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive
Adding Listeners
for Data shipped to PortX/XpoLog
Log Management | Collect and Forward Data
Setup Log Data
Collection Policy
PortX/XpoLog log collection policy is being used to define:
- Schedule – How often PortX/XpoLog will communicate with a data source and collect logs from it
- Storage – Where log files will be saved
- Retention – How long will it keep the log files online
- Archive – Where to archive log files
- Archive Retention – When to delete archived log files
- Forwarding Log Data – Setup Forwarding
PortX/XpoLog comes with a default log collection policy, but it is possible to create multiple collection policies and attach them to different data sources.
Create a log collection policy:
Open PortX/XpoLog Manager. and choose “DATA”, then “Collection Policies”
Click “NEW COLLECTION POLICY” button.
This opens the “Create Collection Policy” screen.
- Provide a name for the collection policy:
Select Folder
- Select the PoertX/XpoLog folder that will contain the logs:
Storage Repository
From the “Storage Repository” section, specify where the logs would be saved in the XpoLog server and how long would they be kept there:
Schedule
Define a schedule PortX/XpoLog will use to collect logs from the source:
Archive
To enable archiving, the following options need to be defined from the “ARCHIVE LOCATION” section of the collection policy settings:
- Enable archiving
- Archive in a local folder or an S3 bucket
- Archive path
- How long to keep archive files before deleting
- Checksum algorithm
TIP:
Log archiving is not a mandatory step for creating a collection policy.
However, this can help in more efficient storage of older, unnecessary logs from a busy network.
Unlike searchable, “current” data, archived logs are not indexed.
Non-indexed, archived data requires 9% of the disk space consumed by the raw ingested data, compared to 35% disk space required by searchable, indexed data.
Archived logs are stored in a compressed format by PortX/XpoLog, which also runs regular checksums on the data to make sure it has not been tampered with.
App-Tag
Attach an AppTag with the collection policy. This can be set later when creating a new AppTag:
Forwarding Log Data
Shiping-forward logs can be done in various methods :
- File Beat Forwarder
- HTTP endpoint: PortX sends logs to another application’s REST API endpoint.
- Syslog server – In this case, PortX works as a log forwarder for a Syslog server.
- XpoLog forwarder: PortX sends data to another PortX/XpoLog instance or other SIEM / Log Management tools.
- Forwarded data can be filtered so not everything is sent across the network.
When done, click “SAVE” and the new collection policy is ready.
Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive
Adding Listeners
for Data shipped to PortX/XpoLog
Adding Listeners
When it is not possible or desired for PortX/XpoLog to periodically connect to a remote data source or device and pull data across.
Shiping (“push”) Data to PortX/XpoLog might be easier for the target systems at regular intervals.
With this approach, Data collection is working with listeners set up in PortX/XpoLog.
Listeners are port interfaces where applications and devices can send their payload using a protocol.
Once a listener is running, XpoLog ensures it’s “listening” on that port for any newly arrived data.
Any data captured on the port is then decoded and saved.
Currently, XpoLog listeners support the following typ3es of traffic types in its listeners:
- Syslog over TCP (any port)
- Syslog over UDP (any port)
- HTTP or HTTPS (the default system HTTP/S ports)
- XpoLog transport protocol
- Apache Kafka
- Cisco routers and switch using Netflow protocol
Creating Data Listeners
To Setup a listener for collecting log data shipped by application/Services:
- Select “DATA”, then “Listen To Data” in the PortX/XpoLog Manager navigation menu:
From the Listeners Configuration Management screen, select “HTTP Listener”:
From the top right corner, click the “Add Account” button.
In the pop-up window, give a name to the new listener, and accept the default values for the token and the URL.
This is the URL the applications or devices will send their data to.
Click on “Advanced Settings”
From the advanced settings fields:
- Accept the parent folder name (it is the same as the new listener name) or select/create a folder where the received data will be stored
- Accept the default collection policy or select a policy to be automatically applied to the data received on this listener
- Accept the log name prefix or change it to a custom prefix
- Specify if data received on this listener will be part of one single log, or if it will be split across multiple log files based on the source device
- Accept all other values and click “Save“
The newly created listener will be displayed
All Guides:
Read more about: