GUIDE 5 | XPLG Products Suite | PortX / XpoLog
Prepare / Log Management
Log data collection policies

After you have installed and configured (PortX/XpoLog), your next step is preparing Log Management and Data Collection policies of your data (for all types or from any data source).

Whether you want to use PortX to collect, manage & optimize your log-data streams and route data/ship the data forward to any log analysis platforms (ELK or others), or whether you want to analyze your log data using XpoLog automatic Log analysis, search, and augmented log insights.

This guide will take you through the simple steps you need to take for preparing your Log Management Data Collection.

(If you haven’t installed XPLGs’ Log Management & Analysis Products suite, download the latest Vr.  XPLG Free Download.)


Overview

Pre-Create Folders

Create App-Tags

Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive

Adding Listeners
for Data shipped to PortX/XpoLog

Collecting Log/IT Data with PortX/XpoLog is Simple.

You can be sure that PortX/XpoLog Log management supports a large variety of data types and collection methods. (Collect, Parses, Process, Search, and Display)

Supported Log Data Types

  • Plain-text log file collection such as Syslog collection and other application log files.
  • Various formats such as Custom-delimited, CSV, JSON, or XML Collection.
  • Windows Event Logs Collection(When Running on a Windows Server)
  • Database tables data collection
    and more…

Dynamic Log Data Collection Methods

  1. Shipping Data to PortX/XpoLog Listeners

    (“Push Collection” method)

    Log sources can send data (“push”) to PortX/XpoLog in real-time. PortX “listeners” are configured to capture such incoming log data.

  2. Setting up Data Collection with PortX/XpoLog

    (“pull Collection” methods)

    • Local server log directories data collection (direct access where PortX/XpoLog is running).
    • Direct access to Remote file shares using a UNC path such as \server_nameshared_folder.
      This is can be used by PortX/XpoLog running on a Windows machine.
    • Mounted directories direct access.
    • SSH access to remote Linux or Unix machine log directories (agent-less collection).
    • JDBC connection to database instances.
    • Integration with HDFS file system.
    • Integration with AWS S3 buckets.
    • Integration with Google App Engine.
    • Access to remote PortX/XpoLog servers (when the remote PortX/XpoLog acts as a Syslog server)

Single log directory or Multiple log directories.

PortX/XpoLog can ingest log files from a single log directory or multiple log directories.
The methods for both are different and this guide will show a series of preparatory steps before adding source data.

These steps can be also done at the end of each data source configuration.

Overview

Pre-Create Folders

Create App-Tags

Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive

Adding Listeners
for Data shipped to PortX/XpoLog


Log Management Data Collection
Creating Log Folders

Logs ingested into PortX/XpoLog are logically arranged in the internal structure of folders (similar to the folders in most computer file systems).

Folders are used to organize and arrange log files into different namespaces.

When adding new logs, the option to choose a target logs folder appears (if left blank the log is added under the root folder.)

Log folders can also be created when adding a new data source.

Pre-Create Folders

It is simple to pre-create folders, simply follow these steps:

  • Open a browser window to the PortX/XpoLog home page:

Nevigate to this address to access XpoLog

  • Click the PortX icon on the top left (or the Gear icon from the top right) corner of the web page:

Setting up portX

  • Select “Folders and Logs” from the navigation panel:

  • On the “Folders and Logs” page
    • Click “ADD” and choose “Folder”.

  • On the “CREATE NEW FOLDER” Panel,
    • Create a folder named for example “MySQL Logs” as shown below.
      Make sure the folder is created under the root (“Folders and Logs”).
    • When done click “SAVE”.

  • Similarly, create three other folders:
    • Remote Syslog folder
    • Database Data folder
    • S3 Logs folder
  • Once created, the folder list should look like this:

create the following folders to add data to XpoLog log analysis tool

Overview

Pre-Create Folders

Create App-Tags

Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive

Adding Listeners
for Data shipped to PortX/XpoLog

Log Management Data Collection
Creating App-Tags

PortX App-Tags allow users to attach metadata information to logs and other PortX/XpoLog components such as dashboards, folders, or user accounts.

It helps to classify and group logs that belong to a source or application, which in turn can streamline search queries.

App-Tags can be also used as a means of implementing security.

User accounts that have an App-Tag attached to it will only have access to the components that belong to the same App-Tag.

Create an AppTag:

  • Choose “TAGS”, and then “AppTags” from the Portx/XpoLog Manager navigation panel
    (PortX/XpoLog have Predefined AppTags installed)

  • Click on the “ADD” button.
  • Name and add a short description to the AppTag on the “Create tag” screen.
    • Note! AppTags can be attached to different components like:
      folders, logs, environment tables, user accounts, or saved searches.
    • For a new AppTag, provide only the name and description:

How to create an AppTag for MySQL logs?

  • When done, click “SAVE”.
    The AppTag will be added to the App-Tags list.

Overview

Pre-Create Folders

Create App-Tags

Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive

Adding Listeners
for Data shipped to PortX/XpoLog

Log Management | Collect and Forward Data 
Setup Log Data
Collection Policy

PortX/XpoLog log collection policy is being used to define:

  • Schedule – How often PortX/XpoLog will communicate with a data source and collect logs from it
  • Storage – Where log files will be saved
  • Retention – How long will it keep the log files online
  • Archive – Where to archive log files
  • Archive Retention – When to delete archived log files
  • Forwarding Log Data – Setup Forwarding

PortX/XpoLog comes with a default log collection policy, but it is possible to create multiple collection policies and attach them to different data sources.

Create a log collection policy:

Open PortX/XpoLog Manager. and choose “DATA”, then “Collection Policies

Click  “NEW COLLECTION POLICY” button.
This opens the “Create Collection Policy” screen.

  • Provide a name for the collection policy:

Click on the “NEW COLLECTION POLICY” button. This opens the “Create Collection Policy” screen

Select Folder

  • Select the PoertX/XpoLog folder that will contain the logs:

selecting folder that will contain_the_log

Storage Repository

From the “Storage Repository” section, specify where the logs would be saved in the XpoLog server and how long would they be kept there:

adding data to XpoLog 7 center

Schedule

Define a schedule PortX/XpoLog will use to collect logs from the source:

defining log collection policy

Archive

To enable archiving,  the following options need to be defined from the “ARCHIVE LOCATION” section of the collection policy settings:

  • Enable archiving
  • Archive in a local folder or an S3 bucket
  • Archive path
  • How long to keep archive files before deleting
  • Checksum algorithm

Configuration of log archive in XpoLog log management

TIP:
Log archiving is not a mandatory step for creating a collection policy.

However, this can help in more efficient storage of older, unnecessary logs from a busy network.

Unlike searchable, “current” data, archived logs are not indexed.

Non-indexed, archived data requires 9% of the disk space consumed by the raw ingested data, compared to 35% disk space required by searchable, indexed data.

Archived logs are stored in a compressed format by PortX/XpoLog, which also runs regular checksums on the data to make sure it has not been tampered with.

App-Tag

Attach an AppTag with the collection policy. This can be set later when creating a new AppTag:

log collection policy creation - XpoLog

Forwarding Log Data

Shiping-forward logs can be done in various methods :

  • File Beat Forwarder
  • HTTP endpoint: PortX sends logs to another application’s REST API endpoint.
  • Syslog server – In this case, PortX works as a log forwarder for a Syslog server.
  • XpoLog forwarder: PortX sends data to another PortX/XpoLog instance or other SIEM / Log Management tools.
  • Forwarded data can be filtered so not everything is sent across the network.

When done, click “SAVE” and the new collection policy is ready.

Overview

Pre-Create Folders

Create App-Tags

Setup Log Data Collection Policy
Collect / Forward / Schedule / Archive

Adding Listeners
for Data shipped to PortX/XpoLog

Adding Listeners

When it is not possible or desired for PortX/XpoLog to periodically connect to a remote data source or device and pull data across.

Shiping (“push”) Data to PortX/XpoLog might be easier for the target systems at regular intervals.
With this approach, Data collection is working with listeners set up in PortX/XpoLog.

Listeners are port interfaces where applications and devices can send their payload using a protocol.
Once a listener is running, XpoLog ensures it’s “listening” on that port for any newly arrived data.
Any data captured on the port is then decoded and saved.

Currently, XpoLog listeners support the following typ3es of traffic types in its listeners:

  • Syslog over TCP (any port)
  • Syslog over UDP (any port)
  • HTTP or HTTPS (the default system HTTP/S ports)
  • XpoLog transport protocol
  • Apache Kafka
  • Cisco routers and switch using Netflow protocol

Creating Data Listeners

To Setup a listener for collecting log data shipped by application/Services:

  • Select “DATA”, then “Listen To Data” in the PortX/XpoLog Manager navigation menu:

Listen to logs, use log data listeners for log data forwarding

From the Listeners Configuration Management screen, select “HTTP Listener”:

forward and collect log data using log data listeners

From the top right corner, click the “Add Account” button.

In the pop-up window, give a name to the new listener, and accept the default values for the token and the URL.

This is the URL the applications or devices will send their data to.

give a name to the new log listener

Click on “Advanced Settings

From the advanced settings fields:

  • Accept the parent folder name (it is the same as the new listener name) or select/create a folder where the received data will be stored
  • Accept the default collection policy or select a policy to be automatically applied to the data received on this listener
  • Accept the log name prefix or change it to a custom prefix
  • Specify if data received on this listener will be part of one single log, or if it will be split across multiple log files based on the source device
  • Accept all other values and click “Save

A newly created listener is displayed

The newly created listener will be displayed

log listeners creation

All Guides: 

Read more about:

Aquick introduction, logstash plugins, filters, logstash inputs, output, logstash alternatives