Collecting Log/IT Data with PortX/XpoLog is Simple.

You can be sure that PortX/XpoLog Log management supports a large variety of data types and collection methods. (Collect, Parses, Process, Search, and Display)

Supported Log Data Types

• Plain-text log file collection such as Syslog collection and other application log files.
• Various formats such as Custom-delimited, CSV, JSON, or XML Collection.
• Windows Event Logs Collection(When Running on a Windows Server)
• Database tables data collection
  and more…

Dynamic Log Data Collection Methods

1. Shipping Data to PortX/XpoLog Listeners

   (“Push Collection” method)

   Log sources can send data (“push”) to PortX/XpoLog in real-time. PortX “listeners” are configured to capture such incoming log data.

2. Setting up Data Collection with PortX/XpoLog

   (“pull Collection” methods)

• • Local server log directories data collection (direct access where PortX/XpoLog is running).
  • Direct access to Remote file shares using a UNC path such as \server_nameshared_folder.
    This is can be used by PortX/XpoLog running on a Windows machine.
  • Mounted directories direct access.
  • SSH access to remote Linux or Unix machine log directories (agent-less collection).
  • JDBC connection to database instances.
  • Integration with HDFS file system.
  • Integration with AWS S3 buckets.
  • Integration with Google App Engine.
  • Access to remote PortX/XpoLog servers (when the remote PortX/XpoLog acts as a Syslog server)

Single log directory or Multiple log directories.

PortX/XpoLog can ingest log files from a single log directory or multiple log directories.
The methods for both are different and this guide will show a series of preparatory steps before adding source data.

These steps can be also done at the end of each data source configuration.

Log Management Data Collection
Creating Log Folders

Logs ingested into PortX/XpoLog are logically arranged in the internal structure of folders (similar to the folders in most computer file systems).

Folders are used to organize and arrange log files into different namespaces.

When adding new logs, the option to choose a target logs folder appears (if left blank the log is added under the root folder.)

Log folders can also be created when adding a new data source.

Pre-Create Folders

It is simple to pre-create folders, simply follow these steps:

• Open a browser window to the PortX/XpoLog home page:

• Click the PortX icon on the top left (or the Gear icon from the top right) corner of the web page:

• Select “Folders and Logs” from the navigation panel:

• On the “Folders and Logs” page
  • Click “ADD” and choose “Folder”.

• On the “CREATE NEW FOLDER” Panel,
  • Create a folder named for example “MySQL Logs” as shown below.
    Make sure the folder is created under the root (“Folders and Logs”).
  • When done click “SAVE”.

• Similarly, create three other folders:
  • Remote Syslog folder
  • Database Data folder
  • S3 Logs folder
• Once created, the folder list should look like this:

Log Management Data Collection
Creating App-Tags

PortX App-Tags allow users to attach metadata information to logs and other PortX/XpoLog components such as dashboards, folders, or user accounts.

It helps to classify and group logs that belong to a source or application, which in turn can streamline search queries.

App-Tags can be also used as a means of implementing security.

User accounts that have an App-Tag attached to it will only have access to the components that belong to the same App-Tag.

Create an AppTag:

• Choose “TAGS”, and then “AppTags” from the Portx/XpoLog Manager navigation panel
  (PortX/XpoLog have Predefined AppTags installed)

• Click on the “ADD” button.
• Name and add a short description to the AppTag on the “Create tag” screen.
  • Note! AppTags can be attached to different components like:
    folders, logs, environment tables, user accounts, or saved searches.
  • For a new AppTag, provide only the name and description:

• When done, click “SAVE”.
  The AppTag will be added to the App-Tags list.

Log Management | Collect and Forward Data 
Setup Log Data
Collection Policy

PortX/XpoLog log collection policy is being used to define:

• Schedule – How often PortX/XpoLog will communicate with a data source and collect logs from it
• Storage – Where log files will be saved
• Retention – How long will it keep the log files online
• Archive – Where to archive log files
• Archive Retention – When to delete archived log files
• Forwarding Log Data – Setup Forwarding

PortX/XpoLog comes with a default log collection policy, but it is possible to create multiple collection policies and attach them to different data sources.

Create a log collection policy:

Open PortX/XpoLog Manager. and choose “DATA”, then “Collection Policies”

Click  “NEW COLLECTION POLICY” button.
This opens the “Create Collection Policy” screen.

• Provide a name for the collection policy:

Select Folder

• Select the PoertX/XpoLog folder that will contain the logs:

Storage Repository

From the “Storage Repository” section, specify where the logs would be saved in the XpoLog server and how long would they be kept there:

Schedule

Define a schedule PortX/XpoLog will use to collect logs from the source:

Archive

To enable archiving,  the following options need to be defined from the “ARCHIVE LOCATION” section of the collection policy settings:

• Enable archiving
• Archive in a local folder or an S3 bucket
• Archive path
• How long to keep archive files before deleting
• Checksum algorithm

TIP:
Log archiving is not a mandatory step for creating a collection policy.

However, this can help in more efficient storage of older, unnecessary logs from a busy network.

Unlike searchable, “current” data, archived logs are not indexed.

Non-indexed, archived data requires 9% of the disk space consumed by the raw ingested data, compared to 35% disk space required by searchable, indexed data.

Archived logs are stored in a compressed format by PortX/XpoLog, which also runs regular checksums on the data to make sure it has not been tampered with.

App-Tag

Attach an AppTag with the collection policy. This can be set later when creating a new AppTag:

Forwarding Log Data

Shiping-forward logs can be done in various methods :

• File Beat Forwarder
• HTTP endpoint: PortX sends logs to another application’s REST API endpoint.
• Syslog server – In this case, PortX works as a log forwarder for a Syslog server.
• XpoLog forwarder: PortX sends data to another PortX/XpoLog instance or other SIEM / Log Management tools.
• Forwarded data can be filtered so not everything is sent across the network.

When done, click “SAVE” and the new collection policy is ready.