Preparing XpoLog Log Management Center 7 for Data Load
With XpoLog Center 7 installed and configured, users can start collecting logs from different systems.
Before doing this, there are some important points to remember.
XpoLog can ingest, process, search and display:
- Any plain-text log files such as syslog and other application log files. These files can be in any format such as custom-delimited, CSV, JSON or XML
- Windows event logs (XpoLog Center 7 needs to be running on a Windows Server)
- Data from database tables
For collecting logs and data, XpoLog can use one of the following “pull methods”:
- Direct access to the local server (where XpoLog is running) log directories
- Direct access to remote file shares using UNC path such as \\server_name\shared_folder\. This is can be used by XpoLog Center 7 running in a Windows machine
- Direct access to mounted directories
- SSH access to remote Linux or Unix machine log directories (agent-less collection)
- JDBC connection to database instances
- Integration with HDFS file system
- Integration with AWS S3 buckets
- Integration with Google App Engine
- Access to remote XpoLog servers (when the remote XpoLog acts as a syslog server)
Log sources can also send data to (“push”) XpoLog Center 7 in real-time. XpoLog “listeners” are configured to capture such oncoming log data.
It is possible to ingest log files from a single log directory or multiple log directories to XpoLog. The methods for both are different.
This chapter will show a series of preparatory steps before adding source data.
These steps can be also done at the end of each data source configuration.
Creating Log Folders
Logs ingested into XpoLog Center 7 are logically arranged in an internal structure called folders which are like regular folders in a computer file system.
Folders can be used to logically arrange log files into different namespaces.
When adding logs, XpoLog gives the option to choose a logs folder.
If none is chosen, the logs directory is added under the XpoLog folders root.
Log folders can be created when adding a data source.
To pre-create folders, follow these steps:
- Open a browser window and browse to the XpoLog Center 7 home page:
- Once the XpoLog Center 7 home page comes up, click on the gear icon from the top right corner of the web page:
- This is will show the XpoLog Manager interface. From the left navigation panel, choose “Folders and Logs”:
- This opens the “Folders and Logs” screen:
- Click “ADD” and then from the submenu choose “Folder”.
- In the “CREATE NEW FOLDER” screen, create a folder for MySQL as shown below. Make sure the folder is created under the root (“Folders and Logs”). Click “SAVE”.
- Once created, the folder list should look like this:
- Similarly, create three other folders:
- one for remote server syslogs
- one for database data
- one for S3 log data
AppTags allow users to attach metadata information to logs and other XpoLog components such as dashboards, folders or user accounts.
It helps to classify and group logs that belong to a source or application, which in turn can streamline search queries.
AppTags can be also used as a means of implementing security.
User accounts that have an AppTag attached to it will only have access to the components that belong to the same AppTag.
To create an AppTag:
- Bring up the XpoLog Manager interface. From the left navigation panel, choose “TAGS”, and then “AppTags” from the submenu:
- This shows the AppTags already created. The image below shows the AppTags XpoLog Center 7 ships with:
- Click on the “ADD” button, this opens the “Create tag” screen.
- From here, give a name and short description of the tag.
- Note how the tag can be attached to different XpoLog components like folders, logs, environment tables, user accounts or saved searches.
- For a new AppTag, provide only the name and description:
- Click “SAVE”. This will add the AppTag and it will be visible from the list.
Creating Log Collection Policy
An XpoLog log collection policy defines:
- How often XpoLog will communicate with a data source and collect logs from it
- Where will it save the log files
- How long will it keep the log files online
- Where will it archive the log files
- When will it delete archived log files
XpoLog Center 7 comes with a default log collection policy, but users can create multiple collection policies and attach them to different data sources.
To create a log collection policy:
Bring up the XpoLog Manager interface. From the left navigation panel, choose “DATA”, then “Collection Policies” under “SETTINGS” from the submenu:
This opens the “Collection Policies” screen. By default, only the “Default Collection Policy” is listed:
Click on the “NEW COLLECTION POLICY” button. This opens the “Create Collection Policy” screen. From here, provide a name for the collection policy:
Select the XpoLog folder that will contain the logs:
From the “Storage Repository” section, specify where the logs would be saved in the XpoLog server and how long would they be kept there:
Define a schedule XpoLog will use to collect logs from the source:
To enable archiving, provide the following options from the “ARCHIVE LOCATION” section of the collection policy:
- Enable archiving
- Archive in a local folder or an S3 bucket
- Archive path
- How long to keep archive files before deleting
- Checksum algorithm
TIP: Log archiving is not a mandatory step for creating a collection policy.
However, this can help in more efficient storage of older, unnecessary logs from a busy network.
Unlike searchable, “current” data, archived logs are not indexed.
Non-indexed, archived data requires 9% of the disk space consumed by the raw ingested data, compared to 35% disk space required by searchable, indexed data.
Archived logs are stored in a compressed format by XpoLog, which also runs regular checksums on the data to make sure it has not been tampered with.
Attach an AppTag with the collection policy. This can be set later when creating a new AppTag:
Finally, select any other log collection endpoints to send logs to. Logs can be forwarded to:
- A syslog server – In this case, XpoLog works as a log forwarder for syslog server
- An HTTP endpoint: XpoLog sends logs to another application’s REST API endpoint
- XpoLog forwarder: XpoLog sends data to another XpoLog instance or other SIEM / log management tools. Forwarded data can be filtered so not everything is sent across the network.
- Once all sections are configured, click “SAVE” on the top right corner. The new collection policy is then created.
Occasionally, it is neither possible nor desired for XpoLog to periodically connect to a remote data source or device and pull data across.
Instead, it may be easier for the target systems to “push” their data to XpoLog at regular intervals.
With this approach, you can create one or more listeners in XpoLog.
Listeners are port interfaces where applications and devices can send their payload using a protocol.
Once a listener is running, XpoLog ensures it’s “listening” on that port for any newly arrived data.
Any data captured on the port is then decoded and saved.
Currently, XpoLog listeners support the following typ3es of traffic types in its listeners:
- Syslog over TCP (any port)
- Syslog over UDP (any port)
- HTTP or HTTPS (the default system HTTP/S ports)
- XpoLog transport protocol
- Apache Kafka
- Cisco routers and switch using Netflow protocol
Creating a Listener
To add a listener:
From XpoLog Manager navigation menu, select “DATA”, then “Listen To Data”:
From the Listeners Configuration Management screen, select “HTTP Listener”:
From the top right corner, click the “Add Account” button.
In the pop-up window, give a name to the new listener, and accept the default values for the token and the URL.
This is the URL the applications or devices will send their data to.
Click on “Advanced Settings”
From the advanced settings fields:
- Accept the parent folder name (it is the same as the new listener name) or select/create a folder where the received data will be stored
- Accept the default collection policy or select a policy to be automatically applied to the data received on this listener
- Accept the log name prefix or change it to a custom prefix
- Specify if data received on this listener will be part of one single log, or if it will be split across multiple log files based on the source device
- Accept all other values and click “Save“
The newly created listener will be displayed
Read more about: