Log4J Patterns: Refined Log Parsing

XpoLog’s updated version does not overlook any piece of raw data, no matter how small or insignificant it may seem.

In this article, we will cover some of the ways you can benefit from XpoLog features and enhancements.

In particular, we will concentrate mainly on how to get the most valuable information from your log4j events.

Can’t wait? want to see for yourself?

From deployment to insights in just a few clicks – we are fully automated

Defining and Editing log4j Patterns in SysLog for more Refined Log Parsing

With XpoLog’s automated log parser it takes minuted to stream event.

Once your log4j logs have been transferred to and defined in XpoLog Center, you can:

  • Troubleshoot your java application by running Augmented Analytic Search on your log4j data.
  • Measure your application performance.
  • Create your own AppTags for better monitoring.
  • And create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis.

So let’s see how you can define and edit your events and log patterns before and after they reach XpoLog Center, when sending them through SysLog.

By creating the most readable data you will allow for XpoLog to perform the highest detailed analysis of your logs.

To follow more easily as I go along you can download the software for free.

Since logs are written in free format, XpoLog has an advanced built-in mechanism to detect the structure, or pattern, of the incoming log.

As a user, you can edit and fine-tune these patterns to suit your needs.

Defining Log Patterns in SysLog Appenders

When sending events to XpoLog through SysLog, be sure to create a detailed conversion pattern while configuring your log4j SysLog appender.

Here is an example:

#Logger definition log4j.logger.events=INFO, SYSLOG #Appender data for syslog log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.syslogHost= log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout log4j.appender.SYSLOG.layout.conversionPattern=[%t] %c%m%n log4j.appender.SYSLOG.Facility=LOCAL1

(t = thread, c = class, m = message, and n = new line) The SysLog appender will write this event logger to the SysLog.

Remember to define a SysLog Listener account inside XpoLog Center.

The events that arrive at XpoLog Center are written internally.

Here is what they might look like when created by the XpoLog SysLog listener:

XPLG:[1436716542132] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit – [Master] [-] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [-] [-] [-] [-] release user admin

XPLG:[1436716542140] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit – [Master] [Admin] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [EDA6FECA79A7BBB4480BAFC0FFB911F1] [administrators] [] [] login with username admin ok

The text at the very beginning is the extra data added by the XpoLog Syslog listener.

The other parts of the text in the SysLog file correspond to the layout you created in the log4j SysLog appender (follow the color scheme).

Once the data arrives into XpoLog, a log is created with the default SysLog pattern:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true} {string:Message}

Edit the log and set the pattern to reflect the layout you defined in the log4j configuration.

To edit the pattern in a log in XpoLog:

1. In XpoLog Center, go to Administration and find the log under Folders and Logs in the tree in the left margin.

Right-click on the log and select Edit. blog 3 log4j Automaticall generated log4j syslog log - logviewer before editing The Edit Log screen opens. 2. Click Next to get to the Log Pattern section.

The pattern can be edited in the Pattern1 field of the Pattern Editor, or you can add a new pattern in addition to the existing one by clicking the New tab.

Toggle between the Manual button (far right) and the Wizard button to see either version of the pattern.

You can add as many patterns as you want by clicking the New tab.

XpoLog will save all these patterns as templates for forthcoming logs. 3. Click Save.

In the screen capture below you can see how to define the log data pattern. It is displayed in the Pattern1 field. The pattern for this log is the following:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true}{text:Application Name}[{text:Process Id}]: {block,end,emptiness=true}{string:Message}

LOG4J blog 2 thread class message format small Most of the pattern, up to and including {block,end,emptiness=true}, is part of the SysLog protocol and functions as an prefix to the message – it contains the SysLog timestamp, facility, priority and the source device.

As mentioned previously, you can edit the pattern inside XpoLog Center after the event logs have been sent.

If your messages all follow the same structure, we recommend further editing the pattern to include this structure, to receive a more refined parsing. Here is a more refined pattern of the log shown above:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true}{text:Application Name}[{text:Process Id}]: {block,end,emptiness=true}[{text:ServerIp}] [{text:User,User}] [{choice:Action Type,LOGIN;VIEW;CHANGE}] [{text:Action description,Action description}] [{choice:Context,LOGS;FOLDERS;VERIFIERS;CONFIGURATION;


The following screen capture shows the same log as above, after editing.

You can see the original message has been split into the relevant columns. LOG4J 2nd blog refined audit log format small Note that by creating the most readable data, you will receive the most detailed analysis of your logs from XpoLog.