Linux Security Must Read Guide:
How to Investigate Suspected Break-in Attempts in Linux

+ Bonus Tip: How you can identify them in minutes

By: Sadequl Hussain

Scenario

Linux has a number of built-in tools, commands and files which can track and store information about every user activity.

These tools are common in most Linux distributions and can be used to investigate suspicious logins or failed login attempts into the system.

In this article, we will talk about some of the initial methods to identify possible security breaches.

We will use an Amazon EC2 instance to show these commands.

(The instance is created from the latest Amazon Linux 2 AMI at the time of writing).

This guide shows you how you can investigate and track down suspected break-in attempts manually.

You can also download XpoLog free and identify them, minutes from now 🙂

We got a pretty powerful tool for you, right here.

Linux Security Investigation, Step 1: Isolate

The first step of investigation is to isolate the machine you suspect has been breached.
This can be done by closing off traffic to and from the instance as much as possible and exposing it to traffic only from the system administrator’s workstation IP.

For production systems exposed to the outside world, this may not be a feasible option, one approach can be standing up another prod server as soon as possible and quarantining the old one. Once separated, the forensics can begin.

Linux Security Investigation, Step 2:

Get an Overview Using Aureport

Most newer Linux distros today should have auditd already installed and running.

If your Linux server does not have the package, you can install it for RedHat or Ubuntu systems.

You can run aureport – a tool that comes with auditd – to get an idea about failed login attempts or changes to system files and configuration.
The following code snippet shows this:

Get an idea about failed login attempts or changes to system files and configuration.

As you can see, there were only a couple of failed login attempts and authentication failures in the selected time period.

If there were say dozens, hundreds or even thousands of such events, it would definitely mean something was wrong.

Next, it would be time to look at the logs.

Linux Security Investigation, Step 3:

Check General Logs

/var/log/secure

For RedHat based systems, the /var/log/secure file contains information about security-related events, including authentication success or failures and the IP addresses where the requests came from.

It also tracks the sudo and SSH login attempts and other security related errors.

For Debian-based systems, similar information is held in the /var/log/auth file.

We recommend you start by looking at this file first. The code snippets below show some examples:

Check the /var/log/secure file contains information about security-related events, including authentication success or failures and the IP addresses where the requests came from. It also tracks the sudo and SSH login attempts and other security related errors.

You can see that there were two authentication failures for the user mytestuser.

Failed attempts to become root privilege holder can be tracked with a command like this:

track failed attempts to become root privilege holder

Another pattern to look for is shown below:

Another pattern to look for

To check if a user account was locked or unlocked, you can search for “lock”:

check if a user account was locked or unlocked

/var/log/wtmp

The wtmp log file contains all login and logout events.

It’s a binary file, so you can’t cat or grep this file. Instead, you can run the last command to get its contents.

The last command shows a chronological history of user logins into the system since the /var/log/wtmp file was created.

If this file is not already created, you can run a simple touch command to create it and Linux will start logging into it.  

The last command comes with a number of options. We will not go into each of these, but here is an example:

last command

It shows each instance of user login and logouts, how long the session was active for, and which host the connection came from.

Also, the entries for the user “reboot” show when the system was restarted.

/var/log/btmp

Similar to wtmp, /var/log/btmp is also a binary file you can touch to create if it doesn’t exist.

This file can be used to find out bad login attempts. Here, we are checking the bad login attempts for the user mytestuser:

Use the file to find out if bed log-in attempts occurred

/var/log/lastlog

The lastlog command can show the contents of the binary file /var/log/lastlog and display information about a user’s last login time.

An example is shown below for the user mytestuser:

 displays a users last log-in time

As you can see, the user’s last login time was Mon Nov 12 13:14:52.

/var/log/cron

Sometimes an account may be accidentally or intentionally locked out or expired.

Such accounts, when used to run cron jobs, will cause the job to fail.

One of the security audits is to look for failed authorizations in the cron service logs.

The example below shows this:

look for failed authorizations in the cron service logs

Running journalctl

In systemd, journalctl is a utility for querying and displaying messages from the systemd journal.

The following command can be used to query the SSH service log:

Use this query to investigate SHH service log

A message like the one above should need investigation.

15 Linux Security Must-Read Resources List

We have created a list of useful resources related to Linux security (the list is current as of November 2018).

These resources will help you:

  • Understand what to do when your Linux server has been hacked.
  • How to run vulnerability checks.
  • How to harden Linux security as well as learn about some useful tools.

Linux Security investigation – Check Audit Logs

Malicious users or malwares typically erase any traces of their intrusion by deleting log files or log entries.

Among other things, malicious intrusions can:

  • Change passwords of existing accounts
  • Unlock or un-expire locked or expired accounts
  • Create new accounts
  • Delete accounts
  • Delete log files
  • Change log files
  • Delete or change system configuration files

So don’t be surprised if you don’t find the files we talked about when you log into a server that has been compromised.

You can’t really stop changes to user accounts, passwords or files once security has been breached, but you can certainly track the changes that have been made.

This can give you valuable leads to further dig in.

Using auditd

The Linux auditing service (auditd) is tightly integrated with the kernel and traps log messages from events it subscribes to.

We already talked about the aureport command of the auditing service.

The log file for auditd is typically /var/log/audit/audit.log.

We recommend creating a cron task to regularly backup the log file to a different server to preserve it.

In the following code snippet, we are listing all the audits defined:

In the following code snippet, we are listing all the audits defined

This is not what we want, so we can define some audit rules:

define some audit rules

The audits are now watching the password files (/etc/passwd and /etc/shadow) as well as the /var/log/lastlog file.

Running a command like ‘rm -f /var/log/lastlog’ to delete the lastlog file will capture the event in auditd log.

The following command shows how this event can be captured:

The following command shows how this event can be captured

Here, we are using the key defined for the lastlog file when the audit was set up (see the previous command).

The output shows the /var/log/lastlog file was deleted by running the rm command from /root directory.

Linux Security Investigation –

Find Currently Logged-in Users

You may also want to run the w or who commands to check currently logged-in users.

Although an intruder may not keep any connection open, it’s still worth to look at all the users connected:

run the w or who commands to check currently logged-in users

Linux Security Investigation –

Check for Malware

A malware or rootkit is another way of intruding into your Linux system.

Rootkits usually take control of files, processes or users.

There are different tools to identify intrusion by rootkits, the most well-known being chkrootkit.

Its source file can be downloaded and compiled.

The snippet below shows how chkrootkit works:

See how chkrootkit works

To automate this checking, chkrootkit can be run on a schedule from a cron job.

The command output can be then sent to a sysadmin group e-mail.

The steps we have covered so far are all manual and assumes you have plenty of time at hand to troubleshoot only one Linux server.

What happens when you suspect more than one server has been breached? How do you look at them? This is what we talk about next, in our bonus tip.

Bonus tip: How to identify suspected break-in attempts in Linux 5 minutes.

You can choose to use a tool like XpoLog for security auditing purposes.

XpoLog is a state-of-the-art log management solution which offers unlimited parsing of log data collected from a wide variety of sources.

XpoLog’s in-depth analytics, super-fast performance and anomaly detection can take away much of the hard work of sifting through thousands of lines of log events.

For example, the analytics graph in the image below shows an incident where there was a large number of possible break-in attempts:

Easily search for break-in attempts with XpoLog log search

XpoLog detects such incidents automatically which is one of its core strengths.

This automation means metrics collected from multiple servers can be shown in a single console.

Once we have an overview like this, we can create filters on an event’s fields to further fine tune the search.

This is shown below:

create filters on an event’s fields to further fine tune the search.

This filters may be used for proactive monitoring to send alerts when critical events occur.

Other holistic but critical information available in XpoLog include:

  • Logins by Hour.
  • Failed Login per Host by Day.
  • Top Authentication Failures.
  • Authentication Failures by Geo IP.
  • Top Failed Passwords.
  • Authentication Failures by Process etc.

XpoLog provides out-of-the-box, ready-to-use, analytics apps for Linux monitoring which can show all this information instantaneously.

Because it comes with “auto connectors” to collect data from Linux servers, the log collection & parsing is automated as well (learn more).

Dashboards can be created to show high-level or granular pictures.

As an example, the dashboard widget below shows relative percentages of possible break-in attempts in multiple sites.

Easily investigate Linux possible break-in attempts with Linux AMI app's users login dashboard

Another dashboard shows login attempts:

View login attempts in a ready to use dashboard

Learn more about XpoLog

You can test-drive the PRO version for 30 days and process 5GB of log data per day. The free version lets you process 1GB of log per day.

Conclusion

Prevention is better than cure.

The methods we talked about are applicable when something already has happened.

As a security engineer, you would always strive to make your systems as secure as possible.

This includes putting appropriate firewall rules to open only certain ports, accepting traffic from trusted zones, locking down unnecessary accounts, removing unnecessary packages, keeping the OS regularly patched or changing root passwords.

Proactive monitoring plays a big role in keeping your network secure, and that’s where XpoLog can make a real difference.