sudoedit.com!

How to log sudo commands with auditd

Trust but verify

This post is about the audit daemon (auditd) that is available for most Linux systems.

Recently I’ve been looking at alternative ways to monitor sudo users on the servers I manage. Generally speaking it’s a good practice to keep an audit trail on managed systems. From a purely security perspective the more auditing you have on a system the easier any incident response should become when you need it. Your I.T. Security groups will need an easily searchable record of who ran which commands and with what privileges when trying to unravel how an exploit was used, or who used it, or both.

Outside of a security perspective you still want these controls in place to make sure that you can retrace any steps taken during changes while troubleshooting a problem. It’s all too common a scenario where a change goes wrong and somehow nobody knows what was changed. A robust audit trail can make hunting down which actions were taken much easier when figuring out what went wrong, and can go a long way towards finding a solution.

The Sudo Log

Historically we’ve always used the sudo log as a means to keep a record of all actions performed by users who need to escalate privileges to root while performing their job. One of the issues with the sudo log is that it does not keep a record of commands that are run when a user switches directly to root after logging in, at least not on a RHEL 7 system.

If a user logs in and immediately switches to root using sudo -i, sudo su - root, or sudo su -, then you won't have any log of their activities while logged in. This is an undesirable situation.

In search of a better solution

I.E. a solution that doesn’t involve just telling people not to switch directly to root. With all that in mind I decided to play around with some auditd rules, and after a little trial and error (and some internet searching) I came upon a really good answer on StackExchange.

I'll let you take a look at the answer for yourself if you so choose. The meat and potatoes of that post is to add this audit rule:

sudo auditctl -a always,exit -F arch=b64 -F euid=0 -S execve


For the most part it's a really good and simple solution. There are really just two problems with it for me. The first is that while this command does do most of what I'm looking for, the end result is only temporary. It will not survive a reboot, or even a restart of the auditd service and I need something permanent. The second problem with this answer (for my purposes) is that it logs everything done as root. I don’t necessarily want to log everything that is run by system daemons as a privilege escalation. Maybe you do, maybe I do and don’t know it yet. Right now what I want is a good way to specifically tag root actions performed by actual human users.

As an aside problem it also generates a lot of noise, which could make your logs unnecessarily large. Like this:

----
type=PROCTITLE msg=audit(09/05/2020 12:36:50.142:1572) : proctitle=sed -n s/device for \(.*\): usb:.*=\(TAG.*\)$/\1 \2/p
type=PATH msg=audit(09/05/2020 12:36:50.142:1572) : item=1 name=/lib64/ld-linux-x86-64.so.2 inode=2491677 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(09/05/2020 12:36:50.142:1572) : item=0 name=/usr/bin/sed inode=2491999 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=CWD msg=audit(09/05/2020 12:36:50.142:1572) :  cwd=/
type=EXECVE msg=audit(09/05/2020 12:36:50.142:1572) : argc=3 a0=sed a1=-n a2=s/device for \(.*\): usb:.*=\(TAG.*\)$/\1 \2/p
type=SYSCALL msg=audit(09/05/2020 12:36:50.142:1572) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x19072f0 a1=0x1906ed0 a2=0x1905e10 a3=0x7ffc9cd52420 items=2 ppid=4471 pid=4473 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sed exe=/usr/bin/sed subj=system_u:system_r:unconfined_service_t:s0 key=root_cmd
-----

Cutting through the noise

Since I was looking for a way to audit commands run as root by real users I needed to filter out the system noise. By default most Linux distributions reserve the first 999 uid’s for system accounts - for reference see: Linux sysadmin basics: User account management with UIDs and GIDs.

After experimenting a little bit I’ve decided to modify that rule to filter down to just what I was looking for, and make the rule permanent.

Heres how:

Add the following lines to the file /etc/audit/rules.d/audit.rules.

-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -F key=sudo_log
-a always,exit -F arch=b32 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -F key=sudo_log

Then restart the auditd service.

service auditd restart


One odd thing to note with the auditd service is that you can not restart it with systemctl. Instead you have to use the older service command to force a restart of the audit daemon, why does it work that way? - I have no idea, but I noticed that Red Hat recommends using the service command to start/stop the auditd service here: Chapter 10. Auditing the system. If you know the answer to why that is the case and want to call me out as a noob please do so on Twitter.

Another odd thing to note here is that on an up-to-date CentOS 7 or Fedora 32 machine that command string will not work if you run it with auditctl from the command line. For one reason or another auditctl doesn’t seem to like the >= operator when it’s passed in from standard input.

So what do these rules do:

  1. -a → Pretty simple that means append. You are appending everything after the -a to the audit rules.
  2. always,exit → This means you want to write to the log all the time when this rule is triggered and include the exit code.
  3. -F → The F option is a way to separate each field. The pattern is: name, operation, value. For instance the first field in our rule is name(arch), operation(=), and value(b64)
  4. -S → The S option defines the syscall we are looking for in this case execve is the syscall we want since it is the syscall for executing a program.
    1. We want to create a log entry everytime any program is run (constrained by the next set of fields)
    2. You can get a complete list of the available syscalls with ausyscall --dump
  5. The next fields euid=0, auid>=1000, and auid!=-1 allows us to constrain our rule based on the following criteria.
    1. The effective user has uid 0 (root)
    2. The original user id has a uid value of 1000 or greater. I chose uid 1000 here because that is the default starting point for non-system users on most Linux systems. You may or may not want to change this value.
    3. The originating user does not have a uid that is “unset”. Without this value you will end up with a ton of noise in your logs similar to what I shared above. I used auid!=-1, but you can also use auid!=4294967295 if your version of auditd doesn’t like the negative value, both do the same thing I just think -1 is more readable.
  6. Lastly we define a key with key=sudo_log. You could call your key anything you want, it is used to search for hits in the audit log.

The second rule is the same except that it watches for 32bit executables.

As always you should check the man pages for any command (man auditctl in this case) you are not familiar with before just copying and pasting something from the internet. You don’t know me, I might be a bad guy, or incompetent, or both. I’d like to think I'm not a bad guy, or incompetent but read the man pages anyway, and while you’re at it check out these documents for extra credit.

Testing the audit rules

After you have those rules in place and have restarted the auditd service, either with a reboot or with the service command, it’s time to test the new rules.

Use the auditclt command to view your rules.

sudo auditctl -l


Next, run any command you want with sudo or after switching to root (sudo -i, or sudo su - root, or sudo su - if you’re really old school)

I’m going to run yum updateinfo list

Displaying the audit log

Then search your audit log with the ausearch command filtering on the key we created:

sudo ausearch -k sudo_log -i

The first couple entries will be related to your ausearch command (remember every command run as root is logged). But scrolling up you should see something that looks like the following output: 

----
type=PROCTITLE msg=audit(09/05/2020 14:12:45.350:551) : proctitle=/usr/bin/python /bin/yum updateinfo list
type=PATH msg=audit(09/05/2020 14:12:45.350:551) : item=2 name=/lib64/ld-linux-x86-64.so.2 inode=2491677 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(09/05/2020 14:12:45.350:551) : item=1 name=/usr/bin/python inode=2494593 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(09/05/2020 14:12:45.350:551) : item=0 name=/bin/yum inode=2502974 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:rpm_exec_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(09/05/2020 14:12:45.350:551) : cwd=/home/luke
type=EXECVE msg=audit(09/05/2020 14:12:45.350:551) : argc=4 a0=/usr/bin/python a1=/bin/yum a2=updateinfo a3=list
type=SYSCALL msg=audit(09/05/2020 14:12:45.350:551) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x56536d272268 a1=0x56536d283ef8 a2=0x56536d2a81a0 a3=0x0 items=3 ppid=3423 pid=3425 auid=luke uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=1 comm=yum exe=/usr/bin/python2.7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=sudo_log
----

A couple things to notice in the log output:

  1. Notice each block is separated by three dashes. That shows you which log enteries are related.
  2. The top line shows which command was run. In my case /usr/bin/python /bin/yum updateinfo list (It shows the full path of the execution - I only typed yum updateinfo list.
  3. On the last line, if you scroll to the right (I know I hate those scroll boxes too, I just don't know how to make the lines wrap... sorry) you will see auid=luke. That shows you who ran the command, and if you scroll a little more to the right you will see euid=root (the effective user when the command was run).
  4. Each line contains a timestamp which is useful for tracing when a command was run - in the event that you are trying to build a timeline.

A note about my environment

My servers do not allow direct login as root. If yours do, then you will need to account for that and will probably need to remove the filter based on the original uid. Also I think you should reconsider the decision to allow root to login - at least via ssh.

Reading the log directly

You can skip the ausearch command if you want and query the audit log directly at /var/log/audit/audit.log using whatever method you like. But the output will not be as friendly as it appears here, uids are numeric, and the time stamps are shown in seconds since epoch, you don’t get the nice dashes between entries. But it’s all there in plain text if you want it.

For further reading on audit logs see this Red Hat doc:


If you found this useful please support the blog.

Fastmail

I use Fastmail to host my email for the blog. If you follow the link from this page you'll get a 10% discount and I'll get a little bit of break on my costs as well. It's a win win.

Backblaze

Backblaze is a cloud backup solution for Mac and Windows desktops. I use it on my home computers, and if you sign up using the link on this page you get a free month of service through backblaze, and so do I. If you're looking for a good backup solution give them a try!

Thanks!

Luke

#CentOS #Fedora #Linux