This post is about the audit daemon (auditd) that is available for most Linux systems.
Recently I’ve been looking at alternative ways to monitor sudo users on the servers I manage. Generally speaking it’s a good practice to keep an audit trail on managed systems. From a purely security perspective the more auditing you have on a system the easier any incident response should become when you need it. Your I.T. Security groups will need an easily searchable record of who ran which commands and with what privileges when trying to unravel how an exploit was used, or who used it, or both.
Outside of a security perspective you still want these controls in place to make sure that you can retrace any steps taken during changes while troubleshooting a problem. It’s all too common a scenario where a change goes wrong and somehow nobody knows what was changed. A robust audit trail can make hunting down which actions were taken much easier when figuring out what went wrong, and can go a long way towards finding a solution.
Historically we’ve always used the sudo log as a means to keep a record of all actions performed by users who need to escalate privileges to root while performing their job. One of the issues with the sudo log is that it does not keep a record of commands that are run when a user switches directly to root after logging in, at least not on a RHEL 7 system.
If a user logs in and immediately switches to root using
sudo su - root, or
sudo su -, then you won’t have any log of their activities while logged in. This is an undesirable situation.
I.E. a solution that doesn’t involve just telling people not to switch directly to root. With all that in mind I decided to play around with some auditd rules, and after a little trial and error (and some internet searching) I came upon a really good answer on StackExchange.
I’ll let you take a look at the answer for yourself if you so choose. The meat and potatoes of that post is to add this audit rule:
sudo auditctl -a always,exit -F arch=b64 -F euid=0 -S execve
As an asside problem it also generates a lot of noise, which could make your logs unnecessarily large. Like this:
---- type=PROCTITLE msg=audit(09/05/2020 12:36:50.142:1572) : proctitle=sed -n s/device for \(.*\): usb:.*=\(TAG.*\)$/\1 \2/p type=PATH msg=audit(09/05/2020 12:36:50.142:1572) : item=1 name=/lib64/ld-linux-x86-64.so.2 inode=2491677 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 type=PATH msg=audit(09/05/2020 12:36:50.142:1572) : item=0 name=/usr/bin/sed inode=2491999 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 type=CWD msg=audit(09/05/2020 12:36:50.142:1572) : cwd=/ type=EXECVE msg=audit(09/05/2020 12:36:50.142:1572) : argc=3 a0=sed a1=-n a2=s/device for \(.*\): usb:.*=\(TAG.*\)$/\1 \2/p type=SYSCALL msg=audit(09/05/2020 12:36:50.142:1572) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x19072f0 a1=0x1906ed0 a2=0x1905e10 a3=0x7ffc9cd52420 items=2 ppid=4471 pid=4473 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sed exe=/usr/bin/sed subj=system_u:system_r:unconfined_service_t:s0 key=root_cmd -----
Since I was looking for a way to audit commands run as root by real users I needed to filter out the system noise. By default most Linux distributions reserve the first 999 uid’s for system accounts - for reference see: Linux sysadmin basics: User account management with UIDs and GIDs.
After expermienting a little bit I’ve decided to modify that rule to filter down to just what I was looking for, and make the rule permanent.
Add the following lines to the file
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -F key=sudo_log -a always,exit -F arch=b32 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -F key=sudo_log
Then restart the auditd service.
service auditd restart
One odd thing to note with the auditd service is that you can not restart it with
systemctl. Instead you have to use the older
service command to force a restart of the audit daemon, why does it work that way? - I have no idea, but I noticed that Red Hat recommends using the service command to start/stop the auditd service here: Chapter 10. Auditing the system. If you know the answer to why that is the case and want to call me out as a noob please do so on Twitter.
Another odd thing to note here is that on an up-to-date CentOS 7 or Fedora 32 machine that command string will not work if you run it with
auditctl from the command line. For one reason or another auditctl doesn’t seem to like the
>= operator when it’s passed in from standard input.
So what do these rules do:
-a→ Pretty simple that means append. You are appending everything after the -a to the audit rules.
always,exit→ This means you want to write to the log all the time when this rule is triggered and include the exit code.
-F→ The F option is a way to separate each field. The pattern is:
name, operation, value. For instance the first field in our rule is
-S→ The S option defines the syscall we are looking for in this case execve is the syscall we want since it is the syscall for executing a program.
auid!=-1allows us to constrain our rule based on the following criteria.
auid!=-1, but you can also use
auid!=4294967295if your version of auditd doesn’t like the negative value, both do the same thing I just think -1 is more readable.
key=sudo_log. You could call your key anything you want, it is used to search for hits in the audit log.
The second rule is the same except that it watches for 32bit executables.
As always you should check the man pages for any command (
man auditctl in this case) you are not familiar with before just copying and pasting something from the internet. You don’t know me, I might be a bad guy, or incompetent, or both. I’d like to think I’m not a bad guy, or incompetent but read the man pages anyway, and while you’re at it check out these documents for extra credit.
After you have those rules in place and have restarted the auditd service, either with a reboot or with the service command, it’s time to test the new rules.
auditclt command to view your rules.
sudo auditctl -l
Next, run any command you want with
sudo or after switching to root (
sudo -i, or
sudo su - root, or
sudo su - if you’re really old school)
I’m going to run
yum updateinfo list
Then search your audit log with the ausearch command filtering on the key we created:
sudo ausearch -k sudo_log -i
---- type=PROCTITLE msg=audit(09/05/2020 14:12:45.350:551) : proctitle=/usr/bin/python /bin/yum updateinfo list type=PATH msg=audit(09/05/2020 14:12:45.350:551) : item=2 name=/lib64/ld-linux-x86-64.so.2 inode=2491677 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(09/05/2020 14:12:45.350:551) : item=1 name=/usr/bin/python inode=2494593 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(09/05/2020 14:12:45.350:551) : item=0 name=/bin/yum inode=2502974 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:rpm_exec_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=CWD msg=audit(09/05/2020 14:12:45.350:551) : cwd=/home/luke type=EXECVE msg=audit(09/05/2020 14:12:45.350:551) : argc=4 a0=/usr/bin/python a1=/bin/yum a2=updateinfo a3=list type=SYSCALL msg=audit(09/05/2020 14:12:45.350:551) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x56536d272268 a1=0x56536d283ef8 a2=0x56536d2a81a0 a3=0x0 items=3 ppid=3423 pid=3425 auid=luke uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=1 comm=yum exe=/usr/bin/python2.7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=sudo_log ----
A couple things to notice in the log output:
/usr/bin/python /bin/yum updateinfo list(It shows the full path of the execution - I only typed
yum updateinfo list.
auid=luke. That shows you who ran the command, and if you scroll a little more to the right you will see
euid=root(the effective user when the command was run).
My servers do not allow direct login as root. If yours do, then you will need to account for that and will probably need to remove the filter based on the original uid. Also I think you should reconsider the decision to allow root to login - at least via ssh.
You can skip the
ausearch command if you want and query the audit log directly at
/var/log/audit/audit.log using whatever method you like. But the output will not be as friendly as it appears here, uids are numeric, and the time stamps are shown in seconds since epoch, you don’t get the nice dashes between entries. But it’s all there in plain text if you want it.
For further reading on audit logs see this Red Hat doc:
If you’d like to get in touch, contact with me via email - or follow on Twitter.