"Cuckoo's Egg"-Style Server Tripwire Instructions

on

 "Cuckoo's Egg"-Style Server Tripwire


TL;DR: auditctl + syslog-ng 🤓

Why

Like many of you, I have a few services running on home servers. I protect our home network with a firewall, but wanted notice in case anyone found their way past the firewall. Plus, I love the book "The Cuckoo's Egg" especially the scene where Mister Stoll runs out of the shower when his tripwire alerted his pager. 


I started with a few free Canary Tokens that would alert me if someone opened a file that came from an area of  my home network the public should not have access to. I gave these files suitable honey pot names and placed them on my primary home Linux server. I knew the tokens were working when someone who _does_ have access opened one in Toronto (our son) and I received an email from Thinkst. Recently, I started getting a few canary token alerts from NYC. I do not know anyone with access to our home network in NYC. The canary token will tell you the geographic location (NYC) of the DNS server of the miscreant and which token was opened, but that is it. After that, it is time to do some of your own investigation.

I started by closing down all open firewall ports that were no longer in use and adding logging to all services that were not currently logging. I then started investigating how I could be alerted with more detail (time, service, user) when a token was accessed.

How

Detailed File Access Logging - auditctl



I started by searching for a way to get more detail on which service was used to access my token file. Was it a service with a forwarded firewall port or was someone in my system? Google led me to the auditctl set of Linux utilities after a quick stop at lsof. I've had hands-on with *NIX since 1986 (AT&T System V) but have never had to implement auditctl. Most likely as my UNIX job was in installation/programming and not operations. 

I created a two auditctl rules to log detailed data on canary token file access:

$ sudo auditctl -w /mount/point/JuicyFileName1.pdf -p war -k Juicy1
$ sudo auditctl -w /mount/point/JuicyFileName2.docx -p war -k Juicy2

I will now get a detailed log entry in /var/log/audit/audit.log whenever some process touches either of these files. The "-p war" means log an entry if some process attempt to w(rite) this file, change the a(ttributes) of this file or r(ead) this file. I really only needed a log on reads, so the rest is gravy. The audit.log entry looks like this when someone accesses one of these files from the ftp server:

type=SYSCALL msg=audit(1601243255.400:35697): arch=c000003e syscall=257 success=yes exit=4 a0=ffffff9c a1=56379062c1c0 a2=800 a3=0 items=1 ppid=22174 pid=22176 auid=4294967295 uid=9001 gid=9002 euid=9001 suid=9001 fsuid=9001 egid=9002 sgid=9002 fsgid=9002 tty=(none) ses=4294967295 comm="vsftpd" exe="/usr/sbin/vsftpd" key="Juicy1"

Lots of good information including which key was triggered and which process triggered the key. To get the same information in human readable format, use the "ausearch --interpret" command such as:

$ sudo ausearch --interpret --key Juicy1
type=PROCTITLE msg=audit(09/27/2020 17:59:23.252:35871) : proctitle=/usr/sbin/vsftpd /etc/vsftpd.conf 
type=PATH msg=audit(09/27/2020 17:59:23.252:35871) : item=0 name=/mount/point/JuicyFileName1.pdf inode=301163 dev=08:12 mode=file,777 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 
type=CWD msg=audit(09/27/2020 17:59:23.252:35871) : cwd=/mount/point 
type=SYSCALL msg=audit(09/27/2020 17:59:23.252:35871) : arch=x86_64 syscall=openat success=yes exit=4 a0=0xffffff9c a1=0x56379062c1c0 a2=O_RDONLY|O_NONBLOCK a3=0x0 items=1 ppid=23463 pid=23465 auid=unset uid=user1 gid=user1 euid=user1 suid=user1 fsuid=user1 egid=user1 sgid=user1 fsgid=user1 tty=(none) ses=unset comm=vsftpd exe=/usr/sbin/vsftpd key=Juicy1 

These rules are temporary and do not survive a reboot but are fine for testing. To make the rules permanent after testing, I added them (minus the "sudo auditctl" prefix) to /etc/audit/audit.rules:

-w /mount/point/JuicyFileName1.pdf -p war -k Juicy1
-w /mount/point/JuicyFileName2.docx -p war -k Juicy2

Intrusion Detection Systems (IDS)

OK. Now, how will I know right away if my auditctl tripwire has been tripped? I started with multiple system management/intrusion detections systems and finally ended up with the simple syslog-ng Community Edition. Here is what I tried along the way to syslog-ng:


  • OSSEC: To save myself a few headaches, I installed the docker version from atomicorp/ossec-docker. OSSEC would definitely do the job as it supports log monitoring, but the solution just seemed to _heavy_ fo my simple needs. Once I started reading about how to configure ossec.conf, I gave up.

  • Snort: Snort is actually very easy to get up and running. In fact, running Snort as a daemon gave me a laugh. Snorts job is to capture packets on the network which is important for IDS, but not really what I was looking for.

  • Suricata: Suricata is the darling of the Open Source IDS community. Again, I took the docker route to save myself a few headaches. Suricata installed fine, but one needs a few more pieces to display what is going on, namely Elasticsearch to catalog Suricata's findings and EveBox to display the results. This started getting very complicated for my simple use case, so I moved on.

  • SELKS ISO: The SELKS ISO from Stamus Networks is a pre-build OS image that includes Suricata and all the additional components to catalog, search, and display events. 
    • S - Suricata IDPS - http://suricata-ids.org/
    • E - Elasticsearch - https://www.elastic.co/products/elasticsearch
    • L - Logstash - https://www.elastic.co/products/logstash
    • K - Kibana - https://www.elastic.co/products/kibana
    • S - Scirius - https://github.com/StamusNetworks/scirius
    • EveBox - https://evebox.org/
  • SELKS works out of the box even in Live CD mode. Tons of functionality, but much more than I needed to alert on someone accessing my trigger files. If I was building a full-blown SIEM, this would probably be the way to go.

    Syslog-NG


    Syslog-ng (Community Edition) ended up being just what I needed for my simple use case of alerting on certain Linux messages. You can find a description of syslog-ng in the Administration Guide here. Setting up alerting on my auditctl rules consisted of two edits to /etc/syslog-ng/syslog-ng.conf and the creation of one shell script:

    One Shell Script

    My shell script (syslog-ng-email.sh) is passed a message from syslog-ng and emails the message with the last field becoming the email subject (awk). The script runs as an infinite loop, but even if the script dies, syslog-ng will start a new copy. (I also created a copy of this script that uses the free Simplepush.io service to send Android notifications🤖)

    #!/bin/bash
while read line ; do
     echo $line > /tmp/syslog-ng-email.txt
     sendemail -l email.log -f "sender@gmail.com" -u "Syslog-NG Alert `awk -F \" \" '{print $NF}' /tmp/syslog-ng-email.txt`" -t "recipient@gmail.com" -s "smtp.gmail.com:587" -o tls=yes -xu "sender@gmail.com" -xp "MySuperSecretPassword" -o message-file="/tmp/syslog-ng-email.txt"
done

    Two syslog-ng.conf Edits

    1. Added syslog-ng-email.sh as "d_prog" to the Destinations section: 

      ########################
      # Destinations
      ########################
      #Email Destination
      destination d_prog { program("/home/dennis/syslog-ng-email.sh" template("<${PRI}>${DATE} ${HOST} ${MESSAGE}\n") ); };

    2. Added combined log/filters of my auditctl keys to the log section to activate syslog-ng-email.sh defined as "d_prog"

      ########################
      # Log paths
      ########################
      #Log Juicy1 trigger file to email
      log {
          source(s_src);
          filter { match("key=\"Juicy1\"" value("MESSAGE")) };
          destination(d_prog);
      };
      #Log Juicy2 trigger file to email
      log {
          source(s_src);
          filter { match("key=\"Juicy2\"" value("MESSAGE")) };
          destination(d_prog);
      };

    Amazingly, it works. I now get detailed emails (and immediate Android notifications) if one of my Canary Token files is accessed on my server.

    Lessons Learned

    This was a great learning experience for me. I learned more about Linux auditing as well as IDS solutions along the way. A few key take-aways were:
    • IDS systems are many-faceted and one should not expect to set them up without RTFM as well as some experience. I continue to be impressed by security professionals.
    • Containers can make one's life easier, but only one container gets port 80 at a time on a container host.
    • Extremely lightweight, built-for-containers, virtual machines like Photon OS (or CoreOS or RancherOS) are awesome and can be used to provide a level of separation between containers.
    • http://hilite.me/ is really nice for formatting source code for blogs🙂

    Thank You

    As always, thanks for taking the time to read this post. I welcome your feedback and ideas for improvement.

    Comments

    Post a Comment