How to perform threat analysis on Apache log

Apache stores details of HTTP requests in its Access Log as defined by the CustomLog directive. The log can be used to check for malicious requests such as hacking or defacement attempts.

Scalp! is a security log analyzer for Apache. It automates reading the Apache log files and perform threat analysis.

Scalp! uses rulesets provided by PHP-IDS project and is available for download from GitHub.

1. Download Scalp from GitHub.

   $ git clone https://github.com/neuroo/apache-scalp scalp
   Cloning into 'scalp'...
   remote: Enumerating objects: 11, done.
   remote: Total 11 (delta 0), reused 0 (delta 0), pack-reused 11
   Unpacking objects: 100% (11/11), 11.85 KiB | 346.00 KiB/s, done.

   Related: How to install Git on Ubuntu

2. Download signature file from PHPIDS project on GitHub.

   $ git clone https://github.com/PHPIDS/PHPIDS phpids
   Cloning into 'phpids'...
   remote: Enumerating objects: 11281, done.
   remote: Total 11281 (delta 0), reused 0 (delta 0), pack-reused 11281
   Receiving objects: 100% (11281/11281), 4.16 MiB | 2.81 MiB/s, done.
   Resolving deltas: 100% (5636/5636), done.

3. Split Apache log file if longer than 10000 lines.

   $ split -l 10000  /var/log/apache2/access_log

4. Analyze Apache log file using Scalp and PHPIDS signature.

   $ sudo python scalp/scalp.py --log /var/log/apache2/access_log --filters phpids/lib/IDS/default_filter.xml
   Password:
   Loading XML file 'phpids/lib/IDS/default_filter.xml'...
   Processing the file '/var/log/apache2/access_log'...
   Scalp results:
   	Processed 1318 lines over 1318
   	Found 6 attack patterns in 0.425544 s
   Generating output in /home/user/access_log_scalp_*

   More options for Scalp:

   Scalp the apache log! by Romain Gaucher - http://rgaucher.info
   usage:  ./scalp.py [--log|-l log_file] [--filters|-f filter_file] [--period time-frame] [OPTIONS] [--attack a1,a2,..,an]
                      [--sample|-s 4.2]
      --log       |-l:  the apache log file './access_log' by default
      --filters   |-f:  the filter file     './default_filter.xml' by default
      --exhaustive|-e:  will report all type of attacks detected and not stop
                        at the first found
      --tough     |-u:  try to decode the potential attack vectors (may increase
                        the examination time)
      --period    |-p:  the period must be specified in the same format as in
                        the Apache logs using * as wild-card
                        ex: 04/Apr/2008:15:45;*/Mai/2008
                        if not specified at the end, the max or min are taken
      --html      |-h:  generate an HTML output
      --xml       |-x:  generate an XML output
      --text      |-t:  generate a simple text output (default)
      --except    |-c:  generate a file that contains the non examined logs due to the
                        main regular expression; ill-formed Apache log etc.
      --attack    |-a:  specify the list of attacks to look for
                        list: xss, sqli, csrf, dos, dt, spam, id, ref, lfi
                        the list of attacks should not contains spaces and comma separated
                        ex: xss,sqli,lfi,ref
      --ignore-ip|-i:  specify the list of IP Addresses to look exclude
                        the list of IP Addresses should be comma separated and not contain spaces
                        This option can be used in conjunction with --ignore-ip
      --ignore-subnet|-n:  specify the list of Subnets to look exclude
                        the list of Subnets should be comma separated and not contain spaces
                        This option can be used in conjunction with --ignore-subnet
      --output    |-o:  specifying the output directory; by default, scalp will try to write
                        in the same directory as the log file
      --sample    |-s:  use a random sample of the lines, the number (float in [0,100]) is
                        the percentage, ex: --sample 0.1 for 1/1000

5. Review generated output.

   $ cat /home/user/access_log_scalp_*
   
   #
   # File created by Scalp! by Romain Gaucher - http://code.google.com/p/apache-scalp
   # Apache log attack analysis tool based on PHP-IDS filters
   #
   Scalped file: access_log
   Creation date: Sat-08-Feb-2020
   
   Attack type: files
   Attack Cross-Site Scripting (xss)
   Attack Cross-Site Request Forgery (csrf)
   Attack Spam (spam)
   Attack Local File Inclusion (lfi)
   
   	### Impact 5
   	127.0.0.1 - - [08/Feb/2020:07:31:12 +0800] "GET /windows/start HTTP/1.1" 200 4541
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:16 +0800] "GET /windows/burn-iso-image HTTP/1.1" 200 4941
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:17 +0800] "GET /_media/windows/burn-iso-image/windows7-burn-iso-explorer-write-to-disc.png?w=400&tok=e21342 HTTP/1.1" 200 16104
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:17 +0800] "GET /windows/disable-system-restore HTTP/1.1" 200 4679
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:18 +0800] "GET /windows/firewall-command HTTP/1.1" 200 5511
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:19 +0800] "GET /windows/restore-mbr HTTP/1.1" 200 5323
   	Reason: "Detects specific directory and path traversal"
   
   Attack SQL Injection (sqli)
   Attack type: format string
   Attack Remote File Execution (rfe)
   Attack Denial Of Service (dos)
   Attack Directory Traversal (dt)
   Attack Information Disclosure (id)