How to perform threat analysis on Apache log file using Scalp

Scalp! is a security log analyzer for the Apache log files. It reads the Apache log and perform threat analysis against rulesets provided by PHP-IDS project. It is available for download from GitHub.

Steps to perform threat analysis on Apache log file:

1. Download Scalp from GitHub.

   $ git clone https://github.com/neuroo/apache-scalp scalp
   Cloning into 'scalp'...
   remote: Enumerating objects: 11, done.
   remote: Total 11 (delta 0), reused 0 (delta 0), pack-reused 11
   Unpacking objects: 100% (11/11), 11.85 KiB | 346.00 KiB/s, done.

   How to install Git client on Ubuntu

2. Download signature file from PHPIDS project on GitHub.

   $ git clone https://github.com/PHPIDS/PHPIDS phpids
   Cloning into 'phpids'...
   remote: Enumerating objects: 11281, done.
   remote: Total 11281 (delta 0), reused 0 (delta 0), pack-reused 11281
   Receiving objects: 100% (11281/11281), 4.16 MiB | 2.81 MiB/s, done.
   Resolving deltas: 100% (5636/5636), done.

3. Split Apache log file if longer than 10000 lines.

   $ split -l 10000  /var/log/apache2/access_log

4. Analyze Apache log file using Scalp and PHPIDS signature.

   $ sudo python scalp/scalp.py --log /var/log/apache2/access_log --filters phpids/lib/IDS/default_filter.xml
   Password:
   Loading XML file 'phpids/lib/IDS/default_filter.xml'...
   Processing the file '/var/log/apache2/access_log'...
   Scalp results:
   	Processed 1318 lines over 1318
   	Found 6 attack patterns in 0.425544 s
   Generating output in /home/user/access_log_scalp_*

   More options for Scalp:

   Scalp the apache log! by Romain Gaucher - http://rgaucher.info
   usage:  ./scalp.py [--log|-l log_file] [--filters|-f filter_file] [--period time-frame] [OPTIONS] [--attack a1,a2,..,an]
                      [--sample|-s 4.2]
      --log       |-l:  the apache log file './access_log' by default
      --filters   |-f:  the filter file     './default_filter.xml' by default
      --exhaustive|-e:  will report all type of attacks detected and not stop
                        at the first found
      --tough     |-u:  try to decode the potential attack vectors (may increase
                        the examination time)
      --period    |-p:  the period must be specified in the same format as in
                        the Apache logs using * as wild-card
                        ex: 04/Apr/2008:15:45;*/Mai/2008
                        if not specified at the end, the max or min are taken
      --html      |-h:  generate an HTML output
      --xml       |-x:  generate an XML output
      --text      |-t:  generate a simple text output (default)
      --except    |-c:  generate a file that contains the non examined logs due to the
                        main regular expression; ill-formed Apache log etc.
      --attack    |-a:  specify the list of attacks to look for
                        list: xss, sqli, csrf, dos, dt, spam, id, ref, lfi
                        the list of attacks should not contains spaces and comma separated
                        ex: xss,sqli,lfi,ref
      --ignore-ip|-i:  specify the list of IP Addresses to look exclude
                        the list of IP Addresses should be comma separated and not contain spaces
                        This option can be used in conjunction with --ignore-ip
      --ignore-subnet|-n:  specify the list of Subnets to look exclude
                        the list of Subnets should be comma separated and not contain spaces
                        This option can be used in conjunction with --ignore-subnet
      --output    |-o:  specifying the output directory; by default, scalp will try to write
                        in the same directory as the log file
      --sample    |-s:  use a random sample of the lines, the number (float in [0,100]) is
                        the percentage, ex: --sample 0.1 for 1/1000

5. Review generated output.

   $ cat /home/user/access_log_scalp_*
   
   #
   # File created by Scalp! by Romain Gaucher - http://code.google.com/p/apache-scalp
   # Apache log attack analysis tool based on PHP-IDS filters
   #
   Scalped file: access_log
   Creation date: Sat-08-Feb-2020
   
   Attack type: files
   Attack Cross-Site Scripting (xss)
   Attack Cross-Site Request Forgery (csrf)
   Attack Spam (spam)
   Attack Local File Inclusion (lfi)
   
   	### Impact 5
   	127.0.0.1 - - [08/Feb/2020:07:31:12 +0800] "GET /windows/start HTTP/1.1" 200 4541
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:16 +0800] "GET /windows/burn-iso-image HTTP/1.1" 200 4941
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:17 +0800] "GET /_media/windows/burn-iso-image/windows7-burn-iso-explorer-write-to-disc.png?w=400&tok=e21342 HTTP/1.1" 200 16104
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:17 +0800] "GET /windows/disable-system-restore HTTP/1.1" 200 4679
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:18 +0800] "GET /windows/firewall-command HTTP/1.1" 200 5511
   	Reason: "Detects specific directory and path traversal"
   
   	127.0.0.1 - - [08/Feb/2020:07:31:19 +0800] "GET /windows/restore-mbr HTTP/1.1" 200 5323
   	Reason: "Detects specific directory and path traversal"
   
   Attack SQL Injection (sqli)
   Attack type: format string
   Attack Remote File Execution (rfe)
   Attack Denial Of Service (dos)
   Attack Directory Traversal (dt)
   Attack Information Disclosure (id)