Investigating a suspected intrusion on Linux establishes whether the host saw a failed probe, a short-lived foothold, or persistent operator access, which changes how urgently containment, credential rotation, and recovery need to happen.
The most useful evidence usually spans several layers at once: SSH and sudo authentication records, session databases, user and key metadata, persistence points such as timers or cron jobs, active listeners and processes, and recently changed files in high-value paths. Reading those sources together turns isolated anomalies into a timeline that can be checked, explained, and escalated.
Volatile state disappears quickly as connections close and processes exit, while shell histories, wtmp records, and rotated logs vary by distribution and retention policy. Start with read-only collection, prefer commands that preserve timestamps, and corroborate each high-signal finding against at least one other source before changing the host.
Related: Enable verbose SSH logging
Steps to investigate a Linux intrusion from host evidence:
- Record the current time, uptime, hostname, and kernel release before collecting anything else.
$ date --iso-8601=seconds 2026-04-14T12:02:50+08:00 $ uptime 12:02:50 up 32 sec, 1 user, load average: 0.14, 0.07, 0.02 $ hostname host $ uname -r 6.8.0-90-generic
Capture the host time zone in the same note set so later log correlation does not mix local time with UTC or another local offset.
- Check current interactive sessions before the source address or controlling terminal disappears.
$ w -h user 127.0.0.1 12:05 3:22 0.00s 0.02s sshd: user [priv] $ who -a system boot 2026-04-14 12:02 run-level 5 2026-04-14 12:02 LOGIN tty1 2026-04-14 12:02 1107 id=tty1w shows who is active right now, while who -a also exposes the current boot, runlevel, and waiting login terminals.
Related: How to list logged-in users in Linux
- Review the historical account trail with last and lslogins before trusting any single login database.
$ last -aiF | head -n 5 reboot system boot Tue Apr 14 12:02:18 2026 still running 0.0.0.0 reboot system boot Thu Jan 15 19:15:36 2026 - Thu Jan 15 19:17:13 2026 (00:01) 0.0.0.0 reboot system boot Thu Jan 15 19:14:27 2026 - Thu Jan 15 19:15:27 2026 (00:01) 0.0.0.0 reboot system boot Thu Jan 8 19:28:40 2026 - Thu Jan 8 19:33:28 2026 (00:04) 0.0.0.0 reboot system boot Thu Jan 8 19:27:20 2026 - Thu Jan 8 19:28:31 2026 (00:01) 0.0.0.0 $ lslogins -u user Username: user UID: 1000 Home directory: /home/user Shell: /bin/bash Supplementary groups: lxd,adm,cdrom,sudo,dip,plugdev Last logs: 12:06 sudo[2509]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000) 12:06 sudo[2509]: pam_unix(sudo:session): session closed for user root
Empty or reboot-only last output does not prove an account was unused. Some current Debian-family systems have moved away from the traditional /var/log/wtmp and /var/log/btmp tools, so lslogins, journalctl, or wtmpdb may carry the more useful trail.
- Filter SSH and sudo journal entries to build the first trustworthy timeline of access and privilege escalation.
$ sudo journalctl --since "24 hours ago" -t sshd -t sudo --output=short-iso --no-pager | tail -n 12 2026-04-14T12:04:46+08:00 host sshd[2236]: pam_unix(sshd:auth): check pass; user unknown 2026-04-14T12:04:46+08:00 host sshd[2236]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1 2026-04-14T12:04:48+08:00 host sshd[2236]: Failed password for invalid user invaliduser from 127.0.0.1 port 59562 ssh2 2026-04-14T12:04:49+08:00 host sshd[2236]: Connection closed by invalid user invaliduser 127.0.0.1 port 59562 [preauth] 2026-04-14T12:04:49+08:00 host sshd[2241]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1 user=user 2026-04-14T12:04:52+08:00 host sshd[2241]: Failed password for user from 127.0.0.1 port 42946 ssh2 2026-04-14T12:04:52+08:00 host sshd[2241]: Connection closed by authenticating user user 127.0.0.1 port 42946 [preauth] 2026-04-14T12:04:52+08:00 host sshd[2245]: Accepted password for user from 127.0.0.1 port 42956 ssh2 2026-04-14T12:04:52+08:00 host sshd[2245]: pam_unix(sshd:session): session opened for user user(uid=1000) by user(uid=0) 2026-04-14T12:04:53+08:00 host sshd[2343]: Received disconnect from 127.0.0.1 port 42956:11: disconnected by user 2026-04-14T12:04:53+08:00 host sshd[2343]: Disconnected from user user 127.0.0.1 port 42956 2026-04-14T12:04:53+08:00 host sshd[2245]: pam_unix(sshd:session): session closed for user user $ sudo journalctl --since "24 hours ago" -t sudo --output=short-iso --no-pager | tail -n 4 2026-04-14T12:04:43+08:00 host sudo[2096]: user : PWD=/ ; USER=root ; COMMAND=/usr/bin/systemctl enable --now ssh 2026-04-14T12:04:43+08:00 host sudo[2096]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000) 2026-04-14T12:04:45+08:00 host sudo[2232]: user : PWD=/ ; USER=root ; COMMAND=/usr/bin/id 2026-04-14T12:04:45+08:00 host sudo[2232]: pam_unix(sudo:session): session closed for user root
Hosts without useful journal history often write the same authentication events to /var/log/auth.log or /var/log/secure.
- Review the shell history file for the account's login shell to see what commands were run from an interactive session.
$ sudo tail -n 12 /home/user/.bash_history sudo /tmp/PrlToolsPackages0/install sudo reboot sudo apt update && sudo apt --assume-yes dist-upgrade && sudo apt autoclean && sudo apt autoremove --assume-yes && sudo poweroff sudo mount /dev/sr0 tmp/ df -h sudo /tmp/PrlToolsPackages0/install sudo apt update && sudo apt --assume-yes dist-upgrade && sudo apt autoclean && sudo apt autoremove --assume-yes && sudo poweroff df -h sudo /tmp/PrlToolsPackages0/install sudo reboot sudo apt update && sudo apt --assume-yes dist-upgrade && sudo apt autoclean && sudo apt autoremove --assume-yes && sudo poweroff
Shell history usually updates only when the shell writes it back to disk, and the relevant file may be .zsh_history, .ash_history, or another shell-specific path instead of .bash_history.
- Audit the target account, its group membership, and privilege path before assuming the compromise stayed inside one login.
$ sudo getent passwd user user:x:1000:1000:user:/home/user:/bin/bash $ sudo getent group sudo sudo:x:27:user $ sudo getent group wheel
Unexpected new accounts, recent additions to sudo or wheel, and new /etc/sudoers.d entries are high-signal changes during triage.
- Inspect SSH key material and its timestamps to see whether remote access was opened through authorized_keys.
$ sudo ls -al /home/user/.ssh/authorized_keys -rw------- 1 user user 91 Apr 14 12:04 /home/user/.ssh/authorized_keys $ sudo stat /home/user/.ssh/authorized_keys File: /home/user/.ssh/authorized_keys Size: 91 Blocks: 8 IO Block: 4096 regular file Device: 252,0 Inode: 1179935 Links: 1 Access: (0600/-rw-------) Uid: ( 1000/ user) Gid: ( 1000/ user) Access: 2026-04-14 12:04:43.803000071 +0800 Modify: 2026-04-14 12:04:43.803000071 +0800 Change: 2026-04-14 12:04:43.803000071 +0800 Birth: 2026-04-14 12:02:23.729000004 +0800
Correlate unexpected key additions with the matching sshd and sudo events before removing anything, especially on shared admin accounts or automation users.
- Inspect cron jobs and systemd timers for persistence that survives logouts and reboots.
$ sudo crontab -l -u user no crontab for user $ sudo systemctl list-timers --all --no-pager | head -n 10 NEXT LEFT LAST PASSED UNIT ACTIVATES Tue 2026-04-14 12:07:18 +08 1min 59s - - update-notifier-download.timer update-notifier-download.service Tue 2026-04-14 12:10:00 +08 4min 40s - - sysstat-collect.timer sysstat-collect.service Tue 2026-04-14 12:17:13 +08 11min - - systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Tue 2026-04-14 12:49:23 +08 44min Mon 2025-09-29 05:30:11 +08 - apt-daily-upgrade.timer apt-daily-upgrade.service Tue 2026-04-14 12:55:25 +08 50min Thu 2026-01-08 19:33:20 +08 - fwupd-refresh.timer fwupd-refresh.service Tue 2026-04-14 13:27:03 +08 1h 21min Sat 2024-04-27 07:58:34 +08 - man-db.timer man-db.service Tue 2026-04-14 14:40:16 +08 2h 34min Sat 2025-07-19 07:06:03 +08 - apt-daily.timer apt-daily.service Tue 2026-04-14 19:37:14 +08 7h Mon 2025-09-29 05:34:11 +08 - motd-news.timer motd-news.service
Disabling or deleting a suspicious timer or cron entry before capture can destroy the exact command, path, and timestamp evidence needed to explain how persistence was installed.
- Check listening sockets and connected endpoints for unexpected exposure or operator beacons.
$ sudo ss -tupn | head -n 8 Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess $ sudo ss -tulpn | head -n 8 Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=541,fd=16)) udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=541,fd=14)) udp UNCONN 0 0 10.211.55.28%enp0s5:68 0.0.0.0:* users:(("systemd-network",pid=529,fd=21)) tcp LISTEN 0 5 0.0.0.0:8888 0.0.0.0:* users:(("python3",pid=2229,fd=3)) tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=541,fd=17)) tcp LISTEN 0 4096 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=2220,fd=3),("systemd",pid=1,fd=165)) tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=541,fd=15))New listeners on public interfaces, user-owned processes binding unexpected ports, and outbound sessions to unfamiliar addresses are strong indicators to correlate with the process list and recent file changes.
- List long-running and recently started processes so suspicious paths can be tied back to sockets, accounts, and service units.
$ ps -eo pid,ppid,user,lstart,cmd --sort=lstart | tail -n 6 2082 1 user Tue Apr 14 12:04:42 2026 /usr/lib/systemd/systemd --user 2083 2082 user Tue Apr 14 12:04:42 2026 (sd-pam) 2220 1 root Tue Apr 14 12:04:43 2026 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups 2229 1 user Tue Apr 14 12:04:43 2026 python3 /home/user/.cache/.svc/agent.py ##### snipped #####
Pay extra attention to processes launched from user-writable paths such as /tmp, /var/tmp, /dev/shm, hidden directories under ~, or application cache directories.
- Check the kernel taint state and recent kernel messages in case the intrusion involved modules, crashes, or low-level runtime changes.
$ cat /proc/sys/kernel/tainted 0 $ sudo dmesg -T | tail -n 8 [Tue Apr 14 12:02:21 2026] input: HDA Intel Speaker as /devices/pci0000:00/0000:00:01.0/sound/card0/input9 [Tue Apr 14 12:02:21 2026] input: HDA Intel Speaker as /devices/pci0000:00/0000:00:01.0/sound/card0/input10 [Tue Apr 14 12:02:21 2026] cfg80211: Loading compiled-in X.509 certificates for regulatory database [Tue Apr 14 12:02:21 2026] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' [Tue Apr 14 12:02:21 2026] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' [Tue Apr 14 12:02:22 2026] loop0: detected capacity change from 0 to 8 [Tue Apr 14 12:02:22 2026] NET: Registered PF_QIPCRTR protocol family [Tue Apr 14 12:03:54 2026] hrtimer: interrupt took 2849250 ns
A non-zero taint value can reflect proprietary modules or kernel conditions that make runtime trust decisions harder to interpret. If direct dmesg access is blocked, pivot to sudo journalctl -k --no-pager.
- Search for recently changed files and privileged binaries inside the suspected time window.
$ sudo find /etc /home/user -xdev -type f -newermt "2026-04-14 12:00:00" -ls 2>/dev/null | head -n 10 656525 4 -rw-r--r-- 1 root root 711 Apr 14 12:04 /etc/hosts.deny 656291 4 -rw-r--r-- 1 root root 219 Apr 14 12:02 /etc/hosts 657037 24 -rw-r--r-- 1 root root 23455 Apr 14 12:04 /etc/ld.so.cache 656900 4 -rw-r----- 1 root shadow 927 Apr 14 12:04 /etc/shadow 656899 4 -rw-r--r-- 1 root root 1650 Apr 14 12:04 /etc/passwd 1341641 4 -rw-rw-r-- 1 user user 241 Apr 14 12:04 /home/user/.cache/.svc/agent.py 1179935 4 -rw------- 1 user user 91 Apr 14 12:04 /home/user/.ssh/authorized_keys 1179943 4 -rw-r--r-- 1 user user 91 Apr 14 12:04 /home/user/.ssh/investigate_demo.pub $ sudo find / -xdev -type f -perm -4000 -ls 2>/dev/null | head -n 8 1071014 68 -rwsr-xr-x 1 root root 67664 Dec 2 2024 /usr/lib/polkit-1/polkit-agent-helper-1 1072165 324 -rwsr-xr-x 1 root root 330104 Mar 5 01:55 /usr/lib/openssh/ssh-keysign 1051868 196 -rwsr-xr-x 1 root root 199752 May 21 2025 /usr/lib/snapd/snap-confine 1052235 328 -rwsr-xr-x 1 root root 335120 Jun 25 2025 /usr/bin/sudo ##### snipped #####
Wide scans can take time and can touch access timestamps on some mounts, so start with the time window and directories that matter most to the incident.
- Record indicators, timestamps, and hashes in a write-protected incident timeline before containment changes the host.
$ printf '%s\t%s\t%s\n' '2026-04-14T12:04:48+08:00' 'ssh' 'Failed password for invaliduser from 127.0.0.1' >> incident-timeline.tsv $ printf '%s\t%s\t%s\n' '2026-04-14T12:04:43+08:00' 'process' 'python3 /home/user/.cache/.svc/agent.py pid 2229' >> incident-timeline.tsv $ sha256sum /home/user/.cache/.svc/agent.py c309f4c1b0b62f007cd9694cfb5b5f6bbac7d5fce91597b5400a1c584280ca94 /home/user/.cache/.svc/agent.py
Keep the timeline, copied log excerpts, and collected hashes in a location that ordinary users cannot edit, and include the command line, account name, source address, and file path for each indicator.
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
