Investigating a suspected Linux intrusion starts with preserving volatile host evidence before containment or cleanup changes the scene. The first pass separates background probes from a real account login, privilege escalation, persistence, or a live process that still needs isolation.
Host evidence usually spans systemd journals, session databases, account records, SSH keys, schedulers, sockets, processes, kernel messages, and recently changed files. Reading those sources in time order turns separate warnings into a timeline that can be escalated, scoped, and later compared with firewall, endpoint, or SIEM records.
Use read-only commands first and save output with timestamps and time zone context. Login databases, rotated logs, shell history, and journal retention vary across distributions, and current Ubuntu releases may use wtmpdb where older systems used only /var/log/wtmp, so corroborate each high-signal finding against another source before removing users, keys, jobs, or files.
Related: Enable verbose SSH logging
Steps to investigate a Linux intrusion from host evidence:
- Record the collection time before reviewing logs.
$ date --iso-8601=seconds 2026-04-14T12:06:22+08:00
Keep the host time zone with every copied excerpt so later correlation does not mix local time, UTC, and SIEM display time.
- Check active sessions before users disconnect or source terminals disappear.
$ w 12:06:24 up 4 min, 2 users, load average: 0.12, 0.08, 0.03 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT user pts/0 203.0.113.44 12:04 1:31 0.03s 0.01s sshd: user analyst pts/1 198.51.100.10 12:05 0.00s 0.02s w
w is a volatile view of who is logged in right now. Capture it before restarting network services or killing sessions.
Related: How to show logged-in users in Linux - Review the login database for recent sessions and reboots.
$ last -a -i -F user pts/0 Tue Apr 14 12:04:52 2026 still logged in 203.0.113.44 reboot system boot Tue Apr 14 12:02:18 2026 still running 0.0.0.0 user pts/0 Mon Apr 13 23:17:09 2026 - Mon Apr 13 23:21:02 2026 (00:03) 198.51.100.25 ##### snipped #####
If last is missing on a host that uses wtmpdb, run wtmpdb last --time-format iso --hostlast for the same session-database check. Empty or reboot-only output does not prove an account was unused.
Related: How to check user login history in Linux - Build the first authentication timeline from SSH and sudo journal entries.
$ sudo journalctl --since "24 hours ago" -t sshd -t sudo --output=short-iso --no-pager 2026-04-14T12:04:48+08:00 host sshd[2236]: Failed password for invalid user backup from 203.0.113.44 port 59562 ssh2 2026-04-14T12:04:49+08:00 host sshd[2236]: Connection closed by invalid user backup 203.0.113.44 port 59562 [preauth] 2026-04-14T12:04:52+08:00 host sshd[2245]: Accepted password for user from 203.0.113.44 port 42956 ssh2 2026-04-14T12:04:52+08:00 host sshd[2245]: pam_unix(sshd:session): session opened for user user(uid=1000) by user(uid=0) 2026-04-14T12:05:03+08:00 host sudo[2331]: user : PWD=/home/user ; USER=root ; COMMAND=/usr/bin/systemctl enable --now ssh 2026-04-14T12:05:03+08:00 host sudo[2331]: pam_unix(sudo:session): session opened for user root(uid=0) by user(uid=1000) ##### snipped #####
On non-systemd hosts or hosts without retained journal entries, check /var/log/auth.log on Debian-family systems or /var/log/secure on RHEL-family systems.
Related: How to check authentication logs in Linux - Read the target account shell history without treating it as complete evidence.
$ sudo cat /home/user/.bash_history sudo systemctl enable --now ssh mkdir -p ~/.cache/.svc python3 ~/.cache/.svc/agent.py ssh-keygen -lf ~/.ssh/authorized_keys ##### snipped #####
Shell history can be disabled, truncated, edited, or written only when an interactive shell exits. Check .zsh_history, .ash_history, and other shell-specific files when the account does not use bash.
Related: How to check commands run by a user in Linux - Confirm the target account identity and privilege groups.
$ id user uid=1000(user) gid=1000(user) groups=1000(user),27(sudo),1001(appdeploy)
Unexpected new accounts, new membership in sudo or wheel, and new files under /etc/sudoers.d/ are high-signal privilege changes during triage.
Related: How to audit local user accounts in Linux - Check the authorized_keys timestamp and ownership for the account.
$ sudo stat /home/user/.ssh/authorized_keys File: /home/user/.ssh/authorized_keys Size: 91 Blocks: 8 IO Block: 4096 regular file Device: 252,0 Inode: 1179935 Links: 1 Access: (0600/-rw-------) Uid: ( 1000/ user) Gid: ( 1000/ user) Access: 2026-04-14 12:04:43.803000071 +0800 Modify: 2026-04-14 12:04:43.803000071 +0800 Change: 2026-04-14 12:04:43.803000071 +0800 Birth: 2026-04-14 12:02:23.729000004 +0800
Correlate key-file Modify and Change times with sshd, sudo, and shell-history entries before deleting the key.
Related: How to check SSH authorized keys in Linux - Record fingerprints for any unexpected SSH public keys.
$ sudo ssh-keygen -lf /home/user/.ssh/authorized_keys 256 SHA256:GBeV49yqFZTNz76D2Nyb/EnWxv5zDjf0FbVxgWIjpQI incident@example (ED25519)
Compare the fingerprint with inventory, ticket, or administrator records before deciding whether the key is legitimate.
Tool: Secure Shell (SSH) Key Fingerprint Checker - Check the account crontab for logout-surviving commands.
$ sudo crontab -l -u user @reboot /usr/bin/python3 /home/user/.cache/.svc/agent.py
Copy the exact command, owner, and schedule before disabling a suspicious cron entry; deleting it first can remove the only readable persistence clue.
Related: How to check for persistence via cron and systemd in Linux - Check systemd timers for service-based persistence.
$ sudo systemctl list-timers --all --no-pager NEXT LEFT LAST PASSED UNIT ACTIVATES Tue 2026-04-14 12:10:00 +08 3min left - - app-cache-refresh.timer app-cache-refresh.service Tue 2026-04-14 12:17:13 +08 10min left Tue 2026-04-14 12:02:31 +08 4min systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Tue 2026-04-14 12:49:23 +08 42min left Mon 2026-04-13 12:49:23 +08 23h apt-daily-upgrade.timer apt-daily-upgrade.service ##### snipped #####
Investigate unfamiliar timer units with sudo systemctl cat <unit>.timer and sudo systemctl cat <unit>.service before stopping them.
Related: How to check for persistence via cron and systemd in Linux - List listening sockets and connected endpoints.
$ sudo ss -tulpn Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp LISTEN 0 4096 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=2220,fd=3),("systemd",pid=1,fd=165)) tcp LISTEN 0 5 0.0.0.0:8888 0.0.0.0:* users:(("python3",pid=2229,fd=3)) udp UNCONN 0 0 127.0.0.53:53 0.0.0.0:* users:(("systemd-resolve",pid=541,fd=15))New public listeners, user-owned processes bound to high ports, and outbound sessions to unfamiliar addresses should be tied back to process paths and login times.
Related: How to check active network connections in Linux - Sort processes by shortest elapsed runtime to surface recent launches.
$ ps -eo pid,ppid,user,etimes,lstart,cmd --sort=etimes PID PPID USER ELAPSED STARTED CMD 2384 2359 user 18 Tue Apr 14 12:06:09 2026 ps -eo pid,ppid,user,etimes,lstart,cmd --sort=etimes 2229 1 user 101 Tue Apr 14 12:04:43 2026 python3 /home/user/.cache/.svc/agent.py 2220 1 root 102 Tue Apr 14 12:04:43 2026 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups ##### snipped #####Give extra attention to processes launched from user-writable paths such as /tmp, /var/tmp, /dev/shm, hidden directories under home directories, and application cache directories.
- Check current-boot kernel warnings for low-level changes.
$ sudo journalctl -k -p warning..alert -b --no-pager -- No entries --
If kernel warnings are present, copy the module, device, audit, or crash lines into the incident timeline and compare them with process and file-change times.
- Search high-value paths for files changed during the suspected window.
$ sudo find /etc /home/user -xdev -type f -newermt "2026-04-14 12:00:00" -ls 656900 4 -rw-r----- 1 root shadow 927 Apr 14 12:04 /etc/shadow 656899 4 -rw-r--r-- 1 root root 1650 Apr 14 12:04 /etc/passwd 1341641 4 -rw-rw-r-- 1 user user 241 Apr 14 12:04 /home/user/.cache/.svc/agent.py 1179935 4 -rw------- 1 user user 91 Apr 14 12:04 /home/user/.ssh/authorized_keys ##### snipped #####
Wide file scans can take time and can touch access timestamps on some mounts. Start with the incident window and the directories that matter most to the suspected access path.
Related: How to check file changes in Linux - Search the current filesystem for setuid-root binaries.
$ sudo find / -xdev -type f -perm -4000 -ls 1071014 68 -rwsr-xr-x 1 root root 67664 Dec 2 2024 /usr/lib/polkit-1/polkit-agent-helper-1 1072165 324 -rwsr-xr-x 1 root root 330104 Mar 5 01:55 /usr/lib/openssh/ssh-keysign 1052235 328 -rwsr-xr-x 1 root root 335120 Jun 25 2025 /usr/bin/sudo ##### snipped #####
Compare this list with a known-good baseline when available. A new setuid-root binary in a writable or unusual path deserves immediate escalation.
Related: How to list files and folders recursively in Linux - Hash suspicious files before moving or deleting them.
$ sha256sum /home/user/.cache/.svc/agent.py c309f4c1b0b62f007cd9694cfb5b5f6bbac7d5fce91597b5400a1c584280ca94 /home/user/.cache/.svc/agent.py
Hash files from their original path first, then copy them to evidence storage that ordinary users cannot edit.
- Record high-signal indicators in an incident timeline before containment.
- incident-timeline.tsv
timestamp source indicator 2026-04-14T12:04:52+08:00 sshd Accepted password for user from 203.0.113.44 2026-04-14T12:05:03+08:00 sudo user ran /usr/bin/systemctl enable --now ssh 2026-04-14T12:04:43+08:00 process python3 /home/user/.cache/.svc/agent.py pid 2229 2026-04-14T12:04:43+08:00 file /home/user/.ssh/authorized_keys modified
Include timestamps, accounts, source addresses, command lines, file paths, hashes, and the evidence source for each row.
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
