Investigating a suspected intrusion on Linux clarifies what access occurred, what changed, and which identities and systems are affected, enabling containment and recovery decisions based on evidence instead of assumptions.
Most intrusion evidence comes from authentication and session telemetry (sshd, sudo, PAM, systemd-journald or syslog), plus account and key material (/etc/passwd, /etc/shadow, /etc/sudoers, /home/*/.ssh) and filesystem metadata that reveals when and where changes landed.
Volatile state disappears quickly as processes exit and connections close, while logs can rotate and history files can be truncated or cleared, so prioritize high-value collection early and favor read-only inspection until evidence handling and escalation paths are clear.
Related: Enable verbose SSH logging
$ date --iso-8601=seconds
2026-01-14T09:00:39+01:00
$ uptime
09:00:39 up 2 days, 18:53, 0 user, load average: 0.37, 0.27, 0.22
$ hostnamectl
Static hostname: 14844ef6f7fc
Icon name: computer-container
Chassis: container
Machine ID: dfae20ca11e742c090f9a9f100772b43
Boot ID: aa8df42ac7eb431c9412beba3d46fae4
Virtualization: docker
Operating System: Ubuntu 24.04.3 LTS
Kernel: Linux 6.12.54-linuxkit
Architecture: arm64
Include time zone and NTP status in notes to avoid a skewed timeline.
$ last -aiF user | head -n 5 wtmp begins Sun Dec 21 08:15:32 2025
Correlate source IPs with sshd logs to confirm the authentication method and session start.
$ w -h
Related: How to list logged-in users in Linux
$ sudo journalctl --since "24 hours ago" _COMM=sshd --no-pager | head -n 6 Jan 13 23:21:07 host.example.net sshd[14183]: Connection closed by ::1 port 40732 [preauth] Jan 13 23:21:07 host.example.net sshd[14185]: Connection closed by ::1 port 40744 [preauth] Jan 13 23:21:07 host.example.net sshd[14184]: Connection closed by ::1 port 40742 [preauth] Jan 13 23:21:07 host.example.net sshd[14186]: Unable to negotiate with ::1 port 40748: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth] Jan 13 23:21:07 host.example.net sshd[14187]: Unable to negotiate with ::1 port 40760: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth] Jan 13 23:22:16 host.example.net sshd[14306]: fatal: Access denied for user backupuser by PAM account configuration [preauth] $ sudo journalctl --since "24 hours ago" _COMM=sudo --no-pager | head -n 5 Jan 13 09:03:54 host.example.net sudo[8729]: user : PWD=/ ; USER=root ; COMMAND=/usr/bin/sync Jan 13 09:03:54 host.example.net sudo[8729]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Jan 13 09:03:54 host.example.net sudo[8729]: pam_unix(sudo:session): session closed for user root Jan 13 09:03:54 host.example.net sudo[8744]: user : PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /proc/sys/vm/drop_caches Jan 13 09:03:54 host.example.net sudo[8744]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Systems using log files instead of journald commonly store auth events in /var/log/auth.log or /var/log/secure.
$ sudo tail -n 15 /home/user/.bash_history #1736198400 sudo -l #1736198460 sudo systemctl status ssh #1736198520 sudo journalctl -u ssh --since yesterday #1736198580 ss -tulpn #1736198640 ps auxf
Shell history may not be written until the shell exits, and history files can be truncated or missing.
$ sudo getent passwd user user:x:1001:1001:,,,:/home/user:/usr/bin/zsh $ sudo getent group sudo sudo:x:27:ubuntu,user $ sudo getent group wheel
Unexpected new users, added group membership (sudo/wheel), or modified sudoers entries are high-signal changes.
$ sudo ls -al /home/user/.ssh/authorized_keys -rw------- 1 user user 90 Jan 14 09:00 /home/user/.ssh/authorized_keys $ sudo stat /home/user/.ssh/authorized_keys File: /home/user/.ssh/authorized_keys Size: 90 Blocks: 8 IO Block: 4096 regular file Device: 0,64 Inode: 382356 Links: 1 Access: (0600/-rw-------) Uid: ( 1001/ user) Gid: ( 1001/ user) Access: 2026-01-14 08:56:47.429324635 +0100 Modify: 2026-01-14 09:00:39.851476169 +0100 Change: 2026-01-14 09:00:39.852479422 +0100 Birth: 2026-01-14 08:56:47.429324635 +0100
Unknown keys and recently changed timestamps on key files merit immediate correlation with login and sudo events.
$ sudo crontab -l -u user no crontab for user $ sudo systemctl list-timers --all --no-pager | head -n 8 NEXT LEFT LAST PASSED UNIT ACTIVATES Wed 2026-01-14 23:32:25 CET 14h Tue 2026-01-13 23:33:16 CET 9h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service - - - - logrotate.timer logrotate.service 2 timers listed.
Disabling or deleting suspicious jobs before capture can destroy attribution and timing evidence.
$ ss -tupn | head -n 8
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
$ ss -tulpn | head -n 6
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=23657,fd=16))
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=23657,fd=14))
tcp LISTEN 0 4096 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=3846,fd=3),("systemd",pid=1,fd=64))
tcp LISTEN 0 5 0.0.0.0:8000 0.0.0.0:* users:(("python3",pid=21697,fd=3))
tcp LISTEN 0 1024 0.0.0.0:8888 0.0.0.0:* users:(("tinyproxy",pid=21562,fd=3))
Outbound connections from unexpected binaries and listeners bound to public interfaces are common intrusion pivots.
$ ps -eo pid,ppid,user,lstart,cmd --sort=lstart | tail -n 8 23419 1 root Wed Jan 14 01:48:35 2026 /usr/sbin/NetworkManager --no-daemon 23657 1 systemd+ Wed Jan 14 01:51:21 2026 /usr/lib/systemd/systemd-resolved 24844 1 root Wed Jan 14 02:04:03 2026 Xvfb :99 -screen 0 1024x768x24 26597 1 root Wed Jan 14 03:01:51 2026 /usr/libexec/upowerd 30508 1 root Wed Jan 14 09:00:10 2026 /usr/lib/systemd/systemd-hostnamed 30761 0 root Wed Jan 14 09:00:40 2026 bash -lc ps -eo pid,ppid,user,lstart,cmd --sort=lstart | tail -n 8 30769 30761 root Wed Jan 14 09:00:40 2026 ps -eo pid,ppid,user,lstart,cmd --sort=lstart 30770 30761 root Wed Jan 14 09:00:40 2026 tail -n 8
Execution from user-writable locations like /tmp, /dev/shm, and hidden directories under /home is a strong anomaly signal.
$ cat /proc/sys/kernel/tainted 4096 $ sudo dmesg -T | tail -n 6 [Wed Jan 14 08:49:25 2026] loop4: detected capacity change from 0 to 262144 [Wed Jan 14 08:49:25 2026] EXT4-fs (loop4): mounted filesystem 768d7f68-caaa-4c34-a5d7-a4eb6307c14b r/w with ordered data mode. Quota mode: none. [Wed Jan 14 08:50:07 2026] loop5: detected capacity change from 0 to 262144 [Wed Jan 14 08:50:07 2026] EXT4-fs (loop5): mounted filesystem a1d70e1f-1791-48a3-b120-442d1b87a8ea r/w with ordered data mode. Quota mode: none. [Wed Jan 14 08:50:23 2026] EXT4-fs (loop5): unmounting filesystem a1d70e1f-1791-48a3-b120-442d1b87a8ea. [Wed Jan 14 08:50:23 2026] EXT4-fs (loop4): unmounting filesystem 768d7f68-caaa-4c34-a5d7-a4eb6307c14b.
A non-zero taint value can indicate proprietary modules or kernel conditions that complicate trust in runtime telemetry.
$ sudo find /etc /usr/local /var/www -xdev -type f -mtime -2 -ls 2>/dev/null | head -n 6
278283 4 -rw-r----- 1 root shadow 744 Jan 14 01:47 /etc/gshadow-
27443 4 -rw-r--r-- 1 root root 249 Jan 14 01:43 /etc/hosts
382576 4 -rw-r--r-- 1 root root 1767 Jan 14 03:02 /etc/passwd
357983 4 -rw-r--r-- 1 root root 877 Jan 14 01:47 /etc/group-
29697 4 -rw-r--r-- 1 root root 77 Jan 13 02:11 /etc/security/faillock.conf
272255 4 -rw-r--r-- 1 root root 1435 Jan 13 01:01 /etc/pam.d/common-session-noninteractive
$ sudo find / -xdev -type f -perm -4000 -ls 2>/dev/null | head -n 6
56087 324 -rwsr-xr-x 1 root root 330104 Aug 26 15:49 /usr/lib/openssh/ssh-keysign
270700 68 -rwsr-xr-x 1 root root 67664 Dec 2 2024 /usr/lib/polkit-1/polkit-agent-helper-1
30253 68 -rwsr-xr-- 1 root messagebus 67504 Aug 9 2024 /usr/lib/dbus-1.0/dbus-daemon-launch-helper
357639 476 -rwsr-xr-- 1 root dip 485200 Apr 3 2024 /usr/sbin/pppd
35808 68 -rwsr-xr-x 1 root root 69088 May 30 2024 /usr/bin/newgrp
24926 68 -rwsr-xr-x 1 root root 67744 Sep 16 02:08 /usr/bin/su
Wide filesystem scans can be slow and can influence access timestamps on some mounts, so scope searches to likely change locations first.
$ printf '%s\t%s\t%s\n' '2026-01-09T16:35:21Z' 'ssh' 'Accepted password for user from 203.0.113.10' >> /root/sg-work/incident-timeline.tsv $ printf '%s\t%s\t%s\n' '2026-01-09T16:38:44Z' 'process' 'python3 /home/user/.cache/.svc/agent.py pid 2331' >> /root/sg-work/incident-timeline.tsv $ sha256sum /usr/local/bin/backup-agent d20bc21bb3c7736d8d03ade3ddb4c68b665cdfbca6f6df0f7fdd192f37f59060 /usr/local/bin/backup-agent
Keep notes and artifacts in a write-protected location and include source IPs, key fingerprints, file hashes, and exact command lines.