Investigating a suspected intrusion on Linux clarifies what access occurred, what changed, and which identities and systems are affected, enabling containment and recovery decisions based on evidence instead of assumptions.
Most intrusion evidence comes from authentication and session telemetry (sshd, sudo, PAM, systemd-journald or syslog), plus account and key material (/etc/passwd, /etc/shadow, /etc/sudoers, /home/*/.ssh) and filesystem metadata that reveals when and where changes landed.
Volatile state disappears quickly as processes exit and connections close, while logs can rotate and history files can be truncated or cleared, so prioritize high-value collection early and favor read-only inspection until evidence handling and escalation paths are clear.
Related: Enable verbose SSH logging
Steps to investigate a Linux intrusion with journalctl, last, and ss:
- Record current system time, uptime, and host identity for investigation notes.
$ date --iso-8601=seconds 2026-01-14T09:00:39+01:00 $ uptime 09:00:39 up 2 days, 18:53, 0 user, load average: 0.37, 0.27, 0.22 $ hostnamectl Static hostname: 14844ef6f7fc Icon name: computer-container Chassis: container Machine ID: dfae20ca11e742c090f9a9f100772b43 Boot ID: aa8df42ac7eb431c9412beba3d46fae4 Virtualization: docker Operating System: Ubuntu 24.04.3 LTS Kernel: Linux 6.12.54-linuxkit Architecture: arm64Include time zone and NTP status in notes to avoid a skewed timeline.
- Review recent login history for the account.
$ last -aiF user | head -n 5 wtmp begins Sun Dec 21 08:15:32 2025
Correlate source IPs with sshd logs to confirm the authentication method and session start.
- Check active sessions for the account.
$ w -h
Related: How to list logged-in users in Linux
- Review authentication logs for SSH and sudo activity.
$ sudo journalctl --since "24 hours ago" _COMM=sshd --no-pager | head -n 6 Jan 13 23:21:07 host.example.net sshd[14183]: Connection closed by ::1 port 40732 [preauth] Jan 13 23:21:07 host.example.net sshd[14185]: Connection closed by ::1 port 40744 [preauth] Jan 13 23:21:07 host.example.net sshd[14184]: Connection closed by ::1 port 40742 [preauth] Jan 13 23:21:07 host.example.net sshd[14186]: Unable to negotiate with ::1 port 40748: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth] Jan 13 23:21:07 host.example.net sshd[14187]: Unable to negotiate with ::1 port 40760: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth] Jan 13 23:22:16 host.example.net sshd[14306]: fatal: Access denied for user backupuser by PAM account configuration [preauth] $ sudo journalctl --since "24 hours ago" _COMM=sudo --no-pager | head -n 5 Jan 13 09:03:54 host.example.net sudo[8729]: user : PWD=/ ; USER=root ; COMMAND=/usr/bin/sync Jan 13 09:03:54 host.example.net sudo[8729]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Jan 13 09:03:54 host.example.net sudo[8729]: pam_unix(sudo:session): session closed for user root Jan 13 09:03:54 host.example.net sudo[8744]: user : PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /proc/sys/vm/drop_caches Jan 13 09:03:54 host.example.net sudo[8744]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Systems using log files instead of journald commonly store auth events in /var/log/auth.log or /var/log/secure.
- Review shell and sudo command history for the account.
$ sudo tail -n 15 /home/user/.bash_history #1736198400 sudo -l #1736198460 sudo systemctl status ssh #1736198520 sudo journalctl -u ssh --since yesterday #1736198580 ss -tulpn #1736198640 ps auxf
Shell history may not be written until the shell exits, and history files can be truncated or missing.
- Audit local user accounts and privilege changes.
$ sudo getent passwd user user:x:1001:1001:,,,:/home/user:/usr/bin/zsh $ sudo getent group sudo sudo:x:27:ubuntu,user $ sudo getent group wheel
Unexpected new users, added group membership (sudo/wheel), or modified sudoers entries are high-signal changes.
- Inspect SSH authorized_keys and key file changes.
$ sudo ls -al /home/user/.ssh/authorized_keys -rw------- 1 user user 90 Jan 14 09:00 /home/user/.ssh/authorized_keys $ sudo stat /home/user/.ssh/authorized_keys File: /home/user/.ssh/authorized_keys Size: 90 Blocks: 8 IO Block: 4096 regular file Device: 0,64 Inode: 382356 Links: 1 Access: (0600/-rw-------) Uid: ( 1001/ user) Gid: ( 1001/ user) Access: 2026-01-14 08:56:47.429324635 +0100 Modify: 2026-01-14 09:00:39.851476169 +0100 Change: 2026-01-14 09:00:39.852479422 +0100 Birth: 2026-01-14 08:56:47.429324635 +0100
Unknown keys and recently changed timestamps on key files merit immediate correlation with login and sudo events.
- Check scheduled tasks and service-based persistence.
$ sudo crontab -l -u user no crontab for user $ sudo systemctl list-timers --all --no-pager | head -n 8 NEXT LEFT LAST PASSED UNIT ACTIVATES Wed 2026-01-14 23:32:25 CET 14h Tue 2026-01-13 23:33:16 CET 9h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service - - - - logrotate.timer logrotate.service 2 timers listed.
Disabling or deleting suspicious jobs before capture can destroy attribution and timing evidence.
- Inspect active network connections for unexpected endpoints.
$ ss -tupn | head -n 8 Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess $ ss -tulpn | head -n 6 Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=23657,fd=16)) udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=23657,fd=14)) tcp LISTEN 0 4096 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=3846,fd=3),("systemd",pid=1,fd=64)) tcp LISTEN 0 5 0.0.0.0:8000 0.0.0.0:* users:(("python3",pid=21697,fd=3)) tcp LISTEN 0 1024 0.0.0.0:8888 0.0.0.0:* users:(("tinyproxy",pid=21562,fd=3))Outbound connections from unexpected binaries and listeners bound to public interfaces are common intrusion pivots.
- List recently started and long-running processes for suspicious command lines.
$ ps -eo pid,ppid,user,lstart,cmd --sort=lstart | tail -n 8 23419 1 root Wed Jan 14 01:48:35 2026 /usr/sbin/NetworkManager --no-daemon 23657 1 systemd+ Wed Jan 14 01:51:21 2026 /usr/lib/systemd/systemd-resolved 24844 1 root Wed Jan 14 02:04:03 2026 Xvfb :99 -screen 0 1024x768x24 26597 1 root Wed Jan 14 03:01:51 2026 /usr/libexec/upowerd 30508 1 root Wed Jan 14 09:00:10 2026 /usr/lib/systemd/systemd-hostnamed 30761 0 root Wed Jan 14 09:00:40 2026 bash -lc ps -eo pid,ppid,user,lstart,cmd --sort=lstart | tail -n 8 30769 30761 root Wed Jan 14 09:00:40 2026 ps -eo pid,ppid,user,lstart,cmd --sort=lstart 30770 30761 root Wed Jan 14 09:00:40 2026 tail -n 8
Execution from user-writable locations like /tmp, /dev/shm, and hidden directories under /home is a strong anomaly signal.
- Review kernel taint state and recent kernel messages for unexpected module activity.
$ cat /proc/sys/kernel/tainted 4096 $ sudo dmesg -T | tail -n 6 [Wed Jan 14 08:49:25 2026] loop4: detected capacity change from 0 to 262144 [Wed Jan 14 08:49:25 2026] EXT4-fs (loop4): mounted filesystem 768d7f68-caaa-4c34-a5d7-a4eb6307c14b r/w with ordered data mode. Quota mode: none. [Wed Jan 14 08:50:07 2026] loop5: detected capacity change from 0 to 262144 [Wed Jan 14 08:50:07 2026] EXT4-fs (loop5): mounted filesystem a1d70e1f-1791-48a3-b120-442d1b87a8ea r/w with ordered data mode. Quota mode: none. [Wed Jan 14 08:50:23 2026] EXT4-fs (loop5): unmounting filesystem a1d70e1f-1791-48a3-b120-442d1b87a8ea. [Wed Jan 14 08:50:23 2026] EXT4-fs (loop4): unmounting filesystem 768d7f68-caaa-4c34-a5d7-a4eb6307c14b.
A non-zero taint value can indicate proprietary modules or kernel conditions that complicate trust in runtime telemetry.
- Capture a recursive file listing of high-value directories for later diffing.
- Find recently modified files and high-privilege binaries.
$ sudo find /etc /usr/local /var/www -xdev -type f -mtime -2 -ls 2>/dev/null | head -n 6 278283 4 -rw-r----- 1 root shadow 744 Jan 14 01:47 /etc/gshadow- 27443 4 -rw-r--r-- 1 root root 249 Jan 14 01:43 /etc/hosts 382576 4 -rw-r--r-- 1 root root 1767 Jan 14 03:02 /etc/passwd 357983 4 -rw-r--r-- 1 root root 877 Jan 14 01:47 /etc/group- 29697 4 -rw-r--r-- 1 root root 77 Jan 13 02:11 /etc/security/faillock.conf 272255 4 -rw-r--r-- 1 root root 1435 Jan 13 01:01 /etc/pam.d/common-session-noninteractive $ sudo find / -xdev -type f -perm -4000 -ls 2>/dev/null | head -n 6 56087 324 -rwsr-xr-x 1 root root 330104 Aug 26 15:49 /usr/lib/openssh/ssh-keysign 270700 68 -rwsr-xr-x 1 root root 67664 Dec 2 2024 /usr/lib/polkit-1/polkit-agent-helper-1 30253 68 -rwsr-xr-- 1 root messagebus 67504 Aug 9 2024 /usr/lib/dbus-1.0/dbus-daemon-launch-helper 357639 476 -rwsr-xr-- 1 root dip 485200 Apr 3 2024 /usr/sbin/pppd 35808 68 -rwsr-xr-x 1 root root 69088 May 30 2024 /usr/bin/newgrp 24926 68 -rwsr-xr-x 1 root root 67744 Sep 16 02:08 /usr/bin/suWide filesystem scans can be slow and can influence access timestamps on some mounts, so scope searches to likely change locations first.
- Record indicators and a timestamped timeline for escalation, containment, and wider hunting.
$ printf '%s\t%s\t%s\n' '2026-01-09T16:35:21Z' 'ssh' 'Accepted password for user from 203.0.113.10' >> /root/sg-work/incident-timeline.tsv $ printf '%s\t%s\t%s\n' '2026-01-09T16:38:44Z' 'process' 'python3 /home/user/.cache/.svc/agent.py pid 2331' >> /root/sg-work/incident-timeline.tsv $ sha256sum /usr/local/bin/backup-agent d20bc21bb3c7736d8d03ade3ddb4c68b665cdfbca6f6df0f7fdd192f37f59060 /usr/local/bin/backup-agent
Keep notes and artifacts in a write-protected location and include source IPs, key fingerprints, file hashes, and exact command lines.
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
