How to troubleshoot missing syslog messages

When an expected syslog record is absent, the failing layer may be the sender, local intake, a rule that stops or reroutes the message, an output file that cannot be written, or a remote receiver that never accepted it. A controlled logger message gives the investigation a marker that can be followed through each layer without waiting for the original application to emit the event again.

On a typical Linux host, logger writes to the local syslog socket, rsyslog parses the message, rulesets apply filters and actions, and destinations write local files or forward to collectors. Distribution defaults differ, so /var/log/syslog, /var/log/messages, and host-specific remote paths should be treated as configured destinations rather than assumptions.

Start on the sender and move outward only after the marker fails the current boundary. Keep the marker unique, validate the full rsyslog configuration before restarts, and use receiver-side evidence for forwarded logs because a successful sender command only proves that the local process accepted the message for delivery.

Steps to troubleshoot missing syslog messages:

Record the message path being tested.
```
Test tag: missing-demo
Facility/priority: local0.info
Marker: SG_MISSING_20260605
Expected local file: /var/log/syslog-troubleshoot-proof.log
Expected remote receiver: logs.example.net:514/tcp
```
Use the real tag, facility, priority, local file, or collector path from the failing application when those values are known. Keep the marker unique enough that an old log entry cannot satisfy the check.
Send the controlled test message from the sender host.
```
$ logger --priority local0.info --tag missing-demo "SG_MISSING_20260605 path check"
```
Run the test from the same host and user context when the original failure depends on a local socket, container, chroot, or application account.
Check the expected local destination.
```
$ sudo grep SG_MISSING_20260605 /var/log/syslog-troubleshoot-proof.log
```
No output means the marker did not reach that file. Continue with the local rsyslog checks before changing the application.
Confirm that rsyslog is running on the sender.
```
$ sudo systemctl is-active rsyslog
active
```
If the service is inactive or failed, inspect the service journal before changing routing rules. Related: How to manage the syslog service
Validate the full rsyslog configuration.
```
$ sudo rsyslogd -N1
rsyslogd: version 8.2512.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
```
Run the check against the master configuration so included files under /etc/rsyslog.d/ are parsed in service order. Related: How to test rsyslog configuration syntax
Search for rules that match the test tag or marker.
```
$ sudo grep --line-number missing-demo /etc/rsyslog.conf /etc/rsyslog.d/*.conf
/etc/rsyslog.conf:4:if ($programname == "missing-demo" and $msg contains "SG_MISSING_20260605") then {
```
Open the matching file and check whether a stop action, selector, ruleset binding, or earlier file/forwarding action is handling the message before the expected destination.

Related: How to inspect the effective rsyslog configuration
Fix the first rule that stops or routes the marker incorrectly.
```
if ($programname == "missing-demo" and $msg contains "SG_MISSING_20260605") then {
    action(type="omfile"
           file="/var/log/syslog-troubleshoot-proof.log")
    stop
}
```
Move the needed file or forwarding action before stop, correct the selector, or bind the input to the ruleset that contains the destination. Keep the production rule narrow so unrelated messages keep their normal route.

Revalidate rsyslog after the routing change.

$ sudo rsyslogd -N1
rsyslogd: version 8.2512.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

Restart rsyslog to load the corrected rule.
```
$ sudo systemctl restart rsyslog
```
Use a maintenance window on busy collectors because a restart briefly interrupts local intake and forwarding. Related: How to manage the syslog service

Send the controlled test message again.

$ logger --priority local0.info --tag missing-demo "SG_MISSING_20260605 path check"

Confirm that the local destination now contains the marker.
```
$ sudo grep SG_MISSING_20260605 /var/log/syslog-troubleshoot-proof.log
2026-06-05T01:49:30.724631+00:00 client01.example.net missing-demo: SG_MISSING_20260605 path check
```
If the local file is still empty, check file ownership, directory permissions, disk space, and mandatory access controls before changing the sender again. Related: How to fix rsyslog output file permission errors
Check the receiver listener when the missing message should be forwarded.
```
$ sudo ss -lntu 'sport = :514'
Netid State  Recv-Q Send-Q Local Address:Port Peer Address:Port
udp   UNCONN 0      0            0.0.0.0:514       0.0.0.0:*
tcp   LISTEN 0      4096         0.0.0.0:514       0.0.0.0:*
```
No listener on the receiver means the sender cannot prove end-to-end delivery. Start or fix the receiving input before retesting forwarding. Related: How to receive remote syslog messages with rsyslog
Confirm the same marker on the receiver-side destination.
```
$ sudo grep SG_MISSING_20260605 /var/log/remote/client01.example.net.log
2026-06-05T01:49:31.112704+00:00 client01.example.net missing-demo: SG_MISSING_20260605 path check
```
If the sender local file succeeds but the receiver file is empty, inspect the forwarding action, queue state, firewall path, protocol, TLS settings, and receiver storage rule in that order. Related: How to forward syslog messages with rsyslog
Related: How to configure an rsyslog forwarding queue
Related: How to forward syslog messages over TLS with rsyslog

Author: Mohd Shakir Zakaria
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.