Missing monitoring data can originate on either side of a Checkmk agent connection. Diagnose the failure from the monitoring site and Linux host by checking retrieval, local output, ports, and registration.

In pull mode, the Checkmk site connects to the agent on TCP port 6556. Registration, push-mode transfers, and certificate updates use the Agent Receiver, normally TCP port 8000 for the first site after discovery through the site REST API, so a port 6556 firewall fix does not repair every registration or push-mode failure.

The installed Linux agent must include the Agent Controller running under systemd. A local data dump shows that the agent collects data, while only a successful query from the monitoring site shows that the connection works.

Related: How to install the Checkmk agent on Linux
Related: How to register a Checkmk agent host

Steps to troubleshoot a Checkmk agent connection on Linux:

  1. Reproduce the failing query from the Checkmk site user. 
    OMD[monitoring]:~$ cmk -d web01.example.net
ERROR [agent]: Communication failed: timed out

    The OMD[monitoring] prompt means the monitoring site user on the Checkmk server; $ sudo commands below run on the monitored Linux host unless stated otherwise. Substitute your configured host name, server, site and IP address. The host name must match its entry in Setup. timed out points toward routing, a firewall, or the configured address; [Errno 111] Connection refused means the address answered but nothing accepted the connection; Agent controller not registered points toward the Agent Controller trust relationship.

  2. Test TCP port 6556 from the Checkmk site against the IP address configured for the host. 
    OMD[monitoring]:~$ nc -vz -w 5 192.0.2.21 6556
Connection to 192.0.2.21 6556 port [tcp/*] succeeded!

    cmk -d connects to the address saved in Setup, so that is the address to test. A successful connection shows only that the port is reachable, not that registration or data retrieval works.

  3. Check the Agent Controller trust relationship on the monitored host. 
    $ sudo cmk-agent-ctl status
Version: 2.5.0p18
Agent socket: operational
IP allowlist: any
Legacy mode: enabled
No connections

    No connections, or Agent controller not registered in Checkmk, calls for registration with the exact host name from Setup. For a listed connection, the Hostname under Remote must match that object; a different name here produces Agent controller not registered on the site even though the host looks registered. Registering under another name can replace the existing trust relationship and leave the original host without agent data.
    Related: How to register a Checkmk agent host

    With legacy mode enabled, port 6556 can expose unencrypted monitoring data. In that state, access should be restricted to the monitoring server.

  4. Dump the agent data through the Agent Controller. 
    $ sudo cmk-agent-ctl dump
<<<check_mk>>>
Version: 2.5.0p18
AgentOS: linux
Hostname: web01
##### snipped #####

    cmk-agent-ctl dump checks the output as the Agent Controller receives it. A direct check_mk_agent run can have a different shell environment and can hide socket or controller problems.

  5. Restart the local agent socket when the dump reports a refused local connection. 
    $ sudo systemctl restart check-mk-agent.socket

    Use this remediation only when cmk-agent-ctl dump reports Connection refused or the agent socket is missing after an update. Restarting the socket does not repair a wrong Checkmk host name, blocked network path, or failed TLS registration.

  6. Verify that the Agent Controller is listening on TCP port 6556 for pull-mode hosts. 
    $ sudo ss -ltnp 'sport = :6556'
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess                                 
LISTEN 0      4096               *:6556            *:*    users:(("cmk-agent-ctl",pid=1288,fd=9))

    For push-mode hosts after registration, no port 6556 listener is expected. In push mode, the agent sends output to the Checkmk site through the Agent Receiver instead of waiting for the site to pull it.

  7. Check the Agent Receiver port on the Checkmk site when registration fails before the certificate prompt. 
    OMD[monitoring]:~$ omd config show
ADMIN_MAIL:
AGENT_RECEIVER: on
AGENT_RECEIVER_PORT: 8000
APACHE_FORWARDED_HEADERS: auto
APACHE_MODE: own
##### snipped #####

    The Agent Controller discovers this port through the Checkmk REST API on HTTP or HTTPS first. If discovery fails but the configured receiver port is reachable, specify that port in --server monitoring.example.net:8000 during registration.

  8. Rerun registration with verbose output when the receiver discovery or proxy path is still unclear.

    Registering under another host name can replace the existing trust relationship. 

    $ sudo cmk-agent-ctl -vv register \
    --hostname web01.example.net \
    --server monitoring.example.net:8000 \
    --site monitoring \
    --user agent_registration

    Confirm the certificate only when the site identity matches the expected Checkmk server. Use --detect-proxy only when the monitored host must reach the Checkmk server through its system proxy settings.

  9. Retest the original Checkmk query from the site user. 
    OMD[monitoring]:~$ cmk -d web01.example.net
<<<check_mk>>>
Version: 2.5.0p18
AgentOS: linux
Hostname: web01
##### snipped #####

    When this output appears, Checkmk is retrieving agent data through the configured host object. If the host was newly registered or services changed, run service discovery and activate the pending changes.
    Related: How to run Checkmk service discovery
    Related: How to activate Checkmk pending changes

    Failed to create site config directory during activation is a site fault, not an agent connection fault. The Failed to clone files line in the site user's ~/var/log/web.log names the file, and ls -l on that file shows its owner. A site file owned by another user needs sudo chown monitoring:monitoring on that file only, never a recursive change, before activation runs again.

Share this guide