How to link Tempo traces to Loki logs

Linking Tempo traces to Loki logs in Grafana adds a navigation path from a span to log lines from the same request or service. It is useful when a trace shows where latency or an error happened and the matching logs contain the application detail.

Trace-to-log linking is configured on the Tempo data source, while log-to-trace linking is configured on the Loki data source. Both sides need to agree on the data source UIDs and on the fields that connect records, such as service.name and a trace ID in the log body.

Do not use trace IDs as Loki labels. Use low-cardinality labels such as service, namespace, or cluster for stream selection, then match trace IDs from structured metadata or message content.

Related: How to correlate LGTM telemetry in Grafana
Related: How to add LGTM data sources in Grafana
Related: How to query logs with LogQL in Loki

  1. Confirm the Tempo and Loki data source UIDs. 
    $ curl --silent --user admin:admin http://127.0.0.1:3000/api/datasources
[
  {"uid":"tempo","name":"Tempo","type":"tempo"},
  {"uid":"loki","name":"Loki","type":"loki"}
]
  2. Open Connections → Data sources in Grafana.
  3. Select the Tempo data source.
  4. Open or scroll to the Trace to logs section.
  5. Select the Loki data source in the Data source field.
  6. Set Span start time shift to -2s and Span end time shift to 2s.

    Widen the window when log writes happen noticeably before or after span timestamps.

  7. Add a tag mapping from service.name to service_name.

    The Loki label must match the actual stream label. A mismatch returns no logs without proving that the trace has no matching log lines.

  8. Enable trace ID filtering when log lines contain the trace ID.
  9. Save and test the Tempo data source.
  10. Select the Loki data source.
  11. Add a derived field that extracts trace IDs from log lines. 
    name: TraceID
matcherRegex: trace_id=([0-9a-fA-F]{32})
datasourceUid: tempo
url: ${__value.raw}
  12. Save and test the Loki data source.
  13. Open a trace in Grafana Explore and click Logs for this span.
  14. Confirm that Grafana opens a Loki query for the matching service and time window. 
    {service_name="checkout-api"} |= "5b8efff798038103d269b633813fc700"
  15. Open a matching Loki log line and confirm the TraceID link opens the Tempo trace.

    If one direction works but the other does not, review the data source UID and regular expression on the failing side.