Logs, metrics, and traces describe different parts of an application failure. The LGTM stack brings them together through Grafana, with Loki storing logs, Tempo storing traces, and Mimir storing metrics; Kubernetes runs each backend as a separate Helm release.

A small internal deployment can run Loki and Tempo in monolithic mode and Mimir with its classic ingestion architecture. Single replicas reduce the initial resource requirement, but provide no high availability. Persistent volumes hold local state, while an existing S3-compatible object store holds telemetry blocks.

The starting requirements are a Kubernetes cluster running version 1.32 or later, a configured kubectl client, Helm 3 or later, and a default StorageClass that provisions writable volumes. The object store must already have the buckets and credentials used by the backends. Application collectors are configured separately after the stack is available.

Steps to deploy the LGTM stack on Kubernetes:

  1. Confirm the Kubernetes context that will receive the stack.
    $ kubectl config current-context

    The installation creates workloads and persistent volumes in the selected cluster. An empty namespace keeps this initial deployment separate from existing monitoring releases.
    Related: How to switch Kubernetes context

  2. Create the monitoring namespace.
    $ kubectl create namespace monitoring
    namespace/monitoring created
  3. Add the Grafana chart repository for Mimir.
    $ helm repo add grafana https://grafana.github.io/helm-charts
  4. Add the Grafana Community chart repository for Grafana, Loki, and Tempo.
    $ helm repo add grafana-community https://grafana-community.github.io/helm-charts
  5. Refresh the chart indexes.
    $ helm repo update
  6. Create a directory for the Helm values files.
    $ mkdir -p values
  7. Create s3-credentials.env in your text editor with the object store's credentials.
    s3-credentials.env
    S3_ACCESS_KEY=your-access-key
    S3_SECRET_KEY=your-secret-key

    The two placeholder values stand for the supplied access key and secret key. This file contains credentials and belongs outside version control and shared terminal recordings, with access limited to its owner.

  8. Import the object store credentials into a Kubernetes Secret.
    $ kubectl create secret generic lgtm-s3 --namespace monitoring \
      --from-env-file=s3-credentials.env
    secret/lgtm-s3 created

    The values below use minio.storage.svc:9000 and the existing buckets lgtm-loki, lgtm-loki-ruler, lgtm-tempo, lgtm-mimir-blocks, lgtm-mimir-ruler, and lgtm-mimir-alertmanager. Bucket names and the endpoint must match the object store.

    The backend APIs have no authentication in this single-tenant setup, and insecure: true selects HTTP for the internal object store. ClusterIP services do not prevent other cluster workloads from connecting. This setup belongs on a trusted test network; public or shared deployments require authentication, network controls, and TLS before ingestion.
    Related: How to configure tenancy and authentication for the LGTM stack
    Related: How to secure the LGTM stack with TLS and secrets

  9. Create values/loki.yaml in your text editor with the following configuration.
    values/loki.yaml
    deploymentMode: Monolithic
    loki:
      auth_enabled: false
      commonConfig:
        replication_factor: 1
      schemaConfig:
        configs:
          - from: "2024-04-01"
            store: tsdb
            object_store: s3
            schema: v13
            index:
              prefix: index_
              period: 24h
      storage:
        type: s3
        bucketNames:
          chunks: lgtm-loki
          ruler: lgtm-loki-ruler
        s3:
          endpoint: minio.storage.svc:9000
          accessKeyId: ${S3_ACCESS_KEY}
          secretAccessKey: ${S3_SECRET_KEY}
          s3ForcePathStyle: true
          insecure: true
    defaults:
      extraArgs: [-config.expand-env=true]
      extraEnvFrom:
        - secretRef:
            name: lgtm-s3
    singleBinary:
      replicas: 1
      persistence:
        size: 5Gi
      resources:
        requests: {cpu: 100m, memory: 256Mi}
        limits: {memory: 768Mi}
    read:
      replicas: 0
    write:
      replicas: 0
    backend:
      replicas: 0
    chunksCache:
      enabled: false
    resultsCache:
      enabled: false
    lokiCanary:
      enabled: false
    test:
      enabled: false

    The replication factor matches the single Loki replica. The zero replica counts disable the chart's other deployment modes. The schema date starts a storage schema period; it is not the installation date.

  10. Install the pinned Loki chart.
    $ helm install loki grafana-community/loki --version 21.0.0 \
      --namespace monitoring --values values/loki.yaml --wait --timeout 10m

    If this initial installation fails, helm uninstall loki –namespace monitoring removes the release. Other failed releases use their own release names. Persistent volumes and object-store buckets may retain data; deleting them can destroy telemetry.

  11. Create values/tempo.yaml in your text editor with the following configuration.
    values/tempo.yaml
    tempo:
      extraArgs:
        config.expand-env: "true"
      extraEnvFrom:
        - secretRef:
            name: lgtm-s3
      receivers:
        jaeger:
          protocols:
            grpc: null
            thrift_binary: null
            thrift_compact: null
            thrift_http: null
        otlp:
          protocols:
            grpc: {endpoint: "0.0.0.0:4317"}
            http: {endpoint: "0.0.0.0:4318"}
      storage:
        trace:
          backend: s3
          s3:
            bucket: lgtm-tempo
            endpoint: minio.storage.svc:9000
            access_key: ${S3_ACCESS_KEY}
            secret_key: ${S3_SECRET_KEY}
            insecure: true
          wal:
            path: /var/tempo/wal
      resources:
        requests: {cpu: 100m, memory: 256Mi}
        limits: {memory: 768Mi}
    persistence:
      enabled: true
      size: 5Gi

    Tempo accepts OTLP over gRPC on port 4317 and HTTP on port 4318 inside the cluster. Its monolithic deployment does not require Kafka.

  12. Install the pinned Tempo chart.
    $ helm install tempo grafana-community/tempo --version 3.2.1 \
      --namespace monitoring --values values/tempo.yaml --wait --timeout 10m
  13. Create values/mimir.yaml in your text editor with the following configuration.
    values/mimir.yaml
    global:
      extraEnvFrom:
        - secretRef:
            name: lgtm-s3
    minio:
      enabled: false
    kafka:
      enabled: false
    mimir:
      structuredConfig:
        multitenancy_enabled: false
        ingest_storage:
          enabled: false
          kafka:
            address: null
            topic: null
            auto_create_topic_default_partitions: null
        distributor:
          remote_timeout: null
        ingester:
          push_grpc_method_enabled: true
          ring:
            replication_factor: 1
        common:
          storage:
            backend: s3
            s3:
              endpoint: minio.storage.svc:9000
              access_key_id: ${S3_ACCESS_KEY}
              secret_access_key: ${S3_SECRET_KEY}
              insecure: true
        blocks_storage:
          s3:
            bucket_name: lgtm-mimir-blocks
        ruler_storage:
          s3:
            bucket_name: lgtm-mimir-ruler
        alertmanager_storage:
          s3:
            bucket_name: lgtm-mimir-alertmanager
    alertmanager:
      enabled: false
    ruler:
      enabled: false
    overrides_exporter:
      enabled: false
    rollout_operator:
      enabled: false
    chunks-cache:
      enabled: false
    index-cache:
      enabled: false
    metadata-cache:
      enabled: false
    results-cache:
      enabled: false
    distributor:
      replicas: 1
      extraArgs:
        config.expand-env: "true"
      resources:
        requests: {cpu: 50m, memory: 128Mi}
        limits: {memory: 512Mi}
    ingester:
      replicas: 1
      extraArgs:
        config.expand-env: "true"
      resources:
        requests: {cpu: 50m, memory: 128Mi}
        limits: {memory: 768Mi}
      zoneAwareReplication:
        enabled: false
      persistentVolume:
        size: 5Gi
    querier:
      replicas: 1
      extraArgs:
        config.expand-env: "true"
      resources:
        requests: {cpu: 50m, memory: 128Mi}
        limits: {memory: 512Mi}
    query_frontend:
      replicas: 1
      extraArgs:
        config.expand-env: "true"
      resources:
        requests: {cpu: 50m, memory: 128Mi}
        limits: {memory: 512Mi}
    query_scheduler:
      replicas: 1
      extraArgs:
        config.expand-env: "true"
      resources:
        requests: {cpu: 50m, memory: 128Mi}
        limits: {memory: 512Mi}
    store_gateway:
      replicas: 1
      extraArgs:
        config.expand-env: "true"
      resources:
        requests: {cpu: 50m, memory: 128Mi}
        limits: {memory: 512Mi}
      zoneAwareReplication:
        enabled: false
      persistentVolume:
        size: 5Gi
    compactor:
      replicas: 1
      extraArgs:
        config.expand-env: "true"
      resources:
        requests: {cpu: 50m, memory: 128Mi}
        limits: {memory: 512Mi}
      persistentVolume:
        size: 5Gi

    Mimir's classic architecture sends writes directly to its ingester instead of Kafka. The bundled object store is disabled because all three backends use the existing S3 service. Ruler and Alertmanager are outside this initial deployment.

  14. Install the pinned Mimir chart.
    $ helm install mimir grafana/mimir-distributed --version 6.2.2 \
      --namespace monitoring --values values/mimir.yaml --wait --timeout 10m
  15. Create values/grafana.yaml in your text editor with the following configuration.
    values/grafana.yaml
    persistence:
      enabled: true
      size: 1Gi
    resources:
      requests: {cpu: 100m, memory: 128Mi}
      limits: {memory: 512Mi}
    service:
      type: ClusterIP
    datasources:
      datasources.yaml:
        apiVersion: 1
        datasources:
          - name: Mimir
            uid: mimir
            type: prometheus
            access: proxy
            url: http://mimir-gateway/prometheus
            isDefault: true
            jsonData:
              httpMethod: POST
              prometheusType: Mimir
          - name: Loki
            uid: loki
            type: loki
            access: proxy
            url: http://loki-gateway
          - name: Tempo
            uid: tempo
            type: tempo
            access: proxy
            url: http://tempo:3200

    These service names follow the release names used above and resolve within the monitoring namespace. Grafana queries Mimir through its Prometheus-compatible path, Loki through its gateway, and Tempo through its HTTP API.

  16. Install the pinned Grafana chart.
    $ helm install grafana grafana-community/grafana --version 14.4.0 \
      --namespace monitoring --values values/grafana.yaml --wait --timeout 10m
  17. Check that the monitoring pods are ready.
    $ kubectl get pods --namespace monitoring

    Every pod should show all containers ready, with Running status. A Pending pod often indicates insufficient resources or an unbound volume; a Helm timeout is not a completed installation.
    Related: How to install a Helm chart on Kubernetes

  18. Retrieve the generated Grafana administrator password.
    $ kubectl get secret grafana --namespace monitoring \
      -o jsonpath='{.data.admin-password}' | base64 --decode

    The output is a credential. It belongs in a private terminal or password manager, not in a transcript, screenshot, or values file.

  19. Forward Grafana to the local workstation.
    $ kubectl port-forward --namespace monitoring svc/grafana 3000:80
    Forwarding from 127.0.0.1:3000 -> 3000

    The command remains running while another terminal performs the connection checks. The default listener is local only; stopping it closes workstation access without stopping Grafana.

  20. Test Grafana's connection to Mimir from another terminal.
    $ curl --fail --silent --show-error --user admin \
      http://127.0.0.1:3000/api/datasources/uid/mimir/health
    Enter host password for user 'admin':
    {"details":{"application":"Mimir","features":{"rulerApiEnabled":true}},"message":"Successfully queried the Prometheus API.","status":"OK"}

    The password prompt accepts the generated Grafana password without placing it in shell history. A successful response has status set to OK; an HTTP error or an error status identifies a connection that still needs attention.

  21. Test Grafana's connection to Loki.
    $ curl --fail --silent --show-error --user admin \
      http://127.0.0.1:3000/api/datasources/uid/loki/health
    Enter host password for user 'admin':
    {"message":"Data source successfully connected.","status":"OK"}
  22. Test Grafana's connection to Tempo.
    $ curl --fail --silent --show-error --user admin \
      http://127.0.0.1:3000/api/datasources/uid/tempo/health
    Enter host password for user 'admin':
    {"message":"Data source is working","status":"OK"}

    All three OK responses confirm that Grafana can reach the deployed backend APIs. Collectors can now send logs to http://loki-gateway.monitoring.svc/loki/api/v1/push, traces to tempo.monitoring.svc:4317 or http://tempo.monitoring.svc:4318/v1/traces, and Prometheus remote writes to http://mimir-gateway.monitoring.svc/api/v1/push.

Share this guide