Logs, metrics, and traces describe different parts of an application failure. The LGTM stack brings them together through Grafana, with Loki storing logs, Tempo storing traces, and Mimir storing metrics; Kubernetes runs each backend as a separate Helm release.
A small internal deployment can run Loki and Tempo in monolithic mode and Mimir with its classic ingestion architecture. Single replicas reduce the initial resource requirement, but provide no high availability. Persistent volumes hold local state, while an existing S3-compatible object store holds telemetry blocks.
The starting requirements are a Kubernetes cluster running version 1.32 or later, a configured kubectl client, Helm 3 or later, and a default StorageClass that provisions writable volumes. The object store must already have the buckets and credentials used by the backends. Application collectors are configured separately after the stack is available.
Steps to deploy the LGTM stack on Kubernetes:
- Confirm the Kubernetes context that will receive the stack.
$ kubectl config current-context
The installation creates workloads and persistent volumes in the selected cluster. An empty namespace keeps this initial deployment separate from existing monitoring releases.
Related: How to switch Kubernetes context - Create the monitoring namespace.
$ kubectl create namespace monitoring namespace/monitoring created
- Add the Grafana chart repository for Mimir.
$ helm repo add grafana https://grafana.github.io/helm-charts
- Add the Grafana Community chart repository for Grafana, Loki, and Tempo.
$ helm repo add grafana-community https://grafana-community.github.io/helm-charts
- Refresh the chart indexes.
$ helm repo update
- Create a directory for the Helm values files.
$ mkdir -p values
- Create s3-credentials.env in your text editor with the object store's credentials.
- s3-credentials.env
S3_ACCESS_KEY=your-access-key S3_SECRET_KEY=your-secret-key
The two placeholder values stand for the supplied access key and secret key. This file contains credentials and belongs outside version control and shared terminal recordings, with access limited to its owner.
- Import the object store credentials into a Kubernetes Secret.
$ kubectl create secret generic lgtm-s3 --namespace monitoring \ --from-env-file=s3-credentials.env secret/lgtm-s3 created
The values below use minio.storage.svc:9000 and the existing buckets lgtm-loki, lgtm-loki-ruler, lgtm-tempo, lgtm-mimir-blocks, lgtm-mimir-ruler, and lgtm-mimir-alertmanager. Bucket names and the endpoint must match the object store.
The backend APIs have no authentication in this single-tenant setup, and insecure: true selects HTTP for the internal object store. ClusterIP services do not prevent other cluster workloads from connecting. This setup belongs on a trusted test network; public or shared deployments require authentication, network controls, and TLS before ingestion.
Related: How to configure tenancy and authentication for the LGTM stack
Related: How to secure the LGTM stack with TLS and secrets - Create values/loki.yaml in your text editor with the following configuration.
- values/loki.yaml
deploymentMode: Monolithic loki: auth_enabled: false commonConfig: replication_factor: 1 schemaConfig: configs: - from: "2024-04-01" store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage: type: s3 bucketNames: chunks: lgtm-loki ruler: lgtm-loki-ruler s3: endpoint: minio.storage.svc:9000 accessKeyId: ${S3_ACCESS_KEY} secretAccessKey: ${S3_SECRET_KEY} s3ForcePathStyle: true insecure: true defaults: extraArgs: [-config.expand-env=true] extraEnvFrom: - secretRef: name: lgtm-s3 singleBinary: replicas: 1 persistence: size: 5Gi resources: requests: {cpu: 100m, memory: 256Mi} limits: {memory: 768Mi} read: replicas: 0 write: replicas: 0 backend: replicas: 0 chunksCache: enabled: false resultsCache: enabled: false lokiCanary: enabled: false test: enabled: false
The replication factor matches the single Loki replica. The zero replica counts disable the chart's other deployment modes. The schema date starts a storage schema period; it is not the installation date.
- Install the pinned Loki chart.
$ helm install loki grafana-community/loki --version 21.0.0 \ --namespace monitoring --values values/loki.yaml --wait --timeout 10m
If this initial installation fails, helm uninstall loki –namespace monitoring removes the release. Other failed releases use their own release names. Persistent volumes and object-store buckets may retain data; deleting them can destroy telemetry.
- Create values/tempo.yaml in your text editor with the following configuration.
- values/tempo.yaml
tempo: extraArgs: config.expand-env: "true" extraEnvFrom: - secretRef: name: lgtm-s3 receivers: jaeger: protocols: grpc: null thrift_binary: null thrift_compact: null thrift_http: null otlp: protocols: grpc: {endpoint: "0.0.0.0:4317"} http: {endpoint: "0.0.0.0:4318"} storage: trace: backend: s3 s3: bucket: lgtm-tempo endpoint: minio.storage.svc:9000 access_key: ${S3_ACCESS_KEY} secret_key: ${S3_SECRET_KEY} insecure: true wal: path: /var/tempo/wal resources: requests: {cpu: 100m, memory: 256Mi} limits: {memory: 768Mi} persistence: enabled: true size: 5Gi
Tempo accepts OTLP over gRPC on port 4317 and HTTP on port 4318 inside the cluster. Its monolithic deployment does not require Kafka.
- Install the pinned Tempo chart.
$ helm install tempo grafana-community/tempo --version 3.2.1 \ --namespace monitoring --values values/tempo.yaml --wait --timeout 10m
- Create values/mimir.yaml in your text editor with the following configuration.
- values/mimir.yaml
global: extraEnvFrom: - secretRef: name: lgtm-s3 minio: enabled: false kafka: enabled: false mimir: structuredConfig: multitenancy_enabled: false ingest_storage: enabled: false kafka: address: null topic: null auto_create_topic_default_partitions: null distributor: remote_timeout: null ingester: push_grpc_method_enabled: true ring: replication_factor: 1 common: storage: backend: s3 s3: endpoint: minio.storage.svc:9000 access_key_id: ${S3_ACCESS_KEY} secret_access_key: ${S3_SECRET_KEY} insecure: true blocks_storage: s3: bucket_name: lgtm-mimir-blocks ruler_storage: s3: bucket_name: lgtm-mimir-ruler alertmanager_storage: s3: bucket_name: lgtm-mimir-alertmanager alertmanager: enabled: false ruler: enabled: false overrides_exporter: enabled: false rollout_operator: enabled: false chunks-cache: enabled: false index-cache: enabled: false metadata-cache: enabled: false results-cache: enabled: false distributor: replicas: 1 extraArgs: config.expand-env: "true" resources: requests: {cpu: 50m, memory: 128Mi} limits: {memory: 512Mi} ingester: replicas: 1 extraArgs: config.expand-env: "true" resources: requests: {cpu: 50m, memory: 128Mi} limits: {memory: 768Mi} zoneAwareReplication: enabled: false persistentVolume: size: 5Gi querier: replicas: 1 extraArgs: config.expand-env: "true" resources: requests: {cpu: 50m, memory: 128Mi} limits: {memory: 512Mi} query_frontend: replicas: 1 extraArgs: config.expand-env: "true" resources: requests: {cpu: 50m, memory: 128Mi} limits: {memory: 512Mi} query_scheduler: replicas: 1 extraArgs: config.expand-env: "true" resources: requests: {cpu: 50m, memory: 128Mi} limits: {memory: 512Mi} store_gateway: replicas: 1 extraArgs: config.expand-env: "true" resources: requests: {cpu: 50m, memory: 128Mi} limits: {memory: 512Mi} zoneAwareReplication: enabled: false persistentVolume: size: 5Gi compactor: replicas: 1 extraArgs: config.expand-env: "true" resources: requests: {cpu: 50m, memory: 128Mi} limits: {memory: 512Mi} persistentVolume: size: 5Gi
Mimir's classic architecture sends writes directly to its ingester instead of Kafka. The bundled object store is disabled because all three backends use the existing S3 service. Ruler and Alertmanager are outside this initial deployment.
- Install the pinned Mimir chart.
$ helm install mimir grafana/mimir-distributed --version 6.2.2 \ --namespace monitoring --values values/mimir.yaml --wait --timeout 10m
- Create values/grafana.yaml in your text editor with the following configuration.
- values/grafana.yaml
persistence: enabled: true size: 1Gi resources: requests: {cpu: 100m, memory: 128Mi} limits: {memory: 512Mi} service: type: ClusterIP datasources: datasources.yaml: apiVersion: 1 datasources: - name: Mimir uid: mimir type: prometheus access: proxy url: http://mimir-gateway/prometheus isDefault: true jsonData: httpMethod: POST prometheusType: Mimir - name: Loki uid: loki type: loki access: proxy url: http://loki-gateway - name: Tempo uid: tempo type: tempo access: proxy url: http://tempo:3200
These service names follow the release names used above and resolve within the monitoring namespace. Grafana queries Mimir through its Prometheus-compatible path, Loki through its gateway, and Tempo through its HTTP API.
- Install the pinned Grafana chart.
$ helm install grafana grafana-community/grafana --version 14.4.0 \ --namespace monitoring --values values/grafana.yaml --wait --timeout 10m
- Check that the monitoring pods are ready.
$ kubectl get pods --namespace monitoring
Every pod should show all containers ready, with Running status. A Pending pod often indicates insufficient resources or an unbound volume; a Helm timeout is not a completed installation.
Related: How to install a Helm chart on Kubernetes - Retrieve the generated Grafana administrator password.
$ kubectl get secret grafana --namespace monitoring \ -o jsonpath='{.data.admin-password}' | base64 --decodeThe output is a credential. It belongs in a private terminal or password manager, not in a transcript, screenshot, or values file.
- Forward Grafana to the local workstation.
$ kubectl port-forward --namespace monitoring svc/grafana 3000:80 Forwarding from 127.0.0.1:3000 -> 3000
The command remains running while another terminal performs the connection checks. The default listener is local only; stopping it closes workstation access without stopping Grafana.
- Test Grafana's connection to Mimir from another terminal.
$ curl --fail --silent --show-error --user admin \ http://127.0.0.1:3000/api/datasources/uid/mimir/health Enter host password for user 'admin': {"details":{"application":"Mimir","features":{"rulerApiEnabled":true}},"message":"Successfully queried the Prometheus API.","status":"OK"}The password prompt accepts the generated Grafana password without placing it in shell history. A successful response has status set to OK; an HTTP error or an error status identifies a connection that still needs attention.
- Test Grafana's connection to Loki.
$ curl --fail --silent --show-error --user admin \ http://127.0.0.1:3000/api/datasources/uid/loki/health Enter host password for user 'admin': {"message":"Data source successfully connected.","status":"OK"} - Test Grafana's connection to Tempo.
$ curl --fail --silent --show-error --user admin \ http://127.0.0.1:3000/api/datasources/uid/tempo/health Enter host password for user 'admin': {"message":"Data source is working","status":"OK"}All three OK responses confirm that Grafana can reach the deployed backend APIs. Collectors can now send logs to http://loki-gateway.monitoring.svc/loki/api/v1/push, traces to tempo.monitoring.svc:4317 or http://tempo.monitoring.svc:4318/v1/traces, and Prometheus remote writes to http://mimir-gateway.monitoring.svc/api/v1/push.