Log aggregation with Loki and structured logging

Ship JSON logs, label them well, and query with LogQL — centralized logs without a heavyweight indexing bill.

Grafana Loki indexes metadata — labels — not full log line text. You choose which dimensions to filter on (job, namespace, level, app) and ship JSON payloads as log content. Queries use LogQL, a Prometheus-inspired language for log streams. The result is centralized logs at a fraction of Elasticsearch indexing cost, provided you resist the urge to label every field (cardinality kills Loki like it kills Prometheus).

Single-binary Grafana stack (Loki + Mimir + Tempo) shares object storage backends — design labels once for metrics and logs so dashboards join without relabel hacks.

This note configures Promtail (or Grafana Alloy) to scrape structured app logs, sets label cardinality rules, and writes LogQL for incident response. Pair with Prometheus alerts on symptom metrics and dive deeper in Detection engineering.

Structured logging from applications

Log one JSON object per line to stdout. Include level, msg, trace_id, and business fields inside the JSON — not as Loki labels. Parsing at query time with | json keeps label cardinality flat while still letting you filter | level="error".

OpenTelemetry trace ids in logs let you jump from a metric spike to the exact trace and span in Tempo or Jaeger when the stack is wired. Standardize field names across services (service.name, level) so platform dashboards work without per-team LogQL forks.

{"time":"2026-07-24T09:15:00Z","level":"error","msg":"payment failed",
 "service":"checkout","trace_id":"abc123","user_id":"u-9981","err":"timeout"}

Promtail scrape with restrained labels

Promtail attaches labels when shipping to Loki. Keep pipeline_stages for parsing if needed, but promote only stable keys to labels. Never label user_id, trace_id, or request ids — they explode stream count.

Retention in Loki is cheap relative to full-text indexing but not free — set retention_period and compactor rules per tenant. Hot queries hit last 24–72 hours; ship older chunks to object storage if compliance requires longer hold. Ruler alerts on log patterns (|~ "failed login") complement metric alerts for security use cases.

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
    pipeline_stages:
      - json:
          expressions:
            level: level
            msg: msg
      - labels:
          level:

Where this goes next

Correlate Loki traces with Prometheus metrics using trace_id in logs and exemplars on histograms. Ship host logs from journald pipelines into the same Loki tenant. Build composite alerts in Detection engineering — metric SLO burn plus log error spike in one Grafana panel.

Dashboard variables should drive label selectors (namespace=$ns) — never hardcode prod streams in shared panels. Test LogQL in Explore with line limit before building alert rules; expensive queries at ingest scale hurt queriers.

Related posts