Log aggregation with Loki and structured logging

Ship JSON logs, label them well, and query with LogQL — centralized logs without a heavyweight indexing bill.

Loki indexes labels, not text. A query names a stream by its labels, and everything after that is a scan of the lines in that stream, so the design question is not how to make Loki index more but where each piece of information should live: in a label (indexed, bounded, defines the stream), in structured metadata (attached to a line, filterable, not indexed), or in the line itself (parsed at query time). Get that split right and Loki is cheap and fast; get it wrong and it is neither, and the usual wrong answer is a label that should have been metadata.

Where each field goes

The pipeline, in Alloy

Promtail reached end of life on 2 March 2026 and receives no updates; the supported collector is Grafana Alloy, and alloy convert turns an existing Promtail configuration into an Alloy one. The pipeline below discovers pods, tails their logs through the Kubernetes API, promotes three keys to labels, moves the trace id into structured metadata, and leaves the rest of the JSON in the line.

discovery.kubernetes "pods" {
  role = "pod"
}
 
discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
}
 
loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.app.receiver]
}
 
loki.process "app" {
  stage.json {
    expressions = { level = "level", trace_id = "trace_id" }
  }
  stage.labels {
    values = { level = "" }                 // bounded: error, warn, info, debug
  }
  stage.structured_metadata {
    values = { trace_id = "" }              // unique per request: metadata, never a label
  }
  forward_to = [loki.write.default.receiver]
}
 
loki.write "default" {
  endpoint {
    url       = "http://loki-gateway.monitoring:3100/loki/api/v1/push"
    tenant_id = "prod"
  }
}

The application's job is the line format: one JSON object per line on stdout, with level, msg, trace_id and a stable service field named the same way across services. The trace id is what joins a log line to a span in Tempo and to an exemplar on a Prometheus histogram, which is the reason it deserves a home that is filterable but not indexed.

LogQL for an incident

Label selection first, then parsing, then filtering, is the order that keeps queries fast, because each step narrows what the next one has to touch. The keep and drop stages matter for metric queries over structured metadata: Loki returns metadata as result labels, and a count_over_time across many trace ids can hit the per-query series limit unless the metadata is dropped from the grouping.

How Loki gets slow

Every distinct label combination is a stream, and every stream has its own chunks and index entries. A request_id label turns a million requests into a million streams of one line each: ingesters run out of memory, chunks never fill, and queries that should read one stream read a million. The symptom is a slow Loki that a bigger node does not fix, and the diagnosis is the label set, not the hardware. The same is true, more slowly, of pod in a cluster with aggressive autoscaling and of level if an application invents its own levels. Retention is the other budget line: retention_period per tenant with the compactor, and a hot window of a few days that most queries hit, with older chunks in object storage for the compliance questions.

Host logs arrive through the same Alloy instance reading the journal, the point where journald persistence and forwarding ends and this pipeline begins; the metric side of the same dashboards, and the alert rules that page before anyone opens Explore, are in Prometheus alerting rules.

Related posts

Quick reference