Buildx builders, outputs, cache and Bake

Builders and drivers, --load and --push, cache export and Bake files.

Buildx sends every build to a builder, every builder has a driver, and the driver decides where results and cache can go. That one chain explains most build surprises in CI: a cache export the builder refuses, a green build that pushed nothing, a docker buildx bake whose targets nobody can locate. This lesson runs each piece on the lab VM against a local registry:3: builders and drivers, the outputs, cache export and import, and Bake.

Use the main lab VM, secopslog-docker, with the lesson files in ~/lab/buildx: a small Go module with two programs (cmd/api and cmd/worker), one Dockerfile that builds either through ARG APP, a buildkitd.toml, and a docker-bake.hcl. Nothing on the host changes. The outputs below are BuildKit's plain progress format, what CI logs show; in an interactive terminal you get the same steps as a collapsing view, or add --progress=plain.

FROM golang:1.27-alpine@sha256:8a5910f31396cd4d89662f56c68b3ae31d374308270a1c3bd96672ee5ed43414 AS build
ARG APP=api
WORKDIR /src
RUN --mount=type=bind,target=. \
    --mount=type=cache,id=lab-shop-go-build,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/${APP}
 
FROM scratch
COPY --from=build /out/app /app
USER 65532:65532
ENTRYPOINT ["/app"]

Builders and drivers

A builder is a named BuildKit instance Buildx can send builds to. Every Docker host has one called default:

The docker driver runs BuildKit inside dockerd. Its worker uses containerd and the overlayfs snapshotter, the same store the daemon's images live in, which is why builds on the default builder can use local images as bases and land in docker image ls without a copy. The GC policy lines are the default garbage-collection rules for build cache (cache mounts and local sources are kept 48 hours within a size cap, the rest within free-space limits). The docker driver has no configuration of its own: no buildkitd.toml, no BuildKit version choice, and it builds for the host's platforms only unless emulation is installed ("Multi-platform images").

The docker-container driver runs a separate BuildKit daemon in a container, from the moby/buildkit image. Two other drivers exist for shared build infrastructure: kubernetes starts BuildKit pods in a cluster (docker buildx create --driver kubernetes --driver-opt namespace=ci,replicas=2), and remote connects to a BuildKit daemon you already run (docker buildx create --driver remote tcp://buildkitd.internal:1234, normally with TLS client certificates). Both behave like docker-container in everything below.

# Pull Docker Hub images through Google's public mirror (avoids Docker Hub's
# anonymous pull limit on shared CI runners) and talk plain HTTP to the lab registry.
[registry."docker.io"]
  mirrors = ["mirror.gcr.io"]
 
[registry."localhost:5000"]
  http = true

This configuration does two things. Builds pull Docker Hub images through mirror.gcr.io, Google's public cache of frequently pulled Docker Hub images, which takes most pulls off a shared CI egress IP and so away from Docker Hub's anonymous pull limit. Images the mirror does not hold still come from Docker Hub and count against the limit, so authenticated pulls or your own pull-through cache remain the complete answer. And the builder talks plain HTTP to the lab registry on localhost:5000; dockerd allows that for localhost by default, a separate BuildKit daemon needs it spelled out. Create the builder:

--bootstrap started it right away instead of on first use. The builder is the container buildx_buildkit_lab-builder0, and its state (cache, snapshots) lives in a volume next to it. --driver-opt network=host puts the container in the VM's network namespace, so localhost:5000 in the builder is the VM's port 5000; Buildx grants the matching network.host entitlement on the daemon flags. The config file was copied into the container at creation, so later edits to your copy need docker buildx rm and create again. The asterisk in docker buildx ls still marks default as the builder plain docker build uses.

Where the result goes

A docker-container builder has its own cache and no image store. Build without saying where the result should go:

The FROM step downloaded the whole golang image (51 seconds here) even though the daemon already had it: this builder does not share the daemon's store, so it pulls through its own registry configuration. The final warning is the point. The image exists only as cache inside the builder. You choose an output:

--load exported to a tarball ("sending tarball") and the daemon imported it. The image ID, 2cb9c78938fa, is the manifest digest shown in the export: a loaded image from this builder is a single manifest with no attestation. That differs from the default builder, where the image goes straight into the containerd image store and keeps its provenance attestation as an index ("Image anatomy: index, manifest, config and layers" shows one). Pushing keeps everything the builder produced. Start the lab registry and push:

Same manifest digest as the loaded image, plus an attestation manifest (BuildKit's default minimal provenance) and the manifest list (the index) that ties them together. The index digest is what the registry tag now points to. SBOM and provenance attestations are build outputs you turn on and tune with --sbom and --provenance; "Pinning, SBOMs, provenance and scanning" in Advanced container security covers what they contain and how to use them.

docker buildx use changes the builder that docker build and docker buildx build pick when --builder is not given. It is a per-user setting in ~/.docker/buildx, and a common cause of "my build suddenly does not show up in docker images":

Here docker build ran on lab-builder and did not warn about a missing output, because Buildx turns on --load by default when it is invoked as docker build. docker buildx build does not do that, as the earlier warning showed. Switching back with docker buildx use default keeps the rest of the course predictable. In scripts, pass --builder explicitly or set BUILDX_BUILDER instead of depending on use.

The filesystem exporters skip images entirely. Build the worker and take the binary straight out of the final stage:

type=local wrote the 1.5MB static worker binary into out/, and it runs directly on the VM. type=tar holds the same kind of filesystem (here the 5.4MB api binary) without any image metadata. type=oci is a complete image in OCI layout: index.json points at the manifest, and the blobs are the manifest, config and one layer. Tools such as Trivy, Syft and Cosign read that layout directly. Like --load, the OCI archive from this builder carries no attestation manifest.

Cache export and import

A fresh CI runner, a new builder or a pruned one starts with an empty cache. Exporting the cache to a registry and importing it on the next build fixes that. mode=max exports cache for every stage, not only the layers of the final image:

The cache lives next to the image as lab-api:buildcache, a manifest whose config type is application/vnd.buildkit.cacheconfig.v0. Its largest layer, 67,652,866 bytes, is the golang base layer; with mode=max the builder stage is in the cache, base included. Now throw the builder's local cache away and rebuild with --cache-from:

482.4MB of local cache gone, and the rebuild still reports every step CACHED after importing cache manifest, in 2.0 seconds. BuildKit downloaded only the cache metadata; because nothing had to run, it never fetched the golang layers either. Compare the two modes with the local cache backend, which writes the same format to a directory:

2.3MB for mode=min, the layers of the final scratch image only, against 74MB for mode=max. With min, a runner whose source changed still has to pull golang and recompile; with max, it reuses every unchanged step of the builder stage, at the price of a bigger upload. Cache mounts (RUN --mount=type=cache) are never exported by either mode. Other backends follow the same pattern: type=gha stores cache in the GitHub Actions cache service and only works inside a workflow, type=s3 and type=azblob use object storage, and type=inline embeds min-mode cache metadata in the pushed image.

The default builder can export cache too, as long as the daemon uses the containerd image store, the default on fresh Docker 29 installs and the lab VM's configuration:

74MB, the same size as the mode=max export from lab-builder, written by BuildKit inside dockerd. On hosts still on the classic store, any --cache-to other than inline fails on the default builder with Cache export is not supported for the docker driver. The usual fix in CI is a docker-container builder, which is what docker/setup-buildx-action creates by default and which works whatever store the runner's daemon uses.

Bake

A real repository builds several images with the same options: platforms, labels, cache settings, tags derived from the commit. Writing those as shell flags in each pipeline drifts. Bake reads them from a file, docker-bake.hcl, and builds the targets in parallel on one builder:

variable "REGISTRY" {
  description = "Registry the images and cache go to"
  default = "localhost:5000"
}
 
variable "TAG" {
  description = "Image tag, set by CI to the commit or release"
  default = "dev"
}
 
group "default" {
  targets = ["api", "worker"]
}
 
target "_common" {
  context    = "."
  dockerfile = "Dockerfile"
  labels = {
    "org.opencontainers.image.source" = "https://example.com/lab/shop"
  }
}
 
target "api" {
  description = "HTTP API image"
  inherits   = ["_common"]
  args       = { APP = "api" }
  tags       = ["${REGISTRY}/lab-api:${TAG}"]
  cache-from = ["type=registry,ref=${REGISTRY}/lab-api:buildcache"]
  cache-to   = ["type=registry,ref=${REGISTRY}/lab-api:buildcache,mode=max"]
}
 
target "worker" {
  description = "Background worker image"
  inherits   = ["_common"]
  args       = { APP = "worker" }
  tags       = ["${REGISTRY}/lab-worker:${TAG}"]
  cache-from = ["type=registry,ref=${REGISTRY}/lab-worker:buildcache"]
  cache-to   = ["type=registry,ref=${REGISTRY}/lab-worker:buildcache,mode=max"]
}

Variables have defaults and are overridden by environment variables of the same name. _common holds the shared settings; api and worker inherit it and add their build argument, tags and cache refs; the group default is what docker buildx bake builds with no target named. Ask Bake what it sees before building anything:

--print shows the fully resolved definition as JSON, with inheritance applied and ${REGISTRY}, ${TAG} filled in. It is the first thing to run when a CI build does something unexpected. _common is not listed as a target because names starting with _ are treated as internal. Build and push both images with a tag from the environment:

Both targets ran in one build session. The ERROR for the worker cache is the first build's normal state: nothing has been exported to lab-worker:buildcache yet, so the import fails, BuildKit builds without it, and the summary at the end repeats the failed import without failing the build. The api import succeeded from the cache pushed earlier. The registry now has both repositories, and lab-worker:2 is an index with its attestation, like any pushed build.

CI changes a definition without editing the file through --set target.key=value (with * for every target) and variables through the environment:

The worker now builds for two platforms with tag 3. --print shows the value as one string, linux/amd64,linux/arm64, exactly as it was set; Buildx splits the comma list into two platforms when it builds. A GitHub Actions job is usually only this much, with the bake file doing the rest. Each action is pinned to a full commit SHA with its version in a comment, for the reason "Docker in CI/CD: build to promote" gives, and the job token gets read access only:

permissions:
  contents: read
 
jobs:
  images:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
      - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
        with:
          registry: registry.example.com
          username: ${{ vars.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_TOKEN }}
      - uses: docker/bake-action@018cb6412ab401ebaa809aa5f85966b74628600f # v7.4.0
        env:
          REGISTRY: registry.example.com/shop
          TAG: ${{ github.sha }}
        with:
          push: true
          set: |
            *.cache-from=type=gha
            *.cache-to=type=gha,mode=max

setup-buildx-action creates a docker-container builder on the runner, so every output and cache backend above is available. The --set lines swap the registry cache for the GitHub cache without touching the file developers use locally. "Docker in CI/CD: build to promote" puts this build into the full pipeline with scanning, signing and promotion.

Clean up

docker buildx rm stops the builder container and deletes its state volume with the cache in it (--keep-state keeps the volume for a later builder of the same name). rm -v on the registry removes its anonymous storage volume with the pushed images. The moby/buildkit image stays in the image store for the next builder; reset-lab.sh removes any lab- builders you forget.

Try this

Work through “Clean up” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from buildx builders, outputs, cache and bake, keep “Clean up”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related