Agents & pools

Hosted vs self-hosted, securing agents.

Azure Pipelines works like a ride-hailing app. A queued pipeline job is a ride request. A pool is a dispatch zone, a named group of drivers on call. An agent is the driver who accepts the request, does the work, and reports back. A job's *demands* are the equivalent of asking for a car with a child seat: only drivers who advertise that feature get matched. Microsoft-hosted agents are fleet cars, scrubbed clean between every trip. Self-hosted agents are your own car. It can pull into your private garage, but the fuel, the cleaning and the repairs are on you.

What an agent actually is

Strip away the branding and an agent is a small open-source program (the Azure Pipelines agent, written in .NET) running on a machine you can point at. When you register it, it joins a pool in your organization and publishes its capabilities: operating system, installed tools, environment variables. After that it sits there and keeps asking Azure DevOps "anything for me?" over ordinary outbound HTTPS (encrypted web traffic) on port 443. Queue a run, and the service picks an idle agent in the target pool whose capabilities cover the job's demands, hands the job down, and the agent executes each step while streaming logs and artifacts back. Two details matter once you run this for real. It is a pull model: the agent calls out, the service never calls in, so an agent behind a corporate firewall needs no inbound firewall rules at all. And concurrency is sold by *parallel jobs*, not by agent count. Ten registered agents on a plan with one parallel job still run one job at a time.

Hosted vs self-hosted: the real trade-off

A Microsoft-hosted agent is a fresh virtual machine (VM) that Microsoft creates for your job and throws away when the job ends. Clean environment every run, common toolchains already installed, nothing for you to patch. You pick one with vmImage: ubuntu-latest (Ubuntu 24.04 as this is written, and the alias moves over time, so pin ubuntu-22.04 if your build cares about the version), windows-latest, or macos-latest. The limits are equally concrete. The machine spec is fixed, so no bigger box and no GPU (graphics processing unit, the chip heavy video and machine-learning builds lean on). It has no line of sight into your private network. Free-tier jobs are killed at 60 minutes. The free grant for private projects is one parallel job and 1,800 minutes a month, and that grant only switches on once your organization is linked to an Azure subscription.

A self-hosted agent is a machine you provide, whether that is a virtual machine, a container, or metal in a rack. It earns its keep in four situations: reaching private resources such as an internal NuGet package feed or a database behind a firewall, tools and licenses the hosted images do not carry, caches that survive between runs and cut a 20-minute build to five, and compliance rules that say builds must happen on hardware you control. The price is ownership. Patching, scaling and hardening are now your problem. Money points the same direction. An extra Microsoft-hosted parallel job costs $40 a month, a self-hosted one $15, because in the first case Microsoft rents you machines and in the second it only sells you the orchestration.

Point a job at a pool in YAML

In YAML (the indented text format your pipeline definition is written in), the choice comes down to one pool block per job. vmImage means Microsoft-hosted. name targets one of your own pools, and demands narrows the match to agents that advertise a given capability.

trigger: [ main ]
 
jobs:
- job: build
  pool:
    vmImage: 'ubuntu-latest'        # Microsoft-hosted: fresh VM per job
  steps:
  - script: dotnet build -c Release
    displayName: Build
 
- job: integration
  dependsOn: build
  pool:
    name: self-hosted-linux         # your own pool
    demands:
    - docker                        # capability must exist
    - Agent.OS -equals Linux        # capability must equal a value
  steps:
  - script: docker compose run --rm tests
    displayName: Integration tests
 
# Run log for the "integration" job:
#   Pool: self-hosted-linux
#   Agent: build-01
#   ##[section]Starting: Integration tests
#   ...test output...
#   ##[section]Finishing: Integration tests

A demand that does not match fails in two very different ways, and telling them apart saves you an afternoon. If *no* agent in the pool advertises what you demanded, the run errors out fast and says that no agent in the pool satisfies the specified demands. If agents that match do exist but every one of them is busy, the run does not fail at all. It sits in the queue with a message along the lines of *the agent request is not running because all potential agents are running other requests*, and that is the first thing to check when a run appears to hang. Notice too that the log header names the exact agent that ran the job. During an incident, "which machine built this artifact?" is a question you want answered in one line.

Register an agent and inspect pools

Registering a self-hosted Linux agent takes about five minutes: create the pool under Project settings, Agent pools, download the agent, run config.sh, install it as a service. Registration authenticates with a personal access token (PAT, a long-lived string that stands in for your password) scoped to Agent Pools (read, manage) and nothing wider. On current agent versions you can do better and use a Microsoft Entra service principal instead, because a PAT sitting in shell history is a classic leak. Either way the credential is only used to register. The agent is then handed its own listener credentials, so that PAT can expire the next day and nothing breaks.

# 1. download — copy the current URL from Agent pools -> New agent
mkdir -p ~/azagent && cd ~/azagent
curl -fsSL https://download.agent.dev.azure.com/agent/4.255.0/vsts-agent-linux-x64-4.255.0.tar.gz | tar xz
 
# 2. register into your pool (PAT scope: Agent Pools - read, manage)
./config.sh --unattended \
  --url https://dev.azure.com/contoso \
  --auth pat --token "$AZP_TOKEN" \
  --pool self-hosted-linux \
  --agent build-01 --replace
#   >> Register Agent:
#   Scanning for tool capabilities.
#   Connecting to the server.
#   Successfully added the agent
#   Testing agent connection.
#   2026-07-14 09:12:44Z: Settings Saved.
 
# 3. run as a systemd service so it survives reboots
#    (no username argument = the service runs as the user who invoked sudo)
sudo ./svc.sh install && sudo ./svc.sh start
 
# 4. inspect pools and agents from any machine with the CLI
az extension add --name azure-devops
az devops configure --defaults organization=https://dev.azure.com/contoso project=payments
az pipelines pool list --query "[].{id:id, name:name, hosted:isHosted}" -o table
#   Id    Name               Hosted
#   ----  -----------------  --------
#   9     Azure Pipelines    True
#   12    self-hosted-linux  False
az pipelines agent list --pool-id 12 \
  --query "[].{id:id, name:name, status:status, version:version}" -o table
#   Id    Name      Status    Version
#   ----  --------  --------  ---------
#   24    build-01  online    4.255.0

The agent updates itself whenever a job needs a newer version, and you can force the same thing with *Update all agents* on the pool's page. When a demand never matches and you cannot see why, run az pipelines agent list --pool-id 12 --include-capabilities true from the Azure command-line interface (CLI). It prints exactly what that box advertises, which usually settles the argument in ten seconds.

Scale out with VM Scale Set agents

One static agent is a pet with a name you know. Production wants cattle. An elastic pool (scale set agents) hands Azure DevOps a VM Scale Set, an Azure resource that runs a herd of identical virtual machines, and lets the service drive it. Azure DevOps keeps a few warm instances idle, grows toward a ceiling under load, and tears idle machines down after a time-to-live window (TTL, the number of minutes an unused machine is allowed to linger). Best of all for security, it can recycle the machine after every job, so each build starts on hardware no previous build has touched. Two scale set settings are non-negotiable, because Azure DevOps insists on doing the scaling and reimaging itself: overprovisioning off, and a manual upgrade policy. In Bicep that reads overprovision: false and upgradePolicy: { mode: 'Manual' } on the Microsoft.Compute/virtualMachineScaleSets resource (the full template pattern comes in the ARM & Bicep lesson).

az vmss create \
  --name vmss-agents-linux \
  --resource-group rg-build-agents \
  --image Ubuntu2204 \
  --vm-sku Standard_D2s_v5 \
  --instance-count 2 \
  --authentication-type SSH --generate-ssh-keys \
  --disable-overprovision \
  --upgrade-policy-mode manual \
  --single-placement-group false \
  --platform-fault-domain-count 1 \
  --load-balancer "" \
  --orchestration-mode Uniform
# (on Windows, az requires the empty value quoted as: --load-balancer '""')
# {
#   "vmss": {
#     "orchestrationMode": "Uniform",
#     "overprovision": false,
#     "provisioningState": "Succeeded",
#     "upgradePolicy": { "mode": "Manual" },
#     ...
# }

Then wire the pool up on the Azure DevOps side. In Terraform, using the microsoft/azuredevops provider, that is one resource pointing at the scale set through an Azure service connection.

resource "azuredevops_elastic_pool" "linux_agents" {
  name                   = "vmss-linux-elastic"
  service_endpoint_id    = azuredevops_serviceendpoint_azurerm.ci.id
  service_endpoint_scope = azuredevops_project.payments.id
  azure_resource_id      = "/subscriptions/<sub-id>/resourceGroups/rg-build-agents/providers/Microsoft.Compute/virtualMachineScaleSets/vmss-agents-linux"
  desired_idle           = 2
  max_capacity           = 10
  time_to_live_minutes   = 30
  recycle_after_each_use = true   # fresh VM per job (defaults to false)
}
 
# terraform apply
#   azuredevops_elastic_pool.linux_agents: Creating...
#   azuredevops_elastic_pool.linux_agents: Creation complete after 9s [id=15]
#   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

recycle_after_each_use buys you a guarantee that no build ever sees another build's leftovers, and it costs a few minutes of provisioning time per job. For anything that touches secrets, that is the right default. If you would rather not run the scale set yourself at all, Managed DevOps Pools is Azure's newer first-party service. Same idea, packaged as a managed resource, with the agent machines living in a Microsoft-managed subscription rather than in yours.

Securing self-hosted agents

The threat model is blunt. An agent runs whatever code your repositories tell it to, using that machine's identity and its network position. So split pools by trust level. Give pull request validation (PR, the review step a change passes through before it merges) a pool with no secrets and no route to production, and keep a separate deployment pool that only privileged pipelines are allowed to target, enforced with pool permissions. Run the agent process as an unprivileged user, never as root or Administrator. In organization settings, leave Limit job authorization scope to current project switched on so a job's access token cannot wander into other projects. And never let a pull request from a fork run on a pool that holds credentials. Hosted agents exist for exactly that kind of untrusted code.

An agent decides *where* your code runs. It has no opinion at all about *what* code shows up. That gate sits one layer up, in the repository, in the branch policies that decide what is allowed to reach main and therefore what is allowed to reach a pool that holds your deployment credentials.

Capabilities come in two flavours, and knowing which is which ends a lot of head-scratching. System capabilities are discovered automatically at registration and on every restart: Agent.OS, the machine's architecture, the version of each tool the agent finds on the path, and its environment variables. User capabilities are the ones you type in yourself on the agent's page in the pool settings, and they exist for facts the agent cannot detect on its own, such as a GPU you want builds to ask for by name, a card-data scope, or a license that lives on one specific box. A demand is checked against both sets, so writing demands: docker in a job is a bet that somebody tagged the right machines.

One more thing about self-hosted boxes that people forget: the agent keeps a working folder called _work, and it is deliberately sticky. Source trees, package caches and anything a step wrote to disk survive between runs. That stickiness is exactly why self-hosted agents are fast, and exactly why a credential file left behind by yesterday's job is dangerous. Encrypt the disk, keep the service account boring, ban interactive logins on build servers, and treat a compromised agent as a compromised developer laptop that already has a network path into staging. Next comes the other half of this picture: Azure Repos, and the branch policies that decide which code is allowed near these machines in the first place.

Try this

List the agent pools in your organization and look at what a Microsoft-hosted pool advertises. If you have a self-hosted agent of your own, confirm it reports as online, and confirm that only the jobs demanding its capabilities actually land on it.

az pipelines pool list -o table
az pipelines agent list --pool-id 1 -o table
# On a self-hosted machine (after install):
./config.sh --unattended --url https://dev.azure.com/<org> --auth pat --token $PAT --pool Default --agent $(hostname)

$ az pipelines pool list -o table
ID  Name                       IsHosted
--  -------------------------  --------
1   Azure Pipelines            True
2   Default                    False
 
# Sample output
$ az pipelines agent list --pool-id 2 -o table
Name        Status  Enabled
----------  ------  -------
build-01    online  True

Takeaway

Remember: hosted agents are disposable and live on the public internet by design, while self-hosted agents reach the things hosted agents cannot and join your attack surface in return. Patch them, split them into pools by trust level, and never let a long-lived secret settle on their disk.

Next: give your production deploy agents a pool of their own with tight role-based access control (RBAC, the rules that say who may use what), so an ordinary project contributor cannot queue an arbitrary script onto a machine that can reach production.

Related