DevSecOps in the pipeline

Shift-left scanning; secure the pipeline.

A car factory does not wait for a finished car to roll off the line before checking whether the brakes went in. Every station inspects its own work, and a fault stops the line right where it happened, when the cost is a few minutes instead of a national recall. DevSecOps (development, security and operations run as one process) does the same thing to software delivery. Security stops being a review somebody books at the end and becomes a set of automated checks living inside the pipeline, so every change has to walk through them. A review gets skipped when a deadline bites. A failing build step does not.

The strategy has a name: shift-left. Draw your delivery timeline left to right, from writing code to running it in production, then push the security checks as far left as they will go, because that is where problems are cheap. A vulnerable package caught on a pull request costs you a five-minute version bump. The same package found in production costs an incident bridge, a hotfix, and a write-up for the auditors. In Azure Pipelines the mechanism is unglamorous. Each check is a step that exits with a non-zero code. The build goes red, the branch policy blocks the merge, and the fix lands before anything ships.

Four scanners, because flaws hide in four different places

A building inspector who only checks the wiring will happily sign off on a cracked foundation. Scanners have the same blind spots, so you run several. SAST (static application security testing) reads the source you wrote and flags dangerous patterns: a SQL query glued together out of strings, a password typed straight into a file. SCA (software composition analysis) looks at your dependencies, the open-source libraries you pulled in, and matches them against public CVE records (Common Vulnerabilities and Exposures, the industry's numbered list of known flaws). Most of a modern app is code you did not write, so that is where the volume lives. IaC scanning (infrastructure as code scanning) reads your Bicep and Terraform templates and catches a storage account left open to the internet before that storage account exists. Image scanning opens the container you are about to ship and finds known CVEs in the operating system packages baked inside it. Run all four and every class of flaw has somebody watching it.

# Runs on every PR and every main build
stages:
- stage: SecurityScan
  jobs:
  - job: scan
    pool:
      vmImage: ubuntu-latest
    steps:
    - task: MicrosoftSecurityDevOps@1     # SAST + IaC analyzers (Bandit, BinSkim,
      displayName: MSDO scan              # Checkov, Trivy…); SARIF → Defender for Cloud
      inputs:
        break: true                       # fail on high-severity findings (default: false)
    - script: trivy fs --scanners vuln --severity HIGH,CRITICAL --exit-code 1 .
      displayName: Dependency scan (SCA)
    - script: |
        trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \
          shopacr.azurecr.io/shop-api:$(Build.BuildId)
      displayName: Container image scan
 
# Run output when the image gate trips:
#   shop-api (debian 12.5)
#   Total: 1 (HIGH: 1)
#   ┌─────────┬───────────────┬──────────┬────────┬──────────────────┐
#   │ Library │ Vulnerability │ Severity │ Status │ Fixed Version    │
#   │ libssl3 │ CVE-2024-6119 │ HIGH     │ fixed  │ 3.0.14-1~deb12u2 │
#   └─────────┴───────────────┴──────────┴────────┴──────────────────┘
#   ##[error]Bash exited with code '1'.
#   Stage: SecurityScan — Failed

Look closely at how the image gate is tuned. --severity HIGH,CRITICAL together with --ignore-unfixed means the build fails only on serious findings that have a patch you can install today. That restraint is what keeps a security gate alive. A scanner that blocks merges over a minor issue with no fix available gets switched off within a month, and a switched-off scanner catches nothing at all. The MicrosoftSecurityDevOps@1 task (MSDO, short for Microsoft Security DevOps) bundles a curated set of open-source analyzers: Bandit, BinSkim, Checkov, ESLint, Template Analyzer, Terrascan and Trivy. It publishes their findings as a pipeline artifact in SARIF format (Static Analysis Results Interchange Format, one file layout that different scanners can all write). Defender for Cloud reads that file once your organization is onboarded, so the security team can look at pipeline findings themselves instead of asking you for screenshots. Watch the default on that task, though: it only reports until you set break: true. And one piece of history worth carrying around. MSDO's built-in secret scanner, CredScan, was deprecated in September 2023. Secret scanning now belongs to GitHub Advanced Security for Azure DevOps, which watches pushes for leaked credentials.

The pipeline can reach production, so treat it like production

Your pipeline deploys to production. That makes the pipeline production infrastructure, and a very attractive target, because breaking into one build hands an attacker everything that build is allowed to touch. Two controls do most of the work. The first is where secrets live: pull them from Key Vault (Azure's managed store for passwords, keys and certificates) through a service connection at run time, and never paste them into YAML or into a variable group as plain text. Be precise about what a *secret* variable in Azure DevOps actually buys you. Azure DevOps stores it encrypted, does not hand it to the agent as an ordinary environment variable unless you map it in yourself, and replaces matching text in the log output with asterisks. Masking is not encryption, and it is not access control. Anybody who can edit the pipeline can print the value in two halves, or base64 it, and the mask catches neither. A secret variable protects you from accidentally echoing a password into a build log. It does nothing about code you allowed to run in the job. The second control beats the first: remove the stored credential entirely with workload identity federation. The mechanic underneath is worth knowing. The job asks Azure DevOps for a short-lived, signed OIDC token (OpenID Connect, a standard way for one system to prove its identity to another) tied to its service connection, then presents that token to Entra ID (Microsoft's identity service, formerly Azure Active Directory). Entra checks the token's *issuer* and *subject* against a federated credential you registered in advance. On a match, Entra hands back a short-lived access token. Nothing is stored anywhere, so there is nothing to leak out of a variable group and nothing to forget to rotate.

# 1. App registration the service connection will federate to
az ad app create --display-name "sc-shop-prod-deploy" --query appId -o tsv
# 6f1b2c3d-9a4e-4f21-b7d0-3c5a8e2f9e8f
 
# 2. Trust tokens for ONE service connection — no secret stored. Copy the
#    Issuer and Subject VERBATIM from the draft service connection in Azure
#    DevOps: new connections use the Microsoft Entra issuer (the old
#    https://vstoken.dev.azure.com issuer is deprecated, retiring July 2027)
az ad app federated-credential create --id 6f1b2c3d-9a4e-4f21-b7d0-3c5a8e2f9e8f \
  --parameters '{
    "name": "azdo-shop-prod",
    "issuer": "https://login.microsoftonline.com/<tenant-id>/v2.0",
    "subject": "<entra-prefix>/sc/<organization-id>/<service-connection-id>",
    "audiences": ["api://AzureADTokenExchange"]
  }'
 
# 3. Least privilege: deploy rights on ONE resource group, never Owner on the sub
az role assignment create \
  --assignee-object-id <sp-object-id> --assignee-principal-type ServicePrincipal \
  --role "Website Contributor" \
  --scope /subscriptions/<sub-id>/resourceGroups/rg-shop-prod
 
# 4. Trigger and watch the gated pipeline from the CLI (azure-devops extension)
az pipelines run --name shop-api-ci --branch main --output table
# Run ID    Number       Status      Result    Pipeline ID    Pipeline Name    Source Branch
# --------  -----------  ----------  --------  -------------  ---------------  ---------------
# 4821      20260714.3   notStarted            42             shop-api-ci      refs/heads/main

That federated credential pins trust to exactly one service connection. With the current Entra issuer, Azure DevOps builds the subject string for you, and it embeds the organization ID and the service-connection ID. That is why you copy it out of the draft connection verbatim rather than composing it from memory. (Older connections used a readable sc://<org>/<project>/<connection-name> form, now deprecated alongside the vstoken issuer.) Either way the effect is the same: a token minted for any other pipeline, project or organization is rejected outright. Pair that with a role scoped to a single resource group, here Website Contributor, which can deploy web apps but cannot touch role assignments or Key Vault access policies, and a compromised build agent has a blast radius of one app rather than one subscription.

Check the infrastructure change before it lands

Infrastructure as code makes environments reproducible, including reproducibly wrong. One bad setting in a template does not stay in one place. It deploys identically to dev, to test and to production, at whatever scale you run. So put two gates in front of it. First, preview exactly what a deployment would change. Then scan that change for known-bad patterns. With Terraform, scan the *plan output* rather than only the .tf files, because variables and computed values resolve at plan time. A var.public_access that defaults to true is invisible in the source file and completely explicit in the plan JSON.

# Bicep: dry-run diff — see the blast radius before anything deploys
az deployment group what-if --resource-group rg-shop-prod \
  --template-file infra/main.bicep --parameters env=prod
#   ~ Microsoft.Web/sites/shop-api
#     ~ properties.httpsOnly: false => true
# Resource changes: 1 to modify.
 
# Terraform: compile the plan, then scan what will ACTUALLY be created
terraform -chdir=infra plan -out=tfplan
terraform -chdir=infra show -json tfplan > tfplan.json
checkov -f tfplan.json --compact
# Check: CKV_AZURE_59: "Ensure that Storage accounts disallow public access"
#   FAILED for resource: azurerm_storage_account.artifacts
# Passed checks: 47, Failed checks: 1, Skipped checks: 0

Treat a FAILED check exactly like a failing unit test. Fix it, or write down why it is allowed to fail. A checkov:skip comment sits next to the resource it excuses, so the exception arrives in the same pull request as the code and a reviewer gets a chance to argue with it. Turn the check off quietly in a pipeline variable instead and the exception becomes permanent, because nobody will ever lay eyes on it again.

Ship the exact artifact you scanned

The last gate answers a blunt question: is the thing about to serve customers the same thing your scanners approved? App Service deployment slots are how you make that true. A slot is a second, fully running copy of your web app sitting beside production. Deploy the exact image tag your gated build produced into a *staging* slot, check it there, then swap. App Service warms the staging instances first and only then flips the routing, so production never serves a cold process or an unscanned one. The security subtlety is configuration. By default, app settings travel with the code during a swap, which is wrong the moment your environments differ. You do not want staging's Key Vault URI riding into production, and you certainly do not want production connection strings landing in a slot where somebody is testing an experimental build. Settings marked as slot settings (Azure calls them *sticky*) stay bolted to the slot instead of following the swap.

# Create a staging slot beside production
az webapp deployment slot create -g rg-shop-prod -n shop-api --slot staging
 
# --slot-settings marks them STICKY: they stay with the slot during a swap
az webapp config appsettings set -g rg-shop-prod -n shop-api --slot staging \
  --slot-settings KEYVAULT_URI="https://kv-shop-stg.vault.azure.net/" \
                  APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=aa1b..."
 
# Verify what is sticky BEFORE trusting a swap
az webapp config appsettings list -g rg-shop-prod -n shop-api --slot staging \
  --query "[?slotSetting].name" -o tsv
# KEYVAULT_URI
# APPLICATIONINSIGHTS_CONNECTION_STRING
 
# Deploy the exact image the security stage scanned, then swap
az webapp config container set -g rg-shop-prod -n shop-api --slot staging \
  --container-image-name shopacr.azurecr.io/shop-api:4821
az webapp deployment slot swap -g rg-shop-prod -n shop-api \
  --slot staging --target-slot production
# Swap completed. Rollback = run the same swap command again.

The swap sequence works in your favour. App Service copies production's non-sticky settings onto the staging instances, restarts them, waits for them to warm up, and only then moves traffic across. So a broken production Key Vault reference falls over in staging, while no customer is looking at it, instead of falling over in front of everyone. Rolling back means running the same swap command a second time, because the previous version is still sitting warm in the other slot.

What none of these gates can see

Be honest about the ceiling. Every scanner in this lesson matches against things that are already *known*: published CVEs, catalogued misconfiguration patterns. None of them finds a business-logic hole, like a checkout endpoint that lets a customer set their own price. None of them finds a zero-day that has no CVE yet. None of them notices your app writing a customer's access token into a log line at run time. There are cost trade-offs as well. Full scans burn build minutes, so a common shape is fast incremental scans on every pull request plus one nightly full scan of every image in the registry. Trivy rebuilds its vulnerability database every six hours, which means an image that passed yesterday can fail today without anyone touching a line of code. Read that as a feature rather than a flake. It is the scan telling you that something already deployed became vulnerable overnight.

Everything above stops a bad change *before* it reaches a user. The other half of the question is whether the release is actually healthy *after* the swap, and that answer comes from monitoring data flowing back into the pipeline as automated release gates. Azure Monitor queries wired up as gate conditions are the subject of the next lesson, Observability and gates.

Shift-left works only if the checks run on their own and are allowed to fail the build. A wiki page asking people to please run the scanner loses to a Friday deadline every time. Put the scanners in the YAML template that every team extends, so a new pipeline inherits them without anybody opting in. Make the thresholds explicit and different by severity: Critical fails the build now, High goes onto a tracked burn-down with a date against it.

Then protect the switches, because a gate is only as strong as the permissions around it. Lock down who can edit a variable group or a service connection. Require a security reviewer on any pull request that touches azure-pipelines.yml, since deleting six lines of YAML is the cheapest way to disable every scanner you built. Put approvals and checks on the production environment so a deploy still needs a human even when the pipeline is green. And give self-hosted agents the same care as a production server, because that is what they are: a machine holding production credentials, in the habit of running whatever the repository tells it to run.

Try this

Add a job to a pipeline that runs a dependency scan (npm audit, a Trivy filesystem scan, or Microsoft Defender for DevOps) and fails on high severity. Pin a package version you know is vulnerable, open a pull request, and watch the build go red. Then bump the version and watch the same pipeline go green. Seeing the gate actually block a merge is what turns it from a report nobody reads into a control.

# Example Trivy filesystem scan in CI (agent must have trivy or use a container job):
trivy fs --exit-code 1 --severity HIGH,CRITICAL .
# Or:
npm audit --audit-level=high
echo "exit code: $?"

$ trivy fs --exit-code 1 --severity HIGH,CRITICAL .
HIGH: [email protected] (prototype pollution)
Exit: 1
 
# Sample output after bump:
$ trivy fs --exit-code 1 --severity HIGH,CRITICAL .
Passed

Takeaway

Security in Azure Pipelines comes down to exit codes. SAST, SCA, image scanning and IaC scanning are steps in a YAML file, and a step that exits non-zero is an argument no deadline can talk its way around. Tune the thresholds so the gate stays believable, then let it fail the build.

Coming next: protecting the pipeline itself with restricted variables, protected environments and trusted agents, so an attacker cannot win by opening a pull request that quietly deletes the scanning stage.

Related