Secrets, references & rotation

Runtime fetch by identity, overlap rotation.

A password you paste into a config file is a key you had copied at the hardware store. Once it exists, you cannot count how many copies are out there, and the only way to revoke it is to change the lock on everyone at the same time. A managed secret works like a hotel keycard instead. It is issued to one named guest, it opens one door, it carries an expiry, the front desk can reprogram it on demand, and every swipe is written to a log. A leaked literal password stays valid until you rotate it everywhere at once; a keycard, you deactivate at the desk and the person holding the copy is locked out. This lesson turns database passwords and connection strings from literals you commit into governed resources your workload asks for by identity, at runtime, and rotates without an outage.

The pointer, not the password

Three words to pin down first. A secret is a named value that Azure Key Vault stores with a lifecycle wrapped around it: an enabled flag, an optional not-before and expires timestamp, and a version identifier that Key Vault mints fresh every single time you set the value. Setting a secret never overwrites the old value. It appends, like a ledger that only ever adds a new line and keeps the old ones readable. That append-only versioning is the quiet detail the whole lesson rests on, and it is also an audit win: every value the secret ever held is still there, timestamped, so you can prove what was live on any given day.

A managed identity is a staff badge the building issues and keeps re-issuing for you. Technically it is a service principal (an app's own login account, one Azure creates and rotates for you) in Microsoft Entra ID, the identity service Microsoft used to call Azure Active Directory (Azure AD). Because Azure owns that credential and cycles it on its own, there is nothing sitting in your pocket to drop. A Key Vault reference is the third piece: a special app-setting value shaped like @Microsoft.KeyVault(SecretUri=...). Read it as a sticky note that says 'ask the front desk for box 14', not the contents of box 14. The App Service platform, not your code, reads that note at startup and swaps in the real value using the app's identity. References come in two shapes, SecretUri=<uri> and VaultName=<v>;SecretName=<s>, and either can carry a version or leave it off. Here is the one detail to tattoo on your hand: a versionless reference, the secret name with no version id after it, always resolves to the newest enabled version, so it follows rotation; a versioned one, with a 32-character version id on the end, pins to that exact value forever. For the identity, grant one role and stop. Key Vault Secrets User lets it read secret values and list what the vault holds, and nothing else: no writing, no deleting, no keys or certificates.

# Store the secret with a 90-day expiry so a near-expiry event can fire later.
az keyvault secret set --vault-name app-kv --name db-pw \
  --value 'S3cr3t-initial' \
  --expires "$(date -u -d '+90 days' +%Y-%m-%dT%H:%M:%SZ)" \
  --query "{id:id, enabled:attributes.enabled, expires:attributes.expires}"

{
  "id": "https://app-kv.vault.azure.net/secrets/db-pw/8f3c9d20e1a44b7c9e2f1a3b5c7d9e0f",
  "enabled": true,
  "expires": "2026-10-20T14:33:07+00:00"
}

# The Function App's system-assigned identity, granted read-only secret access.
pid=$(az functionapp identity assign -g rg -n payments-fn --query principalId -o tsv)
kvid=$(az keyvault show -n app-kv --query id -o tsv)
 
az role assignment create --assignee "$pid" \
  --role "Key Vault Secrets User" --scope "$kvid" -o none
 
# 'list' resolves the friendly role name; the raw create response only carries the role's GUID.
az role assignment list --assignee "$pid" --scope "$kvid" \
  --query "[].{principal:principalId, role:roleDefinitionName, scope:scope}"

[
  {
    "principal": "f8c1d4e2-3a5b-4c6d-8e9f-0a1b2c3d4e5f",
    "role": "Key Vault Secrets User",
    "scope": "/subscriptions/1111aaaa-22bb-33cc-44dd-5555eeee6666/resourceGroups/rg/providers/Microsoft.KeyVault/vaults/app-kv"
  }
]

# Wire a VERSIONLESS reference: the secret name, no version id on the end.
az functionapp config appsettings set -g rg -n payments-fn -o none --settings \
  "[email protected](SecretUri=https://app-kv.vault.azure.net/secrets/db-pw)"
 
# What actually got stored is the pointer, never the secret value:
az functionapp config appsettings list -g rg -n payments-fn \
  --query "[?name=='DbPassword'].value" -o tsv

@Microsoft.KeyVault(SecretUri=https://app-kv.vault.azure.net/secrets/db-pw)

The az functionapp identity assign call hands back a principalId, a globally unique identifier (GUID). That GUID, not the app's name, is what the role assignment binds to. Scope it to the vault's resource id when several apps share one vault, or narrow --scope to .../vaults/app-kv/secrets/db-pw when one app should read exactly one secret and never see the rest. Now the payoff for an attacker who lands read access to your configuration: appsettings list hands them the reference string, not the password. The pointer is worthless without the app's managed identity and the Key Vault Secrets User role behind it, and neither of those is something they can copy out of a config dump. Because the reference carries no version, the platform always resolves whatever is current, which is the mechanism that lets a later rotation reach the app without you touching config at all.

Runtime fetch: the identity reads, your code doesn't

At startup the platform reads the setting, spots the @Microsoft.KeyVault(...) marker, calls the vault as the app's identity, and substitutes the real value in memory. Your code reads an ordinary environment variable and never knows a vault was involved. The platform does not touch the vault on every request. It caches the resolved value and re-reads it roughly every 24 hours, and any change to app settings restarts the app and forces an immediate re-read. When you cannot wait out that cache, a refresh endpoint resolves every reference on demand. That call is also how you verify a change took: after it returns, the portal's Application settings blade shows each reference with a source status of Resolved instead of an error, so you know the identity, the role, and the secret URI all line up before real traffic depends on them.

# Resolve every reference now instead of waiting out the 24-hour cache.
az rest --method post \
  --uri "$(az functionapp show -g rg -n payments-fn --query id -o tsv)/config/configreferences/appsettings/refresh?api-version=2022-03-01"

# HTTP 204, no body. The portal's Application settings now shows source status: Resolved.

App Service and Azure Functions give you references for free. A pod in Azure Kubernetes Service (AKS) or a plain virtual machine (VM) has no such mechanism, so the workload fetches the secret itself with the Azure software development kit (SDK) and DefaultAzureCredential, which quietly picks up the same managed identity the platform would have used. The rule that keeps this cheap and fast: read once, cache in memory, refresh on a timer. Never call the vault on every request.

# AKS pod or VM: no reference mechanism exists here, so the workload fetches
# by identity with the SDK and caches the value in-process.
import time
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
 
_cred = DefaultAzureCredential()                 # picks up the workload's managed identity
_kv = SecretClient("https://app-kv.vault.azure.net", _cred)
 
_cache = {"value": None, "fetched_at": 0.0}
_TTL = 3600                                       # refresh at most once an hour
 
def db_password():
    if time.time() - _cache["fetched_at"] > _TTL:
        _cache["value"] = _kv.get_secret("db-pw").value   # LATEST enabled version
        _cache["fetched_at"] = time.time()
    return _cache["value"]

A vault throttles secret reads at 4,000 GET transactions every 10 seconds, per vault, per region, and every one of those calls is a billed transaction. A hot code path that reads on each request will both run up a bill and start collecting HTTP 429 (Too Many Requests) responses the moment traffic climbs. Spreading load across several vaults in one subscription helps less than you would hope, because the subscription-wide ceiling is only five times a single vault's limit. The in-process cache above, with an hour of time to live (TTL), collapses thousands of vault calls into one and still lets a rotation reach the app within that hour, because get_secret with no version pulls the latest enabled value.

Rotation is a relay, not a switch

Running az keyvault secret set on a name that already exists does not overwrite the value. It appends a new enabled version and makes it the current one, while every earlier version stays enabled and readable. That append-only behavior is what makes safe rotation possible. But the version in Key Vault is only half the job. Writing a new password into the vault is like updating the address book: the door's actual lock has not changed. Until your rotation logic also sets that password on SQL, regenerates the storage account key, or rolls the API (application programming interface) key at the provider, the shiny new value in the vault opens nothing.

# Rotate: append a NEW enabled version. The previous version stays enabled.
az keyvault secret set --vault-name app-kv --name db-pw \
  --value 'S3cr3t-rotated' \
  --expires "$(date -u -d '+90 days' +%Y-%m-%dT%H:%M:%SZ)" \
  --query id -o tsv
 
# Both versions are live during the overlap window:
az keyvault secret list-versions --vault-name app-kv --name db-pw \
  --query "[].{version:id, enabled:attributes.enabled}" -o table

https://app-kv.vault.azure.net/secrets/db-pw/2b90af51c7d34e8a9f0b1c2d3e4f5a6b
 
Version                                                                         Enabled
------------------------------------------------------------------------------  -------
https://app-kv.vault.azure.net/secrets/db-pw/2b90af51c7d34e8a9f0b1c2d3e4f5a6b  True
https://app-kv.vault.azure.net/secrets/db-pw/8f3c9d20e1a44b7c9e2f1a3b5c7d9e0f  True

Overlap is the discipline that keeps a rotation from becoming an outage. Consumers do not switch the instant you write the new version. An App Service reference can lag up to 24 hours. A database connection pool (the set of already-open, already-authenticated connections your app reuses instead of logging in every time) keeps handing out its cached credential until those connections recycle. Other services carry caches of their own. So the rule is: keep both the old and new versions enabled for longer than your slowest consumer's cache, confirm every consumer is actually on the new value, and only then disable the old version. Never delete it, so a straggler you missed is a one-command re-enable, not a restore. For stores that accept a single password at a time, use the two-credential pattern instead: two secrets, or a storage account's built-in key1 and key2, so one is always live while the other rotates.

# ...only AFTER the reference cache AND the DB pool lifetime elapse, and every
# consumer is confirmed on the new value, retire (do NOT delete) the old version:
az keyvault secret set-attributes --vault-name app-kv --name db-pw \
  --version 8f3c9d20e1a44b7c9e2f1a3b5c7d9e0f \
  --enabled false --query "attributes.enabled"

false

Doing this by hand every 90 days is how rotations quietly get skipped. Key Vault emits a Microsoft.KeyVault.SecretNearExpiry event through Azure Event Grid 30 days before a secret's expires timestamp. Subscribe a Function to that event and rotation becomes scheduled and automatic: the event fires, the Function writes the new version and swaps the backing credential, and your overlap logic handles the rest. One catch worth remembering. The event only reaches you if you created the subscription on the vault beforehand, so wire it up before you rely on it, and test that the handler actually runs.

# Fire a rotation Function 30 days before expiry, automatically.
az eventgrid event-subscription create \
  --name db-pw-near-expiry \
  --source-resource-id "$(az keyvault show -n app-kv --query id -o tsv)" \
  --included-event-types Microsoft.KeyVault.SecretNearExpiry \
  --endpoint-type azurefunction \
  --endpoint "$(az functionapp show -g rg -n rotator-fn --query id -o tsv)/functions/RotateDbPw" \
  --query "{name:name, state:provisioningState}"

{
  "name": "db-pw-near-expiry",
  "state": "Succeeded"
}

You can wire one vault and one app by hand. You cannot eyeball whether every secret across forty subscriptions carries an expiry, whether a connection string is sitting directly in app settings instead of behind a reference, or whether an identity can read a hundred secrets when it touches two. That inventory is the job of Defender for Cloud's Cloud Security Posture Management (CSPM). Defender for Cloud is the service Microsoft used to call Azure Security Center. It watches continuously and flags exactly the failures this lesson prevents: secrets stored without an expiry, over-broad vault access, and identities reading far more than they use, then scores the whole estate against those rules. The next lesson turns that scoring on and reads the findings, which is where 'we rotate our secrets' stops being a claim and becomes a control you can watch.

Try this

Work through “Rotation is a relay, not a switch” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a versioned reference silently stops rotating. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related