RBAC & audit trail at scale

AWX/AAP, credentials, and who ran what.

Priya patches the fleet on a Tuesday afternoon. She runs ansible-playbook from her laptop with her personal SSH key (Secure Shell, the encrypted remote-login protocol Ansible uses to reach your servers), types the vault password at the prompt, and forty machines take a kernel update. Three weeks later somebody asks which hosts got patched, who approved it, and whether the reboot flag was set. The only record was Priya's terminal scrollback, and she closed that window weeks ago.

Count what that one run put at risk. A key that can reconfigure forty production machines lives on a laptop that goes through airports. The password to your vault-encrypted variables lives on the same laptop, typed in by hand. Nobody can reconstruct what happened, Priya included. And on the day she leaves, revoking her access means remembering every authorized_keys file her public key ever landed in.

Hotels worked out the fix a hundred years ago. Nobody walks around with a master key on their own keyring. The keys hang in a cabinet behind the front desk, you ask for the room you are entitled to, the desk opens the door without handing the key over, and your name and the time go in the book. In Ansible that front desk is a controller. AWX is the free, open-source project. Ansible Automation Platform (AAP), Red Hat's supported product, ships the same component under the name automation controller, and older documentation calls it Ansible Tower. Same object model, same REST API (application programming interface: the machine-readable HTTP front door that every example below talks to). Everything here applies to both, and every command uses the v2 API that both expose at /api/v2/.

The Five Objects You Wire Together

Five nouns carry the whole design. Getting them straight is most of the work.

A project is a pointer to a git repository holding your playbooks, roles and collections. The controller clones it and records the exact commit that executed. It only re-pulls before a run if you turn on Update Revision on Launch. Leave that off and jobs will happily run whatever was cached the last time somebody hit sync, which is a fine way to patch production with last month's playbook. An inventory is the set of hosts, either typed in or pulled from a dynamic source such as your cloud provider's API. It is a permissioned object in its own right, which is how staging and production stop being the same thing.

A credential is a secret needed to reach or escalate on those hosts, stored encrypted and, as you are about to see, never handed back. A job template is the launchable unit: this project, this playbook, this inventory, these credentials, these flags. A recipe card taped inside the kitchen cabinet, with the ingredients and the oven temperature already filled in. Nobody at your company runs Ansible any more. They launch a job template, and that is the noun your permissions attach to. A survey is a small form bolted onto a job template, turning free-text variables into typed questions with allowed answers.

A Credential You Can Use But Never Read

Secret fields on a credential are encrypted before they reach the database, with a symmetric key derived from the installation's SECRET_KEY. The interesting half is the read path, because there isn't one. Ask the API for a credential and every secret field comes back as the literal string $encrypted$. Not only for junior users. For everyone, superusers included.

# jq is a command-line formatter for JSON (JavaScript Object Notation),
# the text format the controller API speaks.
$ curl -s -H "Authorization: Bearer $AWX_TOKEN" \
    https://awx.acme.internal/api/v2/credentials/7/ \
  | jq '{name, type: .summary_fields.credential_type.name, inputs}'

{
  "name": "fleet-patch (svc-patch)",
  "type": "Machine",
  "inputs": {
    "username": "svc-patch",
    "ssh_key_data": "$encrypted$",
    "become_method": "sudo",
    "become_username": "root"
  }
}

Here is the part people get backwards. To launch a job template you need Execute on that template and nothing else. The credential is already bolted to the template, so the person pressing the button needs no rights over it at all, and no way to read it. The Use role on a credential (its own description in the API reads "Can use the credential in a job template") is for the people who build templates. It lets them attach that credential to a new template without ever seeing what is inside. On a laptop, using a key and holding a key are the same act. Here they come apart.

So how does the secret reach the playbook? Injection. Every credential type carries an injector configuration saying where its fields land at run time, and there are three destinations. env puts the value in an environment variable inside the execution environment (the container image the job runs in). file writes it to a temporary file that is deleted when the job ends, with the path handed to you as {{ tower.filename }}. extra_vars passes it in as an Ansible variable. Built-in types already know their injectors. For anything home-grown you write both halves yourself.

# ---- INPUT CONFIGURATION: the first box on the form, what the
# ---- credential asks its owner to fill in.
fields:
  - id: scanner_url
    type: string
    label: Scanner API URL
  - id: scanner_token
    type: string
    label: Scanner API token
    secret: true            # encrypted at rest, reads back as $encrypted$
required:
  - scanner_url
  - scanner_token
 
# ---- INJECTOR CONFIGURATION: a separate box, a separate document.
# ---- How those fields reach the running playbook.
file:
  template: '{{ scanner_token }}'
env:
  SCANNER_URL: '{{ scanner_url }}'
  SCANNER_TOKEN_FILE: '{{ tower.filename }}'   # path to the temp file above

Pass tokens through a file rather than an environment variable when you get the choice. The reason usually given is that /proc leaks environment variables to anyone on the box, and that one is not true: /proc/PID/environ is readable by root and by the process's own user, nobody else. The real reasons are duller and better. Environment variables are inherited by every child process the playbook spawns, they get scooped into core dumps and crash reports, and plenty of tools print their entire environment when they hit an error. A file has a path you control, a mode you control, and a lifetime that ends with the job.

- name: Install the scanner agent token
  become: true
  ansible.builtin.copy:
    src: "{{ lookup('ansible.builtin.env', 'SCANNER_TOKEN_FILE') }}"
    dest: /etc/acme-scanner/token
    owner: root
    group: root
    mode: '0600'
  no_log: true

Three things in those eight lines are worth slowing down for. The lookup runs on the controller side, inside the execution environment, not on the managed host, so it reads the injected temp file that only exists there. no_log: true keeps that task's arguments and results out of the job output and out of the event rows in the database, including the before-and-after file diff that would otherwise print your token in full whenever somebody runs the job in diff mode. And become: true buys you root on the target host through sudo (the standard Unix way of running one command as another user), which is a completely separate question from whether the controller will let you launch anything at all. People conflate those two and end up widening sudo rights on two thousand servers to fix a permissions error that lived in AWX.

The Vault credential type gets the vault password off Priya's laptop for good. It holds vault_password and an optional vault_id, and the controller feeds it to ansible-playbook when the job starts. Attach several with distinct vault IDs and one job template can open several vaults while no human being knows any of the passwords.

Job Templates and Surveys as the Front Door

The job template is where you decide how much freedom the launcher gets. Every prompt-on-launch checkbox (ask_limit_on_launch, ask_inventory_on_launch, ask_credential_on_launch, ask_variables_on_launch) hands a decision back to whoever presses the button. Turning one on is a permission change, written in a place your access review is probably not looking.

A survey is the safe middle ground. Rather than let an operator paste arbitrary extra variables, you define questions with a type (text, textarea, password, integer, float, multiplechoice, multiselect), a default, and where it fits, a closed list of choices. It is the difference between a blank order pad and a menu.

{
  "name": "Patch window options",
  "description": "",
  "spec": [
    {
      "question_name": "Reboot after patching?",
      "variable": "patch_reboot",
      "type": "multiplechoice",
      "choices": ["no", "yes"],
      "default": "no",
      "required": true
    },
    {
      "question_name": "Maximum hosts at a time",
      "variable": "patch_serial",
      "type": "integer",
      "min": 1,
      "max": 10,
      "default": 2,
      "required": true
    }
  ]
}

Survey answers arrive as extra variables, which sit at the very top of Ansible's variable precedence and beat almost everything set in the repository. That is the point, and also the risk. The controller's ALLOW_JINJA_IN_EXTRA_VARS setting defaults to template, meaning Jinja (the templating language Ansible uses to fill in {{ variables }}) is honored in extra variables only where an admin wrote it into the job template definition itself, never when it arrives from a survey answer or a launch-time variable. Jinja that a stranger controls is a path to running arbitrary Python on the controller. Leave that default alone.

# junior on-call tries to retarget the staging patch job at production
$ curl -s -X POST -H "Authorization: Bearer $JUNIOR_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"limit": "prod_web", "extra_vars": {"patch_reboot": "yes"}}' \
    https://awx.acme.internal/api/v2/job_templates/12/launch/ \
  | jq '{job, ignored_fields, limit}'

{
  "job": 4812,
  "ignored_fields": {
    "limit": "prod_web"
  },
  "limit": "staging_web"
}

Job 4812 exists and is running against staging_web, because that is what the template says. The prod_web value is recorded in ignored_fields and applied nowhere. patch_reboot went through, because it is a question on that template's survey, and survey answers are accepted (and checked against the spec) even with ask_variables_on_launch off. Good outcome for the fleet, nasty one for automation: a wrapper script that checks only the HTTP status code sees 201 Created and reports success while being completely wrong about what it did. Read ignored_fields, or read the limit that came back.

Organizations, Teams and Roles

An organization is a floor of the building. Teams are the departments on that floor. Roles are which doors your badge opens. RBAC (role-based access control, deciding who may do what to which thing) in the controller hangs off individual objects rather than handing out global job titles, so there is no "senior engineer" switch anywhere to flip.

The role names tell you what they permit. On a job template: Admin, Execute, Read. On a credential: Admin, Use, Read. On a project: Admin, Use, Update, Read. On an inventory: Admin, Update, Ad Hoc, Use, Read. For a junior on call, the complete grant is one line: Execute on the staging patch template. Not Use on the credential, not Read on the inventory, because the template already carries both and the launcher never picks them. Those Use roles matter for the people building templates, since creating one requires Use on the project, on the inventory, and on every credential you attach. None of it touches production, because production is a different set of objects.

$ curl -s -H "Authorization: Bearer $AWX_TOKEN" \
    https://awx.acme.internal/api/v2/job_templates/12/ \
  | jq '.summary_fields.object_roles'

{
  "admin_role":   { "id": 214, "name": "Admin",
                    "description": "Can manage all aspects of the job template" },
  "execute_role": { "id": 215, "name": "Execute",
                    "description": "May run the job template" },
  "read_role":    { "id": 216, "name": "Read",
                    "description": "May view settings for the job template" }
}

Grant roles to teams, never to individuals. A joiner is then one team-membership change, a leaver is the same change in reverse, and you never spend an afternoon hunting stray direct grants. Better still, drive team membership from your identity provider over SAML (Security Assertion Markup Language), LDAP (Lightweight Directory Access Protocol) or OIDC (OpenID Connect), so the username stamped on every job record is the same identity your leavers process already switches off.

# grant Execute to the on-call team (id 4), not to a person
$ curl -s -o /dev/null -w '%{http_code}\n' -X POST \
    -H "Authorization: Bearer $AWX_TOKEN" -H "Content-Type: application/json" \
    -d '{"id": 215}' \
    https://awx.acme.internal/api/v2/teams/4/roles/
 
# now prove the junior still cannot launch the PRODUCTION template (id 19),
# which they can see (Read) but must not run
$ curl -s -o /dev/null -w '%{http_code}\n' -X POST \
    -H "Authorization: Bearer $JUNIOR_TOKEN" -H "Content-Type: application/json" \
    -d '{}' \
    https://awx.acme.internal/api/v2/job_templates/19/launch/

204
403

204 No Content means the role association was created. 403 means the launch was refused outright, with no job object created at all, which is exactly the difference between a blocked action and an ignored field. Run that second command as part of your access review. A control nobody has tested is a belief, not a control.

Two more roles deserve naming. System Auditor is a user-level flag giving read-only visibility across everything, which is what a security reviewer needs and also the ceiling they should have. Approve, available on the organization and on individual workflow job templates, is the permission to release a paused workflow. Keeping Approve separate from Execute is what turns a two-person rule from a promise into a setting. One housekeeping note before you script any of this: newer builds also list role_definitions and role_team_assignments under /api/v2/, the reworked role model that Ansible Automation Platform 2.5 brought in, with the object_roles shown above still working alongside it. Open /api/v2/ on your own install and see which endpoints are there.

What the Job Record Actually Holds

Every run leaves a job object behind, and it answers the auditor's questions without anybody reconstructing anything.

$ curl -s -H "Authorization: Bearer $AWX_TOKEN" \
    https://awx.acme.internal/api/v2/jobs/4812/ \
  | jq '{id, name, status, started, finished, limit, scm_revision,
         launched_by:  .launched_by.name,
         inventory:    .summary_fields.inventory.name,
         credentials: [.summary_fields.credentials[].name]}'

{
  "id": 4812,
  "name": "Patch fleet - staging",
  "status": "successful",
  "started": "2026-07-14T13:02:44.918733Z",
  "finished": "2026-07-14T13:09:21.550214Z",
  "limit": "staging_web",
  "scm_revision": "9f1c2ab0d7e64c1e8bb3a5f4d20c77e1b4a9c003",
  "launched_by": "priya.n",
  "inventory": "Staging (dynamic, AWS)",
  "credentials": ["fleet-patch (svc-patch)", "vault-staging"]
}

Note that launched_by sits at the top level of the job, not inside summary_fields, which trips up plenty of first attempts at scripting this. Then read scm_revision once more (SCM being source control management, your git repository). That is the exact commit of your playbooks that executed, so "what did the code look like on the day" turns into a git checkout instead of an argument. The full console output hangs off the same job, readable as plain text and downloadable as a file for an evidence pack.

$ mkdir -p evidence
$ curl -s -H "Authorization: Bearer $AWX_TOKEN" \
    "https://awx.acme.internal/api/v2/jobs/4812/stdout/?format=txt_download" \
    -o evidence/job-4812.txt
$ head -3 evidence/job-4812.txt

PLAY [Patch and optionally reboot] ********************************************
TASK [Gathering Facts] ********************************************************
ok: [web03.staging.acme.internal]

There is a second ledger, this one for changes to the controller itself, called the activity stream. It records creates, updates, deletes and role associations, with the actor and the names of the fields that moved.

$ curl -s -H "Authorization: Bearer $AWX_TOKEN" \
    "https://awx.acme.internal/api/v2/activity_stream/?object1=job_template&operation=update&order_by=-timestamp&page_size=3" \
  | jq -r '.results[] | "\(.timestamp)  \(.summary_fields.actor.username)  \(.changes | keys | join(", "))"'

2026-07-14T09:41:02.771456Z  r.okafor  ask_limit_on_launch
2026-07-12T14:08:19.334902Z  r.okafor  become_enabled
2026-07-09T11:30:47.902118Z  priya.n   name, playbook

Somebody turned on ask_limit_on_launch on 14 July at 09:41. You now know who, and you know to go and read every job launched after that moment. Secret values never enter this stream: that a credential's inputs changed is recorded, what they changed to is not.

A Human Gate in Front of Production

A workflow job template chains job templates into a graph with success, failure and always branches. Drop an approval node into that graph and the workflow stops there, like the signature line partway down a form, until a person holding Approve releases it. The node carries a timeout, so an unanswered request fails the workflow instead of hanging until Monday.

$ curl -s -H "Authorization: Bearer $AWX_TOKEN" \
    https://awx.acme.internal/api/v2/workflow_approvals/93/ \
  | jq '{name, status, timeout, timed_out,
         requested: .started, decided: .finished,
         by: .summary_fields.approved_or_denied_by.username}'

{
  "name": "Approve prod patch window",
  "status": "successful",
  "timeout": 3600,
  "timed_out": false,
  "requested": "2026-07-14T21:58:11.402577Z",
  "decided": "2026-07-14T22:06:47.115330Z",
  "by": "d.morel"
}

A status of successful means approved. failed means denied, and it also means timed out, which is why timed_out sits right next to it and why you check both. The decision itself is a POST to /approve/ or /deny/ on that record, stored with the approver's name and the time. Priya asked, Morel released it eight minutes later, and both facts outlive the people involved.

Getting the Record Out of the Controller

Two things will eat your audit trail. The first is sheer volume. Job events are one database row per task per host, so a sixty-task patch playbook across two thousand hosts writes roughly a hundred and twenty thousand rows a night, close to a million a week. Eventually somebody runs the built-in Cleanup Job Details management job (awx-manage cleanup_jobs --days=90 from a shell) to stop the disk filling, and your history goes with it. The second is an attacker who has taken controller admin and can delete the very records describing what they did.

Both arguments point the same way: copy the record somewhere the controller cannot reach. The logging settings do that natively, with per-logger control so you are not stuck with all-or-nothing.

$ curl -s -H "Authorization: Bearer $AWX_TOKEN" \
    https://awx.acme.internal/api/v2/settings/logging/ \
  | jq '{LOG_AGGREGATOR_ENABLED, LOG_AGGREGATOR_TYPE, LOG_AGGREGATOR_HOST,
         LOG_AGGREGATOR_PROTOCOL, LOG_AGGREGATOR_VERIFY_CERT, LOG_AGGREGATOR_LOGGERS}'

{
  "LOG_AGGREGATOR_ENABLED": true,
  "LOG_AGGREGATOR_TYPE": "splunk",
  "LOG_AGGREGATOR_HOST": "https://splunk.acme.internal:8088/services/collector/event",
  "LOG_AGGREGATOR_PROTOCOL": "https",
  "LOG_AGGREGATOR_VERIFY_CERT": true,
  "LOG_AGGREGATOR_LOGGERS": [
    "awx",
    "activity_stream"
  ]
}

The default list holds four loggers and they do different jobs. awx is service and API activity. activity_stream is the who-changed-what ledger. job_events is the per-task firehose, and it is the one that will surprise your SIEM (security information and event management, the platform your security team searches when something goes wrong) license. system_tracking is gathered fact data. On a busy controller, awx and activity_stream earn their keep on day one, which is why the example above keeps those two and drops the rest. Turn job_events back on deliberately, with the volume worked out first.

Verify before you trust it. A POST to /api/v2/settings/logging/test/ makes the controller push a probe message down the configured path, so you find the wrong port or the untrusted certificate now rather than during an incident. Then go and find that message in the SIEM with your own eyes. Only once you can see it sitting there should you shorten the controller's own retention with awx-manage cleanup_jobs or cleanup_activitystream.

The Controller Is Now the Highest-Value Box You Own

Here is the honest cost of all this. You have built one system that holds a working credential for every host you operate, plus cloud keys, plus git deploy keys, and it will run arbitrary code as root across the fleet on request. Root on that machine, or a copy of its database together with its SECRET_KEY, is root everywhere. You traded many small exposures on many laptops for one very large one, and the trade only pays if you defend the large one properly.

Treat SECRET_KEY accordingly. On an AWX Operator install it lives in a Kubernetes secret. On an RPM-based (Red Hat Package Manager, the packaging format Red Hat systems install from) Ansible Automation Platform install it is the file /etc/tower/SECRET_KEY, which keeps the old Tower path for compatibility.

# kubectl is the Kubernetes command-line client
$ kubectl -n awx get secret awx-secret-key

NAME             TYPE     DATA   AGE
awx-secret-key   Opaque   1      412d

Back up the database and that key, keep them in separate places with separate access, and rehearse the restore before you need it. The dependency cuts both ways, usefully. A stolen database dump on its own gives up no credential secrets, though it still hands over hostnames, inventory variables and every line of job output, so do not file it under harmless. Restore that same database onto a fresh install without the original key and the job history comes back intact while every credential secret is permanently unreadable, meaning every one has to be re-entered by hand. People discover this at three in the morning. Discover it during a drill instead.

You can also shrink what the controller holds in the first place. External secret management credential plugins (HashiCorp Vault Secret Lookup, AWS Secrets Manager lookup, Microsoft Azure Key Vault, CyberArk Central Credential Provider Lookup and others) let a credential field be fetched from your secret store when the job starts, rather than sitting in the controller's database at all. The controller then keeps one token to the secret store instead of a thousand keys to your fleet, and the store's own audit log becomes a second, independent record of every retrieval. Two ledgers that have to agree are much harder to quietly edit than one.

Related