Tuning Falco for signal
Scoped exceptions, custom rules, low noise.
A car alarm that honks every time a cat walks past gets unplugged by its owner inside a week. Falco, straight out of the box, honks at plenty of things that are completely normal in your cluster. Log shippers writing to paths the default rules watch. Health checks opening a shell inside a debug namespace (a namespace is the partition Kubernetes uses to keep one team's workloads separate from another's). If you install Falco and stop there, you have done about half the job.
A few words in plain English before the rest. Tuning means three things together: exceptions scoped as tightly as the evidence allows, custom rules that guard the workloads you cannot afford to lose, and numbers that describe how good the signal is rather than how big it is. That combination is the whole difference between a SOC (security operations centre, the team watching alerts around the clock) that trusts runtime alerts and one that mutes the channel on a Tuesday and never unmutes it.
Trim the rule, do not switch it off
Shadow tuning is the version where somebody runs kubectl edit against a live ConfigMap (the Kubernetes object that holds your rule files) at two in the morning and tells nobody. It fights GitOps, the practice of keeping the cluster's desired state in a Git repository so the cluster is continuously pulled back to whatever the repo says, and it vanishes at the next sync. Ban it both ways. Socially, by saying out loud that nobody does this. Technically, by requiring a pull request for every rule change.
Every exception gets an expiry date. There is no arguing this one. A calendar reminder, or a time annotation carried in the GitOps repo, beats a permanent mute that outlives the temporary CI job (continuous integration, the automated build and test pipeline) that needed it by two years. Walk the list of expired exceptions once a month, the same way you walk expiring certificates. Practise the loop on staging until it feels boring. Boring is what reliable looks like.
When a rule fires on behavior you know is fine, add a narrow exclusion instead of switching the rule off. Narrow means naming the thing: this namespace, this image, this parent process. You do that with append: true in an override file written in YAML (the indented plain-text format Falco uses for rules), loaded after the defaults, so your clause is bolted onto the shipped rule rather than replacing it. The malicious version of the same behavior, coming from a different image in a different namespace, still matches and still fires.
- rule: Terminal shell in containercondition: >spawned_process and container and proc.name in (shell_binaries)and not (k8s.ns.name = "debug-sandbox" and container.image.repository = "acme/toolbox")append: true
Count first, mute second
Run Falco in the environment you actually care about, and leave it running long enough to see a normal week: a deploy, a batch job, a backup window. Ship the counts, broken out by rule name and namespace, into Prometheus (a database that scrapes numbers on a schedule and keeps them over time) using Falcosidekick, the component that fans Falco events out to other systems. Then sit down weekly with the top of that list and ask one question per rule. Is this a real gap you need to close, or benign traffic you can carve out?
Write your own rules for the workloads you cannot lose
Your payments tier is the vault room, not the lobby. You know exactly what it does: java spawns from the entrypoint and nothing else does, and writes land under /app/data and nowhere else. Rules that specific belong in their own file, at a higher priority and a tighter scope than anything that shipped with the install.
- rule: Unexpected child of payments entrypointcondition: spawned_process and k8s.deployment.name = "payments-api" and proc.pname != "java"priority: CRITICAL
Lists and macros: change one line, change every rule
A list in Falco works like the approved-visitor sheet at a front desk. Put your tooling images in one list called allowed_debug_images and point every rule at it, instead of copying the same not() clause into nine conditions and forgetting the tenth. Read those lists out loud once a quarter. An image nobody runs any more is still a hole you left open.
grep -r "append: true" /etc/falco/rules.local.d/ | wc -l# example output:17
Who is allowed to tune what
Overrides live in Git and get reviewed like application code, because that is what they are. Your staging Falco should already be running the rules prod is about to get, and CI should diff the two rule sets and complain loudly when they drift apart.
kubectl -n falco get cm falco-rules -o jsonpath="{.data}" | wc -c# example output:48291
Putting numbers on signal quality
Pick a small set of numbers and publish them as OKR-style targets (objectives and key results, the goal format most leadership teams already use). What share of Critical alerts got a human look inside an hour. How many false positives each rule produced this month. How long it took, in a drill, to go from the first shell alert to a contained pod. Report a tuning win as fewer false positives with the true positive count holding steady. Leadership funds precision work when you show them analyst hours saved. Nobody funds a slide that says alert volume dropped because you turned rules off.
curl -s http://falco-exporter:7979/metrics | grep falco_rules_matches_total | head -3# example output:falco_rules_matches_total{rule="Terminal shell in container",priority="Warning"} 42falco_rules_matches_total{rule="Write below etc",priority="Warning"} 611
Every append exception is a commit, so compare the numbers on both sides of it. If Critical fires climb after a tuning commit and nobody can explain why, roll that commit back and find the explanation afterwards.
Field notes from real clusters
The instinct when a rule screams is to reach for the off switch. Resist it and go one level narrower instead. The rule, then the namespace, then the image, then the exact process name. Most noise dies at the image level. If you find yourself operating at the rule level, you have stopped tuning and started deleting detection.
Sort a week of match counts by volume and the shape is nearly always the same. Three or four rules produce most of the fires, and inside each one the hits pile up in one or two namespaces or behind a single image. Name that image or that service account in the exception and the count collapses without touching anything else. Setting a famous rule to priority:0 across the board is how ransomware, six months later, gets filed as known noise.
Keep your own rules in Git next to the Helm values (the settings file for the packaged install) of the service they protect, so whoever changes the deployment trips over the rule watching it. Anything that lowers sensitivity on prod needs a second reviewer, with no exemption for seniority. A rule pull request is a firewall change wearing YAML, and you already know how carefully you read a firewall change.
One allowlist shared by twenty rules is a gift and a liability at the same time. Update it once and every rule changes behavior in the same instant, which is exactly what you want and exactly what makes it dangerous. Put an owner's name in a comment at the top of each list. Lists without owners become junk drawers, and junk drawers only ever get bigger.
Split the permissions the way you split cluster access. Platform ships baseline exceptions for the system namespaces they run. App teams propose exceptions for their own namespace, with an expiry attached. Security owns any merge that touches a critical rule. Agree that split on a calm afternoon, because during an incident it stops being a process question and turns into an argument about who is to blame.
Once a month, group rules into families and look at three columns per family: true positives, false positives, time to triage. A rule that has never fired and, given what runs in that cluster, never could, should be retired. A rule that fires daily and has never once produced an investigation should be tightened until it does. Tuning is a product you maintain on a schedule, not a weekend spent in YAML.
New platform components arrive with a noise spike attached. A CSI driver (Container Storage Interface, the plugin that lets Kubernetes attach storage to pods) mounts volumes in ways your file rules have never seen. A service mesh sidecar opens sockets and rewrites networking rules the moment a pod starts. Put a tuning window in the rollout plan next to the canary window. When a rollout surprises the on-call engineer, that window usually got cut for time.
One team had a package-manager rule producing twelve thousand hits a day, all of it from a single observability sidecar that shelled out to apk. They disabled the whole rule globally, the graph went flat, everyone moved on. Three months later a genuinely compromised pod installed tooling on itself and not one alert fired. The fix they should have written was a list exception pinned to that image digest, with a comment explaining it and a ninety-day review date. Global disables are how tuning quietly becomes cover for the attacker.
Read every condition you write out loud before you merge it. If you cannot explain each clause to the person sitting next to you, it is too clever for anyone to operate at three in the morning. Named lists and macros like known_debug_images beat three layers of nested negation that nobody will dare to edit. The incident that needs your rule will be handled by someone who was not in the Slack thread where you wrote the YAML.
Here is the failure mode to watch for in yourself. Slack is buzzing, so you lower the priority until it stops, and you change nothing about retention or review in the SIEM (security information and event management, the system that stores and correlates your logs). The noise is all still there. You blinded the humans and left the events running. Either fix the match precision or accept the volume and route it to a lower-tier channel that someone genuinely reads. Lowering a priority to quiet a room is theater, not tuning.
There is a real trade-off in aggressive allowlists. They cut false positives hard, and they build up debt, because every new legitimate image needs somebody to go and update the list. Pair each allowlist with a named owner and a CI check that fails when an exception points at a deployment that no longer exists. An exception nobody can trace back to a live workload is policy rot with a straight face.
Add three tags to your ticketing system and actually use them: falco-fp for false positive, falco-tp for true positive, falco-needs-tune for the ones in between. Without tags, your monthly metrics are vibes with a chart on top. With tags, you can show that tuning cut the number of pages while true positives held flat, which is the one graph that keeps a detection program funded. An hour of tagging after a noisy week buys you a quieter quarter.
Watch the shape of your own rules file. If it is mostly exceptions, you have written down your history instead of your threat model. Go through and delete the exceptions whose owners left the company or whose deployments were decommissioned last spring. A dead allowlist entry is camouflage sitting there waiting for someone to put it on.
Try this
Run this on a lab cluster or one staging node, never on prod first. Read what comes back and sit with it for a minute. A single dry run is not enough evidence to go and change a production policy.
$ falco -r /etc/falco/falco_rules.yaml -r /etc/falco/rules.d/ -L warning --dry-runValidating rules...OK: 312 rules loaded, 4 exceptions appended$ kubectl -n falco logs ds/falco --tail=1 | grep -c Warning2
Takeaway
Tune with exceptions that are narrow, dated and owned, and prove the effect with numbers rather than a feeling that the channel got calmer. Never buy quiet by silencing a high-value rule across the whole cluster. Lists and macros are what keep the file readable a year from now.
Next: find the loudest rule in your cluster this week, write one exception scoped to the image causing it, put the expiry note in the same commit, and check that genuine hits on that rule still land somewhere a human will pick them up.