Project layout and venvs

Isolated, reproducible, not on the system Python.

A system-wide Python is a shared tool drawer in a busy workshop. pip (Python's package installer, the command that pulls libraries off the internet) drops everything anyone installs into one directory called site-packages, and every script on the machine reaches into that same drawer. The interpreter, the python program that reads your .py files and runs them line by line, hands your code whatever version of a library happens to be sitting in the drawer that day. Change the drawer for one tool and you have changed it for all of them.

A virtual environment, or venv, gives each project its own drawer. It is a private copy of the interpreter's entry point plus a private site-packages, built in seconds and thrown away as fast. Nothing you install for one project can reach into another. For an operator that property is a control, not tidiness.

The failure a venv prevents is silent behavioral drift. Your phishing-triage script ran correctly during last month's incident. Then a teammate globally upgraded a date-parsing library that some unrelated tool needed, and now your script reads timestamps in a different order, or dies in the middle of an investigation. Nothing in your code changed. The floor moved under it.

Installing into the system Python is worse than fragile. It usually means sudo pip install, where sudo runs the command as root (the all-powerful administrator account). Now pip executes each dependency's build scripts as root, so anyone who has slipped malicious code into one of those dependencies runs it with complete control of the machine. Modern Linux draws a hard line here: on Debian 12 and Ubuntu 24.04, pip refuses to install into the system at all unless you are inside a venv.

That wall is PEP 668 (a Python Enhancement Proposal, one of the numbered design documents that define changes to Python and its tooling). A stray pip upgrade can replace a library that apt, Debian's own package manager, depends on, and break the operating system's own tools. The distribution would rather you build your own drawer. The --break-system-packages flag exists to override the wall, and its name is the whole review comment you will write when a colleague uses it.

What a venv actually is

python3 -m venv .venv writes a directory with three things in it. A bin/ folder (Scripts\ on Windows) holding a copy or symlink (a small pointer file to the real one) of the interpreter. A private lib/.../site-packages, empty except for pip. And a small text file, pyvenv.cfg, that is the entire mechanism.

When the copied interpreter starts, it looks for pyvenv.cfg sitting next to itself. It reads home = to find the base installation, because the standard library (the batteries Python ships with, like json and socket) is never copied. It is borrowed from the base. Then it sets sys.prefix (Python's record of where your installed packages live) to the venv, while sys.base_prefix keeps pointing at the original install. From that split, import pulls the standard library from the base and everything else from your venv's own site-packages. The include-system-site-packages = false line is what keeps the base machine's packages out of your view.

Activation did almost nothing. The activate script prepends .venv/bin to your PATH (the list of directories your shell searches to find a command) and edits your prompt. That is all it does. Running /home/analyst/ioc-triage/.venv/bin/python by its full path gives you the exact same interpreter with no activation at all. Hold onto that fact. It is the whole point of the scheduled-jobs section below.

A layout that survives its second contributor

Structure the project as a src layout. The package you import, ioc_triage (IOC means indicator of compromise, an artifact like a suspicious IP address or file hash that hints at an intrusion), lives under a src/ directory. Tests live outside it. pyproject.toml, the standard project manifest defined by PEP 621, describes the whole thing in one file. The name src is a convention meaning source.

The src layout closes a quiet trap. With a flat layout, where the package folder sits at the top of the project, import ioc_triage can resolve to that folder in your current working directory instead of the installed copy. Your tests pass on your laptop and fail in CI (continuous integration, the server that rebuilds and tests your code on every push) or on the SOC jump box (SOC is the security operations center; the jump box is the shared, locked-down host analysts use to reach sensitive systems). Put the package under src/ and that accidental import becomes impossible. You have to install the project to import it, so you always test the thing you actually ship.

[project]
name = "ioc-triage"
version = "0.1.0"
description = "IOC enrichment helpers for the SOC"
requires-python = ">=3.12"
dependencies = []          # runtime deps go here; a later lesson fills it
 
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Install the project in editable mode, the -e flag. pip registers the package in the venv but loads its code straight from src/, so every edit takes effect on the next run with no reinstall.

That final error is the isolation proving itself. Inside the venv the import resolves to your source tree under src/. Deactivate, and the package is gone. It exists in the venv and nowhere else on the machine.

Cron and systemd never see your activation

Here is where this stops being tidy code and starts being an on-call problem. Activation changes one shell session. A scheduled job runs in a fresh process that never saw your shell, so it never saw your PATH change either.

systemd (the service manager that starts and supervises background programs on most Linux systems) and cron (the older Unix scheduler that runs commands at fixed times) both launch your script with a bare environment. Call bare python and you get the system interpreter, none of your packages, and a crash at 03:00. Worse, the system Python might carry some old globally-installed version, so the job half-runs against stale code and reports success while producing wrong answers. Nobody notices until an analyst trusts a bad result.

The fix is the full path you saw earlier. Point the job at the venv interpreter directly, and set the working directory so relative file paths resolve the same way they do on your laptop.

[Unit]
Description=Nightly IOC enrichment
Wants=network-online.target
After=network-online.target
 
[Service]
Type=oneshot
User=ioc
WorkingDirectory=/opt/ioc-triage
ExecStart=/opt/ioc-triage/.venv/bin/python -m ioc_triage.enrich

[Unit]
Description=Run IOC enrichment nightly at 03:00
 
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
 
[Install]
WantedBy=timers.target

Reload systemd so it reads the new units, run the service once by hand, and read the log. Verifying the change is the job, not writing it.

Persistent=true tells the timer to run a job it missed while the machine was off, the moment it boots, so a maintenance reboot at 02:45 does not silently skip your nightly run. Type=oneshot tells systemd the service is a task that finishes, not a daemon (a background program that keeps running), so a clean exit reads as success rather than a crash.

What a venv is not

Be blunt about the boundary. A venv is an isolation convenience, not a security wall. Code inside it runs as your operating system user. It can read ~/.ssh, ~/.aws/credentials (your cloud keys), and every environment variable in the process. It can open network connections to anywhere. A malicious package is exactly as dangerous inside a venv as outside one. Real sandboxing of untrusted code needs containers, virtual machines, or controls built into the kernel (the core of the operating system that manages hardware and processes), not a private site-packages.

Venvs also do not travel. The absolute path to the venv is baked into pyvenv.cfg and into the first line of every installed script.

That first line is a shebang (the #! line that tells the kernel which interpreter to run a file with). Copy this venv to another machine, or into a container image, and the shebang points at a path that does not exist there. On Linux a shebang also has a hard length limit of about 127 characters, so a venv buried under a long deploy path can produce scripts that fail to launch with a confusing not-found error. Rebuild the venv in place instead. It costs seconds.

So treat a venv as strictly local. Do not copy one onto another host or into a container image with scp, rsync, or a Dockerfile COPY, and do not restore one from a backup. The baked-in paths and shebangs will point at directories that are not there, and the failures are silent or cryptic. Ship pyproject.toml and a lockfile (a file that pins the exact version of every dependency), then rebuild the environment fresh on the target.

Three habits pay for themselves. Set PIP_REQUIRE_VIRTUALENV=true in your shell profile so pip refuses to touch the system Python even when you forget to activate.

Keep .venv/ in .gitignore. The environment is a build artifact, rebuilt from pyproject.toml in seconds; committing it leaks machine paths and bloats the repository. And treat venvs as disposable. When one misbehaves, delete it and make a fresh one rather than debugging its guts.

At scale, CI should rebuild the environment from scratch on every run so nothing stale survives between jobs. uv, a fast installer written in Rust that the Python community has adopted widely, makes that rebuild almost free and produces the same venv layout, so every rule above still holds.

That empty dependencies = [] line does not stay empty for long. Real triage means reaching out to threat-intelligence and ticketing services over the network, and that is where correctness becomes a security property you can name: setting a timeout so one hung server cannot freeze the whole run, retrying a call that failed for a moment, verifying TLS certificates (TLS is Transport Layer Security, the encryption under HTTPS) so you know you are talking to the real service and not an impostor, and keeping API keys (application programming interface keys, the secret tokens that prove who you are to a service) out of your source code. That is the next lesson.

Try this

Work through “What a venv is not” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: activation stops at your shell prompt. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related