Files and paths with pathlib

Read, write, and walk the filesystem cleanly.

A file path is an address. /var/log/nginx/access.log tells the computer which building, which floor, and which room to open. For years, Python programs built these addresses by gluing text together with slashes, and that broke in quiet, annoying ways: a missing slash in one spot, a Windows backslash in another, a doubled // somewhere else. The pathlib module (a module is a bundle of ready-made Python code you import, and this one has shipped with every standard Python install since version 3.4) treats a path as a real object you can build, question, and act on, rather than a fragile run of text. In security automation work the wrong path is expensive: you read a file you never meant to touch, or you hand an attacker a file you never meant to expose. Getting this part right is worth the few minutes it takes.

Build a path with /, not string glue

The old way reached for os.path.join("/var/log", "nginx", "access.log") to stitch the pieces together with the right separator for the operating system. pathlib does the same job, but it borrows the division sign (/) so the code on the page looks like the path it produces. You start with a base Path, then keep dividing by the next piece. Python drops in the separators for you, whether the machine runs Linux, macOS, or Windows, so you never hand-type a slash and never end up with a mangled /var/log//nginx.

from pathlib import Path
 
base = Path("/var/log")
target = base / "nginx" / "access.log"
print(target)
print(type(target))

The printed type is PosixPath, the Linux and macOS flavor of a path (POSIX is the family of standards those systems share). Run the same code on Windows and you get a WindowsPath that prints with backslashes instead. That is the whole idea. You wrote one expression, and pathlib picked the right shape for the computer it landed on. Nothing in your script has to know or care which operating system it runs on.

Ask a path about itself

A path is like a shipping label. The whole label is the address, but most of the time you want one field off it: the parcel's name, what kind of thing it holds, the shelf it sits on. pathlib hands you each field as a plain attribute. .name is the final piece, .suffix is the file extension including the dot, .stem is the name with that extension stripped off, and .parent is the folder holding the file. They come back ready to use, some as text and some as Path objects, so you feed them straight into the next step without splitting strings on dots or slashes yourself.

from pathlib import Path
 
p = Path("/var/log/nginx/access.log")
print("name  ", p.name)
print("suffix", p.suffix)
print("stem  ", p.stem)
print("parent", p.parent)

A folder to poke at

The rest of the lesson runs against a small tree of pretend log files, so every command below has real output you can check against. Three .log files sit inside two subfolders, and one .txt file rides along to prove the filters actually filter. Here is the whole tree on disk.

Exists? File? Folder? Read it, write it

Before you open a file, you usually want two answers: is it there, and is it the kind of thing you think it is? .exists() gives you the first as a yes or no. .is_file() and .is_dir() tell a real file apart from a folder, which stops you from trying to read a directory as though it were text. For small files, .read_text() hands back the whole contents as one string, and .write_text() drops a string onto disk, making the file or overwriting whatever was there. Each call opens the file, does the read or write, and closes it again, all in one line.

from pathlib import Path
 
p = Path("logs_demo/nginx/access.log")
print("exists ", p.exists())
print("is_file", p.is_file())
print("is_dir ", p.is_dir())
 
first = p.read_text().splitlines()[0]
print("line 1 ", first)
 
report = Path("review_summary.txt")
report.write_text("access.log reviewed\n")
print("wrote  ", report.read_text().strip())

Find files: iterdir, glob, rglob

Three tools cover almost every file search you will write. .iterdir() opens one folder and lists what sits directly inside it, like pulling out a single drawer. .glob(pattern) matches the names in that same folder against a shell-style wildcard, so *.log means any name that ends in .log (the * is a stand-in for any run of characters). .rglob(pattern) runs the exact same match but walks every subfolder underneath as well, like opening every drawer in the building. The r is for recursive, which means the search repeats itself down through each nested folder without you writing the loop.

from pathlib import Path
 
root = Path("logs_demo")
 
print("iterdir (top level only):")
for child in sorted(root.iterdir()):
    print("  ", child.name)
 
print("glob '*.log' (top level):")
print("  ", [p.as_posix() for p in sorted(root.glob("*.log"))])
 
print("rglob '*.log' (every subfolder):")
for p in sorted(root.rglob("*.log")):
    print("  ", p.as_posix())

Look at the glob('*.log') line: an empty list. The .log files live one level down, inside app and nginx, and plain glob only ever looks in the folder you called it on. rglob is the one that reaches into the subfolders and turns up all three. Mixing these two up is a common reason a script quietly reports nothing while the files are sitting right where you left them.

Read a big file line by line

When a file might be enormous, you read it one line at a time and let Python throw each line away as soon as you are done with it. The shape with open(p) as f: opens the file and promises to close it again, even if your code blows up halfway through. Looping with for line in f: pulls in a single line, runs your check, then moves to the next one, so a ten-gigabyte log costs you one line of memory at a time instead of ten gigabytes all at once. A Path object plugs straight into open(), so you pass it in with nothing to convert. Here is a keyword scan across every log file in the tree.

from pathlib import Path
 
root = Path("logs_demo")
needle = "ERROR"
for log in sorted(root.rglob("*.log")):
    hits = 0
    with open(log) as f:
        for line in f:
            if needle in line:
                hits += 1
    if hits:
        print(f"{hits:>3}  {log.as_posix()}")

resolve() and the ../../etc/passwd trap

This is where file code turns into security code. resolve() takes a messy path and returns the one true absolute path (the full address starting from the root of the disk), cleaning up . and .. steps along the way and following any symbolic links (a symbolic link, or symlink, is a signpost file that points at another location) to wherever they really lead. Here is the catch. .. means go up one folder, and an attacker who controls part of a path can string enough of them together to climb out of the folder you meant to keep them inside. A request for ../../etc/passwd aimed at your uploads folder is a try at reading the system's list of user accounts. The fix is to resolve the joined path first, then check that the result still sits inside your base folder using .is_relative_to() (added in Python 3.9).

from pathlib import Path
 
BASE = Path("/srv/uploads").resolve()
 
def safe_path(user_input: str) -> Path:
    candidate = (BASE / user_input).resolve()
    if not candidate.is_relative_to(BASE):
        raise ValueError(f"path escapes {BASE}: {user_input!r}")
    return candidate
 
for req in ["report.txt", "2026/audit.log", "../../etc/passwd"]:
    try:
        print("OK  ", safe_path(req))
    except ValueError as e:
        print("DENY", e)

Put it together: a log report

This last script is the kind of thing you would actually keep around. Point it at a directory, and it finds every .log file underneath, counts the lines in each, adds up the bytes, and prints a tidy table. It leans on rglob to gather the files, the streaming loop to count lines without loading whole files, .stat().st_size for the byte count of each file, and .relative_to(root) so the table shows short names instead of long absolute paths.

import sys
from pathlib import Path
 
def count_lines(path: Path) -> int:
    n = 0
    with open(path) as f:
        for line in f:
            n += 1
    return n
 
def main(root_arg: str) -> None:
    root = Path(root_arg)
    if not root.is_dir():
        print(f"not a directory: {root}")
        raise SystemExit(1)
 
    total_bytes = 0
    total_lines = 0
    logs = sorted(root.rglob("*.log"))
 
    print(f"log report for {root.as_posix()}/")
    print(f"{'file':<24}{'lines':>7}{'bytes':>9}")
    for p in logs:
        lines = count_lines(p)
        size = p.stat().st_size
        total_bytes += size
        total_lines += lines
        print(f"{p.relative_to(root).as_posix():<24}{lines:>7}{size:>9}")
 
    print("-" * 40)
    print(f"{f'{len(logs)} files':<24}{total_lines:>7}{total_bytes:>9}")
 
if __name__ == "__main__":
    main(sys.argv[1])

From here you can bolt real checks onto the same skeleton: flag any log that grew past a size you set, match each line against a list of suspicious patterns, or write the table out with write_text() so a scheduled job can mail it to you every morning. The path handling underneath does not change as you add those, which is the reason to build on pathlib instead of strings you taped together by hand.

Try this

Work through “Put it together: a log report” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: read_text() pulls the whole file into memory. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related