JSON, CSV, YAML and TOML

Parse and write the formats tools speak, and load untrusted YAML safely.

Security tools hand each other data in four text formats. Scanners and APIs speak JSON, people who review findings want CSV, Kubernetes and CI files are YAML, and your own tool's settings fit TOML. This lesson reads and writes all four, and for each one shows the way it goes wrong: a report cut off mid-file, a spreadsheet cell that runs a formula, a CSV field changed on the way in, a YAML file that turns NO into False or asks the loader to call a function, and a TOML file opened the wrong way. It is also the first lesson that needs a package from outside the standard library, so you will install one, pinned, into a venv.

JSON: load, dump and errors with a position

The lab works in ~/ps-data; unpack the lesson files in your home directory, or create the files as they appear. A scanner has written this report about a container image:

{
  "scanner": "labscan 2.1",
  "scanned_at": "2026-09-27T09:14:22Z",
  "target": "registry.internal/api-gateway:1.8.3",
  "findings": [
    {"id": "CVE-2024-3094", "package": "xz-utils", "severity": "CRITICAL", "cvss": 10.0,
     "fixed_in": "5.6.2", "title": "Backdoor in the upstream release tarballs"},
    {"id": "CVE-2023-44487", "package": "nghttp2", "severity": "HIGH", "cvss": 7.5,
     "fixed_in": "1.57.0", "title": "HTTP/2 rapid reset"},
    {"id": "CVE-2022-40897", "package": "setuptools", "severity": "MEDIUM", "cvss": 5.9,
     "fixed_in": "65.5.1", "title": "ReDoS in package_index"},
    {"id": "LAB-UPLOAD-7", "package": "uploads", "severity": "HIGH", "cvss": 8.1,
     "fixed_in": null, "title": "=HYPERLINK(\"http://203.0.113.9/\",\"Open invoice\")"},
    {"id": "CVE-2024-2511", "package": "openssl", "severity": "LOW", "cvss": 3.7,
     "fixed_in": "3.0.14", "title": "Unbounded memory growth with session handling"}
  ]
}

triage.py keeps the findings someone must act on tonight, worst first, prints them and saves them for the next step:

"""Print the findings in a scan report that need action, worst first."""
import json
import sys
 
URGENT = {"CRITICAL", "HIGH"}
 
 
def by_score(finding: dict) -> float:
    return finding["cvss"]
 
 
def urgent_findings(report: dict) -> list[dict]:
    found = [f for f in report["findings"] if f["severity"] in URGENT]
    return sorted(found, key=by_score, reverse=True)
 
 
def main(argv: list[str]) -> int:
    if len(argv) != 1:
        print("usage: triage.py REPORT", file=sys.stderr)
        return 2
    path = argv[0]
    try:
        with open(path, encoding="utf-8") as f:
            report = json.load(f)
    except OSError as err:
        print(f"triage: {err}", file=sys.stderr)
        return 2
    except json.JSONDecodeError as err:
        # lineno and colno point at the place the parser gave up
        print(f"triage: {path}:{err.lineno}:{err.colno}: {err.msg}", file=sys.stderr)
        return 2
    # Valid JSON is not yet a report: check the shape before using it.
    if not isinstance(report, dict) or not isinstance(report.get("findings"), list):
        print(f"triage: {path}: expected an object with a findings list", file=sys.stderr)
        return 2
 
    urgent = urgent_findings(report)
    print(f"{len(urgent)} of {len(report['findings'])} findings need action")
    for f in urgent:
        print(f"  {f['cvss']:>4} {f['severity']:<8} {f['id']:<15} {f['package']}")
    # open("w") is enough for a file only the next step reads (files lesson: os.replace)
    with open("urgent.json", "w", encoding="utf-8") as out:
        json.dump(urgent, out, indent=2)
        out.write("\n")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

json.load() reads a file and returns Python values: a JSON object becomes a dict, an array a list, true/false become True/False and null becomes None. json.loads() does the same for a string. sorted(..., key=by_score) calls by_score on every finding and sorts by what it returns, the CVSS score. json.dump(urgent, out, indent=2) writes the list back as JSON, indented so a person can read it; without indent everything goes on one line, which is what a program downstream prefers.

Now the two cases a first draft usually crashes on: a scan with nothing urgent, and a report that was cut off because the scanner was killed while writing it. This is the clean scan:

{
  "scanner": "labscan 2.1",
  "scanned_at": "2026-09-27T09:20:05Z",
  "target": "registry.internal/status-page:0.4.1",
  "findings": [
    {"id": "CVE-2024-2511", "package": "openssl", "severity": "LOW", "cvss": 3.7,
     "fixed_in": "3.0.14", "title": "Unbounded memory growth with session handling"}
  ]
}

The clean scan printed 0 of 1 and exited 0, because the code never assumes the list has a first element. The truncated file raised json.JSONDecodeError, and its lineno and colno attributes say where the parser gave up: line 9, column 6, where a string starts and never ends. cat -A marks each line end with $; there is none after "fix, so the file stops in the middle of a word. A message with a position turns "the report is broken" into something you can check in ten seconds. JSONDecodeError is a kind of ValueError, so catch the specific name.

A file can be valid JSON and still not be a report. A list, a string or an object without findings would make report["findings"] fail with a traceback, so triage.py checks the shape before it uses the data:

That check covers the top level only. A finding without a cvss key would still end in a KeyError traceback; the capstone validates every field of the data its API returns. triage.py writes urgent.json with a plain open("w") because only the next step of this lesson reads it. A file that other jobs or people read gets the files lesson's temporary file and os.replace().

Writing JSON fails too, for values JSON has no type for:

A set has no JSON form, and neither has a datetime or a Path. Convert first: sorted(ports) gives a list, .isoformat() a timestamp string, str(path) a path. The last line is a note that Python 3.14 adds to the error, naming the key that held the bad value.

CSV for people, and formula injection

The change board wants the urgent findings as a spreadsheet. to_csv.py writes them with csv.DictWriter:

"""Write the urgent findings from urgent.json as a CSV for the change board."""
import csv
import json
import sys
 
FIELDS = ["id", "package", "severity", "cvss", "fixed_in", "title"]
FORMULA_START = ("=", "+", "-", "@", "\t", "\r", "\n")
 
 
def neutralise(value):
    """Stop a spreadsheet from treating text as a formula (CSV injection)."""
    if isinstance(value, str) and value.startswith(FORMULA_START):
        return "'" + value
    return value
 
 
def main() -> int:
    with open("urgent.json", encoding="utf-8") as f:
        urgent = json.load(f)
    # newline="": the csv module writes and reads line endings itself
    with open("urgent.csv", "w", newline="", encoding="utf-8") as out:
        writer = csv.DictWriter(out, fieldnames=FIELDS)
        writer.writeheader()
        for finding in urgent:
            row = {name: finding[name] for name in FIELDS}
            row["fixed_in"] = row["fixed_in"] or "no fix yet"
            writer.writerow({name: neutralise(value) for name, value in row.items()})
    print(f"wrote {len(urgent)} rows to urgent.csv")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

DictWriter takes the column names once, writeheader() writes them as the first row, and writerow() writes one dict per row in that column order. The dict comprehension picks just the fields in FIELDS, and or "no fix yet" replaces a None fix version. Run it first without the neutralise() call:

Look at the LAB-UPLOAD-7 row. Its title is the name of an uploaded file, so the uploader chose it. The csv module quoted it correctly, but quotes are CSV syntax: the spreadsheet removes them and sees a cell that starts with =, which it evaluates as a formula. This one shows a link to 203.0.113.9 labelled "Open invoice" in the middle of your security report. Other formulas can fetch URLs or pull data from other cells. This is CSV injection (formula injection, CWE-1236), and every field that came from outside (user names from logs, file names, page titles) can carry it.

neutralise() puts an apostrophe in front of any text that starts with =, +, -, @, a tab, a carriage return or a line feed, the characters OWASP lists. A spreadsheet shows such a cell as text. The price is in the read-back: a program that reads the CSV now sees the apostrophe as part of the value. So neutralise only the CSV meant for spreadsheets and keep urgent.json unchanged for machines. OWASP also notes that the apostrophe can be lost when Excel saves the file again, and describes a tab-based alternative with the same trade-off. Numbers are left alone by the isinstance check.

What newline="" is really for

The csv documentation says to open CSV files with newline="". The usual explanation, "otherwise you get blank rows", is true on Windows only. Here is the same export written without it, on this Linux machine:

The bytes are identical. The writer ends each row with \r\n itself. On Linux, text mode leaves line endings alone when writing; on Windows it turns every \n into \r\n, so the writer's \r\n becomes \r\r\n and spreadsheets show an empty row. The reason that holds everywhere is on the reading side. This ticket export came from a Windows tool, with \r\n at the end of every line and one note that spans two lines inside quotes. printf creates it byte for byte, which an editor might not:

"""Read tickets.csv twice: with newline="" and with the default newline handling."""
import csv
 
for newline in ("", None):
    with open("tickets.csv", newline=newline, encoding="utf-8") as f:
        notes = [row["note"] for row in csv.DictReader(f)]
    print(f"newline={newline!r:<5} {notes[0]!r}")

cat -A shows ^M for each \r. With newline="" the csv module saw the file's real bytes and returned the note exactly as written. With the default, Python's universal-newlines mode turned the \r\n inside the quoted field into \n before csv saw it: the value changed, and nothing reported it. For a note that matters little; for data you hash, compare or pass on, it is silent corruption. Use newline="" whenever you open a file for csv, reading or writing.

YAML: a justified dependency, and safe_load

The standard library has no YAML parser, and YAML is what the tools around you use for configuration, so here an outside package earns its place. PyYAML is the common choice; 6.0.3 is its current release. Install it pinned with ==, into a venv, as in the setup lesson:

The new venv could not import yaml, even though Ubuntu has it: the last command shows the distribution's copy, installed for the system Python by the python3-yaml package. A venv does not see system packages, which is the point. Your script should depend on the version you pinned, not on whatever the host's package manager installed.

A rules file for a VPN gateway, written the way people write YAML by hand, and a loader that uses yaml.safe_load() and then checks the types it got:

# Geo-blocking rules for the VPN gateway
allow_countries: [SE, NO, DK, FI]
min_client_version: 1.10
maintenance_window: 22:30
ticket_queue: 010
notify:
  on: failure

"""Load a rules file with yaml.safe_load and check the types it produced."""
import sys
 
import yaml
 
TEXT_FIELDS = ("min_client_version", "maintenance_window", "ticket_queue")
 
 
def type_problems(rules: dict) -> list[str]:
    problems = []
    for code in rules.get("allow_countries", []):
        if not isinstance(code, str):
            problems.append(f"allow_countries: {code!r} is not text")
    for field in TEXT_FIELDS:
        if field in rules and not isinstance(rules[field], str):
            problems.append(f"{field}: {rules[field]!r} is not text")
    for key in rules.get("notify", {}):
        if not isinstance(key, str):
            problems.append(f"notify: key {key!r} is not text")
    return problems
 
 
def main(argv: list[str]) -> int:
    if len(argv) != 1:
        print("usage: load_rules.py RULES.yaml", file=sys.stderr)
        return 2
    try:
        with open(argv[0], encoding="utf-8") as f:
            rules = yaml.safe_load(f)
    except OSError as err:
        print(f"load_rules: {err}", file=sys.stderr)
        return 2
    except yaml.YAMLError as err:  # syntax errors and refused tags
        print(f"load_rules: {argv[0]}: {err}", file=sys.stderr)
        return 2
    if not isinstance(rules, dict):  # an empty file loads as None
        print(f"load_rules: {argv[0]}: not a mapping of settings", file=sys.stderr)
        return 2
    for key, value in rules.items():
        print(f"{key:<20} {value!r}")
    problems = type_problems(rules)
    for problem in problems:
        print(f"load_rules: {problem} (quote it)", file=sys.stderr)
    return 2 if problems else 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Nothing failed to parse, and five values are wrong. PyYAML follows YAML 1.1, which guesses a type for every unquoted value. NO, Norway's country code, is a boolean in YAML 1.1, as are yes, off and on, which is why the on key became True. 1.10 is a float, so the version became 1.1. 22:30 is a base-60 number (22 × 60 + 30 = 1350), and 010 is octal, 8. A gateway enforcing this allow list would turn away users in Norway. YAML also accepts a key written twice, silently:

The second allow replaced the first without a warning. The fix for the typing traps is to quote every value that is text, and to check the types after loading, as type_problems() does, so a file someone edits later cannot slip a boolean in:

# Geo-blocking rules for the VPN gateway
allow_countries: ["SE", "NO", "DK", "FI"]
min_client_version: "1.10"
maintenance_window: "22:30"
ticket_queue: "010"
notify:
  "on": failure

The other YAML risk is tags. A tag such as !!python/object/apply:os.getcwd asks the loader to call a Python function and use the result as the value. Here it is in a pull request:

# Geo-blocking rules, as proposed in a pull request
allow_countries: !!python/object/apply:os.getcwd []
min_client_version: "1.10"

safe_load() builds only plain values (dicts, lists, strings, numbers, booleans, None, dates) and refused the tag with a ConstructorError, a subclass of yaml.YAMLError, naming line 2, column 18. The payload here is harmless; the same tag with os.system and a command would run it. PyYAML has loaders that honour such tags, and PyYAML 6 makes you name the loader explicitly for exactly this reason. Never use them on a file you did not write:

# UNSAFE: builds any object a tag names, and calls functions to do it
rules = yaml.load(text, Loader=yaml.UnsafeLoader)
 
# safe: plain values only; unknown tags raise yaml.YAMLError
rules = yaml.safe_load(text)

A loader also says nothing about shape. An empty file is valid YAML and loads as None, and a file that holds a list loads as a list; either would make rules.items() fail with a traceback. load_rules.py checks that it got a mapping first:

TOML for your own settings: tomllib

For a file only your tool reads, TOML is the simpler choice, and tomllib is in the standard library (Python 3.11 and later). TOML has no type guessing: text must be quoted, and a time of day is written in full. The second file below is the same settings file with the country codes left unquoted, as a YAML habit would write them.

# Settings for our own export tool
allow_countries = ["SE", "NO", "DK", "FI"]
min_client_version = "1.10"
maintenance_window = 22:30:00
max_rows = 500
 
[notify]
on = "failure"
channel = "#secops"

# Settings for our own export tool
allow_countries = [SE, NO, DK, FI]
max_rows = 500

"""Load the tool's own settings from a TOML file with the standard library."""
import sys
import tomllib
 
 
def main(argv: list[str]) -> int:
    if len(argv) != 1:
        print("usage: load_settings.py SETTINGS.toml", file=sys.stderr)
        return 2
    path = argv[0]
    try:
        with open(path, "rb") as f:  # tomllib reads bytes and decodes UTF-8 itself
            settings = tomllib.load(f)
    except OSError as err:
        print(f"load_settings: {err}", file=sys.stderr)
        return 2
    except tomllib.TOMLDecodeError as err:
        print(f"load_settings: {path}:{err.lineno}:{err.colno}: {err.msg}", file=sys.stderr)
        return 2
    for key, value in settings.items():
        print(f"{key:<20} {value!r}")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

22:30:00 became a datetime.time, a real time value, and every string stayed a string. The file with unquoted country codes was refused with a position, where YAML would have guessed: TOMLDecodeError has lineno and colno attributes since Python 3.14. The last command opened the file in text mode, and tomllib refused it: it reads bytes and decodes UTF-8 itself, so the file must be opened with "rb". tomllib only reads TOML; writing it needs a third-party package, which a config file you edit by hand rarely does.

Try this

Write load_config.py CONFIG in ~/ps-data. It loads .toml files with tomllib and .yaml files with yaml.safe_load (run it with .venv/bin/python), rejects any other suffix, and exits 2 with one line on stderr when the file cannot be read or parsed (for TOML, file:line:col: message) or when allow_countries holds anything that is not a string. Otherwise it prints FILE: SE, NO, ... and exits 0. Test the failure first: the unquoted rules.yaml, where NO became False, must be refused with country codes must be text: [False] and exit 2. Then verify: rules-quoted.yaml and gateway.toml both print SE, NO, DK, FI and exit 0, and gateway-typo.toml prints gateway-typo.toml:2:20: Invalid value and exits 2. If you raise ValueError for an unknown suffix and catch it in the same place, put the tomllib.TOMLDecodeError handler first: it is a subclass of ValueError, and the first matching except wins.

Takeaway

Parse with the strictest loader the format has (json.load, yaml.safe_load, tomllib with "rb"), report parse errors with their position, and check the types you got before acting on them. Treat every field bound for a spreadsheet as a possible formula, and open CSV files with newline="" in both directions.

Related