Python essentials for engineers

Explanation

When to move from Bash to Python

Bash is ideal for running tools in sequence. Python is the better choice once a script needs data structures, careful parsing, arithmetic, reports or tests of its own.

Scripts and modules

Every .py file is a module. When Python runs a file directly, its __name__ is '__main__'; when another file imports it, __name__ is the module name. Guarding the entry point keeps imports free of side effects, so a regression runner can import filelist and reuse its functions without running its command-line code.

def main() -> int:
    ...
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

raise SystemExit(main()) turns the return value into the process exit status.

Paths with pathlib

pathlib.Path replaces string joining and most os.path calls.

Paths with spaces need no quoting because a Path is never split by a shell.

argparse

argparse builds --help, validates types and reports usage errors with exit status 2 automatically.

parser = argparse.ArgumentParser(description='Build a simulator file list.')
parser.add_argument('roots', nargs='+', type=Path, help='folders to search')
parser.add_argument('-o', '--output', type=Path, default=Path('files.f'))
parser.add_argument('--ext', action='append', default=None)
args = parser.parse_args()

Exceptions

Raise an exception when a function cannot do its job; catch it where you can do something useful, usually in main(). Catch specific types such as FileNotFoundError or ValueError, never a bare except: that would also hide typing mistakes. Print errors to sys.stderr and return a non-zero status.

Worked example

make_filelist.py collects HDL sources into a simulator file list (.f file) that iverilog -c files.f or verilator -f files.f can read. Packages must compile before the modules that import them, so files ending in _pkg.sv are listed first.

#!/usr/bin/env python3
"""Write a simulator file list from one or more source folders."""
import argparse
import sys
from pathlib import Path

DEFAULT_EXTENSIONS = ('.v', '.sv')


def find_sources(roots: list[Path], extensions: tuple[str, ...]) -> list[Path]:
    files: list[Path] = []
    for root in roots:
        if not root.is_dir():
            raise FileNotFoundError(f'not a folder: {root}')
        files.extend(p for p in root.rglob('*') if p.suffix in extensions and p.is_file())
    if not files:
        raise ValueError('no HDL sources found')
    # Packages first, then everything else in a stable order.
    return sorted(set(files), key=lambda p: (not p.stem.endswith('_pkg'), str(p)))


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument('roots', nargs='+', type=Path)
    parser.add_argument('-o', '--output', type=Path, default=Path('files.f'))
    parser.add_argument('--ext', action='append', help='extension to include, e.g. .svh')
    args = parser.parse_args(argv)

    extensions = tuple(args.ext) if args.ext else DEFAULT_EXTENSIONS
    try:
        sources = find_sources(args.roots, extensions)
    except (FileNotFoundError, ValueError) as exc:
        print(f'make_filelist: {exc}', file=sys.stderr)
        return 1

    args.output.write_text(''.join(f'{p}\n' for p in sources))
    print(f'wrote {len(sources)} files to {args.output}')
    return 0


if __name__ == '__main__':
    raise SystemExit(main())

Run it:

python3 make_filelist.py rtl tb -o build/files.f
iverilog -g2012 -s tb_fifo -o build/sim.vvp -c build/files.f
python3 make_filelist.py missing; echo "status=$?"

main() accepts an optional argv list, so a test can call main(['rtl', '-o', 'out.f']) without touching sys.argv. Sorting gives the same file order on every machine, which keeps compile logs comparable. The list[Path] and list[str] | None hints need Python 3.10 or newer.

Check yourself

What does if __name__ == '__main__': protect against when another script imports this file? Why should file lists be sorted? Which exceptions does main() catch, and why not catch every exception? What exit status does argparse use for a usage error?