45 minv1.0

A Python regression runner

Learning objectives

Run EDA tools from Python with subprocess.run, using argument lists, timeouts and check
Run many test-and-seed combinations in parallel with concurrent.futures
Keep each run isolated in its own folder
Make random seeds reproducible
Return an exit status that a CI pipeline can use as its verdict

1 Objective→ 2 Learn→ 3 Example→ 4 Practice→ 5 Verify→ 6 Continue

How to study this lessonUnderstand the idea first. Use the example to see the behaviour, practise where provided, then make sure you can explain the result before continuing.

Explanation

What a regression runner does

A regression is the same set of tests run again after every change, usually with several random seeds per test. The runner must compile once, launch many simulations, stop runs that hang, collect results, and give one clear answer: did anything fail? Make can do part of this; Python adds timeouts, seed handling and structured results.

subprocess.run

import subprocess

proc = subprocess.run(
    ['vvp', '-n', 'build/tb_fifo.vvp', '+seed=7'],
    capture_output=True, text=True, timeout=120,
)
print(proc.returncode, proc.stdout[-200:])

Argument	Purpose
A list, not a string	Each element is one argument; no shell, so no quoting or injection problems
`capture_output=True, text=True`	Collect stdout and stderr as strings
`timeout=seconds`	Kill the process and raise `subprocess.TimeoutExpired` if it runs too long
`check=True`	Raise `subprocess.CalledProcessError` on a non-zero exit status
`cwd=path`	Run in a given folder

Use check=True for steps that must succeed, such as compilation, so a failure cannot go unnoticed. For simulations, leave check off and decide the verdict from the log, because a testbench may report failures while exiting with status 0. Avoid shell=True unless you need shell features; with a list there is nothing to quote.

Parallel runs

Simulations are separate processes, so threads are enough to launch and wait on them: Python only waits while the simulator works. ThreadPoolExecutor runs a function across many inputs with a fixed number of workers.

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=4) as pool:
    futures = [pool.submit(run_one, job) for job in jobs]
    for future in as_completed(futures):
        print(future.result())

as_completed yields results as they finish, so short tests report early.

Seeds and isolation

A random test is only useful if a failure can be reproduced. Pass the seed explicitly (+seed=N, read with $value$plusargs in the testbench), record it in every result, and print the exact command needed to rerun a failure. Give every run its own folder so parallel jobs never write the same waveform or log file.

Exit codes for CI

Status	Meaning
0	Every run passed
1	At least one run failed or timed out
2	The runner itself could not start (bad options, compile failure)

Worked example

regress.py compiles each test once with Icarus Verilog, then runs every test with every seed in parallel.

#!/usr/bin/env python3
"""Compile and run a seeded Icarus Verilog regression in parallel."""
import argparse
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from pathlib import Path

BUILD = Path('build')


@dataclass
class Job:
    test: str
    seed: int


def compile_test(test: str, sources: list[Path]) -> Path:
    vvp = BUILD / f'{test}.vvp'
    subprocess.run(
        ['iverilog', '-g2012', '-Wall', '-s', test, '-o', str(vvp), *map(str, sources)],
        check=True, capture_output=True, text=True, timeout=300,
    )
    return vvp


def run_one(job: Job, timeout: int) -> tuple[Job, str, Path]:
    run_dir = BUILD / f'{job.test}_s{job.seed}'
    run_dir.mkdir(parents=True, exist_ok=True)
    log = run_dir / 'sim.log'
    vvp = (BUILD / f'{job.test}.vvp').resolve()
    try:
        proc = subprocess.run(['vvp', '-n', str(vvp), f'+seed={job.seed}'],
                              cwd=run_dir, capture_output=True, text=True, timeout=timeout)
        output = proc.stdout + proc.stderr
        passed = 'TEST PASSED' in output and 'ERROR' not in output
        status = 'PASS' if passed else 'FAIL'
    except subprocess.TimeoutExpired:
        output, status = f'timeout after {timeout} s\n', 'TIMEOUT'
    log.write_text(output)
    return job, status, log


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument('tests', nargs='+', help='testbench top names, e.g. tb_fifo')
    parser.add_argument('--rtl', type=Path, default=Path('rtl'))
    parser.add_argument('--seeds', type=int, nargs='+', default=[1, 2, 3])
    parser.add_argument('-j', '--jobs', type=int, default=4)
    parser.add_argument('--timeout', type=int, default=120)
    args = parser.parse_args()

    BUILD.mkdir(exist_ok=True)
    rtl = sorted(args.rtl.glob('*.v'))
    try:
        for test in args.tests:
            compile_test(test, rtl + [Path('tb') / f'{test}.v'])
    except subprocess.CalledProcessError as exc:
        print(f'compile failed:\n{exc.stdout}{exc.stderr}', file=sys.stderr)
        return 2

    jobs = [Job(t, s) for t in args.tests for s in args.seeds]
    failed = []
    with ThreadPoolExecutor(max_workers=args.jobs) as pool:
        futures = [pool.submit(run_one, job, args.timeout) for job in jobs]
        for future in as_completed(futures):
            job, status, log = future.result()
            print(f'{status:7} {job.test} seed={job.seed}')
            if status != 'PASS':
                failed.append((job, log))

    print(f'{len(jobs) - len(failed)}/{len(jobs)} passed')
    for job, log in failed:
        print(f'  rerun: vvp -n build/{job.test}.vvp +seed={job.seed}   (log: {log})')
    return 1 if failed else 0


if __name__ == '__main__':
    raise SystemExit(main())

Run it:

python3 regress.py tb_counter tb_fifo --seeds 1 2 3 4 -j 8
echo "status=$?"

subprocess.run with check=True raises CalledProcessError for a compile error; main() catches it, prints the compiler output and returns 2. A hung simulation becomes TIMEOUT instead of blocking the whole regression. Each run writes into build/<test>_s<seed>/, so parallel runs cannot overwrite each other, and the printed rerun: line reproduces any failure exactly.

Check yourself

Why pass a list to subprocess.run instead of a single command string? Why is check=True used for compilation but not for simulation? What happens to a run that exceeds the timeout? Why are threads acceptable here even though Python has a global interpreter lock? What makes a failing random-seed run reproducible?

Engineering practice

Practice: A Python regression runner

Task→Run / reason→Check result→Explain why

Add a --seeds-random N option that draws N seeds from random.SystemRandom() and prints them first, so a run with random seeds can still be repeated. Then write build/results.json with one object per run (test, seed, status, log path, runtime from time.perf_counter()), and make the script exit 1 if more than 10 % of runs fail with TIMEOUT even when the rest pass.

Practice work is non-certified and is not stored by ASIC Campus.