Python log parsing and reports
- Write regular expressions with Python's
remodule, including named groups - Parse simulator logs into structured records with
dataclasses - Group failures by their first error message
- Write a regression summary as CSV for spreadsheets and JSON for dashboards and CI
Explanation
From lines to records
grep answers "does this line exist?". A regression review needs more: which test failed, with which seed, at what simulation time, and do ten failures share one cause? Parse each log into a record once, then every report becomes a simple loop over records.
Regular expressions in Python
Compile a pattern once and reuse it. Write patterns as raw strings (r'...') so backslashes reach the regular-expression engine unchanged. Named groups make the code readable.
import re
ERROR_RE = re.compile(r'^ERROR(?: at (?P<time>\d+)\s*\w*)?:\s*(?P<msg>.*)$')
m = ERROR_RE.match('ERROR at 1250 ns: data mismatch exp=0x3c got=0x3d')
if m:
print(m['time'], m['msg']) # 1250 data mismatch exp=0x3c got=0x3d
| Method | Use |
|---|---|
pattern.match(line) |
Match only at the start of the string |
pattern.search(line) |
Find the first match anywhere |
pattern.fullmatch(text) |
The whole string must match |
pattern.findall(text) |
All matches as a list |
re.sub(pattern, repl, text) |
Replace matches |
(?:...) groups without capturing; \d+ is one or more digits; \s* is optional whitespace.
Dataclasses
A @dataclass declares a record type in a few lines and generates __init__, __repr__ and comparison methods. dataclasses.asdict() converts one to a dictionary for JSON.
Grouping failures
Error messages often contain data that differs between runs, such as values, times or seeds. Replace those with placeholders before grouping so exp=0x3c got=0x3d and exp=0x11 got=0x10 are recognised as the same bug:
signature = re.sub(r'0x[0-9a-fA-F]+|\b\d+\b', 'N', msg)
collections.Counter then counts how many tests hit each signature.
Output formats
| Format | Module | Audience |
|---|---|---|
| CSV | csv.DictWriter |
Spreadsheets, quick sorting by runtime |
| JSON | json.dump |
Dashboards, CI artifacts, other scripts |
| Text | print |
The engineer reading the terminal |
Open CSV files with newline='', as the csv module documentation requires, so rows are not separated by blank lines on some platforms.
Worked example
Each log in logs/ is named <test>_s<seed>.log. A passing run prints TEST PASSED; a failing check prints ERROR at <time> ns: <message>. Save this as summarise.py:
#!/usr/bin/env python3
"""Summarise simulation logs into CSV and JSON reports."""
import csv
import json
import re
import sys
from collections import Counter
from dataclasses import asdict, dataclass, field
from pathlib import Path
NAME_RE = re.compile(r'^(?P<test>.+)_s(?P<seed>\d+)$')
ERROR_RE = re.compile(r'^ERROR(?: at (?P<time>\d+)\s*\w*)?:\s*(?P<msg>.*)$')
@dataclass
class Result:
test: str
seed: int
status: str = 'FAIL'
errors: list[str] = field(default_factory=list)
@property
def signature(self) -> str:
if not self.errors:
return 'no PASS marker' if self.status == 'FAIL' else ''
return re.sub(r'0x[0-9a-fA-F]+|\b\d+\b', 'N', self.errors[0])
def parse_log(path: Path) -> Result:
name = NAME_RE.match(path.stem)
if not name:
raise ValueError(f'unexpected log name: {path.name}')
result = Result(test=name['test'], seed=int(name['seed']))
passed = False
for line in path.read_text(errors='replace').splitlines():
if m := ERROR_RE.match(line):
result.errors.append(m['msg'])
elif line.strip() == 'TEST PASSED':
passed = True
result.status = 'PASS' if passed and not result.errors else 'FAIL'
return result
def main() -> int:
log_dir = Path(sys.argv[1] if len(sys.argv) > 1 else 'logs')
results = [parse_log(p) for p in sorted(log_dir.glob('*.log'))]
if not results:
print(f'no logs in {log_dir}', file=sys.stderr)
return 2
with open('summary.csv', 'w', newline='') as handle:
writer = csv.DictWriter(handle, fieldnames=['test', 'seed', 'status', 'first_error'])
writer.writeheader()
for r in results:
writer.writerow({'test': r.test, 'seed': r.seed, 'status': r.status,
'first_error': r.errors[0] if r.errors else ''})
failures = [r for r in results if r.status == 'FAIL']
buckets = Counter(r.signature for r in failures)
report = {
'total': len(results),
'passed': len(results) - len(failures),
'failed': len(failures),
'buckets': [{'signature': s, 'count': n} for s, n in buckets.most_common()],
'results': [asdict(r) for r in results],
}
Path('summary.json').write_text(json.dumps(report, indent=2) + '\n')
print(f"passed {report['passed']}/{report['total']}")
for s, n in buckets.most_common():
print(f' {n:3d} {s}')
return 1 if failures else 0
if __name__ == '__main__':
raise SystemExit(main())
The walrus operator := assigns and tests the match in one step (Python 3.8+). A log with neither an error nor TEST PASSED fails with the signature no PASS marker, which usually means a crash, a timeout or a $fatal before the verdict. The exit status is 1 when anything failed, so CI can gate on the script directly while still keeping both report files as artifacts.
Check yourself
Why are patterns written as raw strings? What is the difference between re.match and re.search? Why replace numbers and hexadecimal values before counting failure signatures? Why is a log without TEST PASSED reported as a failure even when it has no ERROR lines? Why must CSV files be opened with newline=''?
Practice: Python log parsing and reports
Extend parse_log to read a final line such as Runtime: 12.4 s into a runtime_s: float field and add it to the CSV. Then add a --previous summary.json option that loads an earlier report and prints tests that passed before but fail now (new regressions) separately from tests that were already failing.
Practice work is non-certified and is not stored by ASIC Campus.