mikeleppane/typebench A/B

A/B-measure a type-checker PR build vs its latest release on your projects (wall-time).

Trust Signals

Scorecard Score: not yet scored
Maintenance Recency: Activelast commit Jun 20, 2026
License: MIT

Pinned Snippet

workflow.ymlSHA-pinned

uses: mikeleppane/typebench@8a1d04f3b36c36d845071743fdc03277fa1ed588 # v1.1.0

tags can be moved; commit SHAs can't. why a SHA?

Inputs

name	description	required	default
checker	Adapter id: pyrefly \| ty \| mypy \| pyright (stub is test-only).	yes	—
candidate	Path to a prebuilt candidate binary (use this OR build).	no	""
build	Shell command that builds the candidate (e.g. cargo build --release).	no	""
binary	Path where `build` puts the candidate (required with build).	no	""
baseline	Baseline: 'latest', '==X.Y' (uv), or 'path:/abs/binary' (offline).	no	latest
targets	Newline-delimited project paths and/or repo URLs to type-check.	yes	—
runs	hyperfine timed runs.	no	5
thread-mode	all-cores \| constrained.	no	all-cores
cores	Constrained-track core pin (used only with thread-mode: constrained).	no	1
comment	Post/update a sticky PR comment with the wall delta table.	no	true
python-version	Python for the action's own typebench runtime.	no	3.12

Outputs

name	description
envelope-path	Path to the written A/B results envelope JSON.