armanzareian/AgentCompat

Replay LLM tool-call traces against candidate schemas in GitHub Actions.

Trust Signals

workflow.ymlSHA-pinned

uses: armanzareian/agentcompat@223788566f875d79622cee0406f302978e947cd3 # no releases — HEAD as of 2026-07-10

tags can be moved; commit SHAs can't. why a SHA?

name	description	required	default
config	Optional JSON policy file, usually .agentcompat.json.	no	""
baseline	Baseline tool bundle path.	no	""
candidate	Candidate tool bundle path. May be discovered from policy globs.	no	""
traces	Trace JSONL path.	no	""
trace-format	Trace format: canonical, openai, anthropic, mcp, or langchain.	no	""
fail-under	Minimum compatibility score required for success.	no	""
max-traces	Maximum trace records to replay.	no	""
sample-size	Optional deterministic weighted stratified sample size.	no	""
sample-seed	Seed for deterministic sampling when sample-size is set.	no	""
bootstrap-iterations	Optional deterministic bootstrap iteration count for score confidence intervals.	no	""
confidence-level	Confidence level for bootstrap score intervals, such as 0.95.	no	""
score-tolerance	Optional reporting tolerance for sampled score comparisons.	no	""
redact-paths	Newline-separated exact argument JSON paths to redact before replay.	no	""
redact-key-patterns	Newline-separated regular expressions matching argument keys to redact.	no	""
report-json	Path for the machine-readable AgentCompat JSON report.	no	agentcompat-report.json
sarif	Path for the SARIF report.	no	agentcompat.sarif

name	description
score	Compatibility score formatted with two decimal places.
passed	Number of eligible calls that remain candidate-compatible.
broken	Number of eligible calls that break under the candidate schema.
excluded	Number of baseline-invalid or unknown-tool calls excluded from the score.
score-lower	Lower bootstrap confidence bound when enabled.
score-upper	Upper bootstrap confidence bound when enabled.
report-json	Path to the JSON report.
sarif	Path to the SARIF report.