chris0jeky/llm-release-gate
Block unsafe LLM app changes: run baseline vs candidate configs over a golden dataset, compare quality/abstention/citations/schema/latency/tokens/cost, comment the summary on the PR, and fail the check on a material regression.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| dataset | Path to the golden dataset JSON | yes | — |
| baseline | Path to the baseline run-config JSON | yes | — |
| candidate | Path to the candidate run-config JSON | yes | — |
| scorers | Path to the scorer-config JSON | yes | — |
| thresholds | Path to the thresholds JSON | yes | — |
| pricing | Path to the pricing-table JSON (omit: cost reported unavailable) | no | "" |
| out | Output directory for reports and the run manifest | no | llm-release-gate-out |
| comment | Post/update the Markdown summary as a PR comment ('true'/'false') | no | true |
| github-token | Token used to post the PR comment | no | ${{ github.token }} |
| python-version | Python version to run the gate with | no | 3.12 |
Outputs
| name | description |
|---|---|
| verdict | 'pass' or 'fail' (empty if the gate could not run) |
| result-hash | Content hash identifying this comparison result |
| report-json | Path to report.json |
| report-md | Path to report.md |
| report-html | Path to report.html |