rbumg/evalgate
Eval regression gate: run golden-dataset evals against an OpenAI-compatible endpoint, compare to a committed baseline, and fail the job on regression. Writes a markdown report to the job step summary.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| suites | Suite file or directory, relative to the workspace. | no | evals |
| baseline | Path to the committed baseline file. | no | baseline.json |
| endpoint | Base URL of the OpenAI-compatible endpoint. | no | https://api.openai.com/v1 |
| model | Model to evaluate. | yes | — |
| api-key | API key for the endpoint. Pass a repository secret. | yes | — |
| judge-endpoint | Base URL for the judge model. Defaults to endpoint. | no | "" |
| judge-model | Judge model for llm_judge checks. Defaults to model. | no | "" |
| judge-api-key | API key for the judge endpoint. Defaults to api-key. | no | "" |
| tolerance | Allowed aggregate score drop before failing. | no | 0.01 |
| repeat | Runs per case, aggregated to damp flakiness. | no | 1 |
| extra-args | Extra flags passed verbatim to "evalgate run". | no | "" |
Outputs
no outputs