roshworldwide/holdout eval gate
Run an LLM eval on this PR and fail only when quality regressed beyond statistical noise. Posts the full comparison (effects, confidence intervals, corrected p-values) as a PR comment.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| eval | Eval reference: a .jsonl file or Python 'module:attr' | yes | — |
| target | Target reference: 'ollama:M' | 'openai:M' | 'anthropic:M' | 'module:attr' | yes | — |
| baseline | Baseline run id (or unambiguous prefix) present in the store | yes | — |
| store | Run store directory (commit your baseline artifact inside it) | no | .holdout |
| scorer | Scorer for .jsonl evals: exact | exact-strict | regex:<pattern> | no | exact |
| alpha | Significance level applied to corrected p-values | no | 0.05 |
| correction | Multiple-comparison correction: benjamini-hochberg | holm | none | no | benjamini-hochberg |
| seed | Seed for generation and resampling | no | 0 |
| python-version | Python version to set up | no | 3.12 |
| install | pip spec for holdout ('.' to install from this repo checkout) | no | holdout |
| comment | Post the comparison as a PR comment | no | true |
| github-token | Token used to post the PR comment | no | ${{ github.token }} |
Outputs
| name | description |
|---|---|
| run-id | Run id of the candidate evaluation |
| verdict | improved | regressed | no_significant_change | insufficient_data |