awesome-pro/CIGate eval gate
Eval-gated CI/CD for AI products. Runs CIGate over a sampled golden set, statistically corrects the LLM-judge score, and compares per failure-mode axis against the committed baseline. Emits a markdown report and a `regressed` output. (Comment-posting and the merge-blocking decision are left to the calling workflow for composability and fork safety.)
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| config | Path to evalconfig.yaml | — | evalconfig.yaml |
| baseline-path | Path to the baseline JSON to compare against | — | .cigate/baseline.json |
| full | Evaluate the full golden set instead of a sample | — | false |
| anthropic-api-key | Anthropic API key. If empty (and no OpenAI key), the gate runs in deterministic mock mode ($0). | — | "" |
| openai-api-key | OpenAI API key (optional; for OpenAI-backed generator/judge). | — | "" |
| editable | Install the local checkout (-e .) instead of the published 'cigate' from PyPI. Set 'true' only when gating CIGate's own repo. | — | false |
| python-version | — | — | 3.12 |
Outputs
| name | description |
|---|---|
| regressed | true if any axis regressed beyond tolerance |
| report-path | Path to the markdown report |
| summary-path | Path to the JSON summary |