kareem-rashed/Rubric Eval
Run LLM & agent evals with Rubric, compare against a baseline, and post a regression diff as a PR comment. Codecov for agent behavior.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| eval-file | Python eval file to run (e.g. evals/regression.py) | yes | — |
| baseline | Path to the baseline report JSON committed in the repo | no | evals/baseline.json |
| python-version | Python version to set up | no | 3.11 |
| install | Install command (override to add your project's dependencies) | no | pip install rubric-eval |
| comment | Post/update the regression diff as a PR comment | no | true |
| fail-on-regression | Fail the job if any baseline-passing test now fails | no | true |
| score-drop-threshold | Flag passing tests whose score dropped more than this | no | 0.1 |
| github-token | Token used to post the PR comment | no | ${{ github.token }} |
Outputs
| name | description |
|---|---|
| report | Path to the current run's JSON report |