kutanti/LitmusAI Agent Evaluation
Run AI agent evaluations in CI/CD. Compare against baselines, detect regressions, post PR comments.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| suite | Test suite name or path to YAML file | yes | — |
| agent | Agent module path (e.g. my_agent:agent) | yes | — |
| baseline | Path to baseline results JSON | no | .litmus/baseline.json |
| threshold | Minimum pass rate to succeed (0.0-1.0) | no | 0.7 |
| budget | Maximum total cost budget ($) | no | "" |
| concurrency | Max parallel evaluations | no | 5 |
| save-baseline | Save results as new baseline | no | false |
| runs | Number of evaluation runs for statistical analysis | no | 1 |
| log-dir | Directory to save result logs | no | "" |
| post-comment | Post results as PR comment | no | true |
| python-version | Python version to use | no | 3.11 |
Outputs
| name | description |
|---|---|
| pass-rate | Evaluation pass rate (0.0-1.0) |
| total-cost | Total evaluation cost ($) |
| passed | Number of passed tests |
| failed | Number of failed tests |
| has-regression | Whether regressions were detected |
| results-path | Path to results JSON file |