invincible-jha/AumOS Agent Evaluation
Run agent evaluations and benchmarks in your CI pipeline with multi-run statistical analysis
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| benchmark-path | Path to benchmark YAML file | yes | — |
| eval-runs | Number of evaluation runs for statistical significance | no | 5 |
| python-version | Python version | no | 3.12 |
| fail-threshold | Minimum score threshold to pass (0.0-1.0) | no | 0.7 |
| output-format | Report format: json, markdown, html | no | markdown |
| dimensions | Comma-separated evaluation dimensions: accuracy,safety,consistency,cost | no | accuracy,safety,consistency,cost |
Outputs
| name | description |
|---|---|
| overall-score | Overall evaluation score |
| report-path | Path to evaluation report |
| passed | Whether the evaluation passed the threshold |