andrei-shtanakov/ATP Test
Run ATP agent tests and produce reports, summaries, and badges
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| suite_path | Path to the ATP test suite YAML file | yes | tests/suite.yaml |
| adapter | Agent adapter to use (e.g. cli, http, langraph) | no | "" |
| threshold | Minimum success rate (0.0-1.0). The action fails if the actual success rate falls below this value. | no | "" |
| budget_usd | Maximum allowed estimated cost in USD. The action fails if the estimated cost exceeds this budget. | no | "" |
| baseline_path | Path to a baseline JSON file for regression comparison. When set, the action runs baseline comparison and fails on detected regressions. | no | "" |
| comment_on_pr | Post a markdown summary as a PR comment (true/false) | no | false |
| python_version | Python version to use | no | 3.12 |
| extra_args | Additional arguments passed to `atp test` | no | "" |
Outputs
| name | description |
|---|---|
| success_rate | Overall success rate (0.0-1.0) |
| total_tests | Total number of tests executed |
| passed_tests | Number of passed tests |
| failed_tests | Number of failed tests |
| estimated_cost | Estimated cost in USD |
| junit_path | Path to the generated JUnit XML report |
| json_path | Path to the generated JSON report |
| summary_markdown | Markdown summary of the test results |
| badge_url | shields.io badge URL reflecting the test outcome |
| regression_detected | Whether a regression was detected (true/false) |