coval-ai/Coval Run Evaluation
Launch and monitor AI agent evaluation runs using the Coval v1 Runs API
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| agent_id | The agent ID to test (22 character identifier) | yes | — |
| persona_id | The simulated persona ID to use for testing (22 character identifier) | yes | — |
| test_set_id | The test set ID containing test cases to run (8 character identifier) | yes | — |
| metric_ids | Optional JSON array of metric IDs to evaluate (e.g., ["metricId1", "metricId2"]). If not provided, uses agent's default metrics. | no | — |
| iteration_count | Number of times to run each test case (1-10) | no | 1 |
| concurrency | Number of simulations to run concurrently (1-5) | no | 1 |
| metadata | Optional JSON object for custom metadata (e.g., {"campaign_id": "q4_2025", "created_by": "ci/cd"}) | no | — |
| max_wait_time | Maximum time to wait for run completion in seconds | no | 600 |
| check_interval | Interval between status checks in seconds | no | 30 |
| api_base_url | Base URL for the Coval API | no | https://api.coval.dev/v1 |
| fail_on_metric_id | Metric ID to check for failure conditions. If any simulation has the fail_on_metric_value for this metric, the action will exit with code 1. | no | — |
| fail_on_metric_value | Value that indicates failure for the specified metric (e.g., 'NO'). Case-insensitive comparison. | no | — |
Outputs
| name | description |
|---|---|
| run_id | The unique identifier of the launched run |
| status | Final status of the run (COMPLETED, FAILED, etc.) |
| run_url | URL to view the run in Coval dashboard |
| metric_failures | JSON array of simulation IDs that failed the metric check (empty if none) |