greynewell/SWE-bench Pro Evaluation
Run SWE-bench Pro preflight validation and agent evaluation using mcpbr
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| mode | Operation mode: 'preflight' (validate golden patches) or 'evaluate' (run agent evaluation) | no | preflight |
| benchmark | Benchmark name (e.g., swe-bench-pro, swe-bench-lite) | no | swe-bench-pro |
| sample-size | Number of instances to evaluate (empty = all) | no | "" |
| task-ids | Comma-separated task instance IDs to run | no | "" |
| filter-category | Filter by language or repository substring | no | "" |
| max-concurrent | Maximum concurrent Docker containers | no | 2 |
| timeout | Per-test timeout in seconds | no | 300 |
| fail-fast | Stop on first failure | no | false |
| config | Path to mcpbr YAML config file (evaluate mode) | no | "" |
| anthropic-api-key | Anthropic API key (evaluate mode) | no | "" |
| model | Model override (e.g., claude-sonnet-4-5-20250514) | no | "" |
| output-format | Comma-separated output formats: json, junit, markdown, html | no | json,junit |
| mcpbr-version | mcpbr version to install (e.g., 0.14.1, or empty for latest) | no | "" |
| shard-index | Shard index for parallel runs (0-based) | no | "" |
| shard-total | Total number of shards for parallel runs | no | "" |
Outputs
| name | description |
|---|---|
| results-path | Path to the results directory |
| total | Total number of instances evaluated |
| passed | Number of instances that passed |
| failed | Number of instances that failed |
| success-rate | Success rate as a percentage |