basalt-ai/Cobalt Eval
Run Cobalt AI agent experiments and post rich PR comments with results, comparisons, and AI analysis
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Activelast commit Feb 19, 2026
- License
- MIT
Pinned Snippet
uses: basalt-ai/cobalt@802de59b8615f4622c6abd2b0626f8cf948a3692 # v0.3.0tags can be moved; commit SHAs can't. why a SHA?
Inputs
| name | description | required | default |
|---|---|---|---|
| experiment_files | Experiment file(s) to run (glob pattern or comma-separated). Uses cobalt.config defaults if empty. | no | "" |
| working_directory | Working directory for running experiments | no | . |
| filter | Filter experiments by name (case-insensitive substring match on filenames). Maps to CLI --filter flag. | no | "" |
| concurrency | Override concurrency setting (number of parallel experiment items). Maps to CLI --concurrency flag. | no | "" |
| ci | Enable CI mode with threshold validation | no | false |
| api_key | API key for LLM judges and AI summary. Supports both OpenAI and Anthropic (auto-detected from key prefix). | no | "" |
| ai_summary | Generate an AI-powered analysis of results (requires api_key) | no | false |
| github_token | GitHub token for posting PR comments | no | ${{ github.token }} |
| comment_on_pr | Whether to post a comment on the PR | no | true |
| package_manager | Package manager to use: npm, pnpm, yarn, or auto (detect from lockfile) | no | auto |
| install_deps | Whether to install dependencies before running experiments | no | true |
| step_key | Unique key for comment deduplication. Change only if running multiple Cobalt steps in one workflow. | no | "" |
Outputs
| name | description |
|---|---|
| passed | Whether all experiments passed CI thresholds (true/false) |
| total_experiments | Number of experiments that ran |
| summary_json | JSON string of all experiment summaries |