beyhangl/Evalcraft CI Gate
Run evalcraft agent tests and gate pull requests on agent behavior. Installs evalcraft, runs your pytest suite, evaluates cassette metrics against thresholds, and posts a results table as a PR comment.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Activelast commit Jul 5, 2026
- License
- MIT
Pinned Snippet
uses: beyhangl/evalcraft@986027fd492c916888f90f348777bf539ddd639e # v0.5.0tags can be moved; commit SHAs can't. why a SHA?
Inputs
| name | description | required | default |
|---|---|---|---|
| test-path | Path to agent test files or directory | no | tests/ |
| max-cost | Maximum total cost in USD across all cassettes. Fails the action if exceeded. Example: "0.10" | no | "" |
| max-regression | Maximum allowed increase (%) for any cassette metric (tokens, cost) compared to the baseline cassettes committed in git. Requires record-mode=all or new so cassettes are re-recorded. Example: "5" or "5%" | no | "" |
| cassette-dir | Directory where cassettes are stored and loaded from | no | tests/cassettes |
| record-mode | Cassette record mode passed to --evalcraft-record: none — replay-only, skip tests with missing cassettes (default) new — record cassettes that don't exist yet all — always re-record (live LLM calls required) | no | none |
| evalcraft-version | Evalcraft package version to install. Use "latest" for the most recent release or pin to a specific version (e.g. "0.1.0"). | no | latest |
| python-version | Python version to use when installing and running evalcraft | no | 3.11 |
| extra-pytest-args | Additional arguments forwarded to pytest verbatim | no | "" |
| post-comment | Post a PR comment with the results table (true/false) | no | true |
| github-token | GitHub token used to post PR comments. Defaults to the auto-provided GITHUB_TOKEN secret. | no | ${{ github.token }} |
Outputs
| name | description |
|---|---|
| total-cost | Total cost in USD across all agent test cassettes |
| total-tokens | Total token count across all agent test cassettes |
| total-tool-calls | Total tool call count across all agent test cassettes |
| cassette-count | Number of cassette files evaluated |
| passed | Whether all tests and thresholds passed ("true" / "false") |