preset-io/testmcpy Eval Runner
Run LLM-driven evals against an MCP server with testmcpy: CI gate with exit codes, JUnit output, job-summary report, and optional sticky PR comment.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Activelast commit Jul 8, 2026
- License
- Apache 2.0
Pinned Snippet
uses: preset-io/testmcpy@15c4405735789a66bf2b4afb7605546f50abec4c # v0.11.0tags can be moved; commit SHAs can't. why a SHA?
Inputs
| name | description | required | default |
|---|---|---|---|
| test_path | Path to test files or directory | yes | tests/ |
| model | LLM model to use | no | "" |
| provider | LLM provider | no | "" |
| mcp_url | MCP service URL | no | "" |
| profile | MCP profile ID | no | "" |
| pass_threshold | Minimum pass rate to succeed (0-100) | no | 80 |
| gate_config | Path to a .testmcpy-gate.yaml (overrides pass_threshold) | no | "" |
| report_format | Report format: md, html, json | no | md |
| junit | Emit junit.xml ("true"/"false") | no | true |
| pr_comment | Post/update a sticky PR comment with the report ("true"/"false") | no | false |
| github_token | Token used for the PR comment | no | ${{ github.token }} |
| version | testmcpy version to install (PyPI version), e.g. 0.8.0 | no | "" |
| extra_args | Extra arguments appended to testmcpy run | no | "" |
| anthropic_api_key | Anthropic API key | no | "" |
| openai_api_key | OpenAI API key | no | "" |
| mcp_api_token | MCP JWT API token | no | "" |
| mcp_api_secret | MCP JWT API secret | no | "" |
| mcp_api_url | MCP JWT auth URL | no | "" |
Outputs
| name | description |
|---|---|
| pass_rate | Overall pass rate percentage |
| total_tests | Total number of tests run |
| passed | Number of tests passed |
| failed | Number of tests failed |
| gate_passed | Whether the run passed the gate ("true"/"false") |
| report_path | Path to generated report |
| results_path | Path to results JSON |