mclenhard/MCP Tool Evaluator
Run MCP tool evaluations
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| evals_path | Path to the evals file | yes | evals.ts |
| openai_api_key | OpenAI API key | yes | — |
| model | Model to use for evaluation | no | gpt-4 |
| timeout | Timeout in milliseconds for each tool call | no | 5000 |
| server_path | Path to the MCP server code (relative to repository root) | no | . |
Outputs
| name | description |
|---|---|
| accuracy | Accuracy score from 0-100 |
| completeness | Completeness score from 0-100 |
| relevance | Relevance score from 0-100 |
| clarity | Clarity score from 0-100 |
| reasoning | Reasoning score from 0-100 |
| overall_comments | Detailed comments about the evaluation |