dileepkpandiya/skilleval
Measure whether a Claude SKILL.md file improves model outputs using a blind A/B test and LLM judge.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| skill-path | Path to directory containing SKILL.md | yes | — |
| tasks | Path to tasks YAML file | no | "" |
| fail-below | Fail if skill effectiveness is below this threshold | no | "" |
| fail-if-hurt-pct | Fail if percentage of hurt tasks exceeds this value | no | "" |
| judge-provider | Judge provider - gemini-flash | claude | openai | no | gemini-flash |
| runs | Number of independent A/B runs per task | no | 1 |
| anthropic-api-key | Anthropic API key for the runner | yes | — |
| gemini-api-key | Gemini API key (required if judge-provider is gemini-flash) | no | "" |
| openai-api-key | OpenAI API key (required if judge-provider is openai) | no | "" |
Outputs
| name | description |
|---|---|
| avg-diff | Average effectiveness score diff |
| tasks-improved | Number of tasks improved |
| tasks-hurt | Number of tasks hurt |
| confidence | Overall confidence rating |