unshdee/proofrag
Evaluate a RAG/LLM app and gate CI on answer quality + retrieval regressions.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| goldenset | Path to the golden set JSONL. | yes | — |
| predictions | Path to predictions JSONL ({id, answer, retrieved_contexts}) from your RAG. | yes | — |
| results | Where to write results.json. | — | results.json |
| scorecard | Where to write the HTML scorecard. | — | scorecard.html |
| fail-under | Fail if overall generation score drops below this (0-1). Empty = no absolute gate. | — | "" |
| baseline | Baseline results.json to diff against. Empty = no regression check. | — | "" |
| tolerance | Allowed metric drop vs baseline before it counts as a regression. | — | 0.02 |
| k | Cutoff k for retrieval metrics (Recall@k, NDCG@k, ...). | — | 5 |
| semantic | Use embedding cosine for chunk relevance (needs the openai extra + key). | — | false |
| extra | proofrag extra to install: anthropic or openai. | — | anthropic |
| version | Pin a proofrag version (e.g. 0.3.0). Empty = latest. | — | "" |
| summary | Write a markdown score summary to the GitHub Actions job summary. | — | true |
| upload-artifact | Upload the results JSON and HTML scorecard as a workflow artifact. | — | true |
| artifact-name | Name for the uploaded scorecard/results artifact. | — | proofrag-scorecard |
Outputs
| name | description |
|---|---|
| results | Path to the written results.json. |
| scorecard | Path to the written HTML scorecard. |