verifywise-ai/VerifyWise LLM Evaluation
Run LLM evaluations against a VerifyWise instance and gate CI on quality thresholds. Supports chatbot, RAG, and agent use cases.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| api_url | VerifyWise instance URL | yes | — |
| project_id | Project ID | yes | — |
| dataset_id | Dataset ID to evaluate against | yes | — |
| metrics | Comma-separated metric names (e.g. answer_relevancy,bias,toxicity) | yes | — |
| model_name | Model to evaluate (e.g. gpt-4o-mini, claude-3.5-sonnet) | yes | — |
| model_provider | Provider: openai | anthropic | google | mistral | xai | self-hosted | yes | — |
| judge_model | Judge LLM for metric scoring | no | gpt-4o |
| judge_provider | Judge LLM provider | no | openai |
| threshold | Pass/fail threshold (0.0–1.0) | no | 0.7 |
| timeout_minutes | Max wait time in minutes | no | 30 |
| poll_interval_seconds | Seconds between status polls | no | 15 |
| experiment_name | Custom experiment name (auto-generated if empty) | no | "" |
| fail_on_threshold | Fail the step when thresholds are not met | no | true |
| post_pr_comment | Post results as a comment on the pull request | no | true |
| vw_api_token | VerifyWise API token | yes | — |
| llm_api_key | API key for the model being evaluated | yes | — |
| judge_api_key | API key for the judge LLM (defaults to llm_api_key if not set) | no | "" |
Outputs
| name | description |
|---|---|
| passed | Whether all metrics passed (true/false) |
| results_path | Path to results JSON |
| summary_path | Path to Markdown summary |
| experiment_id | ID of the created experiment |