mila-iqia/DocMetrics
Evaluate documentation quality and post a score comparison comment on pull requests.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Activelast commit Apr 22, 2026
- License
- None
Pinned Snippet
uses: mila-iqia/docmetrics@dd349bc1b4ffdb0f1b654f8abe9e07d7f6e8057e # v0.0.8tags can be moved; commit SHAs can't. why a SHA?
Inputs
| name | description | required | default |
|---|---|---|---|
| questions-file | Path to the questions YAML file, relative to the caller's repository root. | yes | — |
| model | LLM model to use. | no | gemini-2.5-flash |
| docs-url | Space-separated list of documentation URLs to use as context for all questions. | no | — |
| base-docs-url | Documentation URL to use for base branch comparison. Defaults to docs-url if not provided. | no | — |
| gemini-api-key | Google Gemini API key. Required for Gemini models; omit when using 'test:dummy' or non-Gemini models. | no | — |
| ollama-url | Base URL of the Ollama server. Only used with 'ollama:' models. | no | http://localhost:11434 |
| github-token | GitHub token used to post the PR comment. | no | ${{ github.token }} |
| num-candidates | Number of times to ask each question and average the results. Higher values reduce variance at the cost of more LLM calls. When > 1, the PR comment shows a ± std deviation alongside each score. | no | 1 |
| post-comment | Whether to post a PR comment with the results. Set to 'false' when using the workflow_run pattern for fork PRs — upload the docmetrics-* workspace files as an artifact instead, then call the post-pr-comment reusable workflow from a workflow_run trigger. | no | true |
Outputs
| name | description |
|---|---|
| score-with-docs | Score when documentation URLs are provided as context (0.0-1.0). |
| score-without-docs | Score when no documentation context is provided (0.0-1.0). |