lukecarr/Litmus Test Runner
Run Litmus specification tests against LLM models and annotate failures inline.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Activelast commit Jun 5, 2026
- License
- MIT
Pinned Snippet
uses: lukecarr/litmus@e59ae24a2a03cf61a60075400a9fe3e15b4468da # v0.3.3tags can be moved; commit SHAs can't. why a SHA?
Inputs
| name | description | required | default |
|---|---|---|---|
| tests | Path to the test cases JSON file. | yes | — |
| schema | Path to the JSON schema file. | yes | — |
| model | Model(s) to test against, one per line or comma-separated (e.g. openai/gpt-4o). | yes | — |
| prompt | System prompt for the LLM. Mutually exclusive with prompt-file. | no | "" |
| prompt-file | Path to a file containing the system prompt. | no | "" |
| parallel | Number of parallel requests per model. | no | 1 |
| output | Output format: terminal, json, html, or github. | no | github |
| provider | LLM provider: openrouter or cloudflare. | no | openrouter |
| api-key | Provider API key. For openrouter this is the OpenRouter key; for cloudflare it is the downstream provider key. Passed to litmus through the environment, never on the command line. Leave empty to use a key already set in the job's environment. | no | "" |
| cf-account-id | Cloudflare account ID (used with provider=cloudflare). | no | "" |
| cf-gateway | Cloudflare AI Gateway ID (used with provider=cloudflare). | no | "" |
| cf-token | Cloudflare AI Gateway token for authenticated gateways. | no | "" |
| version | Litmus release to download, e.g. v0.3.0. Defaults to the release matching the action ref you pinned (an exact vX.Y.Z tag), or the latest release when the ref is a branch, SHA, or major-version tag. | no | "" |
| working-directory | Directory to run litmus from. | no | . |
Outputs
no outputs