volter-ai/Runhuman QA Test

Run human QA tests on URLs, PRs, and issues with real testers

View on GitHub

Trust Signals

Scorecard Score
not yet scored
Maintenance Recency
Stale
License
None
namedescriptionrequireddefault
urlURL to test (must be publicly accessible)yes
descriptionAdditional test instructions for the testerno
pr-numbersPR numbers to find linked issues from (JSON array or comma-separated)no
issue-numbersIssue numbers to test directly (JSON array or comma-separated)no
include-linked-issuesWhen true (default), issues referenced in a PR body (e.g. "Closes #123") are pulled into the test plan. Set false to test the pull request alone.notrue
on-success-add-labelsLabels to add on test success (JSON array)no
on-success-remove-labelsLabels to remove on test success (JSON array)no
on-failure-add-labelsLabels to add on test failure (JSON array)no
on-failure-remove-labelsLabels to remove on test failure (JSON array)no
on-not-testable-add-labelsLabels to add when issue is not testable (JSON array)no
on-not-testable-remove-labelsLabels to remove when issue is not testable (JSON array)no
on-timeout-add-labelsLabels to add on test timeout (JSON array)no
on-timeout-remove-labelsLabels to remove on test timeout (JSON array)no
fail-on-errorFail the workflow on system errorsnotrue
fail-on-failureFail the workflow if any test fails. Requires a boolean "success" field in the output schema — without one, completed tests always count as passed.notrue
fail-on-timeoutFail the workflow if tester times outnofalse
wait-for-resultWait for test completion before finishing. Set to false for fire-and-forget mode (creates jobs and exits immediately).notrue
post-pr-commentWhen true, post a PR comment containing the tester narrative, AI-extracted bug list, and a link to the full session. Each run posts a new comment so the timeline is preserved across multiple commits. Requires pr-numbers, wait-for-result=true, and a github-token with pull-requests:write.nofalse
api-keyRunhuman API key or PAT (rh_..., rh_pat_...)yes
github-tokenGitHub token for API accessno${{ github.token }}
templateTemplate name to use (resolved from .runhuman/templates/, project, or builtins)no
template-filePath to local template file (e.g., .runhuman/templates/smoke.md)no
target-duration-minutesTarget test duration in minutes (1-60)no5
device-classDevice class: desktop or mobilenodesktop
output-schemaJSON schema for extracting structured data from test results (used with description-only tests)no
auto-create-github-issuesWhen set, enables (true) or disables (false) server-side auto-creation of GitHub issues from extracted QA findings using the detailed renderer (Expected / Actual / Where / Evidence / Related Telemetry). When unset, the project/template default applies.no
auto-create-github-issues-repoTarget repo (owner/repo) for auto-created GitHub issues. Only takes effect when auto-creation is enabled via this input or the project/template. Must be one of the project's linked repos.no
only-tester-surfaced-issuesWhen auto-creation is enabled and this is true, skip auto-filing issues the tester did not explicitly flag as a bug (telemetry-only findings). Defaults to false (file all extracted issues). See runhuman#3160.nofalse
requires-runhuman-apk-installINTERNAL (@volter.ai callers only): request the Runhuman dual-device self-test flow, routing to testers with the APK-install capability. Non-Volter callers are rejected with 403. Distinct from generic third-party APK-install jobs.nofalse
enable-code-contextPreview feature. When true, bug reports include references to the specific files and symbols in your codebase that are most likely related, so issues are easier to debug.nofalse
commit-shaCommit SHA the deployed app corresponds to (typically github.event.pull_request.head.sha). When provided alongside enable-code-context, code references reflect this commit's code instead of the default branch.no
commit-base-shaBase commit SHA the PR is merging into (typically github.event.pull_request.base.sha). When provided alongside commit-sha, code-context only re-indexes files actually changed in the PR rather than every file changed since the repo was last fully indexed.no
wait-for-code-contextWhen true, the test waits up to 90 seconds for code references to be prepared for the given commit-sha before scoring. When false (default), the test proceeds immediately and code references may reflect the default branch instead of this commit.nofalse
namedescription
statusTest status (completed, error, timeout, no-issues, created)
successWhether all tests passed (true/false)
tested-issuesJSON array of tested issue numbers
passed-issuesJSON array of passed issue numbers
failed-issuesJSON array of failed issue numbers
not-testable-issuesJSON array of not-testable issue numbers
timed-out-issuesJSON array of timed-out issue numbers
resultsFull JSON results object
cost-usdTotal cost in USD
duration-secondsTotal duration in seconds
job-idsJSON array of job IDs
job-urlsJSON array of job URLs
extracted-issuesJSON array of AI-extracted issues from test results, enriched with related issue detection