team2027/2027 Evals

Run a 2027.dev eval on a PR's preview deployment and post results back to the PR.

Trust Signals

Scorecard Score: not yet scored
Maintenance Recency: Stale
License: None

Inputs

name	description	required	default
api-key	2027.dev API key (create one at /<orgDomain>/settings)	yes	—
api-base-url	Base URL for the 2027 evals API	no	https://2027.dev/evals
prompt-id	Prompt UUID. List available IDs via GET /api/v1/prompts.	yes	—
url-map	JSON object mapping production hostnames to preview URLs (full http(s) URLs required), e.g. {"acme.com":"https://deploy-preview-1.netlify.app"}. Optional when template-vars carries the prompt's variables (e.g. CLI / non-URL evals); pass {} in that case.	no	{}
template-vars	JSON object of values for the prompt's declared template variables, e.g. {"cliInstall":"npm i -g https://pkg.pr.new/org/repo/@scope/cli@abc1234"}. Required when the prompt declares non-empty templateVars; otherwise omit.	no	""
deployment-url	Preview deployment URL. Required when url-map has more than one entry (validated at runtime).	no	—
wait-timeout-minutes	How long to poll for completion before returning (action exits 0 on timeout)	no	20
poll-interval-seconds	Seconds between status polls	no	20
timeout-fails	When 'true', a polling timeout marks the commit status as failure (blocks merge). Default 'false' marks it success so the check doesn't get stuck pending.	no	false
skip-comment	When 'true', the action does not post or update the sticky PR comment. Use this if you want to render your own comment from the action's outputs.	no	false
skip-status	When 'true', the action does not set the commit status check. Use this if you want to set your own status from the action's outputs.	no	false
github-token	GitHub token for posting PR comments and commit statuses (defaults to github.token)	no	${{ github.token }}

Outputs

name	description
run-id	UUID of the eval run on 2027
status-url	API endpoint that reflects the run's current state
final-status	Terminal state observed before the action exited — completed \| failed \| superseded \| running (the last on timeout)
prompt-title	Human-readable title of the evaluated prompt
report-slug	Report slug if the run produced a scorable report, empty string otherwise
report-url	Full URL to the dashboard report page, empty string if no report
failure-reason	Server-provided failure reason if the run failed, empty string otherwise
score	Final score (0-100) when the run produced a report, empty string otherwise
grade	Final letter grade when the run produced a report, empty string otherwise
baseline-score	Score of the most recent prior published report for the same prompt, empty string if no baseline
report-json	Full report object as stringified JSON (slug, url, score, grade, metrics, dimensions). Empty string if no report. Forward-compatible — picks up any new fields the API adds without an action release.
baseline-json	Baseline object as stringified JSON ({ score, grade }). Empty string if no baseline. Sourced from the most recent prior published run for the same prompt.