reflectt/CI Canary PR Gate

Run a canary/eval command, upload a JSON report artifact, optionally diff against a baseline, and (on PRs) post/update a summary comment.

Trust Signals

Scorecard Score: not yet scored
Maintenance Recency: Stale
License: None

Inputs

name	description	required	default
command	Shell command to run the canary/eval suite. Must produce the report JSON at report_path.	yes	—
report_path	Path to the canonical JSON report produced by the command.	no	eval-report.json
artifact_name	Name for the uploaded workflow artifact.	no	ci-canary-report
retention_days	Artifact retention days (passed to upload-artifact client).	no	14
comment_mode	create-or-update \| create \| off	no	create-or-update
comment_title	Title used in the PR comment heading.	no	CI Canary
github_token	GitHub token for PR comment + optional baseline artifact download.	no	${{ github.token }}
baseline_path	Path to a baseline JSON report in the repo.	no	""
baseline_artifact	Artifact name to download as baseline from a workflow run.	no	""
baseline_workflow	Workflow name, file name (e.g. canary.yml), or workflow id to locate baseline artifacts.	no	""
baseline_ref	Ref for baseline lookup when using baseline artifact.	no	refs/heads/main
fail_on_command_error	If true, fail when command exits non-zero.	no	true
fail_on_any_scenario_fail	If true, fail when any scenario has passed=false.	no	true
min_score	If set, fail when report.summary.score < min_score.	no	""
max_score_drop	If set and baseline exists, fail when (baseline.score - score) > max_score_drop.	no	""
max_metric_regressions_json	JSON map of metric thresholds (fail if metric delta vs baseline > threshold). Example: {"latency_p95_ms":500, "cost_usd":0.1}	no	""
dashboard_url	Optional dashboard link to include in the PR comment.	no	""
health_alerts_url	Optional health alerts link to include in the PR comment.	no	""

Outputs

name	description
passed	true\|false
score	Numeric score as string
score_delta	Numeric score delta vs baseline as string (empty if no baseline)
failed_scenarios	Count of failed scenarios as string
report_path	Resolved report path
artifact_name	Artifact name used