datalayer/datalayer-evals

Run Datalayer eval reports from CI using the agent-runtimes Python API.

View on GitHub

Trust Signals

Scorecard Score: not yet scored
Maintenance Recency: Activelast commit Jul 7, 2026
License: BSD

Pinned Snippet

workflow.ymlSHA-pinned

uses: datalayer/github-actions@14ed6220d0ccaa9d67d3306f25c6c6e4c6615973 # no releases — HEAD as of 2026-07-11

tags can be moved; commit SHAs can't. why a SHA?

Inputs

name	description	required	default
mode	Action mode: run-report, execute-runs, or prepare-spec.	no	run-report
evalset-id	Primary evalset ID to report.	no	""
evalset-spec-file	Path to a primary evalset spec JSON file. If set, the action creates the evalset before reporting.	no	""
secondary-evalset-id	Optional secondary evalset ID to report and compare against the primary report.	no	""
secondary-evalset-spec-file	Optional path to a secondary evalset spec JSON file.	no	""
api-key	Datalayer API key.	yes	""
ai-agents-url	Optional AI Agents base URL override.	no	""
billing-entity-uid	Optional billing entity UID context used for eval operations. May be a user, organization, or team UID. Teams carry their own credits, so a team UID scopes evalset ownership and credit consumption to that team. Takes precedence over account-uid.	no	""
account-uid	Optional account UID context used for eval operations. May be a user, organization, or team UID. Used only when billing-entity-uid is empty.	no	""
run-environment	Optional run environment label passed through to the action runtime.	no	sdk
request-timeout-seconds	Per-request timeout (seconds) for a single agent chat call during execute-runs. A hung agent call is aborted after this window, the case is marked failed, execution continues, and cloud runtimes are always torn down. Defaults to 180 (3 minutes per call).	no	180
prepared-spec-output-dir	Output directory used by mode=prepare-spec.	no	artifacts/specs
prepared-spec-output-file	Optional filename override used by mode=prepare-spec.	no	""
run-limit	Runs fetched per experiment.	no	50
output-markdown	Output markdown report file path.	no	evals-report.md
secondary-output-markdown	Optional secondary markdown report output path.	no	""
comparison-summary-output	Optional comparison summary markdown output path.	no	""
export-csv	Export report.csv from agent-runtimes evals report --export.	no	true
iam-url	Optional IAM base URL override used by agent-runtimes agents create.	no	""
runtimes-url	Optional Runtimes base URL override used by agent-runtimes agents create.	no	""
agentspec-ids	Optional comma-separated agent spec ids used by execute-runs mode.	no	""
agent-environment-name	Environment name used by execute-runs mode for cloud runtime execution.	no	ai-agents-env
execution-target	Runtime execution target for execute-runs mode (cloud or local).	no	cloud
auto-start-local-agent-runtime	When execution-target=local, automatically start a local agent-runtimes server if none is reachable at local-agent-base-url.	no	false
local-agent-base-url	Local agent-runtimes base URL used when execution-target=local.	no	http://127.0.0.1:8765
local-agent-name	Local agent id used when execution-target=local.	no	default
upload-report-artifacts	Upload generated markdown/csv/log artifacts in a final step.	no	true
report-artifact-name	Name used by actions/upload-artifact for report artifacts.	no	datalayer-evals-reports

Outputs

name	description
prepared-spec-path	Path to prepared lane-specific evalset spec generated by mode=prepare-spec.
spec-path	Alias for prepared-spec-path.
report-file	Path to generated markdown report.
evalset-id	Evalset id used for the primary report (created from a spec or provided directly).
executed-evalset-id	Evalset id created in mode=execute-runs (empty otherwise).
csv-file	Path to generated CSV report (empty when export-csv=false).
log-file	Path to generated structured report JSON log.
timestamped-report-file	Timestamped markdown path generated by --export.
timestamped-csv-file	Timestamped CSV path generated by --export.
secondary-report-file	Secondary markdown report file path (empty when secondary input is omitted).
secondary-csv-file	Secondary CSV report file path.
secondary-log-file	Secondary structured report JSON log file path.
secondary-timestamped-report-file	Secondary timestamped markdown path generated by --export.
secondary-timestamped-csv-file	Secondary timestamped CSV path generated by --export.
comparison-summary-file	Comparison summary markdown file path.
failed-run-count	Total number of failed runs across primary and secondary reports.
primary-failed-run-count	Number of failed runs in the primary report.
secondary-failed-run-count	Number of failed runs in the secondary report (0 when no secondary report).