agentkitai/AgentEval

Run an AgentEval suite as a CI gate — fail the build when the agent pass rate or regression breaches your thresholds.

Trust Signals

workflow.ymlSHA-pinned

uses: agentkitai/agenteval@48d54c97915b20f2c182d12e47d88e6a0b09f3df # v0.7.0

tags can be moved; commit SHAs can't. why a SHA?

name	description	required	default
suite	Path to the eval suite file (YAML/JSON).	yes	—
agent	Agent callable as 'module:func'.	yes	—
min-pass-rate	Minimum pass rate (0–1) to pass the gate.	no	0.8
max-regression	Maximum allowed regression percentage vs the baseline.	no	10
baseline	Baseline run ID for regression detection (optional).	no	""
format	Output format: text \| json \| junit.	no	text
output	Write the report to this file (optional).	no	""
parallel	Max concurrent cases.	no	1
version	agenteval version to install from PyPI (e.g. '1.2.3'); empty = latest.	no	""
python-version	Python version to set up.	no	3.12

no outputs