tomerhakak/AgentProbe — AI Agent Testing

Run AgentProbe tests on your AI agent codebase. Posts pass/fail results, cost summaries, and assertion details as PR comments.

Trust Signals

name	description	required	default
test-dir	Directory containing AgentProbe test files	no	tests/
assertions	Comma-separated list of assertions to run (e.g. "cost_under,latency_under,no_hallucination"). Leave empty to run all.	no	""
cost-limit	Maximum total cost per run in USD. The action fails if this budget is exceeded.	no	5.0
fail-on-warning	Whether to fail the action when tests produce warnings (true/false).	no	false
model	Which model to test against (e.g. "gpt-4o", "claude-sonnet-4-6"). Uses the project default when empty.	no	""
github-token	GitHub token for posting PR comments. Defaults to the built-in GITHUB_TOKEN.	no	${{ github.token }}
agentprobe-version	Pin a specific AgentProbe version (e.g. "0.2.0"). Leave empty for latest.	no	""
python-version	Python version to use.	no	3.11
extra-dependencies	Additional pip packages to install (space-separated).	no	""

name	description
result	"pass" or "fail"
total-tests	Total number of tests executed
passed-tests	Number of tests that passed
failed-tests	Number of tests that failed
pass-rate	Pass rate as a percentage (e.g. "95.0")
total-cost	Total cost in USD (e.g. "0.042300")
total-duration-ms	Total duration in milliseconds
exit-code	Exit code of the test run (0 = success)