yotambraun/Toolscore - LLM Agent Evaluation

Catch LLM agent regressions before deployment. Test tool-calling accuracy for OpenAI, Anthropic, Gemini with pytest-like simplicity.

Trust Signals

name	description	required	default
gold-file	Path to gold standard JSON file containing expected tool calls	yes	—
trace-file	Path to agent trace JSON file to evaluate	yes	—
format	Trace format (auto, openai, anthropic, gemini, langchain, custom)	no	auto
threshold	Minimum accuracy threshold (0.0-1.0). Fails if selection_accuracy or invocation_accuracy falls below this.	no	0.9
baseline-file	Path to baseline JSON for regression testing. If provided, runs regression check instead of threshold check.	no	""
regression-threshold	Maximum allowed regression as decimal (e.g., 0.05 = 5%). Only used with baseline-file.	no	0.05
fail-on-regression	Fail the action if accuracy drops below threshold or regression is detected	no	true
generate-report	Generate HTML report artifact	no	true
python-version	Python version to use	no	3.11
mcp-command	MCP server command to test (e.g. "uvx my-mcp-server"). When set, the action runs `toolscore mcp test <command>` instead of the eval/regression path.	no	""
mcp-fail-under	Minimum MCP scorecard grade (A-F or 0.0-1.0). Only used with mcp-command.	no	B

name	description
selection-accuracy	Tool selection accuracy score (0.0-1.0)
invocation-accuracy	Tool invocation accuracy score (0.0-1.0)
argument-f1	Argument match F1 score (0.0-1.0)
passed	Whether all thresholds were met (true/false)
regression-detected	Whether a regression was detected (true/false). Only set when using baseline-file.