langfuse/Langfuse Experiment
Run a Langfuse experiment in CI, gate on regressions, and comment results on the PR.
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Stale
- License
- None
Inputs
| name | description | required | default |
|---|---|---|---|
| langfuse_public_key | Langfuse public API key. | yes | — |
| langfuse_secret_key | Langfuse secret API key. | yes | — |
| langfuse_base_url | Langfuse base URL. | no | https://cloud.langfuse.com |
| experiment_path | Path to an experiment script (.py/.ts/.js/.mjs), a directory, or a glob pattern. | yes | — |
| dataset_name | Dataset to run the experiment against. If omitted, the user script is expected to select its own dataset. | no | — |
| dataset_version | Pin experiment to a specific dataset version. | no | — |
| experiment_metadata | Additional metadata to attach to the experiment run. Multiline, one `key=value` per line. Shown under the Metadata column in the Langfuse UI. | no | — |
| should_fail_on_regression | Fail the CI job when an experiment raises `RegressionError`. | no | true |
| should_fail_on_script_error | Fail the CI job when an experiment script crashes or raises a non-regression error. | no | true |
| should_comment_on_pr | Post the experiment report as a PR comment. | no | true |
| python_sdk_version | Version of the Langfuse Python SDK to install via pip (for `.py` experiments). Use a specific version string or `latest`. | no | 4.6.0b1 |
| js_sdk_version | Version of the Langfuse JS SDK (`@langfuse/client`) to install via npm (for `.ts` / `.js` / `.mjs` experiments). Use a specific version string or `latest`. | no | 5.3.0-beta.0 |
| should_skip_sdk_installation | Skip installing the Langfuse SDK entirely. Use this when you manage the Python/Node environment yourself (e.g. with your own `pip install` / `npm install` step before this action) — the action then trusts the ambient environment to provide `langfuse` / `@langfuse/client` + `tsx`. | no | false |
| github_token | Workflow GitHub token. Used to (1) post the PR comment — needs `pull-requests: write`, and (2) resolve the current job via the REST API so the comment links to this specific job and parallel matrix legs get their own comment sections — needs `actions: read`, otherwise falls back to the workflow-run URL and the job key from `$GITHUB_JOB`. Leave blank to skip the PR comment and the job lookup. See the README for a usage example. | no | "" |
Outputs
| name | description |
|---|---|
| result_json | Versioned JSON envelope with action metadata and normalized experiment results. See README for the schema. |
| failed | 'true' if any experiment script errored or raised a regression, else 'false'. |