microsoft/promptpex
Generate tests for a LLM prompt using PromptPex. <details><summary>Prompt format</summary> PromptPex accepts prompts formatted in Markdown with a YAML frontmatter section (optional). ```text --- ... inputs: some_input: type: "string" --- system: This is your system prompt. user: This is your user prompt. {{some_input}} ``` - The content of the Markdown is the chat conversation. `system:` is the system prompt and `user:` is the user prompt. - The input variables are defined in the frontmatter of the prompt. - If not input variables are defined, PromptPex will append the generated test to the user prompt. ### Frontmatter You can override parts of the test generation process by providing values in the frontmatter of the prompt (all values are optional). ```markdown --- ... promptPex: inputSpec: "input constraints" outputRules: "output constraints" inverseOutputRules: "inverted output constraints" intent: "intent of the prompt" instructions: inputSpec: "Additional input specification instructions" outputRules: "Additional output rules instructions" inverseOutputRules: "Additional inverse output rules instructions" intent: "Additional intent of the prompt" --- ``` </details>
View on GitHubTrust Signals
- Scorecard Score
- not yet scored
- Maintenance Recency
- Activelast commit Feb 23, 2026
- License
- Other
Pinned Snippet
uses: microsoft/promptpex@0f5e0af4f0614e6271197a60d79bccdc27339509 # v0.0.18tags can be moved; commit SHAs can't. why a SHA?
Inputs
| name | description | required | default |
|---|---|---|---|
| prompt | Prompt template to analyze. You can either copy the prompty source here or upload a file prompt. [prompty](https://prompty.ai/) is a simple markdown-based format for prompts. prompt.yml is the GitHub Models format. | no | — |
| effort | Effort level for the test generation. This will influence the number of tests generated and the complexity of the tests. | no | — |
| out | Output folder for the generated files. This flag is mostly used when running promptpex from the CLI. | no | — |
| cache | Cache all LLM calls. This accelerates experimentation but you may miss issues due to LLM flakiness. | no | true |
| test_run_cache | Cache test run results in files. | no | true |
| eval_cache | Cache eval evaluation results in files. | no | — |
| evals | Evaluate the test results | no | true |
| tests_per_rule | Number of tests to generate per rule. By default, we generate 3 tests to cover each output rule. You can modify this parameter to control the number of tests generated. | no | 3 |
| split_rules | Split rules and inverse rules in separate prompts for test generation. | no | true |
| max_rules_per_test_generation | Maximum number of rules to use per test generation which influences the complexity of the generated tests. Increase this value to generate tests faster but potentially less complex tests. | no | 3 |
| test_generations | Number of times to amplify the test generation. This parameter allows to generate more tests for the same rules by repeatedly running the test generation process, while asking the LLM to avoid regenerating existing tests. | no | 2 |
| runs_per_test | Number of runs to execute per test. During the evaluation phase, this parameter allows to run the same test multiple times to check for consistency and reliability of the model's output. | no | 2 |
| disable_safety | Do not include safety system prompts and do not run safety content service. By default, system safety prompts are included in the prompt and the content is checked for safety. This option disables both. | no | false |
| rate_tests | Generate a report rating the quality of the test set. | no | false |
| rules_model | Model used to generate rules (you can also override the model alias 'rules') | no | — |
| baseline_model | Model used to generate baseline tests | no | — |
| models_under_test | List of models to run the prompt again; semi-colon separated | no | — |
| eval_model | List of models to use for test evaluation; semi-colon separated | no | — |
| eval_model_groundtruth | List of models to use for ground truth evaluation; semi-colon separated | no | — |
| compliance | Evaluate Test Result compliance | no | false |
| max_tests_to_run | Maximum number of tests to run | no | — |
| input_spec_instructions | These instructions will be added to the input specification generation prompt. | no | — |
| output_rules_instructions | These instructions will be added to the output rules generation prompt. | no | — |
| inverse_output_rules_instructions | These instructions will be added to the inverse output rules generation prompt. | no | — |
| test_expansion_instructions | These instructions will be added to the test expansion generation prompt. | no | — |
| store_completions | Store chat completions using [stored completions](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/stored-completions). | no | — |
| store_model | Model used to create [stored completions](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/stored-completions) (you can also override the model alias 'store'). | no | — |
| groundtruth | Generate groundtruth for the tests. This will generate a groundtruth output for each test run. | no | true |
| groundtruth_model | Model used to generate groundtruth | no | — |
| custom_metric | This prompt will be used to evaluate the test results. <details><summary>Template</summary> ```text --- name: Custom Test Result Evaluation description: | A template for a custom evaluation of the results. tags: - unlisted inputs: prompt: type: string description: The prompt to be evaluated. intent: type: string description: The extracted intent of the prompt. inputSpec: type: string description: The input specification for the prompt. rules: type: string description: The rules to be applied for the test generation. input: type: string description: The input to be used with the prompt. output: type: string description: The output from the model execution. --- system: ## Task You are a chatbot that helps users evaluate the performance of a model. You will be given a evaluation criteria <CRITERIA>, a LLM prompt <PROMPT>, output rules for the prompt <RULES>, a user input <INPUT>, and <OUTPUT> from the model. Your task is to evaluate the <CRITERIA> based on <PROMPT>, <INPUT>, and <OUTPUT> provided. <CRITERIA> The <OUTPUT> generated by the model complies with the <RULES> and the <PROMPT> provided. </CRITERIA> <PROMPT> {{prompt}} </PROMPT> <RULES> {{rules}} </RULES> ## Output **Binary Decision on Evaluation**: You are required to make a binary decision based on your evaluation: - Return "OK" if <OUTPUT> is compliant with <CRITERIA>. - Return "ERR" if <OUTPUT> is **not** compliant with <CRITERIA> or if you are unable to confidently answer. user: <INPUT> {{input}} </INPUT> <OUTPUT> {{output}} </OUTPUT> ``` </details> | no | — |
| create_eval_runs | Create an Evals run in [OpenAI Evals](https://platform.openai.com/docs/guides/evals). Requires OpenAI API key in environment variable `OPENAI_API_KEY`. | no | — |
| test_expansions | Number of test expansion phase to generate tests. This will increase the complexity of the generated tests. | no | 0 |
| test_samples_count | Number of test samples to include for the rules and test generation. If a test sample is provided, the samples will be injected in prompts to few-shot train the model. | no | — |
| test_samples_shuffle | Shuffle the test samples before generating tests for the prompt. | no | — |
| filter_test_count | Number of tests to include in the filtered output of evalTestCollection. | no | — |
| files | Files to process, separated by semi columns (;). .prompty,.md,.txt,.json,.prompt.yml | no | — |
| debug | Enable [debug logging](https://microsoft.github.io/genaiscript/reference/scripts/logging/). | no | — |
| model_alias | A YAML-like list of model aliases and model id: `translation: github:openai/gpt-4o` | no | — |
| openai_api_key | OpenAI API key | no | — |
| openai_api_base | OpenAI API base URL | no | — |
| azure_openai_api_endpoint | Azure OpenAI endpoint. In the Azure Portal, open your Azure OpenAI resource, Keys and Endpoints, copy Endpoint. | no | — |
| azure_openai_api_key | Azure OpenAI API key. **You do NOT need this if you are using Microsoft Entra ID. | no | — |
| azure_openai_subscription_id | Azure OpenAI subscription ID to list available deployments (Microsoft Entra only). | no | — |
| azure_openai_api_version | Azure OpenAI API version. | no | — |
| azure_openai_api_credentials | Azure OpenAI API credentials type. Leave as 'default' unless you have a special Azure setup. | no | — |
| azure_ai_inference_api_key | Azure AI Inference key | no | — |
| azure_ai_inference_api_endpoint | Azure Serverless OpenAI endpoint | no | — |
| azure_ai_inference_api_version | Azure Serverless OpenAI API version | no | — |
| azure_ai_inference_api_credentials | Azure Serverless OpenAI API credentials type | no | — |
| github_token | GitHub token with [models: read](https://microsoft.github.io/genaiscript/reference/github-actions/#github-models-permissions) permission at least. | no | — |
Outputs
| name | description |
|---|---|
| text | The generated text output. |