Prompt Playground¶
The Prompt Playground enables you to validate prompts before deploying them to production. It supports both batch testing and single prompt testing, helping you evaluate prompt quality, compare model responses, and ensure consistent behavior across different scenarios.
1. Navigate to Prompt Catalog¶
Open Prompt Catalog and locate the workflow prompt you want to evaluate. Click the Edit icon to open the prompt.
2. Generate Test Data¶
Synthetic test datasets can only be generated for Workflow Prompts that contain one or more input variables.
Click Generate Test Data to begin creating a synthetic dataset.
3. Configure Dataset Generation¶
Select the types of test data you want to generate.
Test Data Types¶
- Happy Path – Tests the prompt with typical, valid inputs that represent normal user interactions.
- Edge Cases – Tests boundary and uncommon inputs to verify the prompt handles unusual scenarios correctly.
- Adversarial Cases – Tests intentionally challenging or misleading inputs to evaluate prompt robustness and security.
- Invalid Cases – Tests incorrect or malformed inputs to ensure the prompt fails gracefully and handles errors appropriately.
- Random Cases – Tests diverse, randomly generated inputs to uncover unexpected behaviors and improve overall reliability.
Evaluation Methods¶
Choose how the generated responses should be evaluated.
-
Rubric / Criteria – The LLM generates evaluation criteria for each item and scores the response against those criteria, making it suitable for open-ended or natural language outputs.
-
Exact Match – Compares the generated output with a predefined expected response and passes only when they match exactly.
4. Generate the Dataset¶
Use the slider to specify the number of test records to generate, then click Generate.
5. Save the Dataset¶
Provide a name for your dataset and click Save.
6. Batch Test Execution¶
After creating a dataset, you can execute it immediately or run it later from the dataset list.
Click Run.
Click Run Test Data to execute all generated test cases.
Once execution completes, review the generated outputs and evaluation results.
7. Prompt Playground Evaluation¶
The Prompt Playground supports two evaluation modes:
- Test Dataset – Run an entire synthetic dataset.
- Single Test – Test one custom input.
Select the prompt, evaluation type, dataset, and model, then click Run.
View Generated Outputs¶
Click any result row to inspect the generated response.
Click View Data to inspect the corresponding input used for that execution.
8. Single Test¶
Use Single Test to quickly validate a prompt without creating a dataset.
Enter values for each prompt variable.
Run the prompt to generate the response.
View Execution Details¶
Click the generated result to inspect the complete response and execution details.
Summary¶
The Prompt Playground provides an efficient way to validate prompts before deployment by allowing you to:
- Generate synthetic test datasets
- Test prompts against multiple scenarios
- Evaluate responses using Exact Match or Rubric scoring
- Execute batch tests across datasets
- Compare prompt behavior across different models
- Inspect generated outputs and execution details
- Perform quick one-off validations using Single Test
Using these capabilities helps improve prompt quality, reliability, and consistency before prompts are used in production.