Create test datasets

Use Datasets in Prompt Studio to test a prompt against many inputs. Each case contains values for your prompt’s variables and, optionally, an expected output that an evaluation rule can use as a reference. This guide imports eight synthetic tickets for the Support ticket triage prompt from the Quickstart. You can reuse a dataset with other prompts in your organization when their input names and types match.

Create the support-ticket dataset

Create an empty dataset

Open Datasets under Resources in the Studio sidebar. If the example already exists, open it. Otherwise, click New dataset, name it Support ticket triage, and click Create dataset.The new dataset opens at Variable settings.

Match the prompt's variable

Click Add variable and enter:

Setting	Value
Variable name	`ticket_text`
Type	`string`
Required	On

Enter the name without braces. ticket_text matches {{ticket_text}} in your User message. Click Save variable settings before adding or importing cases.

Preview the sample CSV

Download support-triage.csv, then click Import CSV and select the file.Check the column mapping:

CSV column	Map to
`ticket_text`	The `ticket_text` variable
`expected_output`	Expected output

The preview shows the total case count and a sample of the rows. Confirm that ticket text appears as an input and the JSON reference answer appears as expected output. Resolve any import issues before continuing.

Import and inspect

Click Import 8 test cases, then open Test cases. An empty dataset now contains eight cases.Open a row to read the complete ticket and expected output. The sample includes billing, account access, product failures, general questions, and a ticket that contains an instruction the prompt should treat as data.

CSV import preview showing eight support-ticket cases, with ticket_text mapped to the input variable and expected_output mapped to Expected output. — Preview the CSV mapping before importing. Reference answers belong in Expected output, separate from the prompt's variables.

Inputs and expected output

For the first case, ticket_text contains the login failure affecting five agents. Its reference answer contains:

Reference answer

{
  "category": "account",
  "priority": "high",
  "summary": "Five team members cannot sign in to the dashboard because of an Invalid session error, blocking customer support."
}

Expected output is optional reference content. It is not another prompt variable, it is not sent as an extra message in this example, and adding it does not automatically score a generated answer. Select evaluation rules to decide what to check. The sample references describe intended answers. They do not require the model to reproduce the summary word for word. A Contains JSON rule does not compare the answer with this reference; semantic similarity does.

Support ticket triage dataset showing eight imported test cases, a ticket_text string column, and expected-output references. — The imported dataset contains eight cases. Open a row to read its full input and reference answer.

Add or edit a case

To add another scenario, click Add test case, fill the defined variables, enter an optional Expected output, and save. This adds a case beyond the eight in the sample. Open a case and click Edit test case, or use its row actions to choose Edit, when you need to correct it. Search Test cases to find inputs or expected outputs. Use Export CSV for the dataset, or select rows and choose Export selected for a subset. Keep the input contract stable. While cases exist, existing variable names, types, and required settings are fixed. You can add optional variables. If your prompt needs a different input contract, create a separate dataset with matching settings.

Prepare your own CSV

Use the prompt’s variable names as headers and include expected_output only when you have reference answers. For example:

Example CSV

ticket_text,expected_output
"I forgot my password. How can I reset it?","{""category"":""account"",""priority"":""normal"",""summary"":""A customer needs help resetting a forgotten password.""}"

CSV values containing commas, quotes, or line breaks need proper CSV quoting. Export from your spreadsheet tool, or follow the sample file’s doubled quotes around JSON fields. The import supports files up to 10 MB and datasets with up to 500 cases. Import adds rows to the current dataset; importing the sample again adds another copy of its cases. Use Skip column for file columns you do not want to import.

Evaluation snapshots

When you run an evaluation, Studio saves the dataset inputs used for that run. See them listed under Evaluation snapshots. An unchanged dataset can reuse its existing snapshot. Later edits affect future evaluations. They do not rewrite earlier run inputs, references, or scores. Deleting a case from the current dataset also leaves its existing evaluation snapshots intact. To reuse an agreed correction from an evaluated or production output, follow Human review. Add test case appends the reviewed inputs and final corrected answer and saves a new snapshot; it leaves existing rows and evaluation snapshots intact.

Troubleshooting

What you see	What to do
Import or add-case actions are disabled	Save Variable settings first and check whether the dataset has reached its case limit.
Missing or unknown variables	Match CSV mappings and dataset names to the variables in Build. Check spelling and types.
A type or row error in the preview	Fix the named row or mapping and preview again. Number, boolean, and JSON inputs must match the dataset’s types.
A reference-based rule skips a case	Supply Expected output for rules that require a reference. A reference is optional for other rules.
Import fails after preview	The dataset may have changed. Preview the file again and recheck the available capacity.
No evaluation snapshots	Run an evaluation first. Creating or editing cases alone does not create an evaluation run.

Next, create and test an evaluation rule, then use Evaluate dataset or Evaluate to run the saved prompt against these cases.

Getting Started

Overview

Risk & Governance

Config Workflows

Prompt Studio

API Management