> ## Documentation Index
> Fetch the complete documentation index at: https://docs.traceport.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Production quality and analytics

> Score production prompt requests, read pass rate and coverage together, control scoring costs, and inspect the exact request behind a result.

Use **Analytics** to monitor requests to your released prompts and inspect the evidence behind their scores. A production policy chooses which new requests to score and which rules to apply.

This guide continues with **Support ticket triage**, its released v2, and the required **Support ticket contains JSON** rule. Complete [Versioning and releases](https://docs.traceport.ai/prompt-studio/releases.md) and [Evaluation rules](https://docs.traceport.ai/prompt-studio/evaluations.md) first if you have not prepared those resources.

## Choose what to inspect

Open **Analytics** in Prompt Studio and select your prompt under **Prompt filter**. You can also select **All prompts** to inspect aggregate activity; select a specific prompt to manage policies or filter its versions.

| Tab | What it contains |
| - | - |
| **Production** | Requests that execute a saved, released prompt, including calls pinned to an earlier released version. Usage includes these requests even when no scoring policy matches. Ordinary chat calls without a saved-prompt reference and tests in **Build** are excluded. |
| **Dataset evaluations** | Dataset cases with saved dataset and rule snapshots. These are separate from production requests. Build tests and older runs without this evidence are excluded. |
| **Production policies** | The selected prompt's scoring rules, sampling, scope, budget, and revision history. These policies are separate from the publication requirements in **Release**. |

An organization owner or admin can activate and pause production policies. Running requests still incurs generation cost; LLM judge scoring adds its own model usage.

## Configure production scoring

**1. Set the policy's scope and budget**

Open **Production policies** and click **Create policy**. For the synthetic example, enter:

| Field | Example value |
| - | - |
| **Policy name** | `Support ticket quality` |
| **Prompt version** | **All production versions** |
| **Generation model (optional)** | `claude-haiku-4-5-20251001` |
| **Sampling (%)** | `100` |
| **Daily scoring budget (USD)** | `0.10` |

Use the exact generation model ID from your own prompt requests. Leave the model field blank to match all models, or choose a specific prompt version to narrow the scope.

`100%` sampling makes this small example easier to inspect. At lower percentages, matching requests outside the sample remain unscored. Sampling is stable for each request.

**2. Choose the required rule**

Search for and select **Support ticket contains JSON**. Keep **Required for a request to pass** checked.

You can start from **Start from a saved suite (optional)** instead of selecting rules individually. Expand **Configure this rule for the policy** to adjust its settings for this policy. Those changes do not edit the library rule.

A policy accepts 1–10 rules. At least one must be required for requests to receive a pass/fail verdict. Advisory results are recorded without changing that verdict.

**3. Activate the revision**

Click **Activate revision** and confirm the policy is **Active**.

Activation saves the rule definitions and applies to new matching requests. It does not score earlier requests retroactively. Editing a library rule later does not change this saved policy revision.

![Create production policy for Support ticket quality with all production versions, the Haiku model, 100 percent sampling, a 10-cent daily scoring budget, and the required Contains JSON rule.](https://mintcdn.com/traceport/qnQpjm16A9r4NqtM/images/prompt-studio/support-triage-production-policy.jpg?fit=max\&auto=format\&n=qnQpjm16A9r4NqtM\&q=85\&s=7d1c832e7eb8726804b36f9f9c3beb50)

The example samples every matching request and uses one required deterministic JSON rule. The daily scoring budget is separate from generation cost.

> [accordion] Use LLM judges and understand the scoring budget
>
> For an LLM judge, configure its criteria, threshold, API key, and judge model. Judge calls incur model usage. Deterministic rules do not add model calls; the example's recorded scoring cost is zero.
>
> Production requests have no reference answer for **Semantic similarity**. That method is skipped in production; making it required leaves the request unscored. Use it for [dataset evaluations with expected outputs](https://docs.traceport.ai/prompt-studio/evaluations.md#add-a-judge-or-reference-based-check).
>
> Set a positive daily scoring budget up to `$100`. The budget resets at midnight UTC and is shared across revisions of the prompt's policy. Creating a new revision does not reset the day's spending. The budget controls scoring, not the model call that generates the response. Scoring can stop when the remaining budget is insufficient or priced usage is unavailable; inspect the record's reason.

## Generate requests and refresh

Send new requests to the released prompt using [Chat Completions with `traceport.prompt`](https://docs.traceport.ai/api-reference/endpoint/create-chat-completion#run-a-saved-prompt). In **Prompts**, use **Copy prompt slug** to obtain the saved slug for your instance. Send each ticket as `traceport.prompt.variables.ticket_text`.

Use the login ticket from the [Quickstart](https://docs.traceport.ai/prompt-studio/quickstart.md) and the password-reset ticket from [Results and comparison](https://docs.traceport.ai/prompt-studio/results.md#inspect-the-password-reset-change). Calling **Run prompt** in Build will not create production activity.

Return to **Analytics → Production** and click **Refresh**. Scoring runs asynchronously, so a new request can appear before its required scores are available. Refresh again after processing; the view does not continuously refresh itself.

## Read pass rate and coverage together

The captured example contains three actual requests: login and password reset while policy r1 was active, followed by a billing request while scoring was paused. All three used released v2. A new revision was then activated with the same settings for future requests.

| Metric | Meaning | Recorded example |
| - | - | - |
| **Requests** | All production invocations in the selected scope and period. | `3`, with no generation errors. |
| **Pass rate** | Passed requests divided by requests with every required score. | `2 / 2 = 100%`. |
| **Scoring coverage** | Fully scored requests divided by policy-matched requests, including requests outside the sample. | `2 / 2 = 100%`. The billing request had no matching active policy. |
| **Recorded cost** | Recorded generation cost plus scoring cost in the selected scope. | `$0.0012` generation and `$0.0000` scoring, rounded in the UI. |

A dash means the denominator is zero. Read the sampled count and the pending, error, skipped, and no-required-rule counts below the metrics. These scoring counts describe the eligible population; they do not make unmatched requests disappear from usage.

Treat **partial pricing** or **Unavailable** as incomplete cost information. Totals include the prices that were recorded.

![Production analytics for support-ticket triage showing three requests, two passed out of two fully scored, two fully scored out of two policy-matched requests, and 0.0012 dollars recorded cost.](https://mintcdn.com/traceport/qnQpjm16A9r4NqtM/images/prompt-studio/support-triage-production-analytics.jpg?fit=max\&auto=format\&n=qnQpjm16A9r4NqtM\&q=85\&s=cea98618ddbc5a44d26e54ed1476eb44)

Three requests were generated, but only two matched a scoring policy. Both eligible requests were fully scored. These small, recorded JSON-check results do not establish classification accuracy.

The **Contains JSON** rule checks whether JSON can be found in the response. It does not verify category, priority, or every instruction. Review the [v1 priority mismatch that passed this rule](https://docs.traceport.ai/prompt-studio/results.md#inspect-the-password-reset-change) before interpreting a high pass rate as overall quality.

## Inspect an exact request

1. Scroll to **Request records** and open a row. Use **Outcome** to find **Passed**, **Failed**, **Unscored**, or **Generation / scoring error** records.
2. Check the prompt version, model, policy revision, and separate generation/scoring costs. Read **Generated output** and each rule's required/advisory status, score, verdict, and reason.
3. Expand **Input variables and messages** to see the actual inputs and rendered template. Expand **Record identifiers** when you need its exact execution or trace ID.
4. Click **Open trace** to inspect the linked generation request. The password-reset example opens its production trace and the same prompt execution.

The record URL reopens that exact record. **Send to review** is available for completed production outputs whose content is still retained. Follow [Human review](https://docs.traceport.ai/prompt-studio/review.md) to record independent decisions and resolve disagreements. These decisions do not replace the recorded automatic scores.

![Production password-reset record showing account category, normal priority, a passed required JSON rule with score 1.00, separate costs, and an Open trace action.](https://mintcdn.com/traceport/qnQpjm16A9r4NqtM/images/prompt-studio/support-triage-production-record.jpg?fit=max\&auto=format\&n=qnQpjm16A9r4NqtM\&q=85\&s=bedb59ef2de6e33b3b225125e0e3a499)

The recorded password-reset request used v2 and policy r1. Its required JSON check passed with score 1.00; the actual output and generation trace remain available for inspection.

### Distinguish failure from missing scores

| What you see | How to interpret it |
| - | - |
| **Passed** or **Failed** | Every required rule returned a score. Failed means at least one required rule failed. |
| **Unscored** with **Queued** or **In progress** | Scoring has not finished. Refresh after processing. |
| **Unscored** with **Scoring error** | A rule could not produce a score. Inspect its reason and configuration; a scoring error is not a failed quality check. |
| **Unscored** with **Skipped** | Check the reason: outside the sample, insufficient scoring budget, paused policy, or expired content can prevent scoring. |
| **Unscored** with **No scoring job** | Check whether a policy was active when the request was made and whether its version and model matched. Very recent requests may still be waiting for enrollment. |
| **Unscored** with **Completed** | Inspect the required rules: a skipped similarity check can leave a required score missing. A policy with only advisory rules also has no required-rule verdict. |

The billing example completed generation while scoring was paused. It shows **No scoring job**, no rule results, and the reason **This request did not match a production policy**. Its output is available, but it has no automatic pass/fail verdict.

![Unscored billing production request with a generated JSON response, No scoring job status, no matching production policy, and no recorded rule results.](https://mintcdn.com/traceport/qnQpjm16A9r4NqtM/images/prompt-studio/support-triage-production-unscored.jpg?fit=max\&auto=format\&n=qnQpjm16A9r4NqtM\&q=85\&s=909ca3eb2e5901ad3f67bb71b67da704)

The billing response was generated successfully while scoring was paused. Unscored describes missing evaluation evidence, not a failed response.

If retention has expired, the record shows that its input/output content is unavailable while retaining recorded scores. Review and trace access still depend on the records available to your account.

## Filter activity and interpret trends

* **Period** selects the last 7, 30, or 90 days. **Refresh** updates the end of the window and reloads results.
* **Version**, **Model**, and **Policy** narrow the analytics population. Enter an exact model ID, then press Enter or click the apply checkmark. Choose a policy revision to inspect evidence produced under those saved rules.
* Click a **By model** or **By version** row to apply that filter. **Scoring by day** uses UTC dates; the first and last days can be partial.
* Selecting a day or **Outcome** filters the records table. It does not recalculate the headline metrics. In **Dataset evaluations**, **Dataset split** also filters the case table.

> [accordion] Read pass-rate changes and human-review agreement
>
> **Pass rate change** needs a selected policy revision. It compares the earlier and later halves of the selected period using that revision's rules, in percentage points. Both halves need scored results; otherwise you see **Not enough scored results**. Request content can differ between periods, so this does not isolate the effect of a prompt change. Use [compatible dataset comparisons](https://docs.traceport.ai/prompt-studio/results.md#compare-the-candidate-with-its-baseline) to compare fixed cases.
>
> **Human review agreement** reports agreement between review pairs on their verdicts and compatible corrections, with adjudicated and pending counts. It does not measure automatic scoring accuracy. **No review pairs yet** means there are no pairs to compare; use **Open review queue** to inspect the review workflow.

## Revise or pause scoring

Open **Production policies** and click a row to inspect its saved scope and rules. Click **New revision**, **Create new revision** in its detail panel, or **Create revision from this** in a row's menu to prepare another revision.

Review the settings and click **Activate revision**. Only one revision is active per prompt. New matching requests use it; historical records keep the policy revision and rule definitions used when they were scored.

Use **Pause scoring** in the active policy's row menu or detail panel to stop new scoring work. Judge calls already running can finish. To start again, create and activate a new revision. Requests made during the pause are not retroactively scored by that later revision. Pausing does not stop prompt generation or its costs.

For the recorded example, r1 scored two requests, scoring was paused for the billing request, and r2 was activated with the same settings. The two scored records still identify r1.

## Compare with dataset activity

Select **Dataset evaluations** to inspect offline coverage, pass rate, cost, and case records separately. The existing v1/v2 eight-ticket evaluations provide 16 case records across both versions. Filter **Version** to inspect cases for that version; several runs can share a version. Open a case and choose **Open evaluation result** for its detailed run evidence.

Use this view to track evaluation activity over time. Use [Results and comparison](https://docs.traceport.ai/prompt-studio/results.md) for paired outputs and regressions on matching cases.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.