Prompt analysis
This menu identifies the prompts in use in your application and analyzes call volume, tokens, latency (TTFT), cost, and errors per prompt. Use it to find out which prompts are used the most and which are slow, expensive, or unstable, so you can select optimization targets.
Calls are identified by two units: operation and prompt version. An operation is a group of calls at the feature level, and a prompt version is the version of the system prompt used in that operation. One operation can have several prompt versions, and most charts are aggregated by prompt version.
From top to bottom, the screen is organized as Summary cards → Calls and trends → Distribution and errors → Feature usage → Table, so you can naturally follow it from grasping the status to checking individual operations. Set the time range and filters in the top option bar, and then query the widget data.
Summary cards
Cards that show the key metrics of the selected time range at a glance.
Unique prompts
Shows the number of distinct prompts called at least once within the selected time range. The bottom of the card also shows the total number of calls for those prompts.
- This is the starting point for grasping the scale of your prompt operations at a glance.
- If the number of unique prompts spikes above normal, check whether unintended prompt variants have been deployed.
Total prompt calls
Shows the total number of prompt calls that occurred within the selected time range. The bottom of the card also shows the average per prompt. Viewing it together with the average per prompt lets you gauge whether calls are concentrated on a few prompts or spread evenly.
Average per prompt = Total prompt calls / Unique prompts
Total prompt-related cost
Shows the total cost ($) incurred by LLM API calls in the selected time range. The bottom of the card also shows the version name and amount of the most expensive operation, along with its share (%) of the total.
- A single prompt with a high share is a priority target for cost optimization.
- If a specific prompt accounts for a large portion of the total cost, review the token length or model selection of that prompt first.
TOP 10 prompts by call volume
| Property | Value |
|---|---|
| Chart Type | Vertical bar chart |
| Aggregation Unit | Prompt version |
| Action | Select the displayed metric (dropdown at the top right) |
Shows the top 10 prompt versions by call volume, sorted by bar length. You can switch the displayed metric (call count, total tokens, total cost) using the dropdown at the top right.
- If calls are concentrated on a specific version, that version has a large impact on overall performance and cost, so manage it first.
- If a newly deployed version quickly enters the top ranks, it means the traffic switch (rollout) is proceeding normally.
Prompt usage trend
| Property | Value |
|---|---|
| Chart Type | Stacked area chart |
| Aggregation Unit | Prompt version |
| Action | Toggle legend items |
Shows the call volume trend of each prompt version over time as stacked areas. The legend colors correspond to each prompt version, and the thickness of an area indicates the call volume of that version.
- You can grasp the flow of the total call volume by time of day from the change in the thickness of the whole area.
- If the share of a specific version changes suddenly in a specific time slot, check the impact of a change in traffic pattern or a new version deployment.
- If the area thickens quickly right after a new version is deployed, it means the traffic switch is in progress.
TTFT distribution by prompt
| Property | Value |
|---|---|
| Chart Type | Box plot (x-axis: prompt version) |
| Aggregation Unit | Prompt version |
| Action | View details |
Compares the distribution of TTFT (Time To First Token) by prompt version as box plots. The box area represents p25 to p75 (the middle 50% range), the center line is the median, and the whiskers represent the minimum to the maximum.
- The version with a low median and a narrow box provides the most stably fast initial response.
- A version with long whiskers experiences extreme initial latency intermittently, so check the input length or the model load.
Finish reason distribution
| Property | Value |
|---|---|
| Chart Type | Donut chart |
| Aggregation Unit | Finish reason (length / stop / none) |
| Action | Toggle legend items |
Aggregates the finish reason values that the LLM API returns in its responses as they are, and shows them as a distribution. The values and names shown may differ by provider and model. WhaTap does not define them separately and uses the values returned by the API as they are.
- If the value that indicates normal termination (for example,
stop) accounts for most of the distribution, it means responses are finishing as intended. - If the value that indicates truncation due to a length limit (for example,
length) has a high share, check whether the maximum token (max_tokens) setting is too small and responses are being cut off in the middle. - Calls whose finish reason is empty or not recorded may be grouped and shown under a separate value (for example,
none).
The finish reason values shown follow the response specification of the LLM provider and model you use. For the exact meaning of each value, see the API documentation of that provider.
Error rate by prompt
| Property | Value |
|---|---|
| Chart Type | Vertical bar chart (x-axis: prompt version) |
| Aggregation Unit | Prompt version |
| Action | View details |
Shows the ratio (%) of errors to calls by prompt version.
- A version with a high error rate may be caused by an incorrect tool definition, a schema mismatch, or a token overflow, so check the metrics of that version together in the operation list.
- Cost is incurred even for failed requests, so improving a version with a high error rate also has a direct effect on cost savings.
Error rate = Error calls / Total calls x 100
Feature usage ratio by operation
| Property | Value |
|---|---|
| Chart Type | Card grid (one card per operation) |
| Aggregation Unit | Operation |
| Action | - |
One card is created per operation, and the operation name and total call count are shown at the top of the card. Inside the card, the ratio at which each feature (for example, reasoning) was used in the calls of that operation is shown as a bar and a percentage.
- A single call can use several features at the same time, so the ratios may not add up to 100%, and features that were not used are not shown.
- Feature usage patterns let you quickly understand how an operation behaves and discover feature usage that differs from your intent.
Operation list
| Property | Value |
|---|---|
| Chart Type | Table |
| Aggregation Unit | Operation + prompt version |
| Action | Sort |
Lists the operations identified within the period by prompt version and organizes the key metrics in a table. Even for the same operation, a different prompt version is shown as a separate row.
| Column | Description |
|---|---|
Operation | Name of the group of calls at the feature level |
Prompt version | Version of the system prompt used in that operation |
Prompt preview | The beginning of the original system prompt (— if none) |
Calls | Total number of calls within the period |
Avg input | Average number of input tokens per call |
Avg output | Average number of output tokens per call |
Avg TTFT | Average time to the first token (ms) |
Avg cost | Average cost per request (¢) |
Error rate (%) | Ratio of errors to calls |
- The default sort is descending by call count, and you can change the sort criteria by clicking a column header.
- A row with a high
Avg costis a prompt with an expensive unit price per request, so consider reducing the prompt length or reviewing the model. Prompt previewlets you quickly compare the prompt differences between versions of the same operation.
Reference
How prompts are identified
| Unit | Description |
|---|---|
| Operation | A group of calls at the feature level (for example, cs_chat_support) |
| Prompt version | The version of the system prompt used in that operation (for example, v2.1-ko) |
- One operation can have several prompt versions, and the TOP 10, usage trend, TTFT distribution, and error rate widgets are aggregated by prompt version.
- To quantitatively compare the performance differences between prompt versions, use the Prompt comparison menu. If you need response quality and safety evaluation, use the Response quality analysis menu together.