Prompt comparison
This menu places two prompts side by side and compares their call volume, cost, performance, and response quality. Use it to quantitatively verify whether a prompt change actually improved performance and cost, or to analyze the differences in characteristics between different operations and versions.
You specify each comparison target as a combination of operation + prompt version + date, placing the baseline on A and the comparison target on B. Normally you put the existing version on A and the changed version on B to check the difference.
From top to bottom, the screen is organized as Summary (calls and cost) → Prompt diff → Metric comparison → Trend comparison → Distribution comparison → Prompt logs, so you can follow it from "what changed" (diff) to "what result that change produced" (metrics, trends, and logs).
Setting the comparison conditions
Specify the two prompts to compare in the top option bar.
-
Base date / Operation A / Prompt version A: Select the baseline prompt (A) and the query date.
-
Comparison date / Operation B / Prompt version B: Select the comparison target prompt (B) and the query date.
You can specify different dates for A and B, so you can also compare the same prompt before and after a deployment across different periods. In every widget, A is shown consistently in blue and B in purple.
Total prompt calls
| Property | Value |
|---|---|
| Chart Type | Bar chart (A vs B) |
| Comparison | A (baseline) · B (comparison) |
| Action | Show values on mouseover |
Compares the total number of calls of the two selected prompts as bars side by side. Each bar is labeled with date · prompt version.
-
If the difference in call count is large, also consider the difference in sample size when interpreting average metrics or trends.
-
If the call count of the new version (B) is growing quickly, it means a traffic switch (rollout) is in progress.
Total prompt-related cost
| Property | Value |
|---|---|
| Chart Type | Bar chart (A vs B) |
| Comparison | A (baseline) · B (comparison) |
| Action | Show values on mouseover |
Compares the total cost of the two selected prompts as bars side by side.
-
If the call counts are similar but the total costs differ greatly, the cause may be a difference in prompt length or model unit price.
-
You can compare the cost per call independently of sample size on the
Cost per callcard in Metric comparison below.
System prompt diff
| Property | Value |
|---|---|
| Chart Type | Text diff (left: A / right: B) |
| Action | Switch between Side-by-side and Inline views |
Compares the original system prompts of A and B line by line. Added and deleted lines are distinguished by color, and you can switch between the Side-by-side (split) and Inline (single flow) views using the icon at the top right.
-
Interpret this widget together with the metric and trend widgets below: this one shows "what changed," and those show "what result that change produced."
-
If the prompt of one version is empty, it means that version was newly introduced or completely replaced.
Metric comparison cards
| Property | Value |
|---|---|
| Chart Type | Metric cards (A · B values shown in parallel) |
| Comparison | A (baseline) · B (comparison) |
| Action | - |
Shows the key performance, cost, and quality metrics with the A and B values side by side on each card.
| Metric | Description |
|---|---|
| Average latency | Average time from request start to response completion |
| Average TTFT | Average time to the first token (perceived responsiveness) |
| Average TPOT | Average generation interval per output token (streaming speed) |
| Cost per call ($) | Average cost per request |
| Error rate (%) | Ratio of errors to calls |
| Cache hit rate | Ratio of input tokens served from the cache |
-
By directly contrasting the A and B values, you can see at a glance whether the change improved or degraded each metric.
-
If the cost per call went down while the cache hit rate went up, an increase in the common prompt prefix may have improved caching efficiency.
TTFT trend (A vs B)
| Property | Value |
|---|---|
| Chart Type | Line chart (2 series: A, B) |
| Comparison | A (baseline) · B (comparison) |
| Action | Show values on mouseover |
Overlays the average TTFT by time of day for A and B.
-
Check whether the gap widens only in specific time slots (for example, traffic peaks) or is spread evenly across the whole range.
-
If intermittent spikes appear on one side, you can suspect queue waiting or provider latency.
Cost per call trend (A vs B)
| Property | Value |
|---|---|
| Chart Type | Line chart (2 series: A, B) |
| Comparison | A (baseline) · B (comparison) |
| Action | Show values on mouseover |
Compares the trend of cost per call by time of day for A and B.
-
You can see over time how cache hit patterns or changes in output length affected the cost per call.
-
You can discover patterns where the cost spikes only in specific ranges even though the averages are the same.
Call volume trend (A vs B)
| Property | Value |
|---|---|
| Chart Type | Line chart (2 series: A, B) |
| Comparison | A (baseline) · B (comparison) |
| Action | Show values on mouseover |
Compares the call volume trend of the two prompts by time of day.
- You can see the progress of the traffic switch to the new version (B).
- Ranges with a large volume difference provide the basis for the difference in sample size when interpreting other metrics.
Error trend (A vs B)
| Property | Value |
|---|---|
| Chart Type | Line chart (2 series: A, B) |
| Comparison | A (baseline) · B (comparison) |
| Action | Show values on mouseover |
Compares the error rate trend by time of day for A and B.
-
Check whether errors clustered at a specific point right after the change, or stayed stably low.
-
If there are ranges where the error rate of B is higher than A, trace back the change that caused it in the system prompt diff.
Finish reason distribution
| Property | Value |
|---|---|
| Chart Type | Donut chart (A · B each) |
| Aggregation Unit | Finish reason value |
| Action | Toggle legend items |
Shows the distribution of the response finish reason for A and B side by side as donuts. The values and names shown may differ by provider and model. WhaTap does not define them separately and uses the values returned by the API as they are.
-
If the proportion of the value that indicates truncation due to a length limit (for example,
length) decreased in B, it means output control improved and fewer responses were cut off. -
You can gauge the difference in response completeness from the change in the proportion of the value that indicates normal termination (for example,
stop).
The finish reason values shown follow the response specification of the LLM provider and model you use. For the exact meaning of each value, see the API documentation of that provider.
Feature usage comparison
| Property | Value |
|---|---|
| Chart Type | Bar chart (A vs B) |
| Comparison | A (baseline) · B (comparison) |
| Action | - |
Compares the usage ratio of each feature for A and B.
-
You can verify whether the prompt change (for example, adding examples or strengthening instructions) drove the intended change in feature usage patterns.
-
A single call can use several features at the same time, so the ratios may not add up to 100%.
Model distribution
| Property | Value |
|---|---|
| Chart Type | 100% stacked horizontal bar (one row each for A · B) |
| Comparison | A (baseline) · B (comparison) |
| Action | Toggle legend items |
Compares the composition ratio of the models called by each prompt as A and B rows. The legend colors correspond to each model.
-
If the model composition of A and B differs, you must distinguish whether the latency and cost differences come from the prompt change or from the model difference.
-
If the proportion of a specific model changed significantly, check whether the routing rules or model designation were changed.
Prompt logs (A · B)
| Property | Value |
|---|---|
| Chart Type | Table (A · B each) |
| Action | Expand rows, go to prompt logs |
Shows the recent call logs of A and B in tables so you can check, case by case, how the metric differences appear in the actual responses.
| Column | Description |
|---|---|
Time | Time the call occurred |
Prompt | Summary of the input and the output |
Model | Name of the LLM model used |
Token | Total number of tokens |
Latency | Time from request start to response completion |
Cost | Cost of the call |
-
For calls to which response quality evaluation was applied, the score of each evaluation item (for example, Hallucination, Factuality, Relevance, Toxicity, Prompt Injection, PII Leak, URL Scan, Judge) is also shown in the
Evalarea. For details about the evaluation items, see the Response quality analysis document. -
Use the Go to prompt logs button at the top right of each table to move to the original logs of that prompt.
Reference
How to specify comparison targets
| Term | Description |
|---|---|
| Operation | A group of calls at the feature level (for example, cs_chat_support) |
| Prompt version | The version of the system prompt used in that operation (for example, v3.0-structured) |
| Date | The base date for querying each target (A and B can be specified differently) |
-
To first find comparison target candidates, use Prompt analysis. If you need evaluation from the perspective of response quality and safety, use the Response quality analysis menu.
-
In every widget, A is shown in blue and B in purple.