LLM Dashboard Widget Reference
This document is a reference that organizes the widgets provided by the LLM Dashboard by dashboard area. Check the chart type, filters, and usage for each widget. For the full screen layout and default preset, see LLM Dashboard.
Available Widgets
LLM Performance Metrics
| Widget | Description |
|---|---|
| LLM API Request Count | Request count + error rate by time period |
| Request Count by Status Code | HTTP status code (2xx/4xx/5xx) distribution |
| Latency Average | Average response time trend by time period |
| Latency Percentile | Latency p50/p75/p95/p99 trend |
| Latency Percentile Individual Widgets | Individual widget per percentile |
| Latency Distribution Comparison | Latency box plot by model |
| TTFT Average | TTFT average trend by time period |
| TTFT Percentile | TTFT p50/p75/p95/p99 trend |
| TTFT Percentile Individual Widgets | Individual widget per percentile |
| TTFT Distribution Comparison | TTFT box plot by model |
| TPOT Average | TPOT average trend by time period |
| TPOT Percentile | TPOT p50/p75/p95/p99 trend |
| TPOT Percentile Individual Widgets | Individual widget per percentile |
| TPOT Distribution Comparison | TPOT box plot by model |
| Overall Performance Comparison | TTFT/TPOT/Output TPS comparison by model |
| Output Token Throughput | Output tokens per second (tok/s) |
Tokens & Cost
| Widget | Description |
|---|---|
| Token Usage | Input/output token trend by time period |
| Average Tokens per Request | Average input/output tokens per request |
| Token Usage Distribution | Input/output token total comparison by model |
| Today's Token Usage | Today's cumulative tokens + comparison with previous day |
| Cache Hit Rate | Cache token ratio (%) trend |
| Cache Hit Rate Distribution | Cache hit rate comparison by model |
| Cache Cost Savings | Cost saved through caching ($) trend |
| Cache Cost Savings Distribution | Cache cost savings comparison by model |
| Cost Usage | Input/output cost ($) trend by time period |
| Average Cost per Request | Average cost per request |
| Cost Usage Distribution | Cost total comparison by model |
| Today's Cost | Today's cumulative cost + comparison with previous day |
Model Comparison & Analysis
| Widget | Description |
|---|---|
| Overall Performance Comparison | TTFT/TPOT/Output TPS comparison by model |
| Token Usage Distribution | Input/output token total comparison by model |
| Error Rate Distribution | Error rate (%) comparison by model |
| Error Count | API error and program error count by time period |
| Request Count by Status Code | HTTP status code (4xx/5xx) distribution |
Live Observation
| Widget | Description |
|---|---|
| Active LLM API | Running LLM API calls and the long-running list |
| Active Trajectory | Number of trajectories in progress and the distribution by level |
Widget Details
LLM API Requests
A widget group for monitoring LLM API request volume and status codes.
LLM API Request Count
| Property | Value |
|---|---|
| Chart Type | Bar chart (request count) + Line chart (error rate) combo |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays request count and error rate (%) by time period for LLM API calls. The bar chart (left Y-axis) shows request volume, and the line (right Y-axis) shows error ratio against total requests.
- If request volume spikes with a simultaneous error rate increase, Rate Limit exceeded or provider outage is likely.
- If request volume is unchanged but error rate rises, an issue may be occurring in a specific model or agent. Use View individually to narrow down the cause.
Request Count by Status Code
| Property | Value |
|---|---|
| Chart Type | Stacked bar chart |
| Filter | None |
| Action | View details |
Displays HTTP status code distribution by time period, divided into 2xx (success), 4xx (client error), and 5xx (server error).
- 4xx increase — Check for invalid request parameters, authentication failures, or token limit exceeded.
- 5xx increase — Indicates LLM provider server failure or temporary overload.
- Cross-reference with the LLM API Request Count widget's error rate for quick error classification.
LLM API Response Performance
A widget group for analyzing LLM API overall response time (Latency) from various perspectives. Latency is the total elapsed time from when a request is sent to when the response is fully completed.
Latency Average
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the Latency average trend over time.
- If Latency average rises overall, check for model response speed degradation or network delays.
- Use View individually to compare Latency by model and identify which model is causing delays.
Latency Percentile
| Property | Value |
|---|---|
| Chart Type | Line chart (p50, p75, p95, p99 — 4 series) |
| Filter | None |
| Action | View details |
Displays Latency percentile trends over time in four series: p50, p75, p95, and p99. For example, a p95 value of 3 seconds means 95% of all requests completed within 3 seconds.
- A large gap between p50 and p99 indicates most requests are fast but some experience severe delays.
- If p50 and p99 rise together, it indicates overall performance degradation.
Latency Percentile Individual Widgets
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Provides each percentile as an individual widget. The meaning of each percentile is as follows:
| Percentile | Description |
|---|---|
p50 | Half of all requests completed within this time. Represents average response speed perceived by typical users. |
p75 | 75% of all requests completed within this time. Gap from p50 reveals additional delay in top 25% of requests. |
p95 | 95% of all requests completed within this time. Frequently used as SLA (Service Level Agreement) baseline metric. |
p99 | 99% of all requests completed within this time. Represents the slowest 1% of requests for system stability assessment. |
Latency Distribution Comparison
| Property | Value |
|---|---|
| Chart Type | Box plot (x-axis: model) |
| Filter | Tag filter (always visible) |
| Action | View details |
Compares Latency distribution by tag (model, etc.) using a box plot. The box area represents p25–p75 (middle 50% range), the center line is the median, and whiskers show min–max values.
- Models with wide boxes have high response time variance and may need stability improvements.
- Models with long whiskers experience occasional extreme delays.
- Compare distributions side by side to identify the most stable model.
TTFT (Time To First Token)
TTFT is the time from when a request is sent until the first response token arrives. As TTFT increases, users perceive that "the response hasn't started yet."
TTFT Average
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the TTFT average trend over time.
- If TTFT spikes at a specific time, check for increased model queue wait or provider-side delay.
- Use View individually to compare which model has slower initial responses.
TTFT Percentile
| Property | Value |
|---|---|
| Chart Type | Line chart (p50, p75, p95, p99 — 4 series) |
| Filter | None |
| Action | View details |
Displays TTFT percentile trends over time in four series: p50, p75, p95, and p99. For example, a p95 value of 2 seconds means 95% of all requests received the first token within 2 seconds.
- If p50 is stable but p99 spikes, it signals intermittent queue wait occurrences.
- If all percentiles rise together, overall initial response delay is occurring.
TTFT Percentile Individual Widgets
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Provides each percentile as an individual widget.
| Percentile | Description |
|---|---|
p50 | Initial response wait time perceived by most users. |
p75 | Gap from p50 reveals additional initial delay in top 25% of requests. |
p95 | SLA baseline metric: "This initial response wait is guaranteed for most users." |
p99 | Initial response time of the slowest 1%. Check for intermittent timeouts or delays under specific conditions. |
TTFT Distribution Comparison
| Property | Value |
|---|---|
| Chart Type | Box plot (x-axis: model) |
| Filter | Tag filter (always visible) |
| Action | View details |
Compares TTFT distribution by tag (model, etc.) using a box plot. The box area represents p25–p75, the center line is the median, and whiskers show min–max values.
- Models with low median and narrow box provide the most stable and fast initial responses.
- Models with long whiskers experience occasional extreme initial delays.
TPOT (Time Per Output Token)
TPOT is the average time to generate one output token. Lower TPOT results in smoother streaming responses; higher TPOT may cause a "stuttering response" perception.
TPOT Average
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the TPOT average trend over time.
- If TPOT spikes at a specific time, check for increased model load or Rate Limit impact.
- Use View individually to compare token generation speed by model.
TPOT Percentile
| Property | Value |
|---|---|
| Chart Type | Line chart (p50, p75, p95, p99 — 4 series) |
| Filter | None |
| Action | View details |
Displays TPOT percentile trends over time in four series: p50, p75, p95, and p99. For example, a p95 value of 40ms means 95% of all requests generated each token within 40ms.
- If p50 is stable but p99 rises, it signals intermittent token generation stalls.
- If all percentiles rise together, it indicates overall streaming speed degradation.
TPOT Percentile Individual Widgets
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Provides each percentile as an individual widget.
| Percentile | Description |
|---|---|
p50 | Typical streaming speed. Identify models providing the smoothest streaming. |
p75 | Gap from p50 reveals streaming speed degradation in top 25% of requests. |
p95 | SLA baseline streaming speed metric. |
p99 | Token generation speed of the slowest 1%. Check for intermittent stall issues. |
TPOT Distribution Comparison
| Property | Value |
|---|---|
| Chart Type | Box plot (x-axis: model) |
| Filter | Tag filter (always visible) |
| Action | View details |
Compares TPOT distribution by tag (model, etc.) using a box plot. The box area represents p25–p75, the center line is the median, and whiskers show min–max values.
- Models with low median and narrow box provide the most stable streaming performance.
- Models with long whiskers experience occasional significant token generation slowdowns.
Overall Performance Comparison
| Property | Value |
|---|---|
| Chart Type | Horizontal bar chart (per model, 3 metrics) |
| Filter | Tag filter (always visible) |
| Action | View details, p95/p99 tab toggle |
Compares three key performance metrics per model in a horizontal bar chart. Toggle between p95/p99 tabs to check typical performance and worst-case performance respectively.
| Metric | Description | Direction |
|---|---|---|
TTFT | Time users wait for the first response | Lower is better |
TPOT | Time to generate one token (streaming speed) | Lower is better |
Output TPS | Output tokens generated per second (overall throughput) | Higher is better |
By combining all three metrics, you can identify at a glance models that "start quickly, stream smoothly, and deliver high throughput."
Tokens
A widget group for monitoring token usage patterns and cache efficiency.
Token Usage
| Property | Value |
|---|---|
| Chart Type | Line chart (input, output — 2 series) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays input token count and output token count trends over time. Input tokens are tokens included in the prompt; output tokens are tokens generated by the model as responses.
- If token usage spikes at a specific time, check for traffic increase, prompt changes, or abnormal request influx.
- Use View individually to identify which model or agent is consuming the most tokens.
Average Tokens per Request
| Property | Value |
|---|---|
| Chart Type | Line chart (input, output — 2 series) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the trend of average input/output tokens used per request over time.
- If average input tokens per request spike, prompts may have gotten longer or long contexts may have been included.
- If average output tokens per request spike, response lengths have increased, directly impacting cost.
Output Token Throughput
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View details |
A throughput metric showing output tokens generated per second (Tokens Per Second).
- Periods with decreased throughput should be checked for model response bottlenecks or Rate Limit impact.
- Cross-reference with token usage to identify periods where "tokens are consumed heavily but processing speed is slow."
Token Usage Distribution
| Property | Value |
|---|---|
| Chart Type | Histogram (grouped by tag) |
| Filter | Tag filter (always visible) |
| Action | View details |
A distribution chart comparing input/output token totals by tag (model, etc.).
- Quickly identify which model consumes the most tokens.
- Models with high input ratio have room for cost reduction through prompt optimization.
- Models with high output ratio may benefit from adjusting response length limits (
max_tokens).
Today's Token Usage
| Property | Value |
|---|---|
| Chart Type | Daily comparison chart (today/previous day) |
| Filter | None |
| Action | Date comparison |
Displays today's cumulative token usage (input + output) and change compared to the same time on the previous day.
- If token consumption rate is faster than the previous day, check for traffic increase or prompt size changes.
- Useful for proactively detecting Rate Limit thresholds or daily budget overruns.
Cache Hit Rate
| Property | Value |
|---|---|
| Chart Type | Line chart (%) |
| Filter | Tag filter |
| Action | View individually, View details |
Displays the ratio (%) of tokens retrieved from cache out of total input tokens over time.
- Higher hit rate means prompt caching is working effectively with significant cost savings.
- If hit rate drops suddenly, check for prompt content changes, cache TTL expiration, or new request type influx.
- Use View individually to compare cache efficiency by model.
Cache Hit Rate = (sum of cached_tokens in period / sum of input_tokens in period) × 100
Cache Hit Rate Distribution
| Property | Value |
|---|---|
| Chart Type | Histogram (by tag) |
| Filter | Tag filter (always visible) |
| Action | View details |
A distribution chart comparing cache hit rates (%) by tag (model, etc.).
- Models with high hit rate have well-utilized caching; models with low hit rate need prompt structure review for caching targets.
- If hit rate gaps between models are large, differentiate caching strategies per model for additional cost savings.
Cache Cost Savings
| Property | Value |
|---|---|
| Chart Type | Line chart ($) |
| Filter | Tag filter |
| Action | View individually, View details |
Displays the actual cost savings ($) from prompt caching over time. Cached tokens are charged at a lower rate than regular input tokens, and this difference is calculated as cost savings.
- Consistent cost savings indicate the caching strategy is working effectively.
- If cost savings decrease, cross-reference with cache hit rate to check for increased cache misses.
- Use View individually to identify which model yields the highest savings.
Cache Cost Savings Distribution
| Property | Value |
|---|---|
| Chart Type | Histogram (by tag) |
| Filter | Tag filter (always visible) |
| Action | View details |
A distribution chart comparing cache cost savings ($) by tag (model, etc.).
- Models with high savings benefit from caching, so maintain or strengthen their caching configuration.
- Models with low savings should be reviewed alongside cache hit rate to improve prompt structure for caching or adjust cache TTL.
Cost
A widget group for monitoring LLM API usage cost.
Cost Usage
| Property | Value |
|---|---|
| Chart Type | Line chart (input, output — 2 series) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays input/output cost ($) trends over time. Input cost is prompt processing cost; output cost is response generation cost.
- If cost spikes at certain times, check request volume and tokens per request for that period.
- Compare the ratio of input cost to output cost to decide whether to focus on prompt optimization or response length limits.
- Use View individually to compare cost contribution by model.
Average Cost per Request
| Property | Value |
|---|---|
| Chart Type | Line chart (input, output — 2 series) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the trend of average input/output cost ($) per request over time.
- If cost per request changes suddenly, check for model switching, provider pricing changes, or prompt size changes.
- If total cost increases but cost per request is unchanged, request volume has simply increased — focus on request volume management.
Cost Usage Distribution
| Property | Value |
|---|---|
| Chart Type | Histogram (grouped by tag) |
| Filter | Tag filter (always visible) |
| Action | View details |
A distribution chart comparing input/output cost totals by tag (model, etc.).
- Quickly identify which model has the highest cost.
- Check usage frequency and tokens per request for high-cost models to select cost optimization targets.
Today's Cost
| Property | Value |
|---|---|
| Chart Type | Daily comparison chart (today/previous day) |
| Filter | None |
| Action | Date comparison |
Displays today's cumulative cost ($) and change compared to the same time on the previous day.
- If cost consumption rate is faster than the previous day, check for request volume increase, model switching, or prompt size changes.
- Useful for proactively detecting daily budget overruns.
Errors
A widget group for monitoring LLM API error status.
Error Rate Distribution
| Property | Value |
|---|---|
| Chart Type | Horizontal bar chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Compares error rates (%) by tag (model, etc.) in a horizontal bar chart. Error rate is the ratio of errors (status_code 400 or above) to total requests in the group.
- Quickly identify whether errors are concentrated in a specific model or agent.
- Models with high error rates should be checked for provider outage, invalid parameters, or Rate Limit exceeded.
- Use View individually to narrow down error causes by
Agent,Provider, and other criteria.
Error Count
| Property | Value |
|---|---|
| Chart Type | Stacked bar chart |
| Filter | None |
| Action | View details |
Displays error counts by time period, divided into API errors and program errors.
- API Error — Errors returned by the LLM provider, including Rate Limit exceeded, server errors, and authentication failures.
- Program Error — Client-side errors, including response parsing failures, timeouts, and connection errors.
- Error type trends help quickly distinguish whether the issue is on the provider side or client side.
Live Observation
This widget group lets you check the LLM calls and trajectories that are running right now. A trajectory is the unit that groups the transactions participating in a single LLM workflow. The list screens of both widgets use running data queried directly from the agent, so there is no past range. If the dashboard is paused or you are viewing a past time, no path to the list is provided.
Active LLM API
| Property | Value |
|---|---|
| Chart Type | Swarm chart (track per model) |
| Filter | Dashboard agent filter |
| Action | View details, Open in new window |
Displays the LLM API calls that are currently running as dots on a track for each model. One dot is one call, and the horizontal position indicates the Normal, Slow, or Very Slow range. Data for the three ranges is aggregated according to the values set in the agent options.
- If dots pile up in the
Very Slowrange, check whether calls running longer than the criteria set in the agent are concentrated in a specific model or provider. - Clicking the widget body opens the long-running list in a modal. Expanding the prompt lets you check the system prompt and input prompt that are running, and selecting a call in the list moves to the trajectory view that the call belongs to, where you can check the entire execution flow.
- Clicking the View details button in the widget header opens the same list on a wide screen. Clicking the Open in new window button in the modal header opens it in a separate window.
Active Trajectory
| Property | Value |
|---|---|
| Chart Type | Orbit chart + count list by level |
| Filter | Dashboard agent filter |
| Action | View details, Open in new window |
Displays the number of currently active trajectories and the number of steps of each trajectory, divided into the Light, Middle, and Heavy levels. A level indicates how many times and how repeatedly steps were called; it does not mean speed or execution time.
- If even one of the multi-transactions it belongs to is running, that trajectory is counted as live.
- The level boundaries follow the criteria set in the agent options.
- Clicking a level opens the trajectory list filtered by that level. Selecting a row in the list leads to the trajectory view.
- Clicking the orbit chart opens the entire list without the level filter.
Trajectory List
The list consists of a group header for each trajectory and the transaction rows below it. The group header shows mtid, the number of grouped transactions, the user, and the session. A standalone flow with no mtid is displayed as a standalone run. You can export the list as CSV.
| Column | Description |
|---|---|
Transaction | Transaction name |
| Status | Transaction progress status (Running, Completed, Failed) |
mtid | Key that groups a trajectory |
| Steps | Number of steps the transaction performed |
| Agent | Agent that performed the transaction |
| LLM Calls | Number of LLM calls that occurred within the transaction |
| Elapsed | Time elapsed since the transaction started |
| User | user_id attached to the log |
| Session | session_id attached to the log |
Reference
Collected Fields
| Field | Description |
|---|---|
request_count | LLM API call count |
status_code | HTTP response status code (2xx, 4xx, 5xx) |
latency | Request start to response completion total time (ms) |
ttft | Time To First Token. Request start to first token received (ms) |
tpot | Time Per Output Token. Average interval between output tokens (ms) |
input_tokens | Total input token count (including cached) |
output_tokens | Output token count |
cached_tokens | Input tokens retrieved from cache (subset of input_tokens) |
input_tokens_cost | Total input token cost ($). Includes cached_tokens_cost |
output_tokens_cost | Output token cost ($) |
cached_tokens_cost | Cached token cost ($) |
error_type | Error type (api_error, program_error) |
Widget Group Summary
| Group | Widgets | Description |
|---|---|---|
| LLM API Requests | 2 | Request volume and status code monitoring |
| LLM API Response Performance | 8 | Latency trend and distribution analysis |
| TTFT | 7 | Time to first token analysis |
| TPOT | 7 | Token generation speed analysis |
| Tokens | 9 | Token usage patterns and cache efficiency analysis |
| Cost | 4 | LLM API cost monitoring |
| Errors | 2 | Error status analysis by type |
| Live Observation | 2 | Status of running LLM calls and trajectories |
| Total | 41 |
Terminology
| Term | Description |
|---|---|
Latency | Total elapsed time from when an LLM API request is sent to when the response is fully completed (ms) |
TTFT | Time To First Token. Time from when a request is sent until the first response token arrives (ms) |
TPOT | Time Per Output Token. Average time to generate one output token (ms) |
Output TPS | Tokens Per Second. Output tokens generated per second (tokens/s) |
Percentile | The value below which a given percentage of data falls when sorted in order. Example: p95 = 95% of all data is at or below this value |
Cache Hit Rate | Ratio of tokens retrieved from cache out of total input tokens (%). Higher values mean greater cost savings |
Tag Filter | Feature to filter data by Agent, Model, Provider, Operation criteria |
Preset | Saved widget configuration (type, position, size). Custom presets can be created |
Trajectory | Unit that groups the transactions taking part in one LLM workflow by mtid |