Skip to main content

LLM Dashboard

Home > Select Project > Dashboard > LLM Dashboard

The LLM Dashboard is a menu that provides real-time monitoring of LLM (Large Language Model) API request volume, response performance, token usage, cost, and error status in a single view. Widgets are arranged from top to bottom in the order Real-time Status → Requests & Performance → Tokens & Cost → Comparison & Errors, allowing you to naturally progress from status overview to root cause analysis. This layout is saved as the default preset and is ready to use immediately without additional configuration.

LLM Dashboard

The LLM Dashboard consists of five areas based on the default preset.

AreaNameDescription
①Option BarSet the lookup time, agent filter, screen mode, and preset.
②Real-time StatusView current transaction speed, LLM call types, active HTTPC, and hitmap in real time.
③LLM Performance MetricsAssess LLM response quality through API request count, TTFT/TPOT percentiles, and output token throughput.
④Tokens & CostTrack input/output token usage, average tokens per request, and today's cost.
⑤Model Comparison & AnalysisCompare performance, tokens, and errors across models and check request counts by status code.

Option Bar​

Set the lookup time, agent filter, screen mode (fixed/scroll), and preset in the option bar at the top of the dashboard. For detailed usage of each feature, see Dashboard Common Features.

Real-time Status​

Displays the real-time status of the application.

  • Active Transaction Speed — Displays requests per second (RPS), responses per second (TPS), and ongoing transaction count in real time.

  • LLM Active Status — Shows currently in-progress LLM calls categorized by type (chat, completion, etc.).

  • Active HTTPC — Classifies the real-time status of outbound HTTP requests to external LLM APIs by speed range (Very Slow, Slow, Normal).

  • Hitmap — Plots completed transactions as dots on a time axis (X) and elapsed time axis (Y). Dots higher on the chart indicate slower transactions.

LLM Performance Metrics​

Provides key LLM API performance metrics at a glance.

  • LLM API Request Count — Displays request count (bars) and error rate (line) by time period. If request spikes and error rate increases occur simultaneously, suspect Rate Limit exceeded or provider outage.

  • Percentile TTFT — Shows Time To First Token in p50, p75, p95, and p99 percentiles. High TTFT means longer initial wait time perceived by users.

  • Percentile TPOT — Shows Time Per Output Token in percentiles. High TPOT may cause streaming responses to stall or slow down.

  • Output Token Throughput — Displays output tokens per second (tok/s). A sudden drop in throughput may indicate provider-side performance degradation.

Tokens & Cost​

Tracks token usage and cost.

  • Token Usage — Shows input and output token usage over time in a trend chart.

  • Average Tokens per Request — Displays average input and output tokens per request. Useful for verifying prompt optimization effects.

  • Today's Cost — Displays today's cumulative cost and change compared to the previous day. Enables quick detection of cost anomalies.

Model Comparison & Analysis​

Compares performance, tokens, and errors across models.

  • Overall Performance Comparison — Compares TTFT p95 and TPOT p95 per model in a single chart. Useful for evaluating model replacement or version upgrade impact.

  • Token Usage Distribution — Compares input/output token usage per model by agent, provider, operation, and endpoint.

  • Error Rate Distribution — Compares error rates per model by agent, provider, operation, and endpoint. Quickly identify whether errors are concentrated in a specific model.

  • Request Count by Status Code — Displays request counts by HTTP status code (4xx, 5xx) over time.

Note

For dashboard common features such as widget editing, widget options, and presets, see Dashboard Common Features.

For chart types, filters, and usage of each widget, see LLM Dashboard Widgets.

Last updated on