Skip to main content

LLM Dashboard

Home > Select Project > Dashboard > LLM Dashboard

The LLM Dashboard is a menu that provides real-time monitoring of LLM (Large Language Model) API request volume, response performance, token usage, cost, and error status in a single view. The top of the screen shows the real-time status (Active Transaction Speed, LLM Active Status, Active LLM API, Active Trajectory, Hitmap) along with Percentile TTFT and Percentile TPOT. The middle row contains LLM API Request Count, Token Usage Distribution, Overall Performance Comparison, and Output Token Throughput, and the bottom row contains Token Usage, Average Tokens per Request, and Today's Cumulative Cost, so you can progress from status overview to root cause analysis. This layout is saved as the default preset and is ready to use immediately without additional configuration.

LLM Dashboard

The LLM Dashboard consists of five areas based on the default preset.

AreaNameDescription
①Option BarSet the lookup time, application filter, screen mode, and preset.
②Real-time StatusCheck the current status with Active Transaction Speed, LLM Active Status, Active LLM API, Active Trajectory, and Hitmap.
③LLM Performance MetricsAssess response performance through Percentile TTFT, Percentile TPOT, LLM API Request Count, and Output Token Throughput.
④Model ComparisonCheck differences between models with Token Usage Distribution and Overall Performance Comparison.
⑤Tokens & CostTrack Token Usage, Average Tokens per Request, and Today's Cumulative Cost.

Option Bar​

Set the lookup time, application filter, screen mode (fixed/scroll), and preset in the option bar at the top of the dashboard. For detailed usage of each feature, see Dashboard Common Features.

Chart widgets query the last 10 minutes based on the selected time. Active Transaction Speed, LLM Active Status, Active LLM API, and Active Trajectory show the status in progress at the selected time, and Today's Cumulative Cost shows the cumulative cost from 00:00 of that day to the selected time. In real-time mode, the widgets refresh every 5 seconds.

Real-time Status​

Displays the real-time status of the application.

  • Active Transaction Speed — Displays requests per second (RPS), responses per second (TPS), and ongoing transaction count in real time.

  • LLM Active Status — Shows currently in-progress LLM calls categorized by operation type.

  • Active LLM API — Displays the LLM API calls running right now as dots on a track for each model and divides them into Normal, Slow, and Very Slow ranges by elapsed time. Clicking a model row opens the list of running calls for that model.

  • Active Trajectory — Displays the number of trajectories in progress and their distribution by Light, Middle, and Heavy level. In real-time mode, clicking the orbit chart opens the list of all trajectories, and clicking a level opens the list for that level. In the list, you can stop a running trajectory with the Stop button (project edit permission required). For details, see Trajectory View and Stop.

  • Hitmap — Plots completed transactions as dots on a time axis (X) and elapsed time axis (Y). Dots higher on the chart indicate slower transactions. Dragging an area on the Hitmap opens the transaction list for that range and the LLM trace details in a new window.

LLM Performance Metrics​

Provides key LLM API performance metrics at a glance.

  • LLM API Request Count — Displays request count (bars) and error rate (line) by time period. If request spikes and error rate increases occur simultaneously, suspect Rate Limit exceeded or provider outage.

  • Percentile TTFT — Shows Time To First Token in p50, p75, p95, and p99 percentiles. High TTFT means longer initial wait time perceived by users.

  • Percentile TPOT — Shows Time Per Output Token in percentiles. High TPOT may cause streaming responses to stall or slow down.

  • Output Token Throughput — Displays output tokens per second (tok/s). A sudden drop in throughput may indicate provider-side performance degradation.

Model Comparison​

Compares performance and tokens across models.

  • Overall Performance Comparison — Compares TTFT p95 and TPOT p95 per model in a single chart. Useful for evaluating model replacement or version upgrade impact.

  • Token Usage Distribution — Compares input/output token usage per model by application, provider, operation, and endpoint.

Tokens & Cost​

Tracks token usage and cost.

  • Token Usage — Shows input and output token usage over time in a trend chart.

  • Average Tokens per Request — Displays average input and output tokens per request. Useful for verifying prompt optimization effects.

  • Today's Cumulative Cost — Displays today's cumulative cost and change compared to the previous day. Enables quick detection of cost anomalies.

Note

For dashboard common features such as widget editing, widget options, and presets, see Dashboard Common Features.

For chart types, filters, and usage of each widget, see LLM Dashboard Widgets.

Error Rate Distribution and Request Count by Status Code are available in the LLM API performance dashboard.

Last updated on