Skip to main content

LLM API Performance Dashboard

LLM API Performance Dashboard is a menu that shows the request volume, response performance, tokens, cost, and errors of LLM API calls on one screen. Widgets are arranged from top to bottom in the order running calls and response time → request volume and streaming performance → tokens and cost → distribution and errors, so you can first check whether there are slow calls now and then drill down to the metrics that explain the cause.

In the left sidebar, click Dashboard > LLM API Performance Dashboard.

Note

This dashboard is read-only. You cannot add, delete, or move widgets, and presets are not available. To configure widgets yourself, use the Overview Dashboard. You can look up each widget placed here in the Widget Reference.

Screen layout​

LLM API Performance Dashboard

The LLM API Performance Dashboard consists of the option bar and 13 widgets placed in 4 rows below it.

AreaWidgetDescription
①Option barAt the top, set the query time and application filter.
②Active LLM API · Latency PercentileThe first row shows LLM API calls running at the query time and response time percentiles.
③LLM API Request Count · TPOT Percentile · Output Token Throughput · TTFT PercentileThe second row shows request volume and error rate, token generation speed, throughput, and time to first token.
④Token Usage · Average Tokens per Request · Today's Cumulative CostThe third row shows token usage trends and today's cumulative cost.
⑤Cache Hit Rate · Token Usage Distribution · Error Rate Distribution · Request Count by Status CodeThe bottom row shows cache efficiency, token distribution by criterion, and error distribution.

Option bar​

Set the query time and application filter in the option bar at the top. For details on each feature, see Dashboard Common Features.

  • The query range is the last 10 minutes from the selected time. Today's Cumulative Cost is an exception and shows data from 00:00 today to the query time.
  • In real-time mode, the range moves forward every 5 seconds and the widgets refresh with it. If you pause real-time monitoring or specify a past time, the 10-minute range at that point is fixed.
  • You can narrow the targets with the application filter. In real-time mode, filter changes are applied immediately. If you change the filter while real-time monitoring is paused, the Apply button appears, and the change takes effect only after you click it.
  • Total, Active, and Inactive to the right of the Application selector are the number of applications shown in the list and the number of active and inactive applications among them. Inactive appears only when Show inactive SDKs is on, and if the list shows types or servers instead of applications, Active and Inactive do not appear.

Running calls and response time​

  • Active LLM API: Shows the LLM API calls running at the query time as dots on a track for each model. Each dot is one call, and its horizontal position indicates the Normal, Slow, or Very Slow range divided by the criteria set in the SDK options. The number of calls is shown under the model name, and if not all dots fit in a range, the number of undrawn dots is shown next to it as (+N). If there are too many models to fit the widget height, the remaining models are grouped into one last line shown as N other models. Click a model line to open the running call list for that model, and select a call to move to the trajectory view that the call belongs to. Click the Details icon in the widget header to open the running call list for all models on a wide screen. This icon is shown only in real-time mode.
  • Latency Percentile: Shows p50, p75, p95, and p99 trends of LLM API response time (Latency). Latency is the total time from when a request is sent until the response is fully completed. A large gap between p50 and p99 means that most requests are fast but some requests suffer severe delays.

The running call list shows calls at the current time regardless of the dashboard query time. For the list layout, see Active LLM API in the Widget Reference. For how to check the execution flow in the trajectory view and stop it, see Trajectory View and Stop.

Request volume and streaming performance​

  • LLM API Request Count: Shows the number of requests (bars) and error rate (line) over time. If request volume spikes along with the error rate, suspect a rate limit being exceeded or a provider outage.
  • TPOT Percentile: Shows p50, p75, p95, and p99 trends of the time taken to generate one output token (Time Per Output Token).
  • Output Token Throughput: Shows output tokens per second (tok/s). In ranges where throughput drops, check for model response bottlenecks or rate limit effects.
  • TTFT Percentile: Shows p50, p75, p95, and p99 trends of the time until the first token arrives (Time To First Token).

Tokens and cost​

  • Token Usage: Shows input and output token usage over time.
  • Average Tokens per Request: Shows the trend of average input and output tokens per request over time.
  • Today's Cumulative Cost: Shows the cumulative cost from 00:00 today to the query time, and the change compared to the same time period on the comparison date. The comparison date is 1 day ago by default and can be changed in the Compare selector in the widget header. Amounts under $1 are shown in ¢.

Distribution and errors​

  • Cache Hit Rate: Shows the ratio (%) of tokens served from cache to all input tokens over time. It is initially shown in the merged view. Click the Merged / Individual view icon in the widget header to switch to the individual view, which splits the data by application, and use the criterion buttons that appear in the header to change the criterion.
  • Token Usage Distribution: Compares total input and output tokens by criterion with bars. The default criterion is model.
  • Error Rate Distribution: Compares error rate (%) by criterion with horizontal bars. The default criterion is model.
  • Request Count by Status Code: Shows the number of requests over time split into HTTP status codes 4xx and 5xx. Successful (2xx) requests are not shown.

For Token Usage Distribution and Error Rate Distribution, you can change the comparison criterion in the widget header to one of Application, Model, Provider, Operation, and Endpoint. If the widget is narrow, some criterion buttons are collapsed into a button that shows the remaining count, such as +3.

LLM API Request Count, Token Usage, Average Tokens per Request, and Output Token Throughput are also initially shown in the merged view and can be switched to the individual view by clicking the Merged / Individual view icon. The initial criterion is then Model, and you can choose another criterion with the criterion buttons in the header.

Tip

Error Rate Distribution and Request Count by Status Code are not included in the default preset of the Overview Dashboard. Check error status on this dashboard, or add the two widgets to the Overview Dashboard.

Last updated on