LLM Token Trend Analysis
Home > Select Project > Analysis > Token Trend
Token Trend analyzes LLM API token usage patterns and processing efficiency on a time-series basis. While the dashboard focuses on real-time status monitoring, this page is used for trend comparison across time ranges and detailed analysis across models/agents.
Set the time range, filters, and search conditions in the top option bar, then query widget data. You can narrow the range by dragging on charts, or navigate to the Prompt Log page by clicking on charts/boxplots.
Token Usage
| Property | Value |
|---|---|
| Chart Type | Line chart (Input, Output -- 2 series) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the trend of Input token count and Output token count by time. Input tokens are tokens included in the prompt; Output tokens are response tokens generated by the model.
- If token usage spikes at a specific time, check for traffic changes, prompt modifications, or abnormal request influx.
- Changes in the Input/Output ratio reveal shifts in workload characteristics (input-heavy vs. generation-heavy).
- Use View individually to identify which model or agent concentrates token consumption.
I/O Ratio Trend
| Property | Value |
|---|---|
| Chart Type | Line chart (%, area fill) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
I/O Ratio is the percentage (%) of Output tokens out of total tokens (Input + Output). A 50% baseline (dotted line) is displayed.
- Above 50%: Response-generation-heavy workload. Tasks with long outputs such as summarization or code generation are dominant.
- Below 50%: Input-heavy workload. Tasks with long prompts such as classification, embedding, or RAG retrieval are dominant.
- A sudden ratio change may indicate workload pattern or prompt structure modifications -- investigate accordingly.
I/O Ratio = sum of output_tokens / (sum of input_tokens + sum of output_tokens) x 100
TTFT Percentiles
| Property | Value |
|---|---|
| Chart Type | Line chart (p50, p75, p95, p99 -- 4 series) |
| Filter | Tag filter |
| Action | View individually, View merged, View details, Select percentile |
Displays the percentile trend of TTFT (Time To First Token). If TTFT rises alongside a spike in token usage, it indicates that increased token volume is affecting initial response latency.
- If p50 is stable but p99 spikes, it signals intermittent queue wait or provider delays.
- When an anomalous interval is found, use the TTFT distribution comparison to identify which model is the cause.
TPOT Percentiles
| Property | Value |
|---|---|
| Chart Type | Line chart (p50, p75, p95, p99 -- 4 series) |
| Filter | Tag filter |
| Action | View individually, View merged, View details, Select percentile |
Displays the percentile trend of TPOT (Time Per Output Token). Viewing alongside TTFT Percentiles helps distinguish between "slow first token and slow generation" intervals and "fast first token but slow generation" intervals.
- A p99 spike means intermittent requests where token generation stalls or significantly slows down.
- When an anomalous interval is found, use the TPOT distribution comparison to identify which model is the cause.
TTFT Distribution Comparison
| Property | Value |
|---|---|
| Chart Type | Boxplot (x-axis: model) |
| Filter | Tag filter (always visible) |
| Action | View details |
Compares the distribution of TTFT by tag (model, etc.) using boxplots. The box area represents p25-p75 (middle 50% range), the center line is the median, and whiskers indicate min-max values.
- Models with a low median and narrow box provide the most stable and fast initial response.
- Models with long whiskers experience intermittent extreme initial delays.
- Click a model to navigate to its prompt logs.
TPOT Distribution Comparison
| Property | Value |
|---|---|
| Chart Type | Boxplot (x-axis: model) |
| Filter | Tag filter (always visible) |
| Action | View details |
Compares the distribution of TPOT by tag (model, etc.) using boxplots.
- Models with a low median and narrow box provide the most stable streaming performance.
- Models with long whiskers experience intermittent significant slowdowns in token generation.
Output Token Throughput
| Property | Value |
|---|---|
| Chart Type | Line chart |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
A throughput metric representing output tokens generated per second (Tokens Per Second).
- For intervals where throughput drops, check for model response bottlenecks or Rate Limit effects.
- Viewing alongside token usage helps identify intervals where "many tokens are used but processing speed is slow."
- Use View individually to compare throughput by model.
Cache Hit Rate Trend
| Property | Value |
|---|---|
| Chart Type | Line chart (%, area fill) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the percentage (%) of tokens retrieved from cache out of total input tokens, by time.
- Higher hit rates indicate effective prompt caching with greater cost savings.
- If the hit rate drops suddenly, check for prompt content changes, cache TTL expiration, or new types of request influx.
- Use View individually to compare cache efficiency by model and identify where caching is most effective.
Cache Hit Rate = (sum of cached_tokens / sum of input_tokens) x 100
Average Tokens per Request
| Property | Value |
|---|---|
| Chart Type | Line chart (Input, Output -- 2 series, area fill) |
| Filter | Tag filter |
| Action | View individually, View merged, View details |
Displays the time-based trend of average Input/Output token count per request.
- A spike in Input tokens per request may indicate longer prompts or inclusion of longer context.
- A spike in Output tokens per request means longer responses, directly impacting cost increases.
- If this widget's values increase alongside token usage, it means request size has grown. If only token usage increases, it simply means request count has risen.
Token Analysis
| Property | Value |
|---|---|
| Chart Type | Table |
| Filter | Agent / Model / Provider tab switch |
| Action | View details, CSV download, Column settings |
Organizes key token usage, cache, and cost metrics by tag (Agent, Model, Provider) in a table.
| Column | Description |
|---|---|
Name | Tag value (agent name, model name, provider name) |
Requests | Request count |
Total Input | Total input tokens |
Total Output | Total output tokens |
Total Tokens | Total token count |
I/O Ratio (%) | Output token ratio |
Cached Tokens | Total cached tokens |
Cache Hit (%) | Cache hit rate |
Cached Cost ($) | Cached token cost |
Input Cost ($) | Input cost |
Output Cost ($) | Output cost |
Total Cost ($) | Total cost |
- Items with high
Total Tokensaccount for the largest share of total cost and are priority optimization targets. - Items with high
I/O Ratioare output-heavy workloads, significantly affected by Output token pricing. - Items with low
Cache Hitshould have their caching strategy reviewed for potential cost savings. - Sort by
Total Costto quickly determine cost optimization priorities. - Use the download button in the upper right to export current data as CSV.
- Use column settings to select which columns to display.
Reference
Difference from Dashboard
| Item | LLM Dashboard | Token Trend |
|---|---|---|
| Time Mode | Real-time (Live) + past point | Past range (Range) only |
| Primary Purpose | Real-time status monitoring | Trend analysis across time ranges |
| Filter | Agent selection | Complex condition filter (Agent, Model, Provider, Operation, URL) |
| Prompt Log Integration | None | Navigate to Prompt Log on chart click |
| Data Export | None | CSV download supported |