User and Session Search
This menu aggregates LLM call logs into the three layers of user > session > turn. Because it shows the same logs grouped by user, you can follow the conversation history of a specific user in chronological order and pick out only the sessions with poor evaluation scores. It is intended for operators who check the quality and misuse of LLM services, and for response quality managers.
Log read permission is required. Without the permission, you cannot access the menu.
The menu is provided as a beta. The screen layout and labels may change when it becomes generally available.
Screen Layout
The screen has a three-panel layout: the top option bar, the user list on the left, and the main area on the right. The main area on the right is further divided into the session list at the top and the prompt group at the bottom.
Top Option Bar
-
Time range selection: Specify the start and end date and time. Quick selection options (last 5 minutes, 10 minutes, 30 minutes, 1 hour, 3 hours, 6 hours, and 12 hours) are provided, and you can specify the time down to seconds and milliseconds. The maximum range you can query is 3 days. If you exceed 3 days, the search is disabled.
-
Group by: Select the criterion by which the main area groups sessions.
-
User: Displays only the sessions of the user selected in the list on the left.
-
Session: Expands all sessions into a single list regardless of user. With this criterion, the user list on the left is hidden.
-
-
Filter: Enter a search query to query only the logs that meet a specific condition. You can save frequently used conditions as bookmarks.
-
Search button: Click the magnifier icon button to run the query. While the query is in progress, the button changes to a Cancel button. Changing the time or the filter runs the query again automatically, and clicking the search button runs the query again even when the conditions are unchanged.
User List on the Left
Displays the users that appear in the query range along with a risk badge. Selecting a user changes the main area on the right to that user's sessions.
-
User ID search: Enter part of a user ID to narrow the list.
-
Sort by: Select one of Risk, Cost, or Sessions.
-
Each item shows the user ID along with the session count and the turn count.
View Options
There are two switches at the top right of the main area.
-
Show personal information: Displays the original masked prompt. Personal information read permission is required.
-
Risky sessions only: Displays only the sessions whose risk is not Normal. This option is also reflected in the summary metric values.
Summary Metrics
Four cards aggregated based on the session range currently displayed. If the group-by criterion is User, the target is the sessions of the selected user; if it is Session, the target is all sessions.
| Item | Description |
|---|---|
| Sessions | Number of sessions displayed |
| Total turns | Sum of the turns included in the sessions |
| Total tokens | Sum of turn tokens |
| Total cost | Sum of turn costs ($). - if there is no turn with a cost |
Session List
Session summary cards are listed at the top of the main area. Clicking a card changes the prompt group at the bottom to the turns of that session.
| Item | Description |
|---|---|
| Risk | Session risk badge (Normal, Caution, High) |
| Session ID | Session identifier |
| Duration | From the start of the first turn to the end of the last turn of the session |
| Turns | Number of turns included in the session |
| Tokens | Sum of session tokens |
| Decision | Maximum overall score among the turns in the session |
If the group-by criterion is Session, a user chip is displayed on the card as well. Clicking the chip changes the group-by criterion to User and moves to that user's sessions.
Above the list, you can find the number of sessions displayed and a time sort direction switch. The sort direction switches between newest first and oldest first based on the session start time.
Prompt Group
Displays the turns of the selected session in ascending order of start time. A turn is the unit of one exchange between the user and the model. A single turn can include multiple LLM calls.
| Column | Description |
|---|---|
| Time | Turn start time (date and hour:minute:second.millisecond) |
Operation | Operation Type of the turn. - if there is none |
| Prompt | Input summary and response summary separated by the Input tag and the Output tag. If there is an evaluation result, an evaluation badge is displayed below |
| Model | List of models used within the turn. The provider is displayed on the line below |
| Tokens | Sum of turn tokens |
| Latency & cost | Turn latency (ms) and cost ($) |
Clicking the prompt area opens the transaction profile pop-out. The Input line and the Output line each open their own transaction. In the pop-out, you can check the original prompt and the token and cost details. For more information, see the LLM API Analysis document.
If you do not select a session, the first session in the list is selected automatically. If there is no session that meets the conditions, a message guiding you to clear the filter is displayed.
Risk Decision
Risk is determined based on the overall score of the evaluation feature. The threshold uses the same criteria as response quality evaluation, so the risk on this screen and the decision on the response quality screen do not contradict each other.
| Level | Description |
|---|---|
| Normal | The overall score is below the threshold. A turn with no evaluation result is also regarded as Normal |
| Caution | The overall score is at or above the caution threshold |
| High | The overall score is at or above the risk threshold |
The session risk is the worst risk among the turns that belong to the session, and the user risk is the worst risk among that user's session risks. In other words, if even one turn is High, both the session and the user are displayed as High.
For how to turn on the evaluation feature and collect overall scores, see the Evaluation document.
Query Behavior
This screen queries all logs in the specified time range. If it stopped after receiving only one page, the summary metrics and the session count would differ from the actual values, so it keeps receiving pages until the query is complete.
-
During the query, the progress state (number of logs collected, progress rate against the total, and number of pages) is displayed in place of the results.
-
If you click the Cancel button during the query, the aggregation uses only what has been received so far. In this case, a notice that the result is partial and a Retry button are displayed above the list.
-
If the query fails, a status notice and a Retry button are displayed in the same place.
-
When the query limit is reached, a Load more button appears along with a notice that only part of the data is displayed.
Ranges with a large volume of logs take time to query. Narrowing the query range or reducing the target with a filter shortens the time to completion.
Reference
Identifier Tags
| Tag | Type | Description |
|---|---|---|
user_id | string | User identifier |
session_id | string | Session identifier |
turn_id | string | Identifier of one exchange between the user and the model |
A session is distinguished by user_id and session_id together. Therefore, even if the same session_id appears for different users, they are counted as separate sessions.