Evaluation and Operation Type

WhaTap LLM Observability provides the Evaluation feature, which automatically measures the response quality and stability of the LLM application you operate, and Operation Type labeling, which groups and filters LLM calls by system prompt. Both features are designed not to affect the response time of user transactions, and you can apply them by adding a single decorator line to your code.

Note

As of June 2026, the Evaluation and Operation Type labeling features support Python only.

Evaluation

Evaluation automatically applies evaluations such as hallucination, answer relevance, toxicity, prompt injection, factuality, PII leak, and suspicious URL to LLM responses. The evaluation results are produced as scores (0.0 to 1.0) and collected together with metrics and the logsink, so you can track response quality as a time series and configure threshold-based alerts.

Note

The judge LLM calls reuse the same client instance used by the user calls. Therefore, when you enable the evaluation pipeline, additional LLM resources (tokens and cost) are consumed for the judge calls, separately from the LLM calls that serve user responses.

Configuration options

Key	Default	Description
llm_eval_enabled	false	Master toggle of the evaluation pipeline. If false, queueing and workers do not run.
llm_eval_sample_rate	1.0	Sampling ratio of the evaluation (1.0 == a ratio of 100%).
llm_eval_buffer_limit	1000	Maximum size of the evaluation queue. When exceeded, evaluations are dropped and the LLM030 warning occurs.
llm_eval_workers	4	Number of workers in the ThreadPoolExecutor that runs the evaluators.
llm_eval_judge_timeout_sec	30	Maximum wait time (seconds) for one judge LLM call. When exceeded, it is handled as judge_error. 0 or a negative value means unlimited.

How to use

For Evaluation, you can choose between the decorator method, which applies it only to specific functions, and the method that applies it to the entire application.

Method A — Decorator (evaluates only the LLM calls of a specific function)

from whatap.llm.evaluators import evaluate_with
from whatap.llm.evaluators.builtins import CombinedJudgeEvaluator

@evaluate_with(CombinedJudgeEvaluator())
def chat(q):
    ...

Method B — App-wide registration (always-on for all LLM calls)

from whatap.llm.evaluators import register_evaluator
from whatap.llm.evaluators.builtins import CombinedJudgeEvaluator

register_evaluator(CombinedJudgeEvaluator())  # Call only once at app startup

Evaluator classes you can register

from whatap.llm.evaluators.builtins import (
    # LLM judge based — judge call cost is incurred
    CombinedJudgeEvaluator,
    HallucinationEvaluator,
    AnswerRelevanceEvaluator,
    ToxicityEvaluator,
    PromptInjectionEvaluator,
    FactualityEvaluator,
   
    # Rule based — no LLM call, zero cost
    PIILeakEvaluator,
    URLScanEvaluator,
)

Evaluation class descriptions

Evaluator	LABEL	Type	Score direction	Cost
CombinedJudgeEvaluator	combined_judge	LLM judge	Overall risk — lower is better	5 aspects with 1 judge call
HallucinationEvaluator	hallucination	LLM judge	Lower is better	1 judge call
AnswerRelevanceEvaluator	answer_relevance	LLM judge	Higher is better	1 judge call
ToxicityEvaluator	toxicity	LLM judge	Lower is better	1 judge call
PromptInjectionEvaluator	prompt_injection	LLM judge	Lower is better	1 judge call
FactualityEvaluator	factuality	LLM judge	Higher is better	1 judge call

Each Evaluation measures the following.

• answer_relevance: Answer relevance (higher is better)

  Measures how faithfully the response answers the user question. It is scored 1.0 when the question is fully answered, 0.5 when it is answered only partially or tangentially, and 0.0 when it is unrelated to the question or evades or deviates from it.

• hallucination: Hallucination (lower is better)

  Measures whether the response contains unsupported claims. When a context (ground truth) is given, it is evaluated based on faithfulness to that context; when there is no context, it is evaluated based on the self-consistency of the response itself. 0.0 means completely faithful to the context, and 1.0 means entirely fabricated.

• toxicity: Toxicity (lower is better)

  Determines whether the response contains harmful content across six categories — hate, harassment, violence, sexual, self_harm, and profanity — and also returns the list of detected categories. 0.0 means completely safe, and 1.0 means severely harmful.

• prompt_injection: Prompt injection (lower is better)

  Determines whether an override attempt of the "ignore previous instructions" kind included in the user input succeeded, or whether the response leaked the system prompt, hidden instructions, or confidential information. 0.0 means the original task was performed as is and no protected information was exposed, and 1.0 means it was completely hijacked by the injection.

• factuality: Factuality (higher is better)

  Measures the factual accuracy of the response. It is based on verifiable objective claims such as historical, scientific, geographical, and mathematical facts, as well as dates, names, and numbers, and excludes opinions and hedged expressions from evaluation. 1.0 means all factual claims are accurate, and 0.0 means many clearly false claims are included. Unlike hallucination, which looks at faithfulness to the context, factuality looks at the factual accuracy of the response itself regardless of the context.

Combined Judge

The number of judge LLM calls varies greatly depending on which evaluators you register.

**Registering individual evaluators (5 calls)**When you register hallucination, answer_relevance, toxicity, prompt_injection, and factuality separately, a separate judge LLM call occurs for each evaluation, resulting in 5 calls per response.

Registering combined_judge (1 call, recommended by default)CombinedJudgeEvaluator bundles the 5 aspects into a single combined evaluation request, calls the judge LLM only once, and receives all 5 aspect scores as the result. It can reduce the judge call cost by about 80% compared with registering 5 individual evaluators, so we recommend using it by default.

Overall risk score (combined_judge)

combined_judge combines the 5 aspect scores into a single overall risk. At this point, the two groups with different score directions are unified on a risk basis.

• Risk-direction aspects (hallucination, toxicity, prompt_injection) — A higher score means higher risk, so the score is used as the risk as it is.
• Quality-direction aspects (answer_relevance, factuality) — A higher score is better, so 1 − score is used as the risk.

Assuming each aspect is an independent risk, the overall risk is calculated as the complement of "the probability that all aspects are safe." This is the Probabilistic OR formula from statistics.

Calculation example

For risks of [0.7, 0.3, 0.1, 0.1, 0.2]: 1 − (0.3 × 0.7 × 0.9 × 0.9 × 0.8) = 0.864

Characteristics of risk combination (simulation)

Individual risks	Maximum (max)	Combined (compound)
[0.5, 0, 0, 0, 0]	0.50	0.50
[0.5, 0.5, 0, 0, 0]	0.50	0.75
[0.3, 0.3, 0.3, 0.3, 0.3]	0.30	0.83
[1.0, 0, 0, 0, 0]	1.00	1.00

The overall risk is always greater than or equal to the simple maximum. Even low risks raise the overall risk as they accumulate across several aspects, and if any single aspect is 1.0, the overall risk also becomes 1.0 regardless of the others. This effectively catches "responses that are slightly bad in several areas at once," which a single metric easily misses.

Operation Type

Operation Type is a user-defined label for grouping and filtering LLM calls by prompt, chain, or agent. The label you specify becomes part of the metric dimension, so you can analyze latency, cost, and evaluation scores separately by prompt.

Configuration keys

Key	Meaning	Default
operation_type	Name of the prompt, chain, or agent	'default'
prompt_version	Prompt version	'v1'

How to use

(a) Decorator (recommended)

from whatap.llm import prompt_meta

@prompt_meta(operation_type='checkout_chain', prompt_version='v3')
def checkout(question):
    return client.chat.completions.create(...)

(b) Context manager

from whatap.llm import prompt_meta_scope

def handler(req):
    with prompt_meta_scope(operation_type='greeting', prompt_version='v2'):
        return client.chat.completions.create(...)

Query the currently active meta (in custom logic)

from whatap.llm import get_prompt_meta

op_type, version = get_prompt_meta()  # Returns ('default', 'v1') if there is no scope