Docs
Contents

Metrics Monitoring and Analytical Reports

Feature Overview

Metrics & Analytical Reports (Console entries: Agent Ops → Metrics and Reports) are Agentbase's (v2.10.0) production operations measurement and ROI decision dashboards for technical leads, AI architects, and operations teams. After an LLM application is deployed at scale, teams need real-time visibility into system throughput and financial health to answer three core questions: "How many tokens and how much cost were spent?", "How fast are first-token latency (TTFT) and end-to-end latency (P50/P95/P99)?", and "What is the answer quality of each agent and the distribution of Bad Case attribution?"

In the example project 企业智能客服与知识中台 (enterprise-ai-hub-210) under Agentbase 企业智能中台 (69c6d1aa00062e6bb60), the system derives full-dimension metrics and three thematic analytical reports (/v1/observability/reports/{overview|agents|quality}) in real time from the collected Traces, Observations, and Scores:


Core Concepts and Architecture

Agentbase's metrics and reporting engine is built directly on the underlying analytical trace and score storage, using server-side Bundle merge queries and a two-stage Top-N time-series aggregation architecture, so a single page renders multi-dimensional metrics with just one HTTP request.

Metrics Monitoring and Analytical Reports: workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

Core Metrics and Calculation Rules Table

Metric categoryMetric nameCalculation rules and statistics
Throughput and activityTotal Traces / Total Observations / Active usersDeduplicated counts within the selected time window of end-to-end requests, internal nodes, and non-empty userId (e.g., user_zhangwei_001).
Token consumptionInput (Prompt) / Output (Completion) / Total TokensAggregates real vendor-reported tokens from all GENERATION and EMBEDDING nodes. cached (cache-hit tokens) and reasoning (reasoning chain tokens) map to mutually exclusive sub-buckets and are not double-counted into total billable tokens; calls directly short-circuited by the cache are recorded as avoided call (with 0 new consumption).
Latency percentilesEnd-to-end latency and TTFT (P50 / P95 / P99)- End-to-end latency: the 50th, 95th, and 99th percentile values within the window of the total duration of a Trace or Generation from startTime to endTime;
- First-token latency (TTFT, Time To First Token): the duration percentiles of streaming LLM calls from request initiation to receiving the first generated token (completionStartTime).
Cost estimationTotal model cost estimate / Cost per SessionEstimated amounts computed from the model unit price table matched by each GENERATION node (e.g., qwen-plus, qwen-vl-max) (for directional R&D and operations reference only; actual financial settlement is governed by the platform's quotas and metering bills).
Error rate and qualityError rate / Overall quality score / Evaluation coverage- Error rate: the proportion of Traces/Observations with status ERROR (the gateway layer strictly distinguishes 5xx server-side failures, 429 rate limiting, and ordinary 4xx business rejections, and does not mix ordinary 4xx responses into the service failure rate);
- Overall quality score: the arithmetic mean (rounded to two decimal places) of all normalizable numeric scores (normalizedValue) within the window;
- Evaluation coverage: the percentage of Traces linked to at least one evaluation Score out of the total number of Traces.

Roles, Entry Points, and Relationships

Console Entry and REST API Permission Matrix

  1. Metrics monitoring page (Metrics): located in the left navigation under Agent Ops (Observability / AgentOps) → Metrics (route: /console/project-{region}-enterprise-ai-hub-210/observability/metrics).

  2. Analytical reports page (Reports): the sidebar reports entry is currently hidden by default; the pages and routes are fully preserved, and project members with permission can access the three tabs via direct URLs:

    • /console/project-{region}-enterprise-ai-hub-210/observability/reports/overview

    • /console/project-{region}-enterprise-ai-hub-210/observability/reports/agents

    • /console/project-{region}-enterprise-ai-hub-210/observability/reports/quality

API / pageHTTP method and pathRequired API Key scopes (included by default in user sessions/JWT roles)
Metrics query/v1/observability/metrics (and Console aggregate query)observability.metrics.read
Overview global reportPOST /v1/observability/reports/overviewobservability.metrics.read + observability.traces.read + observability.scores.read
Agents agent ranking reportPOST /v1/observability/reports/agentsthe three above + observability.users.read + observability.sessions.read
Quality and attribution reportPOST /v1/observability/reports/qualityobservability.metrics.read + observability.traces.read + observability.scores.read

Feature Details and Operation Guide

1. View the Metrics page and the memoryOnly-exclusive filter

  1. Log in to the Agentbase Console, go to project 企业智能客服与知识中台 (enterprise-ai-hub-210), and click AgentOps → Metrics in the left sidebar.

Metrics detail page (reusing the registered screenshot)

Interface reference: reuses previously registered historical screenshots; demo data may differ from the current project; no new screenshots were taken this time.

  1. Key filtering and analysis actions:

    • Time range and environment filter: select the statistics time period and target environment (e.g., production);

    • memoryOnly-exclusive filter (Memory chain only): enable the Count Memory only (memoryOnly: true) filter on the page; the dashboard will aggregate only trace data carrying the Memory chain identifier, helping architects separately account for the Tokens, P95 latency, and Memory-specific evaluation scores consumed by Memory fact extraction, profile generation, reranking, and experience distillation;

    • Break down usage and cost by model: in the model detail table, compare the call counts, Prompt/Completion Token proportions, and estimated costs of qwen-plus, qwen-turbo, text-embedding-v3, qwen3.7-text-rerank, and qwen-vl-max.

2. Use the three analytical reports (/v1/observability/reports) for global insights and ROI

When visiting the report pages or calling POST /v1/observability/reports/{overview|agents|quality}, the three tabs share a unified time range contract:

(1) Overview (global overview tab)

(2) Agents (agent ranking and ROI tab, displayed as Trace in the Console tab)

(3) Quality (quality trend and bad case attribution tab)

3. Write-side explicit metadata tagging specification (wasteReason, badCaseCategory, cache)

The Token waste breakdown, cache hit rate, and bad case attribution panels in the reports strictly aggregate based on fields explicitly tagged on the write side in Observation.metadata (untagged normal data is not mixed into the “unknown waste” bucket):

Metadata field pathAllowed standard enum valuesEnum semantics and improvement guidance
metadata.wasteReasonretry, truncation, cache_miss_repeat, context_overflow, user_abortRespectively identify invalid Token consumption caused by error-triggered retries, reruns after output-length truncation, identical-prefix cache misses, context-overlimit discards, and user mid-stream cancellation. A missing field indicates normal, valid consumption.
metadata.badCaseCategorygrounding_missing, hallucination, instruction_violation, tool_failure, query_understanding, safety_refusal, latency_timeoutSeven mutually exclusive bad case attribution buckets: e.g., grounding_missing points to optimizing knowledge base retrieval, and tool_failure points to fixing tool schemas or downstream APIs.
metadata.cache.statushit, partial, missThree-state cache status (e.g., matching the AI gateway semantic cache policy cp-semantic-cache or Prefix Cache): hit (full hit), partial (system Prompt hit but user turns missed), miss (miss). The cache hit rate is computed as the proportion of tagged Observations.

4. Examples of calling the analytical report Bundle API

export APPBASE_ENDPOINT="https://appbuild.oceanbase.com/v1"
export APPBASE_PROJECT_ID="enterprise-ai-hub-210"
export APPBASE_API_KEY="<APPBASE_API_KEY>"

# 1. Query the Overview global report for the last 7 days (7d)
curl -fsS -X POST "${APPBASE_ENDPOINT}/observability/reports/overview" \
  -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
  -H "X-Appwrite-Key: ${APPBASE_API_KEY}" \
  -H "Content-Type: application/json" \
  --data '{
    "range": "7d"
  }' | jq .

# 2. Query Quality and Bad Case attribution reports with a custom time window (from/to, span <= 30 days)
curl -fsS -X POST "${APPBASE_ENDPOINT}/observability/reports/quality" \
  -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
  -H "X-Appwrite-Key: ${APPBASE_API_KEY}" \
  -H "Content-Type: application/json" \
  --data '{
    "from": "2026-10-01T00:00:00.000Z",
    "to": "2026-10-22T23:59:59.999Z"
  }' | jq .

Querying metrics

  1. Open the metrics page via Observability / AgentOps → Metrics in the project sidebar.

  2. Select the data view, metric, aggregation method, grouping dimension, time granularity, and start/end times, then click Apply. The default query covers the last 24 hours, with time granularity defaulting to Hour; available granularities are Minute, Hour, Day, and Week.

  3. View the summary values, time-series charts, and dimension breakdown tables. After selecting a grouping, you can compare changes across dimensions; when you need to inspect a Trace's input/output or specific stages, return to the Trace to check details.

Available metrics and groupings depend on the data view:

Data viewMetricsAvailable groupings
TracesCount, latency, Token usage, costNo grouping, Trace name, Environment
ObservationsCount, latency, Token usage, costNo grouping, model, Trace name, Environment, stage, purpose, result, usage completeness
ScoresCount, scoreNo grouping, score name, Trace name, Environment

Aggregation options adjust based on the selected metric: Count uses Count; latency supports Count, Average, P50, P95; Token usage and cost support Sum and Average; scores support Count and Average. Select the data view and metric first, then choose from the aggregation options provided on the page.

Memory only is available only in the Observations view and is used to count only Observations produced by the Memory chain; the Traces and Scores views do not show this filter. It does not mean that scores are also filtered by Memory.

When there is no metric data

If the chart shows No metric data, first check whether the start/end times cover the period when the data was produced, then confirm whether the data view, metric, and grouping match; when using Memory only, also confirm that the query target is Observations. After changing the conditions, click Apply again. The page's refresh button re-issues the current metrics query.

If there is still no data after adjusting the conditions, verify that the current project has written Traces, Observations, or Scores of the corresponding type, and check the Trace's time and usage records. An empty result only means the current query returned no data; it cannot be used to conclude “no calls”, “zero cost”, or “zero quality”.

FAQ and Troubleshooting

1. Why do the Reports pages show data for total calls and Tokens, but the “Token waste” or “Bad case attribution” panels appear empty?

2. Why does clicking “Apply” for a custom report time range fail or become disabled?

3. Do the model costs shown in Metrics and Reports exactly match the actual charges in the organization's bill?