Docs
Contents

Metrics Monitoring and Analytical Reports

Feature Overview

Metrics & Analytical Reports (Console entry: Agent Ops → Metrics; report visibility depends on deployment configuration) are Agentbase's production operations measurement and ROI decision dashboards for technical leads, AI architects, and operations teams. After an LLM application is deployed at scale, teams need real-time visibility into system throughput and financial health to answer three core questions: "How many tokens and how much cost were spent?", "How fast are first-token latency (TTFT) and end-to-end latency (P50/P95/P99)?", and "What is the answer quality of each agent and the distribution of Bad Case attribution?"

In the example project 企业智能客服与知识中台 (enterprise-ai-hub-210) under Agentbase 企业智能中台 (69c6d1aa00062e6bb60), the system derives full-dimension metrics and three thematic analytical reports (/v1/observability/reports/{overview|agents|quality}) in real time from the collected Traces, Observations, and Scores:


Core Concepts and Architecture

Agentbase's metrics and reporting engine is built directly on the underlying analytical trace and score storage, using server-side Bundle merge queries and a two-stage Top-N time-series aggregation architecture, so a single page renders multi-dimensional metrics with just one HTTP request.

Metrics Monitoring and Analytical Reports: workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

Core Metrics and Calculation Rules Table

Metric categoryMetric nameCalculation rules and statistics
Throughput and activityTotal Traces / Total Observations / Active usersDeduplicated counts within the selected time window of end-to-end requests, internal nodes, and non-empty userId (e.g., user_zhangwei_001).
Token consumptionInput (Prompt) / Output (Completion) / Total TokensAggregates real vendor-reported tokens from all GENERATION and EMBEDDING nodes. cached (cache-hit tokens) and reasoning (reasoning chain tokens) map to mutually exclusive sub-buckets and are not double-counted into total billable tokens; calls directly short-circuited by the cache are recorded as avoided call (with 0 new consumption).
Latency percentilesEnd-to-end latency and TTFT (P50 / P95 / P99)- End-to-end latency: the 50th, 95th, and 99th percentile values within the window of the total duration of a Trace or Generation from startTime to endTime;
- First-token latency (TTFT, Time To First Token): the duration percentiles of streaming LLM calls from request initiation to receiving the first generated token (completionStartTime).
Cost estimationTotal model cost estimate / Cost per SessionEstimated amounts computed from the model unit price table matched by each GENERATION node (e.g., qwen-plus, qwen-vl-max) (for directional R&D and operations reference only; actual financial settlement is governed by the platform's quotas and metering bills).
Error rate and qualityError rate / Overall quality score / Evaluation coverage- Error rate: the proportion of Traces/Observations with status ERROR (the gateway layer strictly distinguishes 5xx server-side failures, 429 rate limiting, and ordinary 4xx business rejections, and does not mix ordinary 4xx responses into the service failure rate);
- Overall quality score: the arithmetic mean (rounded to two decimal places) of all normalizable numeric scores (normalizedValue) within the window;
- Evaluation coverage: the percentage of Traces linked to at least one evaluation Score out of the total number of Traces.

Roles, Entry Points, and Relationships

Console Entry and REST API Permission Matrix

  1. Metrics monitoring page (Metrics): located in the left navigation under Agent Ops (Observability / AgentOps) → Metrics (route: /console/project-{region}-enterprise-ai-hub-210/observability/metrics).

  2. Analytical reports page (Reports): the entry and page depend on deployment visibility settings. When reports are hidden, direct links also redirect; check deployment configuration and permissions if access is needed.

API / pageHTTP method and pathRequired API Key scopes (included by default in user sessions/JWT roles)
Metrics query/v1/observability/metrics (and Console aggregate query)observability.metrics.read
Overview global reportPOST /v1/observability/reports/overviewobservability.metrics.read
Agents agent ranking reportPOST /v1/observability/reports/agentsobservability.metrics.read + observability.traces.read + observability.users.read + observability.sessions.read
Quality and attribution reportPOST /v1/observability/reports/qualityobservability.metrics.read + observability.traces.read + observability.scores.read

Feature Details and Operation Guide

1. View the Metrics page and the memoryOnly-exclusive filter

  1. Log in to the Agentbase Console, go to project 企业智能客服与知识中台 (enterprise-ai-hub-210), and click AgentOps → Metrics in the left sidebar.

Metrics page

Figure: Metrics page in a 2.12.0 test project; contents vary by project.

  1. Key filtering and analysis actions:

    • Time range and environment filter: select the statistics time period and target environment (e.g., production);

    • memoryOnly filter: enable this option in the Observations view to add a service=memory filter to the console query. Traces and Scores do not offer this filter; do not assume that scores have also been filtered by Memory;

    • Break down usage and cost by model: in the model detail table, compare the call counts, Prompt/Completion Token proportions, and estimated costs of qwen-plus, qwen-turbo, text-embedding-v3, qwen3.7-text-rerank, and qwen-vl-max.

2. Use the three analytical reports (/v1/observability/reports) for global insights and ROI

When the reports page is available, or when making an authorized POST /v1/observability/reports/{overview|agents|quality} request, queries share a unified time range contract:

(1) Overview (global overview tab)

(2) Agents (agent ranking and ROI tab, displayed as Trace in the Console tab)

(3) Quality (quality trend and bad case attribution tab)

3. Write-side explicit metadata tagging specification (wasteReason, badCaseCategory, cache)

The Token waste breakdown, cache hit rate, and bad case attribution panels in the reports strictly aggregate based on fields explicitly tagged on the write side in Observation.metadata (untagged normal data is not mixed into the “unknown waste” bucket):

Metadata field pathAllowed standard enum valuesEnum semantics and improvement guidance
metadata.wasteReasonretry, truncation, cache_miss_repeat, context_overflow, user_abortRespectively identify invalid Token consumption caused by error-triggered retries, reruns after output-length truncation, identical-prefix cache misses, context-overlimit discards, and user mid-stream cancellation. A missing field indicates normal, valid consumption.
metadata.badCaseCategorygrounding_missing, hallucination, instruction_violation, tool_failure, query_understanding, safety_refusal, latency_timeoutSeven mutually exclusive bad case attribution buckets: e.g., grounding_missing points to optimizing knowledge base retrieval, and tool_failure points to fixing tool schemas or downstream APIs.
metadata.cache.statushit, partial, missCache status supplied by the caller; presence and meaning depend on the data source.

4. Examples of calling the analytical report Bundle API

(
  set +x
  set -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
  : "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
  APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
  : "${APPBASE_PROJECT_ID:?Set APPBASE_PROJECT_ID for the target environment}"
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY for the target environment}"

  # 1. Query the Overview global report for the last 7 days (7d)
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS -X POST "${APPBASE_ENDPOINT}/observability/reports/overview" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
    -H "Content-Type: application/json" \
    --data '{
      "range": "7d"
    }' | jq .

  # 2. Query Quality and Bad Case attribution reports with a custom time window (from/to, span <= 30 days)
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS -X POST "${APPBASE_ENDPOINT}/observability/reports/quality" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
    -H "Content-Type: application/json" \
    --data '{
      "from": "2026-10-01T00:00:00.000Z",
      "to": "2026-10-22T23:59:59.999Z"
    }' | jq .
)

Querying metrics

  1. Open the metrics page via Observability / AgentOps → Metrics in the project sidebar.

  2. Select the data view, metric, aggregation method, grouping dimension, time granularity, and start/end times, then click Apply. The default query covers the last 24 hours, with time granularity defaulting to Hour; available granularities are Minute, Hour, Day, and Week.

  3. View the summary values, time-series charts, and dimension breakdown tables. After selecting a grouping, you can compare changes across dimensions; when you need to inspect a Trace's input/output or specific stages, return to the Trace to check details.

Available metrics and groupings depend on the data view:

Data viewMetricsAvailable groupings
TracesCount, latency, Token usage, costNo grouping, Trace name, Environment
ObservationsCount, latency, Token usage, costNo grouping, model, Trace name, Environment, stage, purpose, result, usage completeness
ScoresCount, scoreNo grouping, score name, Trace name, Environment

Aggregation options adjust based on the selected metric: Count uses Count; latency supports Count, Average, P50, P95; Token usage and cost support Sum and Average; scores support Count and Average. Select the data view and metric first, then choose from the aggregation options provided on the page.

Memory only is available only in the Observations view and filters Memory-related Observations. Some Console versions also show the option disabled in the Traces and Scores views; this does not mean scores are filtered by Memory.

When there is no metric data

If the chart shows No metric data, first check whether the start/end times cover the period when the data was produced, then confirm whether the data view, metric, and grouping match; when using Memory only, also confirm that the query target is Observations. After changing the conditions, click Apply again. The page's refresh button re-issues the current metrics query.

If there is still no data after adjusting the conditions, verify that the current project has written Traces, Observations, or Scores of the corresponding type, and check the Trace's time and usage records. An empty result only means the current query returned no data; it cannot be used to conclude “no calls”, “zero cost”, or “zero quality”.

FAQ and Troubleshooting

1. Why do the Reports pages show data for total calls and Tokens, but the “Token waste” or “Bad case attribution” panels appear empty?

2. Why does clicking “Apply” for a custom report time range fail or become disabled?

3. Do the model costs shown in Metrics and Reports exactly match the actual charges in the organization's bill?