Docs
Contents

Observability Overview (AgentOps)

Feature Overview

AI Observability (the Console left navigation section is named AgentOps / Agent Ops) is Agentbase's full-chain tracing, quality evaluation, annotation review, and cost measurement platform built for enterprise-grade large model and agent applications. In complex agent production systems, a single user request often spans multiple subsystems, including the Agent Runtime, AI Gateway, Knowledge Base RAG, and long-term Memory, involving multiple large model generations and tool executions. Traditional APM monitoring can only see HTTP status codes and API latency, and cannot provide insight into Prompt quality, retrieval relevance, hallucination risk, or Token cost attribution.

In the flagship project 企业智能客服与知识中台 (enterprise-ai-hub-210) of Agentbase 企业智能中台 (69c6d1aa00062e6bb60), AgentOps provides a complete closed loop from collection, observation, to evaluation and governance:


Core Concepts and Architecture

Agentbase Observability adopts a layered architecture of "Agentbase unified authentication and control plane + high-performance trace storage and asynchronous evaluation engine". Browsers and client SDKs interact only through the /v1/observability/* public gateway, where the platform strictly validates project ownership and Scope permissions.

Observability Overview (AgentOps): workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

Core Data Model Definition Table

Domain entityKey identifiers and hierarchyCore attributes and technical semantics
Trace (call chain)traceId (top-level root container)Represents one end-to-end business request (such as a single complete Q&A round of an intelligent customer service handling support ticket TK-20261022-001). Mounts userId, sessionId, name, input, output, tags, metadata, and persisted ATIF-v1.8-compliant traces.
Observation (observation node)observationId (forms a tree via parentObservationId)Concrete execution steps within a Trace. Supports 10 standard types: GENERATION (large model call, including model name, token details, and cost), TOOL (tool call), AGENT (agent orchestration step), SPAN (generic duration span), EVENT (instantaneous event), RETRIEVER (knowledge/memory retrieval), EMBEDDING, CHAIN, EVALUATOR, GUARDRAIL.
Session (multi-turn conversation)sessionId (aggregates multiple Traces)Chains multiple Traces within the same conversation window in chronological order, and aggregates the total duration, total token usage, and user satisfaction score of the entire session.
User (end user)userId (e.g. user_zhangwei_001)Aggregates by business user dimension all historically initiated Traces, Observation counts, cumulative token consumption, estimated cost, and score distribution.
Score (quality score)scoreId (bound to traceId and optional observationId / sessionId)Measures the quality of a single generation or an entire chain. Supports NUMERIC (numeric, automatically computes the 0~1 normalized value per configuration normalizedValue), CATEGORICAL (categorical), BOOLEAN (boolean); sources are divided into EVAL (automated evaluation), ANNOTATION (manual annotation), and API (external reporting).
Prompt (prompt asset)promptId / name + version / labelCentrally manages versioned prompt templates (including labels such as production), and supports deterministic resolution at inference or evaluation time via GET /v1/observability/prompts/resolve.

Roles, Entry Points, and Relationships

Console Page Structure and Four Navigation Sections

In project enterprise-ai-hub-210, expand Observability / AgentOps in the left navigation bar. The system organizes all subpages into four major business sections:

Trace list

Figure: Trace list in a 2.12.0 test project with user and session identifiers masked; contents vary by project.

Section namePage nameConsole route suffixCore functionality descriptionRequired permission Scope
ObservationTraces/observability/tracesDefault landing page. Filter traces by scope labels, user, or name, and view ATIF traces, observation trees, and waterfall diagrams.observability.traces.read
ObservationObservations/observability/observationsFlat search across Traces for all Observation nodes in the project, with stable pagination by multiple types, levels, environments, and time windows.observability.traces.read
ObservationSessions / Users/observability/sessions · /usersAggregate multiple related traces by session, or drill down into behavior and cost by end user (e.g. user_zhangwei_001).observability.sessions.read / observability.users.read
ObservationMetrics/observability/metricsDashboards of trace-derived latency percentiles (P50/P95/P99), token usage, model cost, and score trends.observability.metrics.read
ObservationData Settings/observability (Data Settings entry)Configure project-level observation data retention (3–36,500 days or unlimited retention).observability.settings.read / write
PromptsPrompts/observability/promptsVersioned management of prompt templates, online Playground experimentation, and deterministic /resolve resolution.observability.prompts.read / write
EvaluationsEvals & Evaluation Skills/observability/evals · /evaluation-skillsBuilt-in and custom evaluation templates, knowledge base parsing and index quality evaluation tasks (general/Memory/knowledge base), and read-only evaluation skill package management.observability.evals.read / write
EvaluationsScores & Score Settings/observability/scores · /score-settingsView automated and manual scores, five-tier normalized distribution, low-score root cause analysis (RCA), and maintain score configurations.observability.scores.read / write
EvaluationsAnnotations/observability/annotationsCreate manual annotation queues, assign project members to claim leases, and review/score Traces/Observations/Sessions.observability.annotations.read / write / submit
Assets / Data (data assets)Datasets/observability/datasetsManage golden evaluation datasets (such as enterprise-support-golden-v1), import CSVs, and trigger benchmark comparison runs.observability.datasets.read / write
Assets / Data (data assets)Reports/observability/reports/overview (and /agents, /quality)Fixed-window (24h/7d/30d or custom ≤30d) dashboards for usage, cost, ROI, and Bad Case attribution (visibility depends on deployment configuration; hidden pages also redirect direct links).Overview: observability.metrics.read
Agents: observability.metrics.read + observability.traces.read + observability.users.read + observability.sessions.read
Quality: observability.metrics.read + observability.traces.read + observability.scores.read

Deep Integration with the Platform's Four AI Capabilities

  1. Integration with the Agent Runtime: When the platform enables agentops.runtimeExportEnabled, the Agent Runtime automatically exports each ReAct run as a Trace tagged with capability:agent and asynchronously consolidates it into a standardized execution trace conforming to ATIF-v1.8; dataset evaluations can also directly use the Agent Runtime as an evaluation target to trigger batch replays.

  2. Integration with the Knowledge Base (Knowledge Base kb_enterprise_support):

    • Ingestion and build tracing: when the platform enables agentops.ingestionTracingEnabled and the knowledge base enables tracingEnabled, document parsing, chunking, and GraphRAG / RAPTOR / LLM Wiki index construction each generate an independent Trace, directly supporting knowledge base parsing and index quality evaluation;

    • Enhanced retrieval tracing: every time /v1/knowledge-base/{kbId}/search is called, in addition to the basic embed_query, search, and rerank nodes, the system records under the same Trace the retrieval.status of graphrag and wiki child nodes (not_requested / not_ready / succeeded / degraded / failed), returned entry counts, and per-stage durations.

  3. Integration with long-term Memory: Memory operation traces can be investigated with filters such as capability:memory; actual traces depend on sampling and deployment configuration. The Metrics page offers memoryOnly filtering in the Observations view, not other views; available UI entry points depend on the environment.


Feature Details and Operation Guide

1. Dual-Channel Ingestion: OpenTelemetry OTLP and /v1/observability/ingestion

In addition to automatic reporting by platform managed components, business applications can report call chains from custom agents or backend microservices to the project's enterprise-ai-hub-210 via the following two standard channels:

Channel One: Using the /v1/observability/ingestion Batch Event Ingestion API

Suitable for reporting structured trace-create, generation-create, span-create, and score-create event batches directly via REST API or agentbase-node-sdk@2.12.0:

(
  set +x
  set -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
  : "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
  APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
  : "${APPBASE_PROJECT_ID:?Set APPBASE_PROJECT_ID for the target environment}"
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY for the target environment}"

  # Batch-report a customer support ticket handling Trace with its internal qwen-plus Generation node
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS -X POST "${APPBASE_ENDPOINT}/observability/ingestion" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
    -H "Content-Type: application/json" \
    --data '{
      "batch": [
        {
          "id": "evt_trace_20261022_001",
          "type": "trace-create",
          "timestamp": "2026-10-22T10:15:00.000Z",
          "body": {
            "id": "tr_support_ticket_20261022_001",
            "name": "enterprise-ticket-triage",
            "userId": "user_zhangwei_001",
            "sessionId": "sess_ticket_TK_20261022_001",
            "input": "工单 TK-20261022-001:核心账务集群 HNSW 向量索引重建报 -4013 内存不足",
            "output": "请结合版本、日志和资源指标核实内存错误的原因。",
            "tags": ["capability:agent", "sla:P0-VIP"],
            "environment": "production",
            "metadata": {
              "ticketId": "TK-20261022-001",
              "gatewayId": "gw_7194685489feb1e169548619"
            }
          }
        },
        {
          "id": "evt_gen_20261022_001",
          "type": "generation-create",
          "timestamp": "2026-10-22T10:15:00.120Z",
          "body": {
            "id": "obs_gen_qwen_plus_001",
            "traceId": "tr_support_ticket_20261022_001",
            "name": "qwen-plus-diagnosis",
            "startTime": "2026-10-22T10:15:00.120Z",
            "endTime": "2026-10-22T10:15:01.480Z",
            "model": "qwen-plus",
            "input": [{"role": "user", "content": "排查 OB -4013 向量内存错误"}],
            "output": {"role": "assistant", "content": "请依据当前数据库版本的官方文档检查相关日志和资源指标。"},
            "usage": {
              "input": 420,
              "output": 185,
              "total": 605,
              "unit": "TOKENS"
            },
            "metadata": {
              "cache": {"status": "miss"}
            }
          }
        }
      ]
    }' | jq .
)

Channel Two: Ingestion via the OpenTelemetry OTLP Standard Protocol

For applications already integrated with the OpenTelemetry SDK (Node.js / Python / Go / Java), you can configure the OTLP HTTP Exporter to point to the project's OTLP receiving endpoint, carrying the project header X-Appwrite-Project: enterprise-ai-hub-210 and an API Key with observability.traces.write permission; the platform automatically maps OpenTelemetry GenAI semantic convention attributes (gen_ai.system, gen_ai.request.model, gen_ai.usage.*) to standard GENERATION and TOOL observation nodes.

2. Cross-Trace Observation List (Observations) and Metrics Dashboard

  1. In the left navigation bar, click Agent Ops → Observations to open the cross-trace observation records list page. This page allows skipping the top-level Trace and filtering all fine-grained execution nodes directly by observation name (name), observation type (GENERATION, TOOL, AGENT, RETRIEVER, etc., with multi-select), log level (DEFAULT, DEBUG, WARNING, ERROR), traceId, environment, and start/end time.

Observation list

Figure: Observation list in a 2.12.0 test project with trace identifiers masked; contents vary by project.

  1. In the left navigation bar, click Agent Ops → Metrics to view call volume, P50/P95/P99 latency, Token consumption, model cost estimates, and quality score distributions derived in real time from traces.

Metrics page

Figure: Metrics in a 2.12.0 test project; data varies by project.

3. Configure the project-level observability data retention policy (Data Retention)

According to compliance audit requirements and storage cost budgets, enterprises can configure the retention period of the current project's observable data in Agent Ops → Observations → Data Settings (corresponding to the control plane API GET/PATCH /v1/observability/settings/data-retention):

Parameter fieldTypeValue range and defaultGovernance rules and safety boundaries
retentionDaysinteger | nullDefault null (retain indefinitely); finite retention supports an integer number of days within 3 ~ 36500 (passing 0 or null is both normalized to indefinite retention)Covers the current project's Traces, Observations, Scores, and associated analysis artifacts and media. Saving only updates the retention policy; a background scheduled job purges expired data daily, and no synchronous blocking deletion occurs at the moment of saving.
expectedRevisionstringThe GET string returned by the most recent revision responseOptimistic-lock concurrency protection. Every time a finite retention period is saved, the Console pops up an irreversible cleanup risk confirmation dialog; once historical data has been purged, it cannot be recovered by later increasing the number of days or switching back to indefinite retention.
(
  set +x
  set -e -o pipefail
  : "${APPBASE_CONSOLE_SESSION:?Set APPBASE_CONSOLE_SESSION}"
# Query and update the observability data retention policy of project enterprise-ai-hub-210 to 180 days
RET_DATA="$(builtin printf 'Cookie: %s\n' "$APPBASE_CONSOLE_SESSION" | curl --disable --header @- -fsS -w '\n%{http_code}' "${APPBASE_ENDPOINT}/observability/settings/data-retention" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Mode: admin")"
case "${RET_DATA##*$'\n'}" in
  2??) RET_DATA="${RET_DATA%$'\n'*}" ;;
  *) builtin printf 'Configuration read returned a non-2xx response\n' >&2; exit 1 ;;
esac

REV="$(echo "${RET_DATA}" | jq -er '.revision')"

builtin printf 'Cookie: %s\n' "$APPBASE_CONSOLE_SESSION" | curl --disable --header @- -fsS -X PATCH "${APPBASE_ENDPOINT}/observability/settings/data-retention" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Mode: admin" \
  -H "Content-Type: application/json" \
  --data "{
    \"retentionDays\": 180,
    \"expectedRevision\": \"${REV}\"
  }" | jq .
)

4. Platform-global observability switches (reference for instance administrators)

In the platform admin backend (the Agent Ops tab of /_internal/admin), instance administrators control the capability boundaries of the entire platform through 4 global dynamic configuration keys (after modification, changes sync to all PHP Worker, Memory, and Agent Runtime processes within about 5 minutes 30 seconds):

Platform configuration keyEnvironment default value variableDescription of effect
agentops.enabled_APP_LANGFUSE_ENABLEDPlatform-wide master observability switch, controlling module entry, the /v1/observability/* API, and internal service Trace export.
agentops.ingestionTracingEnabled_APP_KNOWLEDGE_BASE_INGESTION_TRACING_ENABLED (default enabled)Globally allows knowledge base ingestion and knowledge index build tracing (the knowledge base's own tracingEnabled=true is still required).
agentops.runtimeExportEnabledAGENTOPS_LANGFUSE_ENABLEDControls Trace export and dataset run association for the Agent Workbench (Agent Runtime).
agentops.memoryCaptureContentEnabledMEMORY_OBSERVABILITY_CAPTURE_CONTENT_ENABLED (default true)Globally allows Memory to capture body content (the project-level policy.observability.captureContent=true is still required).

Select a Prompt Version and View Usage Instructions

In Observability / AgentOps → Prompts, open the target Prompt and switch to Usage in the details. The current project must already have a readable Prompt; when providing read credentials to business applications, grant only the required Prompt read permission, and do not write secrets into Prompt content or example variables.

  1. First select a tag or a fixed version, then select the curl, Python, or TypeScript example.

  2. Check the version and variables displayed on the page. When switching versions, wait for the content to finish loading; if loading fails, first verify whether the current project, permissions, and version exist, and do not copy examples from an old version as a substitute.

  3. Replace the endpoint and credential placeholders as instructed on the page, and confirm the project ID is correct. Variable names come from the selected version, and example variable values must be replaced with business inputs.

  4. After running, verify the returned Prompt name, version, body, and variables before integrating it into business calls or evaluations.

Selection methodApplicable scenariosNotes
production, latest, or a custom tagWhen you want calls to follow the version the tag points toTags can be moved and must not be treated as immutable snapshots
Fixed version numberRegression evaluation, reproducing a specific answerMust explicitly use a positive integer version; cannot be specified together with a tag

When resolving by name, if neither a tag nor a version is specified, production is used by default; this does not mean the latest version is fetched automatically. If you need an earlier historical version, first load the corresponding history page in the version list, then select it in the usage instructions.

Python / TypeScript examples replace variables in text Prompts or chat messages according to the selected version; curl examples only indicate variables and do not perform substitution. Message placeholders in chat Prompts also do not mean a complete conversation has been filled in; handle them according to business requirements before integration. After the Prompt version is verified, you can compare results using the same set of datasets; for quality checks of retrieval-based answers, see Evaluations and Scoring.

FAQ and Troubleshooting

1. Why did I perform an action in the Agent Workbench or knowledge base but do not see a new trace in the Traces list?

2. What does missing Trace input/output or [REDACTED] mean?

3. Why do regular project members get 404 project_not_found or 403 when calling GET /v1/observability/settings/data-retention?