Docs
Contents

Traces, Observation Nodes, Sessions, and Manual Annotation

Feature Overview

Tracing and manual annotation (Traces, Observations, Sessions, Users, Scores & Annotation Queues) are the core daily troubleshooting and manual review workbench of Agentbase Agent Ops (Observability / AgentOps). In the development and production operations of agent applications, development teams not only need to view the end-to-end execution waterfall of a single request, but also need to search across traces for specific LLM calls (GENERATION) or tool executions (TOOL), reconstruct multi-turn conversation context by session (Session) or user (User), and continuously accumulate high-quality evaluation criteria through structured scoring (Scores) and collaborative multi-person annotation queues (Annotation Queues).

The following tracing and quality-review scenarios use the example project 企业智能客服与知识中台 (enterprise-ai-hub-210) and example organization Agentbase 企业智能中台 (69c6d1aa00062e6bb60); use your own resources in practice:


Core Concepts and Architecture

A single agent execution is deconstructed by the observability engine into a four-dimensional correlated structure of "top-level Trace container + tree of Observation nodes + normalized ATIF trajectory + multi-source Scores".

Traces, Observation Nodes, Sessions, and Manual Annotation: workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

Observation Node Types and Technical Semantics Table

Observation type (type)Typical producing componentCore recorded fields and troubleshooting purposes
AGENTAgent Workbench (react.run / react.step)Records the agent orchestration loop, thinking steps, and sub-task dispatch, correcting and maintaining the assistant / user / tool message role hierarchy.
GENERATIONLLM Gateway / Memory / Agent RuntimeCan record model, modelParameters, input, output, usage, and estimated cost for model calls; some fields depend on what was reported. Do not treat missing provider usage as zero; cached can be a subset of input.
TOOLAgent tool execution (tool.execute)Records the input parameters, return results, execution status, and latency of MCP tool, database query, or custom function calls.
RETRIEVERKnowledge base /search and Memory retrievalRecords the vector and hybrid retrieval process. In a knowledge base retrieval Trace, when enhanced retrieval is actually attempted, graphrag and wiki sub-nodes are also generated, recording retrieval.status, retrieval.returned_count, and stage.duration_ms.
EMBEDDINGEmbedding stage (embed_query)Records the latency and Token consumption of calling text-embedding-v3 to generate query or document vectors.
SPANGeneric processing stage (context.prepare / HTTP / queue)Records the latency of non-model internal stages (e.g., context preparation, the default skipped stage when enhanced retrieval is not requested, HTTP client calls, and queue delivery).
EVENTInstantaneous key event (memory.ingest.stage_failed)Records instantaneous state transitions or abnormal events with no duration (e.g., stage error codes errorCode, gatewayStatus).
EVALUATOR / GUARDRAIL / CHAINExample observation typesAvailability depends on collected data; a type does not establish that content safety protection is enabled.

Roles, Entry Points, and Relationships

Console entry points and REST API mapping

In the Agent Ops (Observability / AgentOps) sidebar of project enterprise-ai-hub-210, the correspondence between the related pages and the underlying APIs is as follows:

Page nameConsole path suffixCore REST API endpointRequired permission Scope
Trace list and details (Traces)/observability/traces · /traces/{traceId}GET /v1/observability/traces, GET /v1/observability/traces/{traceId}observability.traces.read
Observations list (Observations)/observability/observationsGET /v1/observability/observationsobservability.traces.read
Session View (Sessions)/observability/sessions · /sessions/{sessionId}GET /v1/observability/sessions, GET /v1/observability/sessions/{sessionId}observability.sessions.read
User View (Users)/observability/users · /users/{userId}GET /v1/observability/users, GET /v1/observability/users/{userId}observability.users.read
Scores and Score Settings (Scores)/observability/scores · /score-settingsGET/POST /v1/observability/scores, GET/POST /v1/observability/score-configsobservability.scores.read / write
Annotation Queue (Annotations)/observability/annotations/v1/observability/annotation-queues and /assignments, /items, /items/next, /scores, /completeobservability.annotations.read / write / submit

Scope label filtering (APPBASE_TRACE_TAGS) and cross-system navigation

To help operations staff quickly distinguish Traces generated by platform managed capabilities from Traces reported by external business systems, the top of the Traces list provides scope filters based on authoritative telemetry labels:


Feature Details and Operation Guide

1. Querying Traces and profiling the three detail views

  1. Go to the Agent Ops → Traces page of project enterprise-ai-hub-210.

  2. You can filter precisely by scope label, user ID (userId, e.g. user_zhangwei_001), trace name (name), start and end time, or enter a complete Trace ID in the search box (in the Cloudnav deployment environment, you can also directly enter SOFA-TraceId to search for matching metadata.cloudAtlasTraceId).

  3. Click any Trace to enter its detail page. The top of the page provides three view switch buttons: Trajectory, Observation Tree, and Timeline:

    • Default view and intelligent fallback rules:

      • When the server has generated a valid, non-empty persisted ATIF-v1.8 trajectory and the URL does not specify a particular observation parameter, the page opens the Trajectory view by default, displaying multimodal content (including text, image, and audio structured fields) and tool call processes as normalized system / user / assistant / tool nodes;

      • When the URL carries a valid ?observation=<observationId> parameter, the page automatically opens the Observation Tree and highlights the selected node, making it easy for team members to share precise locations by copying the URL;

      • If the current Trace is a pure retrieval trace or the ATIF trajectory has not been generated or is empty, the page safely falls back to the Observation Tree and disables the Trajectory button, never showing a blank panel.

    • Virtual Trace root explanation: The outermost layer of the Observation Tree and Timeline displays a virtual Trace root node, used to show the end-to-end normalized request (Input) and final answer (Output) of this round. This virtual root exists only in the frontend display layer and is not double-counted in the total Observation count, Token consumption, or slowest node statistics.

2. Filtering call observations across traces (GET /v1/observability/observations)

When you need to directly investigate "all ERROR-level TOOL calls in the past 24 hours" or "the longest-running qwen-plus GENERATION nodes", use the Observations list page or call GET /v1/observability/observations:

Query parameterTypeRequiredValue constraints and pagination contract
namestringNoExact match on observation node name (e.g. "qwen-plus-diagnosis").
typestringNoOne or more valid observation types, comma-separated, e.g. "GENERATION,TOOL" (empty items or unknown types are forbidden, otherwise 400 is returned).
levelstringNoLog level filter: DEFAULT, DEBUG, WARNING, ERROR.
traceId / environmentstringNoExact filter by owning traceId or runtime environment (e.g. "production").
from / tostringNoISO-8601 start and end time range (if from > to, 400 is returned).
limitintegerNoPage size, value 1 ~ 100, default 25.
offsetintegerNoPagination offset, must be an integer multiple of limit (e.g. when limit=25, offset only allows 0, 25, 50..., otherwise 400 general_argument_invalid is returned).
(
  set +x
  set -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
  # Query all GENERATION and TOOL type observability nodes in project enterprise-ai-hub-210 (page 1, 25 items per page)
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS --get "${APPBASE_ENDPOINT:?Set endpoint including /v1}/observability/observations" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID:?Set project ID}" \
    --data-urlencode "type=GENERATION,TOOL" \
    --data-urlencode "environment=production" \
    --data-urlencode "limit=25" \
    --data-urlencode "offset=0" | jq .
)

3. Tracking multi-turn Sessions and end Users

4. Managing the scoring system (Scores) and the five-tier normalized distribution

Agentbase supports score configuration (Score Settings) for three data types:

Score data type (dataType)Values and configuration methodNormalized value (normalizedValue) calculation rules and Console display
NUMERIC (numeric) Define a minimum minValue and maximum maxValue (e.g. 0 ~ 1 or 1 ~ 5)The system strictly linearly normalizes raw scores to the [0, 1] range according to the configId bound to that Score (including archived configurations), and divides them into a five-tier quality distribution (Excellent / Good / Average / Poor / Very Poor). All numeric scores in the Console are uniformly kept to two decimal places (e.g. 0.92).
CATEGORICAL (categorical) Define an enumerated set of categories (e.g. ["准确", "部分遗漏", "严重幻觉"])Records the matched category string stringValue, supporting facet-based statistics of each category's share in the score list.
BOOLEAN (boolean) Binary judgment (true / false, corresponding to numeric values 1 / 0)Suitable for binary compliance checks such as "whether the SOP was followed" or "whether a safety red line was triggered".

5. Using Annotation Queues for collaborative review

When you need to organize multiple domain experts to perform standardized blind reviews or re-checks of a batch of low-scoring or problematic online Traces/Observations/Sessions, use Agent Ops → Annotations:

  1. Create an annotation queue and bind configurations and members:

    • Click Create Annotation Queue (POST /v1/observability/annotation-queues), fill in the queue name (e.g. "VIP工单回答质量专家复核队列"), and bind one or more score configurations (scoreConfigIds);

    • Use member assignment settings (PUT /v1/observability/annotation-queues/{queueId}/assignments) to assign project members allowed to participate in the queue's annotation.

  2. Add items pending review to the queue:

    • Select target items in the Trace list, Trace detail, or Session page, and click Add to Annotation Queue (POST /v1/observability/annotation-queues/{queueId}/items); idempotent enqueueing is supported, and you can view all queues an item belongs to along with their PENDING / COMPLETED status.

  3. 5-minute lease claiming, scoring, and completion closed loop:

    • Only assigned project members can click Start Annotation / Claim Next Item (POST .../items/next) on the queue detail page;

    • The system issues the member an 5-minute exclusive lease to prevent conflicts from multiple people annotating the same record simultaneously (if exiting midway, POST .../items/{itemId}/release can be called to release the lease);

    • The annotator refers to the original Trace/Observation/Session text on the left and enters scores and comments item by item according to all bound score configurations on the right (POST .../items/{itemId}/scores; queue scores carry an exclusive queueId isolation mark and cannot be tampered with via ordinary direct scoring APIs);

    • After completing all required score items, click Complete This Item (POST .../items/{itemId}/complete); the queue item's status changes to COMPLETED and the queue completion progress advances automatically.

(
  set +x
  set -e -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
  : "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
  APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
  : "${APPBASE_PROJECT_ID:?Set APPBASE_PROJECT_ID for the target environment}"
  # Example: create a human annotation queue via the REST API and add a difficult support ticket Trace to the queue

  # 1. Create a human annotation queue (bind numeric and boolean score configs)
  QUEUE_RESP="$(builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS -X POST "${APPBASE_ENDPOINT}/observability/annotation-queues" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
    -H "Content-Type: application/json" \
    --data '{
      "name": "企业客服SOP合规与技术准确性复核队列",
      "description": "针对 P0 工单链路进行人工专家复核",
      "scoreConfigIds": ["sc_sop_compliance_bool", "sc_tech_accuracy_num"]
    }')"

  QUEUE_ID="$(echo "${QUEUE_RESP}" | jq -er '.id')"

  # 2. Idempotently add the ticket Trace to the annotation queue
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS -X POST "${APPBASE_ENDPOINT}/observability/annotation-queues/${QUEUE_ID}/items" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
    -H "Content-Type: application/json" \
    --data '{
      "items": [
        {
          "objectId": "tr_support_ticket_20261022_001",
          "objectType": "TRACE"
        }
      ]
    }' | jq .
)

Finding a Trace

  1. Select a time range and trace scope, and enter a trace name or ID; click Apply. Scope options include Agent, RAG, Memory, external traces, Agent Runtime, and Knowledge Base; PowerContext appears only when the instance provides that capability.

  2. In the results, verify the name, time, latency, Observation count, Environment, User ID, and Session ID. Above the list, the total number of results is also displayed, along with the current page's P50/P95 latency, average Observation count, and number of Environments; these per-page statistics change as you paginate.

  3. Click a trace in the list, or enter the current project's Trace ID in Open trace to open the details directly. If the opened list carries filter labels such as Name or User, you can click the × next to a label to remove that condition and continue browsing.

Name/ID search, category, and time range jointly affect the results. If no records are found, first widen the time range, restore the trace scope, and clear the search conditions; if there are still no results, then verify whether the Trace ID belongs to the current project, and whether the write side has sent Traces to that project.

Analyzing Trace Details

The top of the detail page shows the trace status, Error Observation count, Session, total duration, slowest Observation, and Token usage. If some generation records lack Token usage, the page labels the aggregate as "known usage"; do not interpret incomplete statistics as a complete total. When an associated Session ID exists, you can jump from the details to the corresponding session.

In the trace workspace, view the Observation tree, timeline, or the displayable Agent trajectory depending on the data. After selecting an Observation, the details panel on the right lets you inspect input, output, metadata, and associated scores; Observations may also include usage and cost information. The timeline is suited to locating phases with long durations or Error levels, while the tree view makes it easy to inspect the call structure along parent-child Observations.

The detail status is a signal generated from observed evidence: it shows Error when an ERROR-level Observation exists; Success when no such Observation exists but an Observation or latency evidence does; and Unknown when these evidences are missing. This status does not replace the application's own business response validation; Unknown or an empty Observation cannot be used to conclude that a call succeeded or failed.

The Traces entry on the Memory page opens the trace list filtered to Memory traces, which can be used to inspect the memory pipeline. To compare call volume, latency, tokens, or scores over a time range, continue to Metrics analysis; for dataset run records, see Datasets.

Trajectory normalization and reporting verification

Core concepts and data relationships

When integrating an external Agent or investigating an execution in the Console, distinguish these four concepts and their data boundaries:

Ingestion and derived data: external applications report raw Trace and Observation data through public Events ingestion or OTLP/HTTP Traces. The platform asynchronously derives and persists ATIF v1.8. These ingestion endpoints accept their documented reporting formats; they do not provide a contract for uploading ATIF directly.

Observation trees and conversation trajectories: an Observation Tree represents the technical call structure, while a Trajectory presents the interactions and decisions between system, user, assistant, and tool. Ordinary HTTP requests, context preparation (SPAN), queue operations, and retrieval-only (RETRIEVER) nodes support execution. Their captured data remains available in the tree and timeline; they do not necessarily produce conversation steps.

External Agent integration and reporting requirements

Use these five steps to report an external Agent and verify that its trajectory can be displayed and evaluated:

  1. Configure separate project credentials for writing and reading All calls use public project authentication. A production exporter needs a project API Key with observability.ingest.write. Automated readback uses a separate Key with observability.traces.read, or users can inspect the Console. Keep the write Key separate from the verification read Key; production exporters do not need additional read permissions.

  2. Plan event identities and parent-child relationships Each event has two identities: the outer event.id is the event idempotency ID, while body.id identifies the Trace or Observation entity. Retry the same event with its original event.id. A new update event, such as generation-update, uses a new event.id and retains the entity's body.id. Do not regenerate IDs for each retry. Use non-zero, 32-character lowercase hexadecimal Trace IDs and non-zero, 16-character lowercase hexadecimal Observation IDs when possible, so W3C traceparent can carry the correlation across services. This format is recommended for component correlation and Trace Logs; Events ingestion itself does not require every entity ID to be hexadecimal. Every Observation's body.traceId references the root Trace, and parentObservationId references its direct parent Observation.

  3. Report model messages and tool relationships accurately

    • Roles and actual turns: a model Observation (GENERATION) can report input as a messages array containing role and content. Preserve the actual system, user, assistant, and tool roles and turns. Record only that model call's real assistant response and tool requests (tool_calls) in output; never copy the input Prompt into output as if the model generated it.

    • Context overlap across calls: when an execution makes multiple model calls, each GENERATION records the complete context actually supplied to that call. The platform identifies new turns using prefix-overlap rules. Do not remove real turns simply because their text is identical.

    • Consistent tool call IDs: keep the model's output.tool_calls call id, the TOOL Observation's metadata.toolCallId, and the subsequent model input message's tool_call_id for role tool consistent. The tool Observation supplies the call identity and real result; subsequent model messages preserve the result the model consumed. Parent-child Observation references do not replace tool call IDs. Some compatibility fallbacks exist, but new integrations should not rely on tool-name matching.

    • Per-call usage: report each model call's usage on its own GENERATION. Avoid counting a parent AGENT aggregate together with its child GENERATION usage. Leave unknown Token usage and Cost absent; do not invent zero values.

    • General and source-specific handling: OpenCode traces have additional requirements for a unique, acyclic authoritative Context Snapshot, identity verification, and completeness. Ordinary external Agents should use general ChatML messages. Do not forge source labels or internal signatures to select a special parsing path.

  4. Bound batches and inspect responses Events batches contain 1–100 events. Complete acceptance returns HTTP 200; partial failure returns 207 Multi-Status. Always inspect individual entries in errors[]. OTLP supports HTTP JSON or protobuf, optionally gzip, without gRPC. Its JSON body uses resourceSpans → scopeSpans → spans, without an Events batch wrapper. The current OTLP endpoint returns the platform's converted event result: inspect errors[] rather than assuming the standard OTLP partialSuccess response.

  5. Verify the Console display and evaluation results After persistence, use Observability overview, Online and dataset evaluations, and Datasets for evaluation and benchmarking. For complete request contracts, field descriptions, and code examples, select Observability in the documentation site's SDK Reference or API Reference section.

Four reporting verification layers

HTTP 200, a Trace appearing in the list, or nodes in the Observation tree cannot independently prove that normalization and evaluation have completed. Verify each layer:

LayerGoalVerification and criteriaCommon causes of missing evidence
1. Request acceptedThe ingestion endpoint accepted every eventFor Events, require HTTP 200, an empty errors[], and successes[] covering every original event ID. Treat 207 as partial failure. For OTLP, inspect the converted result and errors[]. Missing fields, non-JSON responses, or unacknowledged events do not prove full acceptance.Missing authentication headers, too many events, invalid field types, or ignored per-event errors in a 207 response.
2. Entities persistedThe asynchronous Worker stored the Trace and ObservationsInspect the tree and timeline in Trace details, or use the read Key to check the response's top-level trace and observations.Data still queued, new entity IDs incorrectly generated during retries, or incorrect traceId / parentObservationId relationships.
3. ATIF trajectory readyThe platform derived a valid, non-empty, ready trajectory in ATIF v1.8Open Trace details and inspect the available Trajectory view. For readback, check trajectory.schema_version=ATIF-v1.8, non-empty steps, and the expected steps. A non-empty field is only a necessary check; displayability does not prove that every evaluator has sufficient evidence.A retrieval-only or ordinary HTTP trace has no model interaction, reporting is incomplete, roles are missing, or tool call IDs cannot be linked.
4. Evaluation evidence sufficientEvidence required by the selected Prompt, Agent, or Program evaluator is availableInspect the Score and execution status in previews and results, including the selected template's scoring source and evidence notes. Exempt / Evidence limited indicates insufficient evidence for that metric; it is neither a normal pass nor a low score. Do not assume all templates use one status field.Missing or duplicate model calls, Usage that cannot be assigned to a user turn, missing cache or command evidence, or trajectories exceeding the reading budget.

Reporting and normalization FAQ

1. Why does an externally reported Trace contain GENERATION nodes but still have no usable trajectory?

2. Why are tool results missing or reported as conflicting?

3. What happens when message roles are wrong, a Prompt is copied into output, or identical turns are removed?

4. Why are Token statistics unreliable or labeled Exempt / Evidence limited?

5. How should HTTP 207 partial failures or network timeouts be retried?

FAQ and Troubleshooting

1. Why is the "Trajectory" button grayed out and unclickable, directly showing the "observation tree", when opening a Trace detail?

2. Why does the GET /v1/observability/observations paginated query return 400 general_argument_invalid?

3. Why can't a project admin or regular member click "Claim next item" or submit scores in Annotation Queues?