Docs
Contents

Memory End-to-End Call Chain Tracing (Memory Traces)

Feature overview and core scenarios

Memory End-to-End Call Chain Tracing (Memory Traces) is a full-chain performance profiling and fault diagnosis capability deeply integrated between Agentbase Memory (v2.10.0) and the agent operations observability center (AgentOps / Observability). In enterprise-grade agent production environments—for example, in the flagship project 企业智能客服与知识中台 (Project ID: enterprise-ai-hub-210, based on OceanBase Shared V2 storage mode) under Agentbase 企业智能中台 (Organization ID: 69c6d1aa00062e6bb60)—a seemingly simple memory read or write often spans multiple heterogeneous computing stages:

When R&D or SRE operations personnel need to locate the following core production issues:

  1. Retrieval latency breakdown: "Why did a certain hybrid hybrid retrieval take 420ms in total? Was the time spent on the OceanBase vector index scan, or on the qwen3.7-text-rerank reranking network round trips of upstream dashscope-prod?"

  2. Async extraction and deduplication attribution: "Why were no new memories added after the VIP architect user_zhangwei_001 submitted multi-turn conversations infer=true? Did the model decide NONE (duplicate skipped), or was LLM_RATE_LIMITED triggered and retrying in the background?"

  3. Token cost accounting and slow call governance: "How many Prompt / Completion Tokens did background knowledge distillation (distillations) and Dreaming aggregation consume respectively? Which slow requests exceeded slowThresholdMs?"

Simply click the Trace tab in the top navigation bar of Memory in the Console, and the system will carry the fixed capability tag capability:memory (APPBASE_TRACE_TAGS.capabilityMemory) and jump in one click to the project's observability trace view, enabling end-to-end pinpointing from business X-AppBase-Trace-Id, jobId, userId down to millisecond-level Span waterfalls and LLM Generation payloads.


Core Concepts and Architecture

Memory call chain tracing uses a unified distributed tracing context (W3C traceparent / X-Trace-ID -> X-AppBase-Trace-Id) to link the synchronous API ingestion plane, the async queue and Pipeline Worker plane, the OceanBase storage and retrieval plane, and the Models / LLM Gateway model invocation plane into a complete OpenTelemetry / Langfuse Trace Tree:

Memory End-to-End Call Chain Tracing (Memory Traces): workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

1. End-to-End Trace ID Propagation and Association Protocol

Agentbase Memory follows deterministic Trace ID inheritance and return rules on all /v1/memory/* APIs:

Propagation StageHeader / Field NameFormat Specification and Priority RulesProduction Troubleshooting Purpose
Inbound request header (priority 1)traceparentStandard W3C Trace Context format (00-<32hex trace-id>-<16hex parent-id>-<2hex flags>)When upstream microservices or the agent orchestration layer already have OpenTelemetry enabled, Memory automatically extracts the 32-digit hexadecimal trace-id within it as the associated Trace ID for this memory operation, achieving unified tracing across the full stack.
Inbound request header (priority 2)X-Trace-ID32-digit non-all-zero lowercase hexadecimal string (^[0-9a-f]{32}$)When traceparent is not passed, a valid X-Trace-ID explicitly passed by the caller is adopted preferentially.
Outbound response header (authoritative receipt)X-AppBase-Trace-Id32-digit non-all-zero lowercase hexadecimal string (e.g., 7a8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e)If the caller does not pass the above request headers, the platform automatically generates a brand-new Trace ID and uniformly returns it via X-AppBase-Trace-Id in all successful response headers; during async retries (attempt 1..N), the same jobId always reuses the same root Trace ID.
Capability filter tagcapability:memoryFrontend constant APPBASE_TRACE_TAGS.capabilityMemoryAutomatically applied to all root Traces produced by managed Memory, supporting one-click Console filtering and independent memoryOnly Metrics cost accounting.

2. Span Tree Structure Breakdown: Synchronous Retrieval vs. Async Write/Distillation

In the waterfall diagram of the Trace detail drawer, different business paths present standardized hierarchical Span tree structures:

Business Chain TypeCore Stage Span / Generation Node NamesObservation Type (type)Associated Model / Storage ComponentsKey Observation Attributes and Diagnostic Metrics
Synchronous hybrid search and retrieval
(search / recall)
memory.query_rewrite (when profile rewriting is enabled)GENERATIONdashscope-prod / qwen-plusRecords the raw query, the query terms after user_zhangwei_001 profile rewriting, and Token consumption.
memory.embedding.querySPANdashscope-prod / text-embedding-v3Records the encoding latency of the 1024-dimensional dense vector and Sparse vector (supports queryEmbeddingCache cache hits).
memory.storage.hybrid_querySPANOceanBase Shared V2Records the durations of vector HNSW scans, full-text search, 2-hop graph traversal (graph), and RRF fusion on project-exclusive table groups _<projectSequence>.
memory.rerank (when policy.rerankEnabled=true)SPANdashscope-prod / qwen3.7-text-rerankRecords rerank duration and score distribution for candidate sets; if reranking times out and degrades, records degraded.rerank=true.
Async write / distillation / Dreaming
(infer=true / distillations / contexts/runs)
memory.job.queue_waitSPAN (or metadata)Async task queueRecords queueDurationMs (the queueing wait in milliseconds from HTTP 202 enqueue until the Worker actually starts execution).
memory.media.parse (on multimodal writes)GENERATIONdashscope-prod / qwen-vl-maxRecords the parse duration and extracted byte count for PNG/JPEG/WebP/PDF files in agent_multimedia_bucket.
memory.pipeline.fact_extraction
memory.pipeline.skill_extraction
memory.pipeline.experience_extraction
GENERATIONdashscope-prod / qwen-plus (tool_call)Records the full Prompt/Completion of LLM calls during the extraction stage (controlled by the captureContent dual gate), first-token latency, and token details.
memory.pipeline.conflict_detection
memory.pipeline.skill_merge
GENERATIONdashscope-prod / qwen-plusRecords the ADD / UPDATE / DELETE / NONE / MERGE decision actions output after comparison with existing similar memories/skills.
memory.ingest.stage_failed (fault diagnosis node)EVENT / SPANFault stage locatorWhen any stage errors, records errorClass, errorCode (e.g., LLM_RATE_LIMITED, LLM_TIMEOUT, LLM_INVALID_JSON), retryable, and gatewayStatus.

Feature Entrypoints and Permission Matrix

The Console entry point is located at: Project enterprise-ai-hub-210 → Memory → Trace tab; clicking it routes directly to /console/project-{region}-enterprise-ai-hub-210/observability/traces?tag=capability%3Amemory.

Function actionConsole UI entry and routeREST API endpointServer SDK / header contractRequired RBAC / API Key ScopeCross-module associated dependency
One-click jump from the memory workbench to the trace pageMemory → Trace tab (routes to /observability/traces?tag=capability%3Amemory)GET /v1/observability/traces?tag=capability:memoryConsole route navigationConsole member + observability.traces.readIntegration Agent Ops → Trace
View the Span waterfall of a single TraceClick the target Trace row in the trace list to expand the detail drawerGET /v1/observability/traces/{traceId}HTTP GETobservability.traces.readDisplays memory.model.call and OceanBase retrieval sub-Spans
Configure Memory observability sampling and slow thresholdMemory → Configuration → Basic Configuration (policy)GET/PUT /v1/projects/{projectId}/memory/config/policyControl-plane HTTP APIConsole administrator or projects.writeManagement observability.enabled, successSampleRate, slowThresholdMs, captureContent
Count only Memory cost and latency on the metrics dashboardAgent Ops → Metrics → Memory only (memoryOnly)GET /v1/observability/metrics?memoryOnly=trueHTTP GETobservability.metrics.readIntegration Metrics monitoring and analysis reports
Cross-check business request receipts against task final statesMemory → Audit tab (?tab=audit)GET /v1/memory/audits, GET /v1/memory/jobs/{jobId}memory.listAudits / memory.getJobmemories.readForms a troubleshooting closed loop with Memory Audit Logs

Feature Details and Operation Guide

Step One: One-click jump from the Memory console and filter traces by capability:memory

  1. Log in to the Agentbase Console, go to the flagship project 企业智能客服与知识中台 (enterprise-ai-hub-210) under organization Agentbase 企业智能中台 (69c6d1aa00062e6bb60), and click Memory in the left navigation bar.

  2. In the horizontal tab bar at the top of the memory workbench (记忆 | 画像 | 经验与技能 | 模型 | 上下文 | 配置 | 审计 | Trace), click the rightmost Trace tab.

  3. The browser automatically jumps to /console/project-{region}-enterprise-ai-hub-210/observability/traces?tag=capability%3Amemory, and the filter bar at the top of the page has the tag = capability:memory filter condition auto-applied, showing only synchronous retrieval and async pipeline traces generated by the Memory service for the current project.

  4. In the filter area at the top of the list, you can further stack refined locating conditions:

    • Trace ID search: paste the X-AppBase-Trace-Id returned in the business response header (e.g., 7a8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e);

    • User ID filter: enter user_zhangwei_001 or user_lina_002 to view only the memory writes, profile updates, and retrieval traces of a specific customer;

    • Session ID filter: enter a business session identifier (e.g., sess_ticket_TK_20261022_001) to view the complete trace sequence of all memory inferences and context retrievals within the same support ticket session.

Step Two: Drill into the Span waterfall (locating latency bottlenecks and skipReason)

Click any Memory Trace record in the list, and the detail panel on the right will display the complete Span tree and stage-duration waterfall. In actual operations, you can quickly locate issues using the following four-step method:

  1. Distinguish queueDurationMs from executionDurationMs:

    • If async writes or distillation tasks take a long time end to end, first observe the blank waiting segment between the root Trace creation time and the first Worker execution Span (queueDurationMs). If queueDurationMs is significantly high while executionDurationMs is normal, it means the project is performing a large batch import (batch / imports) that triggered background queue peak-shaving, not a slowdown of the model or database.

  2. Locate the main time cost of online hybrid retrieval (search / recall):

    • Compare the duration shares of three sub-Spans: memory.embedding.query (calling text-embedding-v3 of dashscope-prod), memory.storage.hybrid_query (OceanBase vector + full-text + graph retrieval), and memory.rerank (calling qwen3.7-text-rerank);

    • If memory.rerank duration approaches or exceeds the budget, check whether the candidate retrieval cap is set too large, or use the progressive retrieval deadlineMs: 450 to enforce a hard timeout protection.

  3. Understand when the LLM GENERATION node (skipReason) will not appear:

    • When viewing a direct infer=false write, a compliance record with handlingPolicy="no_model" set, or an online retrieval with profile rewriting (profileRewriteEnabled=false) not enabled, the absence of an GENERATION LLM generation node in the Span tree is completely expected normal behavior; Span metadata clearly records the skip reason (e.g., skipReason: "request_infer_false" or "policy_no_model").

  4. Locate async task failures and retries (memory.ingest.stage_failed):

    • When a infer=true write or knowledge distillation triggers retries due to upstream model rate limiting (HTTP 429), multiple attempts of the same jobId (attempt=1, 2...) are aggregated under the same root Trace; expand the memory.model.call or memory.ingest.stage_failed node marked with warning/error to directly read errorCode (e.g., LLM_RATE_LIMITED, LLM_INVALID_JSON), retryable: true/false, and gatewayStatus.

Step Three: Configure the privacy content capture dual gate (captureContent) and intelligent sampling policy

In enterprise production environments, not all projects want customer raw conversations and memory content written into observability trace logs; meanwhile, ultra-high-concurrency projects also do not need full Trace collection for 100% of routine fast successful requests. Agentbase Memory provides fine-grained sampling and privacy control switches in policy.observability (located at /v1/projects/{projectId}/memory/config/policy):

Configuration field pathTypeDefault value / rangeGovernance semantics and effective mechanism
policy.observability.enabledbooleantrueProject-level Memory Trace export switch (requires the platform-global agentops.enabled=true to be enabled simultaneously).
policy.observability.successSampleRatenumber0.0 ~ 1.0 (default 1.0)Sampling rate for ordinary successful requests. For example, setting it to 0.2 means successful requests with normal durations are sampled with a 20% probability. Note: failed requests (Error) and slow requests exceeding the slow threshold (Slow) are not dropped by the sampling rate and are always collected with priority.
policy.observability.slowThresholdMsintegerPositive integer (e.g., 800 ms)Forced tracing threshold for slow requests. Even if successSampleRate is set very low, as long as a Memory operation exceeds slowThresholdMs in duration, the system automatically escalates sampling to ensure slow queries always have a trace available for inspection.
policy.observability.captureContentbooleanfalse / trueProject-level switch of the content capture dual gate: only when the platform-level environment variable switches (observability.captureContent / agentops.memoryCaptureContentEnabled) and the project-level switch policy.observability.captureContent are true simultaneously will memory content and Prompt inputs/outputs be recorded in Span and Generation nodes; as long as any gate is false, all business text payloads are automatically replaced with [REDACTED] before persistence, retaining only durations, token counts, and status codes.

Step Four: Inject a custom Trace ID via request headers and verify with API queries

The following demonstrates how to proactively pass W3C traceparent when calling the Memory search API on the business service side (or read the returned X-AppBase-Trace-Id), and then query the complete trace details via the observability REST API:

export APPBASE_ENDPOINT="https://appbuild.oceanbase.com/v1"
export APPBASE_PROJECT_ID="enterprise-ai-hub-210"
export APPBASE_API_KEY="<APPBASE_API_KEY>"

# 1. Call the hybrid retrieval endpoint, pass a custom W3C traceparent and capture the X-AppBase-Trace-Id in the response headers
curl -fsS -D /tmp/memory_trace_headers.txt \
  -X POST "${APPBASE_ENDPOINT}/memory/memories/search" \
  -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
  -H "X-Appwrite-Key: ${APPBASE_API_KEY}" \
  -H "traceparent: 00-7a8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e-1a2b3c4d5e6f7a8b-01" \
  -H "Content-Type: application/json" \
  --data '{
    "userId": "user_zhangwei_001",
    "scope": "user",
    "query": "OceanBase Shared V2 向量索引 HNSW 参数与排障偏好",
    "searchMode": "hybrid",
    "limit": 5
  }' | jq .

# Extract the authoritative X-AppBase-Trace-Id from the response headers
TRACE_ID="$(grep -i '^x-appbase-trace-id:' /tmp/memory_trace_headers.txt | awk '{print $2}' | tr -d '\r')"
echo "Captured X-AppBase-Trace-Id: ${TRACE_ID}"

# 2. Query the project's Memory trace list by the capability:memory tag and userId
curl -fsS --get "${APPBASE_ENDPOINT}/observability/traces" \
  -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
  -H "X-Appwrite-Key: ${APPBASE_API_KEY}" \
  --data-urlencode "tag=capability:memory" \
  --data-urlencode "userId=user_zhangwei_001" \
  --data-urlencode "limit=10" | jq .

# 3. Using the captured TRACE_ID, query the complete Span waterfall tree of a single Trace along with Generation token details
curl -fsS "${APPBASE_ENDPOINT}/observability/traces/${TRACE_ID}" \
  -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
  -H "X-Appwrite-Key: ${APPBASE_API_KEY}" | jq .

FAQ (Frequently Asked Questions and Troubleshooting)

1. Why do I see successful request records in "Memory → Audit" but cannot find the corresponding X-AppBase-Trace-Id in the trace list after clicking "Trace"?

2. Why do the Prompt input and model output show as [REDACTED] when opening the Generation node of a Memory Trace?

3. Why do the model call counts and Token consumption shown in “Agent Operations → Metrics” appear higher than the statistics of calling the Chat API directly from business code?