Docs
Contents

Memory End-to-End Call Chain Tracing (Memory Traces)

Feature overview and core scenarios

Memory End-to-End Call Chain Tracing (Memory Traces) is a full-chain performance profiling and fault diagnosis capability deeply integrated between Agentbase Memory and the agent operations observability center (AgentOps / Observability). In enterprise-grade agent production environments—for example, in the example project 企业智能客服与知识中台 (Project ID: enterprise-ai-hub-210, based on OceanBase Shared V2 storage mode) under Agentbase 企业智能中台 (Organization ID: 69c6d1aa00062e6bb60)—a seemingly simple memory read or write often spans multiple heterogeneous computing stages:

When R&D or SRE operations personnel need to locate the following core production issues:

  1. Retrieval latency breakdown: "Why did a certain hybrid hybrid retrieval take 420ms in total? Was the time spent on the OceanBase vector index scan, or on the qwen3.7-text-rerank reranking network round trips of upstream dashscope-prod?"

  2. Async extraction and deduplication attribution: "Why were no new memories added after the VIP architect user_zhangwei_001 submitted multi-turn conversations infer=true? Did the model decide NONE (duplicate skipped), or was LLM_RATE_LIMITED triggered and retrying in the background?"

  3. Token cost accounting and slow call governance: "How many Prompt / Completion Tokens did background knowledge distillation (distillations) and Dreaming aggregation consume respectively? Which slow requests exceeded slowThresholdMs?"

Use the Memory traces button on the Console Memory page to open the project's observability trace view; the initial filter is tag=memory, enabling end-to-end pinpointing from business X-AppBase-Trace-Id, jobId, userId down to millisecond-level Span waterfalls and LLM Generation payloads.


Core Concepts and Architecture

Memory call chain tracing uses a unified distributed tracing context (W3C traceparent / X-Trace-ID -> X-AppBase-Trace-Id) to link the synchronous API ingestion plane, the async queue and Pipeline Worker plane, the OceanBase storage and retrieval plane, and the Models / LLM Gateway model invocation plane into a complete OpenTelemetry / Langfuse Trace Tree:

Memory End-to-End Call Chain Tracing (Memory Traces): workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

1. End-to-End Trace ID Propagation and Association Protocol

Agentbase Memory follows deterministic Trace ID inheritance and return rules on all /v1/memory/* APIs:

Propagation StageHeader / Field NameFormat Specification and Priority RulesProduction Troubleshooting Purpose
Inbound request header (priority 1)traceparentStandard W3C Trace Context format (00-<32hex trace-id>-<16hex parent-id>-<2hex flags>)When upstream microservices or the agent orchestration layer already have OpenTelemetry enabled, Memory automatically extracts the 32-digit hexadecimal trace-id within it as the associated Trace ID for this memory operation, achieving unified tracing across the full stack.
Inbound request header (priority 2)X-Trace-ID32-digit non-all-zero lowercase hexadecimal string (^[0-9a-f]{32}$)When traceparent is not passed, a valid X-Trace-ID explicitly passed by the caller is adopted preferentially.
Outbound response header (authoritative receipt)X-AppBase-Trace-Id32-digit non-all-zero lowercase hexadecimal string (e.g., 7a8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e)If the caller does not pass the above request headers, the platform automatically generates a brand-new Trace ID and uniformly returns it via X-AppBase-Trace-Id in all successful response headers; during async retries (attempt 1..N), the same jobId always reuses the same root Trace ID.
Example API tag filtercapability:memoryTrace list API filter valueFilter by tag as appropriate; check the actual tags on the trace. The Console entry initially filters by tag=memory.

2. Span Tree Structure Breakdown: Synchronous Retrieval vs. Async Write/Distillation

The Trace detail waterfall shows hierarchical Span trees for different business paths. Providers, models, and dimensions in the table are examples; actual values depend on the project configuration:

Business Chain TypeCore Stage Span / Generation Node NamesObservation Type (type)Associated Model / Storage ComponentsKey Observation Attributes and Diagnostic Metrics
Synchronous hybrid search and retrieval
(search / recall)
memory.query_rewrite (when profile rewriting is enabled)GENERATIONdashscope-prod / qwen-plusRecords the raw query, the query terms after user_zhangwei_001 profile rewriting, and Token consumption.
memory.embedding.querySPANdashscope-prod / text-embedding-v3Records the encoding latency of the 1024-dimensional dense vector and Sparse vector (supports queryEmbeddingCache cache hits).
memory.storage.hybrid_querySPANOceanBase Shared V2Records the durations of vector HNSW scans, full-text search, 2-hop graph traversal (graph), and RRF fusion on project-exclusive table groups _<projectSequence>.
memory.rerank (when policy.rerankEnabled=true)SPANdashscope-prod / qwen3.7-text-rerankRecords rerank duration and score distribution for candidate sets; if reranking times out and degrades, records degraded.rerank=true.
Async write / distillation / Dreaming
(infer=true / distillations / contexts/runs)
memory.job.queue_waitSPAN (or metadata)Async task queueRecords queueDurationMs (the queueing wait in milliseconds from HTTP 202 enqueue until the Worker actually starts execution).
memory.media.parse (on multimodal writes)GENERATIONdashscope-prod / qwen-vl-maxRecords the parse duration and extracted byte count for PNG/JPEG/WebP/PDF files in agent_multimedia_bucket.
memory.pipeline.fact_extraction
memory.pipeline.skill_extraction
memory.pipeline.experience_extraction
GENERATIONdashscope-prod / qwen-plus (tool_call)Ordinary diagnostic traces contain call metadata only; a separate evaluation trace may contain raw input/output when content capture conditions are met.
memory.pipeline.conflict_detection
memory.pipeline.skill_merge
GENERATIONdashscope-prod / qwen-plusRecords the ADD / UPDATE / DELETE / NONE / MERGE decision actions output after comparison with existing similar memories/skills.
memory.ingest.stage_failed (fault diagnosis node)EVENT / SPANFault stage locatorWhen any stage errors, records errorClass, errorCode (e.g., LLM_RATE_LIMITED, LLM_TIMEOUT, LLM_INVALID_JSON), retryable, and gatewayStatus.

Feature Entrypoints and Permission Matrix

On the project's Memory page, use the Memory traces button. The initial filter is tag=memory.

Function actionConsole UI entry and routeREST API endpointServer SDK / header contractRequired RBAC / API Key ScopeCross-module associated dependency
Open traces from the Memory pageMemory traces buttonGET /v1/observability/traces (set filters as needed)Console route navigationConsole member + observability.traces.readIntegration Agent Ops → Trace
View the Span waterfall of a single TraceClick the target Trace row in the trace list to expand the detail drawerGET /v1/observability/traces/{traceId}HTTP GETobservability.traces.readDisplays memory.model.call and OceanBase retrieval sub-Spans
Configure Memory observability sampling and slow thresholdMemory → Configuration → Basic Configuration (policy)GET/PUT /v1/projects/{projectId}/memory/config/policyControl-plane HTTP APIA Console user session with administrative access and the permissions required by each operationManagement observability.enabled, successSampleRate, slowThresholdMs, captureContent
View Memory-related metricsAgent Ops → Metrics → Observations → Memory only (memoryOnly)POST /v1/observability/metricsquery in the JSON body; the console query adapter translates the filterobservability.metrics.readApplies to the Observations view; see Metrics monitoring and analysis reports
Cross-check business request receipts against task final statesMemory → Audit tab (?tab=audit)GET /v1/memory/audits, GET /v1/memory/jobs/{jobId}memory.listAudits / memory.getJobAudit: project administrator session or current-project API Key (memories.read); job query: memories.readForms a troubleshooting closed loop with Memory Audit Logs

Feature Details and Operation Guide

Step One: Open and filter traces from the Memory console

  1. Log in to the Agentbase Console, select the target project, and click Memory in the left navigation bar.

  2. Click the Memory traces button at the top of the page.

  3. The button opens the traces page with tag=memory. You can adjust the filter; this value differs from capability:memory in the API example below.

  4. In the filter area at the top of the list, you can further stack refined locating conditions:

    • Trace ID search: paste the X-AppBase-Trace-Id returned in the business response header (e.g., 7a8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e);

    • User ID filter: enter user_zhangwei_001 or user_lina_002 to view only the memory writes, profile updates, and retrieval traces of a specific customer;

    • Session ID filter: enter a business session identifier (e.g., sess_ticket_TK_20261022_001) to view the complete trace sequence of all memory inferences and context retrievals within the same support ticket session.

Step Two: Drill into the Span waterfall (locating latency bottlenecks and skipReason)

Click any Memory Trace record in the list, and the detail panel on the right will display the complete Span tree and stage-duration waterfall. In actual operations, you can quickly locate issues using the following four-step method:

  1. Distinguish queueDurationMs from executionDurationMs:

    • If async writes or distillation tasks take a long time end to end, first observe the blank waiting segment between the root Trace creation time and the first Worker execution Span (queueDurationMs). If queueDurationMs is significantly high while executionDurationMs is normal, it means the project is performing a large batch import (batch / imports) that triggered background queue peak-shaving, not a slowdown of the model or database.

  2. Locate the main time cost of online hybrid retrieval (search / recall):

    • Compare the duration shares of three sub-Spans: memory.embedding.query (calling text-embedding-v3 of dashscope-prod), memory.storage.hybrid_query (OceanBase vector + full-text + graph retrieval), and memory.rerank (calling qwen3.7-text-rerank);

    • If memory.rerank duration approaches or exceeds the budget, check whether the candidate retrieval cap is set too large, or use the progressive retrieval deadlineMs: 450 to enforce a hard timeout protection.

  3. Understand when the LLM GENERATION node (skipReason) will not appear:

    • When viewing a direct infer=false write, a compliance record with handlingPolicy="no_model" set, or an online retrieval with profile rewriting (profileRewriteEnabled=false) not enabled, the absence of an GENERATION LLM generation node in the Span tree is completely expected normal behavior; Span metadata clearly records the skip reason (e.g., skipReason: "request_infer_false" or "policy_no_model").

  4. Locate async task failures and retries (memory.ingest.stage_failed):

    • When a infer=true write or knowledge distillation triggers retries due to upstream model rate limiting (HTTP 429), multiple attempts of the same jobId (attempt=1, 2...) are aggregated under the same root Trace; expand the memory.model.call or memory.ingest.stage_failed node marked with warning/error to directly read errorCode (e.g., LLM_RATE_LIMITED, LLM_INVALID_JSON), retryable: true/false, and gatewayStatus.

Step Three: Configure the privacy content capture dual gate (captureContent) and intelligent sampling policy

In enterprise production environments, not all projects want customer raw conversations and memory content written into observability trace logs; meanwhile, ultra-high-concurrency projects also do not need full Trace collection for 100% of routine fast successful requests. Agentbase Memory provides fine-grained sampling and privacy control switches in policy.observability (located at /v1/projects/{projectId}/memory/config/policy):

Configuration field pathTypeDefault value / rangeGovernance semantics and effective mechanism
policy.observability.enabledbooleantrueProject-level Memory Trace export switch (requires the platform-global agentops.enabled=true to be enabled simultaneously).
policy.observability.successSampleRatenumber0.0 ~ 1.0 (default 1.0)Sampling rate for ordinary successful requests. For example, setting it to 0.2 means successful requests with normal durations are sampled with a 20% probability. Note: failed requests (Error) and slow requests exceeding the slow threshold (Slow) are not dropped by the sampling rate and are always collected with priority.
policy.observability.slowThresholdMsintegerPositive integer (e.g., 800 ms)Requests exceeding the threshold are not dropped by success sampling, but export can still fail or drop events; a queryable Trace is not guaranteed.
policy.observability.captureContentbooleanfalse / trueContent capture depends on platform agentops.memoryCaptureContentEnabled and project policy.observability.captureContent. Ordinary diagnostic Generations omit model input/output; when capture is enabled, a separate evaluation trace may contain raw input/output. Sanitize before reporting; automatic replacement or recursive redaction is not guaranteed.

Audit coverage: ordinary reads and lists typically do not create request audit entries; a missing entry does not prove that a request did not occur.

Step Four: Inject a custom Trace ID via request headers and verify with API queries

The following demonstrates how to proactively pass W3C traceparent when calling the Memory search API on the business service side (or read the returned X-AppBase-Trace-Id), and then query the complete trace details via the observability REST API:

(
  set +x
  set -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
  : "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
  APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
  : "${APPBASE_PROJECT_ID:?Set APPBASE_PROJECT_ID for the target environment}"
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY for the target environment}"

  # 1. Call the hybrid retrieval endpoint, pass a custom W3C traceparent and capture the X-AppBase-Trace-Id in the response headers
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS -D /tmp/memory_trace_headers.txt \
    -X POST "${APPBASE_ENDPOINT}/memory/memories/search" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
    -H "traceparent: 00-7a8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e-1a2b3c4d5e6f7a8b-01" \
    -H "Content-Type: application/json" \
    --data '{
      "userId": "user_zhangwei_001",
      "scope": "user",
      "query": "OceanBase Shared V2 向量索引 HNSW 参数与排障偏好",
      "searchMode": "hybrid",
      "limit": 5
    }' | jq .

  # Extract the authoritative X-AppBase-Trace-Id from the response headers
  TRACE_ID="$(grep -i '^x-appbase-trace-id:' /tmp/memory_trace_headers.txt | awk '{print $2}' | tr -d '\r')"
  echo "Captured X-AppBase-Trace-Id: ${TRACE_ID}"

  # 2. Query the project's Memory trace list by the capability:memory tag and userId
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS --get "${APPBASE_ENDPOINT}/observability/traces" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
    --data-urlencode "tag=capability:memory" \
    --data-urlencode "userId=user_zhangwei_001" \
    --data-urlencode "limit=10" | jq .

  # 3. Using the captured TRACE_ID, query the complete Span waterfall tree of a single Trace along with Generation token details
  builtin printf 'X-Appwrite-Key: %s\n' "${APPBASE_API_KEY:?Set APPBASE_API_KEY}" | curl --disable --header @- -fsS "${APPBASE_ENDPOINT}/observability/traces/${TRACE_ID}" \
    -H "X-Appwrite-Project: ${APPBASE_PROJECT_ID}" \
     | jq .
)

FAQ (Frequently Asked Questions and Troubleshooting)

1. Why do I see successful request records in "Memory → Audit" but cannot find the corresponding X-AppBase-Trace-Id in the trace list after clicking "Trace"?

2. Why does a Memory Trace Generation omit model input and output?

3. Why do the model call counts and Token consumption shown in “Agent Operations → Metrics” appear higher than the statistics of calling the Chat API directly from business code?