Docs
Contents

Observability

Overview and use cases

Observability covers two things:

  1. AI observation reporting: report Trace / Observation / Generation / Score (Langfuse-style event model) from controlled server, CLI, Function or Site SSR to the current project, for Agent/LLM call chain debugging, token usage and quality evaluation.

  2. Query plane: query event trees by Trace, aggregate by Session/User, metric queries; plus independent Trace log queries (stdout/stderr of Functions/dynamic Sites, by W3C Trace ID or Execution ID).

Prerequisites

Choose an integration method

PrerequisitesIntegration methodApplicability boundary
Already using OpenTelemetry SDK / ExporterStandard exporter → OTLP/HTTP Trace endpointJSON or protobuf, optional gzip; OTLP/gRPC not supported
Agentbase Node.js server applicationsPrecisely install agentbase-node-sdk@2.10.0, call createIngestion or createOtelTracesReuse Client and project authentication; createOtelTraces(body, contentType) (contentType optional) preserves the original body shape
Controlling HTTP requests yourselfREST → Events or OTLP Trace endpointFor REST parameters and error model see API reference

The OpenTelemetry exporter uses the full traces-specific address; the APPBASE_ENDPOINT below already includes /v1, and other variables are injected by the controlled process environment:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="${APPBASE_ENDPOINT%/}/observability/otel/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="X-Appwrite-Project=${APPBASE_PROJECT_ID},X-Appwrite-Key=${APPBASE_API_KEY}"

Do not append /v1/traces to the endpoint again. In this version the OTLP endpoint returns an Agentbase event batch JSON; how the target exporter handles this response must be verified against the actual version—full compatibility cannot be assumed from the OTLP name alone. Production exporters should use bounded queues, batching, short timeouts and limited retries; when the observability backend is unavailable, record sanitized failure/drop counts to avoid blocking business logic. When sending gzip body directly with the Agentbase SDK, the client must also explicitly set the corresponding Content-Encoding; ordinary exporters send according to their own configuration.

Reporting acceptance has two steps: first check the errors[] of the batch response, then use independent read credentials or the Console to confirm asynchronous persistence by the same Trace ID. Status probing requires observability.status.read; Trace lookback requires observability.traces.read; these read capabilities are separate from write permission—do not expand production exporter permissions for debugging convenience.

Usage workflow

  1. Batch reporting: observability.createIngestion({ batch, metadata }) (metadata optional)——batch takes 1–100 events (empty batches or more than 100 are rejected); event structure { id, timestamp, type, body }:

    • The outer id is the idempotency ID (use a new unique value for each create/update event); body.id is the entity ID (keep it unchanged when updating the same entity).

    • Types: trace-create, span-create/span-update, generation-create/generation-update, score-create, etc.; an Observation's body.traceId points to the root Trace, and a child Observation uses parentObservationId to point to its parent.

    • Trace IDs are recommended as 32 lowercase hex characters (not all zeros), Observations 16 hex characters, compatible with W3C traceparent; timestamps use ISO8601 with timezone; level ∈ DEBUG | DEFAULT | WARNING | ERROR.

  2. Handle responses: all success → HTTP 200; any failure → HTTP 207 Multi-Status, you must check each item's errors[] individually, not just whether the request threw. Neither 200 nor 207 means persisted (an async worker writes to the database): when reading back 404, do bounded short polling; if still absent, treat it as "received but persistence unverified".

  3. Retry policy: for 429 and recoverable 5xx use capped exponential backoff andreuse the original outer id; do not blindly retry schema/permission 4xx; for more than 100 items, split into batches in create-before-update order.

  4. OTel traces reporting: observability.createOtelTraces(body, contentType) (contentType optional)——OTLP HTTP form (JSON/protobuf, optional gzip); the SDK sends the raw body directly and returns the raw response; when an OpenTelemetry SDK already exists, the exporter connects directly to the HTTP endpoint. Not an OTLP gRPC endpoint.

  5. Queries: getTrace / listTraces / getSession / listSessions / getUser / listUsers / queryMetrics / listScores / listScoreConfigs; status getStatus.

  6. Trace log query: observability.listLogs({ query, from, to, limit, offset }) (from, to, limit, offset all optional)——query is required and only one traceId:<value> or executionId:<value> is allowed; from defaults to to−3600 (Unix seconds), to defaults to the current time, the time window is at most 24 hours; limit 1–100 (default 25); offset 0–1000.

Key parameters and returns

Errors and boundaries

2.7.0 increments: trajectory evaluation and monitoring metrics

The following capabilities live in the Console evaluation plane and the evaluation worker / deployment monitoring side; the public ingestion and query contract (endpoints on this page) is unchanged (spec v2.6.0 → v2.7.0 observability section adds no new endpoints or parameter changes):

Increment 2.8.0: Evaluation MCP tool discovery

Evaluation tasks can attach MCP tools to evaluators: before configuration, discover the read-only MCP tools available on the gateway using restricted gateway credentials, then select the tools to attach in the evaluation configuration. New method observability.discoverEvalMcpTools(gatewayId, gatewayKeyId) — POST /observability/evals/mcp/tools, request body { gatewayId, gatewayKeyId } (the AI Gateway's gateway ID and restricted Gateway Key ID); returns the list of read-only (no side effects) MCP tools exposed by the gateway. scope is observability.evals.write; gatewayKeyId must be an active sub-key bound to the gateway (explicitly scoped tools, without model / agent permissions).

Increment 2.9.0: Observation list and score analysis

Increment 2.10.0: Knowledge index quality evaluation and Prompt parsing

SDK Category Differences

Example Guide

Complete Agent trajectory reporting example

This complete ESM script uses Node.js 18+ native fetch and node:crypto, without extra dependencies. It calls the public REST endpoints directly (POST /v1/observability/ingestion and optional readback through GET /v1/observability/traces/{traceId}) to demonstrate event relationships and layered verification. It is not a high-level method built into a particular SDK version.

Scenario and events

The example uses a synthetic order-status query: a user asks whether order ORD-20261022-889 has shipped.

  1. First model call (generation-create): receives system and user messages and outputs an assistant tool request through tool_calls for get_order_status.

  2. Tool execution (tool-create): references the same call ID through metadata.toolCallId and returns a shipped status.

  3. Second model call (generation-create): input.messages preserves the earlier system, user, assistant with tool_calls, and tool with the matching tool_call_id. Its output contains only the final assistant answer.

The batch contains one trace-create, two generation-create, and one tool-create: four events. The script generates every ID. The model identity (synthetic-order-model-demo) and Token values are synthetic demonstration data. It does not connect to or impersonate a real model, and does not establish model evaluation success.

Runtime and environment variables

VariableRequiredScopeDescription
APPBASE_ENDPOINTYes—Public API endpoint of the selected instance. Its path must be /v1, such as https://<your-endpoint>/v1.
APPBASE_PROJECT_IDYes—Target project ID.
APPBASE_API_KEYYesobservability.ingest.writeWrite API Key for ingestion.
APPBASE_OBSERVABILITY_READ_KEYNoobservability.traces.readSeparate verification Key for bounded polling of GET /observability/traces/{traceId}. Without it, the script verifies acceptance only.

Complete ESM script (report-agent-trajectory.mjs)

import { randomBytes, randomUUID } from 'node:crypto';

class ExampleError extends Error {}

/**
 * 校验并规范化公开 Endpoint,要求路径必须为 /v1,且不含内嵌凭据、query 或 hash
 */
function normalizeEndpoint(rawEndpoint) {
  if (!rawEndpoint || typeof rawEndpoint !== 'string') {
    throw new ExampleError('缺少必填环境变量 APPBASE_ENDPOINT(路径须为 /v1)');
  }
  const url = new URL(rawEndpoint);
  if (!['http:', 'https:'].includes(url.protocol)) {
    throw new ExampleError('APPBASE_ENDPOINT 必须使用 http 或 https 协议');
  }
  if (url.username || url.password || url.search || url.hash) {
    throw new ExampleError('APPBASE_ENDPOINT 不得包含内嵌凭据、查询参数或片段标识符');
  }
  const cleanPath = url.pathname.replace(/\/+$/, '');
  if (cleanPath !== '/v1') {
    throw new ExampleError('APPBASE_ENDPOINT 路径必须为 /v1(如 https://<your-endpoint>/v1)');
  }
  return `${url.origin}${cleanPath}`;
}

/**
 * 生成指定字节数的非全零小写十六进制字符串(16 bytes = 32 hex,8 bytes = 16 hex)
 */
function generateNonZeroHex(byteLength) {
  while (true) {
    const hex = randomBytes(byteLength).toString('hex');
    if (!/^0+$/.test(hex)) {
      return hex;
    }
  }
}

/**
 * 构造合成订单查询场景的 4 个事件:
 * 1 个 trace-create + 2 个 generation-create + 1 个 tool-create
 * 发送前固定该 batch,重试时复用同一外层 event.id
 */
function buildSyntheticOrderTraceBatch() {
  const traceId = generateNonZeroHex(16); // 32-character non-zero lowercase hex
  const firstGenId = generateNonZeroHex(8); // 16-character non-zero lowercase hex
  const toolObsId = generateNonZeroHex(8); // 16-character non-zero lowercase hex
  const secondGenId = generateNonZeroHex(8); // 16-character non-zero lowercase hex
  const toolCallId = `call_${generateNonZeroHex(6)}`;

  const baseMs = Date.now() - 1000;
  const isoAt = (offsetMs) => new Date(baseMs + offsetMs).toISOString();

  // Synthetic demonstration model identity; no real model call
  const syntheticModel = 'synthetic-order-model-demo';
  const tools = [{
    type: 'function',
    function: {
      name: 'get_order_status',
      description: '查询合成订单状态',
      parameters: { type: 'object', properties: { orderId: { type: 'string' } }, required: ['orderId'] },
    },
  }];

  const systemMsg = {
    role: 'system',
    content: '你是企业订单助手,请通过工具查询订单真实状态后回答用户。',
  };
  const userMsg = {
    role: 'user',
    content: '请帮我查一下订单 ORD-20261022-889 发货了吗?',
  };
  const firstAssistantToolCallMsg = {
    role: 'assistant',
    content: '',
    tool_calls: [
      {
        id: toolCallId,
        type: 'function',
        function: {
          name: 'get_order_status',
          arguments: { orderId: 'ORD-20261022-889' },
        },
      },
    ],
  };
  const toolResultPayload = {
    orderId: 'ORD-20261022-889',
    status: 'SHIPPED',
    carrier: 'SF-Express',
    updatedAt: isoAt(350),
  };
  const toolRoleMsg = {
    role: 'tool',
    tool_call_id: toolCallId,
    content: JSON.stringify(toolResultPayload),
  };
  const finalAssistantAnswer =
    '您的订单 ORD-20261022-889 已由顺丰速运(SF-Express)发出,当前状态为已发货(SHIPPED)。';

  const batch = [
    {
      id: randomUUID(),
      timestamp: isoAt(0),
      type: 'trace-create',
      body: {
        id: traceId,
        timestamp: isoAt(0),
        name: 'synthetic-order-agent.run',
        userId: 'demo_user_order_01',
        sessionId: `sess_order_demo_${traceId.slice(0, 8)}`,
        environment: 'development',
        tags: ['agentbase-external', 'synthetic-demo'],
        input: { content: userMsg.content },
        output: { role: 'assistant', content: finalAssistantAnswer },
        metadata: {
          syntheticDemo: true,
          note: 'Synthetic order inquiry trajectory demo; no real LLM invoked',
        },
      },
    },
    {
      id: randomUUID(),
      timestamp: isoAt(250),
      type: 'generation-create',
      body: {
        id: firstGenId,
        traceId,
        name: 'llm.plan_tool_call',
        startTime: isoAt(20),
        endTime: isoAt(240),
        model: syntheticModel,
        modelParameters: { temperature: 0 },
        input: {
          tools,
          messages: [systemMsg, userMsg],
        },
        output: firstAssistantToolCallMsg,
        // Illustrative usage, recorded separately for each model call
        usage: {
          input: 58,
          output: 24,
          total: 82,
          unit: 'TOKENS',
        },
        metadata: {
          syntheticDemo: true,
          stepRole: 'tool_decision',
        },
      },
    },
    {
      id: randomUUID(),
      timestamp: isoAt(420),
      type: 'tool-create',
      body: {
        id: toolObsId,
        traceId,
        parentObservationId: firstGenId,
        name: 'get_order_status',
        startTime: isoAt(260),
        endTime: isoAt(400),
        level: 'DEFAULT',
        input: { orderId: 'ORD-20261022-889' },
        output: toolResultPayload,
        metadata: {
          // Match output.tool_calls[].id and the later tool message tool_call_id
          toolCallId,
          syntheticDemo: true,
        },
      },
    },
    {
      id: randomUUID(),
      timestamp: isoAt(720),
      type: 'generation-create',
      body: {
        id: secondGenId,
        traceId,
        name: 'llm.final_answer',
        startTime: isoAt(440),
        endTime: isoAt(700),
        model: syntheticModel,
        modelParameters: { temperature: 0 },
        // Report the second call context as supplied; the platform handles prefix overlap
        input: {
          tools,
          messages: [
            systemMsg,
            userMsg,
            firstAssistantToolCallMsg,
            toolRoleMsg,
          ],
        },
        // Record only this assistant response in output, without copying the input Prompt
        output: {
          role: 'assistant',
          content: finalAssistantAnswer,
        },
        usage: {
          input: 112,
          output: 36,
          total: 148,
          unit: 'TOKENS',
        },
        metadata: {
          syntheticDemo: true,
          stepRole: 'final_response',
        },
      },
    },
  ];

  return { traceId, batch };
}

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
const retryable = (status) => [429, 500, 502, 503, 504].includes(status);

async function submitBatchWithRetry({ endpoint, projectId, writeKey, batch }) {
  if (!Array.isArray(batch) || batch.length < 1 || batch.length > 100) {
    throw new ExampleError('Events batch 必须包含 1–100 个事件');
  }
  // Reuse this request body on retries in the same process; do not regenerate event IDs.
  const body = JSON.stringify({ batch });
  const expected = new Set(batch.map((event) => event.id));
  if (expected.size !== batch.length) throw new ExampleError('事件 ID 必须唯一');
  for (let attempt = 1; attempt <= 3; attempt++) {
    let response, text;
    try {
      response = await fetch(`${endpoint}/observability/ingestion`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'X-Appwrite-Project': projectId,
          'X-Appwrite-Key': writeKey,
        },
        body,
        signal: AbortSignal.timeout(10000),
        redirect: 'error',
      });
      text = await response.text();
    } catch {
      if (attempt === 3) throw new ExampleError('上报网络或读取响应失败:受理结果未确认');
      await sleep(300 * 2 ** (attempt - 1));
      continue;
    }
    if (retryable(response.status) && attempt < 3) {
      await sleep(300 * 2 ** (attempt - 1));
      continue;
    }
    if (![200, 207].includes(response.status)) {
      throw new ExampleError(`上报未确认:HTTP ${response.status}`);
    }
    let payload;
    try { payload = JSON.parse(text); }
    catch { throw new ExampleError('上报响应非 JSON:受理结果未确认'); }
    if (!payload || !Array.isArray(payload.successes) || !Array.isArray(payload.errors)) {
      throw new ExampleError('上报响应缺少 successes/errors:受理结果未确认');
    }
    if (response.status === 207 || payload.errors.length > 0) {
      // Do not print raw server errors, which may contain submitted content.
      const statuses = payload.errors.map((item) => Number.isInteger(item?.status) ? item.status : 'unknown');
      throw new ExampleError(`批次未全部受理,HTTP ${response.status},失败状态 ${statuses.join(',')};保留原批次处理失败项`);
    }
    const acknowledged = new Set();
    for (const item of payload.successes) {
      if (!expected.has(item?.id) || acknowledged.has(item.id) ||
          !Number.isInteger(item.status) || item.status < 200 || item.status >= 300) {
        throw new ExampleError('上报成功项身份或状态不符合契约:受理结果未确认');
      }
      acknowledged.add(item.id);
    }
    if (acknowledged.size !== expected.size) throw new ExampleError('上报成功项未覆盖原批次:受理结果未确认');
    return;
  }
}

async function verifyTracePersistence({ endpoint, projectId, readKey, traceId, observations }) {
  for (let poll = 1; poll <= 5; poll++) {
    let response, payload;
    try {
      response = await fetch(`${endpoint}/observability/traces/${encodeURIComponent(traceId)}`, {
        headers: { 'X-Appwrite-Project': projectId, 'X-Appwrite-Key': readKey },
        signal: AbortSignal.timeout(8000),
        redirect: 'error',
      });
      if (response.status === 200) payload = await response.json();
    } catch { /* Readback remains unconfirmed; continue only within this bounded polling loop */ }
    if (response && response.status !== 200 && response.status !== 404 && !retryable(response.status)) {
      throw new ExampleError(`回读失败:HTTP ${response.status},请核对读取权限与请求`);
    }
    if (payload?.trace?.id === traceId && Array.isArray(payload.observations)) {
      const actual = new Map(payload.observations.map((item) => [item.id, item]));
      const ready = observations.every((expected) => {
        const item = actual.get(expected.id);
        return item?.traceId === traceId && item.type === expected.type &&
          (item.parentObservationId ?? null) === (expected.parentObservationId ?? null) &&
          item.input != null && item.output != null;
      });
      if (ready) {
        const trajectory = payload.trajectory;
        return {
          persisted: true,
          // Check version and non-empty steps only; do not claim complete evaluation success.
          hasAtifSteps: trajectory?.schema_version === 'ATIF-v1.8' &&
            Array.isArray(trajectory.steps) && trajectory.steps.length > 0,
        };
      }
    }
    if (poll < 5) await sleep(1500);
  }
  return { persisted: false, hasAtifSteps: false };
}

async function main() {
  const endpoint = normalizeEndpoint(process.env.APPBASE_ENDPOINT);
  const projectId = process.env.APPBASE_PROJECT_ID?.trim();
  const writeKey = process.env.APPBASE_API_KEY?.trim();
  const readKey = process.env.APPBASE_OBSERVABILITY_READ_KEY?.trim();
  if (!projectId || !writeKey) throw new ExampleError('缺少 APPBASE_PROJECT_ID 或 APPBASE_API_KEY');
  const { traceId, batch } = buildSyntheticOrderTraceBatch();
  console.log(`合成示例 Trace ID:${traceId}`);
  await submitBatchWithRetry({ endpoint, projectId, writeKey, batch });
  console.log('本批事件已全部受理;异步持久化与轨迹评估仍需分别验收');
  if (!readKey) {
    console.log('未配置独立读 Key:仅验证受理,未验证持久化');
    return;
  }
  const observations = batch.filter((event) => event.type !== 'trace-create').map((event) => ({
    ...event.body, type: event.type === 'generation-create' ? 'GENERATION' : 'TOOL',
  }));
  const result = await verifyTracePersistence({ endpoint, projectId, readKey, traceId, observations });
  if (!result.persisted) {
    console.log('受理已确认,但在有限回读内未确认本次 Trace 与全部预期 Observation 落库');
    process.exitCode = 2;
    return;
  }
  console.log(`本次 Trace 和预期 Observation 已查到;存在非空 ATIF-v1.8 步骤:${result.hasAtifSteps}`);
  console.log('仍需在 Console 核对消息、工具结果和所选评估器证据;本脚本不证明评估成功');
}

main().catch((error) => {
  // Do not print unknown exception text that may reveal request or server content.
  console.error(error instanceof ExampleError ? error.message : '示例未完成验收:未知异常,受理与落库结果均需核对');
  process.exitCode = 1;
});

Some comments, runtime messages, and synthetic example data retain Chinese to keep the executable example aligned with the source-language guide.

Demonstration and production behavior

Verification scope and repeated runs

Within one process, the script makes at most three requests for network errors, 429, and the listed recoverable 5xx statuses, reusing the same batch. It stops on 207, invalid responses, or incomplete event acknowledgments. Production callers should retain the original batch and failed entries before correcting and retrying, so event identities are not lost. Restarting this synthetic script creates a new demonstration Trace; it does not implement cross-process retries.

Readback checks the three expected Observations by entity ID, type, Trace, parent, and non-null input/output. A node count alone is insufficient. These checks do not prove full content equality, complete ATIF, or model quality. The script does not call a real model; running it writes synthetic data to the explicitly selected project, so use a test project.

Read the product manual's trajectory concepts and four verification layers, then consult Message and tool association formats.

Examples by language

The following provides examples by language, item by item along the usage workflow; each card can switch languages (synced across the page).

Batch reporting (JSON ingestion)

batch accepts 1–100 events; all succeed with HTTP 200; any failure returns HTTP 207 Multi-Status, and you must check errors[] item by item — neither 200/207 means persisted (an asynchronous worker writes to the database), so do bounded short polling when reading back after 404. Use bounded exponential backoff for 429 and recoverable 5xx while reusing the original outer id; do not blindly retry schema/permission 4xx; batch by create-before-update order when exceeding 100 items.

import { Client, Observability } from 'agentbase-node-sdk';

const client = new Client()
    .setEndpoint(process.env.APPBASE_ENDPOINT!)   // Must include /v1
    .setProject(process.env.APPBASE_PROJECT_ID!)
    .setKey(process.env.APPBASE_API_KEY!);        // Write scope observability.ingest.write
const observability = new Observability(client);

const res = await observability.createIngestion({
    batch: [{
        id: crypto.randomUUID(),          // Outer idempotency ID: use a new unique value for each create/update event
        timestamp: new Date().toISOString(),
        type: 'trace-create',
        body: { id: traceId32Hex, name: 'agent-run' }   // body.id is the entity ID; it stays unchanged when updating the same entity
    }]
});
// All succeed: HTTP 200; any failure: HTTP 207 Multi-Status — check res.errors[] item by item; do not rely only on whether an error is thrown

OTel Traces reporting

The SDK sends the raw OTLP body directly and returns the raw response (this page shows the Node SDK and REST forms); other languages can point an OpenTelemetry exporter directly at the same REST endpoint.

// createOtelTraces(body, contentType?) —— OTLP HTTP form (JSON/protobuf, optional gzip)
const raw = await observability.createOtelTraces(otelBody);
// Not an OTLP gRPC endpoint; suitable for OpenTelemetry exporters

Queries and Trace logs

getStatus is a status probe; Trace log queries are a separate data plane (scope traceLogs.read).

const status = await observability.getStatus();   // scope observability.status.read; returns ingestionEndpoints
// The query plane also provides getTrace / listTraces / getSession / listSessions / getUser / listUsers / queryMetrics / listScores / listScoreConfigs

// Trace logs: query is required and allows only one traceId:<value> or executionId:<value>
const logs = await observability.listLogs({ query: `executionId:${execId}`, limit: 50 });