Docs
Contents

AI Gateway (Unified AI Data Plane)

Overview and use cases

AI Gateway is the unified AI data plane and control plane introduced in 2.6.0 (services/ai-gateway), hosting model inference, Target onboarding, Tool/MCP, A2A, HTTP Agent, identity (Consumer/Key), policy (Policy), budgets, caching, and auditing. Resource model: Gateway / Target / Tool / Consumer / Policy / Snapshot — Targets and Tools are project-level assets that enter one or more Gateway Snapshots via mount relationships.

Coexists with the existing LLM Gateway: it does not migrate legacy resources, does not take over legacy production paths, and does not provide compatibility aliases without gatewayId. As long as you use existing Provider/Gateway Key/Chat/Embeddings/Rerank/VLM, continue with llm-gateway.md; use this page only when Gateway/Target/Tool/Consumer/Policy/MCP/A2A is involved. The two systems' names, credentials, and paths are not interchangeable.

Prerequisites

Usage workflow

Data plane entry points (HTTP, added in 2.6.0; SDKs do not wrap these paths):

Control plane (SDK AiGateway service):

  1. Create a Gateway and manage policies: createPolicyEngine / listPolicyEngines / getPolicyEngine; updateGatewayPolicy({ gatewayId, ... }); policy bundles createRevision/publishPolicyBundle/listPolicyBundles/getPolicyBundle; validate with dryRunPolicy before going live; release semantics use Snapshots: sealSnapshot / getSnapshotStatus.

  2. Onboard Targets: createResource / listResources / getResource / replaceResource (kind is the resource category path parameter); revision management createRevision / getRevision / transitionRevision (/resources/{kind}/{id}/revisions/{revisionId}/{action}); health and discovery checkTargetHealth / discoverTargetModels; mounting listTargetMounts / replaceTargetMounts.

  3. Model deployments and standard models: testModelDeployment / getModelDeploymentMetrics; createStandardModel / importStandardModels / listStandardModels / updateStandardModel.

  4. Tool/MCP connections: createToolConnection / discoverToolConnection / updateToolConnection / deleteToolConnection.

  5. Consumer credentials: createKey / listKeys / getKey / updateKeyStatus (issuance and enable/disable of abk_).

  6. Observability and auditing: getObservabilityOverview / listInvocations / getInvocation.

2.7.0 additions (appbase release-2.7.0 PR 526):

2.8.0 additions:

Key parameters and returns

Errors and boundaries

SDK Category Differences

Examples by language

The following provides examples by language, item by item along the usage workflow; each card can switch languages (synced across the page).

Data plane entry points (Inference / MCP / A2A / HTTP Agent)

Inference / MCP / A2A / HTTP Agent are HTTP data plane entry points; SDKs in all languages (including Web) do not wrap these paths, so this card only has a REST tab. Public listeners use the abk_ Key issued by the target Gateway (carried via Authorization: Bearer); the LLM Gateway Key, project API Key, and upstream Provider Key cannot be used as abk_.

# Inference: Responses / Chat / Embeddings / Messages share the /ai-gateway/g/<GATEWAY_ID>/v1 prefix
curl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/g/<GATEWAY_ID>/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $AI_GATEWAY_KEY" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"你好"}]}'
# MCP: /ai-gateway/g/<GATEWAY_ID>/mcp (tool list/search/call share the Agent allowlist and OPA authorization; side effects are protected by an idempotency gate)
# A2A: /ai-gateway/g/<GATEWAY_ID>/a2a/<targetSlug> (Task/Context isolated per caller + Target Revision)
# HTTP Agent: /ai-gateway/g/<GATEWAY_ID>/http/<targetSlug>[/{path...}] (only forwards methods/relative paths/query/body allowed by the Target,
#   Caller authentication Headers are not replayed; credentials are resolved from the managed Credential)

Create a Gateway and manage policies (Policy engine / Policy bundle / Snapshot)

After the policy engine is created, validate the policy bundle via dry-run, then publish it and attach it to the Gateway via updateGatewayPolicy; sealSnapshot seals the current configuration (asynchronously). Changes to Target / Consumer / Tool allowlist / Policy only take effect for public traffic on Snapshots that are published and consistent.

await aiGateway.listPolicyEngines();                             // Policy engine list
await aiGateway.updateGatewayPolicy({ gatewayId: '<gatewayId>' }); // Mount strategy (fields see SDK type definitions)
await aiGateway.sealSnapshot({ gatewayId: '<gatewayId>' });      // Archived as a Snapshot (asynchronous)
await aiGateway.getSnapshotStatus({ gatewayId: '<gatewayId>' }); // Poll archive results
// Policy engine create/detail: createPolicyEngine / getPolicyEngine
// Policy bundles: createRevision / publishPolicyBundle / listPolicyBundles / getPolicyBundle; validate with dryRunPolicy before rollout (evaluates only, does not execute)

Onboard Targets (Resources / Revisions / Mounting)

Targets are onboarded via the unified resource endpoint (kind as a path parameter) and brought online through the revision state machine. Health checks, model discovery, and full replacement of mounts are resource-specific operations.

// Unified resource surface: listResources / createResource / getResource / replaceResource / deleteResource
//   kind is the resource category path parameter; the request body is a generalized schema, subject to SDK type definitions
// Revisions: createRevision / getRevision / transitionRevision (resource go-live uses the revision state machine)
// Target operations: checkTargetHealth / discoverTargetModels; mounting: listTargetMounts / replaceTargetMounts

Model deployments and standard models

Model deployments allow testing one real inference before going live; the standard model catalog supports manual registration as well as batch import, merge, and update of models discovered from targets.

// Model deployment: testModelDeployment (live deployment inference) / getModelDeploymentMetrics (deployment metrics)
// Standard models: createStandardModel / importStandardModels / listStandardModels / updateStandardModel

Tool/MCP connections

A Tool connection is the upstream onboarding for Tool/MCP data plane calls: after creation, you can trigger tool discovery, test connectivity, update configuration, and delete it (once deleted, tools can no longer be called via this connection on the gateway side). Connectivity testing is supported starting from 2.8.0: testToolConnection(id, toolId) (toolId optional) (POST /ai-gateway/tool-connections/{id}/test), which returns diagnostic results (target reachability, credential checks, and per-tool status); the Console connection diagnostics panel uses the same source.

// createToolConnection / discoverToolConnection / updateToolConnection / deleteToolConnection
// 2.8.0 Connectivity test: aiGateway.testToolConnection('<id>'); the optional second parameter toolId tests a single tool only
// payload is defined by the target resource; see the SDK type definitions for fields

Issuing and enabling/disabling Consumer credentials (abk_ Keys)

createKey issues abk_ credentials; use gatewayIds to scope which gateways can be called, and allowedModels / allowedTools / allowedAgents to scope models, tools, and Agent Targets; once disabled, they immediately fail gateway validation.

import { AiGateway } from 'agentbase-node-sdk';

const aiGateway = new AiGateway(client);   // Control plane: aiGateway.read / aiGateway.write

await aiGateway.listKeys();                // abk_ metadata view
await aiGateway.createKey({
    name: 'agent-consumer',
    gatewayIds: ['<gatewayId>']            // For the allowlist and principal fields, see the SDK type definitions
});
// Enable/disable: aiGateway.updateKeyStatus(...) (after disabling, an abk_ key immediately fails gateway validation)

Observability and audit

Call records and the observability overview are queried by aiGateway.read scope, used for troubleshooting and auditing gateway calls.

await aiGateway.getObservabilityOverview();   // Observability overview
await aiGateway.listInvocations();            // Invocation record list (filter parameters see SDK type definitions)
// Single invocation record: getInvocation

Example Guide