AI Gateway (Unified AI Data Plane)
Overview and use cases
AI Gateway is the unified AI data plane and control plane introduced in 2.6.0 (services/ai-gateway), hosting model inference, Target onboarding, Tool/MCP, A2A, HTTP Agent, identity (Consumer/Key), policy (Policy), budgets, caching, and auditing. Resource model: Gateway / Target / Tool / Consumer / Policy / Snapshot — Targets and Tools are project-level assets that enter one or more Gateway Snapshots via mount relationships.
Coexists with the existing LLM Gateway: it does not migrate legacy resources, does not take over legacy production paths, and does not provide compatibility aliases without gatewayId. As long as you use existing Provider/Gateway Key/Chat/Embeddings/Rerank/VLM, continue with llm-gateway.md; use this page only when Gateway/Target/Tool/Consumer/Policy/MCP/A2A is involved. The two systems' names, credentials, and paths are not interchangeable.
Prerequisites
Enablement gating: Docker Compose requires explicitly enabling the
ai-gatewayprofile; aggregated Helm defaults toaiGateway.enabled=false; currently only OceanBase Shared V2 projects are supported. The project control tables are initialized only when the project service is explicitly enabled for the first time.Public listeners use an abk_ Key issued by the target Gateway: verifies the consistency of Key + Gateway + Consumer + published Snapshot. LLM Gateway Keys, project API Keys, and upstream Provider Keys cannot be used as abk_ keys.
Internal services use a separate internal listener + signed runtime context (no credentials are sent); the internal listener must not be exposed to the Ingress, and the public listener must reject internal contexts.
Usage workflow
Data plane entry points (HTTP, added in 2.6.0; SDKs do not wrap these paths):
Inference:
/ai-gateway/g/{gatewayId}/v1(Responses / Chat / Embeddings / Messages).MCP:
/ai-gateway/g/{gatewayId}/mcp(Tool listing/search/calling share the Agent allowlist and OPA authorization; side effects are protected by the idempotency gate).A2A:
/ai-gateway/g/{gatewayId}/a2a/{targetSlug}(Task/Context are isolated by caller + Target Revision).HTTP Agent:
/ai-gateway/g/{gatewayId}/http/{targetSlug}[/{path...}](only forwards methods/relative paths/query/body allowed by the Target, does not replay caller authentication headers, and credentials are resolved from the controlled Credential).
Control plane (SDK AiGateway service):
Create a Gateway and manage policies:
createPolicyEngine/listPolicyEngines/getPolicyEngine;updateGatewayPolicy({ gatewayId, ... }); policy bundlescreateRevision/publishPolicyBundle/listPolicyBundles/getPolicyBundle; validate withdryRunPolicybefore going live; release semantics use Snapshots:sealSnapshot/getSnapshotStatus.Onboard Targets:
createResource/listResources/getResource/replaceResource(kindis the resource category path parameter); revision managementcreateRevision/getRevision/transitionRevision(/resources/{kind}/{id}/revisions/{revisionId}/{action}); health and discoverycheckTargetHealth/discoverTargetModels; mountinglistTargetMounts/replaceTargetMounts.Model deployments and standard models:
testModelDeployment/getModelDeploymentMetrics;createStandardModel/importStandardModels/listStandardModels/updateStandardModel.Tool/MCP connections:
createToolConnection/discoverToolConnection/updateToolConnection/deleteToolConnection.Consumer credentials:
createKey/listKeys/getKey/updateKeyStatus(issuance and enable/disable of abk_).Observability and auditing:
getObservabilityOverview/listInvocations/getInvocation.
2.7.0 additions (appbase release-2.7.0 PR 526):
Credential governance:
createKeyadds an optionallimits— an array of hard rate-limit/quota rules, with elements{ scope, resourceId, rate, quota }(resourceId,rate,quotaall optional):scopetakescredential/model/tool/agent(non-credentialrequires specifyingresourceId);rateisrequestsPerSecond+burst(burstdefaults torequestsPerSecond);quotalimitsrequests/tokensperminute/hour/day/monthperiod (tokensonly forcredential/modelscope). The gateway data plane enforces rules fail-closed.Tool approval closed loop: The Tool's static approval floor (
none/always/policy) combined with the policy engine decision composes a fail-closed runtime gate; hittingREQUIRE_APPROVALgenerates an approval request (ID with aapr_prefix, TTL 15 minutes, replayed calls correlated via theX-AppBase-Approval-Request-Idheader); policy-enforced denials cannot be elevated to allow by human approval. The approval surface adds no new public SDK methods/endpoints.Managed tool templates: Console Tool connections can optionally use managed templates —
standard-models(standard model management MCP tool group) andrag(managed RAG:list_knowledge_bases/search/multi_search, wheretargetKindis the newly addedMCP_RAG).
2.8.0 additions:
Tool connection connectivity test:
testToolConnection(id, toolId)(toolIdoptional) (POST /ai-gateway/tool-connections/{id}/test, 2.8.0) — performs connectivity testing and diagnostics on tool connections (target reachability, credentials, and per-tool checks, same source as the Console connection diagnostics panel); optionaltoolIdtests only a single tool.Logical model mount query and full replacement:
listLogicalModelMounts(id)(GET /ai-gateway/resources/logical-models/{id}/mounts) andreplaceLogicalModelMounts(id)(PUT .../mounts) (2.8.0) — logical models enter the gateway via mount relationships, with semantics identical to Target mounts: the query returns the mount list, and replacement is a full overwrite (existing mounts not included in the request body will be removed; use with caution).
Key parameters and returns
For resource-class methods,
kindis a path parameter (resource category) used together with the resource ID to locate revisions and mounts; available categories are subject to the target instance's specs/responses; do not guess category names.Snapshot: the landing point of release gating — changes to Target/Consumer/Tool allowlist/Policy only take effect for public traffic on a Snapshot that is published and consistent.
Policy bundles are validated via dry-run before publish;
transitionRevisionperforms the revision state transition actions.
Errors and boundaries
General outbound traffic is subject to SSRF protection, redirect, and request/response size limits; Targets are prohibited from accessing private networks and cloud metadata addresses (2.6.0 security defaults).
Provider/Target credentials store references only; plaintext never enters model inputs, audits, or traces. Tool arguments/results and Prompt/Output are not written to observability by default; only hashes/enums/bounded facts are recorded.
A2A/HTTP methods, outbound addresses, and redirects all pass through allowlists; unauthorized access is denied.
The project-level AI Gateway switch is the existing
llmGatewayClient API service switch, not a switch for the new control plane.
SDK Category Differences
Server SDK (server-sdk/nodejs.md) and Client SDK (client-sdk/web.md):
AiGatewayThe service plane is consistent (full control plane); data plane inference/MCP/A2A/HTTP go through HTTP entry points.Console SDK (console-sdk/nodejs.md): 2.6.0 does not include the AI Gateway service.
Both the Java Server / Console SDK have
AiGatewayservices.
Examples by language
The following provides examples by language, item by item along the usage workflow; each card can switch languages (synced across the page).
Data plane entry points (Inference / MCP / A2A / HTTP Agent)
Inference / MCP / A2A / HTTP Agent are HTTP data plane entry points; SDKs in all languages (including Web) do not wrap these paths, so this card only has a REST tab. Public listeners use the abk_ Key issued by the target Gateway (carried via Authorization: Bearer); the LLM Gateway Key, project API Key, and upstream Provider Key cannot be used as abk_.
- REST (curl)
# Inference: Responses / Chat / Embeddings / Messages share the /ai-gateway/g/<GATEWAY_ID>/v1 prefix
curl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/g/<GATEWAY_ID>/v1/chat/completions" \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $AI_GATEWAY_KEY" \
-d '{"model":"<model>","messages":[{"role":"user","content":"你好"}]}'
# MCP: /ai-gateway/g/<GATEWAY_ID>/mcp (tool list/search/call share the Agent allowlist and OPA authorization; side effects are protected by an idempotency gate)
# A2A: /ai-gateway/g/<GATEWAY_ID>/a2a/<targetSlug> (Task/Context isolated per caller + Target Revision)
# HTTP Agent: /ai-gateway/g/<GATEWAY_ID>/http/<targetSlug>[/{path...}] (only forwards methods/relative paths/query/body allowed by the Target,
# Caller authentication Headers are not replayed; credentials are resolved from the managed Credential)Create a Gateway and manage policies (Policy engine / Policy bundle / Snapshot)
After the policy engine is created, validate the policy bundle via dry-run, then publish it and attach it to the Gateway via updateGatewayPolicy; sealSnapshot seals the current configuration (asynchronously). Changes to Target / Consumer / Tool allowlist / Policy only take effect for public traffic on Snapshots that are published and consistent.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
await aiGateway.listPolicyEngines(); // Policy engine list
await aiGateway.updateGatewayPolicy({ gatewayId: '<gatewayId>' }); // Mount strategy (fields see SDK type definitions)
await aiGateway.sealSnapshot({ gatewayId: '<gatewayId>' }); // Archived as a Snapshot (asynchronous)
await aiGateway.getSnapshotStatus({ gatewayId: '<gatewayId>' }); // Poll archive results
// Policy engine create/detail: createPolicyEngine / getPolicyEngine
// Policy bundles: createRevision / publishPolicyBundle / listPolicyBundles / getPolicyBundle; validate with dryRunPolicy before rollout (evaluates only, does not execute)// Web SDK has the same AiGateway control plane
await aiGateway.sealSnapshot({ gatewayId: '<gatewayId>' });
await aiGateway.getSnapshotStatus({ gatewayId: '<gatewayId>' });aiGateway.listPolicyEngines(); // Policy engine list
aiGateway.updateGatewayPolicy("<gatewayId>"); // Mount strategy
aiGateway.sealSnapshot("<gatewayId>"); // Archived as a Snapshot (asynchronous)
aiGateway.getSnapshotStatus("<gatewayId>"); // Poll archive results
// Policy engine create/detail: createPolicyEngine / getPolicyEngine
// Policy bundles: createRevision / publishPolicyBundle / listPolicyBundles / getPolicyBundle; validate with dryRunPolicy before rollout (evaluates only, does not execute)# server-python@release-2.6.0 has no AiGateway service; the control plane is currently accessed via REST (see below) or the nodejs SDK// server-go@release-2.6.0 has no AiGateway service; the control plane is currently called via REST (see below) or the Node.js SDKcurl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/gateways/<GATEWAY_ID>/snapshots" \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY"
# Asynchronous archival: poll GET .../gateways/<GATEWAY_ID>/snapshots/status until completion
# Policy plane (body fields per the API reference / SDK type definitions): PUT .../gateways/<GATEWAY_ID>/policy;
# GET|POST /ai-gateway/policy-engines; POST .../policy-engines/{policyEngineId}/bundles (publish);
# POST .../policy-engines/{policyEngineId}/dry-run (dry run before going live)Onboard Targets (Resources / Revisions / Mounting)
Targets are onboarded via the unified resource endpoint (kind as a path parameter) and brought online through the revision state machine. Health checks, model discovery, and full replacement of mounts are resource-specific operations.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
// Unified resource surface: listResources / createResource / getResource / replaceResource / deleteResource
// kind is the resource category path parameter; the request body is a generalized schema, subject to SDK type definitions
// Revisions: createRevision / getRevision / transitionRevision (resource go-live uses the revision state machine)
// Target operations: checkTargetHealth / discoverTargetModels; mounting: listTargetMounts / replaceTargetMounts// Web SDK has the same AiGateway control plane: resource / revision / mount method names are the same as the Node tab// Java side does not expose a generalized schema request body; methods only take locator parameters (kind is the resource-category path parameter):
aiGateway.createResource("targets"); // Create resource
aiGateway.createRevision("targets", "<id>"); // Create a revision
aiGateway.transitionRevision("targets", "<id>", "<revisionId>", "publish"); // Revision state machine goes live
aiGateway.checkTargetHealth("<id>"); // Health probe
aiGateway.replaceTargetMounts("<id>"); // Mount performs full replacement# server-python@release-2.6.0 has no AiGateway service// server-go@release-2.6.0 has no AiGateway service# Resource creation (kind=targets / model-deployments) and full replacement: POST|PUT /ai-gateway/resources/{kind}[/{id}]
# (request body is a generalized schema; see API reference / SDK type definitions for fields)
# Go live via the revision state machine: POST .../resources/{kind}/{id}/revisions → POST .../revisions/{revisionId}/{action}
curl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/resources/targets/<ID>/revisions/<REVISION_ID>/publish" \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY"
# Health check: POST .../resources/targets/<ID>/health-check; model discovery: POST .../resources/targets/<ID>/models/discover
# Mount full replacement: PUT .../resources/targets/<ID>/mountsModel deployments and standard models
Model deployments allow testing one real inference before going live; the standard model catalog supports manual registration as well as batch import, merge, and update of models discovered from targets.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
// Model deployment: testModelDeployment (live deployment inference) / getModelDeploymentMetrics (deployment metrics)
// Standard models: createStandardModel / importStandardModels / listStandardModels / updateStandardModel// Web SDK has the same AiGateway control plane: model deployment and standard model method names are the same as the Node tabaiGateway.testModelDeployment("<id>"); // Deployment inference as measured
aiGateway.getModelDeploymentMetrics(); // Deployment metrics
// Standard model (status is io.agentbase.enums.Status):
aiGateway.createStandardModel("profile-dashscope", "qwen-plus", null, null, null, null);
aiGateway.importStandardModels("profile-dashscope",
List.of(Map.of("modelId", "qwen-plus"), Map.of("modelId", "qwen-max")), null, null);
aiGateway.listStandardModels("profile-dashscope", null);
aiGateway.updateStandardModel("<standardModelId>", null, null, null);# server-python@release-2.6.0 has no AiGateway service// server-go@release-2.6.0 has no AiGateway service# Test actual model deployment inference (verify the path before going live): POST /ai-gateway/resources/model-deployments/<ID>/test
curl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/resources/model-deployments/<ID>/test" \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY"
# Deployment metrics: GET /ai-gateway/metrics/model-deployments
# Standard model catalog: GET /ai-gateway/standard-models?providerProfileId=<PROVIDER_PROFILE_ID>&status=active
# Batch import discovered models (same-ID entries are merged with the import taking precedence): POST /ai-gateway/standard-models/import
curl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/standard-models/import" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{"providerProfileId":"profile-dashscope","models":[{"modelId":"qwen-plus"},{"modelId":"qwen-max"}]}'
# Manual registration: POST /ai-gateway/standard-models (providerProfileId + modelId required);
# Update: PUT /ai-gateway/standard-models/<STANDARD_MODEL_ID> (displayName / metadata / status)Tool/MCP connections
A Tool connection is the upstream onboarding for Tool/MCP data plane calls: after creation, you can trigger tool discovery, test connectivity, update configuration, and delete it (once deleted, tools can no longer be called via this connection on the gateway side). Connectivity testing is supported starting from 2.8.0: testToolConnection(id, toolId) (toolId optional) (POST /ai-gateway/tool-connections/{id}/test), which returns diagnostic results (target reachability, credential checks, and per-tool status); the Console connection diagnostics panel uses the same source.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
// createToolConnection / discoverToolConnection / updateToolConnection / deleteToolConnection
// 2.8.0 Connectivity test: aiGateway.testToolConnection('<id>'); the optional second parameter toolId tests a single tool only
// payload is defined by the target resource; see the SDK type definitions for fields// Web SDK has the same AiGateway control plane: tool connection method names are the same as the Node tab// Java side does not expose payload: create / discover take no arguments, update / delete only take id
aiGateway.createToolConnection();
aiGateway.discoverToolConnection();
aiGateway.testToolConnection("<id>"); // 2.8.0: Connectivity testing and diagnostics
aiGateway.updateToolConnection("<id>");
aiGateway.deleteToolConnection("<id>");# server-python@release-2.6.0 has no AiGateway service// server-go@release-2.6.0 has no AiGateway service# Create / discover available tools: POST /ai-gateway/tool-connections, POST .../tool-connections/discover
# (payload is defined by the target resource; see SDK type definitions for fields)
curl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/tool-connections/discover" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{ ... 发现语境,字段见 SDK 类型定义 ... }'
# Update / delete: PUT|DELETE .../tool-connections/<ID> (delete returns 204)
# Connectivity test (2.8.0): POST .../tool-connections/<ID>/test (optional body {"toolId":"<TOOL_ID>"} tests a single tool only)Issuing and enabling/disabling Consumer credentials (abk_ Keys)
createKey issues abk_ credentials; use gatewayIds to scope which gateways can be called, and allowedModels / allowedTools / allowedAgents to scope models, tools, and Agent Targets; once disabled, they immediately fail gateway validation.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
import { AiGateway } from 'agentbase-node-sdk';
const aiGateway = new AiGateway(client); // Control plane: aiGateway.read / aiGateway.write
await aiGateway.listKeys(); // abk_ metadata view
await aiGateway.createKey({
name: 'agent-consumer',
gatewayIds: ['<gatewayId>'] // For the allowlist and principal fields, see the SDK type definitions
});
// Enable/disable: aiGateway.updateKeyStatus(...) (after disabling, an abk_ key immediately fails gateway validation)import { AiGateway } from 'agentbase-web-sdk'; // Web SDK's AiGateway service surface matches the Server's (full control plane)
const aiGateway = new AiGateway(client);
await aiGateway.listKeys();
await aiGateway.createKey({ name: 'agent-consumer', gatewayIds: ['<gatewayId>'] });import io.agentbase.services.AiGateway;
AiGateway aiGateway = new AiGateway(client); // Java Server SDK has the AiGateway service
aiGateway.listKeys();
aiGateway.createKey("agent-consumer", List.of("<gatewayId>"),
null, null, null, null, null, null); // region / allowedModels / allowedTools / allowedAgents / defaultLogicalModel / principal# server-python@release-2.6.0 has no AiGateway service; abk_ issuance/enable-disable is currently done via REST (see below) or the nodejs SDK// server-go@release-2.6.0 has no AiGateway service;
// abk_ Key issuance/enabling and disabling is currently done via REST (see below) or the Node.js SDKcurl -sS -X POST "$APPBASE_ENDPOINT/ai-gateway/keys" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" \
-H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{"name":"agent-consumer","gatewayIds":["<gatewayId>"]}'
# allowlist scoping: allowedModels / allowedTools / allowedAgents; principal carries roles and flat claims (fields see API reference)
# List keys GET /ai-gateway/keys; retrieve plaintext GET .../keys/{keyId} (recoverable semantics); enable/disable PUT .../keys/{keyId}/statusObservability and audit
Call records and the observability overview are queried by aiGateway.read scope, used for troubleshooting and auditing gateway calls.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
await aiGateway.getObservabilityOverview(); // Observability overview
await aiGateway.listInvocations(); // Invocation record list (filter parameters see SDK type definitions)
// Single invocation record: getInvocation// Web SDK has the same AiGateway control plane
await aiGateway.getObservabilityOverview();
await aiGateway.listInvocations();aiGateway.getObservabilityOverview(); // Observability overview
aiGateway.listInvocations(); // Invocation record list (filter parameters see SDK type definitions)
// Single invocation record: getInvocation# server-python@release-2.6.0 has no AiGateway service// server-go@release-2.6.0 has no AiGateway service;
// Observability and audit currently go through REST (see below) or the nodejs SDKcurl -sS "$APPBASE_ENDPOINT/ai-gateway/observability/overview" \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY"
# Invocation records: GET /ai-gateway/invocations; single record: GET /ai-gateway/invocations/<INVOCATION_ID>Example Guide
JavaScript (Node control plane calls + data plane HTTP form): examples/javascript.md
Java:examples/java.md