LLM Gateway
Overview and use cases
The LLM Gateway is a project-level LLM proxy (/v1/llm-gateway/*): it manages upstream Providers uniformly (OpenAI-compatible / Anthropic / DashScope), issues Gateway Keys, forwards Chat Completions, Messages, Embeddings, and Rerank requests, and outputs usage statistics. It is suitable for: front ends / server sides calling multiple model providers with the same OpenAI-compatible interface; issuing restricted model Keys to third parties; and tracking project token consumption.
Coexists alongside AI Gateway (the 2.6.0 new unified AI data plane, ai-gateway.md)****: two separate resource sets and two verification chains — keys are not interchangeable even if they look identical; no migration of legacy resources or indiscriminate aliases are provided.
Prerequisites
Scopes: Provider management
providers.read/providers.write; Key managementkeys.read/keys.write; executionexecute; usage queries require the corresponding read scope.Four authentication methods for execution (choose any): ① Gateway Key (
Authorization: Bearer abk_...orx-api-key, project ID embedded in the Key, nox-appwrite-projectrequired); ② Project API Key (x-appwrite-key+x-appwrite-project, must include execute); ③ User Session (cookie); ④ JWT (x-appwrite-jwt).Gateway Keys can only call execution APIs; they cannot manage resources or read usage. The plaintext appears only once in the creation response; afterwards only the keyPrefix is returned.
The project-level
llmGatewayservice switch only blocks Client API identity access to the entire namespace; it does not delete Providers/Keys. Enabled by default for new projects.
Usage workflow
Connect Providers:
createProvider({ providerId, connectionName, providerType, baseUrl, apiKey, defaultModel, allowedCapabilities: ['chat'|'embedding'|'rerank'|'vlm'], supportedModels, priority, timeoutMs, maxRetries }).supportedModelsUse["*"]to allow any model; otherwise an exact match with the requested model is required. Lowerprioritytakes priority; if the primary Provider fails, failover follows priority order. Onlydashscopesupports pulling the model list from the cloud (listProviderModels); OpenAI/Anthropic-compatible types require manually filling insupportedModels. After batch updates, callrefreshProviderCache()to refresh the routing cache. Starting from 2.9.0:providerTypeexpanded to 24 vendor preset types (OpenAI / Anthropic official and API-Compatible, Volcano Ark (Doubao)volcengine,openai-official/anthropic-official, xAI, Zhipu, Moonshot, DeepSeek, etc., plus self-hosted types like Ollama / vLLM); first uselistProviderPresets()(GET /v1/llm-gateway/provider-presets, no auth scope beyond authentication) to get the preset catalog (24 vendors / 313 models, including Base URL, default capabilities, and per-model parameters: context window, max output, Embedding dimensions and input limits, tool calling, multimodal Embedding); after selecting a preset, the Base URL and model list can be brought directly intocreateProvider; Providers support per-modelmodelMetadatasaving merged with preset values, for use by platform-internal tasks (knowledge base indexing, memory).Issue a Gateway Key:
createKey({ keyId, name, description, allowedModels, monthlyTokenBudget, rateLimit, expiresAt });rateLimitis requests per minute (null uses the project default);isActive: falsedisables immediately, deletion revokes.Execute:
createChatCompletion({ model, messages, temperature, maxTokens, stream, tools, toolChoice })— OpenAI-compatible;messagesis passed through as-is (text/multimodal/tool calling all supported),stream: trueuses SSE; vision models (VLM) likewise pass image content via chat completions.createMessage(...)— Anthropic Messages form.createEmbeddings({ model, input })—dimensionsmust match the vector collection dimension.createRerank({ model, query, documents, topN, returnDocuments })。listModels/getModelquery available models.
Usage (Console SDK):
getUsage/getUsageModels/getKeyUsage({ startDate, endDate, period: '1h' | '1d' })— Chat request count, input/output/total tokens, Embedding/Rerank tokens, aggregated by model.
Key parameters and returns
Provider
apiKeyis stored encrypted; responses only return theapiKeySetboolean. Provider path defaults tochatCompletionsPath=/v1/chat/completions;embeddingsPath=/v1/embeddingsandrerankPathbeing empty means not enabled.Routing rules: explicit matches sorted ascending by
priority→["*"]Provider → fallback to available Providers; OpenRouter-styleprovider/modelprefix notation is forbidden.The Key's
allowedModelsmust exactly match the request'smodel; a mismatch returnsModel not allowed.Calling within Functions: use the platform-injected dynamic Key (request header
x-appwrite-key) +APPWRITE_FUNCTION_PROJECT_ID+APPWRITE_FUNCTION_API_ENDPOINT; the Function needs thellm-gateway.executescope configured; when connecting directly with the OpenAI SDK, adddefaultHeaders.
Errors and boundaries
401/403: wrong authentication method or scope (check the four authentication forms and Key permissions).
Troubleshooting order for routing failures/upstream errors: Provider is activated →
allowedCapabilitiesincludes the capability →supportedModelsexact match → whetherrefreshProviderCache()was called after the update.If a Gateway Key leaks, revoke and recreate it (
updateKeyStatus/ delete); the plaintext cannot be retrieved a second time.Streaming responses are parsed event by event via SSE; clients must handle interruptions and partial content.
SDK Category Differences
Server SDK (server-sdk/nodejs.md): full Provider/Key management + all four execution types; the execution plane can also connect directly using a "standalone Client +
Authorization: Bearer <GatewayKey>".Client SDK (client-sdk/web.md): execution APIs (chat / messages / embeddings / rerank / model queries) and trial provider binding; no Provider/Key management.
Console SDK (console-sdk/nodejs.md): full management plane + usage views (
getUsage/getUsageModels/getKeyUsage).Both the Java Server / Console SDK have
LlmGatewayservices.
Examples by language
The following provides examples by language, item by item along the usage workflow; each card can switch languages (synced across the page).
Execution: Chat Completions (OpenAI-compatible)
createChatCompletion routes to the active Provider based on model in the request body, messages is passed through as-is; stream: true uses SSE.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
import { Client, LlmGateway } from 'agentbase-node-sdk';
const client = new Client()
.setEndpoint(process.env.APPBASE_ENDPOINT!) // Must include /v1
.setProject(process.env.APPBASE_PROJECT_ID!)
.setKey(process.env.APPBASE_API_KEY!); // The server stores only the project API key
const llmGateway = new LlmGateway(client);
const chat = await llmGateway.createChatCompletion({
model: 'qwen-max', // Must exactly match the Provider model list / Key allowedModels
messages: [{ role: 'user', content: '用一句话介绍 TablesDB' }],
temperature: 0.2,
maxTokens: 1024,
stream: false // When true, the response is SSE chunks
});import { Client, LlmGateway } from 'agentbase-web-sdk'; // Prerequisite: the user is logged in (Session auth; must not carry a Key)
const llmGateway = new LlmGateway(client);
const chat = await llmGateway.createChatCompletion({
model: 'qwen-max',
messages: [{ role: 'user', content: '用一句话介绍 TablesDB' }]
});import io.agentbase.services.LlmGateway;
LlmGateway llmGateway = new LlmGateway(client); // For client initialization, see server-sdk/java.md
Object chat = llmGateway.createChatCompletion(
"qwen-max", // Must exactly match the Provider model list / Key allowedModels
List.of(Map.of("role", "user", "content", "用一句话介绍 TablesDB")),
0.2, 1024L, false, null, null); // temperature / maxTokens / stream / tools / toolChoicefrom agentbase.services.llm_gateway import LlmGateway
llm_gateway = LlmGateway(client) # Synchronous call
chat = llm_gateway.create_chat_completion(
model='qwen-max', # Must exactly match the Provider model list / Key allowedModels
messages=[{'role': 'user', 'content': '用一句话介绍 TablesDB'}],
temperature=0.2,
max_tokens=1024,
stream=False) # When True the response is SSE chunksllmGateway := agentbase.NewLlmGateway(cl) // facade constructor; cl is a Client already loaded with a project API Key / Gateway Key
chat, err := llmGateway.CreateChatCompletion("qwen-max", // Must exactly match the Provider model list / Key allowedModels
[]interface{}{map[string]interface{}{"role": "user", "content": "用一句话介绍 TablesDB"}},
llmGateway.WithCreateChatCompletionTemperature(0.2),
llmGateway.WithCreateChatCompletionMaxTokens(1024),
llmGateway.WithCreateChatCompletionStream(false)) // When true, the response is SSE chunks
_ = chatcurl -sS -X POST "$APPBASE_ENDPOINT/llm-gateway/chat/completions" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" \
-H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{"model":"qwen-max","messages":[{"role":"user","content":"用一句话介绍 TablesDB"}],"temperature":0.2}'
# Gateway Key auth: use -H "Authorization: Bearer $LLM_GATEWAY_KEY" instead (abk_ credential, Key embeds the project ID, X-Appwrite-Project may be omitted)
# Streaming: add "stream":true to the body; the response is SSE (data: {"choices":[{"delta":{...}}]} chunks, ending with data: [DONE])Execution: Embeddings / Rerank / Messages
Embeddings and Rerank use the OpenAI-compatible form; createMessage is the Anthropic Messages form.
- Node.js (Server)
- Web (user session)
- Java
- Python
- Go
- REST (curl)
await llmGateway.createEmbeddings({
model: 'text-embedding-v3',
input: ['文本'],
dimensions: 1536 // Must match the vector collection's dimension
});
await llmGateway.createRerank({
model: '<rerank-model>',
query: 'q',
documents: ['d1', 'd2'],
topN: 3
});
// Anthropic form: llmGateway.createMessage(...); Anthropic Messages fields are passed through as-is
// Available models: await llmGateway.listModels();// Web SDK has the same execution-plane subset: embeddings / rerank / messages / model queries
await llmGateway.createEmbeddings({ model: 'text-embedding-v3', input: ['文本'] });
await llmGateway.createRerank({ model: '<rerank-model>', query: 'q', documents: ['d1', 'd2'], topN: 3 });// Java side has no dimensions parameter; createRerank(model, query, documents, topN, returnDocuments)
llmGateway.createEmbeddings("text-embedding-v3", List.of("文本"));
llmGateway.createRerank("<rerank-model>", "q", List.of("d1", "d2"), 3L, null);llm_gateway.create_embeddings(model='text-embedding-v3', input=['文本']) # Python surface has no dimensions parameter
llm_gateway.create_rerank(model='<rerank-model>', query='q', documents=['d1', 'd2'], top_n=3)
# create_message(...) follows the Anthropic Messages shape; list_models() queries available modelsemb, err := llmGateway.CreateEmbeddings("text-embedding-v3", []string{"文本"}) // Go side has no dimensions option
rerank, err := llmGateway.CreateRerank("<rerank-model>", "q", []string{"d1", "d2"},
llmGateway.WithCreateRerankTopN(3))
_ = emb
_ = rerank
// Anthropic form: CreateMessage(model, messages, maxTokens, ...); Anthropic Messages fields are passed through as-is
// Available models: ListModels()curl -sS -X POST "$APPBASE_ENDPOINT/llm-gateway/embeddings" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{"model":"text-embedding-v3","input":["文本"]}'
curl -sS -X POST "$APPBASE_ENDPOINT/llm-gateway/rerank" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{"model":"<rerank-model>","query":"q","documents":["d1","d2"]}'
# Anthropic Messages: POST /llm-gateway/messages; model list: GET /llm-gateway/modelsProvider management (server / Console only)
Connect upstream Providers (providerType: 'openai' | 'anthropic' | 'dashscope'); route and failover in ascending order of priority; after batch changes, refreshProviderCache() makes routing take effect immediately. The Web SDK has no management plane.
- Node.js (Server)
- Java
- Python
- Go
- REST (curl)
import { ID, ProviderType, LlmGateway } from 'agentbase-node-sdk';
const controlGateway = new LlmGateway(client); // The management plane must use the project API Key (providers.write)
await controlGateway.createProvider({
providerId: ID.unique(),
connectionName: 'reviewed-upstream',
providerType: ProviderType.Openai,
baseUrl: '<upstream-base-url>',
apiKey: process.env.LLM_PROVIDER_API_KEY, // Stored encrypted; the response only returns the apiKeySet boolean
defaultModel: 'qwen-max',
supportedModels: ['qwen-max', 'text-embedding-v3'] // Exact match against the request model; ["*"] allows any
});
await controlGateway.refreshProviderCache(); // Refresh the route cache after batch changesimport io.agentbase.ID;
import io.agentbase.enums.ProviderType;
import io.agentbase.services.LlmGateway;
LlmGateway controlGateway = new LlmGateway(client); // The management plane must use the project API Key (providers.write)
controlGateway.createProvider(ID.unique(), "reviewed-upstream",
ProviderType.OPENAI, "<upstream-base-url>",
System.getenv("LLM_PROVIDER_API_KEY"), "qwen-max",
null, List.of("qwen-max", "text-embedding-v3"), // allowedCapabilities, supportedModels (["*"] allows any)
null, null, null, null, // chat/embedding/rerank/vlm model list
null, null, null, // chatCompletionsPath / embeddingsPath / rerankPath
null, null, null, null, null, null, null, null); // description / isDefault / isActive / timeoutMs / maxRetries / priority / maxInputTokens / allowByok
controlGateway.refreshProviderCache(); // Refresh the route cache after batch changesimport os
from agentbase.id import ID
control_gateway = LlmGateway(client) # The management plane must use the project API Key (providers.write)
control_gateway.create_provider(
provider_id=ID.unique(),
connection_name='reviewed-upstream',
provider_type='openai', # 'openai' | 'anthropic' | 'dashscope'
base_url='<upstream-base-url>',
api_key=os.environ['LLM_PROVIDER_API_KEY'], # Stored encrypted; the response only returns api_key_set
default_model='qwen-max',
supported_models=['qwen-max', 'text-embedding-v3']) # Exact match on the request model; ['*'] allows any
control_gateway.refresh_provider_cache() # Refresh the route cache after batch changescontrolGateway := agentbase.NewLlmGateway(cl) // The management plane must use the project API Key (providers.write)
_, err := controlGateway.CreateProvider(id.Unique(), "reviewed-upstream", "openai", // openai | anthropic | dashscope
"<upstream-base-url>",
controlGateway.WithCreateProviderApiKey(os.Getenv("LLM_PROVIDER_API_KEY")), // Stored encrypted; the response only returns apiKeySet
controlGateway.WithCreateProviderDefaultModel("qwen-max"),
controlGateway.WithCreateProviderSupportedModels([]string{"qwen-max", "text-embedding-v3"})) // ["*"] allows any
_, err = controlGateway.RefreshProviderCache() // Refresh the route cache after batch changescurl -sS -X POST "$APPBASE_ENDPOINT/llm-gateway/providers" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{"providerId":"openai-prod","connectionName":"reviewed-upstream","providerType":"openai","baseUrl":"<upstream-base-url>","apiKey":"<upstream-key>","defaultModel":"qwen-max","supportedModels":["qwen-max","text-embedding-v3"]}'
# List: GET /llm-gateway/providers; update: PATCH .../providers/{providerId}; delete: DELETE .../providers/{providerId}
# Refresh routes after changes: POST /llm-gateway/providers/cache/refresh; health: GET .../providers/{providerId}/health; upstream model discovery: GET .../providers/{providerId}/modelsGateway Key management
Sub-keys can only call execution plane endpoints and are subject to allowedModels / budget / rate limit constraints; the plaintext appears only once in the creation response; afterwards only keyPrefix is returned. The Web SDK has no management plane, so this card has no Web tab.
- Node.js (Server)
- Java
- Python
- Go
- REST (curl)
const issued = await controlGateway.createKey({
keyId: ID.unique(),
name: 'frontend',
allowedModels: ['qwen-max'], // Exact match against the request model; a mismatch reports Model not allowed
monthlyTokenBudget: 1000000,
rateLimit: 60 // Requests per minute; null uses the project default
});
// issued.key appears in plaintext only once in this response: write it immediately to a controlled secret store; it must not enter the frontend or logs
await controlGateway.listKeys(); // keyPrefix / usage metadata, plaintext is never returnedimport io.agentbase.ID;
controlGateway.createKey(ID.unique(), "frontend", null,
List.of("qwen-max"), // Exact match on the request model; non-matching returns Model not allowed
1000000L, 60L, null); // monthlyTokenBudget / rateLimit (null uses the project default) / expiresAt
// The plaintext key appears only once, in the creation response: immediately write it to a controlled secret store; never expose it to the frontend or logs
controlGateway.listKeys(); // keyPrefix / usage metadata, plaintext is never returnedcontrol_gateway.create_key(
key_id=ID.unique(),
name='frontend',
allowed_models=['qwen-max'], # Exact match on the request model; non-matching returns Model not allowed
monthly_token_budget=1_000_000,
rate_limit=60) # Requests per minute; None uses the project default
# The plaintext key appears only once, in the creation response: immediately write it to a controlled secret store; never expose it to the frontend or logs
control_gateway.list_keys() # keyPrefix / usage metadata, plaintext is never returned_, err := controlGateway.CreateKey(id.Unique(), "frontend",
controlGateway.WithCreateKeyAllowedModels([]string{"qwen-max"}), // Exact match on the request model; non-matching returns Model not allowed
controlGateway.WithCreateKeyMonthlyTokenBudget(1_000_000),
controlGateway.WithCreateKeyRateLimit(60)) // Requests per minute; a zero value uses the project default
// The plaintext key appears only once, in the creation response: immediately write it to a controlled secret store; never expose it to the frontend or logs
_, err = controlGateway.ListKeys() // keyPrefix / usage metadata, plaintext is never returnedcurl -sS -X POST "$APPBASE_ENDPOINT/llm-gateway/keys" \
-H 'Content-Type: application/json' \
-H "X-Appwrite-Project: $APPBASE_PROJECT_ID" -H "X-Appwrite-Key: $APPBASE_API_KEY" \
-d '{"keyId":"key_frontend","name":"frontend","allowedModels":["qwen-max"],"monthlyTokenBudget":1000000,"rateLimit":60}'
# The response contains a one-time plaintext key; list GET /llm-gateway/keys; enable/disable or reconfigure PATCH .../keys/{keyId}; revoke DELETE .../keys/{keyId}Usage statistics (Console only)
Chat request count, input/output/total tokens, Embedding/Rerank tokens, and per-model aggregation — provided only by the Console SDK; the API reference does not offer a project-level usage REST endpoint.
- Console (Node.js)
import { Client, LlmGateway } from 'agentbase-console-sdk';
// Initialization of the project resource-plane client (sdk.forProject automatically calls setProject + setMode('admin')): see console-sdk/nodejs.md
const llmGateway = new LlmGateway(client);
await llmGateway.getUsage({ startDate: '<ISO>', endDate: '<ISO>' }); // Requests and total tokens
await llmGateway.getUsageModels(); // Aggregate by model
await llmGateway.getKeyUsage({ startDate: '<ISO>', endDate: '<ISO>', period: '1d' }); // period: '1h' | '1d'Example Guide
JavaScript (Node execution + Console management): examples/javascript.md
Java:examples/java.md
2.12.0 — Native rerank instruction
rerank accepts optional instruct for DashScope models that support a native sorting instruction. An empty string omits it. Do not assume all providers support the same feature; model/query/documents and return_documents retain their existing contract.