Docs
Contents

LLM Gateway

Overview and use cases

The LLM Gateway is a project-level LLM proxy (/v1/llm-gateway/*): it manages upstream Providers uniformly (OpenAI-compatible / Anthropic / DashScope), issues Gateway Keys, forwards Chat Completions, Messages, Embeddings, and Rerank requests, and outputs usage statistics. It is suitable for: front ends / server sides calling multiple model providers with the same OpenAI-compatible interface; issuing restricted model Keys to third parties; and tracking project token consumption.

Coexists alongside AI Gateway (the 2.6.0 new unified AI data plane, ai-gateway.md)****: two separate resource sets and two verification chains — keys are not interchangeable even if they look identical; no migration of legacy resources or indiscriminate aliases are provided.

Prerequisites

Usage workflow

  1. Connect Providers: createProvider({ providerId, connectionName, providerType, baseUrl, apiKey, defaultModel, allowedCapabilities: ['chat'|'embedding'|'rerank'|'vlm'], supportedModels, priority, timeoutMs, maxRetries }). supportedModels Use ["*"] to allow any model; otherwise an exact match with the requested model is required. Lower priority takes priority; if the primary Provider fails, failover follows priority order. Only dashscope supports pulling the model list from the cloud (listProviderModels); OpenAI/Anthropic-compatible types require manually filling in supportedModels. After batch updates, call refreshProviderCache() to refresh the routing cache. Starting from 2.9.0: providerType expanded to 24 vendor preset types (OpenAI / Anthropic official and API-Compatible, Volcano Ark (Doubao)volcengine, openai-official / anthropic-official, xAI, Zhipu, Moonshot, DeepSeek, etc., plus self-hosted types like Ollama / vLLM); first use listProviderPresets() (GET /v1/llm-gateway/provider-presets, no auth scope beyond authentication) to get the preset catalog (24 vendors / 313 models, including Base URL, default capabilities, and per-model parameters: context window, max output, Embedding dimensions and input limits, tool calling, multimodal Embedding); after selecting a preset, the Base URL and model list can be brought directly into createProvider; Providers support per-model modelMetadata saving merged with preset values, for use by platform-internal tasks (knowledge base indexing, memory).

  2. Issue a Gateway Key: createKey({ keyId, name, description, allowedModels, monthlyTokenBudget, rateLimit, expiresAt }); rateLimit is requests per minute (null uses the project default); isActive: false disables immediately, deletion revokes.

  3. Execute:

    • createChatCompletion({ model, messages, temperature, maxTokens, stream, tools, toolChoice }) — OpenAI-compatible; messages is passed through as-is (text/multimodal/tool calling all supported), stream: true uses SSE; vision models (VLM) likewise pass image content via chat completions.

    • createMessage(...) — Anthropic Messages form.

    • createEmbeddings({ model, input }) — dimensions must match the vector collection dimension.

    • createRerank({ model, query, documents, topN, returnDocuments })。

    • listModels / getModel query available models.

  4. Usage (Console SDK): getUsage / getUsageModels / getKeyUsage({ startDate, endDate, period: '1h' | '1d' }) — Chat request count, input/output/total tokens, Embedding/Rerank tokens, aggregated by model.

Key parameters and returns

Errors and boundaries

SDK Category Differences

Examples by language

The following provides examples by language, item by item along the usage workflow; each card can switch languages (synced across the page).

Execution: Chat Completions (OpenAI-compatible)

createChatCompletion routes to the active Provider based on model in the request body, messages is passed through as-is; stream: true uses SSE.

import { Client, LlmGateway } from 'agentbase-node-sdk';

const client = new Client()
    .setEndpoint(process.env.APPBASE_ENDPOINT!)   // Must include /v1
    .setProject(process.env.APPBASE_PROJECT_ID!)
    .setKey(process.env.APPBASE_API_KEY!);        // The server stores only the project API key
const llmGateway = new LlmGateway(client);

const chat = await llmGateway.createChatCompletion({
    model: 'qwen-max',                        // Must exactly match the Provider model list / Key allowedModels
    messages: [{ role: 'user', content: '用一句话介绍 TablesDB' }],
    temperature: 0.2,
    maxTokens: 1024,
    stream: false                             // When true, the response is SSE chunks
});

Execution: Embeddings / Rerank / Messages

Embeddings and Rerank use the OpenAI-compatible form; createMessage is the Anthropic Messages form.

await llmGateway.createEmbeddings({
    model: 'text-embedding-v3',
    input: ['文本'],
    dimensions: 1536                  // Must match the vector collection's dimension
});

await llmGateway.createRerank({
    model: '<rerank-model>',
    query: 'q',
    documents: ['d1', 'd2'],
    topN: 3
});
// Anthropic form: llmGateway.createMessage(...); Anthropic Messages fields are passed through as-is
// Available models: await llmGateway.listModels();

Provider management (server / Console only)

Connect upstream Providers (providerType: 'openai' | 'anthropic' | 'dashscope'); route and failover in ascending order of priority; after batch changes, refreshProviderCache() makes routing take effect immediately. The Web SDK has no management plane.

import { ID, ProviderType, LlmGateway } from 'agentbase-node-sdk';

const controlGateway = new LlmGateway(client);   // The management plane must use the project API Key (providers.write)
await controlGateway.createProvider({
    providerId: ID.unique(),
    connectionName: 'reviewed-upstream',
    providerType: ProviderType.Openai,
    baseUrl: '<upstream-base-url>',
    apiKey: process.env.LLM_PROVIDER_API_KEY,     // Stored encrypted; the response only returns the apiKeySet boolean
    defaultModel: 'qwen-max',
    supportedModels: ['qwen-max', 'text-embedding-v3']   // Exact match against the request model; ["*"] allows any
});
await controlGateway.refreshProviderCache();       // Refresh the route cache after batch changes

Gateway Key management

Sub-keys can only call execution plane endpoints and are subject to allowedModels / budget / rate limit constraints; the plaintext appears only once in the creation response; afterwards only keyPrefix is returned. The Web SDK has no management plane, so this card has no Web tab.

const issued = await controlGateway.createKey({
    keyId: ID.unique(),
    name: 'frontend',
    allowedModels: ['qwen-max'],       // Exact match against the request model; a mismatch reports Model not allowed
    monthlyTokenBudget: 1000000,
    rateLimit: 60                      // Requests per minute; null uses the project default
});
// issued.key appears in plaintext only once in this response: write it immediately to a controlled secret store; it must not enter the frontend or logs
await controlGateway.listKeys();       // keyPrefix / usage metadata, plaintext is never returned

Usage statistics (Console only)

Chat request count, input/output/total tokens, Embedding/Rerank tokens, and per-model aggregation — provided only by the Console SDK; the API reference does not offer a project-level usage REST endpoint.

import { Client, LlmGateway } from 'agentbase-console-sdk';
// Initialization of the project resource-plane client (sdk.forProject automatically calls setProject + setMode('admin')): see console-sdk/nodejs.md

const llmGateway = new LlmGateway(client);
await llmGateway.getUsage({ startDate: '<ISO>', endDate: '<ISO>' });   // Requests and total tokens
await llmGateway.getUsageModels();                                     // Aggregate by model
await llmGateway.getKeyUsage({ startDate: '<ISO>', endDate: '<ISO>', period: '1d' });   // period: '1h' | '1d'

Example Guide

2.12.0 — Native rerank instruction

rerank accepts optional instruct for DashScope models that support a native sorting instruction. An empty string omits it. Do not assume all providers support the same feature; model/query/documents and return_documents retain their existing contract.