Docs
Contents

Models (LLM Gateway)

Models (LLM Gateway) is the unified foundational entry point through which Agentbase v2.10.0 projects access large language models, embedding models, reranking models, and multimodal vision models. Via the Models page in the left navigation bar of the Console, development and operations teams can centrally manage model provider connection credentials, import model parameters with one click from the built-in vendor preset catalog, run per-model health checks, issue restricted gateway keys (Gateway Key), and provide out-of-the-box model compute support for the project's knowledge base, memory center, agent workbench, and observability evaluation.


Feature Overview

In enterprise AI application development, distributing raw API keys from upstream LLM vendors (such as Alibaba Cloud DashScope, OpenAI, Anthropic, or self-hosted vLLM/Ollama) directly to individual microservices or frontend clients brings serious pain points such as key leakage risk, inconsistent timeout retries, and the inability to aggregate usage billing by project.

The Agentbase Models (LLM Gateway) solves these problems through three core function tabs:

  1. Model Providers:

    • Built-in 24 mainstream vendor presets and 313 preloaded model catalog entries, automatically carrying each model's context window, maximum output Tokens, Embedding default dimensions and input limits, Tool Call support flags, and multimodal capabilities.

    • Supports registering multiple provider connections per project (such as the production primary connection dashscope-prod), with configurable priority failover order, request timeout, and automatic retry count.

    • Provides real-time Model Discovery and per-model Health Probe to ensure the underlying compute remains in the HEALTHY state.

  2. Gateway Keys:

    • Issues restricted execution keys prefixed with abk_ (such as gw-key-support-bot); the plaintext is shown only once at creation, and the server stores only a strong hash and the prefix.

    • Supports configuring per-key allowed model whitelist, requests-per-minute rate limit (RPM), and monthly Token budget cap, and only inference execution endpoints may be called, naturally isolating management plane permissions.

  3. Usage statistics (Usage):

    • Real-time statistics of request counts, input/output Token consumption, and error distribution generated within the project via gateway keys and internal platform services (knowledge base, memory, agent workbench, evaluation).

LLM Gateway page (reusing the registered screenshot)

Interface reference: reuses previously registered historical screenshots; demo data may differ from the current project; no new screenshots were taken this time.


Core Concepts and Architecture

LLM Gateway routing and compute supply architecture

Models (LLM Gateway): workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

Core concepts table

Core conceptEnglish identifierDetailed description
Model provider connectionProviderAn encrypted connection instance between the project and upstream LLM inference services (such as dashscope-prod). Contains protocol type, Base URL, encrypted API Key storage, enabled capability set, and model parameter table.
Vendor preset catalogProvider Preset CatalogPlatform built-in presets covering 24 mainstream vendors (including Qwen DashScope, Volcano Ark, DeepSeek, OpenAI, Anthropic, vLLM, etc.) and 313 standard model templates. After selecting a preset, you can import the model's context window, maximum output, and embedding dimensions with one click.
Model capability classificationCapabilitiesDivided into four standard capabilities: chat (dialogue and text generation), embedding (text and multimodal embedding), rerank (semantic reranking of retrieval results), vlm (visual image understanding). Downstream modules filter selectable models by capability tag.
By model parametersModel ParametersFine-grained specification parameters bound to specific model entries, including contextWindow (context window), maxOutputTokens (maximum output Tokens), embeddingDimension (embedding dimensions, such as 1024), maxInputTokens (maximum embedding input Tokens), etc.
Gateway keyGateway KeyA project-level inference-dedicated key in the format abk_... (such as gw-key-support-bot). It carries project routing context built in, so no additional X-Appwrite-Project header is required when calling.
Platform trial modelsTrial Model BindingManaged models configured globally by platform administrators and authorized for project trial use. Project members can directly use trial models for prototype validation without ever touching the underlying vendor keys.

Roles, Entry Points, and Relationships

Role permission requirements

OperationOwner / AdminDeveloperRead-only member / Guest
View registered provider list, model catalog, and health statusSupportedSupportedSupported
Add, edit, or delete provider connections or update upstream API KeysSupportedAuthorized developers onlyNot supported
Trigger model discovery and health checksSupportedSupportedNot supported
Create, edit limits for, or revoke gateway keys (Gateway Key)SupportedAuthorized developers onlyNot supported
View project model call volume and Token statistics reportsSupportedSupportedSupported

Console entry and downstream module mapping


Feature Details and Operation Guide

Part One: Register a Production-Grade Provider via a Vendor Preset (dashscope-prod)

LLM Gateway Provider preset dialog (reusing the registered screenshot)

Interface reference: reuses previously registered historical screenshots; demo data may differ from the current project; no new screenshots were taken this time.

Procedure

  1. On the Models page, switch to the Providers tab and click the Add provider button in the upper-right corner.

  2. In the creation method section at the top of the dialog, select Vendor preset (recommended), then choose Tongyi Qianwen (DashScope) from the vendor dropdown list.

  3. Fill in the core configuration parameters of dashscope-prod according to the table below:

Configuration fieldParameter nameExample configuration valueField description and validation rules
Connection namenamedashscope-prodUnique identifier within the project; the “vendor-environment” naming convention is recommended
Protocol typeproviderTypeopenai / dashscopeThe preset automatically matches the OpenAI-compatible protocol and Bailian native extension interfaces
Service address (Base URL)baseUrlhttps://dashscope.aliyuncs.com/compatible-mode/v1Auto-filled by the preset; the server enforces URL/DNS security policy validation and forbids pointing to intranet loopback or metadata-sensitive addresses
API KeyapiKey<DASHSCOPE_API_KEY>Alibaba Cloud Bailian API key; stored encrypted with AES-GCM at write time, and no read interface returns the plaintext
Allowed capabilitiesenabledCapabilities["chat", "embedding", "rerank", "vlm"]Check the categories of model capabilities exposed by this connection
Default primary modeldefaultModelqwen-plusThe model used when the caller does not explicitly specify a model, or when an internal platform task requests the default Chat model
Chat model listchatModelsqwen-plus, qwen-turboUsed for agent conversations, knowledge base answer synthesis, knowledge graph construction, and memory extraction
Embedding model listembeddingModelstext-embedding-v3Used for embedding of chunked knowledge base documents and memory vectorization
Rerank model listrerankModelsqwen3.7-text-rerankUsed for reranking of multi-path knowledge base retrieval results (Rerank)
Vision model listvlmModelsqwen-vl-maxUsed for knowledge base image understanding and multimodal parsing
Timeout (seconds)timeoutSeconds120Timeout threshold for a single upstream HTTP inference request, default 120 seconds
Maximum retriesmaxRetries2Number of automatic retries on transient network jitter or 502/503/504
Routing prioritypriority10Lower values indicate higher priority; used for failover ordering of identically named models across Providers
Set as default connectionisDefaulttrue (enabled)Marks this as the default model provider connection for the current project
  1. In the built-in model import and per-model parameters area, check the model entries to import, then review and confirm each model's detailed parameters:

Model identifierCapability typeContext window (contextWindow)Maximum output (maxOutputTokens)Vector dimension (embeddingDimension)Maximum embedding input (maxInputTokens)Tool calling (supportsTools)
qwen-pluschat131072 (128K)8192--true
qwen-turbochat131072 (128K)8192--true
text-embedding-v3embedding--10248192-
qwen3.7-text-rerankrerank32768--4000-
qwen-vl-maxvlm / chat327684096--true
  1. Click Create to complete the registration.


Part Two: Perform Model Discovery and Health Probing

After the provider is created, it is recommended to immediately verify upstream connectivity and the availability of each capability endpoint:

  1. In the model provider list, find the dashscope-prod connection row.

  2. Click the Health Check button in the actions column (or invoke the probe via the management API).

  3. The system will automatically send lightweight probe requests to enabled models such as qwen-plus, text-embedding-v3, and qwen3.7-text-rerank:

    • If upstream authentication passes and the response is normal, the connection status updates to green HEALTHY, and the probe latency is recorded.

    • If the upstream returns 401 InvalidApiKey or 404 ModelNotFound, the status is marked red UNHEALTHY and a specific error summary is displayed.

  4. When the upstream vendor adds new models or adjusts the model list, you can click Discover Models and Refresh Cache to sync the latest available models to the project's model catalog.


Part Three: Issue and Manage Gateway Keys (gw-key-support-bot)

To allow external business services to securely call the model capabilities within the project, issue restricted keys on the Gateway Keys tab:

  1. Switch to the Gateway Keys tab and click Create key.

  2. Fill in the key configuration items:

    • Key Name (Name): gw-key-support-bot (used for the enterprise intelligent customer service bot backend)

    • Allowed Models whitelist (Allowed Models): Check the specific models allowed for access (e.g., qwen-plus, qwen-turbo, text-embedding-v3, qwen3.7-text-rerank); if left empty, all models in the project catalog are allowed by default.

    • Per-minute rate limit (Rate Limit RPM): For example, enter 120 (the default endpoint rate limit is 60 requests/minute).

    • Monthly Token Budget: for example, enter 10000000 (1,000 ten-thousand Tokens); once the budget is exceeded, requests will return 429 Too Many Requests.

    • Expiration time (Expires At): Optionally set the timestamp when the key automatically becomes invalid.

  3. After clicking Create, the dialog will display the plaintext key only once (in a format like abk_live_...). Immediately copy and store it securely in the business system's Key Management Service (KMS) or environment variables.


Part Four: Verify Model Calls via OpenAI / Anthropic Compatible Endpoints

After the abk_ gateway key is issued, business applications can directly use the standard OpenAI SDK, Anthropic SDK, or native HTTP requests to access the following four types of execution endpoints:

1. Chat Completions (POST /v1/ai/chat/completions)

curl -X POST "https://appbuild.oceanbase.com/v1/ai/chat/completions" \
  -H "Authorization: Bearer <APPBASE_GATEWAY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-plus",
    "messages": [
      {"role": "system", "content": "你是企业智能客服与知识中台助手。"},
      {"role": "user", "content": "如何排查 OceanBase 向量索引构建超时?"}
    ],
    "stream": false,
    "temperature": 0.3
  }'

2. Text Embeddings (POST /v1/ai/embeddings)

curl -X POST "https://appbuild.oceanbase.com/v1/ai/embeddings" \
  -H "Authorization: Bearer <APPBASE_GATEWAY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-v3",
    "input": [
      "OceanBase 分布式事务与向量索引排障指南",
      "AI 网关模型路由与限流计费规范"
    ],
    "dimensions": 1024
  }'

3. Semantic Rerank (POST /v1/ai/rerank)

curl -X POST "https://appbuild.oceanbase.com/v1/ai/rerank" \
  -H "Authorization: Bearer <APPBASE_GATEWAY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.7-text-rerank",
    "query": "向量索引内存不足报错 4013 怎么处理?",
    "documents": [
      "当出现 OB_ALLOCATE_MEMORY_FAILED (-4013) 时,需检查 ob_vector_memory_limit_percentage 参数配置。",
      "智能客服工单流转遵循一线坐席初审、二线专家复核的标准作业程序(SOP)。"
    ],
    "top_n": 2
  }'

4. Query the project's available model catalog (Models:GET /v1/ai/models)

curl -X GET "https://appbuild.oceanbase.com/v1/ai/models" \
  -H "Authorization: Bearer <APPBASE_GATEWAY_KEY>"

Verification result:


FAQ and Troubleshooting

Q1: When adding an Alibaba Cloud Bailian (DashScope) connection, which Base URL should be used? Why do Rerank models also work correctly?

When using the built-in Tongyi Qianwen (DashScope) vendor preset, it is recommended to keep the Base URL at the default https://dashscope.aliyuncs.com/compatible-mode/v1. Agentbase's DashScope adapter handles protocol routing automatically: qwen-plus, qwen-turbo, text-embedding-v3, and qwen-vl-max go through the OpenAI-compatible path, while for rerank models such as qwen3.7-text-rerank, the gateway under the hood automatically aligns with the Bailian native Rerank protocol endpoint and returns a standardized ranking structure, so callers need not worry about underlying differences.

Q2: When ingesting long documents into the knowledge base, an error says the Embedding input length exceeds the maximum allowed tokens — how do I adjust this?

In Models -> Model Providers, edit dashscope-prod, expand Model Parameters to find text-embedding-v3, and check its Max Input Tokens configuration (text-embedding-v3 supports up to 8192 Tokens). Meanwhile, in the knowledge base's chunking parser configuration, it is recommended to set chunkTokenNum between 512 and 1024, ensuring that a single chunk plus its heading prefix stays well below the embedding model's input limit.

Q3: Can a gateway key (abk_...) be used to call administrative interfaces such as creating knowledge bases or modifying Providers?

No. Gateway keys follow a strict least-privilege principle and are only authorized to access /v1/ai/* model execution endpoints (Chat Completions, Embeddings, Rerank, Anthropic Messages, and model list queries). Any create, delete, or modify administrative operation against Providers, Gateway Keys, knowledge bases, or the database must use the Server API Key issued under Overview -> API Keys for the project, or a Console administrator session.

Q4: When the same model name exists in multiple Providers, how does the LLM Gateway decide which connection to route to?

If no connection identifier is explicitly specified in the request, the LLM Gateway follows this automatic routing order:

  1. If the project is bound to a matching platform Trial Model and no project connection is explicitly specified, trial model routing is attempted first;

  2. Among the project's self-built Providers, the provider marked as the default connection (isDefault = true) is selected first;

  3. If multiple connections are available, routing proceeds from highest to lowest according to the connection's configured priority (priority), and upon a timeout or 5xx failure on the primary connection, failover to backup connections occurs automatically per the retry policy.