Docs
Contents

Models and AI Gateway Overview

Agentbase v2.10.0 provides enterprise-grade AI applications with a two-layer model access and governance system: Models (LLM Gateway) and AI Gateway. The two systems coexist within a project with clear boundaries of responsibility and neither takes over the other — Models (LLM Gateway) focuses on project-level model provider connection registration, preset model catalog synchronization, model health probing, and underlying compute supply for the platform's internal AI engines (knowledge base vectorization, reranking, memory extraction, agent workbench, observability evaluation); AI Gateway, on the other hand, provides a unified data plane entry with gateway instance isolation for multi-team, multi-agent production environments, supporting logical model dynamic routing, primary-backup disaster recovery switching, Semantic Cache, security guardrails (Guardrail & OPA policies), multi-protocol proxying (OpenAI / Anthropic / MCP / A2A / HTTP Agent), and fine-grained Token/QPS budget governance.


Feature Overview

When implementing an enterprise-grade AI middle platform, development and architecture teams typically face two very different model management needs:

  1. Project foundation compute connections (carried by "Models / LLM Gateway"):

    • The project's knowledge base (Knowledge Base) requires a stable Embedding model (e.g., text-embedding-v3) for chunk vectorization, a Rerank model (e.g., qwen3.7-text-rerank) for multi-path retrieval reranking, a multimodal vision model (e.g., qwen-vl-max) for scanned documents and architecture diagram interpretation, and a large language model (e.g., qwen-plus) to drive GraphRAG knowledge graph extraction, RAPTOR recursive clustering, Wiki compilation, and synthetic question answering.

    • The Memory center (Memory) requires Chat and Embedding models to perform long-term fact extraction, profile summarization, context dream consolidation (Dreaming), and skill distillation.

    • The Agent Runtime (Agent Workbench) and Observability Evaluations likewise read directly from the Provider and model catalog registered on the project's Models page.

    • Meanwhile, business services can call standard interfaces such as /v1/ai/chat/completions, /v1/ai/embeddings, /v1/ai/rerank, and /v1/ai/messages using project-scoped restricted keys issued via abk_....

  2. Multi-gateway-instance production traffic governance (carried by AI Gateway):

    • When an enterprise exposes LLM and agent capabilities to external business lines, partners, or multi-tenant end users, a logical abstraction layer must be built on top of the physical models (for example, wrapping the primary model qwen-plus and backup model qwen-turbo into a unified logical model lm-enterprise-chat).

    • Unified content safety guardrails (Guardrail Profile, such as PII privacy masking, prompt injection defense, and sensitive-word blocking) and OPA policy pre-checks need to be applied to inbound prompts and outbound generated results.

    • Semantic caching based on vector similarity needs to be enabled so that high-frequency repeated queries hit the cache directly, significantly reducing time-to-first-token (TTFT) and token costs.

    • Gateway instances (e.g., /ai-gateway/g/gw_7194685489feb1e169548619/v1/...) need to centrally govern MCP tool approvals, A2A agent calls, principals & credentials, and multi-cycle hard budgets.


Core Concepts and Architecture

Two-Layer Model Access and Governance Architecture Diagram

The following architecture diagram shows how the Models (LLM Gateway) and the AI Gateway collaborate within the Agentbase platform and across external business call chains:

Models and AI Gateway Overview: workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

Core Concepts and Capability Comparison Table

DimensionModels (LLM Gateway)Unified AI Gateway
Console navigation entryProject left navigation bar ModelsProject left navigation bar AI Gateway
Core positioningProject-level foundation model provider connection pool, model catalog, and internal platform AI compute foundationEnterprise-grade AI traffic gateway with multi-gateway-instance isolation: logical routing, guardrails, caching, and tool governance plane
Default enablement statusEnabled by default after project creation (client access controlled by the AI Gateway service switch in project settings)Gateway instances enabled on demand by default; coexists with LLM Gateway and neither overrides nor takes over the other
Data plane request path/v1/ai/chat/completions, /v1/ai/embeddings, /v1/ai/rerank, /v1/ai/messages, /v1/ai/modelsGateway instance ID mandatory: /ai-gateway/g/{gatewayId}/v1/chat/completions, /responses, /embeddings, /rerank, /messages and /mcp, /a2a
Model addressing methodDirectly uses physical model identifiers (e.g., qwen-plus, text-embedding-v3) or connection ID + model nameSupports Logical Models (e.g., lm-enterprise-chat), routing to multiple physical deployment targets at the underlying layer by weight and priority
Platform-native service dependenciesDirectly depended upon: knowledge base, memory, Agent Runtime, and observability evaluations all read Provider configuration from hereNo coupling: enabling or modifying an AI Gateway instance does not change existing model call behavior of the knowledge base or memory modules
Security and caching policiesBasic gateway key (abk_...) whitelist, monthly token limits, and per-minute request rate limits (RPM)Full Guardrail Profiles, Semantic Cache, OPA policy dry-run, tool approval state machine, and multi-granularity budget controls
Typical example configurationConnection dashscope-prod (integrating qwen-plus, qwen-turbo, text-embedding-v3, qwen3.7-text-rerank, qwen-vl-max), gateway key gw-key-support-botGateway instance prod-enterprise-ai-gateway (gw_7194685489feb1e169548619), logical model lm-enterprise-chat, guardrail gp-enterprise-safety, cache cp-semantic-cache

Roles, Entry Points, and Relationships

Roles and Permission Matrix

Within an organization (e.g., Agentbase 企业智能中台, ID: 69c6d1aa00062e6bb60) and a project (e.g., 企业智能客服与知识中台, ID: enterprise-ai-hub-210), the operation permissions of different roles over the two systems are as follows:

Role NameModels (LLM Gateway) PermissionsAI Gateway Permissions
Owner / Admin (organization owner / administrator)Full permissions: create/edit/delete Providers, run health probes, issue and revoke Gateway Keys, view full token billingFull permissions: create gateway instances, configure logical models and deployment targets, publish safety guardrails and cache policies, approve high-risk MCP tools, and archive runtime snapshots
DeveloperCan view registered Providers and model parameters, run health probes, bind models in the knowledge base/memory/Agent Runtime, and view call statisticsCan view gateway instances, debug logical model routing, run policy Dry-Runs, and view traffic and quality metrics
Gateway key callers (abk_... clients)May only call authorized whitelisted model execution endpoints (Chat / Embedding / Rerank); strictly forbidden from accessing management plane APIsMay only call data plane inference and tool endpoints under their bound gatewayId, subject to real-time principal budget and guardrail policy constraints

Console Entries and Upstream/Downstream Relationships

  1. Console entry path:

    • Models (LLM Gateway): Log in to the Console -> select project 企业智能客服与知识中台 (enterprise-ai-hub-210) -> click Models in the left navigation bar (route: /console/project-default-enterprise-ai-hub-210/llm-gateway).

    • AI Gateway: In the same project's left navigation bar, click AI Gateway (route: /console/project-default-enterprise-ai-hub-210/ai-gateway).

  2. Relationships with upstream and downstream modules:

    • Knowledge Base: When creating knowledge base kb_enterprise_support, you must select an embedding model (e.g., text-embedding-v3, 1024 dimensions) from Providers with the embedding capability enabled on the Models page; when enabling multimodal image understanding, you must select a model with the vlm capability (e.g., qwen-vl-max); when enabling reranking in hybrid retrieval (searchMode=hybrid), you must select a model with the rerank capability (e.g., qwen3.7-text-rerank).

    • Memory: Memory extraction, profile generation, dream consolidation, and experience/skill distillation rely on the default Chat and Embedding models configured on the Models page.

    • Agent Runtime (Agent Workbench): Configure the primary conversation model (e.g., qwen-plus) in the workbench and associate knowledge base kb_enterprise_support and lexicons (lex_enterprise_synonyms, lex_support_stopwords) to achieve out-of-the-box enterprise-grade knowledge Q&A.

    • Limits & Observability: Regardless of whether model calls are made through the LLM Gateway or the AI Gateway, the resulting Prompt Tokens, Completion Tokens, latency, and error rates are automatically aggregated into the project's Observability and Usage Limits dashboards.


Feature Details and Operation Guide

Scenario One: Selection Decision — When to Use Models (LLM Gateway) vs. AI Gateway

In real-world engineering delivery, it is recommended to follow the standard path of foundation first, gateway evolution on demand:

Business Stage and ScenarioRecommended ModuleKey Configuration Actions
Stage 1: Connect an LLM compute foundation for the project (supporting knowledge base ingestion, hybrid retrieval reranking, memory extraction, and Agent Runtime Q&A)Models (LLM Gateway)In Models -> Model Providers, register dashscope-prod via the Alibaba Cloud Bailian (DashScope) preset, import qwen-plus, qwen-turbo, text-embedding-v3, qwen3.7-text-rerank, and qwen-vl-max with one click, and complete the health probe.
Stage 2: Single business application directly calls the unified compatible interface (backend microservices need to call Qwen models in OpenAI SDK standard format)Models (LLM Gateway)In Models -> Gateway Keys, issue gw-key-support-bot (limited to the model whitelist and monthly token budget), and point base_url to https://appbuild.oceanbase.com/v1/ai on the business side.
Stage 3: Enterprise multi-agent / multi-channel production-grade traffic governance (requiring seamless primary-backup failover, sensitive information interception, semantic cache cost reduction, and MCP tool authentication approval)AI GatewayIn AI Gateway -> Gateways, create instance prod-enterprise-ai-gateway, configure logical model lm-enterprise-chat (primary route qwen-plus, backup failover qwen-turbo), and attach guardrail gp-enterprise-safety and semantic cache cp-semantic-cache.

Scenario Two: End-to-End Integration of Model Foundation -> AI Gateway Instance -> Business Calls

Step 1: Complete Provider Registration and Health Probe in Models (LLM Gateway)

  1. Go to the Models page of project enterprise-ai-hub-210, and click Add provider on the Providers tab.

  2. Select the built-in vendor preset Tongyi Qianwen (DashScope), enter dashscope-prod as the connection name; the Base URL is auto-filled as https://dashscope.aliyuncs.com/compatible-mode/v1, and enter <DASHSCOPE_API_KEY>.

  3. Check and enable the four capabilities chat, embedding, rerank, and vlm, then import and retain the following 5 core models:

    • Chat models: qwen-plus (default primary model), qwen-turbo (high-concurrency lightweight model)

    • Embedding models: text-embedding-v3 (default dimension 1024, maximum input tokens 8192)

    • Rerank models: qwen3.7-text-rerank (maximum concurrency and document chunk reranking)

    • Vision models (VLM): qwen-vl-max (used for multimodal chart and scanned document understanding in the knowledge base)

  4. After saving, click Health Check on the connection row and confirm the status badge updates to green HEALTHY.

Step 2: Build Highly Available Logical Models and Safety Policies in AI Gateway

  1. Switch to the AI Gateway page in the left navigation bar and create a gateway instance on the Gateways tab:

    • Gateway name: prod-enterprise-ai-gateway

    • Gateway instance ID: gw_7194685489feb1e169548619

  2. Go to the Model routing tab and create logical model lm-enterprise-chat:

    • Point the Primary Target to qwen-plus under dashscope-prod (weight 100, priority 1).

    • Point the Fallback Target to qwen-turbo under dashscope-prod (priority 2; it takes over automatically when the primary model times out or returns 5xx/429).

  3. Go to the Policies & Credentials tab:

    • Create and bind the guardrail configuration gp-enterprise-safety (enabling inbound Prompt injection detection and outbound masking of phone numbers, ID card numbers, and secrets).

    • Create and bind the semantic cache policy cp-semantic-cache (cosine similarity threshold 0.95, TTL 3600 seconds).

    • Issue a dedicated key bound to this gateway instance for the customer service bot principal, and set hard quotas (e.g., QPS cap 20, monthly budget 50,000,000 Tokens).

Step 3: Verify independent invocation of the two data-plane endpoints

Use curl to verify the inference endpoints of the project-level LLM Gateway and the instance-level AI Gateway separately, confirming that both coexist and each applies its own policies:

# 1. Call the project-level LLM Gateway standard endpoint (directly specifying the physical model qwen-plus)
curl -X POST "https://appbuild.oceanbase.com/v1/ai/chat/completions" \
  -H "Authorization: Bearer <APPBASE_GATEWAY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-plus",
    "messages": [
      {"role": "system", "content": "你是企业智能客服助手。"},
      {"role": "user", "content": "请简述 OceanBase 分布式事务的两阶段提交机制。"}
    ],
    "temperature": 0.2
  }'

# 2. Call the unified AI Gateway instance endpoint (specify the logical model lm-enterprise-chat; automatically passes through guardrails and the semantic cache)
curl -X POST "https://appbuild.oceanbase.com/ai-gateway/g/gw_7194685489feb1e169548619/v1/chat/completions" \
  -H "Authorization: Bearer <APPBASE_AI_GATEWAY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lm-enterprise-chat",
    "messages": [
      {"role": "system", "content": "你是企业智能客服助手。"},
      {"role": "user", "content": "请简述 OceanBase 分布式事务的两阶段提交机制。"}
    ],
    "temperature": 0.2
  }'

Verification result:


FAQ and Troubleshooting

Q1: After enabling the "AI Gateway (AI Gateway)" and configuring logical models, will it affect model calls of the knowledge base or memory modules?

No. Agentbase's "Models (LLM Gateway)" and "AI Gateway (AI Gateway)" adopt a fully decoupled, coexisting design. The knowledge base (vectorization, reranking, knowledge graph extraction, Wiki compilation), the memory hub, and the agent workspace always read registered Provider and physical model parameters through the project-level "Models (LLM Gateway)"; the AI Gateway only processes data-plane traffic explicitly sent to /ai-gateway/g/{gatewayId}/... and never hijacks or alters the LLM Gateway's existing routing.

Q2: What should I do if a gateway key starting with abk_ returns 401 Unauthorized or 403 Forbidden when calling /ai-gateway/g/{gatewayId}/v1/chat/completions?

Troubleshoot in the following order:

  1. Check the key's scope: an ordinary project-level key created in the "Models (LLM Gateway)" tab can only access the /v1/ai/* endpoint; accessing /ai-gateway/g/{gatewayId}/v1/* requires gateway credentials issued and bound under that gatewayId instance.

  2. Check the model whitelist and logical model mapping: confirm that the model field in the request body is filled with a logical model name already published under that gateway instance (e.g., lm-enterprise-chat) or an imported standard model, and that the current credential's role policy allows access to that model.

  3. Check the budget circuit-breaker status: go to AI Gateway -> Policies & Credentials and check whether the Token or QPS hard quota (Budget) associated with the credential has reached its cycle limit.

Q3: Why can't I see the models I just filled in for the DashScope Provider in the dropdown when creating a knowledge base or enabling hybrid retrieval Rerank?

This is because when adding or editing a Provider in "Models (LLM Gateway)", you must not only fill in the model names in the model list (e.g., text-embedding-v3 or qwen3.7-text-rerank), but alsocheck the enable switches for the corresponding capabilities (embedding / rerank / vlm), and accurately fill the models into the dedicated model lists for those capabilities (e.g., put text-embedding-v3 into the Embedding model list and qwen3.7-text-rerank into the Rerank model list). It is recommended to import models preferentially via Vendor Presets (Preset Catalog); the system will automatically complete capability classification and parameter binding such as context window and vector dimensions.

Q4: What is the impact of turning off the AI Gateway switch in project settings (Settings -> Services)?

When the service switch in project settings is turned off, it only blocks model inference requests initiated by external clients via the Client API / gateway keys; internal platform system tasks (such as queued knowledge base ingestion, background memory consolidation, and server-side Server API Key management calls) still run securely with their respective permissions, and no saved Provider or gateway policy configuration is lost.