Docs
Contents

Unified AI Gateway (AI Gateway)

Unified AI Gateway (AI Gateway) is the advanced AI data-plane governance center of Agentbase v2.10.0, built for enterprise multi-agent, multi-model, and multi-tool production environments. Through the AI Gateway page in the left navigation bar of the Console, architects and operations teams can build isolated inference and tool entry points per gateway instance (e.g., prod-enterprise-ai-gateway), and uniformly orchestrate Logical Models, Fallback Deployments, Guardrail Profiles, Semantic Cache Policies, MCP tool approvals and agent proxying, Principals & Budgets, and full-dimension traffic quality observability.


Feature Overview

When enterprise AI applications move from single-point prototypes to production-scale delivery, simple model API forwarding can no longer meet the requirements for high availability, compliance security, and lean cost control. The Unified AI Gateway provides full lifecycle governance capabilities through the following five feature tabs:

  1. Gateways:

    • Create and manage independent gateway instances (e.g., prod-enterprise-ai-gateway, instance ID: gw_7194685489feb1e169548619).

    • View the gateway's dedicated data-plane endpoint (/ai-gateway/g/{gatewayId}/v1/...), mounted logical models, guardrails, semantic cache policies, and the state of the immutable Runtime Snapshot.

  2. Model Routing:

    • Define business-decoupled Logical Models (e.g., lm-enterprise-chat) with multiple physical model deployment targets mounted underneath (Deployments, e.g., primary model qwen-plus and fallback model qwen-turbo), supporting weighted traffic splitting and automatic disaster-recovery failover by priority.

    • Supports one-click discovery and import of a standard model catalog from Providers already registered in the project (e.g., dashscope-prod), and configuration of a model Rate Card for precise invocation cost accounting.

  3. Tools & Agents:

    • Centrally register MCP (Model Context Protocol) servers, OpenAPI tools, inline APIs, Serverless Functions, as well as managed AppBase standard models and AppBase RAG connection templates.

    • Built-in strict tool approval state machine (No approval required / Approve by policy / Always approve), forcibly blocking unreviewed or high-risk tools from going live directly in "No approval required" mode.

    • Supports unified access and protocol proxying for A2A (Agent-to-Agent) and HTTP Agents.

  4. Policies & Credentials:

    • Manage guardrail configurations (such as gp-enterprise-safety) and semantic caching policies (such as cp-semantic-cache).

    • Provides policy version diff comparison, policy decision explanations, and OPA (Open Policy Agent) Dry-Run rehearsals.

    • Manage access principals, issue API Key credentials bound to the current gateway instance, and enforce cross-period hard budget and quota controls (Budgets: QPS / burst concurrency / request count / total Tokens, supporting minute, hour, day, and month periods).

  5. Traffic & Quality:

    • Collects in real time the trace of every inference and tool call passing through the gateway, aggregating and analyzing request volume, time to first token (TTFT), P95/P99 latency, semantic cache hit rate, guardrail interception rate, and Token cost across multiple dimensions such as credential / role / logical model / physical deployment / tool / agent.

AI Gateway page (reusing the previously captured screenshot)

Historical error state example: the image shows 503 / AI Gateway project schema is not available. First check the project gateway configuration and service status; this image is for identifying an abnormal page. No new screenshot was captured this time.


Core Concepts and Architecture

Unified AI Gateway Request Processing and Governance Pipeline

Unified AI Gateway (AI Gateway): workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

Core Concepts and Configuration Entities Table

Entity NameEnglish IdentifierBenchmark ExampleCore Responsibility Description
Gateway instanceGateway Instanceprod-enterprise-ai-gateway (gw_7194685489feb1e169548619)An independent data-plane traffic entry point and policy attachment boundary. Each instance has a unique gatewayId and an immutable runtime configuration snapshot (Snapshot).
Logical modelLogical Modellm-enterprise-chatA virtual model name exposed to the business. Business code consistently requests lm-enterprise-chat without needing to be aware of underlying physical model switches, upgrades, or cross-vendor failover.
Model deployment targetModel Deployment / TargetPrimary: qwen-plus
Secondary: qwen-turbo
The specific Provider model bound beneath a logical model, along with routing weights, priority, concurrency limits, and per-target timeout parameters.
Guardrail configurationGuardrail Profilegp-enterprise-safetyDefines inbound (Input) and outbound (Output) content safety detection rules, covering jailbreak/injection attack blocking, automatic masking or rejection of PII (phone numbers / ID numbers / bank cards / secret keys), and compliance keyword filtering.
Semantic caching policyCache Policycp-semantic-cacheAn intelligent caching mechanism based on Embedding vector similarity. When the semantic vector of a new request reaches the cosine similarity threshold against a historical cache entry (e.g., 0.95), the cached response is returned directly.
Principals and credentialsPrincipal & Credentialprincipal-support-agentBusiness identity principals calling the gateway and their bound abk_ gateway keys, role tags, and the logical model/tool scope they are allowed to access.
Hard budget quotaBudgetbudget-monthly-supportHard limits bound at the gateway, principal, or key dimension, supporting circuit breaking on QPS, burst request count (Burst), total request count, and total Token consumption on minute/hour/day/month periods.

Roles, Entry Points, and Relationships

Prerequisites and Permission Matrix

The Unified AI Gateway is a project-level advanced governance service and is disabled by default. To access the AI Gateway entry in the left navigation bar, two conditions must be met simultaneously:

  1. The project service is enabled: The organization administrator or project Owner has enabled the AI Gateway service in Project Settings (Settings -> Services).

  2. The member has the corresponding RBAC permissions: The currently logged-in member has AI Gateway read or management permissions in the organization/project.

Permission DimensionOwner / AdminDeveloperSecurity/Compliance Auditor
Create/modify gateway instances and seal runtime snapshotsSupportedNot supportedRead-only viewing of snapshot history
Configure logical models (lm-enterprise-chat) and primary/secondary deployment routingSupportedSupportedRead-only
Edit guardrails (gp-enterprise-safety), OPA policies, and execute Dry-RunSupportedSupports Dry-Run debuggingSupports approval and audit viewing
Approve high-risk MCP tool releases and manage connection templatesSupportedCan submit tool registration requestsSupports security review
Issue gateway-specific credentials and adjust hard budgets (Budgets)SupportedPersonal test credentials onlyRead-only viewing of budget consumption

Synergy with Other Modules


Feature Details and Operation Guide

Part One: Creating a Gateway Instance (prod-enterprise-ai-gateway) and Snapshot Governance

  1. Log in to the Console, enter project 企业智能客服与知识中台 (enterprise-ai-hub-210), and click AI Gateway in the left navigation bar.

  2. In the default Gateways tab, click the Create gateway button and fill in the following configuration:

Field NameParameter IdentifierBenchmark Configuration ValueDescription
Gateway namenameprod-enterprise-ai-gatewayUnified gateway for production enterprise intelligent customer service and knowledge middle platform
Gateway identifier (ID)gatewayIdgw_7194685489feb1e169548619A system-generated or custom unique instance ID, used as part of the data-plane URL path
Descriptiondescription承载客服对话主备路由、安全护栏与语义缓存的生产网关Describes the gateway's business purpose and owning team
Default guardrailsdefaultGuardrailProfilegp-enterprise-safetyInstance-level default inbound/outbound guardrail policies attached to the instance
Default caching policydefaultCachePolicycp-semantic-cacheInstance-level default semantic caching policy attached to the instance
  1. Runtime snapshot sealing (Runtime Snapshot):

    • After completing the logical model, guardrail, and permission policy configurations, click Seal Runtime Snapshot at the top right of the gateway detail page.

    • The system generates an immutable configuration snapshot with a version number and SHA-256 digest, facilitating online change auditing and one-click rollback when anomalies occur.


Part Two: Configuring Logical Models (lm-enterprise-chat) and Primary/Secondary Failover Routing

  1. Switch to the Model routing tab, click Import from Provider, select dashscope-prod, import qwen-plus, qwen-turbo, text-embedding-v3, and qwen3.7-text-rerank into the gateway's standard model catalog, and configure the corresponding Token rate cards (used to automatically calculate cost per thousand Tokens).

  2. Click Create logical model and configure lm-enterprise-chat:

Configuration ItemParameter NameBenchmark Configuration ValueRouting Behavior Description
Logical model namelogicalModelNamelm-enterprise-chatThe logical name filled in the model field of the client request body
Capability typecapabilitychatSupports the OpenAI Chat Completions, Responses, and Anthropic Messages protocols
Routing policyroutingStrategypriority_failover (priority-based primary/secondary failover)Routes preferentially to the Priority 1 deployment target; when the primary target experiences timeouts, 429 rate limiting, or 5xx errors, it automatically switches without interruption to the Priority 2 secondary target
Primary deployment target (Target 1)deployments[0]Provider: dashscope-prod
Model: qwen-plus
Priority: 1, Weight: 100
Timeout: 60s
The primary high-inference-capability model carrying normal production traffic
Secondary deployment target (Target 2)deployments[1]Provider: dashscope-prod
Model: qwen-turbo
Priority: 2, Weight: 100
Timeout: 30s
The automatic fallback model when qwen-plus encounters upstream rate limiting or high latency

Part Three: Configuring Guardrails (gp-enterprise-safety) and Semantic Caching (cp-semantic-cache)

Switch to the Policies & Credentials tab to configure guardrails, semantic caching, and hard budgets respectively:

1. Guardrail configuration (gp-enterprise-safety)

Guardrail Rule ItemDirectionBenchmark Configuration ValueTrigger Action
Prompt injection and jailbreak detectionInboundenabled = true (strict mode)Block the request (BLOCK), returning a standard compliance message and 400 GuardrailBlocked
PII sensitive private data maskingInbound + Outbound (Both)Detect phone numbers, ID numbers, bank card numbers, and API keysAutomatically replace with masks (REDACT, e.g., 138****5678) before sending to the model or returning to the client
Enterprise prohibited-word and competitor malicious steering filterInbound + Outbound (Both)Reference the enterprise compliance lexiconBlock generation and record a security audit event
OPA policy dry-runPre-release policy validationInput a test JSON payloadPreview policy allow/deny decisions and rule hit details without changing live traffic

2. Semantic caching policy configuration (cp-semantic-cache)

Cache Parameter ItemParameter NameBenchmark Configuration ValueDescription
Policy namenamecp-semantic-cacheSemantic caching policy for high-frequency enterprise customer service Q&A
Embedding modelembeddingModeltext-embedding-v3 (1024 dimensions)The embedding model used to compute semantic vectors of user questions
Similarity hit thresholdsimilarityThreshold0.95Semantic equivalence and a cache hit are determined only when the cosine similarity is $\ge 0.95$, balancing accuracy and retrieval rate
Cache TTLttlSeconds3600 (1 hours)The automatic expiration time of cache entries; after a knowledge base version is released, associated caches can be purged with one click
Exclusion conditionsbypassWhentemperature > 0.7 or carrying X-AI-Gateway-No-Cache: trueHigh-randomness creative requests or requests explicitly requiring real-time computation automatically bypass the cache

3. Credential issuance and hard budgets (Budgets)

In the Credentials & Budgets area, create a dedicated abk_ gateway key for the customer service business entity (enforced binding to gw_7194685489feb1e169548619), and configure multi-dimensional hard quotas:


Step Four: Configure Tools & Agents and the approval workflow

  1. Switch to the Tools & Agents tab.

  2. Click Create from template:

    • You can select the managed template AppBase RAG and bind it to the kb_enterprise_support knowledge base within the project; or choose Remote MCP / OpenAPI / Function to integrate an external support ticket query tool.

  3. Set the tool approval mode (Approval Mode):

    • 无需审批(None): Only low-risk read-only tools that have completed the security review (such as public document retrieval) are allowed; high-risk or unreviewed tools are strictly prohibited by the system from being published in the "No Approval" mode.

    • 按策略审批(Policy-based): Whether manual or upstream confirmation is required is dynamically determined based on the caller's role and the parameter scope.

    • 始终审批(Always): Tools involving production database write operations, support ticket status changes, or sending external notifications must undergo explicit approval.


Step Five: Verify multi-protocol data plane calls and traffic quality observability

Using the gateway key bound to gw_7194685489feb1e169548619, verify the OpenAI Chat Completions, Responses, and Anthropic Messages compatible interfaces:

# 1. Call the gateway instance's OpenAI Chat Completions compatible endpoint
curl -X POST "https://appbuild.oceanbase.com/ai-gateway/g/gw_7194685489feb1e169548619/v1/chat/completions" \
  -H "Authorization: Bearer <APPBASE_AI_GATEWAY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lm-enterprise-chat",
    "messages": [
      {"role": "user", "content": "我的工单号是 TK-20261022-001,联系电话 13800138000,请帮我解释 OceanBase 向量索引排障步骤。"}
    ]
  }'

# 2. Call the Anthropic Messages-compatible endpoint of the gateway instance
curl -X POST "https://appbuild.oceanbase.com/ai-gateway/g/gw_7194685489feb1e169548619/v1/messages" \
  -H "x-api-key: <APPBASE_AI_GATEWAY_KEY>" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lm-enterprise-chat",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "请总结智能客服 SOP 标准作业手册中的工单升级时限要求。"}
    ]
  }'

Verification result:

  1. In the first request, the gp-enterprise-safety guardrail automatically desensitizes the phone number 13800138000 to 138****8000 before sending it to lm-enterprise-chat (served by the underlying qwen-plus).

  2. Switch to the Traffic & Quality tab; in call records and multi-dimensional usage tables, you can view the status code, latency, Token count, rate-card converted cost, and guardrail/cache hit labels of this call by credential, logical model lm-enterprise-chat, and physical deployment qwen-plus.


FAQ and Troubleshooting

Q1: Can the logical model lm-enterprise-chat be called directly via /v1/ai/chat/completions without the /ai-gateway/g/{gatewayId} prefix?

No. The design principle of the unified AI Gateway is "strict isolation per gateway instance; no inference alias routing without a Gateway ID". The logical model lm-enterprise-chat, guardrail gp-enterprise-safety, and cache cp-semantic-cache are all mounted under a specific gateway instance gw_7194685489feb1e169548619, so access must go through the /ai-gateway/g/gw_7194685489feb1e169548619/v1/... path.

Q2: After registering a new MCP tool, why can't its approval mode be set to "No Approval"?

To prevent unauthorized tool calls from causing data privilege violations or production incidents, the AI Gateway has a mandatory security gate: any tool marked as High Risk or Unreviewed is strictly prohibited from being published directly in the "No Approval" mode, both at the API and Console level. First have a project administrator complete the tool security review and confirm the risk level as low-risk read-only, or switch to the "Approval by Policy / Always Approve" mode.

Q3: The semantic cache (cp-semantic-cache) returned answers from an old knowledge version after a hit; how do I refresh it immediately?

When a knowledge base (such as kb_enterprise_support) releases a new version or a major business rule change occurs, you can:

  1. Include X-AI-Gateway-No-Cache: true in the header of a single debug request to force a cache bypass and update the result;

  2. Go to AI Gateway -> Policies & Credentials, find cp-semantic-cache, click Purge Cache, or raise the similarity threshold or shorten the TTL.

Q4: When the primary model qwen-plus triggers automatic failover to the fallback model qwen-turbo, does the client need to modify code and retry?

No. Failover is completed transparently inside the AI Gateway. The client only needs to make a single request to the logical model lm-enterprise-chat; after the gateway detects that the primary deployment target timed out or returned 429/5xx, it automatically switches to the fallback deployment target qwen-turbo within the same request lifecycle, and annotates the actually executed physical deployment target in the response metadata and Traffic & Quality observability logs.