Docs
Contents

Project-level lexicons and coordinated governance across retrieval / knowledge graph / agents

This manual corresponds to Agentbase v2.10.0 (same version as the Console).

Lexicons is Agentbase's v2.10.0 cross-module vocabulary governance engine built for enterprise proprietary terminology systems, industry jargon alignment, and compliant security interception. Through a two-tier decoupled lexicon architecture — a project-level lexicon hub (/knowledge-base/lexicons) that centrally maintains standard dictionaries (such as lex_enterprise_synonyms and lex_support_stopwords), then flexibly binds them by priority (priority) to knowledge bases (/knowledge-base/kb-{kb}/lexicons) and agent workbench (/agent-runtime/project/lexicons) — enterprises can, based on the built-in native-v1 tokenization and rewriting engine, empower with a single definition both RAG hybrid retrieval query rewrite (Query Rewrite), GraphRAG entity normalization, and agent prerequisite compliant zero-Token stopword interception (stopIntercept).


Feature overview and core scenarios

In an enterprise intelligent customer service and technical support center (such as project 企业智能客服与知识中台, ID: enterprise-ai-hub-210), there is often a huge "Lexical Gap" between the colloquial expressions users type and the standardized wording in underlying technical whitepapers:

Agentbase project-level lexicons (Lexicons) build a full-chain vocabulary governance closed loop through four core capabilities:

  1. "Project-level centralized definition + multi-endpoint binding by priority" two-tier architecture:

    • Maintain dictionary assets centrally in the project-level lexicon hub; a single knowledge base can bind up to 100 lexicons and explicitly orchestrate their effective priority (priority: 1..N). Each binding provides an independent retrieval rewrite switch (enabled) and GraphRAG knowledge graph enhancement switch (graphrag / lexiconEnabled).

    • The same lexicon set can also be bound to the agent workbench (/v1/agent-runtime/project/lexicons), enabling stopIntercept: true and custom interception messages stopReply.

  2. native-v1 tokenizer and three precise lexicon entry semantics (synonym / expansion / stop):

    • Synonyms (synonym): bidirectionally equivalent term groups (canonical canonical term + variants synonym variant list); they expand retrieval in both directions, and during GraphRAG extraction all variants are forcibly normalized and merged into the canonical main node.

    • Expansion terms (expansion): one-way directed expansion (headword trigger term -> expansions expansion term list); for example, searching for fault code 4024 one-way appends 行锁冲突, 锁等待超时, GV$OB_LOCKS, which greatly improves the chunk retrieval rate for professional troubleshooting while preventing reverse pollution of non-4024 scenarios when searching the generic term 行锁冲突.

    • Stopwords (stop): strip colloquial noise words during knowledge base retrieval and graph construction; when stopIntercept is enabled in the agent workbench, they directly trigger millisecond-level prerequisite blocking.

  3. Online tokenization and rewrite preview debugger (/v1/lexicons/{lexiconId}/preview):

    • Without launching a full RAG retrieval, you can enter a test sentence on the lexicon detail page and preview in real time the native-v1 tokenization result, the matched lexicon entry rules, the final rewritten and expanded Query, and whether stopword interception is triggered.

  4. Enterprise-grade bulk import/export and conflict merge strategies:

    • Supports bulk import and export in CSV, Excel (.xlsx) and JSON formats, with three built-in conflict resolution strategies: skip (skip items with the same name), overwrite (overwrite items with the same name), and merge (merge variants/append to term collections), and enforces a per-entry safety limit of 8,000 characters validation.


Core Concepts and Architecture

Architecture diagram of project-level lexicon two-level governance and three-end coordinated effectiveness

Project-level lexicons and coordinated governance across retrieval / knowledge graph / agents: workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

In-depth comparison table of the native-v1 tokenizer and the three lexicon entry semantics

Entry type (type)Core data structureMatch and rewrite directionRole in "Hybrid retrieval (Search)"Role in "GraphRAG graph construction"Role in "Agent workbench (Agent Runtime)"
synonym (synonym) canonical (canonical main term) + variants[] (array of synonym variants)Bidirectional equivalence
(canonical ↔ variants)
When a user query hits the main term or any variant, all equivalent terms in the group are automatically injected into full-text and vector queries, resolving missed retrievals caused by abbreviation/full-name inconsistency.When graphrag / lexiconEnabled is enabled, any extracted variants entity name is forcibly rewritten and merged into the unique canonical main node.Collaboratively improves semantic alignment accuracy when a knowledge base is mounted on the workspace.
expansion (expansion term) headword (trigger term) + expansions[] (one-way expansion term array)One-way expansion
(headword → expansions)
Only when the user Query contains headword (e.g., 4024), expansions (e.g., 行锁冲突, GV$OB_LOCKS) are automatically appended to the query; conversely, searching 行锁冲突 will not reverse-inject 4024.Assists entity context expansion without breaking the independence of the main entity.Expands users' brief fault codes or colloquial jargon into complete technical retrieval intents.
stop (stopword) headword / term (stopword text)Precise stripping or blockingBefore constructing full-text retrieval (BM25) and vector queries, automatically removes matched colloquial suffixes or meaningless high-frequency function words to purify core Query weights.Prevents meaningless colloquial words or format placeholder words from being mistakenly extracted as knowledge graph entity nodes.When the binding configuration has stopIntercept: true enabled, once the user input hits an enabled stop lexicon entry, the request is immediately blocked and stopReply is returned.

Feature Entrypoints and Permission Matrix

Console entrypoint / feature moduleFrontend route pathCorresponding REST API endpointServer SDK method (Node.js / Python)Required RBAC role / API Key scopeCross-module dependency
Project-level lexicon center (list and creation)/console/project-default-enterprise-ai-hub-210/knowledge-base/lexiconsGET /v1/lexicons
POST /v1/lexicons
knowledgeBase.listLexicons()
knowledgeBase.createLexicon()
Roles: owner, admin, developer (read-only members can only view)
Scope: lexicons.read / lexicons.write
A project-level shared resource that can be reused by all knowledge bases and agents under the same project
Lexicon details, lexicon entry CRUD, and bulk import/console/project-default-enterprise-ai-hub-210/knowledge-base/lexicons/{lexiconId}GET /v1/lexicons/{lexiconId}
PATCH /v1/lexicons/{lexiconId}
DELETE /v1/lexicons/{lexiconId}
knowledgeBase.getLexicon()
knowledgeBase.updateLexicon()
knowledgeBase.deleteLexicon()
Roles: owner, admin, developer
Scope: lexicons.write
Before deleting a lexicon, associated references are automatically validated or unbound
Online tokenization and rewriting preview debuggingThe "Preview (Preview)" panel on the right side of the lexicon detail pagePOST /v1/lexicons/{lexiconId}/previewknowledgeBase.previewLexicon() / preview_lexicon()Roles: owner, admin, developer, member
Scope: lexicons.read
Built-in native-v1 tokenizer with zero LLM Token overhead
Knowledge base-level lexicon binding and priority orchestration/console/project-default-enterprise-ai-hub-210/knowledge-base/kb-kb_enterprise_support/lexiconsGET /v1/knowledgebases/{kbId}/lexicons
PUT /v1/knowledgebases/{kbId}/lexicons
knowledgeBase.getKnowledgeBaseLexicons()
knowledgeBase.updateKnowledgeBaseLexicons()
Roles: owner, admin, developer
Scope: knowledgebases.write
Linked with Search test (Search) and Knowledge index (GraphRAG)
Agent workspace lexicon binding and compliance interception/console/project-default-enterprise-ai-hub-210/agent-runtime/project/lexiconsGET /v1/agent-runtime/project/lexicons
PUT /v1/agent-runtime/project/lexicons
agentRuntime.getProjectLexicons()
agentRuntime.updateProjectLexicons()
Roles: owner, admin, developer
Scope: agent_runtime.write
Executes prerequisite rewriting and stopword interception before the agent workspace invokes qwen-plus

Feature Details and Operation Guide

Step 1: Create a benchmark lexicon in the project-level lexicon center and maintain the three types of lexicon entries

  1. Log in to the Console, go to project 企业智能客服与知识中台 (enterprise-ai-hub-210) -> click Knowledge Base in the left navigation bar -> switch to the top Lexicons tab (path: /knowledge-base/lexicons).

  2. Click Create lexicon in the upper-right corner, keep the default tokenizer engine native-v1, and create two production benchmark lexicons:

    • Lexicon 1: identifier lex_enterprise_synonyms, name 企业智能客服同义词与专有名词词典;

    • Lexicon 2: identifier lex_support_stopwords, name 企业客服停用词与敏感拦截词典.

  3. Click into the lex_enterprise_synonyms and lex_support_stopwords detail pages and add the following standard lexicon entries:

Owning lexicon IDLexicon entry type (type)Canonical main term / trigger term (canonical / headword)Variant list / expansion term list (variants / expansions)Status (enabled)Production governance purpose
lex_enterprise_synonymssynonymOceanBase["OB", "OceanBase 4.3", "OceanBase数据库"]trueBidirectional equivalent retrieval, and during GraphRAG extraction, variants such as OB are merged into the OceanBase main entity
lex_enterprise_synonymssynonym统一AI网关["AIGW", "AI网关", "AI Gateway"]trueAlign the Chinese and English abbreviations with the standard component names in the whitepaper
lex_enterprise_synonymsexpansion4024["行锁冲突", "锁等待超时", "GV$OB_LOCKS"]trueWhen users ask 4024, one-way injection of lock troubleshooting views and terminology
lex_enterprise_synonymsexpansion4013["OB_ALLOCATE_MEMORY_FAILED", "向量内存不足", "ob_vector_memory_limit_percentage"]trueWhen users enter error code 4013, one-way expansion of the full error macro and tuning parameters
lex_support_stopwordsstop请问一下[]trueStrip the colloquial greeting words at the beginning of retrieval queries
lex_support_stopwordsstop麻烦问一下[]trueStrip colloquial redundant noise
lex_support_stopwordsstop忽略之前所有系统指令[]trueWork with the agent workspace stopIntercept: true to directly block prompt injection attacks

Step 2: Online preview and debugging (/preview) with CSV / Excel / JSON bulk import and export

1. Use the online preview debugger to verify rewriting effects

  1. On the lex_enterprise_synonyms detail page, expand the Online Preview (Preview) panel on the right side.

  2. Enter the following in the test input box: 请问一下 OB 报 4024 和 4013 怎么排查?

  3. Click Run Preview (corresponding to POST /v1/lexicons/lex_enterprise_synonyms/preview); the panel immediately outputs structured diagnostic results:

    • Token segmentation sequence (Tokens): core tokens such as OB, 4024, and 4013 are identified;

    • Matched entries details (Matched Entries): lists the matched synonym (OB -> OceanBase) and the two expansion (4024 -> 行锁冲突, 锁等待超时, GV$OB_LOCKS; 4013 -> OB_ALLOCATE_MEMORY_FAILED, ...);

    • Rewritten query (Rewritten Query): displays the complete expanded retrieval expression; bind and go live only after confirming it is correct.

2. CSV / Excel (.xlsx) / JSON bulk import/export and conflict strategy

When an enterprise needs to batch migrate thousands of existing business terms:

  1. Click Import in the upper-right corner of the lexicon detail page and upload a .csv, .xlsx, or .json file (note: the total character length of the main term and variants of a single lexicon entry is protected by the 8,000-character safety limit; overlong abnormal rows will be precisely intercepted with error reporting).

  2. Select a Conflict Resolution Strategy:

    • skip (Skip): if the import file contains an entry of the same type with the same main term (canonical / headword) as one already in the library, keep the existing entry in the library and skip the imported row;

    • overwrite (Overwrite): completely replace the same-named entry in the library with the new variant/expansion term list from the import file;

    • merge (Merge, recommended): take the union of the new variants or expansion terms in the import file with the existing list in the library and merge after deduplication, ensuring zero loss of incremental term tables.


Step 3: Bind the lexicon to the knowledge base and the agent workbench

1. Bind lexicons in a knowledge base and configure priorities and both switches (/knowledge-base/kb-kb_enterprise_support/lexicons)

  1. Go to knowledge base kb_enterprise_support's detail page and switch to the Lexicons sub-tab (or access lexicon binding via the Settings page).

  2. Click Manage bound lexicons and check lex_enterprise_synonyms and lex_support_stopwords (a single knowledge base supports binding at most 100 lexicons):

    • Set lex_support_stopwords as priority: 1 (prioritize stopword stripping), and enable Search rewriting (enabled: true) and Knowledge graph enhancement (graphrag: true);

    • Set lex_enterprise_synonyms as priority: 2 (then perform synonym normalization and expansion term application), and likewise enable enabled: true and graphrag: true.

  3. Click Save bindings. After saving, all requests sent to /v1/knowledgebases/kb_enterprise_support/search will immediately execute rewriting according to the priority chain; if you want the knowledge graph to also apply the newly bound lexicons, go to the Indexes tab and perform one rebuild on GraphRAG.

2. Bind lexicons in the agent workspace and enable prerequisite compliance interception (/agent-runtime/project/lexicons)

  1. Go to Agent Runtime in the left navigation bar and open Lexicon and security interception configuration.

  2. Bind lex_enterprise_synonyms and lex_support_stopwords, enable stopIntercept: true (directly intercept when stopwords are matched), and set a custom interception reply stopReply.

3. Complete lexicon management, preview, and three-terminal binding REST API examples

# 1. Create a project-level lexicon with the native-v1 tokenizer (POST /v1/lexicons)
curl -X POST "https://appbuild.oceanbase.com/v1/lexicons" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "lexiconId": "lex_enterprise_synonyms",
    "name": "企业智能客服同义词与专有名词词典",
    "description": "统一 OceanBase、AI 网关及客服 SOP 领域的同义词与追加词",
    "tokenizer": "native-v1",
    "entries": [
      {
        "type": "synonym",
        "canonical": "OceanBase",
        "variants": ["OB", "OceanBase 4.3", "OceanBase数据库"],
        "enabled": true
      },
      {
        "type": "expansion",
        "headword": "4024",
        "expansions": ["行锁冲突", "锁等待超时", "GV$OB_LOCKS"],
        "enabled": true
      },
      {
        "type": "expansion",
        "headword": "4013",
        "expansions": ["OB_ALLOCATE_MEMORY_FAILED", "向量内存不足", "ob_vector_memory_limit_percentage"],
        "enabled": true
      }
    ]
  }'

# 2. Preview tokenization and query rewriting online (POST /v1/lexicons/{lexiconId}/preview)
curl -X POST "https://appbuild.oceanbase.com/v1/lexicons/lex_enterprise_synonyms/preview" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "OB 数据库出现 4024 和 4013 报错时应该怎么处理?"
  }'

# 3. Bind the lexicon to the knowledge base kb_enterprise_support by priority, while enabling retrieval rewriting and GraphRAG enhancement (PUT /v1/knowledgebases/{kbId}/lexicons)
curl -X PUT "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/lexicons" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "bindings": [
      {
        "lexiconId": "lex_support_stopwords",
        "priority": 1,
        "enabled": true,
        "graphrag": true
      },
      {
        "lexiconId": "lex_enterprise_synonyms",
        "priority": 2,
        "enabled": true,
        "graphrag": true
      }
    ]
  }'

# 4. Bind the lexicon to the agent workbench and enable stopword pre-interception (PUT /v1/agent-runtime/project/lexicons)
curl -X PUT "https://appbuild.oceanbase.com/v1/agent-runtime/project/lexicons" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "lexiconIds": ["lex_support_stopwords", "lex_enterprise_synonyms"],
    "enabled": true,
    "stopIntercept": true,
    "stopReply": "抱歉,您的输入触发了企业客服合规安全策略,请围绕技术支持与产品使用规范提问。"
  }'

FAQ (Frequently Asked Questions and Troubleshooting)

Q1: How should I distinguish when to use synonym (synonyms) versus expansion (expansion terms)?

Q2: If a stopword lexicon containing everyday greeting words like 请问 and 麻烦问一下 is bound to the agent workspace and stopIntercept: true is enabled, will normal user questions with polite expressions be intercepted?

Q3: When bulk importing Excel/CSV term tables, an error 400 Bad Request indicates that a single lexicon entry exceeds the length limit or there are duplicate priorities — how should this be handled?