Docs
Contents

GraphRAG, RAPTOR, and Advanced Knowledge Index Build Governance

This manual corresponds to Agentbase v2.10.0 (same version as the Console).

Knowledge Index (Index Capabilities) is the advanced derived-index governance center introduced in Agentbase v2.10.0, built for complex multi-hop reasoning and cross-document macro-level summarization, breaking through the limitations of traditional flat chunk retrieval (Flat Chunk RAG). Via the Knowledge Index (Indexes) tab on the knowledge base details page, enterprises can, on top of basic vector and full-text indexes, enable and finely orchestrate on demand the three advanced index capabilities of GraphRAG (entity-relation knowledge graph and Leiden community reports), RAPTOR (GMM+BIC based recursive abstractive clustering summary tree), and LLM Wiki (structured knowledge encyclopedia compilation). Together with version generation numbers (currentGeneration), staleness reason tracking (staleReason), and process-level dual-semaphore model capacity protection (ModelCapacity), they ensure that large-scale background index builds never preempt the inference and embedding quota of online real-time retrieval (/search).


Feature overview and core scenarios

When serving frontline and second-line engineers on an enterprise knowledge middle platform (such as project 企业智能客服与知识中台, ID: enterprise-ai-hub-210), the traditional "fixed-length chunking + Top-K vector retrieval" often falls short on two typical classes of highly difficult queries:

Agentbase's advanced knowledge index build governance thoroughly solves the above pain points through three derived indexes and a rigorous job scheduling mechanism:

  1. GraphRAG (knowledge graph index) with lexicon-enhanced disambiguation:

    • Automatically extracts domain entities and directed semantic relationships from ready chunks using qwen-plus, combines them with the project-level lexicon (lexiconEnabled: true) to normalize and merge synonymous entities (such as OB and OceanBase 4.3) into a canonical main entity OceanBase and filter stopword noise, then applies the Leiden hierarchical community detection algorithm to generate multi-level community summary reports, which are embedded via text-embedding-v3.

  2. RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval):

    • Takes the underlying chunk vectors (Level 0), automatically determines the optimal number of clusters via Gaussian Mixture Model (GMM) soft clustering + Bayesian Information Criterion (BIC), calls qwen-plus to generate abstractive summaries for each semantic cluster and embeds them again, recursively building a multi-level semantic summary tree upward (supporting knowledgeBase cross-document global trees and document single-document hierarchical trees), so that retrieval has both local detail precision and a global thematic view.

  3. LLM Wiki (knowledge encyclopedia compilation) and the await_graphrag dependency fence:

    • Comprehensively consumes underlying document chunks and the entity/community structures extracted by GraphRAG, automatically compiling a structured encyclopedia chapter tree rooted at kb-index (for example, the benchmark knowledge base auto-compiles 10 encyclopedia pages), and uses a await_graphrag fence mechanism to ensure that during joint builds, graph extraction completes before Wiki compilation starts.

  4. Dual-axis state machine, staleness awareness, and ModelCapacity traffic isolation:

    • Independently maintains each capability's configuration state (disabled / configured / enabled) and runtime state (building / ready / stale / failed / cancelled), precisely records 6 types of staleness reasons (staleReason), and strictly limits the concurrency of background index builds via a process-level ModelCapacity dual-semaphore mechanism, prioritizing zero latency jitter for online /search.


Core Concepts and Architecture

Architecture diagram of the three advanced derived-index build pipelines and ModelCapacity concurrency governance

GraphRAG, RAPTOR, and Advanced Knowledge Index Build Governance: workflow diagram, not a runtime screenshot

Flow diagram, not a runtime screenshot; read the relationships between steps or states along the arrows.

View the flow diagram in full size

One. GraphRAG (knowledge graph index): deep principles and configuration parameters

GraphRAG transforms unstructured text into a structured three-layer topology network of "Entity — Relation — Community Report":

  1. Five-Step Build Pipeline:

    • Step 1 (Entity and Relationship Extraction): Iterate over chunks in the knowledge base whose status is ready and invoke the Chat model (qwen-plus under dashscope-prod) to extract entity names, types, contextual descriptions, as well as directed semantic relationships between entities and relationship strength weights (Weight), according to predefined or custom entityTypes.

    • Step 2 (Knowledge Graph Lexicon Enhancement and Cross-Document Disambiguation): When lexiconEnabled: true is enabled, the extractor automatically loads the project lexicon bound to the knowledge base (such as lex_enterprise_synonyms and lex_support_stopwords) before persisting entries: if an extracted entity matches a synonym variant word (such as OB or OceanBase数据库), it is automatically merged into the canonical main term OceanBase; if it matches a stop stopword, the noise entity is discarded directly. It then performs cross-document, cross-language disambiguation of homonyms and description fusion.

    • Step 3 (Leiden Hierarchical Community Detection): Run the Leiden graph clustering algorithm based on the edge weights of the entity relationship graph, partitioning highly cohesive entity nodes into topological communities at different levels (Level 0 fine-grained sub-communities up to high-level global communities).

    • Step 4 (Community Summary Report Generation): Invoke qwen-plus to generate a structured report for each Leiden community containing a title (title), core summary (summary), key findings list (findings), and importance score (rating).

    • Step 5 (Knowledge Graph Vectorization and Persistence): Using graphragEmbedBatchSize (default 32) as the batch size, invoke text-embedding-v3 (1024 dimensions) to generate semantic vectors for all entity descriptions and community summary reports and persist them into the OceanBase vector table.

  2. GraphRAG Core Configuration Parameter Table:

Parameter fieldTypeDefault / benchmark valueDetailed description and tuning advice
enabledbooleantrueWhether to enable the GraphRAG knowledge graph capability for the current knowledge base.
entityTypesstring[]["产品模块", "故障码", "API接口", "配置参数", "SOP步骤", "专家角色"]Restricts the domain entity categories the large model focuses on identifying; focused entity categories significantly reduce the graph extraction rate of meaningless generic nouns.
lexiconEnabledbooleantrueKnowledge graph lexicon enhancement switch: When enabled, the synonym normalization rules and stop filtering rules of the bound lexicon are applied in real time during the entity extraction stage.
language / outputLanguagestring"zh-CN"Options are "zh-CN", "en-US", "auto". Controls the generation language of entity consolidated descriptions and Leiden community reports.
graphragEmbedBatchSizeinteger32 (range 1–128)The number of Embedding chunks per batch when vectorizing entity and community summaries, working with ModelCapacity to control concurrent throughput.
extractionPromptstringPlatform built-in templateOptionally customize the entity relationship extraction prompt, supporting tailored extraction granularity for database troubleshooting or financial compliance domains.

Two. RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) In-Depth Principles and Configuration Parameters

RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) builds a multi-level semantic summary tree bottom-up, compensating for the shortcoming that flat chunks lack cross-paragraph macro context:

  1. Four-Step Recursive Clustering Pipeline:

    • Level 0 (Leaf Node Layer): Use the ready raw document chunks (Leaf Chunks) in the knowledge base and their 1024-dimensional text-embedding-v3 vectors directly as the leaf nodes of the tree.

    • GMM Soft Clustering + BIC Criterion: Perform dimensionality reduction and Gaussian Mixture Model (GMM) soft clustering on the current layer's node vectors (allowing a single cross-topic chunk to belong to multiple relevant semantic clusters simultaneously), and use the Bayesian Information Criterion (BIC) to automatically compute the optimal number of clusters, avoiding under-clustering or over-clustering caused by manually hard-coding the cluster count.

    • Cluster Abstract Summary and Re-Vectorization: For the node text within each cluster, invoke qwen-plus to distill and generate a higher-level abstraction summary node (Summary Node), and invoke text-embedding-v3 to compute that summary node's vector.

    • Recursive Convergence (Level 1..maxLevels): Repeat "GMM+BIC clustering -> LLM abstraction summarization -> vectorization" on the newly generated summary nodes until the configured maximum tree height maxLevels is reached or the remaining node count can no longer be clustered, forming a complete hierarchical semantic tree.

  2. RAPTOR Core Configuration Parameter Table:

Parameter fieldTypeDefault / benchmark valueDetailed description and selection advice
enabledbooleantrueWhether to enable the RAPTOR recursive clustering summary tree capability for the current knowledge base.
scopestring"knowledgeBase"Clustering scope:
• "knowledgeBase": Cross-document global tree, clustering the chunks of all documents in the entire library in the same vector space, suitable for cross-manual comprehensive Q&A;
• "document": Single-document independent tree, building a dedicated section summary tree within each document, suitable for internal summarization of extra-long white papers or hundred-page contracts.
maxLevelsinteger3 (range 1–5)The maximum height of the recursive clustering tree (excluding Level 0 leaf nodes). A mid-sized enterprise knowledge base is recommended to be set to 3.
clusterSizeinteger10 (range 3–50)The upper limit of target child nodes a single semantic cluster can contain, affecting the context aggregation width of each layer's LLM summarization.
summarizationPromptstringPlatform built-in templateOptionally customize the cluster summary generation prompt, guiding the large model to retain key parameters, thresholds, and causal conclusions.

Three. LLM Wiki Compilation and the await_graphrag Dependency Barrier Mechanism

When the user enables GraphRAG and LLM Wiki simultaneously on the Indexes tab and triggers batch builds, Agentbase has a built-in await_graphrag dependency barrier (Dependency Barrier):


Four. Dual-Axis State Machine, Staleness Reasons (staleReason), and ModelCapacity Capacity Governance

1. Index Capability Dual-Axis State Machine and currentGeneration

Each advanced index capability (graphrag / raptor / wiki) is described by two orthogonal state dimensions and a monotonically increasing generation number currentGeneration:

2. Dictionary of Six Standard Staleness Reasons (staleReason)

Staleness reason enum value (staleReason)Triggering scenario descriptionRecommended remediation action (operation + executionMode)
document_ingestedNew documents were uploaded to the knowledge base or existing documents were re-ingested (Re-ingest)operation: "build", executionMode: "incremental"
document_deletedA document that had participated in the build was deleted from the knowledge baseoperation: "repair" or executionMode: "rebuild"
chunk_updatedThe body content of a chunk was manually modified or the chunk was deleted on the document detail pageoperation: "build", executionMode: "incremental"
data_source_syncAn incremental sync (/datasources/{id}/sync) on an external data source such as Yuque produced new or changed documentsoperation: "build", executionMode: "incremental"
config_changedentityTypes, lexiconEnabled, scope or the prompt was modified via PATCH /index-capabilities/{capability}operation: "build", executionMode: "rebuild"
manual_rebuildThe administrator explicitly requested a forced full rebuildoperation: "build", executionMode: "rebuild"

3. Process-Level ModelCapacity Dual-Semaphore Concurrency Protection

In production environments, GraphRAG entity extraction and RAPTOR multi-layer summarization generate intensive LLM (qwen-plus) and Embedding (text-embedding-v3) calls. To prevent background build tasks from exhausting the upstream model TPM/RPM quotas and causing front-end customer support Q&A (/search) timeouts, Agentbase implements a ModelCapacity dual-semaphore isolation mechanism at the Worker and API process layers:


Feature Entrypoints and Permission Matrix

Console entrypoint / feature moduleFrontend route pathCorresponding REST API endpointServer SDK method (Node.js / Python)Required RBAC role / API Key scopeCross-module dependency
Knowledge base → Knowledge index overview page/console/project-default-enterprise-ai-hub-210/knowledge-base/kb-kb_enterprise_support/indexesGET /v1/knowledgebases/{kbId}/index-capabilitiesknowledgeBase.listIndexCapabilities() / list_index_capabilities()Roles: owner, admin, developer, member (read-only)
Scope: knowledgebases.read
Requires at least 1 document with status=ready status in the knowledge base
Configure GraphRAG / RAPTOR / Wiki capability parameters/console/.../kb-kb_enterprise_support/indexes (click "Configure" on the corresponding card)PATCH /v1/knowledgebases/{kbId}/index-capabilities/{capability}knowledgeBase.updateIndexCapability()Roles: owner, admin, developer
Scope: knowledgebases.write
Requires qwen-plus and text-embedding-v3 to be already configured in dashscope-prod; enabling lexiconEnabled requires a bound lexicon
Dispatch incremental build, repair, or full rebuild jobs/console/.../kb-kb_enterprise_support/indexes (click "Build" or "Rebuild")POST /v1/knowledgebases/{kbId}/index-operations
GET /v1/knowledgebases/{kbId}/index-operations
knowledgeBase.createIndexOperation()
knowledgeBase.listIndexOperations()
Roles: owner, admin, developer
Scope: knowledgebases.write
Linked with the ready gating of version management (Versions)
Cancel a running index jobRow action "Cancel" in the recent jobs listPOST /v1/knowledgebases/{kbId}/index-operations/{operationId}/cancelknowledgeBase.cancelIndexOperation()Roles: owner, admin, developer
Scope: knowledgebases.write
Only takes effect on jobs in queued or running status
Clear derived index dataCapability card more menu "Clear Index Data (Clear)"POST /v1/knowledgebases/{kbId}/index-capabilities/{capability}/clearknowledgeBase.clearIndexCapability()Roles: owner, admin
Scope: knowledgebases.write
The request body must explicitly pass in "confirm": true; only derived indexes are cleared, without deleting original documents and base chunks

Feature Details and Operation Guide

Step 1: Configure and enable the GraphRAG, RAPTOR, and LLM Wiki capabilities

  1. Log in to the Console, go to project 企业智能客服与知识中台 (enterprise-ai-hub-210) -> knowledge base -> 企业智能客服与技术支持知识库 (kb_enterprise_support) -> click the top knowledge index (Indexes) tab.

  2. Configure GraphRAG (knowledge graph):

    • Click the Configure button in the upper-right corner of the GraphRAG card;

    • Turn on the Enabled switch;

    • In Entity Types, enter: 产品模块, 故障码, API接口, 配置参数, SOP步骤, 专家角色;

    • Check Enable lexicon enhancement (lexiconEnabled) so that knowledge graph extraction automatically associates the bound lex_enterprise_synonyms and lex_support_stopwords;

    • For Language select 简体中文 (zh-CN), keep vectorization batch size (graphragEmbedBatchSize) at 32, and click Save Configuration.

  3. Configure RAPTOR (recursive clustering summary tree):

    • Click Configure on the RAPTOR card;

    • Turn on the Enabled switch, set Scope to 整库跨文档聚类 (knowledgeBase), set Max Levels to 3, set Cluster Size to 10, and click Save Configuration.

  4. Configure LLM Wiki (knowledge encyclopedia compilation):

    • Click Configure on the LLM Wiki card;

    • Turn on the Enabled switch, set Output language (wikiLanguage) to zh-CN, set max page concurrency (pageMaxConcurrent) to 4, check Compile after GraphRAG is ready (await_graphrag), and click Save Configuration.


Step 2: Dispatch asynchronous build jobs (build / repair) and monitor generation evolution

  1. Above the Indexes tab, for capability cards in the configured or stale state, click Build:

    • Incremental build (operation: "build", executionMode: "incremental"): recommended for daily document additions or after Yuque sync (data_source_sync); it only performs extraction and merging on incrementally changed chunks, with the fastest speed and lowest Token consumption;

    • Full rebuild (operation: "build", executionMode: "rebuild"): use when entityTypes has been modified, lexiconEnabled has been switched, or RAPTOR's scope (staleReason: config_changed) has been adjusted; it generates a brand-new generation (currentGeneration + 1) based on all ready documents in the knowledge base;

    • Index repair (operation: "repair"): use when a build was interrupted by upstream network jitter or when orphaned edges/nodes need to be aligned.

  2. In the Index Operations list at the bottom of the page, you can view job ID, target capability (graphrag / raptor / wiki), execution mode, elapsed time, and progress details in real time. After a successful build, all three capability cards display the green ready status, and currentGeneration is incremented.


Step 3: Fully operate the three index capabilities and job lifecycles via the REST API

# 1. Query the current status, generation number and staleness reason of the knowledge base's three advanced index capabilities (GET /index-capabilities)
curl -X GET "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/index-capabilities" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>"

# 2. Configure and enable the GraphRAG indexing capability, turning on lexicon enhancement and custom entity types (PATCH /index-capabilities/graphrag)
curl -X PATCH "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/index-capabilities/graphrag" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "outputLanguage": "zh-CN",
    "config": {
      "entityTypes": ["产品模块", "故障码", "API接口", "配置参数", "SOP步骤", "专家角色"],
      "lexiconEnabled": true,
      "language": "zh-CN",
      "graphragEmbedBatchSize": 32
    }
  }'

# 3. Configure and enable the RAPTOR recursive clustering summary tree capability (PATCH /index-capabilities/raptor)
curl -X PATCH "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/index-capabilities/raptor" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "outputLanguage": "zh-CN",
    "config": {
      "scope": "knowledgeBase",
      "maxLevels": 3,
      "clusterSize": 10
    }
  }'

# 4. Configure and enable the LLM Wiki capability with the await_graphrag fence (PATCH /index-capabilities/wiki)
curl -X PATCH "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/index-capabilities/wiki" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "outputLanguage": "zh-CN",
    "config": {
      "wikiLanguage": "zh-CN",
      "pageMaxConcurrent": 4,
      "await_graphrag": true
    }
  }'

# 5. Trigger an incremental or full index build job with an idempotency key (POST /index-operations)
curl -X POST "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/index-operations" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "capability": "graphrag",
    "operation": "build",
    "executionMode": "incremental",
    "idempotencyKey": "iop_graphrag_kb_support_20261022_01"
  }'

# 6. Cancel a queued or running build job (POST /index-operations/{operationId}/cancel)
curl -X POST "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/index-operations/iop_7c6b5a4d3e2f1098/cancel" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>"

# 7. Safely clear the specified derived index artifacts (requires explicitly passing confirm: true; does not affect original documents and chunks)
curl -X POST "https://appbuild.oceanbase.com/v1/knowledgebases/kb_enterprise_support/index-capabilities/raptor/clear" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  -H "X-Appwrite-Key: <APPBASE_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "confirm": true
  }'

FAQ (Frequently Asked Questions and Troubleshooting)

Q1: I have newly uploaded documents and the document status changed to ready, but the GraphRAG and RAPTOR card statuses changed to the yellow stale (stale). Will online retrieval report errors at this point?

Q2: I triggered GraphRAG and LLM Wiki builds at the same time; why does the Wiki job stay stuck in the waiting state (await_graphrag) and not start generating pages immediately?

Q3: After enabling GraphRAG's lexiconEnabled: true, I modified synonym entries in the project lexicon lex_enterprise_synonyms. Will the already generated knowledge graph automatically merge old nodes?

Q4: After executing POST .../index-capabilities/{capability}/clear to clear the index, will my 4 uploaded source documents and base vector chunks be deleted?