Docs
Contents

Documents and Ingestion Management

Documents & Ingestion is the core production workshop where a knowledge base (such as kb_enterprise_support) transforms raw unstructured files into high-quality retrievable Chunks and vectors. Through the Documents tab on the knowledge base detail page, users can batch-upload or bind bucket files, flexibly configure document parsers (text / vlm / mineru) and file-appropriate chunk strategies (general / title / smart / regex / faq / sheet), manage hierarchical category tags and precise locator identifiers (documentTag, subProductId), track asynchronous ingestion jobs and document status, and review, edit, and verify image understanding results for generated chunks online.

The ingestion job and document have separate states. HTTP 202 means ingestion was accepted. Poll the job for queued, running, ready, failed, cancelling, or cancelled; the document can be processing. After success, also confirm the document is ready and its activeGeneration matches that job's generation.


Feature overview and core scenarios

High-quality RAG Q&A performance depends on "accurate parsing, reasonable chunking, and fine-grained tags." Agentbase provides fine-grained full-lifecycle control capabilities in the Documents and Ingestion module:

  1. Dual-mode file access and batch upload:

    • Console batch direct upload: the file picker supports uploading up to 50 files per batch, automatically persisting raw files to the knowledge base's dedicated bucket (e.g., kb_enterprise_docs) and creating document records; by default, asynchronous ingestion is triggered immediately after upload.

    • Storage file ID binding: supports referencing files already uploaded to the project bucket (Storage) via fileId to create documents and start ingestion through the API directly, avoiding repeated transfer of large files.

  2. Three types of multimodal document parsers:

    • Supports ultra-fast parsing for plain text/Markdown (text), vision model parsing for complex scanned documents and architecture diagrams (vlm, using the configured vision model), and deep layout restoration for complex two-column PDFs, cross-page tables, and mathematical formulas (mineru).

    • Supports structured chunking of Excel (.xlsx / .xls) by Sheet, and can be combined with Image Understanding to automatically extract semantic descriptions of embedded charts.

  3. File-specific chunk strategies (Chunking Parsers):

    • Built-in general (generic fixed-length sliding window), title (splitting by Markdown/document heading hierarchy), smart (intelligent semantic boundary chunking), regex (custom regular expression splitting), faq (Q&A pair structure splitting), and sheet (spreadsheet chunking). Available strategies depend on the file type; tune parameters such as chunkTokenNum, overlapPercent, titleLevel, and minChunkTokens.

  4. Document tags and locator identifiers:

    • Supports slash-separated parent-child hierarchical tags (e.g., 技术支持/数据库/OceanBase), providing a tag tree directory view and a flat list view.

    • v2.10.0 adds locator identifiers documentTag and subProductId: they support structured locator and free-form tags for filtering by tag and sub-product in document lists, retrieval, and Wiki conflict lists.

  5. Chunk-level transparent review and online hot-fixing:

    • Once document ingestion becomes ready, you can view each chunk's text, Token length, and associated image understanding summaries on the document detail page, and edit/fix chunk text or delete noisy chunks online; after modification, the corresponding vectors are automatically recomputed.

Knowledge base documents and tag directory view

A synthetic document is ready in a 2.12.0 test knowledge base; this example has no tag directory yet.


Core Concepts and Architecture

Parser and Chunker Core Concepts Table

Component dimensionAvailable typesApplicable document scenarios and core logic
Document Parser (documentParser)textBest suited for well-structured .md, .txt, .docx, .html and regular born-digital documents, with the fastest parsing speed and zero extra visual model overhead.
Document Parser (documentParser)vlmInvokes the project's registered visual large model (such as qwen-vl-max under dashscope-prod) to recognize complex layouts, scanned documents, handwritten annotations, and architecture flowcharts page by page.
Document Parser (documentParser)mineruDesigned for academic papers, technical white papers, and complex financial statement PDFs, precisely recognizing multi-column layouts, cross-page tables, and LaTeX formulas and converting them into structured Markdown.
Chunk Parser (parserId)generalGeneral sliding-window chunking. Splits by a fixed chunkTokenNum (default 512) and overlapPercent (default 10%), suitable for long texts without clear heading structure.
Chunk Parser (parserId)titleSplits chapters at the selected heading level (titleLevel, default 3, i.e., ###); a segment may include its current heading, but the full parent heading path is not guaranteed.
Chunk Parser (parserId)smartDynamically merges fragmented paragraphs and splits overly long paragraphs based on paragraph boundaries, semantic coherence, and the minimum chunk threshold (minChunkTokens, default 128).
Chunk Parser (parserId)regexSplits documents by a user-specified regular expression (regexPattern), suitable for log archives, specific-format legal provisions, or standardized entry files.
Chunk Parser (parserId)faqReads questions and answers line by line: the first column becomes the chunk content and question, while the second is stored in metadata.answer. Put each pair in two columns on the same line; adjacent question and answer lines are not combined automatically.

Feature Entrypoints and Permission Matrix

Console Entry Points, REST API, SDK Methods, and Permission Matrix

Console entrypoint / feature moduleFrontend route pathCorresponding REST API endpointServer SDK method (Node.js / Python)Required RBAC role / API Key scopeCross-module dependency
Document List, Folder Tree, and documentTag Filtering/console/project-default-enterprise-ai-hub-210/knowledge-base/kb-kb_enterprise_supportGET /v1/knowledgebases/{kbId}/documentsknowledgeBase.listDocuments() / list_documents()Roles: owner, admin, developer, member (read-only)
Scope: knowledgebases.read
Directory views use directoryPath / directoryUnclassified; the list also supports documentTag and subProductId filters
Batch upload documents/console/.../kb-kb_enterprise_support (click "Upload documents")POST /v1/knowledgebases/{kbId}/documents/batchknowledgeBase.createDocumentsBatch() / create_documents_batch()Roles: owner, admin, developer
Scope: knowledgebases.write
Upload files and batch-create document records first; creation returns HTTP 201, and ingestion is started separately as needed
Bind one storage bucket fileRegister an existing file through the APIPOST /v1/knowledgebases/{kbId}/documentsknowledgeBase.createDocument() / create_document()Roles: owner, admin, developer
Scope: knowledgebases.write
Start ingestion separately as needed after creating the record
Query Asynchronous Ingestion Progress and Trigger Re-ingestionDocument list status badges and row action "Re-ingest"POST /v1/knowledgebases/{kbId}/documents/{docId}/ingestions
GET .../ingestions/{ingestionId}
knowledgeBase.createIngestion()
knowledgeBase.getIngestion()
Roles: owner, admin, developer
Scope: knowledgebases.write / read
Uses the knowledge base's configured embedding model with matching dimensions; vlm parsing or image understanding also requires a configured vision model (see knowledge base settings)
Edit Document Tags and Positioning IdentifiersDocument row action "Edit tags"PATCH /v1/knowledgebases/{kbId}/documents/{docId}knowledgeBase.updateDocument()Roles: owner, admin, developer
Scope: knowledgebases.write
documentTag Integrates with hybrid retrieval, knowledge graph, and the Wiki conflict workbench
Transparent Chunk Review and Online Hot Fixing/console/.../kb-kb_enterprise_support/doc-{docId}GET /v1/knowledgebases/{kbId}/documents/{docId}/chunks
PATCH .../chunks/{chunkId}
knowledgeBase.listChunks()
knowledgeBase.updateChunk()
Roles: owner, admin, developer (read-only members can view only)
Scope: query knowledgebases.read; update knowledgebases.write
After editing a chunk, the vector is recomputed and the related advanced knowledge index may be marked stale (chunk_changed)
Import personal Yuque URLsDocument page "Import Yuque URL"POST /v1/knowledgebases/{kbId}/documents/yuqueknowledgeBase.createYuqueDocuments()Requires the current user session or JWT, Yuque OAuth authorization, and knowledgebases.writeSee external data source integration and Yuque incremental sync
Sync a team data sourceData sources tabPOST /v1/knowledgebases/{kbId}/datasources/{id}/syncsknowledgeBase.createDataSourceSync()Scope: knowledgebases.writeConfigure the team data source and its credentials; see external data source integration and Yuque incremental sync

Feature Details and Operation Guide

Step One: Upload Enterprise Knowledge Documents and Configure Chunking Parameters

In the benchmark knowledge base kb_enterprise_support, we ingested 4 core production documents. Here are the detailed steps:

  1. On the knowledge base Documents tab, click Upload documents in the upper-right corner.

  2. Drag and drop or select local files (up to 50 files per upload):

    • Agentbase企业架构白皮书.md

    • 智能客服SOP标准作业手册.md

    • OceanBase分布式事务与向量索引排障指南.md

    • AI网关模型路由与限流计费规范.md

  3. In the upload configuration panel, set parsing and chunking parameters (you can keep the knowledge base defaults or customize them for the current batch):

Parameter NameField IdentifierRecommended ConfigurationValue Range and Description
Start ingestion immediatelyConsole switchEnable as neededThe Console first creates the documents, then creates a separate ingestion for each document when enabled. Document creation returns 201; ingestion acceptance returns 202.
Document parserparserConfig.documentParserSelect for the document typetext / vlm / mineru; vlm requires an available vision model, while mineru requires an available parser service.
Chunk parserparserIdSelect for the file typeOptions vary by file format and include general / title / smart / regex / faq / sheet; smart can select sheet for spreadsheet-only input.
Target chunk tokensparserConfig.chunkTokenNumFor example, 512Applies to general / title / smart / regex; stay within the embedding model's input limit.
Chunk overlap (%)parserConfig.overlapPercentFor example, 10Applies to general.
Heading split levelsparserConfig.titleLevelFor example, 3Applies only to title.
Minimum chunk tokensparserConfig.minChunkTokensConfigure as neededApplies only to smart.

Image understanding is configured separately and has no toggle in the upload panel. If needed, configure its model and provider in Knowledge Base Settings.


Step Two: Manage Hierarchical Tags and Positioning Identifiers (documentTag & subProductId)

The console writes free-form hierarchical tags and structured locator values to the same documentTag list. Legacy metadata.tags still contributes to the directory projection but is read-only in the edit-tags dialog. Directory lists use directoryPath or directoryUnclassified.

1. Hierarchical Tags

  1. In the document list, click Edit tags to the right of the target document.

  2. Enter parent-child hierarchical tags separated by /, for example:

    • Add a tag to OceanBase分布式事务与向量索引排障指南.md: 技术支持/数据库/OceanBase

    • Add a tag to AI网关模型路由与限流计费规范.md: 技术支持/AI平台/AI网关

    • Add a tag to 智能客服SOP标准作业手册.md: 客服运营/标准作业/SOP

  3. Click the view toggle button in the upper-left corner of the document list to freely switch between the Folder Tree View and the Flat View.

2. Document Positioning Identifiers (documentTag and subProductId)

Starting from v2.10.0, every document can be bound with a structured positioning identifier documentTag and a sub-product code subProductId:

Set APPBASE_ENDPOINT to the target environment API endpoint (including /v1), and confirm the project and credentials belong to that environment. The examples use a versioned knowledge base: replace <DRAFT_VERSION_ID> with the same draft version ID throughout; if versioning is disabled, omit versionId from the request bodies and query.

  1. Submit locator and free-form hierarchical tags together (replaces the documentTag list)

(
  set +x
  set -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
: "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
builtin printf 'X-Appwrite-Key: %s\n' "$APPBASE_API_KEY" | curl --disable -X PATCH "${APPBASE_ENDPOINT}/knowledgebases/kb_enterprise_support/documents/doc_ob_troubleshooting" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  --header @- \
  -H "Content-Type: application/json" \
  -d '{
    "versionId": "<DRAFT_VERSION_ID>",
    "documentTag": "ob_12_4a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d,技术支持/数据库/OceanBase,核心排障",
    "subProductId": "ob"
  }'
)
  1. Trigger re-ingestion of a single document (returns HTTP 202 Accepted and the ingestion task $id)

(
  set +x
  set -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
: "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
builtin printf 'X-Appwrite-Key: %s\n' "$APPBASE_API_KEY" | curl --disable -i -X POST "${APPBASE_ENDPOINT}/knowledgebases/kb_enterprise_support/documents/doc_ob_troubleshooting/ingestions" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  --header @- \
  -H "Content-Type: application/json" \
  -d '{"versionId":"<DRAFT_VERSION_ID>"}'
)
  1. Poll the ingestion task status until it becomes ready

Replace <INGESTION_ID_FROM_RESPONSE> with $id from this response and query the same draft version; omitting versionId selects the current version and may not find the draft task. Stop polling and investigate if the status is failed or cancelled.

(
  set +x
  set -o pipefail
  : "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
: "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
builtin printf 'X-Appwrite-Key: %s\n' "$APPBASE_API_KEY" | curl --disable -G "${APPBASE_ENDPOINT}/knowledgebases/kb_enterprise_support/documents/doc_ob_troubleshooting/ingestions/<INGESTION_ID_FROM_RESPONSE>" \
  --data-urlencode "versionId=<DRAFT_VERSION_ID>" \
  -H "X-Appwrite-Project: enterprise-ai-hub-210" \
  --header @-
)

Step Three: Review and Edit Chunks and Rebuild the Entire Knowledge Base (Reindex)

  1. View the chunk list: In the document list, confirm that the target document has ready status, then click the document name to enter the Document Details page.

  2. Inspect and edit chunks online:

    • The page displays all split chunk cards in order, each showing the chunk index, token count, section path, and body content.

    • If image understanding is enabled, chunks containing images display the Image Caption / VLM Summary generated by the configured vision model below the chunk.

    • Click Edit in the upper-right corner of any chunk card to directly fix typos, add key context, or update proprietary parameters; after saving the content, the system uses the configured embedding model to recompute the chunk vector.

  3. Rebuild the entire knowledge base (Reindex):

    • After you modify the global default chunking strategy in the knowledge base settings (e.g., changing chunkTokenNum from 512 to 800), return to the top of the Documents tab and click Reindex; the system will batch-dispatch asynchronous re-ingestion tasks for all documents in the knowledge base.


External data sources and Yuque sync

A team Yuque data source uses team connection credentials; personal Yuque authorization is a separate import path. For setup, permissions, and sync scope, see External data sources and Yuque.

After starting a sync, check its record on the data source page. queued means waiting, running means processing, ready means the sync job has finished, and failed means the job failed. Its stats can include created, updated, skipped, disabled, and failed counts. A completed sync does not by itself establish document readiness: check each document's ingestion status before testing retrieval.

Uploading and ingesting documents

  1. Go to the knowledge base → Upload documents: the file picker supportsup to 50 files per batch; files are uploaded to the knowledge base's dedicated storage and document records are created. By default, ingestion starts immediately after upload, but automatic ingestion can also be disabled.

  2. Wait until the document status becomes Ready before opening the document details to view chunks. If a document is not ready, first check its status and processing result; do not treat documents whose ingestion has not completed as retrievable content.

  3. If you need to adjust the parser or chunking parameters, modify them in the document's configuration first, then re-ingest the affected documents. Configuration changes are not written back to already-ingested old documents.

  4. Reindex re-runs ingestion and indexing for all documents in the knowledge base; when only a few documents need updating, prefer per-document re-ingestion.

  5. URL / external data sources: connected via Data sources; the currently built-in data source type is Yuque (including OAuth-authorized import and URL import), with support for viewing sync runs and managing data sources.

Document and chunk management

FAQ (Frequently Asked Questions and Troubleshooting)

Q1: A document's ingestion status becomes failed with an error saying the embedding input exceeds the limit—how do I troubleshoot and fix it?

Q2: For a scanned PDF or a document containing many architecture diagrams, what should I do if the text parser cannot extract text?

Q3: What format validation requirements does documentTag have? What happens if the format is invalid?

Q4: How is an uploaded Excel spreadsheet split and vectorized?