Documents and Ingestion Management
Documents & Ingestion is the core production workshop where a knowledge base (such as kb_enterprise_support) transforms raw unstructured files into high-quality retrievable Chunks and vectors. Through the Documents tab on the knowledge base detail page, users can batch-upload or bind bucket files, flexibly configure document parsers (text / vlm / mineru) and file-appropriate chunk strategies (general / title / smart / regex / faq / sheet), manage hierarchical category tags and precise locator identifiers (documentTag, subProductId), track asynchronous ingestion jobs and document status, and review, edit, and verify image understanding results for generated chunks online.
The ingestion job and document have separate states. HTTP 202 means ingestion was accepted. Poll the job for queued, running, ready, failed, cancelling, or cancelled; the document can be processing. After success, also confirm the document is ready and its activeGeneration matches that job's generation.
Feature overview and core scenarios
High-quality RAG Q&A performance depends on "accurate parsing, reasonable chunking, and fine-grained tags." Agentbase provides fine-grained full-lifecycle control capabilities in the Documents and Ingestion module:
Dual-mode file access and batch upload:
Console batch direct upload: the file picker supports uploading up to 50 files per batch, automatically persisting raw files to the knowledge base's dedicated bucket (e.g.,
kb_enterprise_docs) and creating document records; by default, asynchronous ingestion is triggered immediately after upload.Storage file ID binding: supports referencing files already uploaded to the project bucket (Storage) via
fileIdto create documents and start ingestion through the API directly, avoiding repeated transfer of large files.
Three types of multimodal document parsers:
Supports ultra-fast parsing for plain text/Markdown (
text), vision model parsing for complex scanned documents and architecture diagrams (vlm, using the configured vision model), and deep layout restoration for complex two-column PDFs, cross-page tables, and mathematical formulas (mineru).Supports structured chunking of Excel (
.xlsx/.xls) by Sheet, and can be combined with Image Understanding to automatically extract semantic descriptions of embedded charts.
File-specific chunk strategies (Chunking Parsers):
Built-in
general(generic fixed-length sliding window),title(splitting by Markdown/document heading hierarchy),smart(intelligent semantic boundary chunking),regex(custom regular expression splitting),faq(Q&A pair structure splitting), andsheet(spreadsheet chunking). Available strategies depend on the file type; tune parameters such aschunkTokenNum,overlapPercent,titleLevel, andminChunkTokens.
Document tags and locator identifiers:
Supports slash-separated parent-child hierarchical tags (e.g.,
技术支持/数据库/OceanBase), providing a tag tree directory view and a flat list view.v2.10.0 adds locator identifiers
documentTagandsubProductId: they support structured locator and free-form tags for filtering by tag and sub-product in document lists, retrieval, and Wiki conflict lists.
Chunk-level transparent review and online hot-fixing:
Once document ingestion becomes
ready, you can view each chunk's text, Token length, and associated image understanding summaries on the document detail page, and edit/fix chunk text or delete noisy chunks online; after modification, the corresponding vectors are automatically recomputed.
A synthetic document is ready in a 2.12.0 test knowledge base; this example has no tag directory yet.
Core Concepts and Architecture
Parser and Chunker Core Concepts Table
| Component dimension | Available types | Applicable document scenarios and core logic |
|---|---|---|
Document Parser (documentParser) | text | Best suited for well-structured .md, .txt, .docx, .html and regular born-digital documents, with the fastest parsing speed and zero extra visual model overhead. |
Document Parser (documentParser) | vlm | Invokes the project's registered visual large model (such as qwen-vl-max under dashscope-prod) to recognize complex layouts, scanned documents, handwritten annotations, and architecture flowcharts page by page. |
Document Parser (documentParser) | mineru | Designed for academic papers, technical white papers, and complex financial statement PDFs, precisely recognizing multi-column layouts, cross-page tables, and LaTeX formulas and converting them into structured Markdown. |
Chunk Parser (parserId) | general | General sliding-window chunking. Splits by a fixed chunkTokenNum (default 512) and overlapPercent (default 10%), suitable for long texts without clear heading structure. |
Chunk Parser (parserId) | title | Splits chapters at the selected heading level (titleLevel, default 3, i.e., ###); a segment may include its current heading, but the full parent heading path is not guaranteed. |
Chunk Parser (parserId) | smart | Dynamically merges fragmented paragraphs and splits overly long paragraphs based on paragraph boundaries, semantic coherence, and the minimum chunk threshold (minChunkTokens, default 128). |
Chunk Parser (parserId) | regex | Splits documents by a user-specified regular expression (regexPattern), suitable for log archives, specific-format legal provisions, or standardized entry files. |
Chunk Parser (parserId) | faq | Reads questions and answers line by line: the first column becomes the chunk content and question, while the second is stored in metadata.answer. Put each pair in two columns on the same line; adjacent question and answer lines are not combined automatically. |
Feature Entrypoints and Permission Matrix
Console Entry Points, REST API, SDK Methods, and Permission Matrix
| Console entrypoint / feature module | Frontend route path | Corresponding REST API endpoint | Server SDK method (Node.js / Python) | Required RBAC role / API Key scope | Cross-module dependency |
|---|---|---|---|---|---|
Document List, Folder Tree, and documentTag Filtering | /console/project-default-enterprise-ai-hub-210/knowledge-base/kb-kb_enterprise_support | GET /v1/knowledgebases/{kbId}/documents | knowledgeBase.listDocuments() / list_documents() | Roles: owner, admin, developer, member (read-only) Scope: knowledgebases.read | Directory views use directoryPath / directoryUnclassified; the list also supports documentTag and subProductId filters |
| Batch upload documents | /console/.../kb-kb_enterprise_support (click "Upload documents") | POST /v1/knowledgebases/{kbId}/documents/batch | knowledgeBase.createDocumentsBatch() / create_documents_batch() | Roles: owner, admin, developerScope: knowledgebases.write | Upload files and batch-create document records first; creation returns HTTP 201, and ingestion is started separately as needed |
| Bind one storage bucket file | Register an existing file through the API | POST /v1/knowledgebases/{kbId}/documents | knowledgeBase.createDocument() / create_document() | Roles: owner, admin, developerScope: knowledgebases.write | Start ingestion separately as needed after creating the record |
| Query Asynchronous Ingestion Progress and Trigger Re-ingestion | Document list status badges and row action "Re-ingest" | POST /v1/knowledgebases/{kbId}/documents/{docId}/ingestionsGET .../ingestions/{ingestionId} | knowledgeBase.createIngestion()knowledgeBase.getIngestion() | Roles: owner, admin, developerScope: knowledgebases.write / read | Uses the knowledge base's configured embedding model with matching dimensions; vlm parsing or image understanding also requires a configured vision model (see knowledge base settings) |
| Edit Document Tags and Positioning Identifiers | Document row action "Edit tags" | PATCH /v1/knowledgebases/{kbId}/documents/{docId} | knowledgeBase.updateDocument() | Roles: owner, admin, developerScope: knowledgebases.write | documentTag Integrates with hybrid retrieval, knowledge graph, and the Wiki conflict workbench |
| Transparent Chunk Review and Online Hot Fixing | /console/.../kb-kb_enterprise_support/doc-{docId} | GET /v1/knowledgebases/{kbId}/documents/{docId}/chunksPATCH .../chunks/{chunkId} | knowledgeBase.listChunks()knowledgeBase.updateChunk() | Roles: owner, admin, developer (read-only members can view only) Scope: query knowledgebases.read; update knowledgebases.write | After editing a chunk, the vector is recomputed and the related advanced knowledge index may be marked stale (chunk_changed) |
| Import personal Yuque URLs | Document page "Import Yuque URL" | POST /v1/knowledgebases/{kbId}/documents/yuque | knowledgeBase.createYuqueDocuments() | Requires the current user session or JWT, Yuque OAuth authorization, and knowledgebases.write | See external data source integration and Yuque incremental sync |
| Sync a team data source | Data sources tab | POST /v1/knowledgebases/{kbId}/datasources/{id}/syncs | knowledgeBase.createDataSourceSync() | Scope: knowledgebases.write | Configure the team data source and its credentials; see external data source integration and Yuque incremental sync |
Feature Details and Operation Guide
Step One: Upload Enterprise Knowledge Documents and Configure Chunking Parameters
In the benchmark knowledge base kb_enterprise_support, we ingested 4 core production documents. Here are the detailed steps:
On the knowledge base Documents tab, click Upload documents in the upper-right corner.
Drag and drop or select local files (up to 50 files per upload):
Agentbase企业架构白皮书.md智能客服SOP标准作业手册.mdOceanBase分布式事务与向量索引排障指南.mdAI网关模型路由与限流计费规范.md
In the upload configuration panel, set parsing and chunking parameters (you can keep the knowledge base defaults or customize them for the current batch):
| Parameter Name | Field Identifier | Recommended Configuration | Value Range and Description |
|---|---|---|---|
| Start ingestion immediately | Console switch | Enable as needed | The Console first creates the documents, then creates a separate ingestion for each document when enabled. Document creation returns 201; ingestion acceptance returns 202. |
| Document parser | parserConfig.documentParser | Select for the document type | text / vlm / mineru; vlm requires an available vision model, while mineru requires an available parser service. |
| Chunk parser | parserId | Select for the file type | Options vary by file format and include general / title / smart / regex / faq / sheet; smart can select sheet for spreadsheet-only input. |
| Target chunk tokens | parserConfig.chunkTokenNum | For example, 512 | Applies to general / title / smart / regex; stay within the embedding model's input limit. |
| Chunk overlap (%) | parserConfig.overlapPercent | For example, 10 | Applies to general. |
| Heading split levels | parserConfig.titleLevel | For example, 3 | Applies only to title. |
| Minimum chunk tokens | parserConfig.minChunkTokens | Configure as needed | Applies only to smart. |
Image understanding is configured separately and has no toggle in the upload panel. If needed, configure its model and provider in Knowledge Base Settings.
Step Two: Manage Hierarchical Tags and Positioning Identifiers (documentTag & subProductId)
The console writes free-form hierarchical tags and structured locator values to the same documentTag list. Legacy metadata.tags still contributes to the directory projection but is read-only in the edit-tags dialog. Directory lists use directoryPath or directoryUnclassified.
1. Hierarchical Tags
In the document list, click Edit tags to the right of the target document.
Enter parent-child hierarchical tags separated by
/, for example:Add a tag to
OceanBase分布式事务与向量索引排障指南.md:技术支持/数据库/OceanBaseAdd a tag to
AI网关模型路由与限流计费规范.md:技术支持/AI平台/AI网关Add a tag to
智能客服SOP标准作业手册.md:客服运营/标准作业/SOP
Click the view toggle button in the upper-left corner of the document list to freely switch between the Folder Tree View and the Flat View.
2. Document Positioning Identifiers (documentTag and subProductId)
Starting from v2.10.0, every document can be bound with a structured positioning identifier documentTag and a sub-product code subProductId:
Tag format:
documentTagaccepts a free-form value or multiple non-empty tags separated by commas. A structured locator can use{subProductId}_{sectionCode}_{fileKey}; when a structured value such asob_12omits the last segment on write, the server generates a file key. Read the stored value before using it for precise lookup.Automatic derivation rule: On creation or update, if a non-empty
documentTagis submitted whilesubProductIdis omitted or cleared, the system can derive the sub-product code from the first segment of a valid structured locator; free-form tags do not guarantee such a code.Clearing and valid filter values: On update, passing
subProductId: ""without also submitting a non-emptydocumentTagdeletes the standalone field; however, reads and filtering may still fall back to the sub-product code parsed from an existing structured locator. To remove both sub-product associations, clear bothsubProductIdanddocumentTag; this also removes the document positioning identifier, so confirm first that callers no longer depend on that positioning value.Example: Updating a Document Positioning Identifier and Querying Ingestion Status via API: Updates replace the
documentTaglist, so read and retain any values you need first. With versioning active, writes require a DRAFTversionId; if you also passmetadata, it replaces the existing business metadata, so preserve the business fields you need.
Set APPBASE_ENDPOINT to the target environment API endpoint (including /v1), and confirm the project and credentials belong to that environment. The examples use a versioned knowledge base: replace <DRAFT_VERSION_ID> with the same draft version ID throughout; if versioning is disabled, omit versionId from the request bodies and query.
Submit locator and free-form hierarchical tags together (replaces the documentTag list)
(
set +x
set -o pipefail
: "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
: "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
builtin printf 'X-Appwrite-Key: %s\n' "$APPBASE_API_KEY" | curl --disable -X PATCH "${APPBASE_ENDPOINT}/knowledgebases/kb_enterprise_support/documents/doc_ob_troubleshooting" \
-H "X-Appwrite-Project: enterprise-ai-hub-210" \
--header @- \
-H "Content-Type: application/json" \
-d '{
"versionId": "<DRAFT_VERSION_ID>",
"documentTag": "ob_12_4a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d,技术支持/数据库/OceanBase,核心排障",
"subProductId": "ob"
}'
)Trigger re-ingestion of a single document (returns HTTP 202 Accepted and the ingestion task
$id)
(
set +x
set -o pipefail
: "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
: "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
builtin printf 'X-Appwrite-Key: %s\n' "$APPBASE_API_KEY" | curl --disable -i -X POST "${APPBASE_ENDPOINT}/knowledgebases/kb_enterprise_support/documents/doc_ob_troubleshooting/ingestions" \
-H "X-Appwrite-Project: enterprise-ai-hub-210" \
--header @- \
-H "Content-Type: application/json" \
-d '{"versionId":"<DRAFT_VERSION_ID>"}'
)Poll the ingestion task status until it becomes ready
Replace <INGESTION_ID_FROM_RESPONSE> with $id from this response and query the same draft version; omitting versionId selects the current version and may not find the draft task. Stop polling and investigate if the status is failed or cancelled.
(
set +x
set -o pipefail
: "${APPBASE_API_KEY:?Set APPBASE_API_KEY}"
: "${APPBASE_ENDPOINT:?Set APPBASE_ENDPOINT to the target API endpoint including /v1}"
APPBASE_ENDPOINT="${APPBASE_ENDPOINT%/}"
builtin printf 'X-Appwrite-Key: %s\n' "$APPBASE_API_KEY" | curl --disable -G "${APPBASE_ENDPOINT}/knowledgebases/kb_enterprise_support/documents/doc_ob_troubleshooting/ingestions/<INGESTION_ID_FROM_RESPONSE>" \
--data-urlencode "versionId=<DRAFT_VERSION_ID>" \
-H "X-Appwrite-Project: enterprise-ai-hub-210" \
--header @-
)Step Three: Review and Edit Chunks and Rebuild the Entire Knowledge Base (Reindex)
View the chunk list: In the document list, confirm that the target document has
readystatus, then click the document name to enter the Document Details page.Inspect and edit chunks online:
The page displays all split chunk cards in order, each showing the chunk index, token count, section path, and body content.
If image understanding is enabled, chunks containing images display the Image Caption / VLM Summary generated by the configured vision model below the chunk.
Click Edit in the upper-right corner of any chunk card to directly fix typos, add key context, or update proprietary parameters; after saving the content, the system uses the configured embedding model to recompute the chunk vector.
Rebuild the entire knowledge base (Reindex):
After you modify the global default chunking strategy in the knowledge base settings (e.g., changing
chunkTokenNumfrom512to800), return to the top of the Documents tab and click Reindex; the system will batch-dispatch asynchronous re-ingestion tasks for all documents in the knowledge base.
External data sources and Yuque sync
A team Yuque data source uses team connection credentials; personal Yuque authorization is a separate import path. For setup, permissions, and sync scope, see External data sources and Yuque.
After starting a sync, check its record on the data source page. queued means waiting, running means processing, ready means the sync job has finished, and failed means the job failed. Its stats can include created, updated, skipped, disabled, and failed counts. A completed sync does not by itself establish document readiness: check each document's ingestion status before testing retrieval.
Uploading and ingesting documents
Go to the knowledge base → Upload documents: the file picker supportsup to 50 files per batch; files are uploaded to the knowledge base's dedicated storage and document records are created. By default, ingestion starts immediately after upload, but automatic ingestion can also be disabled.
Wait until the document status becomes Ready before opening the document details to view chunks. If a document is not ready, first check its status and processing result; do not treat documents whose ingestion has not completed as retrievable content.
If you need to adjust the parser or chunking parameters, modify them in the document's configuration first, then re-ingest the affected documents. Configuration changes are not written back to already-ingested old documents.
Reindex re-runs ingestion and indexing for all documents in the knowledge base; when only a few documents need updating, prefer per-document re-ingestion.
URL / external data sources: connected via Data sources; the currently built-in data source type is Yuque (including OAuth-authorized import and URL import), with support for viewing sync runs and managing data sources.
Document and chunk management
The document list supports searching by name or tag, deleting documents, and opening document details; the details page organizes ingested content by chunk.
Hierarchical labels and structured locator values can both be written to the
documentTaglist. Legacymetadata.tagsalso contributes to the directory projection but is read-only in the edit-tags dialog. Query directories usingdirectoryPath/directoryUnclassified.Locator fields can be written when creating / updating documents; for format, derivation, and clearing boundaries, see the "Document locators" section on this page. After writing, you can filter documents in the document list and in retrieval by actual locator values (see Search test).
Chunks support editing and deletion; after enabling image understanding, the image understanding results of a chunk can be viewed in the document details.
The REST API supports
enabledto control whether an individual chunk participates in retrieval; disabling it does not delete it. Re-enabling is also subject to the document, ingestion generation, and version publication state. This describes the API; check the actual console and SDK version for their respective controls.After ingestion, use the REST API to check the pre-chunk parse artifact status and read the parsed Markdown when ready. Older documents may require re-ingestion; artifacts that are too large or unsupported cannot be downloaded directly.
Excel ingestion: spreadsheets are parsed and sliced by sheet; you can configureimage enhancement (image interpretation model, prompt, and concurrency 1–8) to improve ingestion quality for image-containing spreadsheets and documents.
Embedding input limit: the embedding model's maximum input Token is configured in the Provider form "Maximum input tokens" under Models (LLM Gateway); ingestion inputs exceeding the limit fail explicitly.
FAQ (Frequently Asked Questions and Troubleshooting)
Q1: A document's ingestion status becomes failed with an error saying the embedding input exceeds the limit—how do I troubleshoot and fix it?
Root cause analysis: When the Token length of a single chunk exceeds the configured
maxInputTokensof the knowledge base's embedding model, the ingestion task fails explicitly to prevent truncation and data loss.Resolution:
Go to Models (LLM Gateway) -> Model providers and check the embedding model's maximum input Token configuration against the provider's actual limits;
Click Parser Config to the right of the failed document, reduce
chunkTokenNum(e.g.,512) or change the chunker fromregextosmart/title, then click Re-ingest.
Q2: For a scanned PDF or a document containing many architecture diagrams, what should I do if the text parser cannot extract text?
Root cause analysis: The
textparser only extracts the document's embedded native text streams; it does not perform visual OCR or topology understanding on rasterized images or scanned pages.Resolution:
Non-plain-text files normally default to
vlm; explicit document or knowledge base settings and file-format normalization can change the effective parser. Check the actual configuration first.For
vlm, configure an available vision model. Formineru, make sure an available MinerU parser service is configured. Change the parser and re-run ingestion.Image understanding is separate. If needed, enable it and configure its model and provider in Knowledge Base Settings.
Q3: What format validation requirements does documentTag have? What happens if the format is invalid?
Root cause analysis:
documentTagaccepts free-form values or multiple non-empty comma-separated tags. The structured locator{subProductId}_{sectionCode}_{fileKey}can support derived behavior such as sub-product inference; its final key is 32 hexadecimal characters. A non-empty standalonesubProductIdis limited to 1–64 letters or digits without hyphens, underscores, or spaces.Resolution: Avoid empty tags in a comma-separated list; read the stored value after writing and use it in later filters. Hierarchical labels and locator values can both be stored in
documentTag; legacymetadata.tagsis read-only in the edit-tags dialog.
Q4: How is an uploaded Excel spreadsheet split and vectorized?
Root cause analysis: Spreadsheet data has rigid row-column structural associations; splitting it with an ordinary plain-text sliding window would cause data rows to become detached from the header and lose semantics.
Resolution: Excel files (
.xlsx/.xls) are structurally sliced by the built-in spreadsheet parser per Sheet, automatically preserving the correspondence between the header and data rows in each chunk; if the spreadsheet contains embedded business screenshots and image understanding is enabled, the system calls the configured vision model at the configured concurrency to generate image captions and assigns them to the chunks of the corresponding Sheet.
Related Documentation
External data source integration and incremental sync with Yuque
GraphRAG, RAPTOR, and Advanced Knowledge Index Build Governance
Knowledge Graph Visualization Exploration and Community Reports
Knowledge Base Wiki Compilation, Conflict Adjudication, and Memory Export
Knowledge Base Immutable Version Management and Release Rollback
Project-level lexicons and retrieval/knowledge graph/agent collaborative governance
Knowledge Base Settings Hub, Default Policies, and Multimodal Image Understanding (VLM)
