Skip to content

[BUG] StructuredDocument returns duplicate chunk IDs across input documents #1551

Open

Description

@Elon-Jiang

What is broken

StructuredDocument.chunk() keeps each chunker's document-local ID when combining multiple documents. With two documents that each produce one token chunk, both output IDs are token-0, and both documents link to that same ID. Indexing the returned chunks by ID silently retains only one of them.

Reproduced on current main e9fc599f6059afcd88b34f2bfc50d0d1d6a0877c after building the unchanged @voltagent/rag package:

import { StructuredDocument } from "@voltagent/rag";
const doc = new StructuredDocument([
  { docId: "policy", text: "Refunds are available within thirty days." },
  { docId: "guide", text: "Contact support to request a refund." },
]);
const { chunks } = doc.chunk({ strategy: "token", maxTokens: 100 });
console.log(chunks.map(chunk => chunk.id)); // ["token-0", "token-0"]
console.log(doc.getLinkGraph()); // {policy:["token-0"],guide:["token-0"]}
console.log(new Map(chunks.map(chunk => [chunk.id, chunk])).size); // 1, expected 2

The chunks from distinct input documents should have distinct IDs, and each document link should refer to its own chunks. This is separate from #1526, which handles inherited-property names in link-graph keys, and from the individual chunker expansion fixes.

I am preparing a focused fix that scopes generated chunk IDs to the document and tests the public multi-document path. The reproducer uses the real exported package and default tokenizer; no model, external service, or vector database is involved. Prepared with AI assistance.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions