You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] StructuredDocument returns duplicate chunk IDs across input documents #1551
StructuredDocument.chunk() keeps each chunker's document-local ID when combining multiple documents. With two documents that each produce one token chunk, both output IDs are token-0, and both documents link to that same ID. Indexing the returned chunks by ID silently retains only one of them.
Reproduced on current main e9fc599f6059afcd88b34f2bfc50d0d1d6a0877c after building the unchanged @voltagent/rag package:
import{StructuredDocument}from"@voltagent/rag";constdoc=newStructuredDocument([{docId: "policy",text: "Refunds are available within thirty days."},{docId: "guide",text: "Contact support to request a refund."},]);const{ chunks }=doc.chunk({strategy: "token",maxTokens: 100});console.log(chunks.map(chunk=>chunk.id));// ["token-0", "token-0"]console.log(doc.getLinkGraph());// {policy:["token-0"],guide:["token-0"]}console.log(newMap(chunks.map(chunk=>[chunk.id,chunk])).size);// 1, expected 2
The chunks from distinct input documents should have distinct IDs, and each document link should refer to its own chunks. This is separate from #1526, which handles inherited-property names in link-graph keys, and from the individual chunker expansion fixes.
I am preparing a focused fix that scopes generated chunk IDs to the document and tests the public multi-document path. The reproducer uses the real exported package and default tokenizer; no model, external service, or vector database is involved. Prepared with AI assistance.
Activity