Repository navigation
Description
Describe the bug
NeuralChunker can return a detector segment with the same ID as an earlier token-fallback chunk. It assigns raw IDs before expansion, but retains those raw IDs for later segments that fit the budget. In one call, four distinct output chunks become only three distinct IDs.
Steps To Reproduce
Using the built public @voltagent/rag exports from current main:
import { NeuralChunker, whitespaceTokenizer } from "@voltagent/rag";
const chunks = await new NeuralChunker(whitespaceTokenizer).chunk("a b c d e f", {
detector: () => [1, 9],
maxTokens: 2,
});
console.log(chunks.map(({ id, content }) => ({ id, content })));
// neural-0: "a"
// neural-1: "b c"
// neural-2: "d e"
// neural-2: " f" <-- collides with the preceding chunkThe first detector segment fits, the middle segment expands into two token chunks, and the last segment fits but keeps raw index 2. The same collision occurs with the real default tiktoken tokenizer (its fallback content retains the leading spaces), and with a custom label.
I also passed the resulting chunks to the actual public InMemoryVectorAdapter.storeBatch, using each chunk ID as the key and a manually supplied [1] vector. count() is 3 for four output chunks, and the last neural-2 replaces the earlier one. This is a local adapter execution, not a claim about a remote vector database or model-derived embeddings.
Expected behavior
Each chunk produced by a single call should have a distinct ID in output order, using the existing label prefix. Contents, ranges, token budgets and metadata should remain unchanged. IDs need not be globally unique across separate calls or documents.
Packages
@voltagent/rag 1.0.2, @voltagent/core 2.11.1, exact source baseline e9fc599f6059afcd88b34f2bfc50d0d1d6a0877c. Node.js 24.13.0, pnpm 8.10.5, macOS arm64. Both packages were built through the repository's existing scripts with frozen dependencies.
Additional Context
I plan to assign the fitting segment's ID when it is appended to finalChunks, matching the existing fallback path, and add focused regressions for both tokenizers and labels. This is separate from #1465/#1466, which concern RecursiveChunker's sentence expansion and do not change NeuralChunker. All 104 current open PR heads were checked against the complete 103-PR file audit plus my three-file SDK PR #1517; none touches NeuralChunker. Focused all-state searches and its source history found no existing fix for this path.
The detector above is an explicitly configured deterministic boundary callback, as supported by the documented API; no neural model was run. Codex assisted with source discovery and executed four real public-API cases (both tokenizers, both label prefixes), all showing the collision and the 3/4 stored-record count. No tokenizer, chunker or vector-adapter implementation was mocked; only manual detector offsets and valid input vectors are supplied. No provider service, browser or production incident is claimed.
Activity