[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer by nishant94 · Pull Request #64667 · apache/doris

nishant94 · 2026-06-22T05:53:45Z

What problem does this PR solve?

Issue Number: #64646

Related PR: None

Problem Summary:
Doris has no Japanese-aware tokenizer for the inverted index. Japanese text has no spaces between words, so the existing parsers can't segment it and MATCH / MATCH_PHRASE on Japanese columns end up with poor recall and precision.

This PR adds a built-in kuromoji parser for Japanese, in the same style as the existing Chinese IK analyzer. It's opt-in per column:

 INDEX content_idx (`content`) USING INVERTED
 PROPERTIES("parser" = "kuromoji", "parser_mode" = "search");

After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms.

How it works:

Native C++ under be/src/storage/index/inverted/analyzer/kuromoji/, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix.
- The dictionary is a process-wide singleton loaded once from ${inverted_index_dict_path}/kuromoji. An offline converter compiles raw IPADIC into a compact C++ runtime format (double-array trie + cost matrix + char/unknown tables) at build time, so no binary blob is committed.
- search (default), normal and extended modes are supported. No thrift/proto changes — parser and mode ride as strings in the index properties.

Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses).

Release note

Support Japanese text tokenization in the inverted index via a new kuromoji parser (PROPERTIES("parser"="kuromoji")), with search/normal/extended modes.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)

  CREATE TABLE test_jp (
    id BIGINT,
    content TEXT,
    INDEX idx_content (content) USING INVERTED
      PROPERTIES("parser" = "kuromoji", "parser_mode" = "search")
  ) ENGINE=OLAP
  DUPLICATE KEY(id)
  DISTRIBUTED BY HASH(id) BUCKETS 1
  PROPERTIES("replication_num" = "1");

  INSERT INTO test_jp VALUES
    (1, '東京都に住んでいます'),
    (2, '日本語の形態素解析エンジン');

  -- search-mode decompounding: 東京都 also matches 東京
  SELECT id FROM test_jp WHERE content MATCH '東京';          -- expect: 1
  SELECT id FROM test_jp WHERE content MATCH_PHRASE '形態素解析'; -- expect: 2

  -- inspect segmentation directly
  SELECT TOKENIZE('東京都に住んでいます', '"parser"="kuromoji","parser_mode"="search"');

Behavior changed:
- No.
- Yes. It adds a new opt-in kuromoji parser. Existing parsers and their output are unchanged; the new behavior only applies to indexes that explicitly set parser="kuromoji".
Does this need documentation?
- No.
- Yes. PR Link to Doris-Website.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-06-22T05:53:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

nishant94 · 2026-06-22T06:23:33Z

run buildall

yiguolei · 2026-06-22T06:27:35Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

nishant94 · 2026-06-22T06:31:13Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

@yiguolei The ICU Analyzer is not good as the Kuromoji. There is huge difference between icu and kuromoji when it comes to morphology of the Japanese words. So I think it worth it adding this new parser.

BiteTheDDDDt · 2026-06-22T07:02:18Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

nishant94 · 2026-06-22T07:29:43Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

This is original code but it is modeled on Apache Lucene's kuromoji.

hello-stephen · 2026-06-22T07:56:03Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

nishant94 · 2026-06-22T09:58:03Z

run buildall

hello-stephen · 2026-06-22T12:00:26Z

BE UT Coverage Report

Increment line coverage 82.40% (791/960) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.51% (21439/39327)
Line Coverage	38.17% (205347/537919)
Region Coverage	34.16% (161044/471416)
Branch Coverage	35.14% (70517/200651)

hello-stephen · 2026-06-22T15:49:21Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.50% (21433/39329)
Line Coverage	38.13% (205092/537920)
Region Coverage	34.11% (160793/471446)
Branch Coverage	35.11% (70468/200678)

hello-stephen · 2026-06-22T16:58:18Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.11% (28441/38375)
Line Coverage	58.02% (309954/534209)
Region Coverage	54.69% (258833/473301)
Branch Coverage	56.10% (112608/200725)

hello-stephen · 2026-06-22T17:05:28Z

FE Regression Coverage Report

Increment line coverage 66.67% (6/9) 🎉
Increment coverage report
Complete coverage report

nishant94 · 2026-06-23T06:16:44Z

run buildall

hello-stephen · 2026-06-23T11:38:51Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T11:55:51Z

FE Regression Coverage Report

Increment line coverage 35.29% (6/17) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T14:39:39Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.72% (21523/39332)
Line Coverage	38.18% (205493/538169)
Region Coverage	34.17% (161179/471738)
Branch Coverage	35.13% (70561/200832)

hello-stephen · 2026-06-23T14:44:04Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.19% (28466/38371)
Line Coverage	58.03% (310151/534436)
Region Coverage	54.77% (259367/473580)
Branch Coverage	56.13% (112755/200875)

nishant94 · 2026-06-24T05:18:18Z

run buildall

hello-stephen · 2026-06-24T06:46:43Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-24T10:23:58Z

BE UT Coverage Report

Increment line coverage 84.20% (842/1000) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.58% (21476/39348)
Line Coverage	38.09% (205045/538313)
Region Coverage	34.07% (160754/471838)
Branch Coverage	35.04% (70387/200890)

hello-stephen · 2026-06-24T14:28:27Z

BE Regression && UT Coverage Report

Increment line coverage 84.02% (468/557) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.22% (28491/38387)
Line Coverage	58.10% (310621/534591)
Region Coverage	55.02% (260603/473686)
Branch Coverage	56.29% (113100/200937)

hello-stephen · 2026-07-03T13:06:02Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

nishant94 · 2026-07-06T12:17:53Z

run buildall

nishant94 · 2026-07-06T12:25:54Z

/review

nishant94 · 2026-07-07T09:37:38Z

run buildall

airborne12 · 2026-07-14T02:48:39Z

/review

github-actions

Automated review result: request changes.

I reviewed the full PR diff and existing review threads, with subagent convergence after the final merged comment set. I did not run builds or tests per the review prompt.

Critical checkpoints:

Goal/coverage: the PR adds Kuromoji analyzer support, dictionary generation/packaging, FE validation, and tests, but several correctness and build/test issues remain.
Data correctness / FE-BE contract: analyzer-only Kuromoji configuration is accepted by FE but fails later because parser-mode defaults ignore the analyzer key.
Runtime correctness / lifecycle: malformed dictionary artifacts can still install mmap-backed pointers that later crash or read invalid metadata.
Search semantics: search/extended mode currently drops the original compound token instead of preserving the default Lucene/OpenSearch-compatible behavior.
Build/config: the new dictionary target depends on a newly staged thirdparty source, but the normal build wrapper can skip refreshing it from older caches.
Testing: the Japanese analyzer regression exercises real morphology, but the deterministic checks need to use the repository's generated-result regression format.
Concurrency/locking/persistence: no additional locking, transaction, or persistence issue was found in this static review.
User focus: .code-review.d0xn9t/review_focus.txt contained no additional focus points.

nishant94 · 2026-07-16T12:14:44Z

run buildall

Add a built-in `kuromoji` inverted-index parser that segments Japanese text into morphemes, mirroring the existing Chinese IK analyzer.

- Added `darts.h` to `.clang-format-ignore` and `.licenserc.yaml`. - Improved code formatting in various Kuromoji source files for better readability. - Updated tests files to include necessary headers.

…mposition - Added support for search mode in the Kuromoji Viterbi segmenter, applying penalties for long all-kanji and other tokens to enhance search recall. - Updated the KuromojiMode enumeration to reflect the new search and extended modes. - Modified the KuromojiTokenizer to utilize the new mode functionality. - Added unit tests to validate the behavior of the search mode, ensuring correct segmentation of compounds. - Updated NOTICE.txt to include Apache Lucene as a dependency for the kuromoji analyzer.

…wn words - Implemented functionality in the Kuromoji Viterbi segmenter to decompose unknown (out-of-vocabulary) words into per-character unigrams when in extended mode, aligning with Lucene's JapaneseTokenizer behavior. - Added unit tests to validate the correct segmentation of unknown words in both normal and extended modes, ensuring expected outputs for various input scenarios.

- Modified error messages to include 'kuromoji' parser in the parser mode validation. - Enhanced tests for the Japanese analyzer to assert expected tokenization results.

- Introduced a new configuration option `enable_kuromoji_analyzer` to toggle the Kuromoji analyzer functionality. - Updated unit tests to validate the behavior of the Kuromoji analyzer when enabled and disabled. - Modified tests to enable the Kuromoji analyzer for specific test cases.

- Updated the namespace for Kuromoji components from `doris::segment_v2::kuromoji` to `doris::segment_v2::inverted_index::kuromoji` across multiple files for better organization and clarity.

…ictionary

- Updated the CMake configuration to ensure the required Kuromoji dictionary files are present at build time, failing the build if any are missing. - Modified the KuromojiAnalyzer and KuromojiTokenizer to throw exceptions when the dictionary is not loaded, preventing silent fallbacks to per-codepoint tokenization. - Improved error handling and validation in the dictionary loading process to ensure robust operation. - Updated unit tests to validate the new behavior, ensuring that missing dictionaries trigger appropriate errors.

- Replaced the `ending_at` vector with `end_head` and `end_next` for better memory management and performance during node processing. - Updated node addition and traversal logic to utilize the new data structures, enhancing the segmenter's efficiency in handling word segmentation.

- Changed the values in the `unk.per_category[CAT_DEFAULT]` entry from `{5, 5, 4769, "unk-default"}` to `{2, 2, 4769, "unk-default"}` to correct the test setup.

- Modified CMake configuration to conditionally include the Kuromoji dictionary files only for non-test builds (MAKE_TEST=ON). - Adjusted the custom target for generating the Kuromoji dictionary to reflect the new conditional behavior, ensuring it remains a manual target during unit-test builds.

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

nishant94 requested review from BiteTheDDDDt, airborne12 and zclllyybb as code owners June 22, 2026 05:53

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from 389fcfb to b79db3c Compare June 22, 2026 09:57

morningman self-assigned this Jun 23, 2026

yiguolei reviewed Jun 24, 2026

View reviewed changes

Comment thread be/src/storage/index/inverted/analyzer/analyzer.cpp Outdated

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from db0ee69 to 06b4ef6 Compare June 24, 2026 03:59

nishant94 requested a review from yiguolei June 24, 2026 05:18

yiguolei reviewed Jun 26, 2026

View reviewed changes

Comment thread be/dict/kuromoji/README.md

nishant94 requested a review from yiguolei July 3, 2026 11:45

nishant94 requested review from csun5285 and eldenmoon as code owners July 6, 2026 10:06

github-actions Bot requested changes Jul 14, 2026

View reviewed changes

github-actions Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java Outdated

nishant94 added 18 commits July 16, 2026 17:46

[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer

424a1f0

Add a built-in `kuromoji` inverted-index parser that segments Japanese text into morphemes, mirroring the existing Chinese IK analyzer.

Update Kuromoji analyzer files and formatting

c57ac07

- Added `darts.h` to `.clang-format-ignore` and `.licenserc.yaml`. - Improved code formatting in various Kuromoji source files for better readability. - Updated tests files to include necessary headers.

Enhance Japanese analyzer tests

6af2bf1

- Modified error messages to include 'kuromoji' parser in the parser mode validation. - Enhanced tests for the Japanese analyzer to assert expected tokenization results.

fix indentation issues

7863da9

Refactor Kuromoji namespace to inverted_index

7b10a7b

- Updated the namespace for Kuromoji components from `doris::segment_v2::kuromoji` to `doris::segment_v2::inverted_index::kuromoji` across multiple files for better organization and clarity.

Update README.md to include Apache License information for Kuromoji d…

888e2a3

…ictionary

Fix comment formatting

13025d3

Fix test case for Kuromoji dictionary

ef86e70

- Changed the values in the `unk.per_category[CAT_DEFAULT]` entry from `{5, 5, 4769, "unk-default"}` to `{2, 2, 4769, "unk-default"}` to correct the test setup.

Enhance Kuromoji dictionary validation and error handling

1c29d68

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

Enhance inverted index parser mode handling

43f2880

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

Add mecab-ipadic staging in build script

468feb2

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

Refactor Japanese analyzer tests for clarity and consistency

4445259

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from c5b7a3d to 4445259 Compare July 16, 2026 12:17

Uh oh!

Conversation

nishant94 commented Jun 22, 2026

What problem does this PR solve?

Release note

Uh oh!

hello-stephen commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

yiguolei commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BiteTheDDDDt commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

FE UT Coverage Report

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

FE Regression Coverage Report

Uh oh!

nishant94 commented Jun 23, 2026

Uh oh!

hello-stephen commented Jun 23, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

nishant94 commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

hello-stephen commented Jul 3, 2026

FE UT Coverage Report

Uh oh!

nishant94 commented Jul 6, 2026

Uh oh!

nishant94 commented Jul 6, 2026

Uh oh!

nishant94 commented Jul 7, 2026

Uh oh!

airborne12 commented Jul 14, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nishant94 commented Jun 22, 2026 •

edited

Loading