OPENNLP-1893: Support Hunspell affix dictionaries for stemming - #1190
Open
krickert wants to merge 19 commits into
Open
OPENNLP-1893: Support Hunspell affix dictionaries for stemming#1190krickert wants to merge 19 commits into
krickert wants to merge 19 commits into
Conversation
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 21, 2026
mawiesne
added
java
krickert
marked this pull request as ready for review
July 24, 2026 11:22
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…ENNLP-1895 recorded Restate the map against apache main a864230, cut as 3.0.0-M5 on 2026-07-24. apache#1177 (OPENNLP-1870) merged upstream and moves into the merged box, apache#1190 and apache#1191 are marked ready for review, and OPENNLP-1895 (quantized embedding tables) joins the diagram in its own colour: filed in JIRA with the pull request deliberately held until apache#1165 and apache#1152 move. Statuses now carry the measured GitHub draft flag and how far each head sits behind main, which surfaces three things the old text did not: apache#1182 is a draft again, apache#1167 is based on main rather than on apache#1155 and carries the seam and isBlank commits as copies, and apache#1152 reports conflicts only because its apache-hosted sentencepiece base has diverged from the refreshed head.
krickert
force-pushed
the
OPENNLP-1893-hunspell
branch
from
July 24, 2026 19:27
8722725 to
65a1c31
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 27, 2026
…preview-docs, record OPENNLP-1897 The 2026-07-24 map refresh (PR-head rebase record, apache#1190/apache#1191 ready, morfologik-fsa, OPENNLP-1895) was committed directly on kristian-3.x-features and would have been discarded by the next regeneration; preview-docs is the durable home. Also adds OPENNLP-1897-term-vectors to the held-PR section and diagram, and moves the state line to 2026-07-26 (apache main unchanged since the M5 cut).
krickert
force-pushed
the
OPENNLP-1893-hunspell
branch
from
July 28, 2026 15:14
3a3f977 to
6f2d9d8
Compare
rzo1
requested changes
Aug 4, 2026
rzo1
left a comment
rzo1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the PR.
-
The PR description says "unsupported dictionary features fail loud at load time rather than degrading silently", but
parseAffixends indefault: i++; break;, which skips every directive it does not know without a word. The class javadoc and README say the opposite of the description, that rules using unsupported features "simply do not fire, so unsupported analyses are missed rather than invented". Only an unknownFLAGmode really fails loud. The gap matters because the silently skipped set includesICONV,OCONVandCOMPLEXPREFIXES, which change results rather than only reducing them, so a dictionary relying on them produces wrong stems with no signal. Please either fail loud on the directives that alter results and keep the silent skip for the cosmetic ones (REP,MAP,KEYand friends), or log them, and correct the description either way. -
Dictionary download tooling should be settled once for both this PR and #1191, not separately. I raised there that
install(URI, Path)fetches without verification and that I would rather dropdev/download-mecab-dictionary.shthan keep integrity checking in a script we do not ship. The same applies here, and this script is the weaker of the two:base="https://raw.githubusercontent.com/LibreOffice/dictionaries/master/${collection}"builds in a download location, pins it to a moving branch, and verifies nothing, while the mecab script at least accepts an expected SHA-256. Whatever we land on in #1191, one Java path with digest verification and no shell helper, please apply it to both so the two features do not ship with different policies.
Smaller things, none of them blocking:
load(InputStream, InputStream)callsreadAllBytes()on both streams with no ceiling, so a caller passing a network stream has no bound on what gets buffered. Unlike #1191 nothing here pre-sizes from a declared count, the affix rule count and the.dicentry count are both handled correctly, so this is the only allocation worth bounding. The.affcontent is also decoded twice, once as ASCII to findSETand once with the declared charset, so a load holds two full copies plus the line array before parsing starts.suffixesEndingWith(char)andprefixesStartingWith(char)go throughMap<Character, List<Affix>>, so each call boxes aCharacter. Same point I left on the trie in #1191.- The PR reads flags per code point, with an explicit comment about supplementary characters under
FLAG UTF-8, butAffixConditionmatches percharandbucketByBoundarykeys on a singlechar. An affix whose material starts or ends with a supplementary character buckets on half a surrogate pair. The two halves of the same change should agree. PART_CHECK_BUDGET = 2048silently truncates the compound search. It is documented on the private constant, but a caller cannot see that analyses were dropped. Please state it in the public class javadoc, and consider making it settable.lookuphands out the internal mutable lists, which does not quite match the "instances are immutable" claim.Morphemein #1191 copies defensively; same treatment would fit here.morphologyIndexis a heuristic and will cut an entry that happens to contain a two-letter run followed by a colon.unescapedSlashtreats\\/as an escaped slash, so a word ending in a literal backslash before its flag separator parses wrong.HunspellStemmeris not final whileHunspellDictionaryis.- This PR carries no
@sincetags and #1191 puts@since 3.0.0on every new type. Pick one for both.
…aries A clean-room reader for .dic and .aff files with PFX/SFX rules, strip strings, character-class conditions matched by a single scan, cross products, and char, long, and num flag modes. No dictionary data is bundled: users point at their own files, so dictionary licenses never attach to the jar. Unsupported affix features fail closed, missing analyses rather than inventing them. (cherry picked from commit 0ecc39c)
…lasses Suffix rules now carry the continuation flags declared on their affix text, and analysis undoes a stacked pair when the inner rule's classes allow the outer one, so derived-then-inflected forms reduce to their dictionary word. (cherry picked from commit b543dea)
…spell engine, precise javadoc
…-preserving download helper
…nd strip-only rules, and read the parser through the whitespace seam
…, and stem nothing from nothing Loading the Spanish dictionary of the LibreOffice collection, the same collection this module's README recommends, exposed three gaps against real data. Flags under FLAG UTF-8 are now one code point each instead of one UTF-16 unit, since that dictionary names prefix rules with supplementary characters that would otherwise split into two flags and abort the load; a variation selector after a flag character selects presentation, not identity, and is dropped, which the same file also relies on. A numeric or long flag run ends at the first space or tabulator, the separators the word-list format defines, so trailing morphological text without a tag no longer aborts the load; the morphology cut itself now splits on exactly those two separators, the set the reference implementation's hashmgr.cxx uses, which the javadoc previously claimed while scanning wider whitespace. Stemming the empty word answers the empty word instead of letting a strip-only rule conjure a stem from nothing. All four downloaded dictionaries of the collection, English, Spanish, Hungarian, and German, now load and stem; new tests pin the escaped slash, the multi-word entry with trailing tags, and each corrected behavior.
…able The published Hungarian dictionary flags all of its entries as numeric references into an AF alias table, so without alias support every entry loaded flagless and stemming answered the surface form unchanged. The affix parser now reads the AF table, the first line as the declared count and every further line as one flag run with trailing comments discarded, and a purely numeric flag field in the word list resolves as a 1-based reference into it, failing loud with the line and table size when the reference is out of range. Without an AF table numeric fields keep their FLAG num meaning. The Hungarian dictionary of the LibreOffice collection now stems inflected forms; remaining gaps there are compound territory, which is tracked separately.
…heir boundary character Undoing a suffix requires the word to end with the rule's affix material, so only rules whose material ends in the word's last character can ever apply, and likewise for prefixes and the first character. The dictionary now buckets its rules by that boundary character at load, and every scan in the stem path, including the twofold and cross-product inner scans, walks the one bucket plus the strip-only rules instead of the whole inventory. Measured on the LibreOffice dictionaries at 4,000 words each: English 553k to 1,024k words per second, Spanish 9.6k to 28.9k, German 132k to 287k.
When the affix analysis finds nothing and the affix file declares compounding, a word now splits into two listed parts that the COMPOUNDFLAG or the positional COMPOUNDBEGIN and COMPOUNDEND flags allow in their positions, honoring COMPOUNDMIN, with the parts reported left to right. Affix analyses keep precedence, listed words never decompose, and unflagged parts block a split. Against the published Hungarian dictionary the unlisted kutyahaz decomposes into its two nouns while listed compounds and inflected forms keep their regular analyses. Longer chains, syllable rules, and the compound-only flags stay unimplemented and simply leave such words unanalyzed.
…itioning A NEEDAFFIX (or PSEUDOROOT) entry is a virtual stem that exists only to be affixed, an ONLYINCOMPOUND entry appears only inside compounds, and a FORBIDDENWORD entry is listed to be blocked; none of them is a standalone analysis anymore, per homonym flag set, and an affix carrying NEEDAFFIX among its continuation classes yields no single-removal analysis while its twofold and cross-product removals stand, the other affix being exactly the further one required. A cross-product now also requires both removed affixes' flags in the same homonym's flag set, and CIRCUMFIX binds marked prefix and suffix halves to one another, so neither half analyzes alone and a marked half never combines with an unmarked affix. Decomposition grows from two verbatim parts to the compound machinery the published German dictionary actually uses: any number of parts under the positional COMPOUNDBEGIN/COMPOUNDMIDDLE/COMPOUNDEND flags and COMPOUNDWORDMAX, parts standing on an entry plus one affix with COMPOUNDPERMITFLAG required at internal boundaries and COMPOUNDFORBIDFLAG barring marked forms, zero and dash linking suffixes included, an uppercased retry for capitalized entries spelled lowercase inside a compound, and the CHECKCOMPOUNDDUP, CHECKCOMPOUNDCASE, and CHECKCOMPOUNDTRIPLE junction guards, case judged against the original surface. A listed forbidden word never decomposes, and a fixed part-licensing budget keeps adversarial input bounded, missing analyses rather than stalling. Abbildungsverzeichnis, Haustuer, and Kinderzimmer now decompose against de_DE_frami at 137k words/s single-threaded. An opt-in test class checks everyday morphology against downloaded dictionaries under -Dopennlp.hunspell.dict.dir; nothing is bundled.
Extend docbkx/stemmer.xml with the hunspell affix-stemmer section, wire the chapter into the manual, and add StemmerFactoryUsageExampleTest and HunspellManualExampleTest asserting the load-and-stem values the chapter prints. Point the dictionary README at the new manual example.
…, split null contracts, thread-safety annotations
…plete javadoc
- Fold the two per-kind bucketing loops in the HunspellDictionary constructor into a
single bucketByBoundary helper that takes the rule list, the kind, and the sink for
the rules with empty affix material.
- Fold collectSuffixedPartStem and collectPrefixedPartStem, which differed only in the
boundary they face, into one collectAffixedPartStem with a suffix marker and an
atEdge marker; document what atEdge means at each end.
- Extract a parseValue helper for the single-integer directives so COMPOUNDMIN and
COMPOUNDWORDMAX no longer share one case body that re-tests which directive it is.
- Extract PREFIX_TAG, SUFFIX_TAG, and NO_MATERIAL constants and use them at the affix
block header, the rule lines, and the strip and affix material checks.
- Give FORBIDDENWORD its own case in the flag directive switch instead of letting the
catch-all default assign it, and make that default throw for a directive listed on
the outer switch but not handled on the inner one.
- Add the missing javadoc on the AffixCondition and HunspellDictionary constructors,
the Affix record components, and the splitLines, splitOn, and split helpers.
- Convert the single-line accessor javadoc on the compounding and affix bucket getters
to the {@return ...} form, and replace the hand-written prose on
HunspellStemmerFactory.newStemmer with {@inheritdoc} plus the instancing note.
- Trim commentary that restates the code: the bucketing rationale duplicated in
HunspellStemmer, the LibreOffice Spanish anecdote on the code point flag reader, and
the sentence left dangling in testGermanCompoundsDecompose.
- Drop the defensive null and directory guards from the test helpers
writeAndLoadFixture and load, which no caller can trip, and document what the real
dictionary tests assert.
- Fold the repeated ByteArrayInputStream plumbing in HunspellStemmerTest into two load
overloads, one UTF-8 and one taking the charset the SET declaration test needs.
- Turn the four table-style stemming tests into parameterized tests over their word and
expected stem pairs, so a failing row names itself.
- Add testNullArgumentsAreRejected, pinning the exact IllegalArgumentException message
of every public entry point including the argument names the stream loader reports.
- Correct the stemmer manual: name the example files after the fixture the test loads
rather than en_US, and state that the printed stems are the fixture's, since which
stem a published dictionary yields is that dictionary's decision.
krickert
force-pushed
the
OPENNLP-1893-hunspell
branch
from
August 6, 2026 02:11
6f2d9d8 to
d8cc10e
Compare
Reject ICONV, OCONV, and COMPLEXPREFIXES at load time; keep skipping cosmetic tables such as REP. Copy lookup results defensively and document the compound search budget on HunspellStemmer.
Contributor
Author
Also in that commit: defensive copy from Stream size ceiling, char vs code-point consistency on affix material, and |
Reject affix and dictionary streams above MAX_STREAM_BYTES. Affix conditions and boundary bucketing use Unicode code points so supplementary characters agree with FLAG UTF-8. Document the ceiling in the stemmer chapter and pin both behaviors in tests.
Contributor
Author
|
Follow-up in 8cede5e.
Download tooling still deferred with #1191 (one Java digest path, no shell helper). |
Contributor
Author
|
Same download rework as #1191 in 0b27d5e: shared
The |
…ownload ceiling The 512 MiB download ceiling becomes a default that opennlp.download.max.bytes can raise at JVM startup; absent or invalid values fall back. Keeps the file identical to the copy in the MeCab PR.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a stemmer that reads standard Hunspell
.dic/.affdictionary pairs, built on theStemmerFactoryseam from OPENNLP-1883.Supported affix features: alias compression (AF), NEEDAFFIX, ONLYINCOMPOUND, FORBIDDENWORD, CIRCUMFIX, twofold suffix analysis through continuation classes, and compound word positioning including linking forms as used in German compounds. Parsing is regex-free; unsupported dictionary features fail loud at load time rather than degrading silently.
Dictionaries are user-supplied and nothing is bundled: the in-tree tests use a project-authored miniature dictionary, and
dev/README-hunspell-dictionaries.mddocuments acquiring published dictionaries together with their license files. The manual gains a Hunspell section whose example is asserted byHunspellManualExampleTest.Additionally verified against published dictionaries for English, Spanish, Hungarian, and German through the gated
HunspellRealDictionaryTest(runs only when-Dopennlp.hunspell.dict.diris set; skipped otherwise, and no dictionary data enters the tree).