Repository navigation
PP migration - #1544
Open
ampli wants to merge 108 commits into
Open
PP migration#1544ampli wants to merge 108 commits into
ampli wants to merge 108 commits into
Conversation
Add dictionary and tokenizer support for replacing the Wj CONTAINS_ONE postprocessing rule with ordinary grammar. The English dictionary now marks prep+wh paths that need a hidden helper token. The tokenizer creates an optional wjqprep helper alternative for those marked paths, and linkage presentation suppresses the helper word and helper links after postprocessing. This lets the dictionary require the Jw/JQ witness for rule 12 without exposing the helper in displayed linkages. Add corpus-knowledge.batch as a focused corpus for 4.0.knowledge rule replacement tests, starting with rule 12. Verified with: link-parser < ./data/en/corpus-knowledge.batch; link-parser < ./data/en/corpus-basic.batch; link-parser < ./data/en/corpus-fixes.batch; link-parser < ./data/en/corpus-fix-long.batch; link-parser < ./data/en/corpus-failures.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Replace the rule-specific WJ shadow tokenizer marker with a generic dictionary convention: INSERTL<token>+ on the previous word and INSERTR<token>+ on the following word request an optional helper token named <token>. Both markers point right, so they carry tokenizer data without forming a possible grammar link. Hide dictionary helper words and their links through a generic helper-label prefix instead of checking for a WJ-specific label. The English wjqprep helper now uses this convention. Verified with make, link-parser corpus-knowledge.batch (0 errors), corpus-basic.batch (88 errors), corpus-fixes.batch (367 errors), and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Add JQ-specific preposition continuations for PP rule 13 so JQ-bearing preposition branches keep only the allowed companions: MVp, Mj, MX#j, or Wj. Include both the direct Wj/Qp question continuation and the helper-token WJI path. Remove the corresponding CONTAINS_ONE postprocessing rule from 4.0.knowledge and add focused corpus-knowledge coverage for JQ preposition questions and relatives. Verified with link-parser corpus-knowledge.batch (0 errors), corpus-basic.batch (88 errors), corpus-fixes.batch (367 errors), corpus-fix-long.batch (9 errors), and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Split the generic preposition continuation into ordinary and relative forms so Mj and MX#j relative-preposition continuations are only licensed when the same preposition also has a Jw wh-object link. Keep the JQ question paths explicit through the existing companion continuation. Remove 4.0.knowledge CONTAINS_ONE rules 10 and 11 and add focused corpus-knowledge coverage for direct, comma-delimited, conjoined, and of-relative preposition examples. Validated with link-parser corpus-knowledge at 0 errors, corpus-basic at 88 errors, corpus-fixes at 367 errors, and corpus-fix-long at 9 errors. Co-authored-by: OpenAI Codex <codex@openai.com>
Add data/en/GRAMMAR-FIXES.md as maintainer-facing documentation for the English PP migration work currently present in this branch. Document the library-assisted helper-token mechanism and the status of rules 12, 13, and 10/11. Rule 12 has dictionary support but remains in 4.0.knowledge; rules 13 and 10/11 have been moved into dictionary grammar and removed from 4.0.knowledge. Co-authored-by: OpenAI Codex <codex@openai.com>
The Wj contains-one condition is now handled by the library-assisted dictionary helper-token path documented in GRAMMAR-FIXES.md, so remove the old postprocessing backstop from 4.0.knowledge. Before removal, link-parser -test=noPP:12,no-metric-extraction matched the accepted corpus expectations for corpus-knowledge, corpus-basic, corpus-fixes, and corpus-fix-long. After removal, corpus-knowledge also passes with ordinary parsing. Co-authored-by: OpenAI Codex <codex@openai.com>
Remove the naked to.r I*a+ fallback and the matching contains-one rule for incorrect use of 'to'6'. This ports the previously validated dictionary-side migration into the en-test grammar state. Validated with link-parser -test=noPP:6,no-metric-extraction before removal, then with link-parser -test=no-metric-extraction after removal on corpus-knowledge, corpus-basic, corpus-fixes, and corpus-fix-long. corpus-knowledge also passes with ordinary parsing. Co-authored-by: OpenAI Codex <codex@openai.com>
Require relative of-whom Jr paths to pass through OFJ, and expose postnominal B#j only with the same OFJ certificate. This removes the Incorrect relative15 and Incorrect relative16 contains-one PP rules. Validated with link-parser -test=noPP:15,noPP:16,no-metric-extraction before removal, then with link-parser -test=no-metric-extraction after removal on corpus-knowledge, corpus-basic, corpus-fixes, and corpus-fix-long. corpus-knowledge also passes with ordinary parsing. Co-authored-by: OpenAI Codex <codex@openai.com>
Remove the no-metric-extraction test flag from English grammar-fix verification commands. The en-test branch does not carry metric-extraction code, so PP-rule migration checks should use ordinary link-parser runs and noPP suppression only. Co-authored-by: OpenAI Codex <codex@openai.com>
Remove the BIh contains-one postprocessing rule after validating that ordinary en-test corpus runs remain unchanged. Keep the neighboring THb and BIq predicate rules in place for separate focused work. Also correct grammar-fix verification notes to use ordinary link-parser runs in this branch, which does not carry the PP suppression or metric-extraction test controls from viterbi-extraction. Co-authored-by: OpenAI Codex <codex@openai.com>
Drop the MX#a contains-one rule because it rejects documented good parenthetical adjective examples. The fixes corpus already marks these sentences as victims of the old adjective65 postprocessing rule. Validated with ordinary en-test link-parser runs: corpus-knowledge remains 0 errors, corpus-basic remains 88, corpus-fix-long remains 9, and corpus-fixes improves from 367 to 365 errors. Co-authored-by: OpenAI Codex <codex@openai.com>
Remove the CONTAINS_ONE pronoun66 PP rule by narrowing second-object dictionary paths. Standalone O*n+ second-object uses now go through <obj2-non-pronoun>, preserving noun-like second objects while excluding Ox pronoun objects; the unrelated dCO*n+ path is unchanged. Add focused corpus-knowledge coverage and document the migration in data/en/GRAMMAR-FIXES.md. Ordinary link-parser corpus checks stayed at 0 corpus-knowledge errors, 88 corpus-basic errors, 365 corpus-fixes errors, and 9 corpus-fix-long errors. Co-authored-by: OpenAI Codex <codex@openai.com>
Require postposed Ma adjective paths to carry a local complement license in the dictionary. POST_ADJ_LIC preserves predicative and conjoined adjective uses while limiting Ma and MJX postposed variants to adjective classes with an explicit license. Remove the adjective63 rule from 4.0.knowledge, keep the neighboring Mam adjective64 rule, and add focused corpus-knowledge coverage. Ordinary link-parser checks stayed at 0 corpus-knowledge errors, 88 corpus-basic errors, 365 corpus-fixes errors, and 9 corpus-fix-long errors. Co-authored-by: OpenAI Codex <codex@openai.com>
Remove seven CONTAINS_NONE checks whose absence leaves ordinary en-test corpus behavior unchanged: rules 69, 70, 74, 75, 76, 77, and 79. Document the removed rules in data/en/GRAMMAR-FIXES.md. Validated with link-parser on corpus-knowledge.batch (0 errors), corpus-basic.batch (88 errors), corpus-fixes.batch (365 errors), and corpus-fix-long.batch (9 errors). Co-authored-by: OpenAI Codex <codex@openai.com>
Rules 49 and 50 rejected valid more/less-than predicative adjective constructions, including corpus examples such as 'He is nothing less than inspired!' and 'He is more than capable!'. Remove both checks from 4.0.knowledge and document the cleanup in GRAMMAR-FIXES.md. Ordinary link-parser corpus checks now expect corpus-fixes.batch to report 363 errors; corpus-knowledge, corpus-basic, and corpus-fix-long stay at 0, 88, and 9 errors respectively. Co-authored-by: OpenAI Codex <codex@openai.com>
Rules 51 through 54 rejected valid reduced comparative adjuncts, including the corpus example 'they report less robust earnings than previously'. Remove the four checks from 4.0.knowledge and document the cleanup in GRAMMAR-FIXES.md. Ordinary link-parser corpus checks now expect corpus-fixes.batch to report 362 errors; corpus-knowledge, corpus-basic, and corpus-fix-long stay at 0, 88, and 9 errors respectively. Co-authored-by: OpenAI Codex <codex@openai.com>
Rules 60 and 61 no longer change the agreed English corpus behavior or focused comparative examples. Remove the broad THc/TOc domain checks from 4.0.knowledge and document the redundancy cleanup. Ordinary link-parser corpus checks remain at 0 errors for corpus-knowledge.batch, 88 for corpus-basic.batch, 362 for corpus-fixes.batch, and 9 for corpus-fix-long.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Rules 45 and 46 do not change the agreed English corpus behavior or focused comparative examples. Remove the broad MV#a/MV#i domain checks from 4.0.knowledge and document the redundancy cleanup. Ordinary link-parser corpus checks remain at 0 errors for corpus-knowledge.batch, 88 for corpus-basic.batch, 362 for corpus-fixes.batch, and 9 for corpus-fix-long.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Rules 55 and 57 do not change the agreed English corpus behavior or focused comparative examples, while neighboring rules 56, 58, and 59 still protect corpus-basic cases and remain active. Remove the redundant U#t and Sp#c domain checks from 4.0.knowledge and document the cleanup in GRAMMAR-FIXES.md. Ordinary link-parser corpus checks remain at 0, 88, 362, and 9 errors for the knowledge, basic, fixes, and fix-long corpora respectively. Co-authored-by: OpenAI Codex <codex@openai.com>
Rule 62 no longer changes the agreed English corpus behavior or focused infinitival-comparative examples. Remove the broad TOtc/TOt domain check from 4.0.knowledge and document the redundancy cleanup. Ordinary link-parser corpus checks remain at 0 errors for corpus-knowledge.batch, 88 for corpus-basic.batch, 362 for corpus-fixes.batch, and 9 for corpus-fix-long.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Add a concise deferred-candidates table to GRAMMAR-FIXES.md for PP rules that were tested for simple removal or identified as likely future dictionary migration work. The section records why rule 14, expletive-it rules, existential-there rules, BIq, and remaining comparative checks are still active instead of leaving those findings only in local notes. Co-authored-by: OpenAI Codex <codex@openai.com>
Record the concrete corpus effects and diagnostic conclusions for the remaining preposition, expletive-it, existential-there, predicate/question, and comparative PP migration candidates. These notes document why the tested families are not safe deletion candidates and what kind of dictionary split is still needed. Co-authored-by: OpenAI Codex <codex@openai.com>
Replace the subscripted wh-preposition object connector with a distinct JW connector family so ordinary J+ object paths cannot match wh-object continuations. Add explicit JW continuations for relative and wh-question preposition paths, preserving the wjqprep helper path for Wj licensing. Remove the rule 14 check from 4.0.knowledge, add focused corpus-knowledge coverage, and document the migration in GRAMMAR-FIXES.md. Verified with link-parser on corpus-knowledge.batch, corpus-basic.batch, corpus-fixes.batch, and corpus-fix-long.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Require postnominal comparative Mam paths to carry their local license in dictionary grammar. Reuse the MJX licensed conjunction path for conjoined comparatives where the license is supplied by one adjective rather than the conjunction. Remove the corresponding postprocessing rule from 4.0.knowledge, add corpus-knowledge coverage, regenerate 4.0.dict, and document the rule migration plus the affected English link types. Verified with top-five linkage comparison against master for representative Mam examples. Also verified link-parser corpus-knowledge.batch (0 errors), corpus-basic.batch (88 errors), corpus-fixes.batch (362 errors), corpus-fix-long.batch (8 errors), regenerated 4.0.dict consistency, and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Document that the removed rule 6 postprocessing check covered only unlicensed I#a paths. Other bad to.r or going-to parses through different I subscripts remain separate grammar issues, so they should not be treated as evidence that rule 6 migration is incomplete. Co-authored-by: OpenAI Codex <codex@openai.com>
Give temporal as.#while a dedicated verb-side MVSWH connector instead of ordinary MVs, so comparative EAy adjective paths can no longer attach to subordinate temporal as through broad @mv+ modifier slots. Remove the corresponding Bad comparative78 CONTAINS_NONE postprocessing rule and add corpus-knowledge coverage for valid comparative-as and temporal-as examples plus the rejected EAy/MVs bad path. Verified with link-parser corpus-knowledge.batch (0 errors), corpus-basic.batch (88 errors), corpus-fixes.batch (361 errors), corpus-fix-long.batch (8 errors), focused top-linkage comparison against master, and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Split comparative-clause subject continuations on than.e and as.e-c so singular and plural S*c paths require explicit dictionary-side antecedent certificates. Add CMPS, CMPP, and CMPX certificate connectors for singular/mass, plural, and agreement-neutral comparative antecedents, and remove the corresponding Bad comparative58 postprocessing rule. Add focused corpus-knowledge examples and document the new connector families and rule migration in GRAMMAR-FIXES.md. Verified with link-parser corpus-knowledge.batch (0 errors), corpus-basic.batch (88 errors), corpus-fixes.batch (361 errors), corpus-fix-long.batch (8 errors), and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Require comparative antecedent certificates on the than.e Cc/CV comparative-clause path, matching the existing rule 58 CMPS/CMPP/CMPX split. This prevents raw MVt + Cc/CV paths such as comparative adjective clauses from relying on postprocessing rule 59. Remove the corresponding CONTAINS_ONE rule from 4.0.knowledge and document rules 58 and 59 as one certificate-based comparative-clause migration. Validated with focused link-parser probes plus corpus-knowledge.batch, corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, m4 dictionary comparison, and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Add the CMPC internal certificate for clausal comparative continuations. Comparative modifier paths corresponding to the old rule 56 witnesses now supply CMPC, and as.e-c / than.e Cc-CV branches require it before they can form the comparative clause. Remove the corresponding CONTAINS_ONE rule from 4.0.knowledge and document the migration in GRAMMAR-FIXES.md with focused corpus coverage. Validated with focused link-parser probes plus corpus-knowledge.batch, corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, m4 dictionary comparison, and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Remove the CONTAINS_ONE postprocessing rule that required Qe domains to contain EEh or EAh. Tighten the ordinary-adverb dictionary expression instead: keep the direct EEh- and Qe+ how-question path, but remove the loose Qe+ alternative from the generic EE/EF adverb branch. This prevents lower-ranked parses such as how--AF--is with tall.e--Qe--is from becoming accepted after the PP rule is removed, while preserving accepted how-adverb and how-adjective question parses. Verified with link-parser corpus-knowledge.batch (0 errors), corpus-basic.batch (88 errors), corpus-fixes.batch (361 errors), corpus-fix-long.batch (8 errors), m4 regeneration consistency, focused Qe witness checks, and git diff --check. Co-authored-by: OpenAI Codex <codex@openai.com>
Add a dedicated ITAF certificate for accepted local filler-it comparative than paths that formerly used AFdi. The certificate preserves the lower copula disjunct cost without exposing an AF-prefixed connector that ordinary lower-subject clauses can satisfy. Document that the remaining infinitival preferred-cost delta belongs to the separate rule-23 TOi/TOIC audit. Verified focused Rule 30 accepted examples and the ordinary-subject negative, plus link-parser corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures count checks. Co-authored-by: OpenAI Codex <codex@openai.com>
Add TOi carrier paths for filler-it predicates so the rule 23 dictionary replacement preserves the old predicative and auxiliary costs without a crossing direct certificate link. Direct object-complement cases still use TOIC, while direct copula, inverted, modal, perfect, object-raising, and comparative examples use TTOI, ITOI, PPTOI, and PTOI carriers. Add focused corpus-knowledge examples and document the carrier link families and cost-preservation checks in GRAMMAR-FIXES.md. Verified focused first-linkage costs for direct, inverted, modal, perfect, object-raising, object-complement, and comparative filler-it TOi examples. Verified corpus-knowledge.batch remains at 0 errors and standard English corpus error counts remain unchanged for corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, and corpus-failures.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Add TSi carrier paths for filler-it predicates so the rule 21 dictionary replacement preserves the old predicative and auxiliary costs without relying on a crossing direct TSIC certificate. Direct object-complement cases can still use TSIC, while direct copula, inverted, modal, perfect, and object-raising examples use TTSI, ITSI, PPTSI, and PTSI carriers. Add focused corpus-knowledge examples and document the carrier link families and cost-preservation checks in GRAMMAR-FIXES.md. Verified focused first-linkage costs for direct, inverted, modal, perfect, object-raising, object-complement, and passive subjunctive-that TSi examples. Verified corpus-knowledge.batch remains at 0 errors and standard English corpus error counts remain unchanged for corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, and corpus-failures.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Add QIi carrier paths for filler-it predicates so the rule 22 dictionary replacement preserves the old predicative and auxiliary costs. Direct object-complement cases can still use QIIC, while direct copula, inverted, modal, perfect, and object-raising examples use TQII, IQII, PPQII, and PQII carriers. Add focused corpus-knowledge examples and document the carrier link families and cost-preservation checks in GRAMMAR-FIXES.md. Verified focused first-linkage costs for direct, inverted, modal, perfect, object-complement, and object-raising QIi examples. Verified corpus-knowledge.batch remains at 0 errors and standard English corpus error counts remain unchanged for corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, and corpus-failures.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Add Ci carrier paths for copular, modal, perfect, and object-raising filler-it finite-clause complements so the dictionary replacement preserves the old predicative-link costs instead of relying on crossing direct CIIC evidence. Keep local CIIC for object-complement cases, add PPCII for perfect auxiliary paths, and document focused cost-preservation checks for direct, inverted, auxiliary, object-complement, and object-raising examples. Verified with focused Rule 24 accepted-linkage checks and link-parser runs over corpus-knowledge.batch, corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, and corpus-failures.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Keep the replacement JW connector on the to.r wh-preposition path in the same cost position as the old optional J connector. This preserves the preferred cost for To what do you owe your success? while retaining the rule 12 dictionary-side witness split. Document the focused cost-preservation check for representative fronted-preposition questions and relatives. Verified with focused Rule 12 accepted-linkage checks and link-parser runs over corpus-knowledge.batch, corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, and corpus-failures.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Allow wh/degree B extraction noun branches to expose a local TOn link to infinitival to. This keeps the broad to.r I*a fallback removed while restoring the pre-migration preferred cost for wh-object infinitives such as which book/books to read and how much water to drink. Updated GRAMMAR-FIXES.md and corpus-knowledge coverage for the cost-preserving path. Verified focused accepted-linkage costs against the pre-migration behavior and checked link-parser corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures error counts. Co-authored-by: OpenAI Codex <codex@openai.com>
Give the certified `so much that ...` replacement branch the inherited result-modifier cost that the old `much` path paid through `[[MVa-]]`. This keeps the local RTHAT certificate while restoring the preferred cost for the focused `so much that` result-clause example. Verified focused Rule 67 accepted and rejected examples, plus the standard English corpus checks. Co-authored-by: OpenAI Codex <codex@openai.com>
Add the missing DWHs extraction branch to way.n so wh-extraction questions such as `Which way did they go?` keep the certified DWH/B path and the previous preferred displayed cost after the PP-rule migration. Verified focused Qd/MX examples and the standard English corpus checks. Co-authored-by: OpenAI Codex <codex@openai.com>
Add a narrow ECQ/EEQ certificate chain for how much more adverb questions so the direct Qe linkage preserves the pre-migration zero-cost path without restoring the broad ordinary-adverb Qe fallback. Verified focused Rule 19 positives and negatives, corpus-knowledge.batch, corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, and corpus-failures.batch. Co-authored-by: OpenAI Codex <codex@openai.com>
Apply the old plural existential inversion branch cost to the Qd/THRP path for are.v, restoring Are there dogs in the park? to the pre-migration first-linkage cost while leaving the singular THRS question path unchanged. Document the late cost-preservation audit result in GRAMMAR-FIXES.md. Verified with focused existential-there positives and negatives, plus link-parser error-count checks for corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Keep the migrated PVTHB passive certificate path at the same disjunct cost as the old Pvf passive path for THb predicate-complement linkages. In particular, `An allegation was made that he did it.` again sorts at the old first cost while direct and modal THb examples stay unchanged. Verified focused zero-null THb positives and negatives, and verified error-counts for link-parser runs over corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Expose the plural comparative antecedent certificate from the specialized currency object branch, matching ordinary plural noun behavior for comparative auxiliary clauses. This restores `I earn as many dollars as John does.` to the comparative as.e-c first linkage and old first cost. Verified focused zero-null Rule 48 positives and the adjective negative, and verified error-counts for link-parser runs over corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Add narrow filler-it BIh predicate paths for direct, inverted, modal, and perfect auxiliary forms without restoring the old broad BI+ copular branch for ordinary subjects. The new IBIH and PPBIH carriers preserve the old preferred costs for examples such as 'It was as if he knew.', modal/perfect variants, and inverted 'Was it as if he knew?', while ordinary-subject as-if predicates continue to use the previous adverbial modifier analysis. Verified focused Rule 41 positive and negative zero-null examples, and verified no error-count changes in link-parser runs over corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Add narrow filler-it QIi paths for the idiomatic matter predicate. Direct matters uses the filler-it subject witness, and modal, inverted, and contracted negative do-support paths use IQII to carry that witness to lower matter without restoring bare ordinary-subject QIi+ on the predicate. This restores the old first costs for 'It matters what Ted does.', 'It doesn't matter what Ted does.', 'It may matter what Ted does.', and inverted do-support variants. Ordinary-subject controls such as 'Joe doesn't matter what Ted does.' remain rejected. Verified focused positive and negative zero-null examples, and verified no error-count changes in link-parser runs over corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Restore the classic Qd,MX contains-none rule because the directive-opener dictionary rewrite is not an exact replacement for the full postprocessing check. The dictionary tightening still handles capitalized no-comma opener paths, but PP remains authoritative for other direct-question Qd+MX domains such as noun subjects with postnominal wh modifiers. Add focused corpus coverage for the noun-subject overgeneration and update GRAMMAR-FIXES.md to document the partial migration status and current validation counts. Verified focused Qd/MX positives and negatives, verified the filler-it matter QIi costs still sort first at DIS=1.50, and verified no error-count changes in link-parser runs over corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Adjust the THR agreement carrier branches so the dictionary replacement for the old existential-there PP checks preserves the inherited preferred costs for going-to-be and likely-to-be chains. PGTHR now offsets the certified final be no-wall cost, and PATHR carries the remaining predicative-adjective cost delta from the old been-Pa path. Document the cost-preservation details in GRAMMAR-FIXES.md. Verified focused existential-there examples against master/pre-migration first costs, including singular/plural/uncountable going-to-be paths and likely/unlikely-to-be paths. Verified link-parser corpus-knowledge.batch, corpus-basic.batch, corpus-fixes.batch, corpus-fix-long.batch, and corpus-failures.batch kept their expected error counts. Co-authored-by: OpenAI Codex <codex@openai.com>
Record the accepted-linkage audit result for Rule 47. The migrated CMPO path intentionally makes "I did as much as he did" prefer the comparative as.e-c object-clause analysis over master's less specific temporal as.#while analysis, while the ordinary "the same as it did" control keeps its first cost and the lexical-verb object-clause example remains new coverage. Verified focused link-parser comparisons against master for "I did as much as he did", "The coffee tastes the same as it did last year", and "I earned as much as John earned"; verified the focused adjective negative remains rejected. Co-authored-by: OpenAI Codex <codex@openai.com>
Restore the inherited [Ma-] cost on MJX-licensed conjoined postposed-adjective paths. The migrated Rule 63 structure still uses MJXl/MJXr to prove that one conjoined adjective carries the required complement license, but the conjunction-side Ma anchor now pays the old postnominal-adjective cost instead of bypassing it. Focused checks verify that Many Democrats unhappy about the economy but doubtful that Clinton can be elected probably will not vote at all sorts at DIS=3.00 with but.j-m cost 1.000, while unlicensed postposed-adjective negatives still reject. Corpus error-count checks were stable for corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Add a PVTHI carrier from filler-it be auxiliaries to passive participles so passive THi predicates no longer fall back to direct THIC evidence and inherited wall/passive costs remain intact. Cover ordinary passive verbs and the taken-for-granted Vtg path. Update GRAMMAR-FIXES.md with PVTHI and focused passive cost checks. Verified focused link-parser examples for believed/taken passive filler-it forms and ordinary-subject negatives. Verified error-count stability for corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Limit CMPS and CMPP on noun-main paths to subject/object roles, so an ordinary prepositional object cannot license a comparative subject clause merely by appearing before than or as. This blocks the party --CMPS-- than path while keeping subject/object comparative antecedents available. Update GRAMMAR-FIXES.md to describe the narrower certificate scope and the passive/expletive analysis retained for the singular More people ... than was expected case. Verified focused link-parser examples for Rules 58 and 59 positives, number-mismatch negatives, and the More people ... than was expected ranking case. Verified error-count stability for corpus-knowledge, corpus-basic, corpus-fixes, corpus-fix-long, and corpus-failures. Co-authored-by: OpenAI Codex <codex@openai.com>
Add as-specific CMPOA and CMPOPA certificates so comparative object nouns can license as.e-c object clauses through an intervening PP modifier without crossing the main object link or losing the old MVp-preposition preference cost. This restores "I earn as much money in a month as John earns in a year." to the pre-migration first cost, keeps "I earn as much money as John earns." at its old first cost, and keeps the mixed "as much money than John earns" and plain-adjective comparative controls rejected. Validated focused link-parser probes and corpus counts: corpus-knowledge.batch 0 errors, corpus-basic.batch 86 errors with the intended Rule56 acceptance, corpus-fixes.batch 355 errors, corpus-fix-long.batch 8 errors, and corpus-failures.batch 1495 errors. Co-authored-by: OpenAI Codex <codex@openai.com>
Show OXi and TOCL as separate edges from want in the cleft-object reference sketch and Rule 31 explanation, rather than placing TOCL on it. Verified the final sketches against dictionary expressions and both accepted link-parser linkages for "I want it to be John who stole the priceless documents." after an Autotools build, with zero null words, a linkage limit of 10000, and link/disjunct output. No dictionary or parser behavior changes are included. Co-authored-by: OpenAI Codex <codex@openai.com>
Move ITAF into the added-family summary and add a connector-reference entry covering its endpoints, local filler-it restriction, separate uppercase family, and inherited lower-copula cost. Cross-reference Rule 30 in both directions while retaining its historical explanation. Verified the three schematic edges against dictionary expressions and an accepted link-parser linkage for the finite comparative example, with zero null words and a linkage limit of 10000. The lower is.v disjunct cost is 1.000 and the first linkage has DIS=1.05. The corresponding ordinary-subject negative has no complete zero-null linkage. No dictionary expressions or parser behavior are changed. Co-authored-by: OpenAI Codex <codex@openai.com>
Add a grouped connector-reference entry for the modal and perfect filler-subject carriers, including endpoints, inverted auxiliary alternatives, and reciprocal links to Rule 41. Verified every schematic edge against dictionary expressions and accepted link-parser output for "It may be as if he knew." and "It has been as if he knew." with zero null words and a linkage limit of 10000. Both first linkages have DIS=0.00. No grammar, costs, or parser behavior changed. Co-authored-by: OpenAI Codex <codex@openai.com>
Add a connector-reference entry explaining the adjective-to-as endpoint, the following Pa relation, and the distinction from ordinary MVzp. Cross-reference Rule 44 in both directions while retaining its historical analysis and the existing summary rows. Verified the final sketch against dictionary expressions and accepted link-parser output for "The claim, unclear as worded, deserves attention." with zero null words and a linkage limit of 10000; its first linkage has DIS=0.00. The negative "I am as intelligent as John does." has no complete zero-null linkage. No grammar, costs, or parser behavior changed. Co-authored-by: OpenAI Codex <codex@openai.com>
Add a grouped reference entry naming all 18 THR agreement families, their endpoints, propagation roles, and lexical restrictions. Distinguish surface connector direction from dependency arrows and link the entry to the existential-there migration discussion in both directions. Verified all 18 concrete families in six accepted singular, plural, and uncountable raising and going-to examples, plus a plural inversion example. Checked final schematic edges against dictionary expressions and link-parser link/disjunct output with zero null words and a linkage limit of 10000. Current probes do not assert historical cost parity. Only connector documentation is changed. Co-authored-by: OpenAI Codex <codex@openai.com>
Describe the considered six-family subscripted alternative and distinguish uppercase-family count from the state-specific dictionary alternatives. Explain the retained naming as protection against future broad matching, not a demonstrated grammatical minimum or a performance optimization. Identify possible safeguards for a future evaluation without implying that the alternative or its validation tools have been implemented. Link the grouped connector reference directly to this rationale. Checked the naming map and matching claims against the design discussion and connector-matching documentation. No grammar or linkage sketches change. Co-authored-by: OpenAI Codex <codex@openai.com>
Describe unmigrated constraints as classic-postprocessor checks and the domain-summary proposal as historical background. Clarify that the Qd/MX dictionary tightening is not a complete rule replacement. Remove stale per-stage corpus totals while retaining sentence-level verification and cost-preservation evidence. No dictionary, corpus, or parser behavior changes. Co-authored-by: OpenAI Codex <codex@openai.com>
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR separates the English PP-rule migration from #1541 so it can be reviewed and adopted without the Viterbi extractor. It preserves the original rule-by-rule commits, including the subsequent cost fixes, and includes the connector-documentation additions.
Most of the changes are under
data/en/. The only C changes provide theINSERTL/INSERTRhelper-token support needed by the rule-12 replacement. They add an optional tokenization path and hide the helper tokens and their incident links from the displayed linkage and the linkage API.The rules not fully migrated to the dictionary remain active in classic postprocessing: the cycle check, inversion rules 1, 2 and 7, the
Qd,MXcheck, and boundedsrule 78. The separate Viterbi changes in #1541 handle the cycle and inversion constraints directly during extraction. Boundedsremains enforced by PP, with feedback allowing the extractor to skip further candidates with the same violation.The migration rationale, connector descriptions, examples, and cost-preservation notes are in
data/en/GRAMMAR-FIXES.md. The newdata/en/corpus-knowledge.batchcontains focused positive and negative tests for the migrated rules.