Merge pull request 'feat(ru): drop abbreviations from the Russian noun list' (#6) from feature/drop-abbreviations into master

The list leaked abbreviations two ways: an OpenCorpora noun tagged Abbr without
Fixd (the declinable acronym ОГВЗ) slipped past the Abbr+Fixd seed filter, and the
РАН «(сокр.)» words (вуз, нэп, колхоз, гост, …) were kept on purpose by the old
policy.

Drop every abbreviation instead. A final pass in ru_stage2.py removes a word when
the orthographic note marks it «(сокр.)» (is_abbrev_note) or when every OpenCorpora
noun reading carries Abbr (oc_abbr_only — also the declinable огвз/заз the seed
filter missed). Homographs with a plain-noun reading survive (ваза «ВАЗ», рис).
Lexicalised words are rescued via manual_confirm.txt (колхоз, совхоз, рация,
спецназ, эсминец, дзот, дот, прораб, токамак, тэн, … and под — a real noun OC knows
only as ПОД); the bound prefix гос is vetoed in manual_reject.txt.

108 words removed from ru_scrabble / ru_erudit; en_sowpods unchanged. tools/README
documents the policy and that libmorph is required for a faithful rebuild.

sources/scrabble_ru/manual_confirm.txt

@@ -133,3 +133,31 @@
 флеш
 флешмобер
 шиноремонт
+военкор
+дзот
+дисбат
+дот
+завгар
+иняз
+каско
+колхоз
+линкор
+морпех
+научпоп
+прораб
+рация
+санбат
+совхоз
+сопромат
+спецназ
+спецхран
+старлей
+старпом
+твэл
+теормех
+токамак
+тэн
+чермет
+эсминец
+юннат
+под

sources/scrabble_ru/manual_reject.txt

@@ -0,0 +1,72 @@
+акцизный
+апикальный
+батальонный
+безумный
+благородный
+большой
+брюшнотифозный
+велярный
+верхоконный
+взрывной
+вольный
+головной
+горбатый
+городской
+гос
+грамотный
+гриппозный
+губной
+двугласный
+желтокожий
+живой
+заразный
+зубной
+иноверный
+кандальный
+каторжный
+клеймёный
+конный
+конюшенный
+лабиальный
+лысый
+любопытный
+маленький
+межзубный
+назальный
+несчастненький
+нёбный
+номерной
+носовой
+овинный
+одинокий
+отвальный
+параличный
+плитовой
+поддужный
+полубезработный
+полуротный
+припадочный
+сверхбольшой
+свободный
+сподручный
+старшенький
+стопорный
+стреловой
+судейский
+сыпнотифозный
+тифозный
+толкучий
+увулярный
+уголовный
+услужающий
+участвующий
+фабричный
+фланговый
+холерный
+чановой
+чесоточный
+чумной
+штрафованный
+щелевой
+щелинный
+эскадронный

tools/README.md

@@ -33,6 +33,7 @@ process's memory (dump it with `--dump`, query it with `--trace WORD`).
 | `orfo_dict_2025.txt` | ✓ | the pdftotext output — the parsed source of truth (the PDF binary is not needed to rebuild). |
 | `all.txt` | ✓ | Stage 1 base: every clean Cyrillic headword/variant; a plural headword with a singular is replaced by that singular. |
 | `manual_confirm.txt` | ✓ | hand-reviewed nouns from the undefined tail; the brain merges them into the result. |
+| `manual_reject.txt` | ✓ | hand-vetoed false nouns (substantivized adjectives a dictionary misreads as nouns, e.g. нёбный); subtracted from the result last. |
 | `scrabble.txt` | ✓ | **Stage 2 result**: common nouns, nominative singular (+ pluralia tantum), length 2–15 — the working dictionary. |
 | `undefined.txt` | — | the ambiguous tail; kept in memory, written only with `--dump`. |
 
@@ -60,7 +61,11 @@ g++ -std=c++17 -O2 tools/libmorph_check.cpp -lmorphrus -lmoonycode -o tools/libm
 ```
 
 If `tools/libmorph_check` is absent, Stage 2 still runs — it simply drops libmorph from
-the stack and reports `libmorph_helper=MISSING`.
+the stack and reports `libmorph_helper=MISSING`. **Build it before a release rebuild:** the
+committed word lists were produced *with* libmorph, and dropping it shifts the result by a few
+hundred words (libmorph-only nouns disappear, and adjectives/eponyms libmorph would veto —
+неперов, венерин — leak in through the orthographic-note path). A rebuild meant to refresh the
+committed lists is only faithful with the helper present.
 
 ## How to run
 
@@ -127,6 +132,10 @@ Each Stage-1 word (length 2–15) is routed by three sources, most authoritative
    травмпункт`, `регги и реггей`) and one is already a confirmed noun, the other is moved
    from review/undefined into the result as well, propagated transitively through chains.
    The plural-form variants the dictionaries already resolve never reach this step.
+5. **Manual overrides**: `manual_confirm.txt` adds maintainer-approved nouns from the
+   undefined tail (before variant rescue, so they can anchor it); `manual_reject.txt` is
+   subtracted **last**, removing false nouns a dictionary misreads — substantivized
+   adjectives with no nominal use (`нёбный`, `акцизный`, …) — overriding every path above.
 
 The nominative singular always comes from the dictionary that recognised the word, or from
 the orthographic `ед.` note — never from a predictor guess (libmorph and the predictor
@@ -155,34 +164,38 @@ passed and lemmas returned in UTF-8.
 
 ## Abbreviations, proper nouns & function words
 
-OpenCorpora lists indeclinable abbreviations as common nouns (`Abbr`+`Fixd`), so seeding the
+**Abbreviations are not Scrabble words and are dropped** (вуз, нэп, гост, жэк, спид, огвз,
-result with its whole noun lexicon leaked non-words like **ндс, ст, ср, кпд, чп** into
+обком, наркомат, роно, тв, …). A final pass in `build` removes every word matched by either
-`scrabble.txt`. They are filtered as follows (see `oc_noun_lemmas`, `oc_abbr_fixd_only`,
+signal, then re-admits the maintainer's rescues — see `is_abbrev_note`, `oc_abbr_only` and the
-`oc_function_word` and the Stage-2 loop in `ru_stage2.py`):
+abbreviation subtraction in `ru_stage2.py`:
 
-- An `Abbr`+`Fixd` noun is **never** admitted on OpenCorpora's word alone; the orthographic
+- **The orthographic note marks it `(сокр.)`** (`is_abbrev_note`): the РАН dictionary's own
-  dictionary decides via its own note (`classify`):
+  abbreviation mark, covering both the bare initialisms (вуз, нэп, гост, бтр, кпд) and the
-  - attested as a **lowercase** РАН headword whose note is a noun ⇒ **kept** — the
+  Soviet-era compounds (обком, райком, наркомат, госплан, …). This is the authoritative signal.
-    lexicalised abbreviations that are real words: **сельпо, под, ска, роно, врио, тв, фио,
+- **OpenCorpora reads it only as an abbreviation** (`oc_abbr_only`): *every* common-noun
-    суперэвм** (present in `all.txt`, `classify == noun`);
+  reading carries the `Abbr` grammeme. This catches acronyms РАН does not head at all — the
-  - everything else ⇒ **dropped**: bare letter-abbreviations not in `all.txt` (ндс, кпд, чп,
+  declinable огвз (a full `Abbr` case paradigm, no `Fixd`, which the `Abbr`+`Fixd` seed filter
-    гибдд, днк, …) and lowercase РАН headwords whose note is *not* a noun (гор, мин, про —
+  misses), заз, and the indeclinable роно, тв, фио, суперэвм. A homograph that *also* has a
-    «приставка»; об, по, со — «предлог»; сто — числительное).
+  plain-noun reading is **not** abbreviation-only and survives — ваза «ВАЗ», рис «РИС».
-- A word OpenCorpora also reads as a **function word** (`CONJ`/`PRCL`) is dropped even if the
-  note looks noun-like — **зато** (союз), and any like it.
-- A word whose OpenCorpora common-noun reading is an **inflected form** (its lemma is a
-  different word) is not a headword and stays out — **ан** (род. мн. от «ана»; the proper
-  noun «Ан», самолёт Антонова, is capitalised and never reaches `all.txt`).
 
-The discriminator is **not** the `(сокр.)` mark — сельпо carries it just like ндс:
+Still dropped by the seed/loop before this pass: a `CONJ`/`PRCL` **function word** even with a
+noun-like note (**зато**), and an OpenCorpora reading that is an **inflected form** of another
+lemma (**ан** = род. мн. от «ана»; the proper noun «Ан» is capitalised and never reaches
+`all.txt`). Proper nouns (`Name`/`Surn`/`Geox`/`Orgn`/…) are excluded from the seed outright.
 
-| word | in `all.txt` (lowercase РАН headword) | `classify` (РАН note) | OC `CONJ`/`PRCL` | verdict |
+**Maintainer overrides decide the borderline cases** (this is policy, not a heuristic):
-|------|:---:|:---:|:---:|---------|
+
-| сельпо, под, ска, роно, врио, тв, фио, суперэвм | yes | noun | no | **keep** |
+- `manual_confirm.txt` **rescues** lexicalised words that carry `(сокр.)` or that OpenCorpora
-| ндс, кпд, чп, гибдд, днк, … | no | — | no | drop (seed) |
+  only knows as an abbreviation but which read as ordinary nouns — **колхоз, совхоз, рация,
-| гор, мин, про, об, по, со, сто, прим | yes | not noun | — | drop (note) |
+  спецназ, эсминец, линкор, дзот, дот, прораб, токамак, тэн, …**, and **под** (a real noun,
-| зато | yes | — | **yes** | drop (function word) |
+  «под печи», that OpenCorpora knows only as the abbreviation ПОД). Edit this list to widen or
-| ан | yes | noun | yes | drop (OC lemma ≠ word) |
+  narrow which abbreviations stay.
+- `manual_reject.txt` **vetoes** the few non-noun leftovers that no signal catches — e.g. the
+  bound prefix **гос** («гос… — первая часть сложных слов», not a standalone word).
+
+> History note: an earlier policy *kept* lexicalised abbreviations (сельпо, вуз, нэп, …) and
+> used `(сокр.)` only as a non-discriminator. The current policy drops all abbreviations and
+> rescues a curated subset, so the dictionary holds words, not acronyms.
 
 ### Auditing a word and repeating this analysis
 

tools/ru_stage2.py

@@ -170,6 +170,38 @@ def oc_function_word(word):
     return False
 
 
+def oc_abbr_only(word):
+    """True when every common-noun reading OpenCorpora gives word is an abbreviation (carries
+    the Abbr grammeme on its paradigm). Generalises oc_abbr_fixd_only past the Fixd
+    (indeclinable) requirement, so it also catches a *declinable* acronym that OpenCorpora
+    inflects through a full Abbr case paradigm without Fixd — огвз, заз — as well as the
+    indeclinable роно, тв, фио. A word with any non-Abbr noun reading (the homographs ваза
+    «ВАЗ», рис «РИС», под «ПОД»/под печи) is not abbreviation-only and is left alone."""
+    parses = D.get_word_parses(word)
+    if not parses:
+        return False
+    gp, pt = D.get_paradigm, D.parse_tag_string
+    has_noun = has_plain = False
+    for pid, idx in parses:
+        suf, tag, pre = gp(pid, idx)
+        pos, gr = pt(tag)
+        if pos != "NOUN":
+            continue
+        _, tag0, _ = gp(pid, 0)
+        _, gr0 = pt(tag0)
+        if PROPER & gr or PROPER & gr0:
+            continue
+        has_noun = True
+        if "Abbr" not in gr0:
+            has_plain = True
+    return has_noun and not has_plain
+
+
+def is_abbrev_note(note):
+    """True when the orthographic note marks an abbreviation («сокр.», e.g. вуз, нэп, колхоз)."""
+    return bool(note) and "сокр" in note
+
+
 def libmorph_analyze(words):
     """Map each word to (known, noun_lemma, codes) per libmorph; noun_lemma is None when it
     is not a common noun there. Empty result if the helper binary is not built."""
@@ -353,7 +385,29 @@ def build():
                     fate[y] = f"scrabble: вариант от «{x}» (через «и»)"
                     changed = True
 
-    undefined = [w for w in amb if w not in scrabble]
+    # Abbreviations are not Scrabble words. Two independent signals catch them: the orthographic
+    # note marks «(сокр.: …)» (вуз, нэп, колхоз, гост, …); and OpenCorpora tags acronyms Abbr —
+    # both the indeclinable (роно, тв, фио) and the declinable ones the Abbr+Fixd seed filter
+    # misses (огвз, заз, гулаг). Drop every such word, except the lexicalised ones the maintainer
+    # rescues in manual_confirm.txt (колхоз, совхоз, рация, спецназ, … and под — a real noun
+    # OpenCorpora knows only as the abbreviation ПОД). The homographs that are real words with a
+    # mere abbreviation reading (ваза, рис) survive — oc_abbr_only requires *all* noun readings
+    # to be Abbr. Subtracted before the manual-reject veto.
+    confirm = {w for w in load(os.path.join(OUT_DIR, "manual_confirm.txt")) if cyr_ok(w)}
+    abbrev = {w for w in scrabble if w not in confirm and (is_abbrev_note(hmap.get(w)) or oc_abbr_only(w))}
+    scrabble -= abbrev
+    for w in abbrev:
+        fate[w] = "отброшено: аббревиатура (помета «сокр.» орфословаря или только Abbr-чтения в OpenCorpora)"
+
+    # Manual rejections: words the maintainer vetoed — substantivized-adjective false nouns
+    # a dictionary misreads as nouns (нёбный, акцизный, …). Subtracted last, so it overrides
+    # every admission path above (OC seed / libmorph / note / manual_confirm / variant).
+    reject = {w for w in load(os.path.join(OUT_DIR, "manual_reject.txt")) if cyr_ok(w)}
+    scrabble -= reject
+    for w in reject:
+        fate[w] = "отброшено: ручной запрет (manual_reject.txt)"
+
+    undefined = [w for w in amb if w not in scrabble and w not in reject]
     return {
         "oc": oc, "scrabble": scrabble, "undefined": undefined,
         "adjectives": adj, "verbs": verb, "singulars": ed_nouns,