Merge pull request 'Restore lost Russian dict sources & drop OpenCorpora abbreviations' (#3) from feature/restore-russian-sources into master

.gitignore

@@ -2,15 +2,13 @@
 /dawg/
 /scrabble-dawg-*.tar.gz
 
-# Russian prep-pipeline intermediates (regenerated locally by tools/; only the curated
-# word lists in sources/scrabble_ru/ are committed).
-/sources/scrabble_ru/orfo_dict_2025.txt
-/sources/scrabble_ru/all.txt
+# Russian prep-pipeline debug dumps (regenerated locally by tools/, never committed).
+# The pdftotext source of truth (orfo_dict_2025.txt) and the Stage-1 base (all.txt) ARE
+# committed under sources/scrabble_ru/, and the source PDF under tools/ — see tools/README.md.
 /sources/scrabble_ru/undefined.txt
 /sources/scrabble_ru/adjectives.txt
 /sources/scrabble_ru/verbs.txt
 /sources/scrabble_ru/singulars.txt
 /sources/scrabble_ru/fate.tsv
 /tools/libmorph_check
-/tools/orfo_dict_2025.pdf
 __pycache__/

sources/scrabble_ru/README.md

@@ -5,5 +5,6 @@ pipeline under [`../../tools/`](../../tools/README.md) from the Russian academic
 dictionary, cross-checked against OpenCorpora and libmorph. `manual_confirm.txt` holds the
 hand-reviewed additions the pipeline merges in. Built to `dawg/ru_scrabble.dawg` (`make dawg-ru`).
 
-The pipeline's uncommitted intermediates (`orfo_dict_2025.txt`, `all.txt`, debug dumps) are
-regenerated here locally and are git-ignored.
+The pdftotext source of truth (`orfo_dict_2025.txt`) and the Stage-1 base (`all.txt`) are
+committed here; only the debug dumps (`undefined.txt`, `adjectives.txt`, `verbs.txt`,
+`singulars.txt`, `fate.tsv`) are regenerated locally and git-ignored.

tools/README.md

@@ -74,8 +74,8 @@ go run ./tools/ruwords
 # Stage 2 — the brain (Python + mawo + libmorph): writes scrabble.txt
 ru-venv/bin/python tools/ru_stage2.py
 
-# ask how a word did or did not reach the dictionary
-ru-venv/bin/python tools/ru_stage2.py --trace травмпункт
+# audit how a single word did or did not reach the dictionary (detailed, per-signal report)
+ru-venv/bin/python tools/ru_stage2.py --trace ндс
 # also write the in-memory buckets (undefined, adjectives, verbs, singulars, fate.tsv)
 ru-venv/bin/python tools/ru_stage2.py --dump
 ```
@@ -111,7 +111,9 @@ Each Stage-1 word (length 2–15) is routed by three sources, most authoritative
 
 1. **OpenCorpora** (`words.dawg`, read directly — *not* the predictor): a common-noun
    reading ⇒ keep the OpenCorpora lemma. The full OpenCorpora common-noun lexicon is also
-   added (so nouns absent from the PDF are included).
+   added (so nouns absent from the PDF are included), **minus indeclinable abbreviations**
+   (`Abbr`+`Fixd`: ндс, кпд, чп, …) — OpenCorpora tags those as nouns but they are not
+   Scrabble words. See *Abbreviations, proper nouns & function words* below for the full rule.
 2. **libmorph** (independent dictionary, via `libmorph_check`): a common-noun reading ⇒
    keep the libmorph lemma. The two dictionaries are treated as **complementary** — a noun
    reading in *either* is enough (their disagreements were reviewed and resolved this way,
@@ -151,6 +153,60 @@ speech. The codes were reverse-engineered (the docs omit the table):
 The analyser instance is requested with the key `libmorph.api.v4:utf-8` so words are
 passed and lemmas returned in UTF-8.
 
+## Abbreviations, proper nouns & function words
+
+OpenCorpora lists indeclinable abbreviations as common nouns (`Abbr`+`Fixd`), so seeding the
+result with its whole noun lexicon leaked non-words like **ндс, ст, ср, кпд, чп** into
+`scrabble.txt`. They are filtered as follows (see `oc_noun_lemmas`, `oc_abbr_fixd_only`,
+`oc_function_word` and the Stage-2 loop in `ru_stage2.py`):
+
+- An `Abbr`+`Fixd` noun is **never** admitted on OpenCorpora's word alone; the orthographic
+  dictionary decides via its own note (`classify`):
+  - attested as a **lowercase** РАН headword whose note is a noun ⇒ **kept** — the
+    lexicalised abbreviations that are real words: **сельпо, под, ска, роно, врио, тв, фио,
+    суперэвм** (present in `all.txt`, `classify == noun`);
+  - everything else ⇒ **dropped**: bare letter-abbreviations not in `all.txt` (ндс, кпд, чп,
+    гибдд, днк, …) and lowercase РАН headwords whose note is *not* a noun (гор, мин, про —
+    «приставка»; об, по, со — «предлог»; сто — числительное).
+- A word OpenCorpora also reads as a **function word** (`CONJ`/`PRCL`) is dropped even if the
+  note looks noun-like — **зато** (союз), and any like it.
+- A word whose OpenCorpora common-noun reading is an **inflected form** (its lemma is a
+  different word) is not a headword and stays out — **ан** (род. мн. от «ана»; the proper
+  noun «Ан», самолёт Антонова, is capitalised and never reaches `all.txt`).
+
+The discriminator is **not** the `(сокр.)` mark — сельпо carries it just like ндс:
+
+| word | in `all.txt` (lowercase РАН headword) | `classify` (РАН note) | OC `CONJ`/`PRCL` | verdict |
+|------|:---:|:---:|:---:|---------|
+| сельпо, под, ска, роно, врио, тв, фио, суперэвм | yes | noun | no | **keep** |
+| ндс, кпд, чп, гибдд, днк, … | no | — | no | drop (seed) |
+| гор, мин, про, об, по, со, сто, прим | yes | not noun | — | drop (note) |
+| зато | yes | — | **yes** | drop (function word) |
+| ан | yes | noun | yes | drop (OC lemma ≠ word) |
+
+### Auditing a word and repeating this analysis
+
+`--trace WORD` prints a per-signal audit — Stage-1/`all.txt` membership, every OpenCorpora
+reading (POS, `Abbr`/`Fixd`, `CONJ`/`PRCL`), libmorph, the РАН note with its `classify`
+verdict, and the final outcome:
+
+```sh
+ru-venv/bin/python tools/ru_stage2.py --trace ндс      # → НЕ в словаре: исключено из посева
+ru-venv/bin/python tools/ru_stage2.py --trace сельпо   # → В СЛОВАРЕ: по помете орфословаря
+```
+
+To find a *whole class* of suspect words (the procedure used to catch the abbreviations above),
+read the signals straight from the dictionaries with the same primitives the pipeline uses:
+
+1. iterate `M._dawg_dict.words_dawg` and collect the lemmas with the grammemes you suspect
+   (e.g. `Abbr`+`Fixd` on the lemma, no clean noun paradigm) — the candidate set;
+2. split it by `w in all.txt` (lowercase РАН headword), by `classify(w, note)` (РАН-note POS),
+   and by an OpenCorpora `CONJ`/`PRCL` reading — these three signals separate real words from
+   abbreviations / proper nouns / function words;
+3. review the candidate list by hand, add the distinguishing grammeme to the filter, re-run,
+   and diff the rebuilt list against the previous one to confirm only the intended words moved
+   (`--trace` each borderline decision).
+
 ## Notes & caveats
 
 - The hard tail (≈ 35 000 Stage-1 words / our candidates) is in **no** morphological

tools/ru_stage2.py

@@ -38,6 +38,8 @@ LIBMORPH_BIN = os.path.join(HERE, "libmorph_check")
 ALPHABET = "абвгдеёжзийклмнопрстуфхцчшщъыьэюя"
 ORDER = {c: i for i, c in enumerate(ALPHABET)}
 PROPER = {"Name", "Surn", "Patr", "Geox", "Orgn", "Trad"}
+ABBR_NOUN = {"Abbr", "Fixd"}      # indeclinable abbreviation marker (НДС, КПД, сельпо, …)
+FUNCTION_POS = {"CONJ", "PRCL"}   # conjunction / particle: a function word, never a Scrabble noun
 LIBMORPH_NOUN_CODES = set(range(7, 22)) | {24}  # 7..21 plus 24 (pluralia tantum)
 ADJ_END = {"ая", "яя", "ое", "ее", "ье", "ья", "ьи"}
 VERB3 = ("ет", "ёт", "ит", "ют", "ут", "ает", "яет", "ует", "уют", "нет", "жет", "чет")
@@ -72,7 +74,10 @@ D = M._dawg_dict
 
 
 def oc_noun_lemmas():
-    """Every common-noun lemma (nom. sing. / pluralia tantum) in OpenCorpora's words.dawg."""
+    """Every common-noun lemma (nom. sing. / pluralia tantum) in OpenCorpora's words.dawg,
+    excluding indeclinable abbreviations (Abbr+Fixd, e.g. НДС, КПД). Such words are admitted
+    only when the orthographic dictionary attests them as nouns (see oc_abbr_fixd_only and the
+    Stage-2 loop); abbreviations carried by OpenCorpora alone (ст, ср, кпд, …) are dropped."""
     gp, pt = D.get_paradigm, D.parse_tag_string
     para0, tagc = {}, {}
 
@@ -104,6 +109,8 @@ def oc_noun_lemmas():
         pre0, suf0, gr0 = g0(pid)
         if (PROPER & gr) or (PROPER & gr0):
             continue
+        if ABBR_NOUN <= gr0:        # indeclinable abbreviation: not seeded; the loop decides via the note
+            continue
         stem = word[len(pre):len(word) - len(suf)] if suf else word[len(pre):]
         out.add(pre0 + stem + suf0)
     return {w for w in out if cyr_ok(w)}
@@ -126,6 +133,43 @@ def oc_status(word):
     return False, True
 
 
+def oc_abbr_fixd_only(word):
+    """True when OpenCorpora's only common-noun reading of word is an indeclinable
+    abbreviation (Abbr+Fixd) — e.g. сельпо, ндс, ан. Such a word is not admitted as a noun on
+    OpenCorpora's say-so; the Stage-2 loop hands it to the orthographic note (and the function-
+    word check) instead, so lexicalised nouns (сельпо, под, ска) survive while bare letter-
+    abbreviations (ндс, кпд) and proper/служебные homographs (ан, зато) do not."""
+    parses = D.get_word_parses(word)
+    if not parses:
+        return False
+    gp, pt = D.get_paradigm, D.parse_tag_string
+    has_noun = has_plain = False
+    for pid, idx in parses:
+        suf, tag, pre = gp(pid, idx)
+        pos, gr = pt(tag)
+        if pos != "NOUN":
+            continue
+        _, tag0, _ = gp(pid, 0)
+        _, gr0 = pt(tag0)
+        if PROPER & gr or PROPER & gr0:
+            continue
+        has_noun = True
+        if not (ABBR_NOUN <= gr0):
+            has_plain = True
+    return has_noun and not has_plain
+
+
+def oc_function_word(word):
+    """True when OpenCorpora reads word as a conjunction or particle — a function word that
+    shares a spelling with an abbreviation/proper noun (ан «ан нет», зато). It is never a
+    Scrabble noun even if the orthographic note looks noun-like (ан → «самолёт Антонова»)."""
+    gp, pt = D.get_paradigm, D.parse_tag_string
+    for pid, idx in (D.get_word_parses(word) or []):
+        if pt(gp(pid, idx)[1])[0] in FUNCTION_POS:
+            return True
+    return False
+
+
 def libmorph_analyze(words):
     """Map each word to (known, noun_lemma, codes) per libmorph; noun_lemma is None when it
     is not a common noun there. Empty result if the helper binary is not built."""
@@ -240,9 +284,27 @@ def build():
     scrabble = set(oc)
     adj, verb, amb = [], [], []
     for w in pdf:
+        if oc_abbr_fixd_only(w):
+            # Indeclinable abbreviation in OpenCorpora: do not trust OC's noun verdict. Drop it
+            # if OC also reads it as a function word (ан, зато — служебное/пропер); otherwise let
+            # the orthographic note decide, keeping lexicalised nouns (сельпо, под, ска) and
+            # dropping the rest (кпд, чп, гор, …).
+            if oc_function_word(w):
+                fate[w] = "отброшено: служебное слово (CONJ/PRCL в OpenCorpora)"
+            elif classify(w, hmap.get(w)) == "noun":
+                s = to_singular(w)
+                scrabble.add(s)
+                fate[w] = "scrabble: несклон. аббрев.-сущ. по помете орфословаря" + ("" if s == w else f" → {s}")
+            else:
+                fate[w] = "отброшено: аббревиатура без подтверждающей пометы орфословаря"
+            continue
         oc_noun, oc_known = oc_status(w)
         if oc_noun:
-            fate[w] = "scrabble: сущ. по OpenCorpora"
+            # A common-noun reading already put w in the seed (scrabble = set(oc)) — unless its
+            # lemma is a different word, i.e. w is an inflected form (ан = род. мн. от «ана»);
+            # such a form is not a headword and stays out.
+            fate[w] = ("scrabble: сущ. по OpenCorpora" if w in scrabble
+                       else "отброшено: словоформа OpenCorpora (лемма — другое слово)")
             continue
         lm_known, lm_lemma, _ = lm.get(w, (False, None, frozenset()))
         if lm_lemma is not None:
@@ -295,21 +357,51 @@ def build():
     return {
         "oc": oc, "scrabble": scrabble, "undefined": undefined,
         "adjectives": adj, "verbs": verb, "singulars": ed_nouns,
-        "fate": fate, "all": set(all_words),
+        "fate": fate, "all": set(all_words), "hmap": hmap,
     }
 
 
 def trace(word, r):
+    """A detailed, per-signal audit of how WORD did or did not reach the dictionary: its
+    Stage-1 (all.txt) membership, every OpenCorpora reading, libmorph, the orthographic note
+    with its classify verdict, and the final outcome. Used by `--trace WORD`."""
     w = destress(word)
-    if w in r["fate"]:
-        return r["fate"][w]
-    if w in r["scrabble"]:
-        return "scrabble: лексикон OpenCorpora" if w in r["oc"] else "scrabble: производная/лемма"
-    if w not in r["all"]:
-        return "нет в russian_all (не извлечено на Stage 1 — нет в .pdf, либо имя собств./дефис/форма)"
-    if not cyr_ok(w):
-        return "отсеяно: длина или символы вне диапазона (2–15 кириллица)"
-    return "не определено"
+    note = r["hmap"].get(w)
+    parses = D.get_word_parses(w) or []
+    gp, pt = D.get_paradigm, D.parse_tag_string
+    pos = sorted({pt(gp(pid, idx)[1])[0] for pid, idx in parses})
+    oc_noun, _ = oc_status(w)
+
+    out = [f"{word}  →  {'В СЛОВАРЕ' if w in r['scrabble'] else 'НЕ в словаре'}"]
+    out.append(f"  Stage-1 / all.txt (строчный заголовок РАН): {'да' if w in r['all'] else 'нет'}")
+    if parses:
+        kind = ("только несклон. аббрев. (Abbr+Fixd)" if oc_abbr_fixd_only(w) else
+                "обычное сущ." if oc_noun else
+                "не существительное")
+        out.append(f"  OpenCorpora: части речи {pos}; как сущ. — {kind}; "
+                   f"служебное (CONJ/PRCL): {'да' if oc_function_word(w) else 'нет'}")
+    else:
+        out.append("  OpenCorpora: нет в словаре")
+    lm = libmorph_analyze([w]).get(w)
+    if lm:
+        known, lemma, _ = lm
+        out.append("  libmorph: " + (f"сущ. → {lemma}" if lemma else ("известно, не сущ." if known else "нет")))
+    else:
+        out.append("  libmorph: helper отсутствует — в анализе не участвует")
+    out.append("  Орфословарь РАН: " + (f"помета «{note}» → classify={classify(w, note)}"
+                                        if note is not None else "нет заголовка"))
+    reason = r["fate"].get(w)
+    if reason is None:
+        if w in r["scrabble"]:
+            reason = "лексикон OpenCorpora (обычное сущ.)"
+        elif oc_abbr_fixd_only(w) and w not in r["all"]:
+            reason = "исключено из посева: несклон. аббрев. (Abbr+Fixd), нет в all.txt"
+        elif not parses and w not in r["all"]:
+            reason = "нет ни в OpenCorpora, ни в all.txt"
+        else:
+            reason = "не прошло ни один путь отбора"
+    out.append(f"  ИТОГ: {reason}")
+    return "\n".join(out)
 
 
 def main():
@@ -320,7 +412,7 @@ def main():
 
     r = build()
     if args.trace:
-        print(f"{args.trace}: {trace(args.trace, r)}")
+        print(trace(args.trace, r))
         return
 
     write(os.path.join(OUT_DIR, "scrabble.txt"), r["scrabble"])