Filter OpenCorpora abbreviations & proper nouns from the Russian noun list
build / dawg (pull_request) Successful in 2m45s

OpenCorpora tags indeclinable abbreviations as common nouns (Abbr+Fixd), and
Stage 2 seeded the result with its whole noun lexicon, so non-words leaked into
scrabble.txt: ндс, ст, ср, кпд, чп, гибдд, днк, …

ru_stage2.py now drops Abbr+Fixd nouns from the OpenCorpora seed and lets the
orthographic dictionary decide instead: a lowercase РАН headword whose note is a
noun is kept (the lexicalised сельпо, под, ска, роно, врио, тв, фио, суперэвм),
everything else is dropped. Function words (CONJ/PRCL, e.g. зато) and OpenCorpora
inflected forms (ан = род. мн. от «ана»; proper «Ан») are excluded too.

--trace WORD now prints a detailed per-signal audit (all.txt, OpenCorpora POS /
Abbr / Fixd / CONJ/PRCL, libmorph, РАН note + classify, outcome) for auditing a
word or repeating the analysis; tools/README.md documents the rule and method.

Net: 179 words removed from scrabble.txt (83385 -> 83206) and erudit.txt
(83343 -> 83164); verified to remove exactly those 179 and add none. DAWGs
rebuild clean.
This commit is contained in:
Ilia Denisov
2026-06-13 13:33:26 +02:00
parent 5b7a741ec2
commit 6b8b176f82
4 changed files with 164 additions and 374 deletions
+59 -3
View File
@@ -74,8 +74,8 @@ go run ./tools/ruwords
# Stage 2 — the brain (Python + mawo + libmorph): writes scrabble.txt
ru-venv/bin/python tools/ru_stage2.py
# ask how a word did or did not reach the dictionary
ru-venv/bin/python tools/ru_stage2.py --trace травмпункт
# audit how a single word did or did not reach the dictionary (detailed, per-signal report)
ru-venv/bin/python tools/ru_stage2.py --trace ндс
# also write the in-memory buckets (undefined, adjectives, verbs, singulars, fate.tsv)
ru-venv/bin/python tools/ru_stage2.py --dump
```
@@ -111,7 +111,9 @@ Each Stage-1 word (length 215) is routed by three sources, most authoritative
1. **OpenCorpora** (`words.dawg`, read directly — *not* the predictor): a common-noun
reading ⇒ keep the OpenCorpora lemma. The full OpenCorpora common-noun lexicon is also
added (so nouns absent from the PDF are included).
added (so nouns absent from the PDF are included), **minus indeclinable abbreviations**
(`Abbr`+`Fixd`: ндс, кпд, чп, …) — OpenCorpora tags those as nouns but they are not
Scrabble words. See *Abbreviations, proper nouns & function words* below for the full rule.
2. **libmorph** (independent dictionary, via `libmorph_check`): a common-noun reading ⇒
keep the libmorph lemma. The two dictionaries are treated as **complementary** — a noun
reading in *either* is enough (their disagreements were reviewed and resolved this way,
@@ -151,6 +153,60 @@ speech. The codes were reverse-engineered (the docs omit the table):
The analyser instance is requested with the key `libmorph.api.v4:utf-8` so words are
passed and lemmas returned in UTF-8.
## Abbreviations, proper nouns & function words
OpenCorpora lists indeclinable abbreviations as common nouns (`Abbr`+`Fixd`), so seeding the
result with its whole noun lexicon leaked non-words like **ндс, ст, ср, кпд, чп** into
`scrabble.txt`. They are filtered as follows (see `oc_noun_lemmas`, `oc_abbr_fixd_only`,
`oc_function_word` and the Stage-2 loop in `ru_stage2.py`):
- An `Abbr`+`Fixd` noun is **never** admitted on OpenCorpora's word alone; the orthographic
dictionary decides via its own note (`classify`):
- attested as a **lowercase** РАН headword whose note is a noun ⇒ **kept** — the
lexicalised abbreviations that are real words: **сельпо, под, ска, роно, врио, тв, фио,
суперэвм** (present in `all.txt`, `classify == noun`);
- everything else ⇒ **dropped**: bare letter-abbreviations not in `all.txt` (ндс, кпд, чп,
гибдд, днк, …) and lowercase РАН headwords whose note is *not* a noun (гор, мин, про —
«приставка»; об, по, со — «предлог»; сто — числительное).
- A word OpenCorpora also reads as a **function word** (`CONJ`/`PRCL`) is dropped even if the
note looks noun-like — **зато** (союз), and any like it.
- A word whose OpenCorpora common-noun reading is an **inflected form** (its lemma is a
different word) is not a headword and stays out — **ан** (род. мн. от «ана»; the proper
noun «Ан», самолёт Антонова, is capitalised and never reaches `all.txt`).
The discriminator is **not** the `(сокр.)` mark — сельпо carries it just like ндс:
| word | in `all.txt` (lowercase РАН headword) | `classify` (РАН note) | OC `CONJ`/`PRCL` | verdict |
|------|:---:|:---:|:---:|---------|
| сельпо, под, ска, роно, врио, тв, фио, суперэвм | yes | noun | no | **keep** |
| ндс, кпд, чп, гибдд, днк, … | no | — | no | drop (seed) |
| гор, мин, про, об, по, со, сто, прим | yes | not noun | — | drop (note) |
| зато | yes | — | **yes** | drop (function word) |
| ан | yes | noun | yes | drop (OC lemma ≠ word) |
### Auditing a word and repeating this analysis
`--trace WORD` prints a per-signal audit — Stage-1/`all.txt` membership, every OpenCorpora
reading (POS, `Abbr`/`Fixd`, `CONJ`/`PRCL`), libmorph, the РАН note with its `classify`
verdict, and the final outcome:
```sh
ru-venv/bin/python tools/ru_stage2.py --trace ндс # → НЕ в словаре: исключено из посева
ru-venv/bin/python tools/ru_stage2.py --trace сельпо # → В СЛОВАРЕ: по помете орфословаря
```
To find a *whole class* of suspect words (the procedure used to catch the abbreviations above),
read the signals straight from the dictionaries with the same primitives the pipeline uses:
1. iterate `M._dawg_dict.words_dawg` and collect the lemmas with the grammemes you suspect
(e.g. `Abbr`+`Fixd` on the lemma, no clean noun paradigm) — the candidate set;
2. split it by `w in all.txt` (lowercase РАН headword), by `classify(w, note)` (РАН-note POS),
and by an OpenCorpora `CONJ`/`PRCL` reading — these three signals separate real words from
abbreviations / proper nouns / function words;
3. review the candidate list by hand, add the distinguishing grammeme to the filter, re-run,
and diff the rebuilt list against the previous one to confirm only the intended words moved
(`--trace` each borderline decision).
## Notes & caveats
- The hard tail (≈ 35 000 Stage-1 words / our candidates) is in **no** morphological