feat(ru): drop abbreviations from the Russian noun list
build / dawg (pull_request) Successful in 6m4s

The list leaked abbreviations two ways: an OpenCorpora noun tagged Abbr without
Fixd (the declinable acronym ОГВЗ) slipped past the Abbr+Fixd seed filter, and the
РАН «(сокр.)» words (вуз, нэп, колхоз, гост, …) were kept on purpose by the old
policy.

Drop every abbreviation instead. A final pass in ru_stage2.py removes a word when
the orthographic note marks it «(сокр.)» (is_abbrev_note) or when every OpenCorpora
noun reading carries Abbr (oc_abbr_only — also the declinable огвз/заз the seed
filter missed). Homographs with a plain-noun reading survive (ваза «ВАЗ», рис).
Lexicalised words are rescued via manual_confirm.txt (колхоз, совхоз, рация,
спецназ, эсминец, дзот, дот, прораб, токамак, тэн, … and под — a real noun OC knows
only as ПОД); the bound prefix гос is vetoed in manual_reject.txt.

108 words removed from ru_scrabble / ru_erudit; en_sowpods unchanged. tools/README
documents the policy and that libmorph is required for a faithful rebuild.
This commit is contained in:
Ilia Denisov
2026-06-22 14:16:55 +02:00
parent 1f5e1906ad
commit 74ff3aefb6
6 changed files with 109 additions and 242 deletions
+34 -26
View File
@@ -61,7 +61,11 @@ g++ -std=c++17 -O2 tools/libmorph_check.cpp -lmorphrus -lmoonycode -o tools/libm
```
If `tools/libmorph_check` is absent, Stage 2 still runs — it simply drops libmorph from
the stack and reports `libmorph_helper=MISSING`.
the stack and reports `libmorph_helper=MISSING`. **Build it before a release rebuild:** the
committed word lists were produced *with* libmorph, and dropping it shifts the result by a few
hundred words (libmorph-only nouns disappear, and adjectives/eponyms libmorph would veto —
неперов, венерин — leak in through the orthographic-note path). A rebuild meant to refresh the
committed lists is only faithful with the helper present.
## How to run
@@ -160,34 +164,38 @@ passed and lemmas returned in UTF-8.
## Abbreviations, proper nouns & function words
OpenCorpora lists indeclinable abbreviations as common nouns (`Abbr`+`Fixd`), so seeding the
result with its whole noun lexicon leaked non-words like **ндс, ст, ср, кпд, чп** into
`scrabble.txt`. They are filtered as follows (see `oc_noun_lemmas`, `oc_abbr_fixd_only`,
`oc_function_word` and the Stage-2 loop in `ru_stage2.py`):
**Abbreviations are not Scrabble words and are dropped** (вуз, нэп, гост, жэк, спид, огвз,
обком, наркомат, роно, тв, …). A final pass in `build` removes every word matched by either
signal, then re-admits the maintainer's rescues — see `is_abbrev_note`, `oc_abbr_only` and the
abbreviation subtraction in `ru_stage2.py`:
- An `Abbr`+`Fixd` noun is **never** admitted on OpenCorpora's word alone; the orthographic
dictionary decides via its own note (`classify`):
- attested as a **lowercase** РАН headword whose note is a noun ⇒ **kept** — the
lexicalised abbreviations that are real words: **сельпо, под, ска, роно, врио, тв, фио,
суперэвм** (present in `all.txt`, `classify == noun`);
- everything else ⇒ **dropped**: bare letter-abbreviations not in `all.txt` (ндс, кпд, чп,
гибдд, днк, …) and lowercase РАН headwords whose note is *not* a noun (гор, мин, про —
«приставка»; об, по, со — «предлог»; сто — числительное).
- A word OpenCorpora also reads as a **function word** (`CONJ`/`PRCL`) is dropped even if the
note looks noun-like — **зато** (союз), and any like it.
- A word whose OpenCorpora common-noun reading is an **inflected form** (its lemma is a
different word) is not a headword and stays out — **ан** (род. мн. от «ана»; the proper
noun «Ан», самолёт Антонова, is capitalised and never reaches `all.txt`).
- **The orthographic note marks it `(сокр.)`** (`is_abbrev_note`): the РАН dictionary's own
abbreviation mark, covering both the bare initialisms (вуз, нэп, гост, бтр, кпд) and the
Soviet-era compounds (обком, райком, наркомат, госплан, …). This is the authoritative signal.
- **OpenCorpora reads it only as an abbreviation** (`oc_abbr_only`): *every* common-noun
reading carries the `Abbr` grammeme. This catches acronyms РАН does not head at all — the
declinable огвз (a full `Abbr` case paradigm, no `Fixd`, which the `Abbr`+`Fixd` seed filter
misses), заз, and the indeclinable роно, тв, фио, суперэвм. A homograph that *also* has a
plain-noun reading is **not** abbreviation-only and survives — ваза «ВАЗ», рис «РИС».
The discriminator is **not** the `(сокр.)` mark — сельпо carries it just like ндс:
Still dropped by the seed/loop before this pass: a `CONJ`/`PRCL` **function word** even with a
noun-like note (**зато**), and an OpenCorpora reading that is an **inflected form** of another
lemma (**ан** = род. мн. от «ана»; the proper noun «Ан» is capitalised and never reaches
`all.txt`). Proper nouns (`Name`/`Surn`/`Geox`/`Orgn`/…) are excluded from the seed outright.
| word | in `all.txt` (lowercase РАН headword) | `classify` (РАН note) | OC `CONJ`/`PRCL` | verdict |
|------|:---:|:---:|:---:|---------|
| сельпо, под, ска, роно, врио, тв, фио, суперэвм | yes | noun | no | **keep** |
| ндс, кпд, чп, гибдд, днк, … | no | — | no | drop (seed) |
| гор, мин, про, об, по, со, сто, прим | yes | not noun | — | drop (note) |
| зато | yes | — | **yes** | drop (function word) |
| ан | yes | noun | yes | drop (OC lemma ≠ word) |
**Maintainer overrides decide the borderline cases** (this is policy, not a heuristic):
- `manual_confirm.txt` **rescues** lexicalised words that carry `(сокр.)` or that OpenCorpora
only knows as an abbreviation but which read as ordinary nouns — **колхоз, совхоз, рация,
спецназ, эсминец, линкор, дзот, дот, прораб, токамак, тэн, …**, and **под** (a real noun,
«под печи», that OpenCorpora knows only as the abbreviation ПОД). Edit this list to widen or
narrow which abbreviations stay.
- `manual_reject.txt` **vetoes** the few non-noun leftovers that no signal catches — e.g. the
bound prefix **гос** («гос… — первая часть сложных слов», not a standalone word).
> History note: an earlier policy *kept* lexicalised abbreviations (сельпо, вуз, нэп, …) and
> used `(сокр.)` only as a non-discriminator. The current policy drops all abbreviations and
> rescues a curated subset, so the dictionary holds words, not acronyms.
### Auditing a word and repeating this analysis