feat(ru): drop abbreviations from the Russian noun list
build / dawg (pull_request) Successful in 6m4s
build / dawg (pull_request) Successful in 6m4s
The list leaked abbreviations two ways: an OpenCorpora noun tagged Abbr without Fixd (the declinable acronym ОГВЗ) slipped past the Abbr+Fixd seed filter, and the РАН «(сокр.)» words (вуз, нэп, колхоз, гост, …) were kept on purpose by the old policy. Drop every abbreviation instead. A final pass in ru_stage2.py removes a word when the orthographic note marks it «(сокр.)» (is_abbrev_note) or when every OpenCorpora noun reading carries Abbr (oc_abbr_only — also the declinable огвз/заз the seed filter missed). Homographs with a plain-noun reading survive (ваза «ВАЗ», рис). Lexicalised words are rescued via manual_confirm.txt (колхоз, совхоз, рация, спецназ, эсминец, дзот, дот, прораб, токамак, тэн, … and под — a real noun OC knows only as ПОД); the bound prefix гос is vetoed in manual_reject.txt. 108 words removed from ru_scrabble / ru_erudit; en_sowpods unchanged. tools/README documents the policy and that libmorph is required for a faithful rebuild.
This commit is contained in:
+34
-26
@@ -61,7 +61,11 @@ g++ -std=c++17 -O2 tools/libmorph_check.cpp -lmorphrus -lmoonycode -o tools/libm
|
||||
```
|
||||
|
||||
If `tools/libmorph_check` is absent, Stage 2 still runs — it simply drops libmorph from
|
||||
the stack and reports `libmorph_helper=MISSING`.
|
||||
the stack and reports `libmorph_helper=MISSING`. **Build it before a release rebuild:** the
|
||||
committed word lists were produced *with* libmorph, and dropping it shifts the result by a few
|
||||
hundred words (libmorph-only nouns disappear, and adjectives/eponyms libmorph would veto —
|
||||
неперов, венерин — leak in through the orthographic-note path). A rebuild meant to refresh the
|
||||
committed lists is only faithful with the helper present.
|
||||
|
||||
## How to run
|
||||
|
||||
@@ -160,34 +164,38 @@ passed and lemmas returned in UTF-8.
|
||||
|
||||
## Abbreviations, proper nouns & function words
|
||||
|
||||
OpenCorpora lists indeclinable abbreviations as common nouns (`Abbr`+`Fixd`), so seeding the
|
||||
result with its whole noun lexicon leaked non-words like **ндс, ст, ср, кпд, чп** into
|
||||
`scrabble.txt`. They are filtered as follows (see `oc_noun_lemmas`, `oc_abbr_fixd_only`,
|
||||
`oc_function_word` and the Stage-2 loop in `ru_stage2.py`):
|
||||
**Abbreviations are not Scrabble words and are dropped** (вуз, нэп, гост, жэк, спид, огвз,
|
||||
обком, наркомат, роно, тв, …). A final pass in `build` removes every word matched by either
|
||||
signal, then re-admits the maintainer's rescues — see `is_abbrev_note`, `oc_abbr_only` and the
|
||||
abbreviation subtraction in `ru_stage2.py`:
|
||||
|
||||
- An `Abbr`+`Fixd` noun is **never** admitted on OpenCorpora's word alone; the orthographic
|
||||
dictionary decides via its own note (`classify`):
|
||||
- attested as a **lowercase** РАН headword whose note is a noun ⇒ **kept** — the
|
||||
lexicalised abbreviations that are real words: **сельпо, под, ска, роно, врио, тв, фио,
|
||||
суперэвм** (present in `all.txt`, `classify == noun`);
|
||||
- everything else ⇒ **dropped**: bare letter-abbreviations not in `all.txt` (ндс, кпд, чп,
|
||||
гибдд, днк, …) and lowercase РАН headwords whose note is *not* a noun (гор, мин, про —
|
||||
«приставка»; об, по, со — «предлог»; сто — числительное).
|
||||
- A word OpenCorpora also reads as a **function word** (`CONJ`/`PRCL`) is dropped even if the
|
||||
note looks noun-like — **зато** (союз), and any like it.
|
||||
- A word whose OpenCorpora common-noun reading is an **inflected form** (its lemma is a
|
||||
different word) is not a headword and stays out — **ан** (род. мн. от «ана»; the proper
|
||||
noun «Ан», самолёт Антонова, is capitalised and never reaches `all.txt`).
|
||||
- **The orthographic note marks it `(сокр.)`** (`is_abbrev_note`): the РАН dictionary's own
|
||||
abbreviation mark, covering both the bare initialisms (вуз, нэп, гост, бтр, кпд) and the
|
||||
Soviet-era compounds (обком, райком, наркомат, госплан, …). This is the authoritative signal.
|
||||
- **OpenCorpora reads it only as an abbreviation** (`oc_abbr_only`): *every* common-noun
|
||||
reading carries the `Abbr` grammeme. This catches acronyms РАН does not head at all — the
|
||||
declinable огвз (a full `Abbr` case paradigm, no `Fixd`, which the `Abbr`+`Fixd` seed filter
|
||||
misses), заз, and the indeclinable роно, тв, фио, суперэвм. A homograph that *also* has a
|
||||
plain-noun reading is **not** abbreviation-only and survives — ваза «ВАЗ», рис «РИС».
|
||||
|
||||
The discriminator is **not** the `(сокр.)` mark — сельпо carries it just like ндс:
|
||||
Still dropped by the seed/loop before this pass: a `CONJ`/`PRCL` **function word** even with a
|
||||
noun-like note (**зато**), and an OpenCorpora reading that is an **inflected form** of another
|
||||
lemma (**ан** = род. мн. от «ана»; the proper noun «Ан» is capitalised and never reaches
|
||||
`all.txt`). Proper nouns (`Name`/`Surn`/`Geox`/`Orgn`/…) are excluded from the seed outright.
|
||||
|
||||
| word | in `all.txt` (lowercase РАН headword) | `classify` (РАН note) | OC `CONJ`/`PRCL` | verdict |
|
||||
|------|:---:|:---:|:---:|---------|
|
||||
| сельпо, под, ска, роно, врио, тв, фио, суперэвм | yes | noun | no | **keep** |
|
||||
| ндс, кпд, чп, гибдд, днк, … | no | — | no | drop (seed) |
|
||||
| гор, мин, про, об, по, со, сто, прим | yes | not noun | — | drop (note) |
|
||||
| зато | yes | — | **yes** | drop (function word) |
|
||||
| ан | yes | noun | yes | drop (OC lemma ≠ word) |
|
||||
**Maintainer overrides decide the borderline cases** (this is policy, not a heuristic):
|
||||
|
||||
- `manual_confirm.txt` **rescues** lexicalised words that carry `(сокр.)` or that OpenCorpora
|
||||
only knows as an abbreviation but which read as ordinary nouns — **колхоз, совхоз, рация,
|
||||
спецназ, эсминец, линкор, дзот, дот, прораб, токамак, тэн, …**, and **под** (a real noun,
|
||||
«под печи», that OpenCorpora knows only as the abbreviation ПОД). Edit this list to widen or
|
||||
narrow which abbreviations stay.
|
||||
- `manual_reject.txt` **vetoes** the few non-noun leftovers that no signal catches — e.g. the
|
||||
bound prefix **гос** («гос… — первая часть сложных слов», not a standalone word).
|
||||
|
||||
> History note: an earlier policy *kept* lexicalised abbreviations (сельпо, вуз, нэп, …) and
|
||||
> used `(сокр.)` only as a non-discriminator. The current policy drops all abbreviations and
|
||||
> rescues a curated subset, so the dictionary holds words, not acronyms.
|
||||
|
||||
### Auditing a word and repeating this analysis
|
||||
|
||||
|
||||
Reference in New Issue
Block a user