Filter OpenCorpora abbreviations & proper nouns from the Russian noun list
build / dawg (pull_request) Successful in 2m45s
build / dawg (pull_request) Successful in 2m45s
OpenCorpora tags indeclinable abbreviations as common nouns (Abbr+Fixd), and Stage 2 seeded the result with its whole noun lexicon, so non-words leaked into scrabble.txt: ндс, ст, ср, кпд, чп, гибдд, днк, … ru_stage2.py now drops Abbr+Fixd nouns from the OpenCorpora seed and lets the orthographic dictionary decide instead: a lowercase РАН headword whose note is a noun is kept (the lexicalised сельпо, под, ска, роно, врио, тв, фио, суперэвм), everything else is dropped. Function words (CONJ/PRCL, e.g. зато) and OpenCorpora inflected forms (ан = род. мн. от «ана»; proper «Ан») are excluded too. --trace WORD now prints a detailed per-signal audit (all.txt, OpenCorpora POS / Abbr / Fixd / CONJ/PRCL, libmorph, РАН note + classify, outcome) for auditing a word or repeating the analysis; tools/README.md documents the rule and method. Net: 179 words removed from scrabble.txt (83385 -> 83206) and erudit.txt (83343 -> 83164); verified to remove exactly those 179 and add none. DAWGs rebuild clean.
This commit is contained in:
+59
-3
@@ -74,8 +74,8 @@ go run ./tools/ruwords
|
||||
# Stage 2 — the brain (Python + mawo + libmorph): writes scrabble.txt
|
||||
ru-venv/bin/python tools/ru_stage2.py
|
||||
|
||||
# ask how a word did or did not reach the dictionary
|
||||
ru-venv/bin/python tools/ru_stage2.py --trace травмпункт
|
||||
# audit how a single word did or did not reach the dictionary (detailed, per-signal report)
|
||||
ru-venv/bin/python tools/ru_stage2.py --trace ндс
|
||||
# also write the in-memory buckets (undefined, adjectives, verbs, singulars, fate.tsv)
|
||||
ru-venv/bin/python tools/ru_stage2.py --dump
|
||||
```
|
||||
@@ -111,7 +111,9 @@ Each Stage-1 word (length 2–15) is routed by three sources, most authoritative
|
||||
|
||||
1. **OpenCorpora** (`words.dawg`, read directly — *not* the predictor): a common-noun
|
||||
reading ⇒ keep the OpenCorpora lemma. The full OpenCorpora common-noun lexicon is also
|
||||
added (so nouns absent from the PDF are included).
|
||||
added (so nouns absent from the PDF are included), **minus indeclinable abbreviations**
|
||||
(`Abbr`+`Fixd`: ндс, кпд, чп, …) — OpenCorpora tags those as nouns but they are not
|
||||
Scrabble words. See *Abbreviations, proper nouns & function words* below for the full rule.
|
||||
2. **libmorph** (independent dictionary, via `libmorph_check`): a common-noun reading ⇒
|
||||
keep the libmorph lemma. The two dictionaries are treated as **complementary** — a noun
|
||||
reading in *either* is enough (their disagreements were reviewed and resolved this way,
|
||||
@@ -151,6 +153,60 @@ speech. The codes were reverse-engineered (the docs omit the table):
|
||||
The analyser instance is requested with the key `libmorph.api.v4:utf-8` so words are
|
||||
passed and lemmas returned in UTF-8.
|
||||
|
||||
## Abbreviations, proper nouns & function words
|
||||
|
||||
OpenCorpora lists indeclinable abbreviations as common nouns (`Abbr`+`Fixd`), so seeding the
|
||||
result with its whole noun lexicon leaked non-words like **ндс, ст, ср, кпд, чп** into
|
||||
`scrabble.txt`. They are filtered as follows (see `oc_noun_lemmas`, `oc_abbr_fixd_only`,
|
||||
`oc_function_word` and the Stage-2 loop in `ru_stage2.py`):
|
||||
|
||||
- An `Abbr`+`Fixd` noun is **never** admitted on OpenCorpora's word alone; the orthographic
|
||||
dictionary decides via its own note (`classify`):
|
||||
- attested as a **lowercase** РАН headword whose note is a noun ⇒ **kept** — the
|
||||
lexicalised abbreviations that are real words: **сельпо, под, ска, роно, врио, тв, фио,
|
||||
суперэвм** (present in `all.txt`, `classify == noun`);
|
||||
- everything else ⇒ **dropped**: bare letter-abbreviations not in `all.txt` (ндс, кпд, чп,
|
||||
гибдд, днк, …) and lowercase РАН headwords whose note is *not* a noun (гор, мин, про —
|
||||
«приставка»; об, по, со — «предлог»; сто — числительное).
|
||||
- A word OpenCorpora also reads as a **function word** (`CONJ`/`PRCL`) is dropped even if the
|
||||
note looks noun-like — **зато** (союз), and any like it.
|
||||
- A word whose OpenCorpora common-noun reading is an **inflected form** (its lemma is a
|
||||
different word) is not a headword and stays out — **ан** (род. мн. от «ана»; the proper
|
||||
noun «Ан», самолёт Антонова, is capitalised and never reaches `all.txt`).
|
||||
|
||||
The discriminator is **not** the `(сокр.)` mark — сельпо carries it just like ндс:
|
||||
|
||||
| word | in `all.txt` (lowercase РАН headword) | `classify` (РАН note) | OC `CONJ`/`PRCL` | verdict |
|
||||
|------|:---:|:---:|:---:|---------|
|
||||
| сельпо, под, ска, роно, врио, тв, фио, суперэвм | yes | noun | no | **keep** |
|
||||
| ндс, кпд, чп, гибдд, днк, … | no | — | no | drop (seed) |
|
||||
| гор, мин, про, об, по, со, сто, прим | yes | not noun | — | drop (note) |
|
||||
| зато | yes | — | **yes** | drop (function word) |
|
||||
| ан | yes | noun | yes | drop (OC lemma ≠ word) |
|
||||
|
||||
### Auditing a word and repeating this analysis
|
||||
|
||||
`--trace WORD` prints a per-signal audit — Stage-1/`all.txt` membership, every OpenCorpora
|
||||
reading (POS, `Abbr`/`Fixd`, `CONJ`/`PRCL`), libmorph, the РАН note with its `classify`
|
||||
verdict, and the final outcome:
|
||||
|
||||
```sh
|
||||
ru-venv/bin/python tools/ru_stage2.py --trace ндс # → НЕ в словаре: исключено из посева
|
||||
ru-venv/bin/python tools/ru_stage2.py --trace сельпо # → В СЛОВАРЕ: по помете орфословаря
|
||||
```
|
||||
|
||||
To find a *whole class* of suspect words (the procedure used to catch the abbreviations above),
|
||||
read the signals straight from the dictionaries with the same primitives the pipeline uses:
|
||||
|
||||
1. iterate `M._dawg_dict.words_dawg` and collect the lemmas with the grammemes you suspect
|
||||
(e.g. `Abbr`+`Fixd` on the lemma, no clean noun paradigm) — the candidate set;
|
||||
2. split it by `w in all.txt` (lowercase РАН headword), by `classify(w, note)` (РАН-note POS),
|
||||
and by an OpenCorpora `CONJ`/`PRCL` reading — these three signals separate real words from
|
||||
abbreviations / proper nouns / function words;
|
||||
3. review the candidate list by hand, add the distinguishing grammeme to the filter, re-run,
|
||||
and diff the rebuilt list against the previous one to confirm only the intended words moved
|
||||
(`--trace` each borderline decision).
|
||||
|
||||
## Notes & caveats
|
||||
|
||||
- The hard tail (≈ 35 000 Stage-1 words / our candidates) is in **no** morphological
|
||||
|
||||
Reference in New Issue
Block a user