Ilia Denisov
7f8165f16d
build / dawg (pull_request) Successful in 24s
Remove the 11 obscene nouns built on the хуй/еб·ёб/пизд/бляд roots (ахуй, блядь, выблядок, ебальник, еблан, охуение, пизда, пиздец, пиздобол, хуй, хуйня) from the Russian Scrabble/Эрудит dictionary. These are morphologically valid common nouns, so the pipeline admitted them like any other word — OpenCorpora's tagset exposes no obscenity grammeme to filter on. They are vetoed as policy via manual_reject.txt, which is subtracted last in ru_stage2.build(), so no pipeline logic changes: scrabble.txt is the deterministic result of that subtraction and erudit.txt is its Ё→Е fold, regenerated with tools/fold_yo.py. The coarser-but-not-mat муд* (мудак, мудила) and legitimate homographs (очко, сука, моча, дрочёна) are deliberately kept. scrabble.txt 83027 -> 83016, erudit.txt 82985 -> 82974.
236 lines
14 KiB
Markdown
236 lines
14 KiB
Markdown
# Russian word-list preparation (`tools`)
|
||
|
||
Builds the Russian **noun** word list for the Scrabble/Эрудит solver out of the official
|
||
Russian academic **orthographic dictionary**, cross-checked against two independent
|
||
morphological dictionaries.
|
||
|
||
The goal of the pipeline is a list of **common nouns in the nominative singular**
|
||
(`sources/scrabble_ru/scrabble.txt`), plus an ambiguous tail for manual review.
|
||
|
||
> This directory is self-contained tooling for *building* the word list. It is not part
|
||
> of the solver library. The committed result lives in `sources/scrabble_ru/`.
|
||
|
||
## Source
|
||
|
||
`orfo_dict_2025.pdf` — *Русский орфографический словарь РАН* (≈ 200 000 entries), the
|
||
authority for **spelling**. It encodes declension type in its grammatical notes but does
|
||
**not** reliably mark part of speech.
|
||
|
||
- Source: <https://ruslang.ru/sites/default/files/doc/normativnyje_slovari/orfograficheskij_slovar.pdf>
|
||
- Mirror: <https://rus-gos.spbu.ru/index.php/dictionary>
|
||
|
||
The PDF is git-ignored (large, third-party); place it here as `orfo_dict_2025.pdf`. Its
|
||
pdftotext output is committed as `russian/orfo_dict_2025.txt`, so the word list rebuilds
|
||
from the text alone — the binary PDF is needed only to regenerate that text.
|
||
|
||
## Outputs (`sources/scrabble_ru/`)
|
||
|
||
The committed result is **three** files; every other bucket stays in the Stage-2
|
||
process's memory (dump it with `--dump`, query it with `--trace WORD`).
|
||
|
||
| File | Committed | Meaning |
|
||
|------|:--:|---------|
|
||
| `orfo_dict_2025.txt` | ✓ | the pdftotext output — the parsed source of truth (the PDF binary is not needed to rebuild). |
|
||
| `all.txt` | ✓ | Stage 1 base: every clean Cyrillic headword/variant; a plural headword with a singular is replaced by that singular. |
|
||
| `manual_confirm.txt` | ✓ | hand-reviewed nouns from the undefined tail; the brain merges them into the result. |
|
||
| `manual_reject.txt` | ✓ | hand-vetoed words subtracted from the result last: false nouns (substantivized adjectives a dictionary misreads as nouns, e.g. нёбный) and obscene lexicon (mat, e.g. хуй, блядь). |
|
||
| `scrabble.txt` | ✓ | **Stage 2 result**: common nouns, nominative singular (+ pluralia tantum), length 2–15 — the working dictionary. |
|
||
| `undefined.txt` | — | the ambiguous tail; kept in memory, written only with `--dump`. |
|
||
|
||
`--dump` also writes `adjectives.txt`, `verbs.txt`, `singulars.txt` and `fate.tsv` (every
|
||
word with the reason it did or did not reach the dictionary); these are git-ignored debug
|
||
artifacts. Stage 1 also writes `/tmp/ru_{skip,singulars,variants}.txt`, intermediate inputs
|
||
the brain consumes.
|
||
|
||
## Prerequisites
|
||
|
||
```sh
|
||
# 1. pdftotext (Poppler)
|
||
sudo apt-get install -y poppler-utils
|
||
|
||
# 2. Go toolchain (Stage 1) — already required by the parent module
|
||
|
||
# 3. Python + the OpenCorpora analyser (Stage 2)
|
||
sudo apt-get install -y python3-venv python3-pip
|
||
python3 -m venv ru-venv
|
||
ru-venv/bin/pip install mawo-pymorphy3 # bundles OpenCorpora 2025 (words.dawg)
|
||
|
||
# 4. libmorph — the independent morphological dictionary (Stage 2 cross-check)
|
||
sudo apt-get install -y morphrus morphrus-dev moonycode-dev morphapi-dev
|
||
g++ -std=c++17 -O2 tools/libmorph_check.cpp -lmorphrus -lmoonycode -o tools/libmorph_check
|
||
```
|
||
|
||
If `tools/libmorph_check` is absent, Stage 2 still runs — it simply drops libmorph from
|
||
the stack and reports `libmorph_helper=MISSING`. **Build it before a release rebuild:** the
|
||
committed word lists were produced *with* libmorph, and dropping it shifts the result by a few
|
||
hundred words (libmorph-only nouns disappear, and adjectives/eponyms libmorph would veto —
|
||
неперов, венерин — leak in through the orthographic-note path). A rebuild meant to refresh the
|
||
committed lists is only faithful with the helper present.
|
||
|
||
## How to run
|
||
|
||
```sh
|
||
# Stage 0 — PDF -> plain text (committed as the source of truth; run once)
|
||
pdftotext tools/orfo_dict_2025.pdf sources/scrabble_ru/orfo_dict_2025.txt
|
||
|
||
# Stage 1 — build the base word list (Go): sources/scrabble_ru/all.txt + /tmp/ru_*.txt
|
||
go run ./tools/ruwords
|
||
|
||
# Stage 2 — the brain (Python + mawo + libmorph): writes scrabble.txt
|
||
ru-venv/bin/python tools/ru_stage2.py
|
||
|
||
# audit how a single word did or did not reach the dictionary (detailed, per-signal report)
|
||
ru-venv/bin/python tools/ru_stage2.py --trace ндс
|
||
# also write the in-memory buckets (undefined, adjectives, verbs, singulars, fate.tsv)
|
||
ru-venv/bin/python tools/ru_stage2.py --dump
|
||
```
|
||
|
||
`-from`/`-to` (defaulting to 452/168808) bound the column word-list section of
|
||
`russian/orfo_dict_2025.txt` (line 452 = the first entry `а1, …`; line 168808 = the last,
|
||
`я́щурный`). The preface above line 452 is prose and is skipped. Verify these bounds if the
|
||
PDF is re-exported.
|
||
|
||
## Algorithm
|
||
|
||
### Stage 1 — `ruwords` (Go)
|
||
|
||
Per dictionary line in `[from, to]` it collects, normalised (stress marks U+0300/U+0301
|
||
stripped, lowercased, `ё` kept, hyphenated/capitalised/non-Cyrillic rejected):
|
||
|
||
- the **headword** (leading token). Leading whitespace including the form-feed `\f`
|
||
pdftotext puts at every page top is trimmed — otherwise the first headword of each page
|
||
is lost;
|
||
- the **singular of a plural headword** when the entry gives it after `ед.`, in full
|
||
(`ящеры, …, ед. ящер`) or as a replacement suffix (`…, ед. -вец`, spliced where the
|
||
suffix best overlaps the headword); the plural is then dropped (a plural that has a
|
||
singular is never needed) and the singular is also recorded (`/tmp/ru_singulars.txt`);
|
||
- **variant headwords** after `и` that carry their own grammatical note
|
||
(`аблатив, -а и аблятив, -а`; `регги и реггей, нескл.`), excluding inflected forms.
|
||
|
||
Everything else (every maximal Cyrillic token not selected above) goes to
|
||
`/tmp/ru_skip.txt`, a safety net for a later morphology re-check.
|
||
|
||
### Stage 2 — `ru_stage2.py` (Python)
|
||
|
||
Each Stage-1 word (length 2–15) is routed by three sources, most authoritative first:
|
||
|
||
1. **OpenCorpora** (`words.dawg`, read directly — *not* the predictor): a common-noun
|
||
reading ⇒ keep the OpenCorpora lemma. The full OpenCorpora common-noun lexicon is also
|
||
added (so nouns absent from the PDF are included), **minus indeclinable abbreviations**
|
||
(`Abbr`+`Fixd`: ндс, кпд, чп, …) — OpenCorpora tags those as nouns but they are not
|
||
Scrabble words. See *Abbreviations, proper nouns & function words* below for the full rule.
|
||
2. **libmorph** (independent dictionary, via `libmorph_check`): a common-noun reading ⇒
|
||
keep the libmorph lemma. The two dictionaries are treated as **complementary** — a noun
|
||
reading in *either* is enough (their disagreements were reviewed and resolved this way,
|
||
since each is incomplete in different places). A singular reconstructed from "ед." that
|
||
neither dictionary knows is accepted as a noun (the orthographic note attests it).
|
||
3. A word **both dictionaries miss** is classified by the orthographic **note**
|
||
(`-ая, -ое` ⇒ adjective; `-ть`, `сов./несов.` ⇒ verb; single genitive `-а/-и` or
|
||
`нескл., м./ж./с.` ⇒ noun). A note-noun goes straight to `scrabble.txt`; an adjective or
|
||
verb is dropped; anything undecided goes to `undefined.txt`.
|
||
4. **Variant rescue**: when the dictionary joins two spellings with "и" (`травмопункт и
|
||
травмпункт`, `регги и реггей`) and one is already a confirmed noun, the other is moved
|
||
from review/undefined into the result as well, propagated transitively through chains.
|
||
The plural-form variants the dictionaries already resolve never reach this step.
|
||
5. **Manual overrides**: `manual_confirm.txt` adds maintainer-approved nouns from the
|
||
undefined tail (before variant rescue, so they can anchor it); `manual_reject.txt` is
|
||
subtracted **last**, removing both false nouns a dictionary misreads — substantivized
|
||
adjectives with no nominal use (`нёбный`, `акцизный`, …) — and the obscene lexicon (mat
|
||
on the хуй/еб·ёб/пизд/бляд roots: `хуй`, `пизда`, `блядь`, …), dropped as policy though
|
||
morphologically valid nouns — overriding every path above.
|
||
|
||
The nominative singular always comes from the dictionary that recognised the word, or from
|
||
the orthographic `ед.` note — never from a predictor guess (libmorph and the predictor
|
||
mis-lemmatise out-of-dictionary words, e.g. `витебчане → витебчан` instead of `витебчанин`).
|
||
|
||
### The libmorph bridge — `libmorph_check.cpp`
|
||
|
||
libmorph (A. Kovalenko, MIT) ships as `libmorphrus.so`. `libmorph_check` is a thin
|
||
stdin→stdout filter: one UTF-8 word per line in, one line out:
|
||
|
||
```
|
||
<known>\t<pos>:<lemma>\t<pos>:<lemma>...
|
||
```
|
||
|
||
`<known>` is `CheckWord` (1 = in the dictionary). `<pos>` is `wdInfo & 0x3f`, the part of
|
||
speech. The codes were reverse-engineered (the docs omit the table):
|
||
|
||
| codes | part of speech |
|
||
|------|----------------|
|
||
| **7–21, 24** | **noun** (all genders / declensions / animacy; pluralia tantum is 24) |
|
||
| 1–3 | verb · 25, 27 adjective · 28–32 pronoun · 33–36 numeral |
|
||
| 38–39 | **proper noun** (excluded) · 48–58 comparative/adverb · 49–53 function words |
|
||
|
||
The analyser instance is requested with the key `libmorph.api.v4:utf-8` so words are
|
||
passed and lemmas returned in UTF-8.
|
||
|
||
## Abbreviations, proper nouns & function words
|
||
|
||
**Abbreviations are not Scrabble words and are dropped** (вуз, нэп, гост, жэк, спид, огвз,
|
||
обком, наркомат, роно, тв, …). A final pass in `build` removes every word matched by either
|
||
signal, then re-admits the maintainer's rescues — see `is_abbrev_note`, `oc_abbr_only` and the
|
||
abbreviation subtraction in `ru_stage2.py`:
|
||
|
||
- **The orthographic note marks it `(сокр.)`** (`is_abbrev_note`): the РАН dictionary's own
|
||
abbreviation mark, covering both the bare initialisms (вуз, нэп, гост, бтр, кпд) and the
|
||
Soviet-era compounds (обком, райком, наркомат, госплан, …). This is the authoritative signal.
|
||
- **OpenCorpora reads it only as an abbreviation** (`oc_abbr_only`): *every* common-noun
|
||
reading carries the `Abbr` grammeme. This catches acronyms РАН does not head at all — the
|
||
declinable огвз (a full `Abbr` case paradigm, no `Fixd`, which the `Abbr`+`Fixd` seed filter
|
||
misses), заз, and the indeclinable роно, тв, фио, суперэвм. A homograph that *also* has a
|
||
plain-noun reading is **not** abbreviation-only and survives — ваза «ВАЗ», рис «РИС».
|
||
|
||
Still dropped by the seed/loop before this pass: a `CONJ`/`PRCL` **function word** even with a
|
||
noun-like note (**зато**), and an OpenCorpora reading that is an **inflected form** of another
|
||
lemma (**ан** = род. мн. от «ана»; the proper noun «Ан» is capitalised and never reaches
|
||
`all.txt`). Proper nouns (`Name`/`Surn`/`Geox`/`Orgn`/…) are excluded from the seed outright.
|
||
|
||
**Maintainer overrides decide the borderline cases** (this is policy, not a heuristic):
|
||
|
||
- `manual_confirm.txt` **rescues** lexicalised words that carry `(сокр.)` or that OpenCorpora
|
||
only knows as an abbreviation but which read as ordinary nouns — **колхоз, совхоз, рация,
|
||
спецназ, эсминец, линкор, дзот, дот, прораб, токамак, тэн, …**, and **под** (a real noun,
|
||
«под печи», that OpenCorpora knows only as the abbreviation ПОД). Edit this list to widen or
|
||
narrow which abbreviations stay.
|
||
- `manual_reject.txt` **vetoes** the few non-noun leftovers that no signal catches — e.g. the
|
||
bound prefix **гос** («гос… — первая часть сложных слов», not a standalone word).
|
||
|
||
> History note: an earlier policy *kept* lexicalised abbreviations (сельпо, вуз, нэп, …) and
|
||
> used `(сокр.)` only as a non-discriminator. The current policy drops all abbreviations and
|
||
> rescues a curated subset, so the dictionary holds words, not acronyms.
|
||
|
||
### Auditing a word and repeating this analysis
|
||
|
||
`--trace WORD` prints a per-signal audit — Stage-1/`all.txt` membership, every OpenCorpora
|
||
reading (POS, `Abbr`/`Fixd`, `CONJ`/`PRCL`), libmorph, the РАН note with its `classify`
|
||
verdict, and the final outcome:
|
||
|
||
```sh
|
||
ru-venv/bin/python tools/ru_stage2.py --trace ндс # → НЕ в словаре: исключено из посева
|
||
ru-venv/bin/python tools/ru_stage2.py --trace сельпо # → В СЛОВАРЕ: по помете орфословаря
|
||
```
|
||
|
||
To find a *whole class* of suspect words (the procedure used to catch the abbreviations above),
|
||
read the signals straight from the dictionaries with the same primitives the pipeline uses:
|
||
|
||
1. iterate `M._dawg_dict.words_dawg` and collect the lemmas with the grammemes you suspect
|
||
(e.g. `Abbr`+`Fixd` on the lemma, no clean noun paradigm) — the candidate set;
|
||
2. split it by `w in all.txt` (lowercase РАН headword), by `classify(w, note)` (РАН-note POS),
|
||
and by an OpenCorpora `CONJ`/`PRCL` reading — these three signals separate real words from
|
||
abbreviations / proper nouns / function words;
|
||
3. review the candidate list by hand, add the distinguishing grammeme to the filter, re-run,
|
||
and diff the rebuilt list against the previous one to confirm only the intended words moved
|
||
(`--trace` each borderline decision).
|
||
|
||
## Notes & caveats
|
||
|
||
- The hard tail (≈ 35 000 Stage-1 words / our candidates) is in **no** morphological
|
||
dictionary; only the orthographic dictionary attests them, so the PDF note is the sole
|
||
signal there. Compound and very recent nouns (`робототехник`, `толкинист`) live here.
|
||
- OpenCorpora and libmorph are near-equal in size (≈ 99 500 words each on `all.txt`)
|
||
and ≈ 96 % overlapping, but **complementary** (each contributes ≈ 2 200 unique nouns),
|
||
which is why both are kept. The mawo *predictor* "knows" ~98 % of everything by guessing
|
||
and is therefore used only as a weak confirming vote, never as dictionary membership.
|
||
- Licensing: OpenCorpora data is CC BY-SA 3.0; libmorph is MIT; the orthographic
|
||
dictionary has its own copyright. A list derived from CC BY-SA data inherits that licence.
|