scrabble-dictionary/tools/README.md

Russian word-list preparation (tools)

Builds the Russian noun word list for the Scrabble/Эрудит solver out of the official Russian academic orthographic dictionary, cross-checked against two independent morphological dictionaries.

The goal of the pipeline is a list of common nouns in the nominative singular (sources/scrabble_ru/scrabble.txt), plus an ambiguous tail for manual review.

This directory is self-contained tooling for building the word list. It is not part of the solver library. The committed result lives in sources/scrabble_ru/.

Source

orfo_dict_2025.pdf — Русский орфографический словарь РАН (≈ 200 000 entries), the authority for spelling. It encodes declension type in its grammatical notes but does not reliably mark part of speech.

• Source: https://ruslang.ru/sites/default/files/doc/normativnyje_slovari/orfograficheskij_slovar.pdf
• Mirror: https://rus-gos.spbu.ru/index.php/dictionary

The PDF is git-ignored (large, third-party); place it here as orfo_dict_2025.pdf. Its pdftotext output is committed as russian/orfo_dict_2025.txt, so the word list rebuilds from the text alone — the binary PDF is needed only to regenerate that text.

Outputs (sources/scrabble_ru/)

The committed result is three files; every other bucket stays in the Stage-2 process's memory (dump it with --dump, query it with --trace WORD).

File	Committed	Meaning
orfo_dict_2025.txt	✓	the pdftotext output — the parsed source of truth (the PDF binary is not needed to rebuild).
all.txt	✓	Stage 1 base: every clean Cyrillic headword/variant; a plural headword with a singular is replaced by that singular.
manual_confirm.txt	✓	hand-reviewed nouns from the undefined tail; the brain merges them into the result.
manual_reject.txt	✓	hand-vetoed words subtracted from the result last: false nouns (substantivized adjectives a dictionary misreads as nouns, e.g. нёбный) and obscene lexicon (mat, e.g. хуй, блядь).
scrabble.txt	✓	Stage 2 result: common nouns, nominative singular (+ pluralia tantum), length 2–15 — the working dictionary.
undefined.txt	—	the ambiguous tail; kept in memory, written only with --dump.

--dump also writes adjectives.txt, verbs.txt, singulars.txt and fate.tsv (every word with the reason it did or did not reach the dictionary); these are git-ignored debug artifacts. Stage 1 also writes /tmp/ru_{skip,singulars,variants}.txt, intermediate inputs the brain consumes.

Prerequisites

# 1. pdftotext (Poppler)
sudo apt-get install -y poppler-utils

# 2. Go toolchain (Stage 1) — already required by the parent module

# 3. Python + the OpenCorpora analyser (Stage 2)
sudo apt-get install -y python3-venv python3-pip
python3 -m venv ru-venv
ru-venv/bin/pip install mawo-pymorphy3            # bundles OpenCorpora 2025 (words.dawg)

# 4. libmorph — the independent morphological dictionary (Stage 2 cross-check)
sudo apt-get install -y morphrus morphrus-dev moonycode-dev morphapi-dev
g++ -std=c++17 -O2 tools/libmorph_check.cpp -lmorphrus -lmoonycode -o tools/libmorph_check

If tools/libmorph_check is absent, Stage 2 still runs — it simply drops libmorph from the stack and reports libmorph_helper=MISSING. Build it before a release rebuild: the committed word lists were produced with libmorph, and dropping it shifts the result by a few hundred words (libmorph-only nouns disappear, and adjectives/eponyms libmorph would veto — неперов, венерин — leak in through the orthographic-note path). A rebuild meant to refresh the committed lists is only faithful with the helper present.

How to run

# Stage 0 — PDF -> plain text (committed as the source of truth; run once)
pdftotext tools/orfo_dict_2025.pdf sources/scrabble_ru/orfo_dict_2025.txt

# Stage 1 — build the base word list (Go): sources/scrabble_ru/all.txt + /tmp/ru_*.txt
go run ./tools/ruwords

# Stage 2 — the brain (Python + mawo + libmorph): writes scrabble.txt
ru-venv/bin/python tools/ru_stage2.py

# audit how a single word did or did not reach the dictionary (detailed, per-signal report)
ru-venv/bin/python tools/ru_stage2.py --trace ндс
# also write the in-memory buckets (undefined, adjectives, verbs, singulars, fate.tsv)
ru-venv/bin/python tools/ru_stage2.py --dump

-from/-to (defaulting to 452/168808) bound the column word-list section of russian/orfo_dict_2025.txt (line 452 = the first entry а1, …; line 168808 = the last, я́щурный). The preface above line 452 is prose and is skipped. Verify these bounds if the PDF is re-exported.

Algorithm

Stage 1 — ruwords (Go)

Per dictionary line in [from, to] it collects, normalised (stress marks U+0300/U+0301 stripped, lowercased, ё kept, hyphenated/capitalised/non-Cyrillic rejected):

• the headword (leading token). Leading whitespace including the form-feed \f pdftotext puts at every page top is trimmed — otherwise the first headword of each page is lost;
• the singular of a plural headword when the entry gives it after ед., in full (ящеры, …, ед. ящер) or as a replacement suffix (…, ед. -вец, spliced where the suffix best overlaps the headword); the plural is then dropped (a plural that has a singular is never needed) and the singular is also recorded (/tmp/ru_singulars.txt);
• variant headwords after и that carry their own grammatical note (аблатив, -а и аблятив, -а; регги и реггей, нескл.), excluding inflected forms.

Everything else (every maximal Cyrillic token not selected above) goes to /tmp/ru_skip.txt, a safety net for a later morphology re-check.

Stage 2 — ru_stage2.py (Python)

Each Stage-1 word (length 2–15) is routed by three sources, most authoritative first:

1. OpenCorpora (words.dawg, read directly — not the predictor): a common-noun reading ⇒ keep the OpenCorpora lemma. The full OpenCorpora common-noun lexicon is also added (so nouns absent from the PDF are included), minus indeclinable abbreviations (Abbr+Fixd: ндс, кпд, чп, …) — OpenCorpora tags those as nouns but they are not Scrabble words. See Abbreviations, proper nouns & function words below for the full rule.
2. libmorph (independent dictionary, via libmorph_check): a common-noun reading ⇒ keep the libmorph lemma. The two dictionaries are treated as complementary — a noun reading in either is enough (their disagreements were reviewed and resolved this way, since each is incomplete in different places). A singular reconstructed from "ед." that neither dictionary knows is accepted as a noun (the orthographic note attests it).
3. A word both dictionaries miss is classified by the orthographic note (-ая, -ое ⇒ adjective; -ть, сов./несов. ⇒ verb; single genitive -а/-и or нескл., м./ж./с. ⇒ noun). A note-noun goes straight to scrabble.txt; an adjective or verb is dropped; anything undecided goes to undefined.txt.
4. Variant rescue: when the dictionary joins two spellings with "и" (травмопункт и травмпункт, регги и реггей) and one is already a confirmed noun, the other is moved from review/undefined into the result as well, propagated transitively through chains. The plural-form variants the dictionaries already resolve never reach this step.
5. Manual overrides: manual_confirm.txt adds maintainer-approved nouns from the undefined tail (before variant rescue, so they can anchor it); manual_reject.txt is subtracted last, removing both false nouns a dictionary misreads — substantivized adjectives with no nominal use (нёбный, акцизный, …) — and the obscene lexicon (mat on the хуй/еб·ёб/пизд/бляд roots: хуй, пизда, блядь, …), dropped as policy though morphologically valid nouns — overriding every path above.

The nominative singular always comes from the dictionary that recognised the word, or from the orthographic ед. note — never from a predictor guess (libmorph and the predictor mis-lemmatise out-of-dictionary words, e.g. витебчане → витебчан instead of витебчанин).

The libmorph bridge — libmorph_check.cpp

libmorph (A. Kovalenko, MIT) ships as libmorphrus.so. libmorph_check is a thin stdin→stdout filter: one UTF-8 word per line in, one line out:

<known>\t<pos>:<lemma>\t<pos>:<lemma>...

<known> is CheckWord (1 = in the dictionary). <pos> is wdInfo & 0x3f, the part of speech. The codes were reverse-engineered (the docs omit the table):

codes	part of speech
7–21, 24	noun (all genders / declensions / animacy; pluralia tantum is 24)
1–3	verb · 25, 27 adjective · 28–32 pronoun · 33–36 numeral
38–39	proper noun (excluded) · 48–58 comparative/adverb · 49–53 function words

The analyser instance is requested with the key libmorph.api.v4:utf-8 so words are passed and lemmas returned in UTF-8.

Abbreviations, proper nouns & function words

Abbreviations are not Scrabble words and are dropped (вуз, нэп, гост, жэк, спид, огвз, обком, наркомат, роно, тв, …). A final pass in build removes every word matched by either signal, then re-admits the maintainer's rescues — see is_abbrev_note, oc_abbr_only and the abbreviation subtraction in ru_stage2.py:

• The orthographic note marks it (сокр.) (is_abbrev_note): the РАН dictionary's own abbreviation mark, covering both the bare initialisms (вуз, нэп, гост, бтр, кпд) and the Soviet-era compounds (обком, райком, наркомат, госплан, …). This is the authoritative signal.
• OpenCorpora reads it only as an abbreviation (oc_abbr_only): every common-noun reading carries the Abbr grammeme. This catches acronyms РАН does not head at all — the declinable огвз (a full Abbr case paradigm, no Fixd, which the Abbr+Fixd seed filter misses), заз, and the indeclinable роно, тв, фио, суперэвм. A homograph that also has a plain-noun reading is not abbreviation-only and survives — ваза «ВАЗ», рис «РИС».

Still dropped by the seed/loop before this pass: a CONJ/PRCL function word even with a noun-like note (зато), and an OpenCorpora reading that is an inflected form of another lemma (ан = род. мн. от «ана»; the proper noun «Ан» is capitalised and never reaches all.txt). Proper nouns (Name/Surn/Geox/Orgn/…) are excluded from the seed outright.

Maintainer overrides decide the borderline cases (this is policy, not a heuristic):

• manual_confirm.txt rescues lexicalised words that carry (сокр.) or that OpenCorpora only knows as an abbreviation but which read as ordinary nouns — колхоз, совхоз, рация, спецназ, эсминец, линкор, дзот, дот, прораб, токамак, тэн, …, and под (a real noun, «под печи», that OpenCorpora knows only as the abbreviation ПОД). Edit this list to widen or narrow which abbreviations stay.
• manual_reject.txt vetoes the few non-noun leftovers that no signal catches — e.g. the bound prefix гос («гос… — первая часть сложных слов», not a standalone word).

History note: an earlier policy kept lexicalised abbreviations (сельпо, вуз, нэп, …) and used (сокр.) only as a non-discriminator. The current policy drops all abbreviations and rescues a curated subset, so the dictionary holds words, not acronyms.

Auditing a word and repeating this analysis

--trace WORD prints a per-signal audit — Stage-1/all.txt membership, every OpenCorpora reading (POS, Abbr/Fixd, CONJ/PRCL), libmorph, the РАН note with its classify verdict, and the final outcome:

ru-venv/bin/python tools/ru_stage2.py --trace ндс      # → НЕ в словаре: исключено из посева
ru-venv/bin/python tools/ru_stage2.py --trace сельпо   # → В СЛОВАРЕ: по помете орфословаря

To find a whole class of suspect words (the procedure used to catch the abbreviations above), read the signals straight from the dictionaries with the same primitives the pipeline uses:

1. iterate M._dawg_dict.words_dawg and collect the lemmas with the grammemes you suspect (e.g. Abbr+Fixd on the lemma, no clean noun paradigm) — the candidate set;
2. split it by w in all.txt (lowercase РАН headword), by classify(w, note) (РАН-note POS), and by an OpenCorpora CONJ/PRCL reading — these three signals separate real words from abbreviations / proper nouns / function words;
3. review the candidate list by hand, add the distinguishing grammeme to the filter, re-run, and diff the rebuilt list against the previous one to confirm only the intended words moved (--trace each borderline decision).

Notes & caveats

• The hard tail (≈ 35 000 Stage-1 words / our candidates) is in no morphological dictionary; only the orthographic dictionary attests them, so the PDF note is the sole signal there. Compound and very recent nouns (робототехник, толкинист) live here.
• OpenCorpora and libmorph are near-equal in size (≈ 99 500 words each on all.txt) and ≈ 96 % overlapping, but complementary (each contributes ≈ 2 200 unique nouns), which is why both are kept. The mawo predictor "knows" ~98 % of everything by guessing and is therefore used only as a weak confirming vote, never as dictionary membership.
• Licensing: OpenCorpora data is CC BY-SA 3.0; libmorph is MIT; the orthographic dictionary has its own copyright. A list derived from CC BY-SA data inherits that licence.