Remove the 11 obscene nouns built on the хуй/еб·ёб/пизд/бляд roots
(ахуй, блядь, выблядок, ебальник, еблан, охуение, пизда, пиздец,
пиздобол, хуй, хуйня) from the Russian Scrabble/Эрудит dictionary.
These are morphologically valid common nouns, so the pipeline admitted
them like any other word — OpenCorpora's tagset exposes no obscenity
grammeme to filter on. They are vetoed as policy via manual_reject.txt,
which is subtracted last in ru_stage2.build(), so no pipeline logic
changes: scrabble.txt is the deterministic result of that subtraction
and erudit.txt is its Ё→Е fold, regenerated with tools/fold_yo.py.
The coarser-but-not-mat муд* (мудак, мудила) and legitimate homographs
(очко, сука, моча, дрочёна) are deliberately kept.
scrabble.txt 83027 -> 83016, erudit.txt 82985 -> 82974.
The list leaked abbreviations two ways: an OpenCorpora noun tagged Abbr without
Fixd (the declinable acronym ОГВЗ) slipped past the Abbr+Fixd seed filter, and the
РАН «(сокр.)» words (вуз, нэп, колхоз, гост, …) were kept on purpose by the old
policy.
Drop every abbreviation instead. A final pass in ru_stage2.py removes a word when
the orthographic note marks it «(сокр.)» (is_abbrev_note) or when every OpenCorpora
noun reading carries Abbr (oc_abbr_only — also the declinable огвз/заз the seed
filter missed). Homographs with a plain-noun reading survive (ваза «ВАЗ», рис).
Lexicalised words are rescued via manual_confirm.txt (колхоз, совхоз, рация,
спецназ, эсминец, дзот, дот, прораб, токамак, тэн, … and под — a real noun OC knows
only as ПОД); the bound prefix гос is vetoed in manual_reject.txt.
108 words removed from ru_scrabble / ru_erudit; en_sowpods unchanged. tools/README
documents the policy and that libmorph is required for a faithful rebuild.
Follow-up to v1.2.0: reviewing the kept low-confidence noun verdicts surfaced
adjective-primary words admitted only by colloquial ellipsis, while their twins
were already rejected — notably большой (kept) vs маленький (dropped).
Adds to manual_reject.txt: большой, старшенький, горбатый, городской, одинокий,
вольный, каторжный, припадочный, сподручный. Their real nouns are different
words (горбун, горожанин, одиночка, каторжник) or the sense is marginal/устар.
scrabble.txt -9 (83144->83135); erudit.txt re-folded.
OpenCorpora/libmorph hand a noun reading to Stage 2 above the РАН note, so
substantivized-adjective false nouns a dictionary misreads (нёбный, акцизный,
велярный, …) reached scrabble.txt with no veto path — the earlier abbreviation
filter only carved out Abbr+Fixd.
ru_stage2.py now subtracts sources/scrabble_ru/manual_reject.txt last, after
every admission path (OC seed / libmorph / note / manual_confirm / variant),
mirroring manual_confirm.txt. The list seeds 62 hand-reviewed words — 43 pure
adjectives plus 19 marginal (slang/archaic/jargon) substantivizations; genuine
substantivized nouns (больной, знакомый, учёный, участковый, …) stay.
scrabble.txt -62 (83206->83144); erudit.txt re-folded -62. The DAWGs are
gitignored and rebuild from these lists in CI.
OpenCorpora tags indeclinable abbreviations as common nouns (Abbr+Fixd), and
Stage 2 seeded the result with its whole noun lexicon, so non-words leaked into
scrabble.txt: ндс, ст, ср, кпд, чп, гибдд, днк, …
ru_stage2.py now drops Abbr+Fixd nouns from the OpenCorpora seed and lets the
orthographic dictionary decide instead: a lowercase РАН headword whose note is a
noun is kept (the lexicalised сельпо, под, ска, роно, врио, тв, фио, суперэвм),
everything else is dropped. Function words (CONJ/PRCL, e.g. зато) and OpenCorpora
inflected forms (ан = род. мн. от «ана»; proper «Ан») are excluded too.
--trace WORD now prints a detailed per-signal audit (all.txt, OpenCorpora POS /
Abbr / Fixd / CONJ/PRCL, libmorph, РАН note + classify, outcome) for auditing a
word or repeating the analysis; tools/README.md documents the rule and method.
Net: 179 words removed from scrabble.txt (83385 -> 83206) and erudit.txt
(83343 -> 83164); verified to remove exactly those 179 and add none. DAWGs
rebuild clean.
The dict pipeline was moved out of scrabble-solver (256999b) into this repo,
but the initial import (d04470b) dropped three primary sources, and the
"Tidy sources" reorg (dd61ff1) git-ignored them while tools/README.md still
documented them as the committed source of truth — so the word-provenance
analysis could no longer run.
Recover them byte-identically from scrabble-solver's history (reachable from
its v1.0.0 tag):
- tools/orfo_dict_2025.pdf primary source (RAS orthographic dictionary PDF)
- sources/scrabble_ru/orfo_dict_2025.txt pdftotext output, pipeline source of truth
- sources/scrabble_ru/all.txt Stage-1 base, input to Stage 2
Stop ignoring them in .gitignore and fix the contradictory note in
sources/scrabble_ru/README.md (only the debug dumps are regenerated locally).
Consolidate the scattered build inputs (dictionaries/english/, dictprep/russian/)
into one sources/ tree keyed by the variant labels (scrabble_en/scrabble_ru/
erudit_ru), and move the Russian prep pipeline to tools/. The dawg outputs and
their filenames are unchanged — rebuilt byte-identical (en_sowpods/ru_scrabble/
ru_erudit) — so the release artifact and the backend are unaffected.
ru_stage2.py OUT_DIR and the ruwords flag defaults are repointed to
sources/scrabble_ru/; Makefile / CI / cmd/builddict default / README updated;
pipeline intermediates git-ignored. Verified: make dawg byte-identical to the
committed baseline, py_compile + go vet of the moved tools. The full Russian
regeneration pipeline (pymorphy3/libmorph/orfo PDF) was not run here.
- build.yaml dropped the release step whose ${{ github.* }} contexts failed the Gitea
workflow compile (the run produced 0 jobs); it now inlines go run (no make dependency)
and reads the committed dictprep/russian/erudit.txt (no python dependency).
- erudit.txt is scrabble.txt with Ё→Е folded (dictprep/fold_yo.py); it reproduces the
canonical ru_erudit.dawg byte-for-byte. Release artifacts are published manually for now
(see README).
- builddict drives the de-internalized scrabble-solver dictdawg/wordlist builders
(pinned v1.0.0) to produce the three DAWGs (en_sowpods, ru_scrabble, ru_erudit),
byte-identical to the solver's committed fixtures (same dafsa/alphabet v1.1.0 -> no
index drift with the running backend).
- Sources: english/sowpods.txt vendored from kamilmielnik/scrabble-dictionaries;
russian/scrabble.txt + the dictprep tooling moved out of scrabble-solver.
- CI builds the DAWGs on push/PR and, on a vX.Y.Z tag, packages them flat into
scrabble-dawg-<tag>.tar.gz and attaches it to the Gitea release.
developer
Ilia Denisov