feat(ru): drop obscene lexicon (mat) from the Russian noun list
build / dawg (pull_request) Successful in 24s

Remove the 11 obscene nouns built on the хуй/еб·ёб/пизд/бляд roots
(ахуй, блядь, выблядок, ебальник, еблан, охуение, пизда, пиздец,
пиздобол, хуй, хуйня) from the Russian Scrabble/Эрудит dictionary.

These are morphologically valid common nouns, so the pipeline admitted
them like any other word — OpenCorpora's tagset exposes no obscenity
grammeme to filter on. They are vetoed as policy via manual_reject.txt,
which is subtracted last in ru_stage2.build(), so no pipeline logic
changes: scrabble.txt is the deterministic result of that subtraction
and erudit.txt is its Ё→Е fold, regenerated with tools/fold_yo.py.

The coarser-but-not-mat муд* (мудак, мудила) and legitimate homographs
(очко, сука, моча, дрочёна) are deliberately kept.

scrabble.txt 83027 -> 83016, erudit.txt 82985 -> 82974.
This commit is contained in:
Ilia Denisov
2026-06-30 07:01:00 +02:00
parent 44713e8a9a
commit 7f8165f16d
4 changed files with 18 additions and 25 deletions
+5 -3
View File
@@ -33,7 +33,7 @@ process's memory (dump it with `--dump`, query it with `--trace WORD`).
| `orfo_dict_2025.txt` | ✓ | the pdftotext output — the parsed source of truth (the PDF binary is not needed to rebuild). |
| `all.txt` | ✓ | Stage 1 base: every clean Cyrillic headword/variant; a plural headword with a singular is replaced by that singular. |
| `manual_confirm.txt` | ✓ | hand-reviewed nouns from the undefined tail; the brain merges them into the result. |
| `manual_reject.txt` | ✓ | hand-vetoed false nouns (substantivized adjectives a dictionary misreads as nouns, e.g. нёбный); subtracted from the result last. |
| `manual_reject.txt` | ✓ | hand-vetoed words subtracted from the result last: false nouns (substantivized adjectives a dictionary misreads as nouns, e.g. нёбный) and obscene lexicon (mat, e.g. хуй, блядь). |
| `scrabble.txt` | ✓ | **Stage 2 result**: common nouns, nominative singular (+ pluralia tantum), length 215 — the working dictionary. |
| `undefined.txt` | — | the ambiguous tail; kept in memory, written only with `--dump`. |
@@ -134,8 +134,10 @@ Each Stage-1 word (length 215) is routed by three sources, most authoritative
The plural-form variants the dictionaries already resolve never reach this step.
5. **Manual overrides**: `manual_confirm.txt` adds maintainer-approved nouns from the
undefined tail (before variant rescue, so they can anchor it); `manual_reject.txt` is
subtracted **last**, removing false nouns a dictionary misreads — substantivized
adjectives with no nominal use (`нёбный`, `акцизный`, …) — overriding every path above.
subtracted **last**, removing both false nouns a dictionary misreads — substantivized
adjectives with no nominal use (`нёбный`, `акцизный`, …) — and the obscene lexicon (mat
on the хуй/еб·ёб/пизд/бляд roots: `хуй`, `пизда`, `блядь`, …), dropped as policy though
morphologically valid nouns — overriding every path above.
The nominative singular always comes from the dictionary that recognised the word, or from
the orthographic `ед.` note — never from a predictor guess (libmorph and the predictor