feat(ru): drop obscene lexicon (mat) from the Russian noun list
build / dawg (pull_request) Successful in 24s
build / dawg (pull_request) Successful in 24s
Remove the 11 obscene nouns built on the хуй/еб·ёб/пизд/бляд roots (ахуй, блядь, выблядок, ебальник, еблан, охуение, пизда, пиздец, пиздобол, хуй, хуйня) from the Russian Scrabble/Эрудит dictionary. These are morphologically valid common nouns, so the pipeline admitted them like any other word — OpenCorpora's tagset exposes no obscenity grammeme to filter on. They are vetoed as policy via manual_reject.txt, which is subtracted last in ru_stage2.build(), so no pipeline logic changes: scrabble.txt is the deterministic result of that subtraction and erudit.txt is its Ё→Е fold, regenerated with tools/fold_yo.py. The coarser-but-not-mat муд* (мудак, мудила) and legitimate homographs (очко, сука, моча, дрочёна) are deliberately kept. scrabble.txt 83027 -> 83016, erudit.txt 82985 -> 82974.
This commit is contained in:
@@ -70,3 +70,16 @@
|
||||
щелевой
|
||||
щелинный
|
||||
эскадронный
|
||||
# обсценная лексика (мат), корни хуй / еб·ёб / пизд / бляд — вычищается как
|
||||
# политика, а не как «ложное существительное»; строки с «#» отбрасывает cyr_ok
|
||||
ахуй
|
||||
блядь
|
||||
выблядок
|
||||
ебальник
|
||||
еблан
|
||||
охуение
|
||||
пизда
|
||||
пиздец
|
||||
пиздобол
|
||||
хуй
|
||||
хуйня
|
||||
|
||||
Reference in New Issue
Block a user