feat(ru): drop abbreviations from the Russian noun list
build / dawg (pull_request) Successful in 6m4s

The list leaked abbreviations two ways: an OpenCorpora noun tagged Abbr without
Fixd (the declinable acronym ОГВЗ) slipped past the Abbr+Fixd seed filter, and the
РАН «(сокр.)» words (вуз, нэп, колхоз, гост, …) were kept on purpose by the old
policy.

Drop every abbreviation instead. A final pass in ru_stage2.py removes a word when
the orthographic note marks it «(сокр.)» (is_abbrev_note) or when every OpenCorpora
noun reading carries Abbr (oc_abbr_only — also the declinable огвз/заз the seed
filter missed). Homographs with a plain-noun reading survive (ваза «ВАЗ», рис).
Lexicalised words are rescued via manual_confirm.txt (колхоз, совхоз, рация,
спецназ, эсминец, дзот, дот, прораб, токамак, тэн, … and под — a real noun OC knows
only as ПОД); the bound prefix гос is vetoed in manual_reject.txt.

108 words removed from ru_scrabble / ru_erudit; en_sowpods unchanged. tools/README
documents the policy and that libmorph is required for a faithful rebuild.
This commit is contained in:
Ilia Denisov
2026-06-22 14:16:55 +02:00
parent 1f5e1906ad
commit 74ff3aefb6
6 changed files with 109 additions and 242 deletions
+46
View File
@@ -170,6 +170,38 @@ def oc_function_word(word):
return False
def oc_abbr_only(word):
"""True when every common-noun reading OpenCorpora gives word is an abbreviation (carries
the Abbr grammeme on its paradigm). Generalises oc_abbr_fixd_only past the Fixd
(indeclinable) requirement, so it also catches a *declinable* acronym that OpenCorpora
inflects through a full Abbr case paradigm without Fixd — огвз, заз — as well as the
indeclinable роно, тв, фио. A word with any non-Abbr noun reading (the homographs ваза
«ВАЗ», рис «РИС», под «ПОД»/под печи) is not abbreviation-only and is left alone."""
parses = D.get_word_parses(word)
if not parses:
return False
gp, pt = D.get_paradigm, D.parse_tag_string
has_noun = has_plain = False
for pid, idx in parses:
suf, tag, pre = gp(pid, idx)
pos, gr = pt(tag)
if pos != "NOUN":
continue
_, tag0, _ = gp(pid, 0)
_, gr0 = pt(tag0)
if PROPER & gr or PROPER & gr0:
continue
has_noun = True
if "Abbr" not in gr0:
has_plain = True
return has_noun and not has_plain
def is_abbrev_note(note):
"""True when the orthographic note marks an abbreviation («сокр.», e.g. вуз, нэп, колхоз)."""
return bool(note) and "сокр" in note
def libmorph_analyze(words):
"""Map each word to (known, noun_lemma, codes) per libmorph; noun_lemma is None when it
is not a common noun there. Empty result if the helper binary is not built."""
@@ -353,6 +385,20 @@ def build():
fate[y] = f"scrabble: вариант от «{x}» (через «и»)"
changed = True
# Abbreviations are not Scrabble words. Two independent signals catch them: the orthographic
# note marks «(сокр.: …)» (вуз, нэп, колхоз, гост, …); and OpenCorpora tags acronyms Abbr —
# both the indeclinable (роно, тв, фио) and the declinable ones the Abbr+Fixd seed filter
# misses (огвз, заз, гулаг). Drop every such word, except the lexicalised ones the maintainer
# rescues in manual_confirm.txt (колхоз, совхоз, рация, спецназ, … and под — a real noun
# OpenCorpora knows only as the abbreviation ПОД). The homographs that are real words with a
# mere abbreviation reading (ваза, рис) survive — oc_abbr_only requires *all* noun readings
# to be Abbr. Subtracted before the manual-reject veto.
confirm = {w for w in load(os.path.join(OUT_DIR, "manual_confirm.txt")) if cyr_ok(w)}
abbrev = {w for w in scrabble if w not in confirm and (is_abbrev_note(hmap.get(w)) or oc_abbr_only(w))}
scrabble -= abbrev
for w in abbrev:
fate[w] = "отброшено: аббревиатура (помета «сокр.» орфословаря или только Abbr-чтения в OpenCorpora)"
# Manual rejections: words the maintainer vetoed — substantivized-adjective false nouns
# a dictionary misreads as nouns (нёбный, акцизный, …). Subtracted last, so it overrides
# every admission path above (OC seed / libmorph / note / manual_confirm / variant).