feat(ru): drop abbreviations from the Russian noun list
build / dawg (pull_request) Successful in 6m4s
build / dawg (pull_request) Successful in 6m4s
The list leaked abbreviations two ways: an OpenCorpora noun tagged Abbr without Fixd (the declinable acronym ОГВЗ) slipped past the Abbr+Fixd seed filter, and the РАН «(сокр.)» words (вуз, нэп, колхоз, гост, …) were kept on purpose by the old policy. Drop every abbreviation instead. A final pass in ru_stage2.py removes a word when the orthographic note marks it «(сокр.)» (is_abbrev_note) or when every OpenCorpora noun reading carries Abbr (oc_abbr_only — also the declinable огвз/заз the seed filter missed). Homographs with a plain-noun reading survive (ваза «ВАЗ», рис). Lexicalised words are rescued via manual_confirm.txt (колхоз, совхоз, рация, спецназ, эсминец, дзот, дот, прораб, токамак, тэн, … and под — a real noun OC knows only as ПОД); the bound prefix гос is vetoed in manual_reject.txt. 108 words removed from ru_scrabble / ru_erudit; en_sowpods unchanged. tools/README documents the policy and that libmorph is required for a faithful rebuild.
This commit is contained in:
@@ -170,6 +170,38 @@ def oc_function_word(word):
|
||||
return False
|
||||
|
||||
|
||||
def oc_abbr_only(word):
|
||||
"""True when every common-noun reading OpenCorpora gives word is an abbreviation (carries
|
||||
the Abbr grammeme on its paradigm). Generalises oc_abbr_fixd_only past the Fixd
|
||||
(indeclinable) requirement, so it also catches a *declinable* acronym that OpenCorpora
|
||||
inflects through a full Abbr case paradigm without Fixd — огвз, заз — as well as the
|
||||
indeclinable роно, тв, фио. A word with any non-Abbr noun reading (the homographs ваза
|
||||
«ВАЗ», рис «РИС», под «ПОД»/под печи) is not abbreviation-only and is left alone."""
|
||||
parses = D.get_word_parses(word)
|
||||
if not parses:
|
||||
return False
|
||||
gp, pt = D.get_paradigm, D.parse_tag_string
|
||||
has_noun = has_plain = False
|
||||
for pid, idx in parses:
|
||||
suf, tag, pre = gp(pid, idx)
|
||||
pos, gr = pt(tag)
|
||||
if pos != "NOUN":
|
||||
continue
|
||||
_, tag0, _ = gp(pid, 0)
|
||||
_, gr0 = pt(tag0)
|
||||
if PROPER & gr or PROPER & gr0:
|
||||
continue
|
||||
has_noun = True
|
||||
if "Abbr" not in gr0:
|
||||
has_plain = True
|
||||
return has_noun and not has_plain
|
||||
|
||||
|
||||
def is_abbrev_note(note):
|
||||
"""True when the orthographic note marks an abbreviation («сокр.», e.g. вуз, нэп, колхоз)."""
|
||||
return bool(note) and "сокр" in note
|
||||
|
||||
|
||||
def libmorph_analyze(words):
|
||||
"""Map each word to (known, noun_lemma, codes) per libmorph; noun_lemma is None when it
|
||||
is not a common noun there. Empty result if the helper binary is not built."""
|
||||
@@ -353,6 +385,20 @@ def build():
|
||||
fate[y] = f"scrabble: вариант от «{x}» (через «и»)"
|
||||
changed = True
|
||||
|
||||
# Abbreviations are not Scrabble words. Two independent signals catch them: the orthographic
|
||||
# note marks «(сокр.: …)» (вуз, нэп, колхоз, гост, …); and OpenCorpora tags acronyms Abbr —
|
||||
# both the indeclinable (роно, тв, фио) and the declinable ones the Abbr+Fixd seed filter
|
||||
# misses (огвз, заз, гулаг). Drop every such word, except the lexicalised ones the maintainer
|
||||
# rescues in manual_confirm.txt (колхоз, совхоз, рация, спецназ, … and под — a real noun
|
||||
# OpenCorpora knows only as the abbreviation ПОД). The homographs that are real words with a
|
||||
# mere abbreviation reading (ваза, рис) survive — oc_abbr_only requires *all* noun readings
|
||||
# to be Abbr. Subtracted before the manual-reject veto.
|
||||
confirm = {w for w in load(os.path.join(OUT_DIR, "manual_confirm.txt")) if cyr_ok(w)}
|
||||
abbrev = {w for w in scrabble if w not in confirm and (is_abbrev_note(hmap.get(w)) or oc_abbr_only(w))}
|
||||
scrabble -= abbrev
|
||||
for w in abbrev:
|
||||
fate[w] = "отброшено: аббревиатура (помета «сокр.» орфословаря или только Abbr-чтения в OpenCorpora)"
|
||||
|
||||
# Manual rejections: words the maintainer vetoed — substantivized-adjective false nouns
|
||||
# a dictionary misreads as nouns (нёбный, акцизный, …). Subtracted last, so it overrides
|
||||
# every admission path above (OC seed / libmorph / note / manual_confirm / variant).
|
||||
|
||||
Reference in New Issue
Block a user