developer/scrabble-dictionary
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Restore lost Russian dict sources & drop OpenCorpora abbreviations #3

Merged
developer merged 2 commits from feature/restore-russian-sources into master 2026-06-13 11:51:29 +00:00
Conversation 2 Commits 2 Files Changed 9
+317880 -381
developer commented 2026-06-13 11:35:00 +00:00
Owner
Copy Link
Copy Source

Two related changes to the Russian word-list pipeline.

1. Restore lost source files (5b7a741)

The repo split dropped three primary sources the analysis pipeline needs; recovered byte-identically from scrabble-solver history (tag v1.0.0):

  • tools/orfo_dict_2025.pdf — РАН orthographic dictionary (primary source)
  • sources/scrabble_ru/orfo_dict_2025.txt — pdftotext output (source of truth)
  • sources/scrabble_ru/all.txt — Stage-1 base

Un-ignored them in .gitignore and fixed the contradictory README note.

2. Filter OpenCorpora abbreviations & proper nouns (6b8b176)

OpenCorpora tags indeclinable abbreviations as common nouns (Abbr+Fixd), so the seed leaked non-words: ндс, ст, ср, кпд, чп, гибдд, днк, …

ru_stage2.py now drops Abbr+Fixd nouns from the seed and defers to the orthographic note: lowercase РАН headwords that are nouns are kept (сельпо, под, ска, роно, врио, тв, фио, суперэвм); function words (зато) and OpenCorpora inflected forms (ан) are dropped. --trace WORD gives a detailed per-signal audit; tools/README.md documents the rule and the reproducible method.

Net: −179 words in scrabble.txt (83385→83206) and erudit.txt (83343→83164); verified to remove exactly those 179 and add none. DAWGs rebuild clean (CI gate).

Two related changes to the Russian word-list pipeline. ## 1. Restore lost source files (5b7a741) The repo split dropped three primary sources the analysis pipeline needs; recovered byte-identically from scrabble-solver history (tag v1.0.0): - `tools/orfo_dict_2025.pdf` — РАН orthographic dictionary (primary source) - `sources/scrabble_ru/orfo_dict_2025.txt` — pdftotext output (source of truth) - `sources/scrabble_ru/all.txt` — Stage-1 base Un-ignored them in `.gitignore` and fixed the contradictory README note. ## 2. Filter OpenCorpora abbreviations & proper nouns (6b8b176) OpenCorpora tags indeclinable abbreviations as common nouns (`Abbr`+`Fixd`), so the seed leaked non-words: ндс, ст, ср, кпд, чп, гибдд, днк, … `ru_stage2.py` now drops `Abbr`+`Fixd` nouns from the seed and defers to the orthographic note: lowercase РАН headwords that are nouns are kept (сельпо, под, ска, роно, врио, тв, фио, суперэвм); function words (зато) and OpenCorpora inflected forms (ан) are dropped. `--trace WORD` gives a detailed per-signal audit; `tools/README.md` documents the rule and the reproducible method. Net: −179 words in `scrabble.txt` (83385→83206) and `erudit.txt` (83343→83164); verified to remove exactly those 179 and add none. DAWGs rebuild clean (CI gate).
developer added 2 commits 2026-06-13 11:35:00 +00:00
Restore lost Russian dictionary sources from scrabble-solver history 5b7a741ec2 
The dict pipeline was moved out of scrabble-solver (256999b) into this repo,
but the initial import (d04470b) dropped three primary sources, and the
"Tidy sources" reorg (dd61ff1) git-ignored them while tools/README.md still
documented them as the committed source of truth — so the word-provenance
analysis could no longer run.

Recover them byte-identically from scrabble-solver's history (reachable from
its v1.0.0 tag):

- tools/orfo_dict_2025.pdf                 primary source (RAS orthographic dictionary PDF)
- sources/scrabble_ru/orfo_dict_2025.txt   pdftotext output, pipeline source of truth
- sources/scrabble_ru/all.txt              Stage-1 base, input to Stage 2

Stop ignoring them in .gitignore and fix the contradictory note in
sources/scrabble_ru/README.md (only the debug dumps are regenerated locally).
Filter OpenCorpora abbreviations & proper nouns from the Russian noun list
build / dawg (pull_request) Successful in 2m45s
Details
6b8b176f82 
OpenCorpora tags indeclinable abbreviations as common nouns (Abbr+Fixd), and
Stage 2 seeded the result with its whole noun lexicon, so non-words leaked into
scrabble.txt: ндс, ст, ср, кпд, чп, гибдд, днк, …

ru_stage2.py now drops Abbr+Fixd nouns from the OpenCorpora seed and lets the
orthographic dictionary decide instead: a lowercase РАН headword whose note is a
noun is kept (the lexicalised сельпо, под, ска, роно, врио, тв, фио, суперэвм),
everything else is dropped. Function words (CONJ/PRCL, e.g. зато) and OpenCorpora
inflected forms (ан = род. мн. от «ана»; proper «Ан») are excluded too.

--trace WORD now prints a detailed per-signal audit (all.txt, OpenCorpora POS /
Abbr / Fixd / CONJ/PRCL, libmorph, РАН note + classify, outcome) for auditing a
word or repeating the analysis; tools/README.md documents the rule and method.

Net: 179 words removed from scrabble.txt (83385 -> 83206) and erudit.txt
(83343 -> 83164); verified to remove exactly those 179 and add none. DAWGs
rebuild clean.
owner approved these changes 2026-06-13 11:35:36 +00:00
Dismissed
owner approved these changes 2026-06-13 11:43:27 +00:00
developer merged commit b5600771a6 into master 2026-06-13 11:51:29 +00:00
developer deleted branch feature/restore-russian-sources 2026-06-13 11:51:30 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
owner
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
owner (Ilia) developer
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: developer/scrabble-dictionary#3
Delete Branch "feature/restore-russian-sources"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?