Dictionary Processing For Library of Babel

Root words only, remove inflections, plurals such as -ing -ly
Primitives only, remove theasaurus. Two words with the same definition. Keep the simpler word.

The scope of the library likely always require some tricks, workarounds.

The search engine format

The master word list has all the words in the English language. The user enters a search request. By the terms in the search request, a selection of keywords are selected by the probability that those keywords are highly probable as being part of the answer. The highest probability words are selected and the portion of library is generated. After the generation the results are also ranked by probability using a range of filters.

You type a request, for example “a lonely lighthouse keeper writing to his daughter”, and a length, say 5 words.
The system picks a short list of words it thinks could appear in a good answer. This is the keyword selection.
It builds sentences only from that short list. Instead of 82,639 words per position, it uses a few hundred, which makes the library small enough to generate.
It ranks the results so the most sensible-looking sentences come first.

Rule of thumb

2–5 words: per-slot lists, generate everything, then rank.

6+ words: per-slot lists plus beam search, so you never generate the full product.

Short sentences (2–5 words): generate everything from the small per-slot lists, then rank.

Long sentences (6+ words): beam search, building one word at a time and keeping only the best partial sentences.

The cost of speed: you may miss unusual answers whose beginnings look weak.

Building in versions is the right call. Each version adds one technique and has a clear pass/fail test before the next starts.

Version	What it does	New idea added	Works up to	Done when
---	---	---	---	---
v1	Keywords chosen from the request, one shared list, every sentence generated, simple filters rank them	The basic loop (this is `babel_find.c`, plus a thin web page and API)	to 4 words		A query returns sensible results in a few seconds
v2	Same as v1, but each slot has its own keyword list	Per-slot lists	to 5 words		The same compute budget gives visibly better results than v1
v3	Exact k-best Viterbi over small per-slot lists, bigram scoring	Dynamic programming, with a bigram table	+ words			Matches v2's top results on short lengths, then handles long lengths v2 cannot
v4	Viterbi over the full vocabulary, no keyword cutoff, results streamed best first	Whole-library coverage for the cheap score	+ words			Top results never depend on a keyword cutoff
v5	Beam search with a richer, whole-prefix score	Scoring that Viterbi cannot express	+ words			Beats v4's results on a test set of queries
v6	Mix of searches (for example Viterbi guiding beam), plus the rerank cascade of embeddings and an LLM judge	Combining engines	+ words			Beats each single engine on the same test set
v7	GPU and multi-server scaling, caching, hardening	Throughput	Many users	Meets a latency target under load

Rules for moving between versions**

- **One new idea per version.** If results change, you know why.

- **Keep a fixed test set** of about 30 queries at different lengths, and rerun it on every version. That is how you tell whether an upgrade actually helped.

- **Keep old versions runnable.** Each version serves as the baseline for the next one and as a fallback.

- **Don't mix engines before v6.** Until then, every version uses a single search technique.

What each stage needs before it starts**

- **v1:** nothing more than your word list. It is already built.

- **v3:** a bigram count table from a text corpus. That is the first real dependency, so decide where it comes from before v3.

- **v5:** a scorer that looks at a partial sentence as a whole (for example a small language model). This can wait until v4 is working.

I can write the v2 change next, which means per-slot keyword lists in `babel_find.c` and a small web API around it. Or I can add this roadmap to the design spec document, replacing the combined Viterbi and beam plan.

https://claude.ai/artifact/64woTeEFWrdMN4FxNFCrEN#bbeda3ea-2009.mc1kdp68dr3.9917

📝 📜 ⏱️ ⬆️