stenographer: prefix-40000

The word model for stenographer. You type a word by pressing its opening letters at once, like a stenographer. This model works out which word you meant from those letters and the sentence so far.

This is the checkpoint the daemon loads by default (scripts/stenod.py, --checkpoint runs/prefix-40000/checkpoint.pt).

What it is

• DistilBERT (distilbert-base-uncased) encodes the sentence so far, with the word being typed masked.
• A linear head scores the vocabulary.
• At typing time, candidates are limited to the words that begin with the pressed letters.

Training

• Data: Salesforce/wikitext, config wikitext-103-raw-v1, as written by scripts/prepare_corpus.py.

• Run: 40,000 steps and 10.24 million (context, word) examples, about 7.3 hours on one GPU. The context was trimmed to what the daemon actually has at typing time (./retrain).

• Held-out accuracy at the last step, measured on 3,000 word positions:

  • top 1: 85.7%
  • top 3: 94.8%
  • top 5: 96.8%

  history.json has the curve, and summary.json has the run's figures.

Files

• checkpoint.pt: the PyTorch checkpoint, loaded by BertPredictor.load in the repository. It is a pickle, so load it only from a source you trust.
• vocab/vocab.json and vocab/letters.npy: the vocabulary the head scores, and the letters index the chords use. Pass this folder as --vocab.

Use

git clone https://github.com/maximus-powers/stenographer
cd stenographer
# put checkpoint.pt in runs/prefix-40000/ and the vocab folder in data/vocab/
./steno