Nokia123543/muscriptor-large-fork

MuScriptor Large — transcription toolkit (fork)

Two post-processors that make a MuScriptor Large transcription measurably better without touching a single weight. Both run on CPU, in seconds, next to the model.

• transcript_fix.py — repairs what the decoder loses in difficult passages: onset timing, ghost notes, duplicates, and gaps.
• muscriptor_dynamics.py — adds the velocity MuScriptor does not predict, so forte and piano reach the MIDI.

Upstream weights: MuScriptor/muscriptor-large (5.09 GB, 48 layers, dim 1536, card 1395). They are not copied here — see What is not here.

Why post-processing, and not a better model

MuScriptor's own output describes a note to about one analysis hop. In the model's tokenizer, velocity is a two-valued flag (1 for onset, 0 for offset) inside EventRange("velocity", 0, 1), and the whole output is a single head, linears.0.weight of shape [1395, 1536]. There is no loudness dimension to recover, and no code edit can create one: the extra rows would be untrained numbers. So loudness is measured from the recording at the onsets the model already placed, and the note-level errors are repaired against the same audio.

transcript_fix.py

stage	what it does
onset refinement	the flux peak lags the attack by roughly the analysis window, so each note's onset is moved to the sharpest rise in local energy (1 ms grid)
ghost removal	a note whose onset window is far below the piece's own floor was never struck
duplicate merge	same pitch within 45 ms is one note; the longer one wins
gap filling	an onset with no note nearby is filled only if autocorrelation is confident and the pitch is one this piece already uses, capped at a third of the note count

Benchmark (python3 transcript_fix.py --selftest): a synthesized 12-note passage with three notes lost, one ghost, one duplicate and ±20 ms onset jitter, scored one-to-one at 50 ms tolerance.

до правки:  P=0.82 R=0.75 F1=0.78, ошибка онсета 10 мс, нот 11
после:      P=0.77 R=0.83 F1=0.80, ошибка онсета  2 мс, нот 13

Onset error falls five-fold (10 ms → 2 ms), recall rises 0.75 → 0.83, F1 0.78 → 0.80, the ghost is removed. Precision dips slightly (0.82 → 0.77) because three notes are added and one of them is a semitone off — an honest trade, reported here rather than hidden by tuning the benchmark.

muscriptor_dynamics.py

Measures each note inside a 250 ms window at its onset: level in dB, and attack as peak/RMS, which separates a hammer blow from a bowed swell at equal loudness. The piece's own 4th and 99.7th percentiles become the reference, the two are combined multiplicatively, and the result is written as velocity 1..124.

Self-test (python3 muscriptor_dynamics.py --selftest): eight notes of known amplitudes return correlation 0.958, soft [3, 48, 27] against loud [124, 124, 124].

Usage

from muscriptor_dynamics import load_audio, read_midi, write_midi
from transcript_fix import improve

samples, rate = load_audio("song.wav")
notes, tempo, _ = read_midi("muscriptor.mid")     # what the box returned
fixed, report = improve(notes, samples, rate)      # repair + velocity
write_midi("song.fixed.mid", fixed, 60000000.0 / tempo[0][1])

Both print progress in tens of percent. Both are one pass over the data: no retries, no loops, no GPU.

merge_transcriptions.py — the ensemble

A single pass makes two kinds of mistake that do not repeat: one pass drops the blurry note in a fast run, another invents a ghost in a reverb tail. Running the model a few times — different sizes, different instrument conditioning — and keeping only what the passes agree on removes both.

python3 merge_transcriptions.py song.wav out.mid large.mid medium.mid piano.mid

Notes cluster by pitch and onset within 60 ms; a cluster survives on --votes (default 2) distinct passes; its onset is the median of the passes, which is steadier than any single one; velocity is then measured from the audio at that agreed onset. --keep-strong rescues singletons whose attack is loud and clean — a fortissimo note in a quiet passage is heard once and is real.

Benchmark (--selftest): three passes over the same 14 notes, each losing a different three and inventing one ghost.

проход 1: P=0.92 R=0.79 F1=0.85
проход 2: P=0.92 R=0.79 F1=0.85
проход 3: P=0.92 R=0.79 F1=0.85
слияние:  P=1.00 R=1.00 F1=1.00, ошибка 4.6 мс, нот 14
отчёт: clusters 17 -> agreed 14, dropped_doubtful 3

F1 0.85 → 1.00, every ghost gone, onset error 4.6 ms. The gain is not free: it costs one extra model pass per opinion, which is why this is the one global improvement that does not need training but does need the GPU the model already runs on.

Limits, stated plainly

• Overlapping notes share a measurement. A chord's members measured in one window come out at similar velocities; per-voice dynamics needs source separation or a model that predicts velocity.
• The audio must be the one that was transcribed. No alignment step, so a different edit puts the windows in the wrong places.
• An ensemble needs passes that actually differ. Three runs with the same conditioning produce the same mistakes and vote them through.
• Gap filling is conservative by design and refuses ambiguous pitches; in dense legato writing it will leave gaps rather than invent notes.
• Dynamics are relative to the piece, since the reference points are its own percentiles.
• The timing repair assumes percussive attacks. On long bowed or sung notes the energy rise is gradual and the refinement moves less, which is correct but less dramatic.

What is not here, and what a real retrain needs

Weights, configs and the training code are not mirrored: the weights are CC-BY-NC-4.0 (non-commercial) and 5 GB, and copying them adds nothing to the code that is published here.

Real model-level gains — velocity that survives overlapping voices, better recall in blurred passages — need a velocity head added to the card (e.g. eight buckets: card 1395 → 1403), trained on performance MIDI with per-note dynamics (MAESTRO and similar) against the same audio, with the box's training code and a GPU for days. That is a different project from this repository, and it is the only route to dynamics that a post-processor cannot reach.

License

Code in this repository: MIT, same as MuScriptor's own code. The upstream weights it is designed for remain CC-BY-NC-4.0, so the combined pipeline is non-commercial.