FuseAlign

Abstract

Word-level forced alignment estimates when each transcript word occurs in a recording. It underpins text-based media editing, subtitling, speech-data curation, and phonetic analysis. Existing evaluations understate how hard the task is: they use short, clean speech, perfect transcripts, and metrics that hide the errors that matter. Real pipelines run on long, diverse recordings and on error-prone ASR output.

We address these gaps with improved evaluation metrics, a scoring protocol for real ASR transcripts, and AlignBench, a benchmark spanning diverse speaker, acoustic, and text conditions. We further introduce FuseAlign, a transformer aligner trained on large-scale pseudo-labeled speech with online label correction. It jointly contextualizes audio and text to localize words coarsely, then refines boundaries at millisecond resolution and detects missing transcript words, without lexicon-based or Viterbi decoding. On AlignBench, FuseAlign substantially outperforms all baselines and stays robust under real ASR transcripts.

Measuring alignment the way an edit experiences it

The conventional metrics are the absolute boundary error and per-boundary tolerance rates, computed over the words an aligner chose to place. Each metric misleads when the alignment is used for editing:

• Don't penalize placement in silence. A boundary predicted inside a pause next to a word is not harmful when the alignment is used for deleting the word. However, the same shift into a neighboring word clips it, so we penalize only the duration of speech clipped or swallowed and nothing for landing in silence.
• Worst edge instead of per-edge tolerance. An edit fails if either boundary is wrong, so a word is scored by max(start error, end error) and we report the share of words above 25 and 50 ms and of catastrophic errors (above 300 ms) rather than counting a word with one bad edge as half correct.
• Score every word. Aligners may reject certain words, so scoring only placed words lets a selective system look precise. To ensure fair comparison, an unplaced word gets a fixed fallback interval that divides the gap between its placed neighbours uniformly, and every system is scored on all the words.

Finally, we score each aligner on real ASR transcripts, not just hand-corrected ones. This mirrors real-world use cases where the transcript may not be perfect. We match ASR words to the reference by edit distance and report the new metrics on words that were present in both the reference and the ASR transcript.

Hear the difference

Given below are some snippets from AlignBench which motivate the new metrics. For each one, play the context, then the word as the reference places it and as Gentle places it, then delete the word with each set of boundaries. The deletion is the honest test: it is the edit a user would make.

FuseAlign

We then built FuseAlign, a model designed from scratch for forced alignment. It is a transformer-based aligner that separates global word localization from local boundary refinement. A joint audio–text transformer matches transcript words to coarse regions of the recording; convolutional heads then refine each start and end on a 1 ms grid. Every word, and each of its two edges, is localized independently, so a transcript mismatch does not shift its neighbours, and a presence head rejects words that were never spoken, as with I in the example below. It is trained on roughly 78,000 hours of pseudo-labeled speech.

Hear FuseAlign against the baselines

Given below are short AlignBench excerpts where the systems disagree. Each track shows where a system put every word; colours are the worst-edge asymmetric error against the hand-corrected reference, and a dash marks a word the system did not place. Click any word to play exactly the span that system chose, then click the same word in another track and compare.