[nirajkhetre]

Addresses discussion #2875.

With word_timestamps=True, split_tokens_on_spaces starts a new word at every punctuation token, even in the middle of a space-delimited word. merge_punctuations only handles punctuation that stands alone, so it can't repair the pieces, and they reach segments[].words and word-level subtitles.

Punctuation now only stands alone at the end of a space-delimited word. After merge_punctuations with the default settings:

Text	Before	After
$1,234.50	$1, ,234, .50	$1,234.50
10:30	10, :30	10:30
example.com.	example, .com.	example.com.
U.S.	U, .S.	U.S.
and/or	and, /or	and/or
"안녕"이라고	"안녕, "이라고	"안녕"이라고
3.5%	3, .5, %	3.5, %

Languages that use split_tokens_on_unicode (Chinese, Japanese, Thai, Lao, Burmese, Cantonese) aren't affected.

Why not remove the punctuation check entirely? The discussion suggests that, but merge_punctuations needs trailing punctuation as a separate word, so --append_punctuations would stop having any effect. With this change, append_punctuations="" still gives Hello, , as it does today.

One case for you to decide: a bare % before a space stays separate ( 3.5, %, and 4, % as before), because % isn't in the default append_punctuations. Anyone can add it through that option. I left the default alone since it's a separate behaviour change, but I'm happy to add it here if you'd prefer.

Testing

• New test_split_on_spaces_only_separates_trailing_punctuation in tests/test_tokenizer.py: 3 of its 5 cases fail on main. The other 2 pin behaviour that shouldn't change (Hello, world!, 4% today.).
• pytest tests/test_transcribe.py -k tiny passes. On jfk.flac with tiny, the transcript and all 22 word timings are identical before and after.
• find_alignment with tiny on a forced sentence containing the cases above returns the merged words, with word times still in order. That audio doesn't contain those words, so this checks the code path, not timing accuracy.
• The tokenizer, timing, normalizer and audio tests pass with -m "not requires_cuda" on Windows 11, Python 3.12, torch 2.15 CPU. The requires_cuda tests can't run on this machine.
• black (25.1.0), isort and flake8 pass with the pre-commit settings.