[GCSimba]

get_tokenizer(multilingual=True).split_tokens_on_unicode([1041, 243, 230]) raises IndexError, although those tokens decode to �안. Checking only the first replacement character can flush an incomplete character too early. A literal U+FFFD split across byte tokens can also crash or silently lose subsequent tokens. This splitter is used by word-level timestamp alignment.

Use Python’s incremental UTF-8 decoder to distinguish incomplete bytes from actual replacement characters. Preserve the original token order, flush incomplete trailing bytes, and keep end-of-text and timestamp tokens separate from preceding text.

Validation:

• Added regressions for both tokenizer vocabularies, the crashing sequences, split literal replacement characters, invalid and truncated UTF-8, and special-token boundaries.
• pytest tests/test_tokenizer.py -q: 32 passed.
• Black 25.1.0, isort 6.0.0, Flake8 7.1.1 with the repository’s hook arguments, and git diff --check pass.

Tested locally on Windows with Python 3.10. No GPU validation.