Skip to content

Preserve Unicode and token boundaries when splitting malformed UTF-8 - #2895

Open
GCSimba wants to merge 1 commit into
openai:mainfrom
GCSimba:codex/fix-whisper-unicode-splitting
Open

Preserve Unicode and token boundaries when splitting malformed UTF-8#2895
GCSimba wants to merge 1 commit into
openai:mainfrom
GCSimba:codex/fix-whisper-unicode-splitting

Split token bytes incrementally without losing Unicode or special tokens

946b103
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs