Skip to content

Preserve sign words that do not introduce a number - #2902

Open
pandaxiong0729 wants to merge 1 commit into
openai:mainfrom
pandaxiong0729:fix/numeric-prefix-requires-number
Open

pandaxiong0729 wants to merge 1 commit into
openai:mainfrom
pandaxiong0729:fix/numeric-prefix-requires-number

Conversation

@pandaxiong0729

Copy link
Copy Markdown

English number normalization currently changes ordinary phrases such as the positive and negative effects to the +and negative effects, and a positive point of view to a +point of view. The sign handler accepts every parser keyword as the beginning of a number, including and, currency names, and point without a following value.

Only consume a sign word when it introduces a number. Look ahead through point, double, and triple using the same rules as their existing numeric handlers, so signed decimals and repeated digits keep working. Add regression and control cases for both English normalizers.

Validation on Windows, Python 3.14.2 and PyTorch 2.15.0:

  • The new cases on unchanged main fail 14 times; 16 existing/control cases pass.
  • Normalizer and tokenizer suites: 34 passed.
  • CPU timing suite: 8 passed, 8 CUDA cases deselected.
  • Black 25.1.0, isort 6.0.0 and Flake8 7.1.1 checks on the changed files, and git diff --check: passed.

pre-commit run --all-files could not initialize its first hook because GitHub reset the connection while fetching pre-commit-hooks. The matching pinned formatter/linter versions were run directly as listed above.

Model transcription and the other Python versions in CI were not run. Prepared and tested with Codex assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant