Repository navigation
Keep "point" when it does not start a decimal in the English normalizer - #2889
Open
MohammadHijjawi97 wants to merge 1 commit into
Open
MohammadHijjawi97 wants to merge 1 commit into
MohammadHijjawi97 wants to merge 1 commit into
Conversation
EnglishNumberNormalizer only emitted "point" when the next word was not a
number word at all. When the next word was number-related but not a
decimal digit ("and", "minus", "first", "dollars", ...), the branch did
nothing and the word was silently dropped:
"that was the point and then" -> "that was the and then"
"match point minus two" -> "match -2"
Emit "point" (after flushing any pending value) in that case, as the
"double"/"triple" branch already does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EnglishNumberNormalizer.process_wordshandlespointin two cases: if the next word is not a number word,pointis emitted as-is; if the next word is a decimal digit (or numeric), a.is appended. When the next word is a number word but not a decimal digit (and,minus,first,dollars, ...) neither case applies and the word is silently dropped:that was the point and then we leftthat was the and then we leftthat was the point and then we leftmatch point minus twomatch -2match point -2the point firstthe 1stthe point 1stgood point dollarsgood dollarsgood point dollarstwo point and2 and2 point andThe fix flushes any pending value and emits
point, the same way thedouble/triplebranch already handles a non-numeric follower. Decimal cases are unchanged (3 point 14,one point oh five,two point two five dollarsstill give3.14,1.05,$2.25).Added four asserts to the parametrized
test_number_normalizer(they fail without the change);tests/test_normalizer.py,test_tokenizer.pyand the non-CUDAtest_timing.pytests pass, and black/isort/flake8 at the pre-commit pins are clean.