Skip to content

Add typhoon-asr-streaming-nemotron #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wannaphong merged 2 commits into from
Oct 4, 2026
Merged

Add typhoon-asr-streaming-nemotron #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6d4ee51
Add typhoon-asr-streaming-nemotron
wannaphong Oct 4, 2026
302f8c3
Update name
wannaphong Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ file = "sample.wav"
# Uses Typhoon ASR (FastConformer RNN-T ONNX) by default
print(asr(file))

# Use Typhoon Nemotron 3.5 Streaming ASR (INT4 ONNX)
print(asr(file, model="typhoon_nemotron_asr"))

# With timestamps (returns dictionary with 'text', 'chunks', and 'timestamps')
result = asr(file, return_timestamps=True)
print(result["text"])
Expand All @@ -69,6 +72,7 @@ for chunk in result["chunks"]:
# print(asr(file, model="airesearch/wav2vec2-large-xlsr-53-th"))
# print(asr(file, model="biodatlab/whisper-small-th-combined"))
# print(asr(file, model="biodatlab/whisper-th-medium-timestamp", return_timestamps=True))

```

### Live Audio Streaming
Expand Down Expand Up @@ -206,6 +210,8 @@ stream_asr(

**Options for model**
- *typhoon_asr* / *typhoon-asr-realtime* (default) - Typhoon FastConformer RNN-T ONNX model (offline & realtime)
- *typhoon_nemotron_asr* / *wannaphong/typhoon-asr-streaming-nemotron-0.6b-int4-onnx* - Typhoon Nemotron 3.5 Streaming ASR INT4 ONNX model
- *typhoon_nemotron_asr_fp32* / *wannaphong/typhoon-asr-streaming-nemotron-0.6b-fp32-onnx* - Typhoon Nemotron 3.5 Streaming ASR FP32 ONNX model
- *airesearch/wav2vec2-large-xlsr-53-th* - AI RESEARCH - PyThaiNLP model (requires pythaiasr[torch])
- *wannaphong/wav2vec2-large-xlsr-53-th-cv8-newmm* - Thai Wav2Vec2 with CommonVoice V8 (newmm tokenizer) (requires pythaiasr[torch])
- *wannaphong/wav2vec2-large-xlsr-53-th-cv8-deepcut* - Thai Wav2Vec2 with CommonVoice V8 (deepcut tokenizer) (requires pythaiasr[torch])
Expand All @@ -217,6 +223,8 @@ stream_asr(
You can read about models from the list:

- [*typhoon-ai/typhoon-asr-realtime* / *wannaphong/typhoon-asr-realtime-onnx* - Typhoon FastConformer RNN-T ONNX model](https://huggingface.co/wannaphong/typhoon-asr-realtime-onnx)
- [*wannaphong/typhoon-asr-streaming-nemotron-0.6b-int4-onnx* - Typhoon Nemotron 3.5 Streaming ASR INT4 ONNX model](https://huggingface.co/wannaphong/typhoon-asr-streaming-nemotron-0.6b-int4-onnx)
- [*wannaphong/typhoon-asr-streaming-nemotron-0.6b-fp32-onnx* - Typhoon Nemotron 3.5 Streaming ASR FP32 ONNX model](https://huggingface.co/wannaphong/typhoon-asr-streaming-nemotron-0.6b-fp32-onnx)
- [*airesearch/wav2vec2-large-xlsr-53-th* - AI RESEARCH - PyThaiNLP model](https://medium.com/airesearch-in-th/airesearch-in-th-3c1019a99cd)
- [*annaphong/wav2vec2-large-xlsr-53-th-cv8-newmm* - Thai Wav2Vec2 with CommonVoice V8 (newmm tokenizer) + language model](https://huggingface.co/wannaphong/wav2vec2-large-xlsr-53-th-cv8-newmm)
- [*wannaphong/wav2vec2-large-xlsr-53-th-cv8-deepcut* - Thai Wav2Vec2 with CommonVoice V8 (deepcut tokenizer) + language model](https://huggingface.co/wannaphong/wav2vec2-large-xlsr-53-th-cv8-deepcut)
Expand Down
98 changes: 94 additions & 4 deletions pythaiasr/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@
get_typhoon_model_files,
get_diarization_model_files,
get_nemotron_diarization_model_files,
get_nemotron_asr_model_files,
get_typhoon_nemotron_asr_model_files,
download_file,
)
from pythaiasr.typhoon import (
Expand All @@ -39,6 +41,18 @@
extract_features,
load_audio,
)
from pythaiasr.typhoon_nemotron_asr import (
NemotronStreamingASR,
NemotronASR,
NemotronStreamASR,
RealtimeStreamNemotron,
extract_nemotron_features,
TyphoonNemotronStreamingASR,
TyphoonNemotronASR,
TyphoonNemotronStreamASR,
RealtimeStreamTyphoonNemotron,
extract_typhoon_nemotron_features,
)
from pythaiasr.diarization import (
Diarization,
NemotronDiarization,
Expand All @@ -50,10 +64,11 @@
extract_speaker_dict,
)

# Friendly alias
# Friendly aliases
TyphoonASR = FastConformerRNNT



class ASR:
def __init__(self, model: str="typhoon_asr", lm: bool=False, device: str=None) -> None:
"""
Expand All @@ -63,6 +78,7 @@ def __init__(self, model: str="typhoon_asr", lm: bool=False, device: str=None) -

**Options for model**
* *typhoon_asr* / *typhoon-asr-realtime* (default) - Typhoon FastConformer RNN-T ONNX model (offline & realtime)
* *typhoon_nemotron_asr* / *wannaphong/typhoon-asr-streaming-nemotron-0.6b-int4-onnx* (INT4) / *wannaphong/typhoon-asr-streaming-nemotron-0.6b-fp32-onnx* (FP32) - Typhoon Nemotron 3.5 Streaming ASR ONNX models
* *airesearch/wav2vec2-large-xlsr-53-th* - AI RESEARCH - PyThaiNLP model (requires pythaiasr[torch])
* *wannaphong/wav2vec2-large-xlsr-53-th-cv8-newmm* - Thai Wav2Vec2 with CommonVoice V8 (newmm tokenizer) + language model (requires pythaiasr[torch])
* *wannaphong/wav2vec2-large-xlsr-53-th-cv8-deepcut* - Thai Wav2Vec2 with CommonVoice V8 (deepcut tokenizer) + language model (requires pythaiasr[torch])
Expand All @@ -83,6 +99,22 @@ def __init__(self, model: str="typhoon_asr", lm: bool=False, device: str=None) -
"typhoon_asr",
"typhoon-asr-realtime",
"wannaphong/typhoon-asr-realtime-onnx",
"typhoon_nemotron_asr",
"typhoon-nemotron-asr",
"typhoon_nemotron_asr_int4",
"typhoon-nemotron-asr-int4",
"typhoon_nemotron_asr_fp32",
"typhoon-nemotron-asr-fp32",
"nemotron_asr",
"nemotron-asr",
"nemotron_asr_int4",
"nemotron-asr-int4",
"nemotron_asr_fp32",
"nemotron-asr-fp32",
"nemotron-3.5-asr-streaming-0.6b",
"wannaphong/nemotron-3.5-asr-streaming-0.6b-onnx-int4",
"wannaphong/typhoon-asr-streaming-nemotron-0.6b-int4-onnx",
"wannaphong/typhoon-asr-streaming-nemotron-0.6b-fp32-onnx",
]
self.whisper_models = [
"biodatlab/whisper-small-th-combined",
Expand All @@ -95,10 +127,29 @@ def __init__(self, model: str="typhoon_asr", lm: bool=False, device: str=None) -
"typhoon-asr-realtime",
"wannaphong/typhoon-asr-realtime-onnx",
]
self.nemotron_models = [
"typhoon_nemotron_asr",
"typhoon-nemotron-asr",
"typhoon_nemotron_asr_int4",
"typhoon-nemotron-asr-int4",
"typhoon_nemotron_asr_fp32",
"typhoon-nemotron-asr-fp32",
"nemotron_asr",
"nemotron-asr",
"nemotron_asr_int4",
"nemotron-asr-int4",
"nemotron_asr_fp32",
"nemotron-asr-fp32",
"nemotron-3.5-asr-streaming-0.6b",
"wannaphong/nemotron-3.5-asr-streaming-0.6b-onnx-int4",
"wannaphong/typhoon-asr-streaming-nemotron-0.6b-int4-onnx",
"wannaphong/typhoon-asr-streaming-nemotron-0.6b-fp32-onnx",
]
assert self.model_name in self.support_model, f"Model {self.model_name} is not in supported models: {self.support_model}"
self.lm = lm

self.is_typhoon = self.model_name in self.typhoon_models
self.is_nemotron = self.model_name in self.nemotron_models
self.is_whisper = self.model_name in self.whisper_models

if self.is_typhoon:
Expand All @@ -107,6 +158,14 @@ def __init__(self, model: str="typhoon_asr", lm: bool=False, device: str=None) -
self.device = dev
return

if self.is_nemotron:
dev = device if device is not None else "auto"
prec = "fp32" if "fp32" in self.model_name else "int4"
self.model = NemotronStreamingASR(device=dev, precision=prec)
self.device = dev
return


if torch is None or torchaudio is None:
raise ImportError(
f"torch and torchaudio are required for {self.model_name}. "
Expand Down Expand Up @@ -168,13 +227,22 @@ def __call__(
:rtype: Union[str, dict]
"""
use_timestamps = return_timestamps if return_timestamps is not None else timestamps
if self.is_typhoon:
if getattr(self, "is_typhoon", False):
return self.model.transcribe(
data,
sample_rate=sampling_rate,
return_timestamps=use_timestamps,
)

if getattr(self, "is_nemotron", False):
return self.model.transcribe(
data,
sampling_rate=sampling_rate,
return_timestamps=use_timestamps,
)



b = {}
if isinstance(data, np.ndarray):
b["speech"] = data
Expand Down Expand Up @@ -341,6 +409,7 @@ def asr(

**Options for model**
* *typhoon_asr* / *typhoon-asr-realtime* (default) - Typhoon FastConformer RNN-T ONNX model
* *typhoon_nemotron_asr* / *wannaphong/typhoon-asr-streaming-nemotron-0.6b-int4-onnx* (INT4) / *wannaphong/typhoon-asr-streaming-nemotron-0.6b-fp32-onnx* (FP32) - Typhoon Nemotron 3.5 Streaming ASR ONNX models
* *airesearch/wav2vec2-large-xlsr-53-th* - AI RESEARCH - PyThaiNLP model (requires pythaiasr[torch])
* *wannaphong/wav2vec2-large-xlsr-53-th-cv8-newmm* - Thai Wav2Vec2 with CommonVoice V8 (newmm tokenizer) (+ language model, requires pythaiasr[torch])
* *wannaphong/wav2vec2-large-xlsr-53-th-cv8-deepcut* - Thai Wav2Vec2 with CommonVoice V8 (deepcut tokenizer) (+ language model, requires pythaiasr[torch])
Expand Down Expand Up @@ -385,6 +454,7 @@ def stream_asr(

**Options for model**
* *typhoon_asr* / *typhoon-asr-realtime* (default) - Typhoon FastConformer RNN-T ONNX model (recommended for streaming)
* *typhoon_nemotron_asr* / *typhoon-nemotron-asr* - Typhoon Nemotron 3.5 Streaming ASR ONNX model
* *airesearch/wav2vec2-large-xlsr-53-th* - AI RESEARCH - PyThaiNLP model (requires pythaiasr[torch])
* *wannaphong/wav2vec2-large-xlsr-53-th-cv8-newmm* - Thai Wav2Vec2 with CommonVoice V8 (newmm tokenizer) (+ language model, requires pythaiasr[torch])
* *wannaphong/wav2vec2-large-xlsr-53-th-cv8-deepcut* - Thai Wav2Vec2 with CommonVoice V8 (deepcut tokenizer) (+ language model, requires pythaiasr[torch])
Expand Down Expand Up @@ -417,18 +487,25 @@ def stream_asr(
_model_name = model

if chunk_duration is None:
chunk_duration = 0.48 if _model.is_typhoon else 5.0
chunk_duration = 0.56 if getattr(_model, "is_nemotron", False) else (0.48 if _model.is_typhoon else 5.0)

use_timestamps = return_timestamps or bool(timestamps)

# If Typhoon model, use stateful RealtimeStreamASR
# If Typhoon or Nemotron model, use stateful realtime stream session
streamer = None
if _model.is_typhoon:
streamer = RealtimeStreamASR(
model=_model.model,
sample_rate=sampling_rate,
step_sec=chunk_duration,
)
elif getattr(_model, "is_nemotron", False):
streamer = NemotronStreamASR(
model=_model.model,
sample_rate=sampling_rate,
step_sec=chunk_duration,
)


# Initialize PyAudio
audio = pyaudio.PyAudio()
Expand Down Expand Up @@ -511,6 +588,16 @@ def stream_asr(
"TyphoonASR",
"RealtimeStreamASR",
"StreamingTranscriber",
"NemotronStreamingASR",
"NemotronASR",
"NemotronStreamASR",
"RealtimeStreamNemotron",
"extract_nemotron_features",
"TyphoonNemotronStreamingASR",
"TyphoonNemotronASR",
"TyphoonNemotronStreamASR",
"RealtimeStreamTyphoonNemotron",
"extract_typhoon_nemotron_features",
"stream_from_mic",
"stream_from_file",
"list_audio_devices",
Expand All @@ -520,5 +607,8 @@ def stream_asr(
"get_typhoon_model_files",
"get_diarization_model_files",
"get_nemotron_diarization_model_files",
"get_nemotron_asr_model_files",
"get_typhoon_nemotron_asr_model_files",
"download_file",
]

Loading
Loading