[kawshikbuet17]

Summary

Whisper has no built-in way to fine-tune on your own data. This PR adds a small training command to the package that works with the existing models, checkpoints and CLI, with no new runtime dependencies.

• whisper-train fine-tunes any Whisper model on a JSONL or CSV manifest (audio, text, optional language).
• Two modes:
  • LoRA (--finetune lora, the default) freezes the model and trains low-rank adapters, about 0.2–0.4% of the parameters by default, saved as a small adapter file.
  • Full fine-tuning (--finetune full, optionally --freeze_encoder) saves a regular Whisper checkpoint.
• Inference: whisper audio.wav --model small --lora adapter.pt or whisper.load_model("small", lora="adapter.pt"). Full checkpoints load with --model model.pt as before, and --export_merged folds an adapter into a plain checkpoint.
• Training: fp32/fp16/bf16 (--precision auto picks bf16 on GPUs with native support, else fp16 with a gradient scaler), gradient checkpointing and accumulation, warmup with linear/cosine schedules, validation loss and WER/CER before and after training, best-checkpoint selection, exact resume (--resume) and --keep_checkpoints.

Existing behaviour is unchanged unless whisper-train or --lora is used.

Changes

• whisper/lora.py: LoRALinear, apply_lora, save/load and merge_lora. It works with the existing KV-cache hooks.
• whisper/precision.py: precision selection, autocast and GradScaler helpers.
• whisper/train.py: the whisper-train CLI, with its entry point added in pyproject.toml.
• whisper/__init__.py, whisper/transcribe.py: lora= and --lora support.
• README.md: a short "Fine-tuning" section with trainable parameter counts per model.
• tests/test_lora.py, tests/test_train.py: unit and end-to-end tests. They run on CPU, and the fp16/bf16 tests also run on GPU when one is available.

Details handled in the new code:

• torch.cuda.is_bf16_supported() also reports emulated bf16 (e.g. on a T4), so native support is checked instead.
• In scripts where the leading space doesn't merge into the first word (e.g. Bengali), the lone space token is dropped from targets, since SuppressBlank stops the decoder from ever predicting it.
• WER for non-English text keeps combining marks (Indic vowel signs), which BasicTextNormalizer removes.

Testing

• pytest tests: the full suite passes, including the existing tests and the new GPU fp16/bf16 tests (run on a Tesla T4 and an RTX 4090).
• Smoke runs:

Run	Trainable	Peak memory	Time/step
small, LoRA, batch 4 (T4, fp16)	0.88 M	4.15 GB	0.41 s
small, full, batch 4 (T4, fp16)	240.6 M	6.21 GB	0.54 s
medium, LoRA rank 16 on all layers, batch 16, gradient checkpointing (4090, bf16)	17.3 M	8.1 GB	~2 s

• Overfitting a few clips reaches 0% WER, and a killed run resumes where it stopped.
• On a real task (medium, LoRA, about 3 h of speech), held-out WER improved on both in-domain and general speech.

Usage

whisper-train --model small --language en \
    --train_manifest train.jsonl --val_manifest val.jsonl --output_dir runs/small-lora
whisper audio.wav --model small --lora runs/small-lora/adapter_best.pt

[agammann]

One blocker for the documented per-sample language training mode.

[kawshikbuet17]

@agammann Thanks, good catch. Fixed in 911a042: the padding token is the EOT token, which is the same for every language, so it is now taken from a tokenizer built without a language. I added test_per_sample_language_without_language_flag, which trains with per-row en/fr languages and no --language; it failed with this ValueError before the fix.

[agammann]

Thanks for the fix. I rechecked head 911a042: padding now obtains the EOT token without asking the per-sample tokenizer cache for a global language, and the new test runs training with English/French rows and no language flag. That addresses the startup failure I raised. I reviewed the patch but did not run the test locally. AI-assisted review.