End-to-end vegetable audio classifier — from raw WAV recordings to a web-accessible inference device running on an ESP32-S3.
flowchart TD
subgraph Data["Data Preparation"]
A[Raw WAV Recordings] --> B[reorganize_data.py\nFlatten & rename files]
B --> C[gan_augment.py\nOptional — balance to 400 samples/class]
C --> D[split_dataset.py\nStratified 80/20 train/test split]
end
subgraph Training["Model Training"]
D --> E[train_classifier.py\nCNN on Mel Spectrogram]
E --> F[(best_classifier.pt\nPyTorch checkpoint)]
end
subgraph Export["Model Export & Quantization"]
F --> G[export_onnx.py\nPyTorch → ONNX → Simplify → INT8]
G --> H[(classifier_int8.onnx)]
G --> I[(classifier_int8.tflite)]
H --> J[onnx_to_esp32.py\nONNX → C headers]
J --> K[(model_data.h\nmodel_info.h)]
end
subgraph ESP32["ESP32-S3 Deployment"]
K --> L[Arduino Sketch\nesp32_deployment.ino]
L --> M[ESP32-S3\nWeb Server + Inference Engine]
end
subgraph Client["Client"]
N[Browser\nSame WiFi network] -->|Upload .wav| M
M -->|Prediction + confidence| N
end
The classifier recognises 10 spoken vegetable names:
| # | Class |
|---|---|
| 1 | Carrots |
| 2 | Garlic |
| 3 | Green Chillies |
| 4 | Ladies Fingers |
| 5 | Leeks |
| 6 | Long Beans |
| 7 | Onions |
| 8 | Potatoes |
| 9 | Pumpkin |
| 10 | Tomatoes |
UniVoice/
├── reorganize_data.py # Step 1 — flatten raw recordings into class dirs
├── gan_augment.py # Step 2 — GAN-based augmentation (optional)
├── split_dataset.py # Step 3 — stratified train/test split
├── train_classifier.py # Step 4 — train CNN classifier
├── export_onnx.py # Step 5 — export to ONNX + INT8 quantize
├── convert_to_tflite.py # Step 5b — ONNX → TFLite conversion
├── onnx_to_esp32.py # Step 6 — generate C headers for ESP32
├── infer.py # Desktop inference (sanity-check)
├── esp32_deployment/
│ ├── esp32_deployment.ino # Main Arduino sketch
│ ├── inference_esp/ # Inference-only sketch variant
│ ├── web_esp/ # Web interface sketch variant
│ ├── model_data.h # Quantized model as uint8_t array
│ ├── model_info.h # Preprocessing constants + class labels
│ ├── wifi_config.h # WiFi credentials (fill in before flashing)
│ ├── webpage.h # Embedded HTML/CSS/JS
│ └── *.onnx / *.tflite # Exported model artifacts
├── docs/ # Per-script documentation
└── USER_MANUAL.md # Full step-by-step manual
Use the pre-configured virtual environment:
/ml/HelaBERT_Analysis/.venv/bin/python <script.py>Key packages: torch, torchaudio, onnx, onnxruntime, onnxsim, soundfile, sklearn
Install via Tools → Manage Libraries:
- ESP32-audioI2S — audio capture
- ArduinoJson — JSON responses
- ESPAsyncWebServer — async HTTP server
Board: ESP32-S3 Dev Module, Flash: 8 MB, Partition: Huge APP (3MB No OTA), PSRAM: OPI PSRAM
python reorganize_data.pyFlattens a nested recording structure into data/<class>/0001.wav, 0002.wav, ...
python gan_augment.pyTrains a GAN per class and synthesises samples until each class has 400 recordings.
python split_dataset.pyProduces a stratified 80/20 train/test split under data_split/.
python train_classifier.pyTrains a MobileNet-style CNN on Mel spectrograms. Saves the best checkpoint to best_classifier.pt.
python export_onnx.pyConverts best_classifier.pt → classifier.onnx → simplifies → INT8 quantizes to classifier_int8.onnx.
python onnx_to_esp32.pyConverts classifier_int8.onnx into model_data.h and model_info.h for embedding in the Arduino sketch.
- Open
esp32_deployment/wifi_config.hand set your WiFi credentials. - Open
esp32_deployment/in Arduino IDE (all.hfiles must be in the same folder). - Select the correct board settings and click Upload.
- Open Serial Monitor at 115200 baud to get the device IP.
Navigate to http://<ESP32-IP> in any browser on the same network. Upload a .wav file and click Classify.
Accepted audio: WAV (PCM), mono or stereo, 8–48 kHz (resampled on-device to 16 kHz), up to 512 KB.
To test the trained model locally without an ESP32:
python infer.py --model best_classifier.pt --audio path/to/audio.wavSee USER_MANUAL.md for detailed troubleshooting and advanced configuration.