Skip to content

About

Spoken vegetable classifier — CNN on Mel spectrograms, quantized to INT8 ONNX, deployed on an ESP32-S3 with a browser-based WAV upload interface

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

UniVoice

End-to-end vegetable audio classifier — from raw WAV recordings to a web-accessible inference device running on an ESP32-S3.


System Architecture

flowchart TD
    subgraph Data["Data Preparation"]
        A[Raw WAV Recordings] --> B[reorganize_data.py\nFlatten & rename files]
        B --> C[gan_augment.py\nOptional — balance to 400 samples/class]
        C --> D[split_dataset.py\nStratified 80/20 train/test split]
    end

    subgraph Training["Model Training"]
        D --> E[train_classifier.py\nCNN on Mel Spectrogram]
        E --> F[(best_classifier.pt\nPyTorch checkpoint)]
    end

    subgraph Export["Model Export & Quantization"]
        F --> G[export_onnx.py\nPyTorch → ONNX → Simplify → INT8]
        G --> H[(classifier_int8.onnx)]
        G --> I[(classifier_int8.tflite)]
        H --> J[onnx_to_esp32.py\nONNX → C headers]
        J --> K[(model_data.h\nmodel_info.h)]
    end

    subgraph ESP32["ESP32-S3 Deployment"]
        K --> L[Arduino Sketch\nesp32_deployment.ino]
        L --> M[ESP32-S3\nWeb Server + Inference Engine]
    end

    subgraph Client["Client"]
        N[Browser\nSame WiFi network] -->|Upload .wav| M
        M -->|Prediction + confidence| N
    end
Loading

Classes

The classifier recognises 10 spoken vegetable names:

# Class
1 Carrots
2 Garlic
3 Green Chillies
4 Ladies Fingers
5 Leeks
6 Long Beans
7 Onions
8 Potatoes
9 Pumpkin
10 Tomatoes

Project Structure

UniVoice/
├── reorganize_data.py       # Step 1 — flatten raw recordings into class dirs
├── gan_augment.py           # Step 2 — GAN-based augmentation (optional)
├── split_dataset.py         # Step 3 — stratified train/test split
├── train_classifier.py      # Step 4 — train CNN classifier
├── export_onnx.py           # Step 5 — export to ONNX + INT8 quantize
├── convert_to_tflite.py     # Step 5b — ONNX → TFLite conversion
├── onnx_to_esp32.py         # Step 6 — generate C headers for ESP32
├── infer.py                 # Desktop inference (sanity-check)
├── esp32_deployment/
│   ├── esp32_deployment.ino # Main Arduino sketch
│   ├── inference_esp/       # Inference-only sketch variant
│   ├── web_esp/             # Web interface sketch variant
│   ├── model_data.h         # Quantized model as uint8_t array
│   ├── model_info.h         # Preprocessing constants + class labels
│   ├── wifi_config.h        # WiFi credentials (fill in before flashing)
│   ├── webpage.h            # Embedded HTML/CSS/JS
│   └── *.onnx / *.tflite   # Exported model artifacts
├── docs/                    # Per-script documentation
└── USER_MANUAL.md           # Full step-by-step manual

Prerequisites

Python

Use the pre-configured virtual environment:

/ml/HelaBERT_Analysis/.venv/bin/python <script.py>

Key packages: torch, torchaudio, onnx, onnxruntime, onnxsim, soundfile, sklearn

Arduino

Install via Tools → Manage Libraries:

  • ESP32-audioI2S — audio capture
  • ArduinoJson — JSON responses
  • ESPAsyncWebServer — async HTTP server

Board: ESP32-S3 Dev Module, Flash: 8 MB, Partition: Huge APP (3MB No OTA), PSRAM: OPI PSRAM


Pipeline

1 — Reorganise data

python reorganize_data.py

Flattens a nested recording structure into data/<class>/0001.wav, 0002.wav, ...

2 — Augment (optional)

python gan_augment.py

Trains a GAN per class and synthesises samples until each class has 400 recordings.

3 — Split dataset

python split_dataset.py

Produces a stratified 80/20 train/test split under data_split/.

4 — Train classifier

python train_classifier.py

Trains a MobileNet-style CNN on Mel spectrograms. Saves the best checkpoint to best_classifier.pt.

5 — Export to ONNX

python export_onnx.py

Converts best_classifier.pt → classifier.onnx → simplifies → INT8 quantizes to classifier_int8.onnx.

6 — Generate ESP32 headers

python onnx_to_esp32.py

Converts classifier_int8.onnx into model_data.h and model_info.h for embedding in the Arduino sketch.

7 — Flash the ESP32

  1. Open esp32_deployment/wifi_config.h and set your WiFi credentials.
  2. Open esp32_deployment/ in Arduino IDE (all .h files must be in the same folder).
  3. Select the correct board settings and click Upload.
  4. Open Serial Monitor at 115200 baud to get the device IP.

8 — Classify

Navigate to http://<ESP32-IP> in any browser on the same network. Upload a .wav file and click Classify.

Accepted audio: WAV (PCM), mono or stereo, 8–48 kHz (resampled on-device to 16 kHz), up to 512 KB.


Desktop Inference

To test the trained model locally without an ESP32:

python infer.py --model best_classifier.pt --audio path/to/audio.wav

See USER_MANUAL.md for detailed troubleshooting and advanced configuration.

About

Spoken vegetable classifier — CNN on Mel spectrograms, quantized to INT8 ONNX, deployed on an ESP32-S3 with a browser-based WAV upload interface

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Contributors

Languages