A PyTorch-like deep learning framework in pure Rust
Volta is a minimal deep learning and automatic differentiation library for Rust, heavily inspired by PyTorch.
Built from scratch to expose the internals - DAG linearization, GradFn dispatch, and SafeTensors interop - that production frameworks abstract away.
It provides a dynamic computation graph, NumPy-style broadcasting, and common neural network primitives.
This project is an educational endeavor to demystify the inner workings of modern autograd engines. It prioritizes correctness, clarity, and a clean API over raw performance, while still offering optional GPU (WGPU) and Accelerate (BLAS) backends. The GPU backend is experimental and currently slower than the CPU (see What's in Progress).
Caution
This repo is pre-v1.0, and currently... not in a spot that I'm proud of. Expect MAJOR refactors in order to get it into a more polished spot.
- Dynamic Computation Graph: Build and backpropagate through graphs on the fly, just like PyTorch.
- Reverse-Mode Autodiff: Efficient reverse-mode automatic differentiation with topological sorting.
- Rich Tensor Operations: A comprehensive set of unary, binary, reduction, and matrix operations as methods on
Tensor. - Broadcasting: Full NumPy-style broadcasting support for arithmetic operations.
- Neural Network Layers:
Linear,Conv2d,ConvTranspose2d,MaxPool2d,Embedding,LSTMCell,PixelShuffle,LayerNorm,Flatten,ReLU,GELU,Sigmoid,Tanh,Dropout,BatchNorm1d,BatchNorm2d. - Optimizers:
SGD(momentum + weight decay),Adam(bias-corrected + weight decay), and experimentalMuon. - External Model Loading: Load weights from PyTorch, HuggingFace, and other frameworks via
StateDictMapperwith automatic weight transposition and key remapping. Supports SafeTensors format. - Named Layers: Human-readable state dict keys with
Sequential::builder()pattern for robust serialization. - SafeTensors dtypes: f16, bf16, f64, i32, i64, u8, and bool files load (converted to f32); every op, layer, and optimizer computes in f32, f64, f16 or bf16 on the CPU and in f32, f16 or bf16 on the GPU (
Module::to_dtypeconverts a model);amp::autocastandGradScalergive mixed precision (docs/plans/ADDENDUM.md§11). - IO System: Save and load model weights (state dicts) in SafeTensors format.
- BLAS Acceleration (macOS): Optional acceleration for matrix multiplication via Apple's Accelerate framework.
- GPU Backend: Experimental WGPU-based GPU support for core tensor operations (elementwise, matmul, reductions, movement ops) with automatic backward pass on GPU.
- Validation-Focused: Includes a robust numerical gradient checker to ensure the correctness of all implemented operations.
This library trains MLPs, CNNs, RNNs, GANs, VAEs, and other architectures on CPU. Known defects are tracked as GitHub issues; the merge-gate label marks the ones that block a release.
-
✅ What's Working:
- Core Autograd: Operations are tested against numerical gradients, and the core ops and layers against PyTorch reference values (
tests/torch_ref_*.rs) - Layers: Linear, Conv2d, ConvTranspose2d, MaxPool2d, Embedding, LSTMCell, PixelShuffle, LayerNorm, BatchNorm1d/2d, Dropout, MultiHeadAttention, TransformerBlock, PositionalEncoding
- Optimizers: SGD (with momentum), Adam, Muon
- External Loading: PyTorch/HuggingFace model weights via SafeTensors with automatic transposition
- Named Layers: Robust serialization with human-readable state dict keys
- Loss Functions: MSE, Cross-Entropy, NLL, BCE, KL Divergence
- Examples: MNIST (CNN, LSTM, bf16 autocast), character LMs (LSTM, Transformer), VAE, DCGAN, super-resolution, LSTM sine-wave prediction, polynomial regression; see Available Examples
- GPU Training Pipeline: Fully GPU-accelerated training for Conv2d layers with device-aware constructors and GPU optimizer state storage
- Benchmarking Suite: Criterion benchmarks in 4 targets (tensor_ops, neural_networks, gpu_comparison, conv_algorithms) with HTML reports
- GPU Resource Handling: GPU buffer pooling, command queue throttling, and CPU cache invalidation
- Code Quality:
cargo clippy --all-targets -- -D warnings(with the pedantic set configured inCargo.toml) is clean with no features and withgpu+accelerate, and runs in CI - GPU Convolution: Fully GPU-accelerated Conv2d with Direct, im2col, and iGEMM algorithms
- All algorithms support both forward and backward passes on GPU
- Auto-selection chooses optimal algorithm based on input size and device
- Memory-efficient alternatives (Direct, iGEMM) prevent OOM on large inputs
- Core Autograd: Operations are tested against numerical gradients, and the core ops and layers against PyTorch reference values (
-
⚠️ What's in Progress:- Performance: Comprehensive benchmarking suite for performance tracking with
just benchcommands - Benchmarks:
just bench-gpucompletes a full run. The GPU is slower than the CPU at every size measured, because each op pays about 1.5 ms of fixed overhead; seedev-docs/BENCHMARKING.md - GPU Support: Experimental WGPU backend via the
gpufeature:- ✅ Core ops on GPU: elementwise (unary/binary), matmul, reductions (sum/max/mean), movement ops (permute/expand/pad/shrink/stride)
- ✅ GPU backward pass for autograd with lazy CPU↔GPU transfers
⚠️ Broadcasting preprocessing happens on CPU before GPU dispatch
- Performance: Comprehensive benchmarking suite for performance tracking with
-
❌ What's Missing:
- Production-ready GPU integration, distributed training, learning-rate schedulers (planned; see the roadmap)
Add Volta to your Cargo.toml:
[dependencies]
volta = "0.3.0"The default build is CPU-only. Both backends are opt-in features, and they add to each other: gpu (the experimental WGPU backend, which compiles wgpu) and accelerate (Apple's Accelerate framework for matrix multiplication on macOS; a no-op on other platforms).
[dependencies]
volta = { version = "0.3.0", features = ["gpu", "accelerate"] }Pick either one alone, for example features = ["accelerate"] for BLAS on macOS without the GPU backend.
Volta is built around a dynamic computation graph that tracks operations for reverse-mode automatic differentiation.
When operations are performed on tensors requiring gradients, Volta constructs a Directed Acyclic Graph (DAG) of the computation. During the backward pass, this graph is linearized via topological sorting to compute gradients efficiently without deep recursion.
[Input x] (requires_grad=true) [Weights w] (requires_grad=true)
\ /
\ /
v v
[matmul + add] <--- (GradFn::Matmul, GradFn::Binary) --
\
v
[ReLU] <--- (GradFn::Unary)
|
v
[Loss] <--- (the GradFns of the ops the loss is built from)
The core of the autograd engine is the GradFn enum (src/autograd.rs). The output of each differentiable operation stores a GradFn variant holding what its backward pass needs. During the backward pass, the topological sort yields a linearized sequence of nodes; the engine visits them iteratively and calls each node's GradFn::backward, a match over the variants, which maps the gradient of the output to the gradients of the inputs. This iterative approach avoids call stack exhaustion on deep, complex graphs.
Volta saves and loads state dicts in one format, HuggingFace's SafeTensors (io::save_safetensors / io::load_safetensors, used by the MLP example below), for these reasons:
- Simple, checked layout: a JSON header followed by raw tensor bytes. The format allows memory-mapping, but Volta does not use it:
load_safetensorsreads the whole file into memory and copies each tensor out of it. - Safety: Unlike Pickle, SafeTensors cannot execute arbitrary code.
- Interoperability: It enables Volta to natively read weights from models trained in PyTorch or JAX.
Volta provides multiple backends for computation:
- CPU Backend: Matrix multiplication uses Apple's Accelerate framework on macOS (the
acceleratefeature) and the single-threadedmatrixmultiplycrate elsewhere. Everything else runs in plain Rust loops on one thread. - GPU Backend (Experimental): Uses
wgputo dispatch operations to the GPU. Tensors can seamlessly move between devices usingTensor::to_device().
Criterion benchmarks run through just bench (CPU) and just bench-gpu. GPU results and how to read them are in dev-docs/BENCHMARKING.md; the GPU does not beat the CPU yet.
Here's how to define a simple Multi-Layer Perceptron (MLP) with named layers, train it on synthetic data, and save the model.
use volta::nn::{Adam, Layer, Linear, Module, ReLU, Sequential};
use volta::io;
use volta::tensor::{RawTensor, mse_loss};
fn main() {
// 1. Define a simple model with named layers: 2 -> 8 -> 1
let model = Sequential::builder()
.add_named("fc1", Box::new(Linear::new(2, 8, true)))
.add_unnamed(Box::new(ReLU))
.add_named("fc2", Box::new(Linear::new(8, 1, true)))
.build();
// 2. Create synthetic data
let x_data = vec![0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0];
let x = RawTensor::new(x_data, &[4, 2], false); // Batch size 4, 2 features
let y_data = vec![0.0, 1.0, 1.0, 0.0];
let y = RawTensor::new(y_data, &[4], false); // Flattened targets
// 3. Set up the optimizer
let params = model.parameters();
let mut optimizer = Adam::new(params, 0.1, (0.9, 0.999), 1e-8, 0.0);
// 4. Training loop
println!("Training a simple MLP to learn XOR...");
for epoch in 0..=300 {
optimizer.zero_grad();
let pred = model.forward(&x).reshape(&[4]); //alignment
let loss = mse_loss(&pred, &y);
if epoch % 20 == 0 {
println!("Epoch {}: loss = {:.6}", epoch, loss.item());
}
loss.into_loss().expect("loss is scalar").backward();
optimizer.step();
}
// 5. Save and Load State Dict (human-readable keys: "fc1.weight", "fc1.bias", etc.)
let state = model.state_dict();
io::save_safetensors(&state, "model.safetensors").expect("Failed to save");
// Verify loading
let mut new_model = Sequential::builder()
.add_named("fc1", Box::new(Linear::new(2, 8, true)))
.add_unnamed(Box::new(ReLU))
.add_named("fc2", Box::new(Linear::new(8, 1, true)))
.build();
let loaded_state = io::load_safetensors("model.safetensors").expect("Failed to load");
new_model.load_state_dict(&loaded_state).expect("Failed to load state");
}The following utilizes the current API to define a training-ready CNN.
use volta::nn::{Adam, Conv2d, Flatten, Layer, Linear, MaxPool2d, Module, ReLU, Sequential};
fn main() {
// 1. Define Model
let model = Sequential::new(vec![
// Input: 1x28x28
Box::new(Conv2d::new(1, 6, 5, 1, 2, true)), // Padding 2
Box::new(ReLU),
Box::new(MaxPool2d::new(2, 2, 0)),
// Feature map size here: 6x14x14
Box::new(Flatten::new()),
Box::new(Linear::new(6 * 14 * 14, 10, true)),
]);
// 2. Data & Optimizer
let input = volta::randn(&[4, 1, 28, 28]); // Batch 4
let target = volta::randn(&[4, 10]); // Dummy targets
let params = model.parameters();
let mut optim = Adam::new(params, 1e-3, (0.9, 0.999), 1e-8, 0.0);
// 3. Training Step
optim.zero_grad();
let output = model.forward(&input);
let loss = volta::mse_loss(&output, &target);
loss.clone().into_loss().expect("loss is scalar").backward();
optim.step();
println!("Loss: {}", loss.item());
}Volta can load weights from PyTorch, HuggingFace, and other frameworks using SafeTensors format with automatic weight mapping and transposition.
use volta::nn::{Layer, Linear, Module, ReLU, Sequential};
use volta::io::load_safetensors;
use volta::io::mapping::StateDictMapper;
fn main() {
// 1. Build matching architecture with named layers
let mut model = Sequential::builder()
.add_named("fc1", Box::new(Linear::new(784, 128, true)))
.add_unnamed(Box::new(ReLU))
.add_named("fc2", Box::new(Linear::new(128, 10, true)))
.build();
// 2. Load PyTorch weights with automatic transposition
// PyTorch Linear stores weights as [out, in], Volta uses [in, out]
let pytorch_state = load_safetensors("pytorch_model.safetensors")
.expect("Failed to load SafeTensors");
let mapper = StateDictMapper::new()
.transpose("fc1.weight") // [128,784] → [784,128]
.transpose("fc2.weight"); // [10,128] → [128,10]
let volta_state = mapper.map(pytorch_state);
// 3. Load into model
model.load_state_dict(&volta_state).expect("converted weights should fit the model");
// 4. Run inference
let input = volta::randn(&[1, 784]);
let output = model.forward(&input);
println!("Output shape: {:?}", output.shape());
}Weight Mapping Features:
rename(from, to)- Rename individual keysrename_prefix(old, new)- Rename all keys with prefixstrip_prefix(prefix)- Remove prefix from keystranspose(key)- Transpose 2D weight matrices (PyTorch compatibility)transpose_pattern(pattern)- Transpose all matching keysselect_keys(keys)/exclude_keys(keys)- Filter state dict
See examples/load_external_mnist.rs for a complete end-to-end example with validation.
use volta::{Device, randn};
fn main() {
let device = Device::gpu().expect("GPU required");
// Create tensors on the CPU and move them to the GPU. `randn` tensors don't require
// grad, so mark `a_gpu` (a leaf) as one, or backward has nothing to reach.
let a_gpu = randn(&[1024, 1024]).to_device(device.clone());
a_gpu.set_requires_grad(true);
let b_gpu = randn(&[1024, 1024]).to_device(device);
// Operations execute on GPU automatically
let c_gpu = a_gpu.matmul(&b_gpu); // GPU matmul
let sum_gpu = c_gpu.sum(); // GPU reduction
// Gradients computed on GPU when possible
sum_gpu.into_loss().expect("loss is scalar").backward();
let grad = a_gpu.grad().expect("a_gpu requires grad");
println!("Gradient elements: {}", grad.len());
}The library is designed around a few core concepts:
Tensor: The central data structure, a handle to a shared, reference-countedRawTensor(data, shape, dtype, gradient, and device). ATensorexposes no fields, only a private handle, and downstream code cannot borrow theRawTensorinside it; read through methods such asdata(),item(),shape(),grad()anddtype(). The Dtypes note under Outstanding Issues says which dtypes compute.- Tensor methods:
Tensorhas inherent methods for all operations (e.g.,tensor.add(&other),tensor.matmul(&weights)); there is no trait to import. nn::Module: The base trait for anything with learnable parameters. Providesparameters(),state_dict(),load_state_dict()(strict: missing or unexpected keys are an error, as PyTorch's default),load_state_dict_partial()(lenient; both atomic: nothing is written unless every entry fits),zero_grad(),to_device(), andtrain()/eval().nn::Layer: AModulethat maps one tensor to one tensor withforward(). OnlyLayers can go in aSequential. Implement it to write your own layer.nn::RecurrentLayer(LSTMCell): a sealed trait for layers that carry state between steps (initial_state/step).Embeddingis a plainLayerwhose input is an I64 or I32 index tensor, as in PyTorch.Sequential::builder(): Builder pattern for composing layers with named parameters for robust serialization. Supports bothadd_named()for human-readable state dict keys andadd_unnamed()for activation layers.- Optimizers (
Adam,SGD,Muon): Structures that take a list of model parameters and update their weights based on computed gradients duringstep(). Device: Abstraction for CPU/GPU compute. Tensors can be moved between devices withto_device(), and operations automatically dispatch to GPU kernels when available.- External Model Loading:
StateDictMapperprovides transformations (rename, transpose, prefix handling) to load weights from PyTorch, HuggingFace, and other frameworks via SafeTensors format. - Vision Support:
Conv2d,ConvTranspose2d(for GANs/VAEs),MaxPool2d,PixelShuffle(for super-resolution),BatchNorm1d/2d, andDropout. - Sequence Support:
Embeddinglayers for discrete inputs,LSTMCellfor recurrent architectures.
Volta has an extensive test suite that validates the correctness of every operation and its gradient. To run the tests:
cargo test -- --nocaptureThat is a CPU-only run (the default features are empty). To build and run the GPU backend's tests as well (they skip when no adapter is found):
cargo test --features gpu,acceleratecargo test <filter> runs the tests whose names contain <filter>; there are no category modules. To run one test file, name it:
cargo test --test torch_ref_layers # layers against PyTorch values
cargo test --lib conv # library tests with "conv" in the namejust ci runs the full gate (fmt, clippy, tests and docs with no features and with gpu + accelerate, and the examples).
The examples/ directory contains complete programs demonstrating various capabilities. Those in just examples-smoke run in CI on every change; the list and the reason for each one left out are in dev-docs/DEVELOPMENT.md ("Examples smoke run").
The MNIST examples (mnist_cnn, mnist_rnn, mnist_autocast, vae, dcgan) need the dataset in data/mnist/: just mnist-data downloads and checks it. Build anything that trains on MNIST with --release.
# Basic examples
cargo run --example readme1 # Simple MLP training
cargo run --example readme2 # LeNet-style CNN
cargo run --example showcase # Feature showcase
cargo run --example layernorm_demo # LayerNorm forward and gradients
# Vision tasks
cargo run --release --example mnist_cnn # MNIST digit classification
cargo run --release --example mnist_autocast # MNIST under bf16 autocast vs f32
cargo run --example super_resolution # Image upscaling with PixelShuffle
cargo run --release --example dcgan -- 1 # Deep Convolutional GAN (argument: epochs, default 5)
# Generative models
cargo run --release --example vae # Variational Autoencoder
# Sequence models
cargo run --example char_language_model # Character-level LSTM language model
cargo run --example char_transformer_model # Character-level Transformer language model
cargo run --release --example mnist_rnn # MNIST rows as a sequence, LSTM classifier
cargo run --release --example time_sequence # LSTM sine-wave prediction (trains to NaN today, #88)
# External model loading
cargo run --example load_external_mnist # Load PyTorch weights via SafeTensors
# GPU backend (needs the `gpu` feature)
cargo run --features gpu --example gpu # GPU tensor operations
cargo run --features gpu --example gpu_training # Training on the GPU
# Regression
cargo run --example regression # Polynomial curve fittingThe next major steps for Volta are focused on expanding its capabilities to handle more complex models and improving performance.
- Complete GPU Integration: Port the remaining layers to the GPU (
ConvTranspose2dhas no GPU path), optimize GEMM kernels with shared memory tiling. - Performance Optimization: Implement SIMD for element-wise operations, optimize broadcasting on GPU, kernel fusion for composite operations.
- Learning Rate Schedulers: Cosine annealing, step decay, warmup schedules.
- Conv2d Memory Inefficiency:
im2colimplementation insrc/nn/layers/conv.rsmaterializes the entire matrix in memory. Large batch sizes or high-resolution images will easily OOM even on high-end machines. Somewhat mitigated through algorithm selection in convolutional layer. - GPU Kernel Efficiency: Current GPU matmul uses naive implementation without shared memory tiling. Significant performance gains possible with optimized GEMM kernels.
- Dtypes: tensors of every dtype can be built, read back (
to_vec_as), and cast (RawTensor::from_typed,to_dtype). Ops compute in f32, f64, f16 and bf16 on the CPU (half types in f32, rounding each output) and in f32, f16 and bf16 on the GPU (bf16, and f16 withoutSHADER_F16, as packed 16-bit emulation); integer and bool tensors are data (movement, comparisons,where, indices, class targets), and integer arithmetic panics. Mixed operands promote as in PyTorch unlessDTypePolicy::Strict.state_dictkeeps each parameter's dtype, so a model of any float dtype saves and loads bit-exact (loading into another dtype casts). - Single-threaded: Uses
Rc<RefCell>instead ofArc<Mutex>, limiting to single-threaded execution on CPU.
Contributions, issues, and feature requests are welcome! Feel free to check the issues page.
This project is licensed under the MIT License - see the LICENSE file for details.