Tiny-NN — Fully Connected Neural Networks in C++20 + CUDA 12.8

Tiny-NN is a high-performance implementation of fully connected neural networks supporting both CPU and GPU execution. It's designed for easy experimentation and benchmarking, featuring:

• CPU execution (parallelized)
• CUDA execution with memory reuse (weights and biases uploaded only once per layer)
• Training with backpropagation and SGD
• Model serialization using json.hpp (MIT licensed) included in the repository
• Simple MNIST dataset integration and ASCII preview

Requirements

• C++20 compatible compiler
• CUDA 12.8 (for GPU support)
• CMake >= 3.24
• Python 3.12 (optional, for dataset download and preview)

Although development was done on Windows 10/11 using Visual Studio 2022, the project can be built on any OS with a compatible C++20 compiler and CUDA installation.

Setup

1. Clone or copy the repository to your machine.

git clone https://github.com/Vicen-te/tiny-nn.git
cd tiny-nn

2. Download the MNIST dataset: Using Python script (recommended):

python scripts/download_mnist.py

• This will download and save the MNIST dataset in data/minst/.
• Alternatively, you can download the dataset manually from Kaggle

3. Optional: generate a small model using Python (arguments: input layer, hidden layer, output layer):

python data/generate_model.py 128 64 10

4. Optional: preview MNIST digits:

• Python: python scripts/preview.py
• C++: ascii_preview() function in MNISTLoader

Build

Visual Studio:

• Open Visual Studio -> File -> Open -> Folder... and select the project folder.
• Visual Studio will detect CMake. For GPU usage, choose x64 configuration.
• Build -> Build All.

PowerShell / Developer Command Prompt (recommended):

Option 1: Specify all options manually

mkdir build
cd build
cmake .. -G "Visual Studio 17 2022" -A x64 -DCUDA_TOOLKIT_ROOT_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.8"
cmake --build . --config Release

• -G "Visual Studio 17 2022" selects Visual Studio 2022
• -A x64 selects 64-bit architecture (recommended for CUDA)
• -DCUDA_TOOLKIT_ROOT_DIR is optional, CMake can auto-detect CUDA

Note: The -A x64 option is recommended if you want to use CUDA on Windows. On Linux or macOS, this is not necessary.

Option 2: Let CMake detect everything automatically (recommended)

cmake -B build -S .
cmake --build build --config Release

• CMake will detect Visual Studio and CUDA if installed in standard locations
• -S is the source folder, -B is the build folder

Both methods produce the same result. Use Option 2 for simplicity and fewer manual settings.

Visual Studio 2026 (v18) + CUDA 12.8: nvcc 12.8 does not recognise the newer MSVC, and the CUDA MSBuild integration is only installed for VS 2022. Build with Ninja from a Developer Command Prompt and allow the compiler explicitly:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler" -B build-ninja -S .
cmake --build build-ninja

Run

From the build/Release folder:

tinny-nn.exe <mode>

Modes:

• train or t → Train model
• inference or i → Run inference on a sample

The CPU-vs-CUDA benchmark and the correctness check live in the test binary (unit_tests, built from test/), run from its own build folder:

unit_tests.exe compare   # or c: CPU vs CUDA inference timing + numerical check
unit_tests.exe verify    # or v: argmax check against MNIST one-hot labels

Expected output

Training (train / t)

• Training progress printed to console
• Training duration in seconds
• Saved model JSON to ./data/models/fc_digit_classification.json
• ASCII MNIST preview of a single sample image

Inference (inference / i)

• Output values of selected sample
• Maximum value and its index
• ASCII preview of the sample

Benchmark (unit_tests compare)

• CPU vs GPU inference correctness check
• Average inference timings per method
• CSV results saved to ./data/results/bench.csv

Currently, benchmark only measures inference, not training. Measuring training performance would require additional implementation.

Results

Single model: 784 → 128 → 10 (MNIST classifier)

From data/results/bench.csv, average of 200 inferences after 5 warm-up runs, batch size 1. CPU path is the multithreaded ParallelExecutor; CUDA path uses cuBLAS GEMM with weights uploaded once per layer.

input	output	CPU (ms)	CUDA (ms)	CUDA speed-up
784	10	0.036	0.588	0.06× (CPU wins)

Read this number honestly. For a ~100 K-parameter network at batch size 1 the GPU is ~16× slower than the CPU. That is the expected result, not a bug: each inference pays a host→device copy of the input, two kernel launches, a device→host copy of the output, and a cuBLAS call whose fixed launch cost dwarfs the ~100 K multiply-adds it performs. CUDA only pays off once there is enough arithmetic per launch to amortise that overhead — larger layers or batched inputs.

Both implementations agree numerically: the benchmark asserts CPU and CUDA outputs match within 8.3e-3 before timing anything (see benchmark::compare_models).

Size sweep: where does CUDA start to win?

scripts/benchmark_sweep.py generates random models of increasing size, runs the benchmark on each and writes data/results/bench_sweep.csv:

python scripts/benchmark_sweep.py --bin build/test/Release/unit_tests.exe

Measured on an RTX 4070 Ti SUPER (CUDA 12.8) with the multithreaded CPU path, batch size 1, 200 timed inferences after 5 warm-ups. CPU and CUDA outputs matched within tolerance in every run.

Model (in → hidden → out)	Params	CPU (ms)	CUDA (ms)	CUDA speed-up
784 → 128 → 10	101,770	0.036	0.600	0.06× (CPU wins)
784 → 512 → 10	407,050	0.140	0.578	0.24× (CPU wins)
784 → 1024 → 10	814,090	0.281	0.538	0.52× (CPU wins)
2048 → 2048 → 10	4,216,842	1.657	0.587	2.8×
4096 → 4096 → 10	16,822,282	6.754	0.651	10.4×
4096 → 4096 → 4096 → 10	33,603,594	13.178	0.822	16.0×
8192 → 8192 → 10	67,198,986	26.468	1.050	25.2×
8192 → 8192 → 8192 → 10	134,316,042	53.255	2.441	21.8×

Reading the curve. CUDA time is essentially flat (~0.55–0.65 ms) up to ~4 M parameters: that is the fixed cost of the host→device copy, the kernel/cuBLAS launches and the device→host copy, and it does not depend on the matrix size. CPU time grows linearly with the parameter count. The two lines cross between ~0.8 M and ~4 M parameters; from there the GPU pulls away, reaching 25× at 67 M parameters. Below the crossover (which is where the MNIST classifier lives) the CPU is the right choice — and the framework lets you pick either backend per model.

The benchmark measures inference only; training time is not benchmarked (see Notes).

Notes & Improvements

• Currently, weights W and biases b are uploaded to the GPU once per layer. The input vector is uploaded for each inference.
• cuBLAS GEMM is already used for matrix multiplications, replacing the simple custom FC kernel.
• Intermediate GPU buffers (dX/dY) are allocated per layer and batch and are not fully reused, though CUDA streams enable asynchronous execution.
• For higher performance (future improvements):
  • Reusing intermediate GPU buffers across layers and batches via CUDA streams.
  • Implementing more efficient batching and overlapping of data transfers with computation.
• Profiling can be done with Nsight Systems / Nsight Compute.