I Built a Neural Network Inference Engine From Scratch in C++ (No PyTorch, No ONNX, Just AVX2)

Why does inference need a framework at all?

Every time I ran a tiny linear model through PyTorch, I felt like I was driving a go-kart with a jet engine strapped to it. The model was a few hundred KB. PyTorch's runtime was gigabytes. Somewhere between model(x) and the actual floating-point math, an entire universe of abstraction — autograd graphs, dispatch layers, tensor metadata — was quietly eating my CPU cycles.

So I asked a simple question: what does inference actually look like with nothing in the way?

That question turned into ML-model-loader — a bare-metal C++ inference engine that loads raw binary weights and runs forward passes directly on the CPU, using the same low-level techniques that power ggml and llama.cpp: cache-tiled GEMM, AVX2 SIMD intrinsics, and INT8 quantization.

No PyTorch. No ONNX Runtime. No GPU. Just C++, some pointer arithmetic, and a CPU that's faster than people give it credit for.