Hi folks, this is No Saved DATA. I dedicate this post to describe some of the features I put in Neve to make it an expressive high-level language (close to Python/PyTorch syntax), while also allowing efficient low-level code. I am sharing this now, because I believe the language has already strongs traits that allow it to be extended to other problem domains.
Current results:
Close to Python/SentencePiece in text processing + Byte-Pair Encoding (BPE) training;
Competitive with NumPy and OpenBLAS in CPU matrix multiplicaton, but with pure high-level SIMD code;
It was able to train a CIFAR Resnet faster than PyTorch, but I did not debug whether this was due to better CPU or GPU orchestration. But the LSTM was slower (mine lacked kernel fusion and other optimizations). Also, that old Neve deep learning framework was mostly implemented in C++. I am now changing it to be mostly implemented in Neve. That is, compute intensive preprocessing, automatic differentiation, parallel dataworkers and GPU kernels all in high-level.
