Nearly three quarters of my small transformer was a lookup table, so the dial that mattered was embedding precision, not network precision. int4 with one scale per row costs 16 KB more than one scale per tensor and recovers 2.2 of the 2.5 points per-tensor int4 loses.
An earlier experiment in this series had established something slightly deflating about a small transformer: quantising the whole network to int8 is free, and the bytes you save are better spent on count tables than on network precision. That is a useful result and it invites a sharper question, which is where the bytes actually were.
tok.weight (embedding) 1,024,000 params 71.7%
everything else 404,992 params 28.3%
Enter fullscreen mode
Exit fullscreen mode
class TinyTransformer(nn.Module): def __init__(self): super().__init__() #...
Transformer architectures are the backbone of many modern large language and generative AI models. As these models grow in size,…
An 11.9M-parameter transformer scores 83% strict on the domains it trained on and 0/8 on a single unseen noun. Three experiments…
Turning click logs into training pairs, the in-batch-negatives loss that makes the training cheap, hard-negative mining, and the…
A Blog post by Ai2 on Hugging Face
If you've built a RAG pipeline before, you know the pattern: hook up an embedding model, dump vectors...
