I am delighted to announce that a user reported dysfunction so that I could go down the rabbit hole of fixing it. Messing around locally is one thing, but building a tolerable app for end users has other considerations when using a ‘local’ AI. This led to interesting findings about the limitations of hardware and gaining a high-level understanding of quantization and it's importance. I also explain how to creatively get around limitations. It's called flippy card, and it's an app that helps you study via custom uploaded content. Wanna see it? It's here: Flippycard
Problems:
Ollama was stuck behind cloudflare in a local loop.
Ollama needs to be accessed outside of systemctl with config file.
Ollama was running painfully slow, and needed to set it to use GPU, not CPU.
