I’ll be honest: I thought I could just throw an OpenAI API call into a serverless function and call it a day. Two hours later, I was staring at a 429 error, wondering why my demo chatbot kept freezing. This is the story of how I learned to build a streaming AI chat endpoint—the hard way.
The Problem
I was building a simple chatbot for my personal site. Users type a question, I send it to the AI, and display the answer. Simple, right? I hooked up a Node.js Express endpoint that called the OpenAI API, waited for the full response, then sent it back as JSON. It worked… for about 10 requests. Then the rate limits kicked in. And even when it worked, users waited 5–10 seconds staring at a spinner. Not acceptable.
What I Tried That Didn’t Work
First, I tried caching common queries. That helped a little, but every unique question still hammered the API. Then I switched to a queue system with retries—overkill for a side project, and it still didn’t solve the waiting problem. I even considered switching to a local model, but my server couldn’t handle it.
