Reduce LLM Token Waste in RAG with Markdown

TL;DR

Feeding raw HTML to Large Language Models wastes tokens on markup, scripts, and styling. By rendering dynamic web pages in a headless browser and converting the final DOM to clean Markdown, you reduce token consumption by up to 90% while preserving semantic structure and improving retrieval accuracy in RAG pipelines.

The Problem: LLMs, Context Windows, and the HTML Tax

Building Retrieval-Augmented Generation (RAG) pipelines over web data introduces a specific data engineering problem. The web is built on HTML. Large Language Models operate on tokens.

When you pass raw HTML to an embedding model or an LLM context window, you pay a steep tax. You pay for <div class="mt-4 flex flex-col justify-center">, <script type="application/json">, SVG paths, and inline CSS. These non-semantic tokens dilute the actual content. They increase latency, exhaust context limits, and drive up API costs.