Turn any website into clean Markdown for your RAG pipeline (with code)

If you've built anything with retrieval-augmented generation (RAG), you know the unglamorous truth: 80% of the work is getting clean text in. Your embeddings are only as good as the content you feed them, and raw web pages are a mess of nav bars, cookie banners, ads, share widgets, and <div> soup.

This post shows a fast, reliable way to convert any URL — or a whole site — into clean, LLM-ready Markdown, with code you can drop into a pipeline today.

Why not just requests + BeautifulSoup?

You can, and for a single static page it's fine:

import requests

Turn any website into clean Markdown for your RAG pipeline (with code)

Related reading

How to Convert Webpages into Clean Markdown for LLMs (in 5ms)

Reduce LLM Token Waste in RAG with Markdown

I was burning context feeding HTML to LLMs, so I built a Markdown converter for…

Build a Token-Efficient RAG Pipeline with pgvector & Markdown

How to Build a RAG Knowledge Base from Any Documentation Site in 5 Minutes

How Our Document Ingestion Pipeline Turns Files into LLM-Ready Markdown