How to Scrape E-Commerce Sites for AI Agents Using Playwright and LLMs

TL;DR

AI agents require structured JSON data (prices, specifications, availability), but modern e-commerce sites serve heavily obfuscated, JavaScript-rendered HTML. To bridge this gap, modern scraping pipelines use headless browsers like Playwright to execute JavaScript and normalize browser fingerprints, combined with LLMs to extract schema-validated JSON directly from the rendered DOM. This approach eliminates brittle CSS selectors and scales across diverse retail layouts.

The AI Agent Data Bottleneck

Autonomous agents and LLM-powered applications rely on real-time external data. When an AI agent needs to analyze market trends, compare product specifications, or track inventory, it cannot parse raw, minified HTML effectively. Traditional rules-based web scraping relies heavily on XPath or CSS selectors to parse this HTML.

The problem is that retail engineering teams constantly deploy A/B tests, obfuscate class names using CSS-in-JS frameworks, and alter page structures. A pipeline relying on soup.select('.price-tag-v2') will inevitably fail.