A competitor intelligence agent enables real-time pricing visibility, automated positioning tracking, and structured competitor briefs for brands without manual research. At the center of these workflows are a researcher agent responsible for gathering data and an analysis agent responsible for generating a summary and a downstream report.
However, one of the things that makes the researcher agent's output trustworthy is its retrieval layer, since a poor retrieval-layer output can make the analysis agent's recommendation questionable. If the researcher agent searches the webpage and retrieves a challenge page, an empty response, or blocked content, every downstream conclusion becomes less trustworthy. With a 99.93% success rate on protected websites, ZenRows provides the reliable retrieval layer that makes these workflows practical in production.
This tutorial shows why a CrewAI researcher agent can fail on protected competitor pages. It starts with a custom ZenRows-based tool setup, then later shows the MCP server as an alternative approach.
Prerequisites
This tutorial works best with Python 3.10 or newer. If you are on an older Python version, create a dedicated virtual environment with a current Python installation to avoid dependency conflicts.
