Chhaya Methani is an AI leader and the principal applied science manager at Microsoft.getty​Gartner predicted 2026 will be the year of the inflection point for enterprise AI, with global AI spending slated to grow by 47%. Transitioning multistep workflows to agentic AI will be one of the key drivers of this push. However, as far as I've seen in the industry, the conversation about the shift to agentic AI systems has been focused primarily on new model releases and harness engineering. What often gets missed is the crucial role that data plays in achieving the accuracy required to realize the return on investment from AI. I believe the effective use of data is what will separate successful AI systems from cool demos.The next wave of enterprise AI adoption will reward teams that can put the right business context in front of their models at the right moments. That makes data curation, retrieval quality, ranking and context management core parts of the agent infrastructure layer, not afterthoughts.Where RAG Starts To Break DownFrontier AI models are typically trained on web-scale datasets containing trillions of tokens. However, they have not been exposed to a company’s private policies, customer context or user intent. This information is what forms a business's competitive advantage. Most AI systems try to bridge this gap by bolting enterprise context to the model using retrieval-augmented generation (RAG).This works for simple queries, but as the data grows and tasks get more complex, context starts to bloat and quality quietly drops. Two competing factors are at work here: The system may fail to pull the right document, which is a recall problem, or it may return too many noisy and irrelevant documents, which is a precision problem. Both lead to context getting polluted, eroding response quality at scale. This can result in hallucinations and low task-completion rates. The challenge for business leaders lies in finding the right balance for optimal performance.Why Bigger Context Windows Won’t Fix Bad RetrievalBigger context windows do not solve bad context. These are not model failures alone; they are architecture problems in the AI system around it. More context can mean more noise, lowering response quality, increasing latency and increasing cost per response. This turns a quality problem into an economic one. To make RAG work reliably in production, enterprises need to design for retrieval, ranking and data curation. These three priorities will help them manage the context window effectively.Recommendation Systems For AgentsRecommendation systems have long helped search, advertising and content personalization at scale. In my experience working to improve the quality of agentic systems at an enterprise scale, I've found it is important to think about data curation in the same way. An agent’s value is bounded by the fidelity of information presented to it; it is not uncommon to see multipoint accuracy gains with a streamlined retrieval service. This effectively means that the agent will become the main consumer of recommendation engines serving your enterprise data.These systems can be improved through ranking pipelines. Multistage ranking pipelines work by generating a pool of potential candidate chunks (a chunk is typically a group of 200 words) from a set of documents and then ranking them with scoring models to select the top relevant chunks. The first stage can improve recall by casting a broader net to ensure that the right answer is captured. Following stages can help find the best match for the user query.What This Looks Like In A Real WorkflowLet's look at an example in an agentic AI workflow. In a workflow, agents need to interpret user intent, break it into queries and look for relevant context in enterprise documents and databases. These candidate chunks are often retrieved through a hybrid stack. Vector stores handle semantic similarity, while keyword indexes like Lucene use BM25-style scoring to capture exact terms like names, years and other signals. The second stage ranker can then score the candidates, personalizing them for the user where appropriate, and present the most relevant context to agents. This provides the agent with the appropriate context it needs to reason through the task. Consider a simple request: “Highlight the differences in the compliance report between 2025 and 2026 and outline the risks therein.” The agent may break this intent into multiple search queries, like “2025 compliance report” and “2026 compliance report” before proceeding to reason through the risks. If retrieval returns noisy or wrong information, the agent may start with the wrong evidence and can produce a confident but incorrect answer.Without ranking, key distinctions like year, region or policy number can get lost in passages that have similar verbiage. In enterprise workflows, focusing on nuanced details is paramount.How To Know Your Data Layer Needs More InvestmentAs a business leader, it's important to continuously vet the complexity of your enterprise context and user scenarios. The more complex these are, the more important it is to optimize your data layer. As you do this, pay close attention to the following aspects:• Large Corpus: If your corpus has over a few thousand chunks, retrieval quality becomes more important.• Overlapping Vocabulary: Documents may have similar language across nuanced categories like regions, years or categories.• Multifaceted Queries: User queries or tasks may combine multiple constraints, such as geography, policy discounts and exceptions.• Relationship-Aware Data: If answers depend on work graphs, policy hierarchies or organizational permissions, graph-based databases may become part of the architecture.• High Hallucination Risk: If wrong answers create compliance issues or impact user trust, your data layer needs more guardrails.Final ThoughtsEnterprise AI agent architecture needs the same discipline as classical machine learning systems. As models become better at following instructions, I've found they also become more easily misled by bad context. Business-specific data simultaneously represents a competitive advantage and a bottleneck for agentic AI pipelines. As such, it needs to be treated as a critical part of your agent infrastructure layer. Every step of your decision process must be evaluated for impact to remove blind spots.​​​Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?