While developing multi-agent systems and LLM integrations in Rust, I found myself repeatedly confronting two related architectural challenges.
The first was an operational divide between two layers of infrastructure: a lightweight model gateway for routing and caching LLM requests, and an agent orchestration runtime responsible for tool execution, planning, and task resolution. Teams often end up deploying two separate systems for these workloads, duplicating provider configurations, authentication layers, and connection pools.
The second challenge was the "stateless turn" problem: many agent loops initialize each execution turn from scratch ([system_prompt, current_user_message]), immediately discarding previous tool outputs and conversational turns. While simple, this makes multi-turn reasoning brittle and deprives agents of contextual continuity.
This article shares how we approached these challenges around a unified Tokio-based architecture, and how we recently implemented an opt-in, four-tier context engine to give agents persistent conversational memory without sacrificing backward compatibility.
The Dual-Mode Architectural Pattern
