In the last part, we worked up from the foundational problem to the key components of a RAG system. We examined eight components and how they fit together.
These eight components can be grouped into two main pipelines: the knowledge pipeline and the query pipeline.
The Knowledge Pipeline
This pipeline turns raw source information into knowledge the system can store and retrieve later. It runs when information enters or changes and aims to produce retrievable knowledge data. For example, when a new ADR document is created in Notion, we fetch it, parse its structure into a uniform format, convert it into a consistent form, store it, and index it so it can be retrieved efficiently later.
In essence, this pipeline prepares the knowledge that the retrieval part of the RAG system will eventually search. It works much like the write path in a database system.
