Beyond RAG: Why I Started Thinking in Context System Design

When I started my second AI project, I thought I was going to dive deep into RAG, vector databases, embeddings, and the latest AI frameworks.

Instead, I found myself thinking about something much more fundamental.

Context.

Not prompts.

Not models.

Not even AI itself.

Just context.

It Started With a Simple Project

The original goal was straightforward: build an engineering knowledge assistant.

An assistant capable of answering questions like:

• Why was this service designed this way?
• Who owns this system?
• What happened the last time this failed?
• What breaks if I change this?
• Is this documentation still accurate?

Like many engineers entering AI, my first thought was simple: Use RAG. Retrieve relevant documents, feed them into an LLM, and generate an answer. Problem solved.

Except… it wasn’t.

The more I explored the problem, the more I realized that retrieving information wasn’t the hard part. The hard part was determining:

1. What information should be retrieved?
2. How fresh does it need to be?
3. Can it be trusted?
4. How should information from multiple sources be combined?
5. How do we know the answer is actually correct?

That realization completely changed how I viewed AI systems.

AI Isn’t the Bottleneck

Like many people, I initially viewed AI as the center of the solution. Eventually, I realized something important:

AI does not create organizational knowledge. It inherits it.

If documentation is outdated, if ownership is unclear, if architecture decisions were never captured, or if incident reports are inconsistent… the AI won’t magically solve those problems.

It will simply scale them.

A powerful model with poor context is still a poor system. That is when I stopped asking:

“How do I build an AI application?”

And started asking:

“What context does the AI need to make a good decision?”

The System Design Parallel

When I first learned backend engineering, I didn’t focus on memorizing technologies. I focused on understanding problems and trade-offs.

• When should I use Redis?
• Why PostgreSQL instead of Elasticsearch?
• When does a message queue actually make sense?
• When do I need an event-driven architecture?

Eventually, you stop thinking in tools. You start thinking in system design.

Redis isn’t universally “good” — it is good for a specific problem. Kafka isn’t automatically better — it solves a different problem. The technology comes after understanding the architectural need.

While studying AI systems, I realized I was asking the same types of questions. The question wasn’t “Should I use RAG?”The question was: “What context problem am I trying to solve?”

A New Mental Model

That is when a simple analogy clicked:

• Traditional System Design optimizes software systems.
• Context System Design optimizes the information those systems rely on.

The technologies are different. The architectural thinking is remarkably similar.

Problems Over Technologies

Once I started looking at AI systems through this lens, the ecosystem became much easier to understand. Instead of memorizing every new framework, I started mapping technologies directly to the problems they solve:

The pattern is familiar. You don’t start with Redis; you recognize a caching problem. You don’t start with Kafka; you recognize a messaging problem.

Likewise, you shouldn’t start with RAG. You should recognize a context retrieval problem.

The Context Pipeline

The more I explored, the more I realized the language model is only one part of a much larger architecture.

Knowledge Sources
      ↓
   Acquire
      ↓
    Store
      ↓
  Keep Fresh
      ↓
   Retrieve
      ↓
   Compose
      ↓
 Reason (LLM)
      ↓
  Validate
      ↓
    Learn

The LLM occupies exactly one stage.

Everything before it is software engineering. Everything after it is software engineering. The AI model is important, but the surrounding system determines whether it can succeed.

The Shift Toward Knowledge Health

This realization led me somewhere unexpected. Organizations don’t just manage software — they manage knowledge. That knowledge has a lifecycle. It is:

• Created
• Captured
• Organized
• Validated
• Consumed
• Updated
• Eventually degraded

Knowledge naturally decays. Systems change, people leave, documentation becomes outdated, and decisions lose their original context. If that lifecycle is unhealthy, no AI system can consistently produce trustworthy answers.

This shifted my interest away from simply retrieving knowledge and toward something broader: Knowledge Health.

Instead of asking: “Can the AI answer this question?”

A better question might be: “Is our organization’s knowledge healthy enough for the AI to answer it reliably?”

To me, that is the more interesting engineering problem.

Context System Design Is Still Evolving

I don’t claim that Context System Design is an established discipline. Many of the individual concepts already exist: RAG, Knowledge Graphs, Vector Search, Agent Frameworks, Tool Calling, LLMOps, and Knowledge Management.

The idea is not to replace these technologies. The idea is to provide a way to reason about them.

Just as traditional system design helps engineers choose the right architecture based on system requirements, Context System Design is an attempt to help engineers choose the right architecture based on context requirements.

This is only Version 0.1. The framework will need to evolve through real-world application. Each system designed with it will reveal:

• What works
• What is missing
• What assumptions were wrong
• What patterns repeat

The framework should improve alongside the systems it helps create.

Final Thoughts

AI introduces a new category of engineering questions:

• How do we acquire context?
• How do we keep context fresh?
• How do we determine which sources are trustworthy?
• How do we combine information from different systems?
• How do we measure the quality of AI-generated answers?

These are not just prompt engineering questions. They are architecture questions.

Models will continue to improve. Frameworks will change. Technologies will come and go. But the need for systems to provide the right context will remain.

Maybe the future of AI engineering is not only about building better models.

Maybe it is about building better context systems.