How to Use an LLM Effectively: Context, Memory, and Tools

Author’s Note: This is the third article in a series on working with large language models, drawn from daily practice across software development, long-form creative writing, and conceptual research.

Article 1: What Is an LLM, Really? established the core mental model: an LLM is a probabilistic prediction engine, not an artificial mind.

Article 2: How to Use an LLM established the interaction model: directing the tool through bounded tasks, stopping points, and iterative verification.

Article 3 (This piece): Examines the deeper systems question: How do we build the environment of context, memory, and tools around the model so long-running projects don’t fall apart?

Once you understand how to direct an LLM through bounded tasks, you immediately run into a second barrier:

What information should the model actually have?

When projects grow beyond a single conversation, context begins to degrade.

Details get lost. Motifs drift. Code breaks existing conventions. The model forgets what happened three turns ago, or becomes paralyzed by conflicting rules you gave it earlier.

The natural impulse is to solve this by dumping more information into the prompt: more rules, more background, more documentation.

In practice, that makes the problem worse.

Effective human-agent collaboration requires Context System Design: treating context not as a giant text dump, but as a deliberate architecture of tiered memory, modular skills, and external tools.

1. The Myth of Maximum Context

Language model context windows have expanded dramatically. We can now feed entire codebases or complete books into a single prompt.

That technical capability has created an expensive illusion: the idea that if we give the model everything, it will understand everything.

In reality, two failure modes occur:

1. Attention Degradation (“Lost in the Middle”): Models do not process long contexts uniformly. Critical instructions and subtle constraints placed in the middle of massive prompts are easily diluted or ignored.
2. Context Pollution: When an agent is forced to hold fifteen auditing rubrics, complete style guides, and historical changelogs in context while simply trying to draft a scene or write a function, its reasoning becomes slow, defensive, and brittle.

The goal is not maximum context. The goal is sufficient, relevant context.

Don’t force the model to hold your entire operating manual in its head when it only needs to turn a single screw.

2. Context Tiering & Modular Skills

How do you keep context lean without leaving the model blind?

The solution is Context Tiering — dividing information by how and when it is needed:

Tier 1: Global Core (Always Present)
  → Identity, project thesis, non-negotiables, high-level guardrails (~100 lines)
 
Tier 2: Modular Skills (Loaded on Demand)
  → Detailed checklists, drafting rules, diagnostic templates (Lazy-loaded)
 
Tier 3: Reference Artifacts (Retrieved When Needed)
  → Schemas, object ledgers, API docs, style guides

The Modular Skill Pattern

Instead of stuffing your global agent instructions with every procedural checklist you have ever invented, package those procedures into modular skills.

In our creative writing and engineering workflows, we keep the global constitution lean. When it’s time to plan, the agent loads a 3-point planning skill. When it’s time to draft, it loads a prose drafting skill. When it’s time to review, it loads an audit skill.

Each stage gets 100% of the relevant procedural guidance without cluttering the context of the other stages.

3. Grounding in Reality: Probabilistic Generation vs. Deterministic Tools

An LLM is extraordinarily capable at generative work: exploring architectures, drafting dialogue, synthesizing notes, and interpreting ambiguous requests.

It is notoriously bad at being an authoritative database.

If you ask an LLM to remember the exact balance of an account, whether code passes tests, or where a specific physical object was left five chapters ago, it will generate a plausible-sounding guess.

That is why effective systems use a simple architectural pattern:

LLM interprets → deterministic system operates → LLM communicates the result.

The model doesn’t need to guess facts that another system can determine directly. The tool provides the deterministic anchor; the LLM provides the creative synthesis around it.

4. The Library Card Pattern: Bridging Static Data via MCP

What happens when you have a massive dataset that cannot — and should not — fit into a context window?

I ran directly into this problem with my own personal ChatGPT history: 63.5MB of exported data across 584 conversations and 28,000 messages spanning several years.

I wanted my agents to draw upon those authentic conversations — the raw decompression, real-world friction, and unvarnished thoughts that birthed my essays and books.

Attempting to paste that archive into a prompt is impossible. Even breaking it into Markdown files creates context bloat that crushes attention.

The breakthrough came from building a local Model Context Protocol (MCP) server called search-gpt.

Instead of cramming the archive into memory, we indexed the messages into a local SQLite database with full-text search (FTS5) and exposed a few opinionated tools to the agent.

I wrote about this exact lightbulb moment in How 60MB of ChatGPT Exports Finally Made MCP Click for Me:

“MCP shines at bridging static data to active reasoning. When you have a large dataset that doesn’t fit into memory… MCP is the cleanest way to give your AI agent an on-demand ‘library card.’ You don’t make the agent memorize the library; you give it tools to check out the exact books it needs.”

When drafting, the agent searches the archive on demand, retrieves authentic human dialogue from three years ago, uses it as a thematic seed, and releases the rest from memory.

5. Living Artifacts as Persistent Memory

An LLM agent has session amnesia. When a session ends, its working memory vanishes.

To build long-running projects (like a multi-chapter book or complex codebase), the agent needs persistent external memory.

We accomplish this through Living Artifacts:

• The Ledger of Objects: A structured file tracking the physical state, location, and owner of recurring motifs (e.g., a chipped diner mug, keys, overcoats).
• Sensory Lexicon: Documenting established character tells and behavioral shorthand.
• Architecture Matrix: Documenting what components exist and what rules govern them.

The agent doesn’t guess continuity from conversational memory. It reads the ledger before it works, and anchors its generation in physical facts.

The Discipline of Deferred Synchronization

When you are exploring ideas, drafts change rapidly. If you force the agent to update every living artifact after every rough iteration, you cause massive thrashing and burn tokens.

A vital rule of context management is Deferred Synchronization: allow the model to iterate freely in a scratchpad or draft; only update your living state files once a milestone is formally ratified by the human.

6. The Missing Setup: Why Adding Instructions Fails

When an LLM produces bad output, our immediate instinct is to write a corrective prompt:

• “Stop explaining character emotions.”
• “Don’t use clichés.”
• “Make sure you handle edge cases.”

If you keep adding patch instructions, the model becomes paralyzed by competing negative constraints.

In our work, we discovered a universal diagnostic law:

The urge to add instructions is almost always a symptom of a missing setup.

When an LLM resorts to cheap narrator editorializing or clumsy code workarounds, it is rarely because it lacked an instruction. It is almost always because an upstream physical anchor or context was missing:

• In writing: The narrator tried to explain that a character was anxious because we never gave the character a physical action (tapping a steering wheel, letting coffee go cold) earlier in the scene.
• In code: The agent generated an awkward helper function because an existing utility was never introduced into its context.

Don’t patch the output downstream with ten prompt bandages. Step upstream, provide the missing physical anchor or context, and let the model’s prediction self-correct naturally.

7. The Non-Destructive Diagnostic Audit

When an agent produces a candidate draft or code implementation, how do you review it without destroying the baseline?

Having an agent immediately “fix” an output often makes it worse. It overwrites good work while chasing a vague critique.

Instead, separate diagnosis from execution:

1. The Non-Destructive Audit: The agent runs a dedicated diagnostic pass against explicit failure modes (e.g., checking for negative assertions, narrator commentary, broken motif continuity, or unhandled errors).
2. The Diagnostic Report: The agent quotes exact lines, identifies the mechanism that failed, and proposes physical alternatives — without modifying the source file.
3. Human Decision: The human reviews the audit report, agrees or disagrees with the findings, and only then authorizes targeted edits.

This preserves human taste, prevents destructive churn, and gives you total visibility before changes are committed.

8. The System Gets Smaller

When you begin building with AI agents, the instinct is to build elaborate machinery: complex multi-agent swarms, endless frameworks, and recursive reviewer loops.

What I actually discovered through daily practice is that the more you understand how LLMs work, the less machinery you need.

You don’t need an agent to interpret another agent that interprets a third agent.

You need:

• Clear human direction that constrains the destination.
• Bounded tasks that establish stopping points.
• Tiered context that prevents cognitive interference.
• External tools (like MCP) that act as on-demand library cards.
• Living artifacts that track persistent state.
• Non-destructive audits that verify work before altering it.

The chainsaw was never the problem.

The scalable approach isn’t making the machine responsible for everything. It is understanding what the machine is good at, what it isn’t, and building an environment where human taste and machine exploration work in harmony.