What Is an LLM, Really?

TL;DR

• What it actually is: An LLM isn’t a human-like mind; it is an exceptionally capable prediction engine optimized to generate plausible continuations based on learned patterns and available context.
• The core risk: Because its baseline objective is to continue generating, a model will produce confident, coherent outputs even when the initial interpretation or underlying context is flawed. A better continuation does not mean the right direction.
• Where it excels: High-constraint tasks with external verification loops (e.g., code compilation, test suites, schema validation) where reality can grade the output.
• The chainsaw takeaway: Treat the model like high-powered machinery. It handles the heavy cutting, but human supervision is required to supply context, set direction, and verify the final result.

Maybe we were given a chainsaw without being taught how to use it.

There is a strange assumption surrounding large language models. We interact with them through something that looks remarkably human: a conversation.

We ask a question.

It answers.

We ask a follow-up.

It remembers the discussion.

We ask it to write code, explain a concept, analyze an argument, or create something new.

The natural conclusion is that there is some kind of intelligence on the other side of the conversation that works roughly like ours.

That assumption is useful right up until it isn’t.

I think a better way to understand an LLM is much simpler:

An LLM is a prediction system that has become extraordinarily good at generating plausible continuations from the information available to it.

That sounds less impressive. In practice, it is incredibly powerful. But understanding the distinction changes how you use the tool.

Here’s Your Chainsaw

Imagine someone hands you a chainsaw. It is an extraordinarily powerful machine.

You can cut down a tree in minutes. You can cut branches. You can process wood. You can accomplish work that would have taken hours with a hand saw.

But none of that means the chainsaw understands trees.

• It doesn’t know which tree you intended to cut.
• It doesn’t know whether the tree should be cut down.
• It doesn’t know whether you should be cutting a tree at all.

It is a tool. Its power comes from doing a particular kind of work extremely well.

LLMs are similar. The mistake is not expecting them to be powerful; the mistake is expecting them to be something they aren’t.

Prediction Is the Trick

At a simplified level, an LLM takes the information available to it and predicts what should come next. For language, that means predicting tokens.

The consequences of learning that prediction problem at scale are far more interesting than the description suggests. The model learns patterns across:

• Language and syntax
• Code and system structure
• Abstract concepts and relationships
• Reasoning paths and rhetorical styles
• World representations contained in its training data

This allows it to do much more than autocomplete sentences. It can transform information, summarize, translate, refactor, generate code, imitate styles, and explore possibility spaces.

The key word is patterns. The model is exceptionally good at continuing from patterns represented in its context and learned parameters. That is why it can feel so intelligent.

Where Prediction Shines

Prediction is not inherently a limitation — it is the exact reason the technology is useful.

Suppose you ask an LLM to continue a well-established programming pattern. There may be millions of examples of similar structures in its training. The model operates within a rich space of well-mapped patterns.

The same applies to:

• Standard technical writing and summarization
• Translation and idiom mapping
• Boilerplate code and refactoring
• Conventional architectures and data schemas
• Familiar conceptual explanations

When the target matches well-trodden ground, prediction does exactly what we want.

The Prediction Blind Spot

Now imagine asking the model to do something highly specific that is poorly represented in its training data or immediate context.

The model still has to produce an output. It cannot simply halt because the target is difficult to predict.

This creates one of the most critical characteristics of LLM behavior:

The model can produce a plausible answer even when the available information is insufficient to determine the correct answer.

This is why hallucination remains a persistent challenge. More importantly, it creates a subtle failure mode: the model can continue in the wrong direction.

The Wrong Direction Can Still Look Good

Imagine this sequence:

1. You ask: “Design this feature.”
2. The model interprets your prompt one way and produces a coherent, well-structured design.
3. You respond: “That’s not quite what I meant.”
4. The model pivots slightly and produces another coherent design.
5. You clarify again, and it modifies the artifact again.

Twenty minutes later, you are polishing something built on an incorrect initial framing. The model didn’t fail at generation — it generated effectively from the wrong starting point.

A better continuation does not necessarily mean a better direction.

Why Does It Sound So Confident?

Humans have several conversational escape hatches when they don’t know something:

• “I don’t know.”
• Asking a clarifying question.
• “I’m not sure, but here’s my best guess.”
• Refusing to answer.

An LLM has no intrinsic requirement to do any of those things. Its baseline task is simple: produce a continuation.

Research into calibration shows that model confidence does not reliably correlate with accuracy [1]. Producing an uncertainty statement and actually having a calibrated measure of ground truth are fundamentally different mechanisms.

A human feels the subjective state of “I don’t know this,” which prompts them to stop, investigate, or ask for help. An LLM requires no such internal experience to keep generating.

Sophistication vs. Understanding

None of this makes the model dumb.

A system trained on token prediction at scale can solve complex algorithmic puzzles, parse obscure reasoning chains, and synthesize non-obvious solutions.

The mistake is conflating capability with human-style comprehension:

• Observation: This system can perform remarkably sophisticated prediction.
• False leap: This system understands every problem the way a human does.

Verification Changes the Equation

Consider two prompts:

1. “Generate valid, schema-compliant XML for this data.”
2. “Make visitors feel confident when they land on this homepage.”

The first prompt has rigid constraints. It can be parsed, compiled, and objectively verified by the environment.

The second prompt might yield valid HTML, pristine CSS, and clean copy — yet completely fail its psychological objective.

This is why software engineering is such a compelling domain for LLMs. The model does not need to be the final authority when the execution environment provides immediate feedback:

Model → Code → Compiler → Test Suite → Verification

SWE-bench demonstrates this principle at scale by evaluating models against real repositories and verifiable test suites [2].

Prediction becomes exponentially more useful when the output can be mechanically evaluated.

The Chainsaw Lesson

You don’t evaluate a chainsaw by asking if it can brew coffee. You evaluate it by how well it cuts, learn its handling characteristics, choose appropriate tasks, and supervise the cut.

With AI, the right questions are:

• What does this tool do exceptionally well?
• Where are its context blind spots?
• Which workflows provide automated feedback loops?
• Where must a human strictly maintain supervisory control?

The mistake is expecting the model to provide direction, context, and quality assurance on its own. That is the equivalent of handing someone a chainsaw and telling them to “figure it out.”

The saw can drop the tree in seconds. But someone still has to decide: Which tree? Why this tree? Where should it fall? And should we be cutting it at all?

References

[1] Kadavath, S., et al. (2022). Language Models (Mostly) Know What They Know.

[2] Jimenez, C. E., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?