Hallucination Is the Feature, Not the Bug

Frank Coyle of UC Berkeley has been doing computer science for thirty-five years. In a recent talk on why agentic systems need ontologies, he says something almost no one is thinking about:

People worry about hallucinations, but that's the feature. It's who we are. We imagine things that may not exist, and then we turn them into reality.

He's right. The willingness to produce something that isn't in the training data is the same property that lets a model draft a plan, propose a hypothesis, or write code nobody has ever written before. Imagination and hallucination are one and the same: you cannot keep one and delete the other.

But this is also the tension every company deploying agents is living with today. The imagination is why you want the model. The imagination is also how you end up with data you simply cannot trust: a renewal date nobody agreed to and a deal stage that doesn't even exist.

The resolution isn't to suppress the mechanism. It's to be precise about where imagination is allowed to operate. An agent deciding how to answer, which systems to consult, what to check first, what a complete answer would include, is imagination doing its job. An agent deciding what is true about a specific customer, system, or process should not have the latitude to imagine anything. Those are two completely different jobs, and most AI deployments hand both to the same probabilistic engine.

Enterprise AI without a layer of ontological truth is not imperfect. It is fundamentally broken. Point the best model in the world at a business where the CRM, billing, and support systems each hold a different version of the customer, and there is no fact for it to retrieve. There is nothing for it to be right about. Every answer becomes imagination looking like fact: fluent, confident, cited to nothing, and wrong in ways nobody can audit.

Agents raise the stakes. A chatbot's hallucination is one bad answer on a screen, read by a person who might catch it. An agent's hallucination compounds. The invented fact becomes the premise for the next tool call, then the next, until a deal stage that never existed has routed a real approval. Errors that used to be reviewed by humans are now consumed by machines...at machine speed.

And none of it is fixed by a "better" model, a bigger context window, or more tokens. Those buy fluency, and fluency without grounded truth just makes the guess more convincing. The missing ingredient is not intelligence. It is a formal, current, machine-checkable statement of what exists in your business, how it relates, and what is actually true right now.

Coyle's prescription is the classic one: pair the neural network with a symbolic layer: neuro-symbolic AI, where the model reasons and a formal ontology retains the facts. He reaches back to Tom Gruber's 1993 definition of an ontology as "a formal specification of a shared conceptualization," which is a precise way of saying: write down what exists in your world, how those things relate, and what the rules are, in a machine-readable form.

It's a fantastic talk. It's also only twenty minutes, so, predictably, it ends right where the hard engineering begins. Three things in particular deserve more than the talk could give them, and they happen to be the three things we've been hard at work developing.

Validation Is the Wrong Place for the Ontology

The architecture Coyle sketches puts the ontology downstream of the agent. The loop runs, a tool executes, and a validator checks the result against the rules before anything is committed as output. His examples are good ones: a functional property catches a second refund on the same order, a disjointness rule catches a payout routed to a support rep instead of a buyer, an enumeration catches a made-up status like "probably shipped."

Every one of those checks is worth having. But notice what they have in common: they all fire after the agent has already done the work. The agent burned its loop iterations (read: your token budget) exploring a path that the ontology could have ruled out before the first tool was even called.

He warns about this himself, in a different section. Loops made agents Turing complete, he says, and loops are also where agents drift, break, and run up the bill. The two halves of his talk are more connected than they seem: the reason agent loops thrash is that the agent is navigating by guesswork. Anyone who has asked an agent to do some hard work knows this: the harness will try something novel, back away from any failures, and set off down a new path. Rinse and repeat.

Put the ontology under the agent instead of after it. When the agent starts from a map of what exists, how it connects, and which paths are legitimate, planning stops being exploration. The question "which renewals are at risk" doesn't trigger a fishing expedition across every tool the agent can reach. It compiles into a specific walk through specific relationships, because the map already says where the answer lives. The ontology shouldn't be a linter on the output. It should be the thing informing and constraining the plan.

An Unmaintained Ontology is Bound to Fail

Coyle offers two ways to build your ontology. Top-down: get the domain experts in a room and model the business, the way his generation did in the expert systems era. Bottom-up: mine the data and let the structure emerge. He notes, almost in passing, that the top-down approach is what everyone did in the 1980s, and that it ended in failure.

That aside deserves to be the headline, because it's the obvious objection to this whole idea: the industry bet on symbolic AI once and lost. Why would it go differently now? Because expert systems didn't fail on the merits. They failed because the rules were written once, by hand, by people who then went back to their day jobs. The knowledge base was accurate on the day of the workshop and decayed every day after. A model of your business that no one maintains isn't a model of your business. It's a model of a business that used to exist.

Training doesn't escape this problem either. Fine-tune a model on your business and you've built the same decaying artifact in a different shape. The weights capture your past, not your present, frozen at whatever was true on training day. The account that churned last week, the rep who left, the pricing that changed yesterday: none of it exists inside the model, and no amount of scale fixes that. Retraining on every change is the expert-systems workshop all over again...just with a GPU training budget attached.

What's different now is that the maintenance no longer has to be done by hand. The same probabilistic models that need the ontology are also what finally make it cheap to keep one alive: they can read the connected systems continuously, observe what actually exists, and propose structure instead of waiting for a committee to write it. The two halves of neuro-symbolic AI aren't just compatible. Each one solves the other's fatal flaw.

So "top-down or bottom-up" is the wrong question. Any ontology that works in production has to be both, at the same time, forever. Someone declares what the business should look like. The platform observes what the connected systems actually contain. And the two views are reconciled continuously, with disagreements surfaced as first-class information rather than discovered later inside a wrong answer.

This is the piece that never appears on the architecture diagram. Drift between the declared model and the observed data isn't a failure state to be prevented. It's a permanent condition to be detected. Fields get repurposed, teams reorganize, a system gets replaced, and reality walks away from the model. The question is whether you find out from your platform or from your customer.

The Textbook Inference Is the Production Nightmare

My favorite moment in the talk is the smallest one. Coyle explains functional properties with fatherhood: a person has exactly one father, so if the data says Bob is Jim's father and also that BB is Jim's father, the reasoner concludes Bob and BB must be the same individual. Two names, one person. A free inference, courtesy of the ontology.

On a slide, that's elegant. Against real enterprise data, that exact inference is the hardest problem to solve when truth is paramount.

Here's what the same logic meets in the wild: one environment we run has 160 contact records spread across six systems, describing 81 actual people. The CRM, the support desk, two enrichment tools, and an outreach platform each hold a partial copy, with no shared key, names spelled three ways, and emails that agree only sometimes. Run a strict merge over that, whether driven by logical rules or string matching, and confident, terrible things happen. Two different people who happen to share a name become one record. One person becomes five different entities, only because her work and personal email addresses don't match.

This isn't a contrived example. There are only two types of businesses: those that know their data is messy and those that pretend it's not.

The production version of "Bob and BB are the same person" can't be a rule. It has to be a judgment with a confidence attached: these two records are probably the same person, here's the evidence, and here's the case that requires a human in the loop. This kind of uncertainty can't stop at the merge. It has to travel into the answers. When someone asks how many customers are at risk, a trustworthy system also says "and there are two records I haven't been able to resolve that could change this count." That sentence, not the merge itself, is what separates an unreliable system from one you can act on.

This Is Exactly What We've Been Building

The reason this talk hit home is that SixDegree is, more or less, this argument shipped as a product.

We connect to your systems of record and maintain a live operational graph of the business: people, accounts, deals, teams, services, and the relationships among them. Agents don't consult that graph as a linter after the fact. They plan against it, so the model's imagination develops the approach and the graph supplies only what is factual.

The ontology is declared and observed at once. You say what your business should look like; discovery watches what your systems actually contain; drift between the two is detected and put in front of you. Identity is resolved with calibrated confidence and a human review queue for the close calls. And answers carry their provenance: which systems they drew from, how many records stand behind each entity, and what the platform is still unsure about.

Coyle closes his talk with his teaching philosophy, borrowed from Sister Corita Kent: there's no win, no fail, only make. We've been making this for a while now.

Let the model imagine the approach. Never let it imagine the facts. If you're tired of choosing between an assistant that's creative and one you can trust, we should talk.