The rule layer: deterministic guardrails around a probabilistic model

A language model is, by nature, probabilistic: the same input can give different outputs, and sometimes it errs. If you want to build a reliable system, the common mistake is to try to force the model to always be flawless. The better way is to accept that the model is probabilistic and build a deterministic layer around it that catches what slips through.

The division of labour: cognition versus rules

The foundational idea is separating two kinds of work. The model does the cognitive work: understanding context, reasoning, generating language, and decisions that aren’t rule-bound. But the rule layer — which we’ll call the harness — takes on the deterministic work: validation, recovery from error, policy enforcement, and anything that can be expressed as a clear rule. The model is free to generate; the harness is responsible for making sure what’s generated conforms to the rules.

A trigger, not just a prompt: where the rule belongs

The key question is where each rule should live. Separate three categories:

• Hard policy should be enforced in the harness, not the prompt. For example, “any output that contradicts the given context must be rejected” — a deterministic check downstream does this, not a hope that the model will respect it itself.
• Soft guidance belongs in the prompt, where the model is the decider. For example, “prefer fidelity over fluency” — the model tries to balance, but isn’t mechanically prevented from choosing otherwise.
• Format constraints are validated by the harness. The model generates structured output with its best effort, but the harness validator checks it and, on error, routes to a fallback.

Walk the data path

To design this layer well, there’s a simple but decisive exercise: walk the data path of each call. Before calling the model, what does the harness assemble? What input does the model receive, and what can it invent versus what comes from the context? After the model returns, what does the harness check or compensate for? When you walk this path, it becomes clear which responsibility belongs to the model and which to the rule layer — and that removes the most common design error.

The verifier as a guardrail

A powerful form of the rule layer is using a verifier model. When a single model can’t guarantee a critical constraint, add a second model as a guardrail: the first model generates, and the second reads both the input and the output and rejects it if a rule is violated. This is the same primary-and-verifier pattern, but here the verifier’s role is a policy guardrail: a gatekeeper that won’t let an undesirable output through the gate.

The key invariant

Remember the final principle: the goal isn’t to force the model to follow all rules flawlessly — that’s impossible. The goal is for the model to generate freely and for the harness to be responsible for making the output conform to the rules. The rule layer catches exactly what slips through the model’s training. This is the difference between a system built on the hope of a flawless model and one designed assuming the model will err — and the second is always more stable.