The Output Matched the Schema. What Did the Schema Guarantee?

Compile-Time Intelligence, Part 5

A clinical guideline, an insurance policy or a legal agreement can contain knowledge that an organization wants to apply consistently. Turning it into software means translating thresholds, exceptions and qualifications into decisions a computer can execute. Done well, that could make guidance easier to apply and individual decisions easier to inspect.

AI can help draft the rule, the instructions that software will later apply to individual cases. But first, that software needs an output it can read. A missing bracket, an invented field or an unexpected category can stop the process before anyone evaluates the meaning.

Constrained decoding addresses this problem by restricting what a model can generate. Instead of asking it to follow a format and hoping it complies, the surrounding software rules out continuations that violate specified constraints. That can make generated outputs substantially easier to integrate into a working system.

An output can be perfectly readable, however, and still implement the wrong requirement. A system can accept a number without establishing that it is the right threshold. It can require a source citation without establishing that the source supports the rule. If that rule is reused, the mistake can affect many later decisions.

This is the next step in Eventum’s Compile-Time Intelligence series. Part 1 distinguished a verified formal statement from a faithful translation. Part 2 examined which mistakes tests can expose, and Part 3 asked what justifies their expected answers. Part 4 proposed a contract that keeps the source, its interpretation, tests and review decisions connected as a rule changes. [1] [2] [3] [4]

Part 5 asks how much of that contract can be enforced as a model generates its proposal, and what still needs checking afterward. The distinction is not simply between a weak format check and a powerful meaning check. It is between the requirements a check has actually been given and the ones it has not.

The opportunity is to prevent more mistakes during generation, while keeping clear what each passing check actually establishes.

A rule that fits the format and contradicts the source

Return to Part 4’s historical example: the 2017 ACC/AHA blood-pressure classification, reproduced in a 2019 performance-measures paper. Pressures here are in millimeters of mercury (mm Hg). Systolic pressure is the first number in a measurement such as 134 over 76; diastolic pressure is the second. Under that classification, 134 over 76 belongs in stage 1 hypertension. When the two values indicate different categories, the higher category takes precedence. [5]

The table’s footnote also says to base blood pressure on an average of at least two careful readings on at least two occasions. The recorded interpretation, the account of how the source becomes software requirements, includes Part 4’s decision to return “insufficient measurements” when the required evidence is unavailable. That outcome is an implementation decision made explicit, not a phrase quoted from the guideline. This is a software example, not current clinical advice. [4] [5]

Suppose a model must express the rule as JSON, a text format that stores named fields and their values. Separate software reads those fields to choose measurements and assign a category. The JSON describes the proposed rule; it is not itself a patient’s result.

A schema specifies which fields are required, what types of values they accept, and any additional restrictions. Requiring a numeric threshold lets the receiving software read it as a number. Fixing that threshold to a particular value would be a stronger, different requirement.

Our illustrative schema requires a source passage, a measurement-selection method, category definitions and a way to resolve conflicting categories. It allows either “latest” or “average” for measurement selection, and either “higher” or “lower” for conflict resolution. An outcome for insufficient measurements is optional. In this illustrative system, omitting it means the software applies no sufficiency check before classification.

A generated rule can satisfy all of those requirements and make these choices:

The candidate keeps the normal and elevated categories unchanged but shifts the systolic boundaries for stage 1 and stage 2. We will follow the consequences of those choices as each kind of check is introduced.

The candidate even quotes the correct averaging footnote in its source field.

Every required field is present. Every value has an allowed type. Every category name is permitted. Yet the rule disagrees with the recorded interpretation in four ways.

It passed the schema because the schema allowed those choices, not because the choices were faithful to the source.

Prevent malformed outputs before they exist

A language model generates tokens, pieces of text such as a word fragment or punctuation mark. A grammar defines which sequences form valid expressions. A parser reads a completed expression and turns it into a structure software can use.

In grammar-constrained decoding, the decoding engine, the software controlling generation, tracks what has already been written and restricts the next token to permitted continuations. For a field allowing only “latest” or “average”, it can prevent “yesterday” from appearing. It still permits “latest”. [6] [7]

When generation is driven by a schema, the engine translates the constraints it supports into restrictions on the text it may emit. The schema describes acceptable data; the grammar governs the text used to express it.

That intervention must be fast enough to repeat for every token. Outlines, from Brandon Willard and Rémi Louf, organizes generation around a finite-state machine, a representation of allowed transitions, and an index over the model’s token vocabulary. SynCode, from Shubham Ugare and colleagues, precomputes information from context-free grammars, which can describe nested structures such as JSON. In its JSON experiments, SynCode eliminated syntax errors. Both approaches reduce the work needed to determine which tokens remain valid. [6] [7]

Other systems connect these checks to the programs that manage generation. Guidance, developed by Scott Lundberg, Marco Tulio Ribeiro and colleagues, lets application code control and constrain a model’s output; its llguidance engine (Michał Moskal) performs the grammar checks. XGrammar, from Yixin Dong and colleagues including Tianqi Chen, and SGLang, from Lianmin Zheng and colleagues including Cody Hao Yu, also target efficient structured generation. These tools make constraints practical to enforce while writing, rather than relying only on repairs afterward. [8] [9] [10]

For our candidate, correctly enforced constraints can require the expected fields, value types and permitted labels. Numeric restrictions can also be enforced when both the schema and the engine support them.

But when “latest” and “average” are both allowed, neither is an invalid continuation. The decoder cannot reject “latest” on the basis of a requirement it has not been given.

This is also a conditional guarantee: the engine must correctly support the constraints, and generation must finish successfully. An interrupted response is not a valid complete rule merely because every token generated so far was allowed.

Check what the engine actually enforces

There is a distinction even before we reach the clinical meaning: a schema can specify more than a particular decoding engine enforces.

JSONSchemaBench, from Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins and colleagues, evaluated six structured-generation frameworks using 10,000 real-world schemas. It found substantial differences in support for those schemas. The benchmark separates three questions that are easy to conflate: how fast generation runs, which constraints are supported, and how well the resulting answers perform. [11]

Google’s Gemini documentation illustrates the issue directly. It describes support for a subset of JSON Schema and states that string-length constraints such as minLength and maxLength are not enforced during constrained decoding. It recommends application-side validation, including handling schema-compliant outputs that are semantically incorrect, meaning wrong in what they say rather than how they are formatted. [12]

A validator checks an output against specified requirements. Running an appropriate full-schema validator after generation can detect violations of constraints the decoding engine did not enforce. It does not repair the answer automatically, and it cannot supply a requirement missing from the schema itself.

For our candidate, this distinguishes an unenforced constraint from an absent one. A full-schema validator can catch the former. It still accepts “latest” when the schema permits it, so perfect enforcement alone would leave the four source mismatches intact.

Does constraining the format improve the answer?

Enforcement raises another question. Restricting the available continuations changes how the model chooses an answer. Does that improve its decisions, or only make them easier for software to read?

Grammar-Aligned Decoding, from Kanghee Park and colleagues including Nadia Polikarpova and Loris D’Antoni, shows that ordinary token filtering can distort the probabilities assigned to complete valid outputs. Its method seeks to sample more faithfully from the model’s probabilities among outputs that satisfy the grammar. That is a result about sampling, not a guarantee of factual correctness. [13]

Zhi Rui Tam and colleagues found that format restrictions could degrade reasoning performance in their experiments. CRANE, from Debangshu Banerjee and colleagues including Shubham Ugare, investigates a response: allow intermediate reasoning while constraining the final answer. Conversely, JSONSchemaBench reports downstream improvements in its evaluated setups. [11] [14] [15]

These findings support evaluating both format compliance and task performance. A format can make an answer easier to consume without making its interpretation more reliable. Any explanation a model supplies is another proposal to inspect, not a substitute for checking the rule.

To prevent more than format errors, a checker needs more than a format description. The next useful source of information is the environment in which the rule will run.

Use context to restrict what the rule can refer to

Suppose the generated rule refers to a blood-pressure field the application has never declared. The text may be valid JSON, but the application cannot use that name as intended. Detecting this requires a list of available inputs and what kinds of values they contain.

Scope determines which names are available at a particular point in a program. Type rules determine which kinds of values can be used together, such as whether an operation expects a number or text. Static analysis examines code without running it to check properties of this kind.

Synchromesh, from Gabriel Poesia and colleagues including Oleksandr Polozov and Vu Le, uses constraints on partial programs to enforce syntax, scope, typing rules and contextual logic. Its evaluations cover database queries, visualization specifications and dialogue programs. The added information is not just “what does a valid program look like?” but “what is valid in this particular context?” [16]

Monitor-guided decoding, from Lakshya Agrawal, Aditya Kanade, Navin Goyal, Shuvendu Lahiri and Sriram Rajamani, consults static analysis of the surrounding codebase during generation. Type-constrained decoding, from Niels Mündler and colleagues, applies type-system constraints to generated code, including TypeScript, a programming language with a type checker. In its synthesis and translation experiments on HumanEval and MBPP, this approach reduced compilation errors by more than half on average. IterGen, from Shubham Ugare and colleagues, allows structured generation to backtrack and revise earlier output when an application detects a problem. These approaches bring additional checks into the generation process instead of waiting for a finished program. [17] [18] [19]

For our guideline example, a comparable check could limit the rule to declared inputs and enforce units if the representation records them. It could reject an undeclared field or a value of the wrong type.

But a declared input is not necessarily the right input. A hospital may expose both a latest measurement and an average. Both can be valid numbers in the same units, while only one satisfies the recorded requirement. Nor does a declaration prove that the site’s data mapping supplies what it promises. That is the binding problem from Part 4. [4]

Some contextual constraints can be built into a task-specific grammar; others are more conveniently enforced by a separate checker. The distinction is the information used, not an absolute boundary between what grammars and other tools can express.

Even after every referenced input is valid, the rule’s conditions must still work together. That is a different question from whether its individual fields are acceptable.

Check how the completed rule fits together

Consider a requirement for the completed classification rule: every pressure pair in its declared input domain should receive a category. The evaluator is the software that reads the proposed rule and classifies each case. This illustrative evaluator accepts nonnegative whole-number pressure pairs; fractional averages and measurement validity need their own specified handling in a real system.

The candidate calls a pair normal when systolic is below 120 and diastolic is below 80, and elevated when systolic is 120-129 and diastolic is below 80. Its erroneous hypertension categories use systolic 140-159 or diastolic 80-89 for stage 1, and systolic at least 160 or diastolic at least 90 for stage 2. When more than one category matches, it chooses the lower one.

A solver, software for analyzing logical conditions, can search for a pair that satisfies none of those categories.

The candidate leaves a gap. At 134 over 76, the systolic value is too high for its elevated category but too low for its stage 1 category; the diastolic value does not qualify for either hypertension category. The solver can expose that input without knowing the correct clinical thresholds.

But it has found a gap, not established how to repair it.

A second check might require an insufficient-measurements outcome whenever the rule’s measurement-selection method is set to “average”. That check does not fire here because the candidate chose “latest”. One error has hidden another from a validator that trusts the candidate’s own configuration.

These properties can also guide generation incrementally. Checking the completed rule is often a convenient implementation choice, not proof that earlier enforcement is impossible.

Why assess the assembled rule rather than only its parts? TypeWeaver, by Ming-Ho Yee and Arjun Guha, illustrates the importance of choosing the right unit of evaluation in a different setting. It inserts predicted TypeScript annotations, declarations of the types code should use, and checks the result. With the best evaluated model, 69% of files type checked, but only 21% of the 513 packages, collections of related files, did. These are type-checking results, not rates of functional correctness. Success for individual files did not establish success for their packages. [20]

Rejection helps only when the check asks the right question

Once a validator finds a problem, its result can guide another attempt. Instructor, Jason Liu’s Python library, supports this generate-validate-retry workflow: parse the output into an object with declared field types, run validators, and return failures to the model, up to a configured retry limit. Its documentation includes a validator that checks whether a quotation appears in supplied source text. [21]

That quotation check would pass for our candidate. The averaging footnote is real. The rule still uses the latest measurement.

My colleague Chen Li calls this false grounding in Seven Ways Clinical AI Fails, and the Control Each Requires: attaching a valid citation to a claim the passage does not support. [22]

Validation and retries can improve adherence to the checks supplied. They cannot turn a quotation-existence check into a judgment about whether the quoted requirement was implemented.

Establish which requirements came from the source

Suppose a retry responds to the gap check by extending the candidate’s elevated systolic range from 129 to 139. Every nonnegative whole-number pressure pair now receives a category, and its numerical thresholds remain ordered.

It is still not the 2017 classification. A pressure of 134 over 76 now receives “elevated” rather than “stage 1”. The upper boundary of 139 is familiar from the prehypertension category of JNC 7, a 2003 U.S. hypertension guideline. But inserting it here creates a hybrid rule, not a faithful implementation of either guideline. [5] [23]

This is the limit of checking internal consistency. The candidate cannot determine from its own structure which edition it was supposed to implement.

Could a stronger schema prevent the mistake? Yes. It could permit only “average”, require an insufficient-measurements outcome, enforce “higher” precedence and fix the category thresholds to the reviewed values.

But then the schema contains part of the interpretation. It needs the same source links, review and versioning as the generated rule.

Source-based constraints can therefore be enforced before or during generation, not only afterward. Establishing them is still a separate task. Moving a requirement into the schema changes where it is enforced, not where its justification comes from.

There are two complementary ways to gather evidence for that connection: judge whether the source supports a proposed rule, or turn a reviewed interpretation into expected answers and test the rule against them. The first examines the proposed translation; the second makes particular decisions repeatably checkable.

A model can help judge support, without becoming the authority

A support check must consider the proposal and its evidence together. ODKE+, from Samira Khorshidi and colleagues, separates that judgment from structural constraints in knowledge extraction. An ontology, a description of entity types and permitted relationships, guides the extracted facts. A second model, called the grounder, then judges whether the evidence supports each fact. [24]

A comparable check in our example would ask whether a rule using “latest” is supported by a passage requiring an average, not merely whether the quotation exists. That is a possible transfer of the approach, not a clinical result from ODKE+. The grounder’s judgments still need evaluation against reviewed examples. Its role is to add evidence about support, not to turn a second model into an unquestionable authority. [24]

Turn reviewed requirements into revealing tests

The second approach makes reviewed source requirements executable as tests. A test oracle, the criterion used to judge an answer, can be a recorded expected result rather than another model. Part 3 explained why that expectation needs its own justification. [3]

For the toy evaluator, four cases expose the four mistakes. The first two hold the average and latest value equal so that they test category logic without also testing measurement selection:

For the last case, systolic measurements of 110 and 114 on one occasion and 124 and 128 on another average 119; every diastolic measurement is 70. Assume these synthetic measurements meet the recorded quality requirements. Both the candidate and intended rule use the same normal and elevated ranges, so this case exposes the choice of latest versus average without relying on the wrong hypertension thresholds.

These four cases are not a complete clinical validation set. Their expected answers still depend on the justified interpretation, and passing them does not establish behavior on every untested case.

Here, the errors were known when the cases were chosen. Can automation help discover revealing cases when they are not? 3DGen, from Sarah Fakhoury and colleagues including Shuvendu Lahiri, combines generated specifications with generated tests. Its agents translate Internet-protocol documents known as RFCs, along with examples, into 3D, a language for specifying the layout and allowed contents of binary messages. EverParse then generates verified C parser code from that precise description. [25]

The proof connects the parser to the supplied specification, not to the original RFC. To examine that other connection, 3DGen uses symbolic analysis, reasoning over possible input values, to generate test messages. The external oracle Wireshark, a network-packet analyzer, labels them as accepted or rejected. The authors also found 11 cases where RFC constraints were not enforced by Wireshark, so the comparison tool was not interchangeable with the source. [25]

For the guideline example, the analogous opportunity is to generate cases around disputed thresholds and input-selection choices, then establish their expected results against the reviewed interpretation. The representation enables automated checks; the source gives those checks their intended target. Neither replaces the other.

Make each guarantee specific

The checks form a division of work, not a rigid sequence. Some can guide generation; others are convenient to run on a completed rule. What matters is the requirement supplied to each check and the evidence its result provides.

In the worked example, the format check accepts the proposal. The whole-rule check exposes a gap. A retry can close that gap without fixing the interpretation. Source-linked tests then reveal what is still wrong. A single “valid” label would hide those differences.

Constrained generation remains a substantial engineering gain. It can prevent entire classes of malformed proposals and make further checks easier to automate. The opportunity is to move well-supported requirements into enforceable constraints while keeping their justification visible.

The useful question is not simply whether an output matched its schema. It is which requirements were encoded, which were actually enforced, and what evidence connects them to the source.

Part 4 argued that the interpretation must remain attached to the rule. Part 5 extends that obligation to the machinery that constrains and judges the rule. A schema, a validator and an answer key can each contain decisions about meaning. Those decisions should not become invisible merely because they are enforced automatically.

The next question is how to evaluate the translation itself: by its answers, by comparison with a reference program, or by whether it expresses the same meaning as its source.

Building a system that turns policies, contracts, or clinical guidance into executable logic? Talk to Eventum about the translation, evaluation, and review process around it.

Sources

1. Bressler D. AI Is Discovering New Mathematics. The Hard Part Is Knowing Exactly What It Proved. Eventum AI, October 13, 2026. Compile-Time Intelligence, Part 1.
2. Bressler D. The Rule Passed Its Tests. What Did the Tests Establish? Eventum AI, October 19, 2026. Compile-Time Intelligence, Part 2.
3. Bressler D. AI Can Write the Tests. Who Checks the Answer Key? Eventum AI, October 19, 2026. Compile-Time Intelligence, Part 3.
4. Bressler D. The Guideline Compiled Cleanly. Where Did the Interpretation Go? Eventum AI, October 22, 2026. Compile-Time Intelligence, Part 4.
5. Casey DE Jr, Thomas RJ, Bhalla V, et al. 2019 AHA/ACC Clinical Performance and Quality Measures for Adults With High Blood Pressure. Journal of the American College of Cardiology 74(21), 2661-2706 (2019). Table 2 reproduces the historical classification and its averaging and category-precedence requirements. The toy evaluator’s domain and insufficient-measurements outcome are explicit implementation choices, not a complete clinical workflow.
6. Willard BT, Louf R. Efficient Guided Generation for Large Language Models. arXiv:2307.09702 (2023). Introduces the approach implemented in Outlines.
7. Ugare S, Suresh T, Kang H, Misailovic S, Singh G. SynCode: LLM Generation with Grammar Augmentation. arXiv:2403.01632 (2024). The syntax-error finding concerns the evaluated JSON-generation experiments.
8. Guidance contributors. Guidance and llguidance. Open-source project documentation, accessed October 26, 2026. Describes programming interfaces and grammar-constrained generation.
9. Dong Y, Ruan CF, Cai Y, Lai R, Xu Z, Zhao Y, Chen T. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv:2411.15100 (2024).
10. Zheng L, Yin L, Xie Z, Sun C, Huang J, Yu CH, Cao S, Kozyrakis C, Stoica I, Gonzalez JE, Barrett C, Sheng Y. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024; arXiv:2312.07104.
11. Geng S, Cooper H, Moskal M, Jenkins S, Berman J, Ranchin N, West R, Horvitz E, Nori H. JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models. arXiv:2501.10868v3 (2025). Evaluates schema support, efficiency and downstream quality; these are separate dimensions.
12. Google. Structured outputs, Gemini API documentation. Accessed October 26, 2026. See JSON schema support, best practices and limitations. Support statements are specific to the documented implementation and can change.
13. Park K, Wang J, Berg-Kirkpatrick T, Polikarpova N, D’Antoni L. Grammar-Aligned Decoding. NeurIPS 2024; arXiv:2405.21047. Studies the distribution induced by grammar-constrained sampling.
14. Tam ZR, Wu CK, Tsai YL, Lin CY, Lee HY, Chen YN. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. arXiv:2408.02442 (2024). Findings concern the tested tasks, models and formatting conditions.
15. Banerjee D, Suresh T, Ugare S, Misailovic S, Singh G. CRANE: Reasoning with Constrained LLM Generation. ICML 2025; arXiv:2502.09061. Studies reasoning-augmented constrained generation.
16. Poesia G, Polozov O, Le V, Tiwari A, Soares G, Meek C, Gulwani S. Synchromesh: Reliable Code Generation from Pre-trained Language Models. arXiv:2201.11227 (2022). Evaluates constrained semantic decoding and example selection on SQL, Vega-Lite and SMCalFlow.
17. Agrawal LA, Kanade A, Goyal N, Lahiri SK, Rajamani SK. Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context. NeurIPS 2023. The arXiv version is titled Guiding Language Models of Code with Global Context using Monitors.
18. Mündler N, He J, Wang H, Sen K, Song D, Vechev M. Type-Constrained Code Generation with Language Models. Proceedings of the ACM on Programming Languages 9 (PLDI), 2025. Open preprint.
19. Ugare S, Gumaste R, Suresh T, Singh G, Misailovic S. IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking. ICLR 2025; arXiv:2410.07295. Describes grammar-guided forward generation and backtracking.
20. Yee MH, Guha A. Do Machine Learning Models Produce TypeScript Types That Type Check? ECOOP 2023. Open preprint. The 69% file and 21% package results concern the best evaluated prediction model on a dataset of 513 packages.
21. Instructor. Validation and Reasking. Documentation, accessed October 26, 2026. Covers validators, bounded retries and a source-text quotation check.
22. Li C. Seven Ways Clinical AI Fails, and the Control Each Requires. Eventum AI, October 22, 2026. The false-grounding discussion distinguishes a source’s existence from its support for a claim.
23. Chobanian AV, et al. The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure: The JNC 7 Report. JAMA 289(19), 2560-2572 (2003). Its historical prehypertension range is not the 2017 elevated category.
24. Khorshidi S, Nikfarjam A, Shankar S, et al. ODKE+: Ontology-Guided Open-Domain Knowledge Extraction with LLMs. arXiv:2509.04696 (2025). Section 3.3.1 describes the grounder. The paper reports 19 million ingested facts and 98.8% precision for the combined production pipeline, not the grounder alone or a clinical evaluation.
25. Fakhoury S, Kuppe MA, Lahiri SK, Ramananandro T, Swamy N. 3DGen: AI-Assisted Generation of Provably Correct Binary Format Parsers. ICSE 2025; arXiv:2404.10362. Distinguishes verified parser generation from test-based refinement of the specification and documents limitations of the Wireshark oracle.