Prompt Engineering & Structured Output

The Merge Gate team makes their CI review job return structured findings, each with a severity field restricted to the values blocker, major or minor by the schema fragment below, and the pipeline rejects any response that fails validation. Across one month every response validates cleanly and no run is rejected. A triage review then finds that 30 percent of the findings labelled blocker are formatting preferences that no engineer would hold a merge for. What does schema validation guarantee about those findings?

"severity": {
  "type": "string",
  "enum": ["blocker", "major", "minor"]
},
"file":    { "type": "string" },
"line":    { "type": "integer" },
"message": { "type": "string", "minLength": 20 }

• AIt guarantees that the labels were applied consistently within a response, because a single validated payload cannot mix two different readings of the criteria across its own entries.
• BIt guarantees only that the severity value is one of the three permitted labels, and says nothing about whether the finding satisfies the team's written definition of a blocker.check_circle Correct
• CIt guarantees that each finding came from the diff rather than the wider repository, since the file and line fields have to resolve to positions the pull request actually touched.
• DIt guarantees that a re-run against the same diff yields the same severity for the same defect, because a constrained field removes the variation that produces label drift.

Why A is wrong: Consistency within one response sounds like something structure would buy, but validation inspects each field independently; nothing compares one finding's severity reasoning against another's in the same payload.

Why B is correct: Correct. An enum is a constraint on the set of allowed strings. It admits the label blocker regardless of what the model applied it to, so semantic misclassification passes validation untouched.

Why C is wrong: The schema requires a string and an integer in those fields, not a position inside the changed lines; resolving a line to the diff would need a separate check the pipeline is not running.

Why D is wrong: Constraining the output alphabet narrows what can be emitted, not which of the permitted values the model chooses, so a re-run can move the same defect between bands and still validate.

A document intelligence team runs an invoice extraction pipeline that reads scanned supplier invoices and emits fields against a JSON schema in which purchase_order_number is nullable. Across a sample of 2,000 scanned pages, where the purchase order number is smudged or cropped the service returns a well-formed but invented number in 6.4 percent of documents, and downstream ledger matching then fails without raising an error. The system prompt already states that unreadable fields must be returned as null. Which change most effectively addresses the logged behaviour?

• AAdd few-shot examples that pair a page whose purchase order number is obscured with the intended output holding null for that field, so the ambiguous case is demonstrated rather than described.check_circle Correct
• BRestate the instruction about unreadable fields at both the start and the end of the system prompt and mark it as a strict processing rule.
• CMake purchase_order_number a required string in the schema so validation rejects any extraction that omits it and the document can be routed for reprocessing.
• DRetry the extraction up to three times whenever the purchase order number reads as absent, and accept the first value on which two attempts agree.

Why A is correct: Correct because the failure is confined to a specific evidence state, an unreadable field, and a worked example fixes the mapping from that state to the intended output token instead of asking the model to derive it from a general rule.

Why B is wrong: Tempting because instruction placement and emphasis do measurably affect adherence, and repeating a rule is the cheapest thing to try. It is wrong here because the rule has already been stated and is being violated on a case the model has never been shown, so compliance stays probabilistic on exactly the documents that fail.

Why C is wrong: Tempting because schema validation is deterministic and a rejected document is visible rather than silent. It is wrong because removing the null option forces the model to produce a value for a field it cannot read, and validation cannot distinguish an invented identifier from a correctly read one.

Why D is wrong: Tempting because sampling several times and taking agreement is a real technique for unstable outputs. It is wrong because no retry recovers characters the page does not carry, and repeated samples of a plausible invention are quite likely to agree with each other.

A CI code review job built on Claude Code takes eleven minutes against a five minute pre-merge budget, and merges are queueing behind it. The job currently reviews every file in the repository that the changed files import, across nine finding categories. An engineer proposes moving the work to the Message Batches API, citing the lower cost per request and the freedom to run a longer pass. What should the architect do?

• AAdopt the batches route as proposed and post its findings whenever they arrive, accepting that some pull requests will have merged before their review lands on the thread.
• BKeep the current prompt and raise the job timeout to fifteen minutes, so that the review completes and reviewers are shown the full set of findings before the merge proceeds.
• CKeep the current prompt and cache the repository checkout between runs, on the basis that the checkout is the part of the job most likely to dominate the elapsed time.
• DKeep the review synchronous and cut its scope with explicit criteria, restricting it to the changed files and a named set of finding categories, since the batches route is asynchronous and cannot gate a merge.check_circle Correct

Why A is wrong: Tempting because the cost saving is real and the reviews do eventually appear, but the Message Batches API processes asynchronously within a window of up to 24 hours, so the job stops being a merge gate and defects reach the default branch unreviewed.

Why B is wrong: Tempting because it makes the job pass reliably and preserves coverage, but it concedes the budget the team set rather than meeting it and lengthens the merge queue that prompted the complaint.

Why C is wrong: Tempting because caching is standard CI practice and does trim setup, but the elapsed time here is dominated by model calls over a transitively expanded file set, so the saving is small against a six minute overrun.

Why D is correct: Correct because a pre-merge gate needs a result within the build, which rules out an asynchronous route, and narrowing the reviewed surface and the category list is the lever that reduces elapsed time while also raising precision.

The Ledgerline document platform extracts supplier name, invoice number, currency and total from scanned invoices, then validates each extraction against a JSON schema before writing it to the finance ledger. Nightly volume of 8,000 invoices is submitted through the Message Batches API, one request per invoice, and the pipeline reads the results file in the order the lines arrive and zips it against the submitted list by position. Finance reports that roughly one ledger row in nine carries the totals of a different invoice. What explains this outcome?

• ASchema validation ran before the results were paired with their source documents, so a mismatched total passed the numeric type check and was written to the ledger unflagged.
• BThe batch exceeded the size at which requests are split across shards, and each shard returns a separate results file that the pipeline concatenated in the wrong sequence.
• CResults from a batch are not guaranteed to come back in submission order, so positional zipping is unsafe and each result must be matched to its source document through the custom_id supplied on the request.check_circle Correct
• DRequests that took longer than the median to process were retried internally and appended at the end of the file, which shifted the position of every result after the first slow invoice.

Why A is wrong: Tempting because validation ordering does matter for catching bad values, but schema validation was never the pairing mechanism here, and running it later would still leave every row attached to the wrong invoice.

Why B is wrong: Plausible because large workloads often shard in other systems, but a batch returns one results set for the caller and there is no shard concatenation step the pipeline could have got wrong.

Why C is correct: Correct: ordering is explicitly not guaranteed for batch results, and the caller-supplied custom_id is the field provided for correlating each result back to its originating request.

Why D is wrong: Tempting because it offers a mechanism for reordering, but no such internal retry-and-append behaviour is defined, and inventing one distracts from the real point that order is simply not promised.