Sometimes it’s easier to show than explain: teaching Jev a visual standard

Eight labeled images improved Jev’s decisions from 57.1% to 72.1% when written instructions left part of an inspection standard unspecified.

“Pack this kit properly” sounds straightforward until you have to specify what properly means. A loose cable might be acceptable for a return. The same cable might need to be coiled before the kit can be shipped to another customer.

Sometimes it’s easier to show the distinction: this arrangement passes, this one fails. That suggests a useful way to configure AI for a business workflow. A customer could provide a short instruction and examples of the standard they want applied.

At Eventum, we tested that idea with Jev, TypeSafe’s decision model. On controlled illustrations of electronics kits, adding eight labeled images to an incomplete written instruction improved agreement with our inspection policies by 15 percentage points. Descriptions of the examples and complete written rules scored higher. The experiment shows where visual examples helped, and where they still fell short.

Figure 1. Every condition evaluated the same 120 new kit illustrations under two policies, giving 240 decisions. The query was an image in every condition. All calls used Jev 1.14.0 without fine-tuning.

We gave Jev two short instructions, which we call briefs. For the original-parts policy:

Require our approved original accessory type. Ordinary cable-packing variation is acceptable.

For the packing policy:

Compatible accessory substitutions are acceptable. The cable must meet our approved packing arrangement.

We deliberately left out what the approved adapter looked like and which cable arrangement met the packing standard. We chose those omissions before testing. The question was whether labeled images could supply the missing information.

The complete policies were simple. The original-parts policy required a blue rectangular adapter and allowed either a loose or coiled cable. The packing policy allowed that adapter or an orange rounded substitute, but required a coiled cable. Both policies required the device, cable and adapter to be visibly present.

Every condition received the same evidence rule: reject a visible failure; accept when all requirements can be verified; otherwise answer insufficient evidence. An obscured compartment could therefore require another view, while a visible violation elsewhere was enough to reject the kit.

We then supplied eight example images labeled accept, reject or insufficient evidence. In the request, each image appeared under the answer it illustrated. TypeSafe’s image documentation supports this way of defining answers through examples. [1]

Figure 2. The eight reference images used in the main comparison, with packing-policy labels. The same images were used for the original-parts policy; two labels changed. Each policy retained two accept, five reject and one insufficient-evidence example. Display labels were added for the reader.

This is visual in-context learning: examples accompany the new image and help communicate how to judge it. The model’s weights stay fixed, and the application supplies the examples again on subsequent requests.

The main comparison used 120 new illustrations, separate from the examples, with variations in background, view, tray and device appearance. Across both policies, 100 decisions should have been accept, 100 reject and 40 insufficient evidence. Expected answers came from the constructed scenes and policies.

The brief alone produced 137 correct decisions out of 240. Adding images raised that to 173. The examples corrected 43 decisions and made seven previously correct decisions wrong. The estimated 95% interval for the gain was 9.6 to 20.4 percentage points, calculated by resampling whole images with their two policy decisions together.

The improvement held in additional checks. Two other sets of eight examples scored 170/240 and 166/240. A fresh repeat of the main comparison scored 138/240 with the brief alone and 170/240 with images. These checks reused the same 120 test scenes.

The packing policy accounted for most of the gain, improving from 67/120 to 101/120. The original-parts policy moved only from 70/120 to 72/120. The difference matters: examples communicated the cable arrangement much more successfully than the required adapter type.

Figure 3. Two selected cases from the final evaluation. The examples preserved the correct acceptance and corrected the loose-cable decision. Device appearance also differs between the images. The kit illustrations are the actual inputs; captions and model answers were added for display.

Complete written rules plus images scored 213/240, or 88.8%, the highest result in this comparison. Complete rules alone scored 200/240, or 83.3%. Adding images raised correct insufficient-evidence answers from 0/40 to 18/40. The combined condition also returned insufficient evidence on five cases that the written rules alone correctly rejected.

Descriptions of the same eight examples scored 196/240, or 81.7%. Those descriptions were generated from exact visible facts in the renderer, such as a blue rectangular adapter and a loose cable. They explicitly supplied the relevant attributes, removing the need to perceive the demonstration images. Writing similarly useful descriptions of real photographs could take work; we did not measure that effort here.

With the brief plus images, Jev still returned accept on 38 of the 100 decisions requiring rejection, down from 70 with the brief alone. Of the remaining 38, 35 accepted the orange substitute under the policy requiring the blue original. Correct accept decisions remained at 100/100.

Recognizing missing evidence remained difficult too. Images increased correct insufficient-evidence answers from 10/40 to 19/40. The other 21 obscured-evidence decisions were rejected. A workflow would need to distinguish a visible failure from a case where another photograph could resolve the question.

The appeal of “show, then explain” is that the example can become part of the specification. Consider a returns team defining acceptable packing, or a refurbishment team defining condition grades. A proposed workflow would let a reviewer select images, label the accepted variations and failures, and supply them with a short instruction. Before using an updated example set, the team would check its decisions on a separate collection of reviewed cases.

That could let different customers supply their own standards to one underlying model. Our experiment demonstrates a limited version of that adaptation on rendered scenes. Real photographs and the time people spend selecting examples or writing instructions remain untested.

Sometimes a written rule is enough. Sometimes a picture helps make the intended distinction concrete. For Jev, eight examples made an incomplete instruction more useful on this task. The next practical question is which parts of a customer’s standard those examples communicate reliably.

Study details. We fixed Jev at version 1.14.0 and completed 2,360 planned requests: 200 development requests, 1,200 requests in the five-condition final comparison, 480 with alternative example sets and 480 repeats. The 120 final illustrations varied scene appearance while keeping the object designs constant. The 20 development and 24 reference illustrations came from our existing kit-image library. Labels came from scene construction. All completed responses were checked against registered inputs. Image requests included the TypeSafe-Danger-Zone-Images: true header, which enables image inputs. [1]

Relationship disclosure. Eventum is an official TypeSafe technology partner. [2] Erik Gafni, TypeSafe’s CTO, is also an advisor to Eventum. [3] The experiment and interpretation are Eventum’s work.

Sources

1. TypeSafe. Images documentation.
2. Eventum. In-context learning at scale: what Jev changes.
3. Eventum. About Eventum.