In-context learning at scale: what Jev changes

Eventum’s experiments show that ICL works with TypeSafe’s Jev - and how low-cost inference, larger casebooks, and model-driven example selection could change AI customization.

Decision Models in Practice

In-context learning works with Jev. Supplying relevant labeled examples improved accuracy across all four public benchmarks we tested, without updating the model’s weights. [1]

At Eventum, an official technology partner of TypeSafe, we investigated how far that approach could go for bounded decisions: choosing an answer from a defined set. On BANKING77, a 77-category banking-intent task, accuracy rose from 80.75% without examples to 88.73% with retrieved examples, and to 90.71% with a request configuration selected on separate development cases. Relevant examples also improved emotion classification, recognition of requests outside a known intent taxonomy, and document-level contract inference. [1, pp. 1-3]

We then used Jev to choose examples for another Jev call. That pipeline reached 92.95% on BANKING77, versus 91.95% for a fixed lexical-selection control, and 76.00% on Emotion, versus 73.10%. In a separate matched experiment, supplying all 2,446 labeled banking examples outperformed retrieving 32 from the same library. These were evaluated on complete official test sets, not just a handful of demonstrations. [1, pp. 2, 68-70]

The economics make those findings especially interesting. On October 14, 2026, TypeSafe released Jev 1.14 alongside experimental extended-context access, increasing the opt-in per-question allowance from 32,000 to approximately 262,000 tokens. At the published input rate, 100,000 input tokens cost $0.0042 - less than half a cent. A request near the expanded limit costs about 1.1 cents in input tokens. [2] [3]

This is more than an opportunity created by cheap tokens. We have evidence that relevant examples help, that Jev can help select them, and that a substantial casebook can improve results when the task benefits from it.

The potential shift is from customizing every decision through a separately trained model toward customizing a shared decision model through maintained examples and explicit criteria. Our experiments show useful ways to do that with Jev, while identifying where training and additional production engineering remain necessary.

A different way to customize AI

Fine-tuning and in-context learning introduce task-specific information in different places. Fine-tuning updates model weights using training examples. ICL leaves those weights unchanged and supplies examples alongside the new input. Both approaches use a trained model; ICL is not learning without pretraining. [4]

Fine-tuning can be an excellent fit for stable tasks with substantial labeled data. A specialized model can apply the resulting behavior without receiving its training examples again on every call. The training and deployment work can be spread across many predictions, and trained models can outperform ICL as more data becomes available. [5]

ICL offers a different kind of flexibility. An application could supply one customer’s examples for one request, another customer’s for the next, and a revised set when a policy changes. A team developing a document taxonomy can evaluate corrected examples without first producing a new checkpoint. The underlying model stays the same; the task-specific material changes.

We call the labeled examples supplied in a request a casebook. A larger training library can hold the examples from which each casebook is selected. That selection can depend on the current input: a case near the boundary between two categories may need different precedents from a straightforward one. [1, pp. 5-7] [7]

The recurring cost has been an important trade-off. An ICL system may process its examples again for every decision. Hundreds or thousands of demonstrations consume input tokens, compete with the current document for context space, and add inference work. Adding examples also does not guarantee improvement: many-shot research has found substantial gains, but their size depends on the task and model. [5] [6]

Jev changes the cost of exploring that flexibility. The prospect is not that every classifier should become an ICL system. It is that runtime customization can become a serious option for more workflows, especially where requirements change or differ across customers.

When reading the casebook becomes inexpensive

At Jev’s published rate of $0.042 per million input tokens, a workload processing 100,000 input tokens per decision across 100,000 decisions would cost $420 in input tokens. There is no separate output-token charge. [3]

Calculated from the published rate checked on October 16, 2026. Each input includes the target, instructions, and examples. These single-stage scenarios exclude retries, development, retrieval infrastructure, and human review. They are not throughput measurements or complete production-cost estimates.

For perspective, the same 10-billion-token workload would cost $30,000 at an illustrative input rate of $3 per million tokens. That is a price scenario, not a like-for-like comparison of model capabilities. Prompt caching can also reduce repeated-context costs on other platforms; a deployment comparison should account for actual cache-hit patterns rather than assume all tokens are uncached. [8]

The expanded window addresses another constraint: what fits. It creates room for thousands of short demonstrations or richer collections of document examples, rather than requiring every casebook to fit the default allowance. It does not mean that each request should use the maximum length.

TypeSafe explicitly describes extended context as experimental and warns that it is slower and less accurate than staying within the default limits. We pinned jev-1.14.0 and tracked the context setting separately. The larger allowance requires the TypeSafe-Danger-Zone-Long-Context: true header and is also available for Jev 1.13. [2] [3]

Low cost makes large casebooks affordable to investigate. The experiments determine whether they are useful.

Relevant examples improved all four public tasks

We began with a direct comparison: hold the reference request fixed and change whether it receives retrieved examples.

The tasks cover different kinds of decisions. BANKING77 distinguishes closely related banking intents. Emotion classifies short texts into six emotion categories. CLINC150 includes requests that fall outside the supported intents. ContractNLI asks whether a complete contract supports, contradicts, or does not mention a claim. [11] [12] [13] [14]

Original ordinary-context comparisons. “Selected request” means a complete request configuration chosen on separate development cases, not the later Jev example-selector pipeline. All official test records remain included. [1, pp. 1-3, 23-24]

Retrieved examples improved accuracy on all four tasks, with paired intervals remaining positive after adjustment across the eight designated primary comparisons. Request selection added a supported gain on three tasks, but not CLINC150. [1, pp. 1-3]

The original search tested 257 named configurations per task and example method, varying wording, layout, category representation, and ordering. Finalists advanced from 48 search cases to 200 separate confirmation cases before official testing. This evaluates complete configurations; it does not isolate a universally better phrase. [1, pp. 8-9]

For ContractNLI, retrieval requested 32 demonstrations, but only 2 to 28 whole examples fit alongside each target. The selected configuration also retained one fewer example on 442 targets. Its improvement therefore includes packing and ordering effects. [1, pp. 20-22]

The results establish useful ICL with Jev. They also show why the next question is not simply how many examples the model can accept, but which examples it should receive.

Jev choosing examples for Jev

A similarity search asks which records resemble the target. A demonstration selector has a different objective: choose examples that help the final classifier make the right distinction.

For a signature-status workflow, “sent for signature” and “signed by both parties” may share much of their vocabulary while falling on different sides of the decision boundary. A useful casebook should help distinguish them, not merely repeat nearby language. That is the motivation for the architecture; its value must be measured in the final answers.

Our pipeline retrieved 32 labeled candidates using TF-IDF, a lexical method based on weighted word and word-pair overlap. Jev scored their estimated usefulness. Application code retained the eight highest-ranked examples, and another Jev call made the final classification. [1, pp. 68-70]

We compared that pipeline with two fixed controls: the first eight lexical matches, and a selection favoring category diversity. All three used the same candidate pool, eight complete examples, and the same final classifier prompt. [1, pp. 68-70]

The gains over the first-eight control were 1.01 percentage points on BANKING77 and 2.90 on Emotion, calculated before rounding. All four selector-control intervals remained positive after adjustment for those comparisons. [1, p. 69]

The controls were specified retrospectively, after the selector outcomes were known. Selection and initial example ordering changed together, and the controls did not receive their own prompt searches. This supports the tested pipeline against these fixed controls, not superiority over every retrieval system. [1, p. 70]

Nor was this an eight-label learning problem. The retrieval libraries contained 5,996 BANKING77 records and 10,792 Emotion records. The final classifier could choose every task category. These runs used ordinary context, so the selector benefit is separate from the expanded-window results. [1, p. 68]

The additional stage was inexpensive

Selection plus classification accounted for approximately $0.63 in recorded Jev input-token usage for all 3,080 BANKING77 cases, and $0.24 for all 2,000 Emotion cases. Including reservations for attempts with unresolved usage gave about $0.66 and $0.29. A separate BANKING77 calculation counting fresh selection and classification at every stage, without response reuse, was also approximately $0.63. These figures exclude development, labeling, and local retrieval computation. [1, pp. 68-69]

That is the architectural opportunity: an extra model stage can be cheap enough to consider as part of the normal decision path, provided its accuracy benefit and latency justify it.

More elaborate stages did not automatically help. Supplying the selector with histories of previous example effects did not establish a further gain. A second selection round also failed to establish an improvement on either full test while roughly doubling hypothetical fresh-call token costs. [1, pp. 65-77]

Cheap inference makes more architectures affordable. Evaluation tells us which ones are worth keeping.

Sometimes, the full casebook was better

The expanded context window enabled a different experiment: let Jev read the entire available casebook.

With the same fixed pool of 2,446 BANKING77 training examples, we compared retrieving 32 per target with supplying all 2,446. Both conditions used extended mode. The full casebook required about 84,000 tokens and reached 91.36% accuracy, compared with 88.70% for retrieval from that pool. The gain was 2.66 percentage points, with an unadjusted paired 95% interval of 1.82 to 3.54 points. [1, matched-pool comparisons]

Recorded final-inference usage cost $10.92 across 3,080 full-casebook decisions, versus $0.37 for retrieval. The larger book was substantially more expensive in relative terms, but its absolute token cost may still be reasonable for a valuable workflow. Those amounts exclude development and local computation. [1, p. 2]

This comparison used a smaller training library than the preceding selector experiment. The results should not be read as a ranking of full casebooks against selectors with different available data.

More context did not consistently win. On 200 Emotion development cases, 32 retrieved examples produced 139 correct answers, while a random casebook containing approximately 6,388 examples produced 125. Composition and ordering also differed, so this is not an isolated test of context length. It shows that making more examples affordable does not make them more useful. [1, p. 3]

The engineering choice is to test a complete casebook against a selected one. Capacity is an opportunity, not an instruction to fill the window.

Define the decision before optimizing the examples

One of our largest diagnostic gains came from something more basic than selection.

The initial CLINC150 setup described its rejection category only as oos. Our request search varied presentation without repairing that weak definition. In a later diagnostic, explicitly explaining that the category meant a request belonged to none of the listed intents raised retrieved-example accuracy from 83.73% to 94.93%. Correct rejections rose from 265 to 898 of 1,000 unsupported requests, while false rejections rose from seven to 38 of 4,500 supported requests. [1, pp. 19-26]

This was a post-hoc rerun of an inspected benchmark, with a fresh baseline; it does not replace the original CLINC150 result. Its practical lesson is clear: define what the permitted answers mean before adding more elaborate machinery.

Jev’s Choice interface lets an application specify alternatives, criteria, and instructions. For a signature-status task, fully_executed might require explicit evidence that both parties signed; awaiting_signatures and insufficient_evidence describe different states. [9]

The examples must also remain separate from evidence about the current record. A precedent showing a completed agreement teaches the distinction. It does not prove that the current agreement is complete.

Where training remained stronger

The promise of ICL is not that training becomes unnecessary. It is that more customization problems may have a useful runtime alternative.

In a low-data BANKING77 comparison, Jev reading all 616 training examples reached 89.32%, versus 83.38% for the tested MiniLM training protocol. Both used the same training cases; MiniLM used 96 additional tuning labels, while the fixed Jev request used no prompt-tuning labels. This was one encoder and a bounded training search, not the best attainable fine-tuning result. The casebook fits ordinary context, although that run used extended mode. [1, matched-pool comparisons]

With 2,446 examples, the corresponding scores were 91.36% for Jev and 90.88% for MiniLM. The interval for the difference included zero, establishing neither superiority nor equivalence. With the larger BANKING77 library, the 92.95% Jev selector was numerically ahead of a classifier trained on frozen BGE representations at 91.92%, but the adjusted comparison did not establish a winner. Jev configuration selection used 248 development labels, compared with 96 for that comparator. [1, pp. 49-50; matched-pool comparisons]

On Emotion, training was decisively stronger: 92.35% for fine-tuned MiniLM, versus 76.00% for the highlighted Jev selector, using the same available training library. [1, pp. 49-50]

The choice therefore depends on the workflow. A stable, high-volume task may favor a compact trained classifier. A collection of changing, customer-specific decisions may place more value on a shared model with editable casebooks. The latter is an architectural opportunity; we did not measure customer-onboarding or policy-maintenance savings in this study.

Can examples teach a changed policy?

Public benchmark categories do not directly establish that Jev can learn an organization’s own decision rule. We therefore constructed two purchase policies and evaluated the same facts under both, including unseen numeric values and held-out combinations of conditions. [1, pp. 77-81]

In one illustrative case, an otherwise-eligible $12,597 purchase required review under a $12,000 ordinary approval limit but qualified for approval under an $18,000 limit. Given 128 examples without policy text, Jev returned both correct decisions. [1, p. 79]

Across affected facts, current-policy casebooks produced both policy decisions correctly on 20 of 40 numeric cases and 22 of 34 held-out combinations, versus one in each group with labels from the other policy. This supports partial learning from the examples, with substantial errors remaining. Increasing to 2,048 examples did not close the gap with trained tree models: reference Jev scored 192/256 and 176/256 decisions across the two splits, while gradient boosting scored 256/256 and 255/256 on the same training records. [1, pp. 77-81]

These were structured facts and programmatic labels, not a live document workflow. When the rules and facts are already explicit, ordinary code is an appropriate solution. The operational hypothesis is about applying reviewed distinctions to new records, including messy documents. That still requires representative evidence and regression tests when the casebook changes.

From useful ICL to dependable decisions

Improving the average answer is not the same as knowing which answers can be acted on automatically.

We separately fitted acceptance thresholds on 200 calibration cases per task. Neither highlighted selector, with or without selection history, found a nonempty acceptance set satisfying a 1% observed calibration-error target on either task. Under that tested rule, both accepted no test cases. Small calibration samples, tied scores, and differences between calibration and test populations also limit the interpretation. [1, acceptance analyses]

The thresholds used the returned score for the chosen category, not Jev’s separate Choice-confidence field. The finding is not that no useful acceptance policy exists. It is that higher overall accuracy did not automatically produce one at the tested strict error target. [1, acceptance analyses]

For a document-processing deployment, we would test the complete path: source evidence, an explicit decision definition, an authorized and versioned example library, retrieval or selection, the Jev answer, and an acceptance or escalation rule. The system should retain the configuration and evidence used, route uncertainty appropriately, and test changes before they can trigger workflow actions.

Latency, throughput, retries, and review effort also belong in that evaluation. These experiments did not establish large-scale production performance. The low token bill makes broader adoption economically plausible; it does not replace workload-specific validation. [1, pp. 101-102]

A different default for AI customization

Many-shot ICL and demonstration retrieval have established research histories. Agarwal and colleagues studied large demonstration sets; Bertsch and colleagues compared long-context ICL, retrieval, and fine-tuning; Rubin and colleagues learned retrieval using downstream-model feedback. AnthusAI’s public Few-Shot-Jev repository already reports Jev 1.13 experiments, and Jev-Mem explores the related use of Jev to control memory and retrieval. These are foundations for this work, not directly comparable leaderboard entries. [5] [6] [7] [10] [15]

Eventum’s contribution is evidence about how these ideas work together with Jev: relevant examples, model-assisted selection, larger casebooks, trained alternatives, policy changes, and acceptance rules. The useful gains are real within the tested comparisons. So are the limits.

The larger opportunity is to make example-driven customization a routine engineering option. Instead of immediately training a separate classifier for each decision, a team can first test whether a shared model can apply that decision’s reviewed casebook accurately enough, at an acceptable total cost.

That could mean different customers, policies, and document categories sharing one underlying model while maintaining distinct definitions and examples. Jev can support both applying those examples and helping choose which ones the next decision should use.

ICL with Jev works. For the right task, the next customization step may be a better casebook - not another training run.

Building a document workflow that needs dependable decisions? Talk to Eventum about evaluating Jev, designing the casebook, and testing the path from evidence to action.

Methods and scope

The experiments primarily used pinned preview model jev-1.14.0, recording context mode separately. The original four-task comparisons retained every official test record and removed 40 development records whose source groups overlapped the tests. Request selection, calibration, and final testing had distinct roles. Paired intervals grouped related records, including hypotheses from the same contract. [1]

The study was not preregistered: the eight-comparison primary plan was recorded after 85 final predictions existed, with only their count reported as inspected. Several later comparisons, including the highlighted selector controls, were retrospective. The intervals do not estimate repeated-call variability or correct for every exploratory choice across the program. Recorded usage, uncertain-charge reservations, and fresh-call scenarios are distinct cost quantities. The full report preserves the broader comparisons and remaining verification limits. [1]

Relationship disclosure

Eventum is an official technology partner of TypeSafe. Erik Gafni, TypeSafe’s CTO, is also an advisor to Eventum. This article reports Eventum’s experiments and analysis; the relationship does not imply TypeSafe endorsement of these comparisons. [16]

References

1. Eventum AI. Jev in-context learning: full experimental results. Study closed October 16, 2026. Primary experimental report; the named sections retain populations, comparisons, source versions, cost definitions, and limitations.
2. TypeSafe (October 14, 2026). Jev: For the builders AND dreamers.
3. TypeSafe. Models. Pricing, identifiers, and context settings checked October 16, 2026.
4. Brown et al. (2020). Language Models are Few-Shot Learners.
5. Bertsch et al. (2025). In-Context Learning with Long-Context Models: An In-Depth Exploration.
6. Agarwal et al. (2024). Many-Shot In-Context Learning.
7. Rubin et al. (2022). Learning To Retrieve Prompts for In-Context Learning.
8. Anthropic. Prompt caching.
9. TypeSafe. Choice.
10. AnthusAI. Few-Shot-Jev. Earlier author-reported experiments with a different evaluation setup.
11. Casanueva et al. (2020). Efficient Intent Detection with Dual Sentence Encoders. BANKING77.
12. Saravia et al. (2018). CARER: Contextualized Affect Representations for Emotion Recognition. The study used the DAIR.AI Emotion split configuration documented in the full report.
13. Larson et al. (2019). An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. CLINC150 with out-of-scope requests.
14. Koreeda and Manning (2021). ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts.
15. Jiang, Li, and Li (2026). Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents.
16. Eventum AI. About Eventum. Public advisory relationship.