Every Alert Has to Survive an Analyst

Chen Li | Executive AI Research Advisor, Eventum

A compliance analyst opens an alert. A news article says that John Smith was sentenced for fraud in Leeds. The bank has a customer called John Smith in Leeds.

The system has found a match. But has it found the right person? And even if it has, what should happen next? And perhaps even more importantly, what does it mean for the people and companies one or more steps away from him in the network? Is this a single bad actor, or part of a pattern among a group of the bank’s customers who live in one area, make the same kind of deals or hold assets with the same broker?

Those questions separate a useful risk signal from a costly mistake. Before anyone acts, the analyst needs to understand what connected the article to the customer, what the article actually establishes, and which parts of the decision remain uncertain. Months later, someone else may need to reconstruct the same decision. And years later, we may need to ask how legislation or regulation could make such cases few and far between.

That was the kind of problem I worked on at Moody’s Analytics. Between 2018 and 2021, I helped build adverse-media screening systems for compliance, which surfaced the entities, relations and patterns that mattered to Moody’s Analytics clients, warning them of risks or confirming that past choices had been sound. In 2020, with David Bressler and colleagues, I helped build CoronaPulse, a system that tracked how the pandemic was affecting companies and industries through the news, giving each group of users the real-time information relevant to its own decisions as events moved quickly. [1] [2]

The models were different from today’s, and far less mature. We used pre-trained embedding models and built classifiers, segmentation models, boundary detectors, density estimators and other narrow models on top of them, including BERT-based models rather than generative language models. But the central engineering question was already familiar: how do you turn a large stream of unstructured documents into decisions that people can inspect and use? And how do you find the patterns and common threads in a mountain of data too large to read?

The answer was not one model that read everything and produced a coherent verdict. It was a sequence of narrower tasks, each with its own evidence and evaluation. That experience, which David Bressler, Eventum’s Managing Partner, and Zaid Marji, our Fractional AI CTO, also bring from Moody’s Analytics, shapes how we take apart what a large language model is doing when it appears to make all of those judgments at once.

The alert was the end of a chain, not a single prediction

The adverse-media system is described in a workshop paper by Rupinder Khandpur and colleagues, including me. The paper reports its deployment in Compliance Catalyst, Moody’s KYC monitoring tool. [1] [3] Its design separated questions that are not easy to distinguish from one another. Is the article about a relevant risk domain? Is the person or company being screened actually involved, or were they merely mentioned? Is the coverage adverse in terms of the types of deals that this client cares about?

These judgments cannot be separated simply by reading the overall series of events; each has to be made from the perspective of a particular business decision. An article can describe a serious fraud case while mentioning the bank’s customer only as a witness. It can concern the right company but describe a relation that does not matter for this decision. It can report an allegation that a later article shows was dismissed, or was misplaced from the start. Getting the topic right does not establish the identity, the relationship to the event, or the status of the proceeding, and each has to be judged through the lens of the client’s own decision.

The paper organized four perspectives into five components. It reported separate results for risk relevance (for a scenario), entity relevance (for an event), risk categorization, and proceedings stage; the dictionary-based adverse-sentiment component had no reported performance figure. For example, entity relevance reached an F1 score of 0.80, while proceedings-stage classification reached 0.93 on the authors’ respective held-out evaluation datasets. These were component results, reflecting performance on well-circumscribed tasks, not a claim that the entire workflow was correct 93% of the time. [1]

That separation matters when something goes wrong. An undifferentiated alert score tells you the system made a mistake. Separate decisions help you find the mistake: an identity match, a relevance judgment, a category, or a misunderstood stage of the case. The full component results and their evaluation conditions are retained in the research notes below.

CoronaPulse used a related design in terms of the high level workflow, but this time with an underlying message passing system architecture. Articles passed through deduplication, company-entity resolution, company relevance, finance-tuned sentiment analysis and other enrichment components. The pipeline progressively removed irrelevant material, according to the risk being assessed and each type of client’s workflow, rather than asking one model to make disparate judgments at once. Even its emerging topics were selected by a human expert, for their business impact, before reaching business users. [2]

The common idea was simple: make each question explicit enough that its answer can be evaluated. Each smaller problem can then be assessed on its own, and each stage contributes to the pipeline while preserving enough evidence to inspect the result.

Design for the person who has to use the alert

Adverse-media screening makes the trade-offs unusually visible. An analyst has limited time, and a queue full of weak matches is not simply inefficient. It injects irrelevant information into the decision-making process, and makes the useful matches even more of a needle in a haystack than they already were.

The screening paper describes its aim as high-precision adverse-news filtering. [1] That focus does not make missed cases disappear: a false negative is inherently harder to detect, and the right level of sensitivity, the share of genuine cases the system catches, is harder to achieve. A filter can improve the quality of what reaches the analyst while discarding genuine signals along the way. To understand that trade-off, the evaluation has to include samples of what was filtered out, with a record of the stage at which the system discarded each item, not just the alerts that survived the pipeline. That record is part of basic explainability, and it is what makes long-term maintenance and performance monitoring possible.

The final output also needs to be more useful than a label. The paper describes a risk profile organized by categories and proceedings stages, with article counts behind them. [1] The broader design requirement is to let the analyst move from that profile back to the variables and the material supporting them. An alert should come with the evidence needed to assess it, rather than ask the analyst to simply trust the score or reconstruct the entire search by post hoc reasoning of their own.

This is why I think about reliability, including whether very similar cases get the same outcome, as a property of the workflow. A model may recognize adverse language correctly and still produce an unusable alert. The system becomes useful when its decisions, evidence, and review process fit together.

Better models make this architecture more valuable, not less

Language models have made it much easier to prototype the reading step, because of in-context learning and few-shot inference. Tasks that once required a separately trained relevance or classification model can now often be attempted with a well-built prompt and a few minimal pairs, examples that differ in one telling detail. That is a substantial change in what a small team can build, without collecting and managing a large training set for each part of a finance or compliance model.

It also makes it easier to package the chain as a general-purpose agentic workflow. A demonstration can look convincing when one model reads an article, matches an entity relevant for a specific type of event, assigns a risk category for that found relation, and explains its reasoning in a coherent response. The hard question is whether each of those decisions stays above an acceptable level of specificity and sensitivity when the input is incomplete, ambiguous, or simply outside what the model saw in training.

Decision models offer a different way to use today’s capabilities. Rather than asking for prose which may or may not contain the right answer, a system can pose a bounded question and receive a typed result: a choice among a defined set, a score on a scale tied to a business measure, or a yes-or-no judgment with an associated probability. Eventum is TypeSafe’s official technology partner, and we use TypeSafe’s Jev as our default decision model across a variety of scenarios. [4] [5]

The economics make it practical to ask many more narrow questions and to provide targeted responses. The Jev input price cited here is $42 per billion tokens: roughly four thousandths of a cent for 1,000 input tokens, before other calls and system costs. Eventum’s published recruiting case study describes nightly enrichment that asks 44 questions of each job post, through a process handling more than 154,000 posts. [4] [6]

That is the opportunity I find most compelling: not ever more nuanced answers, but making the results of such reasoning systems affordable at high volume. The same economics make automated interactions with other systems easier to build.

Probabilities help only when their meaning can be reasonably assessed. TypeSafe explicitly distinguishes calibration across groups of predictions from a guarantee that an individual answer is correct. [5] In a deployed workflow, the practical question is whether the scores separate cases that can be handled automatically from cases that need human intervention, at an error rate appropriate to the action. A threshold for routing a document need not be the threshold for linking a customer to a fraud report, because the cost of getting those two wrong is worlds apart. [7]

A typed answer makes a decision easier to handle consistently across many different domains of knowledge. It does not make the decision infallible, but it makes the model’s uncertainty explicit and gives a person the means to judge how far to trust it. Evaluation, thresholds, and an explicit unresolved state still do essential work, adapting the system to business processes whose outcomes are defined differently.

Let’s revisit John Smith

Suppose the customer record describes John Smith, born in 1971, a director of a logistics company in Leeds. The article describes a John Smith, aged 54, a former haulage director sentenced in Leeds.

There are three possible routes: reject the candidate match, accept the identity match and continue to the risk questions, or leave identity unresolved for an analyst. TypeSafe’s entity-alignment example uses the same three-way pattern for product records, where a mistaken merge can contaminate everything subsequently attached to the merged entity. [8]

The evidence here is suggestive, but it does not necessarily indicate a specific conclusion. The names match, the industries are related, the demographic information may be consistent (provided the article’s date and metadata are reliable), but being sentenced in Leeds is not evidence of living there, and the article provides no shared address or associates. Some comparisons belong in the system logic once dates and identifiers have been rationalized and normalized, while others require adjustments according to context. The important thing is not to collapse all of them into an unexplained match score.

For this example, suppose identity remains unresolved. The analyst should see the agreements, the gaps, and the source sentence behind each. Only after identity is established should the system proceed to the next questions: is this person the accused, is the event within the bank’s screening scope, how recent is it, and what stage has the proceeding reached?

Errors in both specificity and sensitivity have a cost. A false match can wrongly associate a customer with a fraud case, scuttling a deal that would have been perfectly legitimate. A missed match can hide a relevant risk, which a client may find out only after much more costly consequences are revealed. Thresholds should be evaluated against labeled cases with those consequences in mind, recording both how often the system acts without review, and how often those actions lead us astray.

Name collisions are not an abstract problem in this business. The research indices cited below split and mix publications even under my name, including papers attributed to other researchers of the same name. [9] A matching name is the beginning of an identity investigation, not its conclusion. A name collision is the most straightforward of many such problems, which computational semantics treats as ambiguity and allomorphism. They are pervasive across data domains, and left unresolved they can lead to serious errors.

The same questions follow the data into clinical AI

After Moody’s, I worked on clinical and biomedical language models at Verantos and Avodah. The documents changed, but the need to separate claims did not.

Does a clinical encounter’s note actually support the relevance of a specific procedure? Does a diagnostic criterion in a guideline apply to this patient? How does a SOAP note, the visit record of what the patient reported, what was found, the assessment and the plan, show the clinician’s decision at that visit? These are different questions, and a system needs to distinguish them before handing later clinicians the answers they will rely on.

David’s post on machine-checked mathematics makes the companion point: a solver can establish what follows from an encoding without establishing that the encoding faithfully represents its source. [10] The same distinction matters in a screening pipeline, perhaps even more so, because the consequences sometimes surface decades later. Correct downstream logic cannot repair a mistaken identity or an unsupported fact supplied upstream. Mistaking one procedure, medication, condition or diagnostic criterion for another can lead to catastrophic health consequences and steep regulatory penalties.

Eventum’s production document-processing guide expresses the division as “Models interpret; the system decides.” [11] Models contribute judgments. Explicit logic, policies, regulatory considerations, and people determine how those judgments may be used, including when the system must withhold an answer, or make clear the lack of confidence in a specific answer.

When the system can identify uncertainty and route it deliberately, it becomes possible to automate the cases the evidence supports, while surfacing the risky ones to the people who are ultimately accountable for them.

What I would ask to see before expanding automation

Before coming to a decision as to which model to use, I would ask for the decision record and the evaluation metrics and methodologies behind it.

For each critical decision, how many cases were handled automatically, and how many went to a person as the final arbiter? How often was an automatically accepted result, one marked as not needing review, materially wrong in an audited sample? How do those rates change as cases become less certain? Where do reviewers change a material claim, and what do those changes reveal about the earlier decisions? [11]

Then I would pose two less glamorous questions. Can we reconstruct an outcome from its inputs, model and policy versions, and thresholds? And can the person reviewing an alert get directly to the evidence and the chain of decisions that support it? A lower review rate is not an improvement if the erroneous decisions are moving into the accepted stream, or if higher specificity comes at the cost of a collapse in sensitivity.

What makes this space exciting is that these are not simple questions with cursory answers. Many problems are still open for people who can build on what the latest language models make possible. We can bring more capable semantic understanding to many more decisions while preserving the discipline that makes those decisions usable. The alert still has to survive an analyst. A better model should help it do that.

Eventum helps teams turn document understanding into dependable workflows: defining the decisions, evaluating their boundaries, and building the evidence and review paths around them. Get in touch to discuss a screening or document-processing system your team needs to put into production.

References and research notes

1. Khandpur RP, Nanda AA, Davis M, Li C, Nurmanbetov D, Gaur S, Talukder A. Adverse Media Mining for KYC and ESG Compliance. KDD 2020 Workshop on Machine Learning in Finance; arXiv:2110.11542, version 1, October 2021. Extended abstract. Component figures, training-set sizes and splits are from Section 2.
   Research note. Component detail: risk relevance used 2,500 articles and reported F1 0.81 on an 80/20 split; entity relevance used 1,200 hand-tagged articles and reported F1 0.80 on an 80/20 split. The 65-category risk classifier used weak supervision over more than 9 million news documents and reported top-3 accuracy 0.86 on a 70/30 split. The five-stage proceedings classifier reported F1 0.93 on a 70/30 split and also used weak supervision. No performance figure was reported for the dictionary-based adverse-sentiment score. The small labeled sets, different metrics, and author-defined splits limit comparison; none is an end-to-end workflow score.
2. Li C, Bressler D, Nurmanbetov D, Gaur S, Gurunath N, Khandpur R, Nanda A, Davis M, Freedman J, Talukder A. CoronaPulse: Real-Time Sentiment Analysis and Emergent Multi-Sector Financial Risk Detection From CoVID-19 Events. KDD 2020 Workshop on Machine Learning in Finance, paper 28. Extended abstract; the workshop schedule at sites.google.com/view/kdd-mlf-2020/schedule links the paper.
3. Moody’s. Compliance Catalyst: Compliance Risk Platform. Product page, moodys.com/web/en/us/kyc/products/compliance-catalyst.html, read 16 October 2026.
4. TypeSafe. Introducing System One Models and Jev. typesafe.ai/blog/introducing-system-one-models-and-jev, 15 September 2026 (the release date); and TypeSafe. Models. docs.typesafe.ai/models, read 16 October 2026 (the price).
   Research note. The cited release date is 15 September 2026 and the input price is $42 per billion tokens. The 1,000-token arithmetic describes input charges for that amount of text, not the complete cost of a multi-call application.
5. TypeSafe. System One. docs.typesafe.ai/concepts/system-one, read 16 October 2026.
6. Eventum. How Eventum cut LLM screening costs 104x with TypeSafe. Case study, eventum.ai/case-studies/how-eventum-cut-llm-screening-costs-104x-with-typesafe, 2026. The 44 questions and the count of job posts are from the case study’s text.
7. TypeSafe. Confidence. docs.typesafe.ai/confidence, section “Thresholds scale with risk”, read 16 October 2026.
8. TypeSafe. Knowledge graph entity alignment. Cookbook, docs.typesafe.ai/cookbooks/entity_alignment, read 16 October 2026.
9. ACL Anthology author directory, aclanthology.org/people/chen-li/ and its unverified page, read 16 October 2026; OpenAlex and Semantic Scholar author records checked 6 and 16 October 2026 (OpenAlex files the CoNLL 2009 and adverse-media papers in two clusters carrying other researchers’ identifiers; Semantic Scholar files the adverse-media paper with a paper on Weibo authorship that is not mine).
   Research note. Author-index checks dated 16 October 2026 identified six disambiguated ACL Anthology pages named Chen Li and an unverified page containing 48 papers from multiple authors, including two of mine. Related checks identified misattribution in OpenAlex and Semantic Scholar. The ACL author directory contained approximately 135,000 listed authors. These are dated observations of changing indexes, not permanent counts.
10. Bressler D. AI Is Discovering New Mathematics. The Hard Part Is Knowing Exactly What It Proved. Eventum, eventum.ai/blog/ai-is-discovering-new-mathematics-the-hard-part-is-knowing-exactly-what-it-proved, 13 October 2026.
11. Eventum. Guardrails for Production Document Processing: A Technical Guide. eventum.ai/blog/guardrails-for-production-document-processing-a-technical-guide, 2026. Controls 8, 13 and 14.