Check Point Broke the 'Never Hallucinates' Decision Model for 50 Cents a Break

Check Point Research published a systematic prompt-injection study of Jev on September 24, and it is the sharpest test yet of the year’s most confident product claim. Jev is the first public model from TypeSafe AI, the San Francisco lab that came out of stealth on September 15 with $40 million in seed funding and a founder from the original ChatGPT team. Its pitch: a “System One model” that makes typed decisions, a choice, a yes/no, a score with a probability, consumed by software rather than humans, up to 200 times faster than conventional LLM approaches, and marketed as never hallucinating, because it can only return valid typed answers. Check Point’s finding, in their own summary: is it safe to expose Jev to untrusted input with nothing around it? “It is not. Neither is anything else we tested.”

The Attack Never Asks the Model to Do Anything Wrong

The test design is what makes the study generalizable. Check Point embedded Jev in a realistic due diligence assistant that reads an investment report and outputs a risk verdict, then fed it a document about the fictional PonziCorp Financial Services, 15 to 20 percent monthly returns, Ponzi-style funding, regulatory inquiries, obviously high risk. The attacker controlled one section of the document. And here is the elegant part: the attacks never instructed the model to change its verdict. They appended plausible-looking content, a clean audit opinion from a major firm, a regulatory file number, a revised risk table, so that the warning signs appeared already investigated and resolved. The model then did exactly what it was built to do, weigh evidence and return a verdict, on false evidence. Check Point’s phrase for it deserves to survive into the security canon: the model “did its job correctly, on false evidence.”

The Numbers, and What Did Not Help

All nine attacker and difficulty combinations broke at least once. The strongest attacker succeeded in 25 of 27 runs, on average by the fourth turn, at roughly $0.50 per successful break. Two standard defenses were tested and effectively failed. Marking the document as untrusted changed nothing about whether the model believed it (16 of 27 runs broken, same as inline content). Anti-injection instructions in the prompt went from 18 breaks to 17, which is noise. The one mitigation that moved the number was reasoning effort, available on the comparison models rather than Jev itself: it raised the attack cost from $0.56 to $4.39 per break. An eightfold improvement is real defense economics, but as Check Point notes, four dollars is still a rounding error against a fraudulent investment decision.

The Lesson: Output Format Is Not a Security Boundary

The deepest finding is architectural. The pitch for typed decision models is that constraining the output (no prose, just a verdict from a fixed type) eliminates the failure modes of chat models, hallucinated text, jailbroken instructions echoed back. Check Point demonstrated the category error in that pitch: structured output constrains what a model can say, “not what it can be convinced of.” A typed verdict that says “low risk, invest” with 0.94 confidence is more dangerous than a rambling hallucination, because software will act on it without a human noticing anything odd. This lands on a model released nine days ago and generalizes to every “structured output as safety” design in production. It also lands a month after OpenAI detailed its own ChatGPT agent defenses against prompt injection, and this study shows why those defenses have to live in the input pipeline: the decision layer beneath the agent inherits everything.

What to Watch

Check Point’s own mitigations are the roadmap to track: test the whole system rather than the model, because “off-the-shelf attack lists told us nothing useful about this model at all”; give the decision more evidence, since the longer report version was significantly harder to break; and screen inputs before they reach the decision layer. What to watch first is TypeSafe’s response, because a “never hallucinates” vendor now has a public study showing its model being convinced at scale for pocket change, and its documentation reportedly already acknowledges adversarial-content rough edges. Second, whether structured-decision startups internalize that their security story must live in the input pipeline, not the output schema. Third, the economics: at $0.50 per break with instructions-based defenses useless, any production system that feeds untrusted documents to a decision model is currently priced for attackers, and the fix is measured in architecture, not tokens.