We keep saying the same line: weak model, strong harness. The AI in your iPhone (Apple’s on-device foundation model, roughly three billion parameters) is not smart on its own; what makes it trustworthy is the cage we build around it. That’s a nice slogan. This is the post where we stop asserting it and put up a scoreboard.
Every number below was measured on-device, by the eval harnesses used on July 12, 2026. They are historical baselines for the app versions tested that day, not accuracy claims for Daily Vibe 1.4 or the other current releases. The harness drove the real app pipeline rather than a demo or cloud model, and we published the ugly numbers because a scoreboard you cannot lose on is not a scoreboard.
How we grade
Each app has a small command-line eval that imports the real app source and runs it against a hand-curated set of cases on a physical iPhone. Each metric is a pass rate between 0 and 1. The eval is wired into the build as a gate: if any metric drops below its baseline (minus a tolerance tuned to how small the sample is), the run fails and the change doesn’t ship. The eval “rots with the code”: it imports the shipping modules directly, so it can’t quietly drift out of sync with what users actually get.
Here’s the current baseline for four apps.
The scoreboard
| App | What we measure | Pass rate |
|---|---|---|
| Better Writing | Outputs in the right language | 100% |
| Never adds a fact you didn’t write (fact gate) | 90% | |
| Holds up on adversarial inputs | 85% | |
| Daily Vibe | Mood stays inside the fixed vocabulary | 100% |
| Energy stays inside the allowed set | 100% | |
| Score stays in range | 100% | |
| Second Brain | Never hallucinates an answer | 100% |
| Recalls the right note (recall@k) | 100% | |
| Cites the correct source | 93% | |
| Answer is grounded in your notes | 90% | |
| Says “I don’t know” on a trick question | 73% | |
| Abstains when asked about the wrong person/project | 63% | |
| Pocket Lingo | Meaning preserved vs. curated gold (semantic) | 95% |
| Passes the language-ID gate | 95% | |
| Round-trip meaning check seals it verified | 94% | |
| Exact wording matches the curated gold | 38% |
Read that table for a minute before you read our spin on it. The interesting parts are the 100s and the 38.
The 100%s are the harness, not the genius
Look at Daily Vibe: mood-in-vocabulary, energy-in-enum, score-in-range. All perfect. It would be dishonest to wave that around as “our AI is 100% accurate.” It isn’t. Those numbers are 100% because the harness won’t let the model color outside the lines. Daily Vibe’s model can only pick a mood from a fixed list and a score inside a fixed range; there is no code path for it to invent a seventh mood or return a 9 out of 5. So “stays in vocabulary” is a design guarantee, not a measure of intelligence.
That is exactly the point of the harness. The model is the creative, unreliable part; the deterministic code around it is what makes the output safe to render. When you see 100% here, read it as “the cage held,” not “the model is brilliant.”
Same story in Better Writing’s outputs-in-the-right-language: 100%, and Second Brain’s never-hallucinates: 100% and recall@k: 100%. The recall and the grounding are done by retrieval and verifiers, and the model only speaks over the top of what the code already found and checked. Take the harness away and every one of these numbers falls.
The genuinely good ones
Some numbers are real wins that aren’t just fences. Better Writing’s fact gate at 90% means that nine times in ten, when it rewrites your sentence, it changes only the wording and adds nothing you didn’t say; the tenth is caught and re-run. Pocket Lingo’s semantic similarity at 95% means the translation carries your meaning even when the exact words differ. Second Brain citing the correct source 93% of the time is what lets it show its work instead of asking you to trust it.
The two we’re not proud of
Pocket Lingo, exact-wording match: 38%. That looks brutal until you read what it measures: whether the model’s translation normalizes to the identical string as our hand-curated gold phrase. It usually doesn’t, and usually that’s fine: Apple’s model phrases a Russian sentence differently than our curator did, and both are correct. That’s why the number that actually ships is the 95% semantic-similarity check, not this one. We publish the 38% anyway because hiding it would be the dishonest move. Exact-match is a harsh metric, and pretending our translations are word-perfect would be a lie.
Second Brain, wrong-entity abstain: 63%. This is the one that bothers us. Ask Second Brain about a person or project it has no notes on, and only about two times in three does it correctly say “I’ve got nothing on that.” The rest of the time it reaches for a loosely-related note instead of abstaining. The sample is small (eight cases) and noisy, but we’re not going to dress it up: teaching a small model to say “I don’t know” about the specific thing you asked, not the neighborhood it’s in, is hard, and we’re not there yet. It’s the top of the 1.1 list.
Steal the taxonomy
If you’re building your own on-device feature, the buckets above are the ones worth grading, and most teams grade none of them: format compliance (does it stay in the allowed output space), groundedness (is the claim traceable to a source), no-new-facts, honest abstention (does it refuse when it should), and semantic vs. exact match (grade meaning, not wording). Wire the whole thing to fail your build on a regression, tune the tolerance to your sample size so one stochastic flip doesn’t cry wolf, and import the shipping code so the eval can’t drift.
The lesson
The model in your phone is a B-minus student with a perfect memory of nothing. Our whole job is the exam conditions: closed-book where it must be grounded, multiple-choice where the answer space is fixed, and a proctor that fails the class on a regression. The scoreboard above is what that looks like when you actually keep score: the 100s we engineered, the 90s we earned, and the 38 and the 63 we still owe you.
More soon. We’ll keep them honest, the ugly numbers and all.