An AI-written newsletter that isn't allowed to make anything up
The model picks facts by ID, code checks every number by its unit, and a second model edits.
The problem
I built a weekly email letter for a small local business. A model writes it every Thursday, the owners approve it with one click, and it goes out Friday morning. The interesting part is everything around the model, because the model is the part you can't trust.
The pattern
The model never supplies a fact
A script gathers the week's facts first: the forecast and any weather alerts, tides, observed rainfall, the local events calendar and recent posts from the business's own site. Free government data covers most of it. The model then writes prose and picks events and posts by ID from a fixed list. Every time, temperature, height, title and link in the email is printed by the template from the facts. It can't invent an event, because it can't type one.
A validator that knows units
After the model writes, checks in code run. Numbers are checked by kind: a number followed by inches has to match a rainfall figure in the facts, feet has to match a water level, and degrees (including “the low 80s”) have to match a temperature. A number that appears nowhere in the facts stops the letter.
The other checks are each a mistake a model actually made: no personal story unless the owners wrote it, no six-word phrase copied from the brief's examples or the last three letters, the subject and first sentence lead with any storm, surge or flood warning in force, the same banned-phrase list the prompt uses, and under 95 KB, because Gmail clips messages at 102 KB.
The editor is a different model
A second model reads the draft as an editor and flags invented stories, unsupported claims and contradictions. The writer gets up to three rewrites. If the editor only has tone notes, the letter gets one rewrite and stops, because editors never run out of opinions.
People write the human part
On Wednesday the owners get an email with three questions about their week. Whatever they answer becomes the heart of the letter, used once and as written. If they don't answer, the letter says nothing personal. The model may never make up a “we went to the farmers market this weekend.”
Approval is tied to the exact text
The preview email has an Approve button, and it approves a hash of that exact text. If anything changes after approval, say a new weather alert on Friday morning makes the job rewrite the letter, the approval stops counting and a new preview goes out.
A lab for the prompt
Prompt changes are tested like code. Nine frozen weeks of facts run through the real pipeline after every change, and a scorer counts what matters. A run costs about $0.20 a letter. Each check in the suite is a mistake a model made, or could make.
- Make the model choose from facts, never supply them.
- Check numbers by unit, deterministically.
- Use a second model as the editor, and cap the rewrites.
- Get the personal content from people, and forbid the model from inventing it.
- Freeze test weeks and score every prompt change.
- Make approval one click, and bind it to the exact text.How to use it
- Gather the facts in code first, from sources you trust, and give each one an ID.
- Let the model write the prose and pick facts by ID. The template prints every number, date and link.
- Check each number against the facts by its unit, and stop the send on anything that doesn't match.
- Freeze a handful of real weeks and score every prompt change against them before it ships.
Where it came from
For the first three Fridays nothing went out, and the scheduler never failed once. Every send ran on time and stopped because nobody had approved the letter: the approve switch lived in the CMS, and nobody opened the CMS. Twice, a flood watch issued after the draft stopped it too, which was correct.
The fix was a one-click Approve button in the preview email. Automation fails at the human step more often than at the AI step.
Excellent checks AI agents' work on the same principle: what the agent says is a claim until something else has looked. excellent.so