Lore Week 4: keeping data by category, checking AI written content at scale, and describing a product as data

Up to the Sunday 21:00 cutoff we merged 52 pull requests this week. Lexicon, our exam preparation app, took 20 of them, AutoReal 11, lorelabs.co 10, guno 6 and Playables 2. The other 3 went to internal tools. Playables became a live product page on lorelabs.co this week.

We want to teach three new ideas this week:

1. Keeping data by category, not by one number. Why one retention rule for every upload did not fit, and what replaced it.
2. Checking AI written content at scale. How we checked 508 generated practice cards, and what a small sample can and cannot prove.
3. Describing a product as data. How one exam engine reads a written description of each exam, so a new exam starts as a file and not as new screens.

If you are new here: branches and approval gates were Lore Week 1, CI and check minutes were Lore Week 2, and changing model providers and usage caps were Lore Week 3. We do not repeat them here.

What it is. A retention rule says how long a product keeps something a customer gave it. Two ideas from data protection help here: keep only what the purpose needs, and keep it only as long as the purpose needs it. Türkiye's personal data law, KVKK, lists both among its general principles in Article 4: personal data should be "related, limited and proportionate" to its purpose, and kept "for the period required" by the law or by the purpose. We use these as design ideas here; this paper is not legal advice and makes no claim about compliance.

Why it matters. The easy way to follow these principles is one number for everything: delete every upload after 30 days. It is simple to explain and simple to build. But a single number treats very different things as if they were the same.

How it works, in simple words. A purpose decides a period. A photo of a person is personal data, and a marketing post needs it for a short time. A logo or a product photo usually serves the brand for as long as the brand uses our product. That is not always a clean line: a logo can contain a name or a face, and a product photo can show a person, so brand assets can also contain personal data. What an image contains and what it is used for both matter, not only its file type. So the rule is written per category: each category names its purpose, and each purpose sets its own period.

What happened to us. AutoReal makes social media posts for small businesses, and this week it started using the owner's own photos. Its first plan kept every upload for 30 days. Reading the plan against real use showed the problem: every month, the rule would have deleted the brand's logo and its product photos, and the owner would have had to upload them again. We split the rule. Under our design, photos of people are set to go after 30 days, and logos and other brand photos are kept while the brand exists. This describes our internal rule, not a legal assessment or a record of how it has been enforced. We wrote the change into our company standard, not only into one product, so the next product starts from the split rule.

How we compare. A rule per category is only as good as the sorting behind it, so the sorting step needs its own measurement, and we keep that work on our list. One more note in the same spirit: when guno started storing a user's morning setup answers this week, its public privacy page was updated in the same week to say so.

What it is. Lexicon teaches words with practice cards: a meaning, an example sentence and a common mistake. An AI model writes the first version of each card. This week we produced 508 cards for a second exam that is not yet switched on for users.

Why it matters. A model can write a fluent sentence that teaches something wrong. One wrong card is small, but 508 cards are too many to read with full attention every time a prompt changes.

How it works, in simple words. We used three layers. First, a model writes the card. Second, a separate model pass reads each staged card against a list of accuracy rules and either corrects it or flags it for a person to settle. Third, an internal reviewer who did not write the cards reads a sample. A card cannot move into the app until the checker has passed it, and a flagged card is refused.

What happened to us. The first try showed why the second layer was needed. Our working notes on a starter set of 65 cards report that many common mistake notes confused percent with percentage points, and four cards put a percentage on "the number of", which is wrong English. Tightening the prompt alone still left 10 of 27 redone cards with a clear error. With the checker in place, across all 508 cards, 37 passed as written and 471 were corrected. Six cards were then fixed by hand after a fresh review. We counted these figures again from the card files themselves. Separately, an internal reviewer read a sample of 50 cards, 5 from each of the ten sets, and reported 1 borderline error, which was fixed. That was a separately reviewed internal sample, not an independent evaluation, and our records do not show exactly how its cards were drawn. Our agreed bar was an error rate under 3 percent.

What others say. Research on using a model as a judge (Zheng and colleagues, 2023) found that strong models agree with expert preferences about as often as experts agree with each other, and also that model judges have known biases, such as favouring longer answers and their own style. That is why our checker is not the last step: an internal reviewer who did not write the cards reads a sample.

How we compare. Here is the part a small sample cannot prove. 1 error in 50 cards is 2 percent, but with only 50 cards the uncertainty is wide. If the 50 cards had been drawn at random, an approximate 95 percent Wilson interval (described in the NIST statistics handbook) would run from about 0.4 percent to about 10.5 percent. That interval rests on sampling assumptions and is not a guaranteed bound on the true error rate. If the sample was not random, it cannot establish the error rate of all 508 cards. So the sample does not show that we are under our 3 percent bar. A proposed next step is a larger random sample, sized from this kind of calculation.