Threat patterns · Worked example

A good AI policy, and the four ways it had already been outrun

Every rule below is one a careful firm would write. Each was correct when written. Each has since been defeated by a documented change, and in three cases the firm would have had no way of noticing.

The firm in this example is invented. It is not a client and not a case study, and nothing here is a description of anyone’s actual breach. What is not invented is every fact it runs into: each one is dated, sourced, and carried by one of the threat patterns. The scenario exists only to put those facts in the order a real firm meets them.

Picture a twelve-person accountancy practice. In January 2025 someone sensible wrote a one-page AI policy. It had four rules. It was better than most.


Rule 1: “Use the paid plans. We pay, so we are the customer, not the product.”

Outrun by training, generation 2 — paying does not make you the customer.

This is the control most people actually rely on, and it is not written in any policy: the belief that money changes the relationship. It does not. The paid individual tier of every major consumer AI product we rate carries the same training default as the free one. ChatGPT Plus and Pro, Claude Pro and Max, GitHub Copilot Pro, Pro+ and Max, Perplexity Pro — all consumer terms, all training on by default, all with an opt-out you have to go and find.

This rule was not defeated on a single date, which is what makes it the hardest of the four to catch. Two of those products moved into that list after a policy written in January 2025 — consumer Claude on 28 August 2025, and GitHub Copilot Pro and above with effect from 24 April 2026. On others, training by default on the consumer tiers is the older arrangement and long predates any 2025 policy; that is generation 1, the bargain nobody reads. So this rule was already partly wrong on the day it was written, and then got wronger twice — with no single event our firm could have been watching for.

The line that matters is not free versus paid. It is consumer versus commercial. On the business and enterprise tiers, training exclusion is a contractual term rather than a toggle — which means it cannot be changed by a product decision, and it survives a terms update.

The money argument usually inverts once someone checks: the step up to a commercial tier is frequently cheaper than the premium individual plan people buy instead. We have deliberately not printed the figures on this page. Prices and tier names are the fastest-decaying thing we publish — when we last re-verified thirteen profiles, the privacy postures had largely held while eleven of the thirteen plan tables were wrong — so the current numbers belong on the vendor profile, where they are dated, and not in an example that will be read a year from now.

Rule 2: “Check the training setting when we adopt a tool.”

Outrun by training, generation 3 — the default flips under you.

A setting checked at adoption is a fact about the day you checked it. Anthropic announced a change to consumer Claude terms on 28 August 2025, and contemporaneous reporting described the training toggle in the acceptance dialog as pre-set to on, beneath a prominent Accept button. GitHub changed Copilot Free, Pro, Pro+ and Max the same way with effect from 24 April 2026.

Our firm did nothing wrong and changed nothing. Its policy was accurate in March and inaccurate in May, and the event that changed it was a dialog box that somebody clicked through in a hurry. “We would have noticed” is not a control. These changes arrive as terms-acceptance prompts, which are designed to be dismissed.

The repair is to write the policy against the question rather than the vendor’s current answer. “Training must be off wherever client data is entered” survives a vendor changing its mind. “Vendor X does not train on our data” does not.

Rule 3: “Our notetaker is visible in the participant list, so everyone knows.”

Outrun twice — by consent, generations 2 and 3.

Disclosure by presence was never consent. It was a signal, and it worked only while the tool stayed visible. Generation 2 broke it for the people who most needed it: an assistant that joins from a calendar turns up in meetings organised by other people, and those people are frequently not account holders — so there is no setting for them to change and no opt-out available to them. Generation 3 removed the signal altogether, with recorders that capture audio on the user’s own device and never appear in the participant list at all. The absence of a visible bot is marketed as a feature.

Note what this does to our firm’s position. The rule was written about their notetaker. It says nothing about the tool on the other side of the call, and by generation 3 there is nothing they could look at to find out.

Rule 4: “Training is a privacy preference, so it lives in our privacy policy.”

Outrun by training, generation 4 — on 13 August 2026 it stopped being only a preference.

In In re Otter.AI Privacy Litigation the court granted a motion to dismiss in part and denied it in part; federal wiretap, California Invasion of Privacy Act and Illinois biometric claims survived. The reasoning is the part that reaches past Otter. Otter argued it was merely a tool acting for the meeting host. The court rejected that and held it could be treated as a third-party eavesdropper — because it retained the conversations and used them for its own commercial purposes, including training its models.

So “does this tool train on what it records” is no longer only a question about privacy preferences. A US court has treated a vendor’s own use of the recording as the thing separating a participant from an eavesdropper. Nothing has been proven on the merits — this is the pleading stage, and an allegation is not a finding. But the theory survived, and it reaches any meeting tool that trains on what it captures.

The rule the firm did not write, and would have got wrong

“We are in New Zealand, so US recording law is not our problem.” Half right, and wrong in both directions.

New Zealand is a one-party consent jurisdiction: under section 216B(2) of the Crimes Act 1961, a person who is party to a private communication may lawfully record it without the others agreeing. We had this wrong ourselves, in five places, until 3 September 2026 — see Corrections. So the criminal law is not what requires you to disclose.

Three other things do. Recording a meeting collects personal information, so information privacy principle 3 of the Privacy Act 2020 requires you to tell people you are collecting it, why, and who will receive it — a notification duty rather than a consent one, and one a silent recorder makes very easy to breach. Covertly recording colleagues can breach the employment duty of good faith. And if any participant is sitting in a genuine all-party consent state — California, Washington, Florida, Illinois, Maryland, Massachusetts, Montana, Nevada, New Hampshire or Pennsylvania — their law may reach the call wherever you are.


What actually survived

Four rules, four defeats, and not one of them caused by carelessness. The pattern in the failures is worth more than the individual fixes: every control that broke was a control that depended on a vendor’s setting, a vendor’s tier, or a tool being visible. The ones that held are organisational.

  • Decide which meetings may be recorded, and apply that to the calendar rather than to the tool. This survives every generation of the consent pattern, including the ones with nothing to see.
  • Say it out loud at the start, and put a line in the invite. The only disclosure that still works is the one a person performs. No technical signal does it for them any more.
  • Use commercial tiers for anything touching client data, where the training exclusion is contractual rather than a toggle — and price the step up before assuming it is expensive.
  • Re-check settings on a schedule, not at adoption, and treat a terms-update dialog as a trigger to re-check rather than an interruption.
  • Write the policy against the question, never the vendor’s current answer. A policy naming a vendor’s posture is a snapshot, and it decays silently.

The uncomfortable part is the timing. Rule 2 was outrun in August 2025 and again in April 2026. Rule 4 in August 2026. Rule 1 was never wholly true and became less true twice over the same period. A firm reviewing its AI policy each January would have carried a wrong policy for most of two years while believing it had a current one — and at no point would anything have arrived to tell them. That is the whole reason this section of the site is organised by generation rather than by incident: you cannot check a control against a list of things that have gone wrong, only against the thing that defeats it.

Every claim above is carried by a dated pattern on the threat patterns page, where you can see its sources and whether a person has read them in full. Where a pattern rests on a court order we have read only through reports, it says so rather than letting the citation imply otherwise. This is general information, not legal advice for your situation.