The Decisions You Make Before You Automate

Before asking AI to apply your contracting playbook, give the same borderline example to three people who use it. Ask them to classify it independently and explain their answer. If the answers differ, you have found something worth fixing before you automate the process.

Consider a hypothetical supplier described as “low risk”. Procurement points to the modest annual spend. The business owner points to a service that could be replaced next week. Security points to the supplier’s access to a production system. Everyone has read the policy. They are simply using different tests for the same label.

A more detailed AI prompt will not settle which test the organisation intended. That requires a decision about the definition. AI may still help with the work leading to that decision, particularly by helping a team find examples that expose the disagreement.

What the new research actually tested

Google Research’s 4 November 2026 article on Agile Deliberation describes a system for refining subjective image categories. Users establish what belongs within a category, then examine borderline images and refine their definitions through feedback.⁠[1]

The underlying study involved nine participants across 18 live sessions, each approximately 1.5 hours. It tested “healthy food” and “paid to play”, a category involving images promising unrealistic rewards for online entertainment. These were image classification tasks, not assessments of contracts or business policies.⁠[2]

The researchers reported gains in classifier alignment against automated baselines and lower participant-reported effort than manual deliberation. They also identify limitations, including the small participant group, limited concepts and lack of full-pipeline testing across other generative models.⁠[2]

My proposed application for Legal and Procurement is narrower: use a similar sequence to test the meaning of internal policy labels before asking AI to apply them. That is an operational analogy, not a legal capability demonstrated by this study.

Find the definition problem before the technology problem

Start with a specific decision your team wants to make more consistently. “Assess supplier risk” is too broad for a first exercise. “Decide whether this supplier must complete our security review before contracting” gives the team a concrete outcome and a clearer set of relevant facts.

This also helps separate different reasons for disagreement. One reviewer may lack information about system access. Another may interpret “access” differently. A third may agree the trigger applies but believe an exception is justified because the project is urgent. Those are three different issues, requiring three different responses.

Missing information calls for a question. Ambiguous wording calls for a clearer definition. An exception calls for the authorised decision-maker and a recorded rationale. If you put all three into a single “uncertain” bucket, you lose the explanation that would make the process better.

There is an older research foundation for taking definitions seriously. Kulesza and colleagues’ 2014 work examined how people’s concepts change while labelling data. Their controlled experiment found structured labelling improved consistency, although it took longer than the baseline approach.⁠[3] That trade-off is useful to keep in mind when planning a workshop: allow time to clarify the rule, rather than judging the exercise by how quickly people finish.

Choose one label and collect the awkward cases

For a first pilot, choose one label that changes the review path. Examples might include “standard terms”, “material departure” or “critical supplier”. Avoid starting with the entire playbook. You want a manageable question with an owner who can approve any resulting change.

Gather a small working set of examples. Include clear cases on both sides, then add the awkward ones: the clause that prompted three emails, the low-value supplier that needed unusual scrutiny, or the renewal everyone assumed was routine until someone read the variation. Use approved, appropriately protected material, with unnecessary identifying details removed.

Keep each example short enough to review, but preserve the facts that could change the outcome. If the question concerns supplier access, the spend figure alone is insufficient. If it concerns a liability departure, the cap cannot be assessed meaningfully without the relevant scope, exclusions and contractual context.

In an approved AI environment, you could ask for candidate examples that appear to sit near the current definition’s boundary. Treat the output as a proposed reading list. A reviewer should check the underlying document and reject examples built on missing text, inaccurate summaries or invented facts.

You can also ask for hypothetical variations of a verified example, clearly marked as invented. Change one fact at a time. That makes it easier to identify which fact actually changes the classification, without treating an AI-generated scenario as evidence of what the organisation has previously approved.

Include a case where the supplier’s name is removed, then compare it with an otherwise identical case involving a familiar supplier. If the answers differ, ask which policy condition explains the difference. Familiarity may be relevant to a documented assessment, but it should not become an unstated substitute for the test you are trying to define.

A worked example what counts as a standard renewal

Imagine your internal playbook allows Procurement to process “standard renewals” without a fresh Legal review. The phrase sounds clear until you put these hypothetical cases beside each other:

• The service, price and contract wording remain unchanged, and the existing agreement expressly permits the extension.
• The service remains unchanged, but the renewal order incorporates a new set of online terms.
• The price increases under an agreed formula, while the supplier also introduces a different support location.
• The contract wording remains unchanged, but the service will now support a business-critical function.

Ask each reviewer which cases fit the existing definition and which facts matter. Do this before sharing an AI-generated answer or opening the group discussion. The purpose is to see each person’s interpretation clearly, rather than having everyone react to the first confident explanation.

One reviewer may define “standard” by whether the document changes. Another may focus on whether the commercial arrangement changes. A third may consider the current use of the service. The disagreement tells you that a single label is carrying several separate conditions.

For this hypothetical organisation, the policy owner might decide that the simplified renewal path requires unchanged approved terms, pricing within the agreed mechanism, no relevant change in service or data handling, and a current risk assessment. That is an illustrative decision, not a recommended universal rule. Your organisation’s requirements may be different.

Notice the difference between recording that the third example needs review and recording why. The useful clarification is that an agreed price adjustment does not, by itself, decide whether a changed support arrangement needs review. That explanation can guide the next case, even when its details differ.

Write down the disagreement before drafting the replacement

Use a simple record for each disputed example: the relevant facts, the competing interpretations, the existing wording, the point requiring a decision and the person authorised to decide it. Keep the initial views visible until the decision is made.

AI could help organise those records into recurring questions. For example, are people disagreeing about the meaning of “change”, about which changes matter, or about who can accept the consequences? Ask it to preserve the separate positions and identify where the supplied material does not resolve the issue.

Do not ask it to produce a neat consensus statement too early. A polished paragraph can conceal the very disagreement the exercise was meant to expose. Where two policy owners have incompatible requirements, record that conflict and obtain the decision needed to resolve it.

NIST’s AI Risk Management Framework 1.0 supports documenting intended purposes, deployment context and relevant assumptions. It also calls for clear responsibilities and lines of communication for managing AI risks.⁠[4] For this pilot, put those ideas into practice by naming both the policy decision-maker and the person responsible for testing the resulting instruction.

Once the decision is made, draft a short definition with inclusion conditions, exclusions, the facts needed to apply it and an escalation route. Keep the supporting examples beside it. Someone reading the definition later should be able to understand why a superficially similar case has a different outcome.

Ask for a short explanation of what the revised wording changes. Which cases now fall within the definition, which move outside it, and which still require escalation? Check those consequences against the policy owner’s decision. A small drafting change, such as replacing “and” with “or”, can substantially expand a category even when the surrounding paragraph looks reassuringly similar.

Test the revised wording with people first

Put the revised definition in front of a colleague who did not attend the workshop. Give them a fresh set of examples, including some ordinary ones. Ask for the classification, the relevant condition and any information they would need before deciding.

If the colleague needs the workshop facilitator to explain what the words really mean, revise the wording. The goal is a definition that travels beyond the meeting. It should not depend on remembering a conversation that happened while a particular clause was on screen.

Next, test the intended AI workflow against the approved examples and their reasons. Keep a separate set of cases out of the drafting exercise so you can see whether the instruction works beyond the examples used to develop it. A good result on familiar material is only a starting point.

Distinguish an incorrect classification from an appropriate request for more information. In the renewal example, missing information about support locations should not quietly become an assumption that nothing has changed. Specify the expected response when a necessary fact is unavailable.

Pay particular attention to mistakes that send a matter down the wrong path. Classifying an exceptional renewal as standard may be more consequential than requesting an unnecessary review. Set the acceptable level of error for the particular use, rather than chasing a single overall score that hides those differences.

For a small pilot, keep a simple before-and-after record: cases where reviewers disagreed, cases blocked by missing facts, and cases needing a policy decision. Revisit the disputed cases after the revision and record whether the reason for disagreement has actually been addressed. Do not treat a lower disagreement count alone as success. Check that people can identify the relevant condition and explain the same result without relying on an informal understanding shared only by the original workshop participants.

Keep the first pilot advisory. Compare its suggestions with the existing review process, examine the disagreements and decide whether the definition or the implementation needs work. Avoid changing approval rights merely because the pilot produces convincing explanations.

Keep the definition connected to later decisions

When a genuine new case exposes a gap, add it to the review set. Record whether the team clarified the existing rule, changed the policy or approved a one-off exception. Those outcomes should not be interchangeable. An exception granted for a particular project should not silently rewrite the playbook for every future project.

Give the approved definition a version and an effective date. Retain the supporting examples and decision record with it. If an instruction changes, rerun the cases that previously worked as well as the case that prompted the change. This is a practical way to check whether solving one ambiguity has introduced another.

Be equally clear about what the workshop cannot settle. It should not treat a legal requirement, contractual obligation or formal delegation as an optional preference. Where the wording raises a legal interpretation issue, obtain the appropriate advice and record that basis separately from the organisation’s discretionary risk settings.

For next week, choose one recurring classification that causes avoidable back-and-forth. Bring together its owner and the people applying it. Review a small set of clear and borderline cases, identify the unresolved question, then test the revised definition with someone outside the discussion. You may finish with a simpler process, a better escalation trigger or a conclusion that the decision is not ready for automation.

Each is a useful outcome. The objective is to give the next reviewer, human or AI, a clearer instruction than “use your judgement and follow the policy”. Start by finding where reasonable readers disagree, then make the decision that the wording has been leaving to them.

If you are reviewing how your team uses policies and playbooks in its contracting work, book a demo with Contract Cloud to discuss your workflow and what you would need to test before wider use.

Sources

[1] Google Research: Agile Deliberation: How AI helps humans define the gray areas. 2026-11-04.

[2] Wang et al., Agile Deliberation: Concept Deliberation for Subjective Visual Classification, CVPR 2026, revised arXiv version 2. 2026-04-04 (v2; first submitted 2025-12-11).

[3] Kulesza et al., Structured Labeling to Facilitate Concept Evolution in Machine Learning, CHI 2014. 2014 (CHI conference 26 April to 1 May).

[4] NIST AI Risk Management Framework 1.0: AI RMF Core. 2023 (framework; current official excerpt reviewed 2026-11-07).