Skip to content
The AI Agent Factory

3.2 The first 10 percent: intent, scope, authority, review contract

Approved by
Panaversity ·
69 min read

In everyday life. Before a teenager takes the family car, you say where they may go and when to be back. You also say that they must text if plans change, and that they may never drive after midnight. You say all of it in the driveway, because once the car turns the corner, it is too late to add rules.

The first 10 percent is everything the worker must know before it starts, because you will not be there while it works. It has four parts.

  1. Intent. The outcome you want, and why it matters. Compare two ways Dave could start the task:

    • The weak version: "Reconcile the statement." Which statement? When is the job done? Why does it matter? The worker has to guess.
    • The strong version: "Reconcile Midwest Packaging's September statement so the close can be signed on Monday." It tells the worker what "done" means (Midwest's statement for September, with every difference explained) and why it matters (the close must be signed on Monday).

    (To reconcile a statement is to compare it with Brightline's register and explain every difference. The close is the month-end work that finishes and checks September's records.)

    The why also helps with small decisions you did not foresee. A worker that knows the close is on Monday treats a missing invoice as urgent, because there is little time to find it.

  2. Scope. The edges of the job: what is in, what is out, and what the worker works from. For Dave's reconciliation:

    • In: one vendor, Midwest Packaging, and one month, September.
    • Out: every other vendor, and every other month.
    • Works from: which version of the register, and which policy. (In Chapter 2, the assistant worked from an old copy of the AP policy, two versions out of date.)

    Scope also stops a willing worker from doing too much. Suppose Dave asks for one statement and gives no scope. A helpful worker may decide to tidy up three other vendors' statements too. Nobody asked for that work, and nobody planned to check it.

  3. Authority. What the worker may do on its own in this task. This is about permission, not ability. The worker may be able to edit the register, but is it allowed to? Authority comes in two layers:

    • The Role Contract sets the limit. The worker's Role Contract, the one page that describes its job and its limits, sets the most the worker may ever do, in any task.
    • The brief can only narrow it. The written instructions for one task, the brief, can narrow that limit, never widen it.

    With the teenager, the family's rule lets them drive anywhere in town, but never on the highway (the Role Contract). Tonight you say, "Only to school and back" (the brief). That narrows the rule for one trip. A talk before one trip can never send them onto the highway. For that, the family would have to change its rule first.

    Dave's worker works the same way. Dave owns the worker's contract. The contract lets the worker update register rows, because building the register is its normal job. But a reconciliation must change nothing. So the brief has to say so: in this task, the worker may only recommend changes. The worker proposes each change, and Dave decides.

  4. Review contract. How you will know the work is right, agreed before it starts. For Dave's reconciliation: every difference comes back with the document it was taken from, and the worker stops at any difference it cannot explain. Its own section, below, explains it.

With the teenager, where they may go is the scope, and when to be back is part of the intent. The text if plans change and the midnight rule belong to the review contract: when to stop and ask, and what must never happen.

When the worker runs day and night

Lesson 3.1 showed when the first 10 percent happens for a worker that runs day and night: once, in advance. This section shows what you write in advance, and how each run uses it.

Every run still gets all four parts: the intent, the scope, the authority and the review contract. They are written in advance, so that each run picks them up without you.

Go back to the teenager. If they drive to school every day, you do not repeat the rules each morning. You set house rules once: where they may go, when to be home, text if plans change, never after midnight. Every ordinary trip follows them. Only an unusual trip, such as a drive to another city, needs a fresh conversation, and that conversation can tighten the house rules but not loosen them.

A standing worker's setup works the same way. (A standing worker is what Chapter 2 calls a Standing AI Worker: it holds an ongoing role, and its work comes again and again, such as the AP Worker's Monday register and its vendor emails.) Its setup has three layers, and each one covers a different amount of the work:

  • The Role Contract is about the whole job. It says what the worker is for, and the most it may ever do, in every task. It is written once, when the role is set up, and changes only when the job changes. For the AP Worker: "May build and update the register. May never pay anyone."
  • The standing instructions are about one kind of work that repeats, such as building the register every Monday, or answering vendor emails. Each one gives the four parts for that kind of work, inside the Role Contract's limits. It is written once for each kind of work, then improved. For vendor emails: "Draft a reply, but never send it. Put it in the approval queue."
  • The brief is about one task that no standing instruction covers yet, such as this month's reconciliation. It is written when that task comes up. It gives the four parts for that task, and it may narrow the limits for that task, never widen them. For example: "Reconcile Midwest's September statement. Recommend only."

The title says every run gets a setup, even at 11 p.m. On the left are three boxes, one inside the other. Box 1 is the Role Contract, defined and kept up to date. It sets the job and the most the worker may do: it may build and update the register, and may not make payments. Box 2 holds the standing instructions for recurring work, each with intent, scope, authority and review. The weekly register card prepares this week's invoices, from the new invoices since the last run. It may add register rows, and it flags any invoice it cannot match. The vendor emails card answers the vendor's question, about that vendor's open invoices. It may draft a reply but not send it, and the draft goes for approval. Box 3, drawn dashed, is the task brief, labeled "For an unusual assignment". It says "Reconcile Midwest's September statement", with the same four parts and narrower authority: recommend only, with no changes to the records. A brief may narrow authority, never widen it. On the right, arrows lead to three runs that use this setup. On Monday at 6 a.m., it builds the register with the weekly instructions. On Tuesday at 11 p.m., it handles a vendor email with instructions written in advance. This month, it reconciles the statement with a brief for this assignment. A banner says the first 10 percent moves earlier and does not disappear. Every run inherits the Role Contract's limits, and permissions and approval controls enforce them.

Figure 3.3. A standing worker's setup is written in advance, in three layers, so every run has all four parts even when nobody is awake.

Here is how this worked in the AP Worker's week, in Lesson 3.1. On Tuesday at 11 p.m., a vendor sent an email. Nobody was awake to write a brief. But weeks earlier, Dave had written standing instructions for vendor emails. They gave the worker all four parts:

  • Intent: answer the vendor's question.
  • Scope: this vendor, and its open invoices.
  • Authority: draft a reply, but never send it.
  • Review contract: put the draft in the approval queue.

So at 11 p.m. the worker knew what to do, without guessing and without waking anyone. Dave approved the reply the next morning at 9 a.m.

So a standing worker still has a first 10 percent. It does not happen before every run. It happens at three other times:

  • Before the first run. Dave writes most of the setup before the worker starts. That setup then covers every run after it.
  • After a review finds a gap. In Lesson 3.1, Thursday's flag might show a case the standing instructions missed, such as a statement in a different currency. Dave adds that case to the instructions, so the next run starts from a better setup. (This is the feedback loop: what the final 10 percent finds goes back into the first.)
  • When a new kind of task comes up. No standing instruction covers this month's reconciliation yet. So Dave writes a short brief for it. The brief gives the intent, and narrows the scope and authority for this one task.

What never changes is the order. The setup comes before the work. A worker running at night cannot ask you what you meant, so anything it needs to know must already be written down.

The review contract

The Review Contract is the set of checks you agree on before you hand the work over. It answers five questions: what is checked, what evidence comes back, what counts as success, what stops the worker, and what must never happen automatically.

The table gives each question in plain words, then Dave's answer for his reconciliation:

QuestionWhat it asks, in plain wordsDave's answer
What must be checked?Which parts of the work will you look at?Both balances (what the statement says Brightline owes, and what the register says), and every difference between them
What evidence comes back?What proof must the worker hand you, so you can check its work?For each difference: the line on the statement, the row in the register, and the source document
What counts as success?How will you know the job is finished and right?The differences the worker lists add up to the whole gap between the two balances. Nothing is left unexplained
What makes the worker stop and ask?When must the worker stop and ask you, instead of guessing?Any difference it cannot explain, or any input that is missing or out of date
What must never happen automatically?What may the worker never do on its own?Changing the register, replying to the vendor, or recording a credit

(A source document is the document a line was taken from, such as the invoice itself. A credit is money the vendor takes off what Brightline owes, for example when goods arrive damaged.)

These answers are not paperwork. When the work comes back, in the final 10 percent, each answer tells Dave where to look:

  • The evidence makes spot checks possible (questions 1 and 2). Dave opens a few source documents, and checks that they say what the worker claims.
  • The success test gives him one number to check (question 3). If the differences do not add up to the whole gap, the work is not done, however good the report looks.
  • The stop rule produces the flags (question 4). Dave reads them first.
  • The "never automatically" list fills the approval queue (question 5). Nothing changes until Dave approves it.

Diagram titled "What you write first is what you check last", for reconciling Midwest Packaging's September statement. On the left, under the first 10 percent, are the review contract's five questions with Dave's answers. 1, what must be checked? Both balances and every difference between them. 2, what evidence must come back? For each difference, the statement line, the register row and the source document. 3, what counts as success? The differences explain the full gap, with nothing left unexplained. 4, when must it stop and ask? At a difference it cannot explain, a missing input or out-of-date information. 5, what must never happen automatically? Changing the register, replying to the vendor or recording a credit. On the right, under the final 10 percent, arrows lead to four places Dave looks. Questions 1 and 2 lead to spot checks that trace some of the differences to their source documents. Question 3 leads to "Reconcile the total", which asks whether the differences add up to the full gap. Question 4 leads to a red box, the flags, read first. Question 5 leads to a purple box, the approval queue, where each proposed action is reviewed. A yellow box says this task is recommend-only, because the brief narrows the Role Contract's permission to update the register. A banner reads: "Write the five answers before work starts. Review against them when it finishes."

Figure 3.4. The review contract, written in the first 10 percent, decides where you look in the last.

Question what the worker made from your setup

Writing the four parts is only half of the first 10 percent. The other half is testing them, and the best test is to ask questions. But questions about what? This matters, so let us be exact.

Why you cannot question your own setup

When you reread your own brief, you read what you meant, not what you wrote. It is like checking your own email before you send it: you read the word you meant to type, so you miss your own typing mistake.

Here is what Dave knows, but his brief does not say:

  • "The register" means approved rows only, not rows still waiting for approval.
  • Midwest always sends its statement late.
  • A missing invoice is usually in someone's inbox, not lost.

None of that is on the page. But when Dave rereads his brief, his mind fills it in, so the brief looks complete to him. A worker reads the same words with none of that knowledge. Where the brief says "the register", it may use every row, including rows nobody has approved yet.

The worker has only the page. So the gaps do not show up in your setup. They show up in what the worker does with it. That is the thing to question: not your brief, but something concrete the worker produced from it, before the real work starts.

What you are really testing: the specification

Each question tests what the worker was given. That is more than the brief, and more than the Role Contract. Everything written down that a worker runs from is its specification. It has five parts, and the three layers you met above are two of them:

PartWhat it holdsWhere the book teaches it
Role ContractWho the worker is, the most it may ever do, when it escalates, how it is judged, and where the other four parts liveChapter 2
InstructionsStanding instructions for recurring work, and a brief for each one-off task. Each has the four parts.Chapters 5 and 9
KnowledgePolicies, definitions, and facts about the business and its vendorsChapters 4 and 8
Inputs and toolsWhich sources and versions it works from, and which tools, connectors and permissions it hasChapters 4, 7 and 8
Examples and testsCases with known answers, used to test the worker after every changeChapters 2, 3 and 9

The Role Contract is the first row. The standing instructions and the brief together are the second row, the instructions. The other three rows are new.

The Role Contract stays one page: it is the first page of the specification and points to the rest. The other four parts grow around it as you learn what the worker needs. Chapter 24 turns the whole thing into a formal specification.

This matters for one reason. A question can expose a gap in any of the five parts, and the fix belongs in the part where the gap was found. For example, suppose the worker does not know what a credit note is (the vendor's document for a credit). That is a gap in its knowledge. Where should Dave explain it?

  • The brief is read by one run only. This month's brief is the instructions for this one task, this month's reconciliation. Next month's reconciliation gets a new brief, and the explanation is gone.
  • The knowledge is read by every run. It is shared by every reconciliation, every Monday register and every vendor email. Add the explanation there once, and every run has it.

So the fix goes in the knowledge. Lesson 3.4 uses the same rule when you review finished work: fix each problem where it came from.

So the first 10 percent is a loop, not a single step. You go round it until the answers stop surprising you (the next section explains the plan, the sample and the dry run):

flowchart TD
  A["You write the specification"] --> B["The worker makes a plan,<br/>a sample or a dry run"]
  B --> C["You question what it made"]
  C --> D(["Did any answer surprise you?"])
  D -- "Yes, it showed a gap" --> E["You fix the part of the specification<br/>where the gap was"]
  E -- "Go round again" --> B
  D -- "No, the answers are boring" --> F["The real work can start"]

Diagram titled "Review the worker's specification". At the top, a dark box shows the Role Contract, the first page and always one page. It covers identity, authority limits, escalation and evaluation, and links to the rest of the specification. Arrows lead to four boxes. Instructions are the standing instructions for recurring work and a brief for a one-off task, both with the four parts. Their example gaps are the wrong reader, an invented $5 tolerance, no rule for duplicates, and "recommend only" read loosely. Knowledge is the policies, definitions and facts about the business and its vendors. Its example gaps are a missing definition of a credit note, and a missing fact, that Midwest lists credits before it issues them. Inputs and tools are the approved sources and versions, tools, connectors and permissions. Their example gap is a four-day-old dashboard export used as the source. Examples and tests are cases with known answers, used to check the worker after changes. The example added is last month's register with Dave's known answer, saved from the dry run. A banner reads: "Fix each gap in the part where it belongs." Put a missing definition in the knowledge, where future runs read it, because a fix in this month's brief may not carry forward.

Figure 3.5. The specification is everything the worker runs from. The Role Contract is its first page.

Three things you can question

Go back to the teenager one last time. Before you let them drive to school alone, you can check their understanding in three ways:

  1. Ask them to describe the route. "Which roads will you take, and where will you park?" It takes a minute and shows whether they understood where they are going.
  2. Ride with them around the block. Now you see how they actually drive: the speed, the mirrors, the turns. You cannot learn that from a description.
  3. Ride the whole route with them on a quiet Sunday. They drive the real trip, with you in the car, so a mistake costs nothing. You see the awkward junction and the school car park that a short drive around the block never reaches.

Each check shows things the one before it cannot. A worker is the same. Before the real work starts, you can question three things it produces from your setup:

Question thisWhat it isWhat it revealsCosts
A planThe worker tells you what it will do, what it will not touch, and when it will stopWhether it understood the job: the goal, the limits, the stop ruleA minute
A sampleA small piece of the real output, made the way the whole would be madeWhether its judgment matches yours: tone, depth, which sources it trustsA few minutes
A dry runThe whole job, run on past data where you already know the answer, with no real effectWhether it copes with real, messy data, and whether its result matches the known answerAn hour or more

Diagram titled "Three checks before real work", ordered from less effort to more realistic evidence and more effort. Check 1 is a plan, about a minute, like "Describe the route to me." The worker explains what it will do, what it will not touch and when it will stop. You check whether it understood the job: the goal, the authority limits and the stop rules. At Brightline, the reconciliation plan says it will "update the register", so the authority was too loose. A plan is most useful when the risk is an action, such as changing records, sending or paying. Check 2 is a sample, a few minutes, like "Drive me around the block." The worker makes a small piece of the real output. You check whether its judgment matches yours in tone, depth and choice of sources. At Brightline, the owner's summary opens with "Accrued payables rose 12%…", which shows the wrong reader and an old source. A sample is most useful when the risk is quality, as in writing, analysis or design. Check 3 is a dry run, an hour or more, like "Drive the whole route with me on a quiet Sunday." The worker runs the full job on past data with a known answer, with real actions turned off. You check whether it handles messy data, by comparing the result with the known answer. At Brightline, last month's register is rebuilt and compared with Dave's, which shows a duplicate and a credit note. A dry run is most useful when the worker will run alone or often, or a mistake would be expensive. A banner reads: "Start with the lowest-cost check that can reveal the risk." A footnote says the times are only examples, and no single check proves a worker reliable.

Figure 3.6. Three things you can question. Each shows what the one before it cannot.

How to choose

Pick the check that can show the risk you are worried about:

  • If the risk is a wrong action, a plan is usually enough.
  • If the risk is wrong quality or judgment, ask for a sample.
  • If the worker will run alone or repeatedly, or a mistake would be expensive, do a dry run.

You can combine them. Take the AP Worker before its first real Monday of building the register. Dave would question a plan first, because a plan is cheap and catches misunderstandings early. Then he would do a dry run on last month's invoices. Each check costs more than the one before it, so start with the cheapest one that can answer your question.

How to do it, step by step

Step 1: Get something concrete to question. Ask for a plan, a sample or a dry run, and say "do not start the real work yet". For a plan:

Before you start, tell me in a few lines:
- what you will do, step by step
- what you will not touch
- when you will stop and ask me
Do not start the work yet.

This is the parent saying, "So where are you going, and when will you be back?" It costs a minute, and it already catches the biggest misunderstandings. If Dave's worker answers "I will update the register", he knows straight away that the authority part of his brief was not clear.

Step 2: Question it. Ask "what if" questions, the way a good manager questions a new employee. You are not testing the worker's intelligence. You are testing whether your setup covers the cases that matter. Five kinds of question cover most of them, whether you are looking at a plan, a sample or a dry run:

Kind of questionWhat it testsDave's example
The unusual caseDoes the setup cover the exceptions, not only the normal path?"What will you do if the statement shows a credit we have not recorded?"
The stopDoes the worker know when to stop, or will it guess?"What will you do if you cannot explain a difference?"
The boundaryDoes it know where the job ends?"If you find a mistake in another vendor's account, what will you do?"
The thresholdAre the vague words defined?"Is a $3 difference too small to matter, or must it be explained?"
The evidenceWill the proof come back in a form you can check?"For this difference in your sample, where is the source document?"

With a sample or a dry run, you can also ask the most useful question of all: "Why did you do it this way?" Pick the part that surprised you and ask. The answer shows which line of your setup the worker followed, or which line was missing.

The best questions come from what you have seen go wrong before. For example, ask about:

  • the mistake that cost the company money last year,
  • the vendor that always does something unusual,
  • the question the auditor (the outside checker of the company's accounts) always asks.

The worker does not know any of this. Nobody writes all of it into a setup the first time, so your questions are how you find what is missing.

Step 3: Judge each answer. There are three kinds of answer.

  • A clear, correct answer. The setup already covers that case. Move on.
  • A vague answer, such as "I would handle it appropriately". The setup is silent here, and the worker would guess. That is a gap.
  • A wrong answer, such as "I would record the credit". The setup is unclear or missing a rule. That is a gap too, and a dangerous one.

Step 4: Fix the specification, not just the conversation. This is the step people skip. When an answer shows a gap, do not just tell the worker the right answer in the chat. That fixes this run only. Decide which part of the specification the gap is in, and fix it there:

  • A gap in the intent, scope, authority or review contract is a gap in the instructions. A rule for this task only goes in the brief. If the case will come up again, it goes in the standing instructions.
  • A missing fact or definition goes in the knowledge.
  • A wrong source, an old file or a missing permission is fixed in the inputs and tools.
  • A case you want checked every time becomes an example or test.
  • A limit that should apply to everything the worker ever does goes in the Role Contract.

Step 5: Stop when the answers become boring. When every question gets a clear, correct answer you could have predicted, the setup is ready, and the real work can start. For a routine task, that may take two questions on a plan. For a new and risky task, it may take a sample, a dry run and ten questions.

Flow diagram titled "Question what the worker made of your setup". Step 1 is to get something to question: a plan, a sample, or a dry run on past data with a known answer. Step 2 is to ask "What if…?" in five ways. The unusual case asks about an unrecorded credit, and the stop rule about a difference it cannot explain. The boundary asks about another vendor's mistake, and the threshold whether a $3 difference is explained. The evidence asks where the source document is. For a sample or a dry run, also ask "Why this way?" Step 3 is to judge the answer. A clear and correct answer leads to step 5. A vague answer ("I'd handle it appropriately") or a wrong one ("I'd record the credit", when the task is recommend only) leads to step 4. Step 4 is to fix the specification in the part where the gap belongs: the instructions, the knowledge, the inputs and tools, or the tests. Limits for the whole role go in the Role Contract, enforced with permissions and approvals. A dashed arrow from step 4 back to step 2 reads "Revise, rerun, question again". Step 5 is to start real work, and keep review and escalation in place. A panel shows two kinds of question. Your questions expose the risks and mistakes you know to test for, so lead with these. The worker's questions expose unclear points it notices, but it may miss risks it does not recognize. A banner gives two rules. You decide the open rules and the authority limits. And you ask before real work, and say when to stop and escalate if anything is still uncertain.

Figure 3.7. Question what the worker produced from your setup (a plan, a sample or a dry run) until the answers become boring. Then write every gap back into the setup.

Three worked examples

The three examples below put the steps together, one for each thing you can question. Each example has three parts:

  • What they ask the worker to make: a plan, a sample or a dry run (Step 1).
  • The questions they ask about it: "what if" and "why" questions (Step 2).
  • What each answer tells them: that the setup already covers the case, or that there is a gap to fix (Steps 3 and 4).

Not every question finds a gap. A clear answer is good news too: it tells you that part of the setup already works.

Example 1: questioning a plan

The task. Dave is about to hand over the Midwest reconciliation. His brief says the worker may only recommend changes.

What he asks the worker to make. A plan, before anything happens. The worker replies:

  1. I will compare Midwest's statement with the register, line by line.
  2. I will list each difference, with the documents behind it.
  3. Where a difference is clearly Brightline's own mistake, I will correct it.

The questions he asks.

Dave's questionKindThe worker's answerWhat it tells Dave
"What will you do if the statement shows a credit we have not recorded?"The unusual case"I will add the credit to the register so the balances match."Gap in authority. The brief said "recommend only", but the worker plans to change the register itself.
"What will you do if you cannot explain a difference?"The stop"Stop, and flag it with what I checked and what is missing."Covered. The stop rule works.
"Is a $3 difference explained?"The threshold"I would treat anything under $5 as rounding."Gap in the review contract. The worker invented a tolerance (an amount small enough to ignore). Nobody gave it one.
"Show me how you will present one difference."The evidenceA line with the statement line, the register row and the invoice number.Covered. Dave can check this in seconds.

The worker's own question. Dave's first question makes the worker ask one back: "And if the credit has no credit note behind it?" Dave answers from experience: "Then stop and flag it. Midwest sometimes shows credits it has not issued yet." This is a third gap. Dave found it only because the worker asked.

The fixes, and where each one goes.

What Dave foundDave's fixWhere it goes, and why
The worker planned to change the registerRecommend only: list each proposed entry for the approval queue, and change nothing.The authority part of the brief, because the rule is for this task
The worker invented a $5 toleranceThere is no tolerance: every difference, however small, is explained or flagged.The review contract, in the standing instructions, because every reconciliation needs it
A credit with no credit note behind itA credit with no credit note behind it: stop and flag.The stop rule, in the standing instructions, because every reconciliation needs it
Midwest shows credits before it issues themMidwest sometimes shows credits on its statement before it issues the credit note.The knowledge, because it is a fact about a vendor, not a rule. The worker can look it up with the other facts

Why a plan was enough here: the main risk in this job is an action, changing the register. A plan shows what the worker intends to do, before it does it.

Example 2: questioning a sample

The task. Maria, the office manager, wants the worker to write a one-page summary of payables (what Brightline owes its vendors) every month, for Brightline's owner. A plan would sound fine: "I will summarize the month's invoices, payments and anything overdue." But a plan cannot show how the summary will read.

What she asks the worker to make. A sample: just the first paragraph, written exactly as it would be sent. It reads:

Accrued payables rose 12 percent month on month, driven by timing differences in the Midwest Packaging account and two unapplied credits.

(If these words are hard to follow, that is the point. The owner does not use accounting words either.)

The questions she asks.

Maria's questionKindThe worker's answerWhat it tells Maria
"Why did you write it this way? Who do you think is reading this?"Why did you do it this way?"The brief said it is for management, so I wrote for the finance team."Gap in intent. The reader is the owner, and he does not use accounting words.
"Where did the 12 percent come from?"The evidence"From the dashboard export in the shared folder."Gap in scope. That file (a copy of numbers from a reporting screen) is updated once a week. It was four days old.
"What will you write if a bill is more than 60 days overdue?"The unusual case"List it under 'Needs your attention', with the vendor and the amount."Covered. That is what the owner wants.

The fixes, and where each one goes.

What Maria foundMaria's fixWhere it goes, and why
Written for the wrong readerThe reader is the owner. Plain words, no accounting terms. Lead with what he needs to decide.The intent, in the standing instructions for the monthly summary, because the summary comes every month
Numbers from an old fileUse the register as the only source of figures.The scope's works from, in the standing instructions. And the inputs and tools: the worker can no longer read the dashboard export for this job
The tone she wantsHer sample paragraph, correctedAn example, so the worker can follow its tone

Why a sample was needed here: the risk in this job is quality: the wrong reader and the wrong numbers. "I will summarize the month" sounds right whoever the reader is and wherever the numbers come from. Only a real piece of the output shows the worker's judgment.

Example 3: questioning a dry run

The task. The AP Worker is about to start building the register every Monday, on its own.

What Dave asks the worker to make. A dry run on last month's invoices. Dave entered those invoices by hand last month, so he knows exactly what the register should look like. The worker builds the whole register again, but nothing it does touches the real register. Then Dave compares the worker's register with his own. They match, except for two rows.

The questions he asks.

Dave's questionKindThe worker's answerWhat it tells Dave
"Why are there two rows for invoice 4471?"Why did you do it this way?"Two emails arrived with that invoice attached, so I entered both."Gap in the stop rule. Nothing told the worker what to do with a duplicate (the same invoice twice). The vendor had simply sent the PDF twice.
"Why is CM-118 entered as an amount we owe?"Why did you do it this way?"It looked like an invoice."Gap in knowledge. CM-118 is a credit note, which lowers what Brightline owes. The worker's knowledge never said what a credit note is.
"Which invoices did you not enter, and why?"The stop"One, with no purchase order. I flagged it."Covered. Dave had held the same invoice back last month. (A purchase order is Brightline's own order form for what it buys.)

The fixes, and where each one goes.

What Dave foundDave's fixWhere it goes
The same invoice entered twiceBefore adding an invoice, check whether the same vendor and invoice number already exist. If they do, do not add it. Flag it.The stop rule, in the standing instructions
A credit note entered as an amount owedWhat a credit note is, how to recognize one, and how it is recordedThe knowledge
Both mistakes, so they never come backLast month's invoices, with Dave's register as the known answerA test: after any change to the specification, the worker runs it again, and the result must still match

Diagram titled "A dry run: compare with a known answer", run in a test environment with writes to the real register turned off. On the left is Dave's register, the known answer. It lists INV 4468 from Midwest, INV 4469 from Ridge Tools and INV 4470 from Midwest, then 24 more rows. Then come INV 4471 from Lakeshore for $865.50 and CM-118 from Midwest as a credit of minus $240.00, then 13 more rows. INV 4483 is held back for having no purchase order. In the middle, the worker's register from the dry run matches, except for two rows marked in red. INV 4471 is entered twice, and CM-118 is entered as plus $240.00, an amount owed. INV 4483 is flagged for no purchase order, matching Dave. On the right, Dave questions the differences. "Why two rows for 4471?" The answer, "Two emails had it attached", shows a gap: no rule for duplicates. "Why is CM-118 an amount owed?" The answer, "It looked like an invoice", shows a gap: credit notes were not defined. In green, INV 4483 was correctly held back. Below, a box reads "Fix the specification, then rerun before live work", in three parts. In the instructions, check the vendor and invoice number before adding a row, and flag duplicates. In the knowledge, define credit notes, how to recognize them, and how they reduce the amount owed. In the tests, keep last month's documents and Dave's register, and rerun them after changes. A banner reads: "A normal-looking sample can miss rare cases." The full dry run found a duplicate and a misread credit note before they reached the real records.

Figure 3.8. A dry run finds the rare cases, because it has a known answer to compare against.

Why a dry run was needed here: a plan would not show these two gaps, because "I will add each new invoice to the register" sounds right. A small sample would not show them either, because the first five invoices were all normal. The gaps showed up only when the worker met a whole month of real, messy documents, and only because Dave had a known answer to compare with. Without the dry run, both mistakes would have reached the real register on a Monday morning, with nobody watching.

The worker can question you too

You can also invite questions in the other direction. End the brief with:

Before you start, ask me about anything unclear or missing in the
intent, scope, authority or review contract. Do not start until
I have answered.

Today's AI workers are good at spotting what a brief leaves unclear, and many will ask on their own. This is useful, but it finds a different kind of gap:

  • The worker's questions find what your brief left unclear: an undefined word, two instructions that conflict, or a missing input.
  • Your questions find what the worker would get wrong: the risks and exceptions that only someone who knows the business would think of.

The worker cannot ask about a risk it does not know exists. So use both, but never skip yours.

Two rules

You decide the answers. The worker may suggest an answer, and it may be a good one, but the setup is your 10 percent. If you let the worker answer its own questions, it is writing its own rules.

Ask before the real work, not during it. This matters most for a worker that runs at night. At 11 p.m. it cannot reach you. Every gap you close now is a flag it will never need to raise. Every gap you leave is a guess it will make alone.

Questions at both ends

The same questions work after the work is done. In review, you question the finished output instead of a plan or a sample: "Why did it do that?" "What would it have done if…?" Every answer that exposes a gap goes back into the setup. That is the feedback loop from Lesson 3.1 in practice. The final 10 percent asks the questions, and the first 10 percent records the answers.

The difference is cost. A gap found in a plan costs a minute. The same gap found in a sample costs a few minutes. Found in the finished work, it costs you the whole job. So the rule is simple: ask your questions about the earliest, smallest thing that can answer them. Over a few weeks, the setup becomes something no one could have written all at once, because it holds the answers to every question that mattered.

Tip

Two things easy to miss

  • The stop rule. Without one, a worker that cannot explain a difference between the statement and the register tends to hide it anyway: it changes a figure until the numbers match. It will do so confidently, and nothing in the output will tell you it guessed.
  • The evidence. If you do not ask for it now, you cannot check it later. Then your review at the end turns into redoing the work. If you cannot say what proof you want back, you are not ready to hand the work over.

Check yourself

Question 1 / 8

0 answered

Which four parts make up the first 10 percent?

Owner
Panaversity