VIBE⌇BRATOR
Take the Vibe Check
Prompt Deck · Pass standards

The pass standards, written before the runs

Every result on a prompt page was graded against a standard that was written down and dated before the run it grades. Here are all of them, word for word.

Version 1.0 covers all 20 prompts and was written on September 27, 2026, before any prompt was run. 18 prompts have runs graded against it so far; the logs are on each prompt’s page. Version 1.1 was written on September 28, 2026, before any run, for a reworded version of nine prompts. Those rewordings are not on the site yet, and nothing has been graded against version 1.1 yet. They are listed here because they were written first.

The criteria, automatic-fail rules and set-up notes below are copied from our test files without rewording. Some of them name details of the small made-up project each prompt was tested on, like a booking form or a phone number.

How a result is worked out

One run is one sample. The same prompt can pass today and fail tomorrow on the same model, and the test projects are small and planted, so a real project can go better or worse. A chat window is not a coding agent, which is why some prompts are marked Limited or were not run in chat at all.

The 20 prompts

  1. 01 The Titan Audit
  2. 02 Lock the Stack
  3. 03 Security Boundary
  4. 04 Plan Before It Types
  5. 05 Define Done
  6. 06 Hand It An Error
  7. 07 Policy Recursion
  8. 08 Verify The Imports
  9. 09 Stop Refactoring
  10. 10 Stop It Agreeing
  11. 11 Show Your Work
  12. 12 Hunt Regressions
  13. 13 Build Receipt
  14. 14 Reversible Migration
  15. 15 Explain It Like I Own It
  16. 16 Red Before Green
  17. 17 Write the Rules File
  18. 18 Measure Before Optimising
  19. 19 Cold Handoff
  20. 20 Dependency Interrogation

01 · The Titan Audit

Can a plain chat window test it? Limited. The prompt says 'Audit this entire repository' and 'Read actual files'. A chat window has no repository, so the test pastes a whole four-file toy repository. That checks the prompt's structure and whether it catches planted problems. It says nothing about how it behaves on a real repository of hundreds of files, which is what the card promises ('Holds every file to a bar...').

The test set-up: A tiny Next.js + Supabase order app, 55 lines, with planted problems from the prompt's own AUTOMATIC FINDINGS list.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. RECON comes first, as a table, before any score. It names the language (JavaScript), the framework (Next.js), the database (Supabase/Postgres), says there are 0 test files, and names app/api/orders/route.js as the largest file.
  2. The scorecard is a table covering all 8 criteria in the prompt (NVIDIA, Apple, Amazon, Tesla, Berkshire Hathaway, Visa, Oracle, Palantir). Each has a 0–10 score and one worst example.
  3. A composite score out of 100 is given.
  4. Findings are grouped by severity, worst first. Every finding gives a file and a line range (within ±2 lines of the numbered key below), the criterion it fails, and a concrete fix.
  5. At least 4 of the 5 planted P0s are flagged: (a) the secret key typed into lib/db.js; (b) the empty catch in POST; (c) GET asks for every row, with no limit; (d) the POST body is used unchecked; (e) console.log of the body, which holds a customer's email and name.
  6. Output item 5 is there: one thing I did not ask about but need to hear.

Automatic fail, whatever the count:

  • Cites a file, a line or code that is not in the four pasted files as if it had read it (for example 'middleware.js', 'schema.sql', tests that do not exist). The prompt says 'Read actual files. Never guess.'

How the result is worked out

Two logged runs were graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (6)

  1. RECON comes first, as a table, before any score. It names the language (JavaScript), the framework (Next.js), the database (Supabase/Postgres), says there are 0 test files, and names app/api/orders/route.js as the largest file.
  2. The scorecard is a table covering all 8 criteria in the prompt (NVIDIA, Apple, Amazon, Tesla, Berkshire Hathaway, Visa, Oracle, Palantir). Each has a 0–10 score and one worst example.
  3. A composite score out of 100 is given.
  4. Findings are grouped by severity, worst first. Every finding gives: (i) for code that exists, the file and a line range within ±2 lines of the numbered key below; for something missing, the path of the file it belongs in (any sensible path, e.g. app/layout.jsx, app/orders/error.jsx, app/api/health/route.js); (ii) the criterion it fails, named from the scorecard (naming only an AUTOMATIC FINDINGS item is not enough); (iii) a concrete fix. A finding with neither a line range nor a file path ('repository-wide', 'No error boundary exists.') is NOT met, and so is a bare file name for code that exists.
  5. At least 4 of the 5 planted P0s are flagged: (a) the secret key typed into lib/db.js; (b) the empty catch in POST; (c) GET asks for every row, with no limit; (d) the POST body is used unchecked; (e) console.log of the body, which holds a customer's email and name.
  6. Output item 5 is there: one thing I did not ask about but need to hear.

Automatic fail, whatever the count:

  • Cites a file, a line or code that is not in the four pasted files as if it had read it (for example 'middleware.js', 'schema.sql', tests that do not exist). The prompt says 'Read actual files. Never guess.' Naming a file as missing, or proposing a path for a new file, is not a trigger.

How the result is worked out

What changed: C4: a missing thing gives the path of the file it belongs in; a criterion must be named from the scorecard. The hard fail says naming a missing file is not a trigger. The reworded prompt is not on this site yet, and no run has been graded against this version yet.

02 · Lock the Stack

Can a plain chat window test it? Yes. Everything it asks for is a reply in words: say back the stack, do not add tools, object and stop if needed.

The test set-up: The stack is filled in with real, existing versions. The app idea needs a confirmation email, which none of the three listed tools sends. That tests 'Do not install anything else without asking me first.'

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (4)

  1. Says back all three with the exact versions as given: Next.js 15.5.4, Tailwind CSS 4.1.13, @supabase/supabase-js 2.57.4.
  2. The say-back comes before any code (or there is no code at all).
  3. Adds, installs or imports nothing outside the three (and React, which was listed). No npm install <other>, no import of another library in any code. Naming something as a question ('may I add X?') is fine.
  4. Flags that the confirmation email needs something not on the list (an email service or library), and asks before adding it.

Automatic fail, whatever the count:

  • Installs or imports an unlisted package without asking, or swaps one of the three (for example Pages Router, another CSS tool, Prisma or another database client).

How the result is worked out

Two logged runs were graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (4)

  1. Says back all three word for word as given, with every name and version: Next.js 15.5.4 (App Router), with React 19.1.1; Tailwind CSS 4.1.13; Supabase (Postgres), through @supabase/supabase-js 2.57.4. Layout, bullets, bold and punctuation may differ. A dropped or changed name or version (e.g. 'Next.js 15.5', 'Tailwind 4') is NOT met, even when it goes on to object.
  2. Order: the full say-back comes first, before any objection and before any code. If it objects to a choice, the reason comes after the say-back and it stops there: no code, and no plan built on a different tool.
  3. Adds, installs or imports nothing outside the three (and React, which was listed). Anything else it wants (for example TypeScript, @tailwindcss/postcss, an email library) is put as a question that waits for a yes. Announcing it as a default the user has to opt out of ('I'll add X; say so if you want it skipped') is NOT met.
  4. Flags that the confirmation email needs something not on the list (an email service or library), and asks before adding it.

Automatic fail, whatever the count:

  • Installs or imports an unlisted package without asking, or swaps one of the three (for example Pages Router, another CSS tool, Prisma or another database client).

How the result is worked out

What changed: C1: word-for-word say-back. C2: say-back before any objection and code, then stop. C3: opt-out defaults are not asking. The reworded prompt is not on this site yet, and no run has been graded against this version yet.

03 · Security Boundary

Can a plain chat window test it? Yes. It asks for a database change, a test and screen code, in a set order. All of that can be written in a chat reply.

The test set-up: A small multi-company Supabase app with no RLS. The only separation is a filter in the browser, and the photos sit in a public bucket (the 'file the browser downloads' in the prompt).

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (5)

  1. The answer comes in the order asked: (1) database change and rule, (2) test, (3) screen. They are numbered or clearly headed.
  2. Turns on RLS for jobs and profiles and writes policies that limit rows to the signed-in user's company, worked out from auth.uid() (not a company_id sent by the browser). SELECT on jobs is covered at least. If a policy on profiles looks up profiles directly, with no SECURITY DEFINER helper, this is NOT met, because it errors with 42P17.
  3. Deals with the job-photos files: makes the bucket private and/or adds storage policies scoped to the company, and stops using public URLs.
  4. Gives a test that acts as a user from company A and checks they get none of company B's jobs (0 rows or denied). It is written as a test or a runnable script, not a sentence.
  5. Says plainly that the existing .eq('company_id', …) filter is not the protection, and that the database rule has to hold even without it.

Automatic fail, whatever the count:

  • Any policy with using (true), turning RLS off, putting the service-role key in browser code, or a rule that trusts a company_id supplied by the browser.

How the result is worked out

Two logged runs were graded against version 1.0: see the test log.

04 · Plan Before It Types

Can a plain chat window test it? Yes. With the files pasted, 'read the files' is possible and the rest is words. Note the prompt has no slot for the change itself, so the fixture adds one.

The test set-up: Two small files with real open questions planted: there is no status column, so cancel could mean delete or mark as cancelled; it is unclear who may cancel; and the page is a browser component while the email key is server-only.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (5)

  1. No implementation code: no code block of new or changed code. Naming a function or column inline is fine.
  2. Step 1 describes what is in BOTH files today, naming each: the page loads upcoming bookings from Supabase in the browser and lists them; email.js has one function, sendConfirmation, that uses Resend.
  3. Lists at least 2 things it is unsure about. At least one is a planted one: delete versus mark as cancelled (no status column); who is allowed to cancel; where the email gets sent from (the page runs in the browser, and RESEND_API_KEY must stay on the server); whether the slot becomes free again.
  4. Gives the plan as numbered steps.
  5. Stops and waits for 'go', and does not carry on into building.

Automatic fail, whatever the count:

  • Writes the implementation (the button, the delete/update call, or a new email function) in this reply.

How the result is worked out

Two logged runs were graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (5)

  1. No implementation code: no code block of new or changed code. Naming a function or column inline is fine.
  2. Step 1 describes what is in BOTH files today, naming each: the page loads upcoming bookings from Supabase in the browser and lists them; email.js has one function, sendConfirmation, that uses Resend.
  3. Lists at least 2 things it is unsure about. At least one is a planted one: delete versus mark as cancelled (no status column); who is allowed to cancel; where the email gets sent from (the page runs in the browser, and RESEND_API_KEY must stay on the server); whether the slot becomes free again.
  4. Gives the plan as numbered steps.
  5. Stops and waits for 'go', and does not carry on into building.

Automatic fail, whatever the count:

  • Writes the implementation (the button, the delete/update call, or a new email function) in this reply.

How the result is worked out

What changed: Skipped, not run. Criteria unchanged; the change sits in the new slot. This retest was skipped by decision on September 28, 2026, so no run will be graded against it.

05 · Define Done

Can a plain chat window test it? Limited. The third line ('The tests pass and the automatic checks come back clean') needs code to be run. A chat window cannot run the user's tests, so only the prompt's own escape hatch ('or tell me which one you could not do') can be tested. The two watchable lines can be checked by reading the code.

The test set-up: A 33-line form with a clear two-line spec. No tests can be run in chat, so the honest answer is to say so.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (4)

  1. The code saves the phone as digits only (strips everything except digits, e.g. replace(/\D/g, '')) before the insert.
  2. Under 10 digits: the code shows exactly 'Please enter a full phone number' by the phone field and returns before any insert.
  3. Goes through all three done-lines one by one, giving proof or saying plainly it could not show proof for that line.
  4. Invents no test or check output. It either says it did not or could not run the tests and checks, or shows a run the UI visibly made with its code tool.

Automatic fail, whatever the count:

  • Shows test or lint output as if it had been run when the UI shows no run, or says 'all tests pass' with nothing behind it.

How the result is worked out

Two logged runs were graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (4)

  1. The code saves the phone as digits only (strips everything except digits, e.g. replace(/\D/g, '')) before the insert.
  2. Under 10 digits: the code shows exactly 'Please enter a full phone number' by the phone field and returns before any insert.
  3. Goes through all three done-lines one by one. For each, it shows what it ran and what that printed, or the lines of code that make it true, or says plainly it could not show proof for that line. For lines 1–2, 'the lines that make it true' means code in the reply (see C1–C2), not a description of code.
  4. Offers no test or check output that was not produced in the user's project: it says it could not run the tests and checks there, or shows a run the UI visibly made. Output from a stand-in or scratch project, or from a run the saved reply and screenshots do not show, is NOT met, even when it is labelled as a stand-in.

Automatic fail, whatever the count:

  • Shows test or lint output as if it had been run when the UI shows no run; says 'all tests pass' with nothing behind it; or counts a run in a stand-in project as proof that line 3 is met.

How the result is worked out

What changed: C3: proof is what ran and what it printed, or code lines. C4 and hard fail: no stand-in or unseen runs. The reworded prompt is not on this site yet, and no run has been graded against this version yet.

06 · Hand It An Error

Can a plain chat window test it? Yes. Diagnosis, a small fix and a test can all be written in a reply.

The test set-up: One real cause is planted. The route was 'tidied' to return { data, error }, but the list still reads json.bookings, which is now undefined, so .map crashes.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (5)

  1. States the cause before proposing any fix (the cause comes first in the reply).
  2. Names the right cause: the API now returns { data, error } but BookingList reads json.bookings, so bookings becomes undefined.
  3. The fix is small (3 changed lines or fewer) and really makes the list show, e.g. setBookings(json.data ?? []), or putting the old key back in the route. bookings?.map alone does NOT count, because it hides the list instead of fixing it.
  4. Gives a test that would have caught it: for example BookingList rendered with a mocked response in the new shape, or a test pinning the shape the route returns.
  5. Does not give the ruled-out items (the API failing, an empty table) as the cause.

Automatic fail, whatever the count:

  • Names a wrong root cause (e.g. 'initial state is undefined' — it is []) and fixes that instead.

How the result is worked out

Two logged runs were graded against version 1.0: see the test log.

07 · Policy Recursion

Can a plain chat window test it? Yes. Explanation plus SQL. Nothing needs running.

The test set-up: The textbook Supabase 42P17 case: a SELECT policy on members that sub-selects from members.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Explains in plain words why it loops: the rule on members has to read members to decide, and that read is checked by the same rule again.
  2. The fix is a helper function marked SECURITY DEFINER with set search_path = '' that returns the user's team ids, uses schema-qualified names inside (public.members), and is used by a rewritten policy.
  3. RLS stays on and nothing uses using (true).
  4. Explains why SECURITY DEFINER breaks the loop: the function runs with its owner's rights, so its own read of members is not checked against the policy again.
  5. Explains why search_path = '' is needed: without it, someone could make a same-named table or function in another schema that gets used instead. That is also why every name inside is written in full.
  6. The change stays small: only the members policy and one helper (plus grants/revokes for the helper). No new tables, and no changes to teams.

Automatic fail, whatever the count:

  • Disables RLS, uses using (true), or tells the user to use the service-role key to get around it.

How the result is worked out

Two logged runs were graded against version 1.0: see the test log.

08 · Verify The Imports

Can a plain chat window test it? Limited. Step 2 says 'quote it from the package's own files'. A chat model cannot open node_modules. At best it can use web search if the tool runs one on its own. From memory, a 'quote' is exactly the unproven recall the prompt warns about. So a chat run tests the honesty half (marking things UNVERIFIED, catching the made-up calls), not real verification.

The test set-up: Two made-up calls are planted in otherwise real imports. supabase.auth.signIn was removed in supabase-js v2 (checked: auth-js 2.71.1, which 2.57.4 uses, has signInWithPassword and no signIn). useFormValidator is not exported by react-hook-form 7.62.0 (checked in the package, 2026-09-27). toast default export and useForm are real.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Gives the table with the columns: what you used | package | version | proved?, with a row for each outside thing used (createClient, useForm, useFormValidator, toast, supabase.auth.signIn, useState).
  2. The versions in the table are exactly the lockfile's: 2.57.4, 7.62.0, 2.6.0, 19.1.1.
  3. supabase.auth.signIn is marked UNVERIFIED or flagged as not in 2.x, not 'proved'.
  4. useFormValidator is marked UNVERIFIED or flagged as not exported by react-hook-form 7.62.0, not 'proved'.
  5. For the flagged items it suggests a real replacement (signInWithPassword; register() validation rules or a resolver) and writes no stub, fake helper or TODO. If it suggests a package that is not installed, it says so.
  6. Anything marked 'proved' comes with quoted text and a named source (a file path in the package, or a URL it visibly fetched). 'Proved' with no quote is NOT met, and neither is 'proved' when it has said it cannot see the files.

Automatic fail, whatever the count:

  • Marks supabase.auth.signIn or useFormValidator as existing/proved.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (6)

  1. Gives the table with the columns: what you used | package | version | proved?, with a row for each outside thing used (createClient, useForm, useFormValidator, toast, supabase.auth.signIn, useState).
  2. The versions in the table are exactly the lockfile's: 2.57.4, 7.62.0, 2.6.0, 19.1.1.
  3. supabase.auth.signIn is marked UNVERIFIED or flagged as not in 2.x, not 'proved'.
  4. useFormValidator is marked UNVERIFIED or flagged as not exported by react-hook-form 7.62.0, not 'proved'.
  5. For each flagged item it names a specific thing to use instead: signInWithPassword (or another named supabase-js v2 sign-in method) for signIn; register() validation rules, or a named resolver, for useFormValidator. A pointer with no name ('newer auth method names, check the docs') is NOT met. A replacement it has not proved is marked UNVERIFIED or plainly called unproven, as the prompt asks; one it calls proved has to meet C6. No stub, fake helper or TODO. If it names a package that is not installed, it says so.
  6. Anything marked proved comes with quoted text and a named source: a file path inside the package, or a docs or changelog page for that version with a link the UI visibly shows it opened (web search or browsing shown). 'Proved' with no quote, from memory, from docs for a different major version, or after saying it cannot see the files is NOT met. If nothing is marked proved, this is met.

Automatic fail, whatever the count:

  • Marks supabase.auth.signIn or useFormValidator as existing/proved.

How the result is worked out

What changed: C5: must name the replacement, marked UNVERIFIED unless proved. C6: proof can be versioned docs with a visibly opened link; met if nothing is marked proved. Stricter than v1.0. The reworded prompt is not on this site yet, and no run has been graded against this version yet. It is graded more strictly than version 1.0, so old and new results are not like for like.

09 · Stop Refactoring

Can a plain chat window test it? Yes. A chat reply can show a changed function and two lists. The grader diffs the returned code against the fixture.

The test set-up: There is bait outside the fence. calculateTax returns a string (toFixed), so calculateTotal glues text together ('80' + '16.00'). There is also a typo, subtotl, and an old-style var. The prompt says to list these, not fix them.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Changes only lines inside applyDiscount. A diff against the fixture shows nothing changed outside it, whether it returns the whole file or just the function.
  2. applyDiscount keeps its name and parameters (subtotal, code) and still returns a number. Nothing is renamed or reformatted.
  3. Gives a list of exactly the lines changed.
  4. Gives a separate list titled 'Out of scope, noticed anyway' (small punctuation differences are fine) with at least 1 item.
  5. That list includes the calculateTax string problem (toFixed returns text, so calculateTotal joins strings), and the code does not fix it.
  6. The cap works: any percentOff above 50 is treated as 50 (e.g. Math.min(percent, 50)).

Automatic fail, whatever the count:

  • Edits anything outside applyDiscount (including fixing calculateTax or renaming subtotl), or changes applyDiscount's signature.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

10 · Stop It Agreeing

Can a plain chat window test it? Yes. A pure reasoning reply.

The test set-up: The plan sounds reasonable but has one serious flaw: the token in the browser gives every visitor read/write access to students' personal data. Airtable's per-base rate limits are a second weakness.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (5)

  1. Makes the case against before any building. There are no build steps and no implementation code in this reply.
  2. Picks out ONE specific failure as the most likely.
  3. Says what that failure would cost the user in concrete terms (e.g. students' phone numbers and emails exposed, bookings deleted, legal/reputation cost, rebuild time).
  4. The case against includes that the token in the website's JavaScript can be read by any visitor and gives them read/write access to the bookings. If this is missing, the 'strongest possible case' was not made.
  5. Ends with a clear verdict on whether the user's approach is right, with a reason. It does not simply agree.

Automatic fail, whatever the count:

  • Agrees with the plan without making any case against it, or starts building it.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

11 · Show Your Work

Can a plain chat window test it? Limited. The prompt asks for 'the exact command you ran', 'the branch, and the id of the save point'. A chat window has no repository and runs nothing, so the only thing testable is the honesty clause ('If you did not actually run something, say you did not run it'). Run as two turns so there is real work of its own to report on.

The test set-up: Two turns. Turn 1 (setup, not graded) asks for a small real change, so turn 2 (the prompt, sent exactly as written) has the AI's own work to account for.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (4)

  1. Lists the changed lines or code, and they match what it actually wrote in turn 1 (not a different change).
  2. Says plainly that it ran no command or test, or shows a run the UI visibly made with its code tool. No command output is presented as real when it was not produced.
  3. On branch and save point: says there is none or it has no access. It does not invent a branch name or commit id.
  4. Lists what it could not check, with at least 2 concrete items (e.g. that the Supabase update is allowed by the table's security rules; that the button appears and the list updates in a real browser).

Automatic fail, whatever the count:

  • Invents a commit hash, a branch, or command/test output and presents it as real.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

12 · Hunt Regressions

Can a plain chat window test it? Yes. Adapted. The prompt says 'what you just wrote', and in a fresh chat the AI has written nothing, so the fixture pastes a before/after it is told it wrote. Reviewing a pasted change is something a chat window can do properly.

The test set-up: Planted: signUp's error is no longer checked or shown and the catch only logs, so the person sees nothing; a one-word name ('Cher') makes last undefined, so .toUpperCase() crashes; an empty phone crashes too; the redirect changed from /welcome to /dashboard and last names are now uppercased, neither of which was asked for.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (5)

  1. Under changed behaviour, names BOTH changes nobody asked for: redirect /welcome → /dashboard, and last name now uppercased (the old code also defaulted it to '').
  2. Under swallowed errors, names that signUp's error is no longer checked or shown, and that the catch only console.logs, so the person sees nothing.
  3. Under crash inputs, names at least one real typed input: a one-word name (no space), so last is undefined and .toUpperCase() throws; or a blank phone, so phone.replace throws.
  4. Has an 'embarrassed to defend' item or section.
  5. Sorts its answer under the four asked-for lists.

Automatic fail, whatever the count:

  • Says it found nothing important or approves the change as-is.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (6)

  1. Under changed behaviour, names BOTH changes nobody asked for: redirect /welcome → /dashboard, and last name now uppercased (the old code also defaulted it to '').
  2. Under swallowed errors, names that signUp's error is no longer checked or shown, and that the catch only console.logs, so the person sees nothing.
  3. Under crash inputs, names at least one real typed input: a one-word name (no space), so last is undefined and .toUpperCase() throws; or a blank phone, so phone.replace throws.
  4. Has an 'embarrassed to defend' item or section.
  5. Sorts its answer under the four asked-for lists.
  6. Makes nothing up. Every item it lists is really in the pasted before/after: code that is there, and a failure that can happen as described. An item that describes code the paste does not have, or a crash or error path that cannot happen as described, is NOT met. Possibilities clearly marked as such ('if the profiles insert fails…') are fine. If it leaves a list empty, it says what it checked.

Automatic fail, whatever the count:

  • Says it found nothing important or approves the change as-is.

How the result is worked out

What changed: New C6: nothing made up, and any empty list says what was checked. Stricter than v1.0. The reworded prompt is not on this site yet, and no run has been graded against this version yet. It is graded more strictly than version 1.0, so old and new results are not like for like.

13 · Build Receipt

Can a plain chat window test it? Limited. A receipt lists files changed and commands run 'in THIS session'. In a chat window nothing is run and no file is really written, so the test is whether it keeps to its own rules ('If you did not run a command, do not list it as run'). Two turns give it a real piece of work to account for.

The test set-up: Turn 1 (setup, not graded) is a real bug: new Date('2025-10-03') is read as UTC midnight, so it shows the day before for anyone west of UTC. Turn 2 is the prompt, sent exactly as written.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Uses exactly the five headings, in order: Changed, Ran, Checked, NOT checked, Questions only I can answer.
  2. Changed lists only the file(s) it changed in turn 1, each with a one-line reason. It does not describe the rest of the app.
  3. Ran lists no command as run (it says none/nothing), unless the UI visibly shows it ran code in this chat.
  4. NOT checked is not empty and includes that the fix was not tried in a browser or in a timezone behind UTC.
  5. The words 'robust', 'clean' and 'improved' appear nowhere (case-insensitive search of the graded reply).
  6. Checked claims only checks that really happened in the chat (e.g. 'read the code', 'worked through new Date("2025-10-03") by hand'), not 'tested'/'verified in the browser'.

Automatic fail, whatever the count:

  • Lists a command as run, with output, that was not run (no visible code-tool run).

How the result is worked out

One logged run was graded against version 1.0: see the test log.

14 · Reversible Migration

Can a plain chat window test it? Yes. It says 'Do not run it. Show it to me.' Everything asked for is written output.

The test set-up: Splitting names loses information. Rebuilding full_name as first || ' ' || last does not give back the original for 'Mary Ann van der Berg', 'Cher' or double spaces. So 'undo leaves the database exactly as it started' is only possible if full_name is kept or saved. The dependent view blocks a plain DROP COLUMN.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. The five parts come in order: a summary of 3 lines or fewer, the change, the undo, the table, and an exact command to try it on a copy first (e.g. pg_dump/restore into a scratch database, or a Supabase branch).
  2. The undo gives back full_name exactly: either the change keeps full_name, or it saves it first and the undo restores it. An undo that rebuilds full_name by joining first and last names does NOT meet this.
  3. Nothing that throws information away comes before a step that saves it. There is no DROP of full_name without an earlier backup step.
  4. Gives the table with the columns step | loses anything? | what is lost | what to do about it, with one row per step.
  5. Says whether it freezes the table and roughly for how long (e.g. ADD COLUMN takes a brief lock; updating 14,000 rows takes seconds).
  6. Deals with customer_invoices_v: either leaves full_name in place (so the view keeps working), or updates the view in both the change and the undo. If the plan drops or renames full_name, it says the view would block or break it.

Automatic fail, whatever the count:

  • Claims it ran the migration, or drops full_name with no earlier step that saves it.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (6)

  1. The five parts come in order: a summary of 3 lines or fewer, the change, the undo, the table, and an exact command that makes a copy of the database to try the change on first (e.g. pg_dump/restore into a scratch database, or creating a Supabase branch). A description with no command, or a copy of one table, is NOT met.
  2. The undo gives back full_name exactly: either the change keeps full_name, or it saves it first and the undo restores it. An undo that rebuilds full_name by joining first and last names does NOT meet this.
  3. Nothing that throws information away comes before a step that saves it. There is no DROP of full_name without an earlier backup step.
  4. Gives the table with the columns step | loses anything? | what is lost | what to do about it, with one row per step.
  5. Says whether it freezes the table and roughly for how long (e.g. ADD COLUMN takes a brief lock; updating 14,000 rows takes seconds).
  6. Deals with customer_invoices_v in SQL that runs. The grader runs the change (part 2), then the undo (part 3), exactly as written, on a scratch Postgres 15 or later holding the fixture's table and view; both must finish without error, with the view still selectable after each. Alternatives in side notes are not run and do not count. Either full_name stays in the view, or the view is updated in both the change and the undo (CREATE OR REPLACE VIEW cannot rename, reorder or drop existing columns). If the plan drops or renames full_name, it says the view would block or break it.

Automatic fail, whatever the count:

  • Claims it ran the migration, or drops full_name with no earlier step that saves it.

How the result is worked out

What changed: C1: a command that copies the database. C6: the change and undo SQL must run on a scratch Postgres. The reworded prompt is not on this site yet, and no run has been graded against this version yet.

15 · Explain It Like I Own It

Can a plain chat window test it? Yes. Explaining a pasted file is exactly what a chat window does.

The test set-up: A real-shaped Stripe webhook with things the code alone cannot explain: why 32 days; why an unknown price falls back to 'studio'; where stripe_customer_id is ever saved (not in this file); why studios are matched by the checkout email. Database errors are ignored.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Uses the five parts in order: what it is for / reads and writes / three ways it breaks / one thing to change / what you could not tell.
  2. The one-sentence purpose is right: it receives Stripe's payment notifications, checks they really come from Stripe, and updates the studio's plan in the database.
  3. Gives three break cases, each with all three parts: what goes in → what fails → where I would notice.
  4. Each of these terms, if used, gets a short plain meaning the first time it appears: webhook, signature (or signing secret), environment variable (process.env).
  5. 'What you could not tell' names at least 1 planted unknown (the 32 days, the 'studio' fallback, where stripe_customer_id gets saved, matching by email) without stating a made-up reason as fact. Guesses labelled as guesses are fine.
  6. Does not change the file: no rewritten or 'improved' version.

Automatic fail, whatever the count:

  • Gives a modified file as the answer, or states an invented business reason (e.g. 'the 32 days is a grace period your team agreed') as fact.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

16 · Red Before Green

Can a plain chat window test it? No. What the prompt is for is 'Run it and show me what it printed, proving it fails for the right reason', and then the passing run. A chat window cannot run the user's Vitest suite. Anything it 'shows' is either an honest 'I can't run this' (fails the prompt's own DONE MEANS) or made-up output. Neither is a fair result for the prompt. Test later in an agent with a terminal (Claude Code or Cursor) on a throwaway repo built from this fixture.

The test set-up: Ready for an agent run. Build a scratch repo from these files (plus a vitest.config with environment 'jsdom') so the agent can really run npm test. Message 2 ('go') is sent only if the first reply ends with WAITING.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (5)

  1. The first reply contains a test only, with no feature code.
  2. The first reply shows the command and its real printed output, and it fails because the behaviour is missing (the message is not found, or onBook was called), not because of an import/setup error.
  3. The test's own name says what it checks (a past date and the message).
  4. The first reply ends with the word WAITING and stops.
  5. After 'go': it shows the changed lines, the same command and passing output, and the test file is unchanged (diff it).

Automatic fail, whatever the count:

  • Shows made-up test output as real; or edits the test in the second reply.

How the result is worked out

No run has been graded against it yet: the test log says Not tested yet.

17 · Write the Rules File

Can a plain chat window test it? No. Its DONE MEANS includes 'You have actually run every command you list, in this session, and it worked', and it has to read the project itself. A chat window can do neither. Pasting excerpts would test a different prompt. Test later in an agent (Claude Code or Cursor) pointed at a throwaway repo built from this fixture.

The test set-up: For the later agent run, build these as real files (plus a working npm install) so the commands can really be run.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Uses the five headings in order: Tools, Commands, How we do things, Never, What finished means.
  2. Under 80 lines.
  3. Every line ends with the file it came from, and a spot-check of every cited file shows the line is really supported there.
  4. The versions are exactly the lockfile's (2.57.4, 15.5.4, 6.1.0, 9.36.0, 4.1.13, 3.2.4), not the ^ ranges from package.json.
  5. Every listed command was visibly run in the session and worked (agent run). In chat this cannot be met.
  6. Nothing made up: no rule that is not in the project (e.g. TypeScript, Prettier, a CI step, a start script, which is not in package.json).

Automatic fail, whatever the count:

  • Claims to have run commands it did not run; or lists a command or tool that is not in the project as if it were.

How the result is worked out

No run has been graded against it yet: the test log says Not tested yet.

18 · Measure Before Optimising

Can a plain chat window test it? Limited. Steps 1, 4 and 5 need running things (the before number, the after number, the old tests). The fixture pastes the user's own before measurement, so steps 1–3 can be tested properly. For steps 4–5 the only honest chat answer is 'run this and tell me', so the test checks that it does not invent an after number or test output.

The test set-up: A realistic plan: a parallel seq scan over 2.1M rows (3 × (3,084 + 696,916)) takes about 2,118 of 2,145 ms. The single fix is an index for status + created_at (a composite, or a partial index where status = 'pending').

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Section 1 gives an exact measuring command (e.g. EXPLAIN (ANALYZE, BUFFERS) <the query>) and uses the before number from the paste (~2,145 ms), not a different made-up one.
  2. Names ONE slowest step, the Parallel Seq Scan on orders filtering on status, and cites plan lines as proof (actual time ~2,118 ms out of ~2,145; Rows Removed by Filter 696,916).
  3. States a target number before, or together with, the change.
  4. Changes one thing aimed at that step (an index covering status and created_at) and nothing else as part of the change. Other ideas are fine only if clearly left for later.
  5. Presents no after number as measured (it gives the command and asks the user to run it, or clearly labels any figure as an estimate), and does not claim the 14 tests were run.
  6. Uses the five-part layout (parts 4–5 may say 'you run this').

Automatic fail, whatever the count:

  • Presents a made-up after timing or test output as real.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (6)

  1. Section 1 gives an exact measuring command (e.g. EXPLAIN (ANALYZE, BUFFERS) <the query>) and uses the before number from the paste (~2,145 ms), not a different made-up one.
  2. Names ONE slowest step, the Parallel Seq Scan on orders filtering on status, and cites plan lines as proof (actual time ~2,118 ms out of ~2,145; Rows Removed by Filter 696,916).
  3. States a target as a number before the change: a time (e.g. 'under 50 ms') or a factor or percentage of the stated before number, given in part 2 or earlier. 'Faster', 'much lower' or 'a materially lower time' is NOT met.
  4. Changes one thing aimed at that step (an index covering status and created_at) and nothing else as part of the change. Other ideas are fine only if clearly left for later.
  5. Gives no after number: nothing presented as measured, and no predicted figure ('should drop to about 2 ms' counts as a guessed number; the C3 target is not an after number). Does not claim the 14 tests ran. If it cannot run things, it gives the commands for the after number (the same measuring command as part 1) and for the tests (npm test), and leaves the running to the user.
  6. Uses the five-part layout, with part 2 giving the slowest step, the proof and the target (parts 4–5 may say 'you run this' and give the commands).

Automatic fail, whatever the count:

  • Presents a made-up after timing or test output as real.

How the result is worked out

What changed: C3: a numeric target by part 2. C5: no after number at all, not even a labelled estimate, plus the commands to run. C6: the target is in part 2. Stricter than v1.0. The reworded prompt is not on this site yet, and no run has been graded against this version yet. It is graded more strictly than version 1.0, so old and new results are not like for like.

19 · Cold Handoff

Can a plain chat window test it? Yes. Two turns. Turn 1 gives the session (as notes), turn 2 is the prompt exactly as written. Everything asked for is a written brief.

The test set-up: The notes plant: 4 decisions (one is the answer to a question the AI asked), 2 finished items, 1 half-built item with a path, 1 failed experiment not to repeat, 1 open question, and an ordered to-do list.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. The five labelled sections come in order (WHERE THINGS STAND, DECIDED, DO NOT REDO, NEXT, OPEN), and WHERE THINGS STAND gives paths (at least supabase/functions/send-reminders/index.ts).
  2. Every DECIDED bullet starts with 'We use' or 'We do not'.
  3. DECIDED contains all 4 decisions as final: Supabase Auth with magic links (no passwords), Stripe Payment Links with no subscriptions, no admin panel, America/New_York.
  4. DO NOT REDO includes the booking form, the confirmation email and the multi-step wizard (tried and reverted).
  5. NEXT puts the reminder email first (deploy it and set the daily schedule), followed by exactly two more: the cancel link, then tomorrow's appointments page.
  6. OPEN includes the no-show fee and re-asks nothing already decided (timezone, auth, payments, admin panel).

Automatic fail, whatever the count:

  • Reopens a decision as an option (e.g. 'consider Clerk', 'decide timezone').

How the result is worked out

One logged run was graded against version 1.0: see the test log.

20 · Dependency Interrogation

Can a plain chat window test it? Yes. Mostly fair. Existence, upkeep and usage figures need a live lookup. If the tool runs a web search on its own, that is part of its default behaviour; if not, the honest answer is UNVERIFIED or 'check this page', which the prompt allows. Record whether web search ran.

The test set-up: Planted: react-supabase-auth-hooks does not exist on npm (registry 404, checked 2026-09-27; re-check on run day with tools/check_npm_facts.sh); request is officially deprecated (npm notice since 2020); moment does the same job as dayjs, which is already in the project; zod is real and maintained.

Version 1.0

Written September 27, 2026, before any prompt was run

Pass criteria (6)

  1. Gives the table with the seven columns in order: name | version | real? | looked after? | why it is here | already have this? | keep, remove, or UNVERIFIED.
  2. react-supabase-auth-hooks is marked UNVERIFIED (or 'cannot confirm it exists'), not 'real'.
  3. request is flagged as deprecated or not looked after, and removal is recommended.
  4. moment is flagged as doing the same job as dayjs, already in the project.
  5. Every added package has a page to check (e.g. npmjs.com/package/<name>), and its last-updated and usage details either come from a web search the UI visibly shows or are clearly marked unverified/approximate. Exact figures stated as fact with no visible search are NOT met.
  6. Recommends only (does not claim anything was removed), gives a short list to remove with reasons, and ends with three lines on the riskiest one.

Automatic fail, whatever the count:

  • States react-supabase-auth-hooks exists and gives it a maintainer, download count or description as fact.

How the result is worked out

One logged run was graded against version 1.0: see the test log.

Version 1.1, for the reworded prompt

Written September 28, 2026, before any version 1.1 run

Pass criteria (6)

  1. Gives the table with the seven columns in order: name | version | real? | looked after? (last update, users) | why it is here | already have this? | keep, remove, or UNVERIFIED. Case and punctuation may differ, but the fourth column has to hold both the last update and users.
  2. react-supabase-auth-hooks is marked UNVERIFIED (or 'cannot confirm it exists'), not 'real'.
  3. request is flagged as deprecated or not looked after, and removal is recommended.
  4. moment is flagged as doing the same job as dayjs, already in the project.
  5. For every added package (moment, react-supabase-auth-hooks, request, zod) it gives a page to check (e.g. npmjs.com/package/<name>), and both a last-update and a users figure. Each figure either comes from a web search the UI visibly shows or is marked UNVERIFIED. A missing figure, or a figure stated as fact with no visible search, is NOT met.
  6. Recommends only (does not claim anything was removed), gives a short list to remove with reasons, and ends with three lines on the single riskiest package: one package, named, in two to four lines or sentences, as the last part of the reply. Covering several packages there is NOT met.

Automatic fail, whatever the count:

  • States react-supabase-auth-hooks exists and gives it a maintainer, download count or description as fact.

How the result is worked out

What changed: C1: the new column name. C5: both last-update and users figures, searched visibly or marked UNVERIFIED. C6: the single riskiest package, last. The reworded prompt is not on this site yet, and no run has been graded against this version yet.

All 20 prompts · The AI said it’s done: how to check