How Would You Know?

Instructor: the room controls for this activity, and the room display for the projector.

Friday, October 23. Your agent tells you something is done. How would you know? Not because it said so, and not because you clicked once and it looked fine, but in a way you’d trust next week, after the agent has changed ten other things. That’s today’s question. Project two ships next Friday, October 30, at an address the person it’s for can open, and Wednesday, October 28 is for finishing it, so today is about the week in between: the changes you’ll ask for, and knowing each one didn’t break something else.

First thing: type the code on the projector into the box at the top of this page. That’s how the room knows you’re here. When the page asks to show notifications, allow it: you’ll spend most of today in your agent, and the page will tell you when the room needs you to look up.

Why Today

On Wednesday the person your project is for sat at your laptop and found things your agent hadn’t. That works once. It doesn’t work every time you change something, and between now and next Friday you’ll change a lot. So the question isn’t whether your project works right now. It’s how you’d find out, quickly and without anyone’s help, whether it still works after the next change.

You’ve practiced looking for yourself all semester. Today is about turning that look into something that happens every time:

Never accept “done” without looking. “That should work now” is sincere and sometimes wrong. Thirty seconds a turn.

Here’s the catch, and it’s worth saying out loud to your agent. A check your agent builds in the same conversation where it built the project tends to share its blind spots: if it misunderstood what you wanted, its test will misunderstand the same way. So part of today is thinking about checks that don’t depend on the agent being right: a person, a fresh agent that knows only what a stranger would, or a different AI altogether.

Your agent knows what today is for. Every word of what the course adds to it is below. Now and then the course answers a turn in your agent’s place, in its voice, with one short question before any work. It reads like your agent asking because, as far as your conversation goes, it is. Answer in your own words and carry on; your next message always goes through.

How Today Works

Geoff moves the room from one to the next. Nobody is paired today: the brainstorm and the building are yours alone, and the room comes together once, in the go-around. You’ll get a notification when the go-around starts and when it’s time for feedback; the rest of the time, stay in your agent.

Brainstorm

Twenty minutes with your agent and one question: how could you know your project works without taking anyone’s word for it, including the agent’s? Ask for as many ways as fit, at every level, from one small piece to the whole thing in a stranger’s hands. For each one, ask what it would catch and what it wouldn’t. Ideas, not building yet.

There’s no list to start from, on purpose. We want to see what you come up with. While you talk, the course reads the ideas out of your conversation and puts them on the projector, grouped by kind, without names. You don’t have to do anything for that: just talk to your agent.

Go Around

When the go-around starts, look up. Each of you, under a minute: the one way of checking you’d build and why. While you speak, the projector shows your name and the ideas the course read from your conversation, so the room can see what you came up with. If you’d rather say a different one than what’s on the screen, say it; the screen is a prompt, not a script.

When everyone has spoken, the projector puts the room’s ideas beside the course’s own list, and marks the ones nobody named. That’s not a scold. Some of them are hard to think of until you’ve seen someone do them.

Build It

Back to your agent. Pick one: yours, one you heard, or one nobody named. Make it real so it runs from now on, not once: written down somewhere it lives with your project, where you or your agent can run it again next week. Then run it, and say what it caught, in plain words. If it caught nothing, say that too, and ask yourself whether it could have.

Your agent knows the room’s list now, merged with the course’s, so you can ask it about any of them. Two things worth knowing. Some checks need somewhere to send what they collect, like errors or what people do on your site. The course won’t run that for you, ever: part of today is finding a tool that does it, or building it, which is a lot easier now than it used to be. And if you want a checker that doesn’t share your agent’s blind spots, a different provider’s AI is a better bet than another copy of the same one, though not a guarantee. Your agent only reaches the course’s models, so that would mean finding your own access.

If You Want an AI to Use Your Site

We tried two ways before class, with the same setup you have (2026-10-21). For your agent to read:

• A fresh agent as a stranger works as you are. Open a terminal in a new, empty folder, not your project’s, start codex there, and tell it only what a stranger knows: the address (your site running on your laptop, or its real address) and one sentence of what it’s for. Ask it to try the site and say what it did, what it expected, and what it saw. Ours read the page, ran its code, and found the bug we planted. It can’t see the page drawn, so it won’t catch layout problems, and it shouldn’t claim to.

• An AI driving a real browser works through Playwright, not Codex’s own browser. Codex’s built-in browser needs the desktop app or a ChatGPT sign-in, which this course’s setup doesn’t use. What worked is Playwright’s browser server, added to ~/.codex/config.toml, with Chrome installed and Node’s npx available:

  [mcp_servers.playwright]
  command = "npx"
  args = ["-y", "@playwright/mcp@0.0.83", "--browser", "chrome", "--isolated", "--headless"]
  startup_timeout_sec = 60
  default_tools_approval_mode = "approve"

  The last line matters: without it every browser step fails with “requires approval”, and launching Chrome from inside the agent’s sandbox crashes. On Windows, --browser msedge should work for Edge; we haven’t tried it. Restart codex after changing the file.

What Your Agent Knows Today

Everything the course adds to your agent’s context today is here, word for word. The Brainstorm note goes in at the end of every request your agent makes until the go-around; the Go Around note while the room goes around; the Build It note from Build It on, for the rest of the afternoon. The Build It note carries the room’s ideas: before any are in, the course’s list alone, and once they are, the kinds the room named with how many people named each, then what the course adds. The counts below are an example; yours are the room’s. Your agent never sees any of these as coming from you. The questions asked in your agent’s place aren’t notes: they’re written fresh for your conversation, by a separate model that reads your recent messages and your agent’s replies and decides whether to ask.

Brainstorm

Course note (context for you; the student’s request comes first): Today’s class is How Would You Know? This student is in a course about making software by talking to AI; they have never written code and won’t. Their project, something small for another student in the room, ships Friday 10-30 at an address the person it is for can open, and Wednesday 10-28 is for finishing it. The class practices the third of the course’s eight practices: never accept “done” without looking. Today’s question is how they would know it works without taking anyone’s word for it, yours included. The course would like you to help them think of as many ways as fit, at every level, from the smallest piece to the whole thing in a stranger’s hands, and to say plainly what each would and wouldn’t catch, including what a checker built in this same conversation would miss. Ideas, not building yet: the building comes later in the class. Now and then the course asks one short question in your place; where the conversation shows one, it was yours, and their answer is the brief.

Go Around

Course note (context for you; the student’s request comes first): Today’s class is How Would You Know? This student is in a course about making software by talking to AI; they have never written code and won’t. Their project, something small for another student in the room, ships Friday 10-30 at an address the person it is for can open, and Wednesday 10-28 is for finishing it. The room is going around aloud now, each student saying the one way of checking they would build and why, in under a minute. If they ask, the course would like you to help them say theirs in a sentence; anything longer can wait for the building, which comes next. Now and then the course asks one short question in your place; where the conversation shows one, it was yours, and their answer is the brief.

Build It, before the room’s ideas are in

Course note (context for you; the student’s request comes first): Today’s class is How Would You Know? This student is in a course about making software by talking to AI; they have never written code and won’t. Their project ships Friday 10-30; Wednesday 10-28 is for finishing it. They pick one way of checking and build it so it keeps running: written where it lives, run once now, what it caught said plainly. The room’s ideas aren’t in yet, so here are the course’s: Tests the agent writes and runs: unit tests for the pieces, end-to-end tests for whole flows; An AI driving the app in a browser, the way a user would; A simulator that exercises the core and shows how the system behaves; Instrumentation that catches every error; Logging that shows what people do and where they fail; Asking the people it’s for, through AI and other ways; Agents checking agents, ideally a different model or provider than the one that built it. Another provider’s checker is a better bet, not a guarantee, and needs the student’s own access. The course runs no services for projects. Before an AI uses the site, what has worked here is at https://www.conversationalprogramming.org/activities/how-would-you-know/#tested. Now and then the course asks one short question in your place; where the conversation shows one, it was yours, and their answer is the brief.

Build It, with the room’s ideas

Course note (context for you; the student’s request comes first): Today’s class is How Would You Know? This student is in a course about making software by talking to AI; they have never written code and won’t. Their project ships Friday 10-30; Wednesday 10-28 is for finishing it. They pick one way of checking and build it so it keeps running: written where it lives, run once now, what it caught said plainly. The room’s ideas, with how many named each: Tests the agent writes and runs: unit tests for the pieces, end-to-end tests for whole flows (11); Asking the people it’s for, through AI and other ways (7); An AI driving the app in a browser, the way a user would (4). Ideas the course adds: A simulator that exercises the core and shows how the system behaves; Instrumentation that catches every error; Logging that shows what people do and where they fail; Agents checking agents, ideally a different model or provider than the one that built it. Another provider’s checker is a better bet, not a guarantee, and needs the student’s own access. The course runs no services for projects. Before an AI uses the site, what has worked here is at https://www.conversationalprogramming.org/activities/how-would-you-know/#tested. Now and then the course asks one short question in your place; where the conversation shows one, it was yours, and their answer is the brief.

While the Feedback section is up, whichever note is active ends with one more sentence: “The class is ending: the course would like to hear how it went, at https://www.conversationalprogramming.org/feedback/, a short conversation.” No sentence about a partner today, and no “Right now” lines.

Before You Go

Tell the course how today went. The last ten minutes are for it, and it’s the main way this class gets better while you’re still in it.

The ideas the course read from your conversation are kept with your name on them, and Geoff reads them afterwards, because the way this class learns to teach is by reading them. On the projector they had no names, except your own during your turn in the go-around.

If you’d rather see it than read it, there’s a walkthrough of the whole day, rehearsed end to end on students’ screens and mine: checking in with the projector’s code, the idea wall filling from three conversations, the go-around one student at a time, and the room’s list reaching an agent for the building.