# aite plan (draft 1, 2026-10-09) aite = AI Integrated Test Environment. An AI writes small readable scripts that drive a real browser; when one fails, a human can tell in under a minute whether the app or the script is wrong. Status: **plan only, nothing built.** Draft 1 turns draft 0 into a build plan: decisions with defaults, a design with concrete interfaces, phases with checklists and acceptance tests. Anything marked **(assumed)** is a default I chose; change it before the phase that depends on it starts. Contents 1. Principles 2. What BITE gives us, what does not carry over 3. Decisions (with defaults) 4. Design 5. Repo layout 6. Phases and checklist 7. How the AI writes and repairs scripts 8. How aite itself is tested 9. Security and accounts 10. Risks 11. Needs from Alex 12. Out of scope The plan is split across files. This one is the overview; the details live beside it. **Where this file and `docs/design/*.md` disagree, the design file wins** (and the task list in TASKS.md wins over both for scope and acceptance): | File | What it holds | |---|---| | `docs/decisions.md` | D1 onward, each with status, the phase that needs it, and alternatives | | `docs/design/script-format.md` | the grammar, every command and target, verify/wait forms, timing rules, error codes E001 to E007 (parser and runner), an example | | `docs/design/protocol.md` | the runner and extension websocket protocol: handshake, messages, failure reasons, keep-alive, delivery rules, security | | `docs/design/data-formats.md` | element descriptor and lock file, `run.json`, verdicts, repair requests, `aite.config.json` | | `docs/design/locator-repair.md` | resolution order, scoring weights, outcomes (found, repaired, drifted, not-found, ambiguous), rules | | `docs/journeys.md` | the mallbucks journeys the first scripts cover (send, gift, shop) with expected texts | | `docs/TASKS.md` | the tasks (50 after the QA review) with acceptance criteria, test tiers, test cases, dependencies and labels; GitLab holds the live status | | `docs/test-plan.md` | the QA test plan: risk tiers, cases with oracles, manual QA call-outs, gates, and the post-review task changes (section 6) | | `docs/adversarial-review.md` | the independent review of this plan (20 findings, verified); where it conflicts with this file, the task list in TASKS.md and the changes in test-plan.md section 6 win | --- ## 1. Principles 1. **A failure must be confirmable by a person fast.** Every step carries its intent in words, so a failure reads as "wanted X, found Y", with a screenshot, not as a selector error. 2. **Real session, real browser.** Scripts run in a signed-in browser profile, so flows that need a genuine session (Bluesky sign-in, the posting permission, passkey wallets, popups) are tested for real, by the AI, without waiting for a human tester. 3. **Scripts are plain text in git.** Reviewable, diffable, written and repaired by the AI, readable by Qian. 4. **Small command vocabulary.** If a step cannot be said in a few words, the script is too clever. 5. **Never fake a pass.** A step that cannot find its element fails; waiting is explicit; no silent retries that hide flakiness (retries are counted and shown). 6. **Reuse ideas, not the stack.** BITE's design is worth keeping; its code (Closure, App Engine, Manifest V2) is not. ## 2. What BITE gives us, what does not carry over Read from `source-archive-bite.zip` (unpacked outside the repo; nothing was run; the archive is left out of the published repository, see README). License: Apache 2.0 (`COPYING`). **Keep (as ideas, rewritten in TypeScript):** - A tiny command language: `click doubleClick drag enter type select change submit input find verify verifyNot wait sleep mouse move comparePosition changeUrl redirectTo run call validateBlock replaceHtml` (`tools/rpf/extension/src/libs/codegenerator.js`). - Element descriptors (XPath plus the element's attributes) instead of a single selector, and a locator-repair step (`locatorsupdater.js`, `getactioninfo.js`, `common/` ElementDescriptor). - Record then edit: `recordmanager.js`, `eventsmanager.js` turn live clicks into commands, which a person then tidies. - Playback with waits and retries (`playbackmanager.js`, 1,700 lines) and an action layer built on Selenium's browser atoms (`bothelper.js`, `actionshelper.js`). **Drop:** - The extension shell: Manifest V2 (Chrome no longer loads it), uses removed APIs (`chrome.extension.sendRequest`, 17 uses of `executeScript`, 5 of `eval`; V3 forbids `eval` and remote code). Rewrite for Manifest V3. - The server: Python 2.7 on Google App Engine. Replace with a small Node process. - Closure Library, the compiler, GData and App Engine clients, the Ace editor: all of `deps/` (89 MB). - Bug tracking, crawlers and the Google-specific parts of the server. ## 3. Decisions (with defaults) | # | Decision | Default **(assumed)** | Why / what would change it | |---|---|---|---| | D1 | Script format | A small line-based format for steps (section 4.1), plus optional TypeScript for custom checks | Easiest for people to read and for the AI to write and diff. Cost: a parser (about 300 lines). Alternative: TypeScript functions only. | | D2 | Driver | MV3 Chrome extension, one runner interface so a Playwright backend can be added later | Alex's preference. An extension cannot answer an OS prompt (Touch ID) or run headless in CI; Playwright with a virtual passkey can. | | D3 | Language | TypeScript, no UI framework, esbuild for the bundle | Small, typed, no Closure. | | D4 | Where it runs | Headed Chrome with a persistent profile on `balrog`, started by a script; also runnable on a laptop | Real session needs a real profile; headless comes later with Playwright. | | D5 | Bridge | A local Node process, websocket on `127.0.0.1` only, a per-run token | Lets Claude start a run and read results; never exposed to the network. | | D6 | Results | JSON + screenshots on disk, rendered as one static HTML report; verdict buttons live in a small local page | Keeps v1 simple; a GitLab artifact can come later. | | D7 | Repo | This repo (`root/aite`). BITE stays as an archive/reference; new code lives beside it | | | D8 | Large files | Decided 2026-10-09: leave the two zips in place; keep a short `reference/` of files read | 84 MB sits in history already. | ## 4. Design ``` Claude (AI) --writes--> scripts/*.aite (git) | | | start run, read result v v +---------+ websocket (127.0.0.1) +----------------------+ bridge (Node) <---------> | runner | <------------------------> | MV3 extension | | +---------+ | service worker | | | content scripts | v +----------+-----------+ results// (json, png, report.html) | acts on v real signed-in Chrome profile ``` ### 4.1 Script format (sketch, final syntax is a Phase 2 task) One file per flow, `scripts//.aite`. Plain text; `#` starts a comment. ``` test "send 1 MALLBUCKS to a handle" covers J-09 J-13 start https://mallbucks.malldao.xyz/ needs session test-qian step "remember the starting balance" capture before css:".balance" step "find the recipient" click text:"Send" type placeholder:"Search" "alex" click text~:"@distributed.camp" step "review shows the right amount" type label:"Amount" "1" click role:button name:"Send" verify text:"1 MALLBUCKS" within:"Review send" step "balance goes down by one within 5 seconds" verify css:".balance" number:changed-by from:before equals:-1 timeout:5s ``` - **Targets** are descriptors, written compactly: `text:`, `role:`+`name:`, `label:`, `placeholder:`, `testid:`, `css:`, `xpath:`. The runner expands each to a full descriptor (section 4.2) when it first resolves it. - **Commands (v1):** `start goto click doubleClick type press select check uncheck verify verifyNot wait sleep capture set drag switchTab closeTab screenshot call run` (the full table is in `design/script-format.md`, which wins). `call` runs a TypeScript custom check; `run` includes another script. - **Waits are explicit** (`wait`, a `timeout:` modifier on a command, or a `timeout` line in the file header). No hidden global sleeps. - **Every step has a name (the intent).** A script without step names does not parse. ### 4.2 Element descriptor When a target resolves, the runner stores a descriptor beside the script (`scripts//.aite.lock.json`), so the next run can find the element even if the text changed: ``` { "text": "Send", "role": "button", "tag": "button", "attrs": { "aria-label": "...", "data-testid": null }, "xpath": "/html/body/.../button[2]", "css": "...", "near": { "heading": "Send money" }, "seen": "2026-10-09T13:00:00Z" } # sketch; design/data-formats.md has the full shape and wins ``` Repair order when the first locator fails: other attributes of the descriptor, then text similarity, then position `near`; if a candidate is found with confidence below a threshold the step fails as **"locator drifted"** with the proposed element outlined, instead of passing. ### 4.3 Messages (runner <-> extension), v1 JSON over the websocket, request/response with an id: ``` { "id": 7, "cmd": "resolve", "target": { "text": "Send" }, "timeoutMs": 5000 } -> { "id": 7, "ok": true, "descriptor": {...} } { "id": 8, "cmd": "act", "action": "click", "descriptor": {...} } -> { "id": 8, "ok": true } { "id": 9, "cmd": "verify", "kind": "text", "value": "1 MALLBUCKS", "within": "..." } -> { "id": 9, "ok": true, "found": "1 MALLBUCKS" } ``` Failures carry `{ "ok": false, "reason": , "detail": {...} }` (the design file lists all twelve and wins). ### 4.4 Result schema `results//run.json`: the shape is defined in `design/data-formats.md` (which wins): tests contain steps, steps contain commands, each with its status, reason, expected, found, timing, retries and screenshot path. The static report is generated from it. ### 4.5 The confirm loop (the human part) A failed step renders as: ``` x step "review shows the right amount" (test: send 1 MALLBUCKS to a handle, covers J-13) wanted: verify text "1 MALLBUCKS" within "Review send" found: "1 MUCK" [screenshot with the "Review send" box outlined] ( App is broken ) ( Script is out of date ) ( Flaky, retry ) ``` - **App is broken** files a GitLab issue in the right project with the screenshot, the step, and the run link. - **Script is out of date** hands the failure to the AI, which proposes a script diff for the human to approve. - **Flaky** reruns the whole test three times from a clean pre-state and records the pass rate (a step with side effects is never repeated alone). ## 5. Repo layout ``` aite/ README.md docs/PLAN.md (this file) NOTICE credit to Google's BITE (Apache 2.0) for derived ideas/code LICENSE Apache 2.0 (matches BITE) reference/ the few BITE files worth re-reading (copied, not built) extension/ MV3 extension: manifest.json, src/{background,content,locator,actions}.ts runner/ script parser, step engine, result writer (Node/TypeScript) bridge/ local websocket server + CLI (`aite run scripts/...`) report/ static report generator, confirm page scripts/ the test scripts (mallbucks/, fixtures/) fixtures/ tiny static pages used to test aite itself (including a deliberately broken one) docs/ ``` ## 6. Phases and checklist Each phase ends with something running and a check a person can do. Do one phase at a time. ### Phase 0: housekeeping (half a day) - [x] D8 decided 2026-10-09: leave the zips in place (recorded in `docs/decisions.md`). - [ ] Add `LICENSE` (Apache 2.0) and `NOTICE` crediting BITE. - [ ] Copy only the files named in section 2 into `reference/` (about 10 files, under 1 MB) with a note of their original path. - [ ] Create the TypeScript project skeleton: `package.json`, `tsconfig.json`, esbuild script, a unit test runner (vitest). - [ ] GitLab CI: type check + unit tests on every push (the stores repo lacked this and its tests rotted). - **Done when:** `npm test` passes in CI on an empty skeleton, and a deliberately failing test turns the pipeline red. - The checklists in this file are summaries. `docs/TASKS.md` has the full task list per phase (50 tasks, including the spikes, hardening and test tasks) and wins on scope and acceptance. ### Phase 1: walking skeleton (1 to 2 days) - [ ] MV3 extension that loads in current Chrome (`chrome://extensions`, developer mode) and shows its status. - [ ] Bridge + websocket on `127.0.0.1` with a per-run token; extension connects from its service worker (and reconnects: service workers sleep). - [ ] Two commands end to end: `click` and `verify text`, against `fixtures/hello.html`. - [ ] Screenshots, taken by the runner over a private debugging pipe (the extension cannot: see protocol.md), saved in `results//shots/`. - **Done when:** `aite run fixtures/hello.aite` clicks a button on the fixture page and reports a pass with a screenshot. ### Phase 2: the script language (2 days) - [ ] Parser for the format in 4.1 with good errors (line numbers, "step names are required"). - [ ] Step engine: sequential steps, explicit `wait`/`timeout`, step status, retry counting. - [ ] Full v1 command set from 4.1 implemented in the extension's content script. - [ ] Unit tests for every command against fixture pages; a parser test table. - [ ] Fixture pages for every command and a CI self-check job that loads the extension and runs them (TASKS.md P2-4). - **Done when:** the whole command set passes on fixtures, and a malformed script gives a one-line error pointing to its line. ### Phase 3: element descriptors and repair (2 days) - [ ] Descriptor capture on first resolve, saved to the `.lock.json` file next to the script. - [ ] Repair order from 4.2, with a confidence score and a `drifted` failure. - [ ] Fixture `fixtures/drift.html`: a page whose button text changes between runs; the test proves repair works and that low confidence fails instead of passing. - **Done when:** a renamed button is found by repair and reported as "repaired (text changed)", not silently. ### Phase 4: reporting and the confirm loop (2 days) - [ ] `run.json` writer and the static `report.html` (steps, intents, expected/found, screenshots, outlines). - [ ] The three verdicts: "App is broken" creates a GitLab issue (title, screenshot, step, repro command); "Script is out of date" writes a repair request file the AI picks up; "Flaky" reruns the whole test three times from a clean pre-state. - [ ] `fixtures/broken.html`: a deliberately broken version of the fixture; a run against it fails with a readable report. - **Done when:** a person, given only the report, correctly says "app" or "script" for the broken and the drifted fixtures in under a minute each. ### Phase 5: real session and test accounts (1 to 2 days, needs Alex) - [ ] Persistent Chrome profile for the runner on `balrog` (`--user-data-dir`), loading the extension unpacked. - [ ] Dedicated Bluesky test accounts and a wallet; the profile signs in once by hand (OAuth consent screens are web pages, fine), then the saved session is reused. - [ ] A `session-check` script that fails loudly when the saved session has expired, with instructions. - [ ] Documented procedure for renewing a session. - **Done when:** a script runs signed in as the test account with no human present, and `session-check` correctly fails when the saved session is deliberately cleared. ### Phase 6: first real milestone, mallbucks journeys (3 to 4 days) - [ ] `scripts/mallbucks/send.aite` (J-09, J-13): search, send 1 MALLBUCKS, check the review wording and the balance change. - [ ] `scripts/mallbucks/gift.aite` (J-36, J-37, J-39, J-40): create a gift, Share to Bluesky, check the post exists with a tappable link, open the claim page. - [ ] `scripts/mallbucks/shop.aite` (J-24 to J-28): open a shop, buy, pay in the wallet pop-up (a second window), download the file. - [ ] A negative run: point the scripts at a deliberately broken build (a text change the app really makes, for example a different amount wording; see `docs/journeys.md`) and confirm the report catches it. - **Done when:** all three pass on production, the negative run fails at the right step, and Qian confirms the report is clearer than a written test log. ### Phase 7: authoring workflow and scheduling (2 days) - [ ] Record mode (BITE's idea): record clicks in the live page to a draft script, with names left for the AI/human to fill in. - [ ] `aite author`: Claude writes a script from a journey test description, runs it, fixes it until it passes or reports why it cannot. - [ ] Run on a schedule on `balrog` (after each deploy and nightly); failures open or update one GitLab issue per script, not a flood. - **Done when:** a new journey from `docs/qa-journey-test-plan.md` (in the mallbucks repo) becomes a passing script with no hand-editing beyond approving it. ## 7. How the AI writes and repairs scripts 1. **Author:** given a journey (steps in words), write the script with a named step for each sentence; use `text:` and `role:` targets first (they survive restyling), `testid:` where the app has one; run it; fix until green or report the blocker. 2. **Ask the app for stable hooks:** when a target is fragile, propose adding a `data-testid` in the app repo rather than a long selector. 3. **Repair:** on a "script is out of date" verdict, read the failing step, the screenshot and the current page; change the smallest thing (a text, a target); rerun that test only; show the diff for approval. Never weaken a `verify` to make it pass. 4. **Never invent expected values.** Expected text comes from the product spec or QA plan, not from whatever the page currently says. ## 8. How aite itself is tested - Unit tests: parser, step engine, descriptor matching, report generation (vitest). - Fixture pages cover every command, plus `drift.html` and `broken.html`. - A "self-check" CI job that loads the extension in the pinned Chrome and runs the fixture scripts through `aite run` itself (decisions.md D13: not Playwright, so the launch path under test is the real one), so a broken extension build is caught before it reaches `balrog`. Each fixture script carries a `# expect:` line; the job runs each 20 times, and a second run with the extension broken on purpose must fail. ## 9. Security and accounts - The bridge listens on `127.0.0.1` only, with a per-run token; the bridge closes any connection that lacks it. - Test accounts are dedicated throwaways with little value (the wallet is on Monad testnet). Their sessions live in the Chrome profile on `balrog`, never in git; the profile directory is mode 0700 and excluded from backups that leave the machine. - Scripts never contain secrets. A `needs session ` line names a profile; the runner fails if it is missing. - Screenshots can show addresses and handles. Reports stay on the machine; sharing is an explicit step. - The extension is loaded unpacked into one profile; it is not published. ## 10. Risks | Risk | Mitigation | |---|---| | MV3 service worker sleeps and drops the websocket | Reconnect with backoff; keep the websocket alive with a 15 s ping (a `chrome.alarms` heartbeat every 30 s is only the backup, see protocol.md); prove it in Phase 1 before building more | | Locator drift makes the suite noisy | Repair loop in Phase 3, built early; drift is a distinct failure kind | | OS-level prompts (Touch ID) cannot be answered by an extension | Use a test wallet created with a passphrase backup for the extension runner; passkey flows go to a Playwright backend later (D2) | | The pay pop-up is a second window | Phase 6 covers `switchTab`; the extension already sees all tabs | | Real accounts expire or get rate limited | `session-check` fails loudly; runs are serial and few | | AI "fixes" a failing test by loosening it | Rule 7.3 and a CI check that flags removed `verify` lines in script diffs | ## 11. Needs from Alex 1. Approve or change D1 to D7 (the defaults above stand if you say nothing). D8 is decided; D9 and D10 are in `docs/decisions.md`. 2. Create or name the Bluesky test accounts (Phase 5) and say where their sessions may live. 3. Confirm `balrog` can run a headed Chrome (a display, or a virtual one such as Xvfb) for the runner. ## 12. Out of scope (for now) Bug tracking and crawling (BITE server features); a hosted multi-user dashboard; mobile browsers; load or performance testing; visual diffing of whole pages; headless CI for the signed-in flows (comes with the Playwright backend).