# Start here: a Brodie coach on LiveKit

What the starter coach is, how to set it up, your first call on the phone page, and where to take it beyond reps.

You are a team with a LiveKit Cloud project of your own (cloud.livekit.io, the free tier is enough), a laptop running
macOS or Linux (no Windows), Python 3.12 with uv for the coach, and a phone (or a Mac with Safari) to stand in front of.
The caller joins a room of your project through our static phone page and your worker joins the same room (section 3):
that is the main path for every team. Three teams also have one of the three Chert FaceTime lines; for them the same
worker answers a real FaceTime call (section 5, with the Chert CLI facts from docs.trychert.com/facetime-cli). This page
is the whole read-me; the event schema is in [/hackathon/events](/hackathon/events) and what you can build is in [/hackathon/build](/hackathon/build).

## 1. What you are getting, and what you change

A working real-time video coach in one Python process, one LiveKit participant, one audio track. It runs the same
whether the caller arrives through the phone page in a room of your own LiveKit project or through a Chert line (a
Chert room has exactly two seats, the caller's bridge and your agent, so there everything you add runs inside this
process; in your own room there is no such limit, but keep one audio publisher). Its eyes are
`facetime/perception.py`: MediaPipe pose on every sampled frame at 10 fps, a rep counter and a fault detector, turned
into a versioned stream of events (`rep`, `exercise_seen`, `fault`, `cue`, `visibility`, `gesture`, `probe`, `session`,
`status`, and `voice` for the model's turns with `--voice gemini`). Its fast mouth is the pre-rendered Brodie cue bank (`data/cues_audio_gemini`, 379 clips, Gemini TTS, voice Puck;
an Australian-direction rendering of the same strings, `data/cues_audio_gemini_au`, is optional via `CUE_AUDIO_DIR`):
counts, corrections, praise and announcements are heard about 0.2 s
after the decision (145 to 225 ms measured over the LiveKit SFU). Its conversation is Gemini Live in the same Puck
voice, gated to the gaps between sets (`--voice gemini`, `facetime/voice.py`, section 2c of [/hackathon/events](/hackathon/events));
the raw event stream is also there for your own layer (`@facetime.worker.consumer`, section 2 of that page).

You change three files in `starter/` (you write coaching, not perception):
- `starter/persona.md`: the system prompt of the conversational voice (who Brodie is, how it talks between sets).
- `starter/plan.txt`: the session script, in `--plan` syntax: `squats:10,pushups:8,jumping jacks:20,lunges:10` (27 names
  are accepted; these four count and are validated on a device, nine more count untested, the rest run as timed correction
  blocks: [/hackathon/build](/hackathon/build) section 1).
- `starter/handlers.py` (loaded by default whenever the file exists, with or without `--voice gemini`; `--handlers none`
  turns it off): `def on_event(event, say_now, ask_voice)`, called once per event with the event dict
  (every event carries `v`, `t`, `t_wall`, `seq`, `type` and, except `status`, `frame`). Two verbs with two latencies:
  `say_now(text)` plays a bank line through the fast path's priority slots, heard about 0.2 s later, only strings in
  the bank (`data/cues_audio_gemini/manifest.json`, keyed on the exact string); `ask_voice(prompt)` hands the LLM a
  prompt and is heard 1 to 2 s later; the shipped handler asks for one sentence at each `rest` and a warm close at `done`.
  Call `ask_voice` only when the session opens a slot: `session.state` is `rest` or `done`, a `gesture` `tpose`, or a
  caller's question (a visibility pause never opens the floor; a line asked during the `wait` before a set is cut by its
  first count). Never during `active`: the fast path owns the set, the first count of a set cuts any reply still
  playing, an ask queued during a set is dropped at the next set start or after 15 s, and `cue` events are the
  transcript of what the fast path already said, so do not repeat them.
  The same events are read raw with `@facetime.worker.consumer async def fn(job): ev = await job.events.get()`.

## 2. Setup

Ten minutes of typing, 25 to 45 minutes of waiting.
```
git clone https://github.com/seraphimlabs-dev/brodie-starter.git brodie-rt-coach && cd brodie-rt-coach
git checkout -b team-<your-team>   # work on a branch of your own
uv venv --python 3.12 .venv && uv pip install -r requirements.txt
mkdir -p models && for m in lite full heavy; do [ -f models/pose_landmarker_$m.task ] || curl -sSfL -o models/pose_landmarker_$m.task https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_$m/float16/latest/pose_landmarker_$m.task; done
```
`uv pip install` is the long step: torch, mediapipe, transformers, modal and livekit (on Linux torch pulls about 3 GB of
CUDA wheels; run `uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu` first to skip them),
then the tarballs and the pose models (about 120 MB); the three Chert teams also install Node and the Chert CLI and run
the `chert login` browser flow (section 5). No `uv` yet: `pip install uv` or `brew install uv`.
You do not need Modal or any cloud of ours: the worker runs on your laptop (a Mac processed 10.7 fps with pose at 10 ms), the `modal` package in requirements.txt is a dependency of our demo path and stays unused, and nothing here
calls our deployment. The two shared conveniences are the phone page (section 3), whose link `mint_token --url` prints is
served from our host but is a static page with the url and token of YOUR LiveKit project in it (opening
`facetime/phone_test.html` as a file in Safari on a Mac works the same), and the Gemini relay (section 4), which stands in
for a Google account.
The data folder is never in git. The starter bundle is three tarballs (`rtcoach_starter_part1_models_recordings`,
`part2_bank_au`, `part3_bank_puck`, dated 2026-10-09); unpack part 1 and part 3 at the repo root (part 2 is optional) and
everything lands under `data/`: `data/models/clf_pose.pt` and `trigger_lr.json`, the voice (`data/cues_audio_gemini/`,
plain Puck, 379 clips, part 3, the bank the demo uses), the optional Australian-direction bank (`data/cues_audio_gemini_au/`,
378 clips, part 2; select it with `CUE_AUDIO_DIR=data/cues_audio_gemini_au`), the sample recordings (section 6) and
`data/STARTER_BUNDLE_README.txt`. The player prefers the plain Puck bank; use it for both the bank and the conversation, since mixing voices sounds like two speakers. The thresholds, `cues.json` and `cue_prior.json` are in `src/rtcoach/` in git. Check:
```
ls data/models/clf_pose.pt data/models/trigger_lr.json data/cues_audio_gemini/manifest.json models/pose_landmarker_full.task   # plus data/cues_audio_gemini_au/manifest.json if you unpacked part 2
```
Before the first call, prove the install on a bundle recording (about 60 s; expect squats 9 of 10 on the 75 s timeout and the
first pushup at 137.8 s), then the contract test (PASS in about 30 s) and the offline voice tests (83 checks, no key needed):
```
python -m facetime.replay_local --video data/tmp/mac_handoff/crops/run3.mp4 --plan "squats:10,pushups:8,jumping jacks:20,lunges:10" --fast --events data/tmp/run3.events.jsonl
python scripts/test_perception.py --video data/tmp/mac_handoff/segments/run3_squats.mp4 --plan squats:10
python scripts/test_voice.py
```
Then read [/hackathon/build](/hackathon/build): what you can build on this and how.
Create `.env.local` in the repo folder (never commit it). The three LiveKit values come from one LiveKit Cloud project of
your own: create it at cloud.livekit.io (the free tier is enough), then Settings -> Keys gives the URL
(`wss://<project>.livekit.cloud`), the key and the secret:
```
LIVEKIT_URL=wss://<your-project>.livekit.cloud
LIVEKIT_API_KEY=<key>
LIVEKIT_API_SECRET=<secret>
AGENT_NAME=
GEMINI_API_KEY=<the team token on your card>
GEMINI_BASE_URL=https://james-72765--rtcoach-gemini.modal.run
```
`AGENT_NAME=` is set EMPTY on purpose: an empty name is automatic dispatch, the worker joins every new room of your
project, which is what the phone page needs (section 3). A named worker (the default `brodie-coach` when the variable is
missing) waits for a named dispatch and never joins a phone room; only the three Chert teams set
`AGENT_NAME=<the dispatch name on the card>`, and only for the line (section 5). The worker itself needs nothing but these
three LiveKit values and the bundle on disk; `GEMINI_API_KEY` and `GEMINI_BASE_URL` are for `--voice gemini` and for
re-rendering the bank, through our relay (section 4). You re-render the bank with `python -m facetime.cues_tts --backend
gemini` only if you add strings to `cues.json` (plain Puck into `data/cues_audio_gemini`, only the strings still missing;
about 4 minutes and $0.16 for the whole bank). For the optional Australian bank add `--style "(Australian accent, warm,
upbeat trainer) " --out-dir data/cues_audio_gemini_au` (the bracketed direction is not read aloud, a spoken prefix would be).

## 3. First call on the phone page (LiveKit)

The phone (or a Mac in Safari) joins a room of your LiveKit project through `facetime/phone_test.html`, publishes its camera
and plays the coach; your worker joins the same room by automatic dispatch. Nothing loads `.env.local` for you (the LiveKit
Agents CLI does not read it, and `mint_token` needs the key and secret exported), so export it in both terminals.

Terminal A, the coach:
```
cd brodie-rt-coach && source .venv/bin/activate && set -a && source .env.local && set +a
python -m facetime.worker dev                                   # the plan in starter/plan.txt (ships as squats:10,pushups:8,jumping jacks:20,lunges:10)
python -m facetime.worker --plan "squats:10,pushups:8" dev      # your plan
python -m facetime.worker --voice gemini dev                    # plus Gemini Live (Puck): talks only between sets, needs GEMINI_API_KEY and GEMINI_BASE_URL (section 4)
```
`AGENT_NAME=` empty in `.env.local` (or `--agent-name ""` before `dev`) is what makes the worker join your room; with a
dispatch name set it registers and then waits forever.

Terminal B, the link (same folder, same exports; a fresh room name every attempt):
```
cd brodie-rt-coach && source .venv/bin/activate && set -a && source .env.local && set +a
python -m facetime.mint_token --room team1-try1 --url
```
It prints THREE lines: the token, the query string, and the https link on our host,
`https://james-72765--rtcoach-webhook.modal.run/phone?url=...&token=...&room=...`. The THIRD line is what the phone opens:
AirDrop it, send it in Messages, or make a QR code of it. Add `&facing=user` for the front camera (the default is the
back camera) and `&autojoin=1` to join without a tap; `&agent=<identity>` changes which participant the page calls the
coach (below). The page is static: our host only serves the HTML, and the `url` and `token` in the link are your own
LiveKit project (the `room` in the link is informational, the token carries the room). On a Mac, open
`facetime/phone_test.html` as a file in Safari with the same query string: it works the same, because `file://` is a
secure context for the camera, which is also why the phone needs the https link and not http from your laptop. The page
loads `livekit-client` from jsdelivr, publishes one camera track named `facetime-remote-video` asking for 1280x720 at
30 fps (a phone held in portrait delivers 720x1280; the worker samples at 10 fps), plays every audio track it subscribes
to, and shows "Tap to enable audio" when the browser blocks autoplay: tap it.

Order matters: start Terminal A before the phone joins, and use a new room name per attempt. Automatic dispatch fires when
a room is CREATED (the phone's join creates it), so a room the phone opened while the worker was down, or a room reused from
an earlier attempt, never gets a job. The job ends 2 s after the phone leaves (`--leave-grace-s -1` keeps it; a page reload
within 2 s binds again). Prop the phone (section 7) and step in.

What it prints when it works. Terminal A: `[worker] agent ''; mode plan (...); audio cue player; data channel off; events ->
data/tmp/worker_events.jsonl`, `[worker] prewarm: loaded in 5.x s (pose full, counter v2, trigger
data/models/trigger_lr.json, ...)`, the LiveKit line `registered worker` (the notices `dev mode is deprecated ... use lk agent
dev` and `in-process auto-reload has been removed` are livekit-agents 1.8.5 talking; ignore them); when the phone joins
`[worker] job ...: joined room 'team1-try1'`, `[worker] published audio track coach-audio`, `[worker] participant joined:
chert-bridge`, `[worker] bound to 'chert-bridge' (call id n/a): video 'facetime-remote-video' 720x1280`, `[worker] first frame
720x1280`, `[cue] HH:MM:SS [announce] "Let's do squats"`, then `[event] rep squats 1 (agree 1.0, ...)` and `[cue] ... [count]
'One' decision->push N ms` per rep, and a `[status]` line every 5 s (expect processed about 10 fps, pose about 20 ms,
underruns 0). On the phone: "Let's do squats" from the speaker, then the counts.

The page's "Coach present" indicator is green ONLY for a participant whose identity equals `agent` exactly (default
`brodie-agent`). The worker under `dev` joins as `agent-<job id>`, so the indicator stays "no" on a working call; the real
signal is the page's "Coach audio: playing from agent-..." line, and the audio itself. To make it green, join one room
directly under that identity instead of waiting for a dispatch: `python -m facetime.worker connect --room team1-try1
--participant-identity brodie-agent` (`connect` is a LiveKit Agents CLI subcommand).

On this path the caller's participant is plain `chert-bridge` and there is no call id: `call_id` is null in your logs, and
your hand-in is keyed on the room name and the time of the call (section 10). In your own room there is no two-seat limit (a
second phone or a dashboard participant can join; the worker binds to the first non-agent participant that publishes video
and ignores other video), but keep one audio publisher.

`--voice gemini` on a call: on a FaceTime call over the Chert CLI the model session started 0.9 s after the worker bound to
the caller, the caller's words were transcribed but not logged, and the worker ended 252 ms after the hang-up. The first count
of a set cuts the voice, and the persona keeps it in English. It waits for a clip to end instead of playing under it, the caller's
audio reaches the model only when no Brodie audio has played for 0.3 s, the rest clock pauses while either side is talking
(up to 25 s), and only one Brodie line plays at a time (an ask waits while any Brodie audio plays); prompts are never read
aloud. These rules pass 61 offline and 63 live checks but have not been verified on a device yet, so listen for overlapping
voices on your first call.

## 13. Your own coach: beyond reps

The worker has two modes.

**Reps (as shipped).** `facetime/perception.py` watches the caller at 10 fps (pose, rep counter, fault detector, the session
script from `starter/plan.txt`), the fast path speaks counts and corrections from the bank about 0.2 s after the decision, and
the Gemini voice is gated to the gaps between sets. Everything in sections 1 to 12 is about this mode.

**Voice plus vision, perception off.** `--perception off` runs no pose model, no counter and no script: the worker still joins,
publishes its audio track, binds to the first caller that publishes video (or audio, when there is no video), starts the Gemini
Live session on that bind, and from then on the model is always allowed to talk (`--floor open`, the default here): the caller's
audio is always forwarded, `ask_voice` is spoken at once, nothing is ever cut. `--vision` hands the caller's video to the model at
about 1 fps (`--vision-fps`, at most 2): each frame goes out as a JPEG scaled by the plugin to fit 1024x1024 (a phone-page
720x1280 frame as 576x1024, a FaceTime 1280x720 frame as 1024x576), which is what
the Live API wants for a scene that changes slowly, like a kitchen. The plan file is appended to the persona as free text (a
recipe, a checklist), not parsed as exercises. Your `handlers.py` sees ONE synthetic `session` event, `{state: "open",
exercise: null, count: 0, mode: "voice"}`, when the model session is up (greet there), a `status` event every 5 s (frames
received, frames sent to the model, no pose fields) and the `voice` events; no `rep`, `cue`, `fault` or `visibility` events.

The meal-prep example in `starter/examples/meal_prep/` (a persona that names what it sees before it instructs, one step at a
time, safety first with knives and heat; a 15-minute garlic butter pasta with greens as numbered, timed steps; a handler that
greets on the session open event and a `later(seconds, prompt, ask_voice)` helper for timed nudges on the worker's event loop):
```
.venv/bin/python -m facetime.worker --perception off --vision --voice gemini --persona starter/examples/meal_prep/persona.md --plan-file starter/examples/meal_prep/recipe.md --handlers starter/examples/meal_prep/handlers.py dev
```
Tell the caller: prop the phone so the camera sees the counter and the stove (landscape, about an arm's length away, not pointed
at the ceiling), keep your hands in view when you cut or stir, and say "ready" or hold the board up to the camera when you want
the next step. The coach asks you to show the pan or the board when it is unsure.

What to expect. The first audio arrives about 1 s after the model decides to speak (measured 0.9 to 1.3 s to first audio on a
text turn with `gemini-3.8-live`). Video goes at 1 fps, so the model sees a step change (the pasta going in, the garlic
colouring) within a second or two; it only sees what the phone camera sees, so framing is everything. `[vision] forwarding video
to the model at 1 fps (1280x720)` is logged once on the first forwarded frame and the `[status]` line carries the count of frames
sent. The greeting comes from your handler on the session open event, about 1 to 2 s after the caller is bound.

Limits. There is no fast path for cooking: every word goes through the model (about 1 s), the bank `say_now` still plays the
strings that are in the bank (`data/cues_audio_gemini/manifest.json`: counts, "Nice work", "Rest for a moment" and the like), and the
arms-up probe and the T-pose gesture do nothing in this mode (no pose model runs). The model cannot read a thermometer or taste;
it works from what it sees at one frame per second and what the caller says. This mode has been tested offline only (fake
room, fake frames, a no-room model session with one pushed frame); it has not been on a call, phone page or FaceTime. Expect to tune the persona
on the first call: if the model narrates too much, tell it so in `persona.md`.

# What you can build on Brodie, and how

The skills a team can add to Brodie in a day, with the commands and the evidence loop for each one.

Read this after section 2 of [/hackathon/start](/hackathon/start) (the setup). Every command below runs from the repo root with the
virtualenv activated (`source .venv/bin/activate`, so `python` is `.venv/bin/python`) and the starter bundle unpacked under
`data/`. Nothing here needs a call until it says "on a call": a call is the phone page in a room of your own LiveKit
project (README section 3; a fresh room name per attempt, nothing to wait for) or, for the three teams that have one, the
Chert FaceTime line (README section 5; Offline for a few minutes after every call). Calls are for checking, recordings are
the inner loop. The event schema is [/hackathon/events](/hackathon/events).

## Brodie in five sentences

1. **Eyes.** `facetime/perception.py` runs MediaPipe pose on the caller's video at 10 fps, a rep counter (`src/rtcoach/repcount2.py`)
   and a fault classifier, and turns them into a stream of events: `rep`, `fault`, `cue`, `visibility`, `gesture`, `probe`,
   `exercise_seen`, `session`, `status` (plus `voice` for the model's turns). Your code reads events; it never touches frames.
2. **Fast mouth.** The bank: 377 pre-rendered strings (379 clips, Gemini TTS, voice Puck, `data/cues_audio_gemini/manifest.json`)
   pushed into the one audio track by the cue player about 0.2 s after a decision (145 to 225 ms measured). Counts, corrections,
   praise, announcements and your `say_now(text)` lines all come from the bank, and only strings in the bank can be said.
3. **Conversation.** Gemini Live (same Puck voice, `facetime/voice.py`) in the same process, mixed into the same track, allowed to
   speak only when the session opens a slot (the rest after a set, the wait before one, the end, a gesture, a caller's question
   during a rest). About 1 s from a prompt to the first audio, so it never counts.
4. **The session script.** `starter/plan.txt` (or `--plan`) is a list `name:count,...`: for each item the coach announces it, waits
   for movement (up to 3 s), counts to the target with corrections, says "That's ten, nice work", rests 5 s, moves on, and ends with
   "All done, great session"; a set that has not reached its target after 75 s is closed with the count reached (`--set-timeout`).
5. **One participant, one audio track.** Everything you build runs inside `python -m facetime.worker` (`facetime/worker.py`): on a
   Chert line the room has exactly two seats, the caller's bridge and your worker, so there is no second process on that call and
   no third seat; in your own LiveKit room there is no such limit, but keep one audio publisher.

## Twenty minutes before any call, before anything else

Prove the install with the bundle, then build against recordings. Times below are on the bundle files, on 4 CPU cores:

```
python -m facetime.replay_local --video data/tmp/mac_handoff/crops/run3.mp4 --plan "squats:10,pushups:8,jumping jacks:20,lunges:10" --fast --events data/tmp/run3.events.jsonl
```
About 60 s. It prints every cue with its clip time (`t=11.90s [count] 'One'`) and writes 263 events. Expect squats 9 of 10
(the set closes on the 75 s timeout), the first pushup at 137.8 s. The shipped `data/tmp/mac_handoff/crops/run3.events.jsonl`
was written by an earlier policy version: it has 27 cue events where the current code writes 18 (the nine that went are
a standing "Sink lower" before the first count and idle praise during the uncounted pushup set). Reps
and session states are identical. Regenerate the logs rather than diffing against the shipped ones.

```
python scripts/test_perception.py --video data/tmp/mac_handoff/segments/run3_squats.mp4 --plan squats:10   # 32 s, prints PASS
python scripts/test_voice.py                                                                              # 31 s, 83/83 offline, no key needed
```
(`test_perception.py` without `--video` looks for `data/tmp/squats_clip.mp4`, which is not in the bundle.)

Hear the coach on a recording, no call: `--audio-out` mixes the bank clips at their cue times into a wav.
```
python -m facetime.replay_local --video data/tmp/mac_handoff/segments/run3_squats.mp4 --plan "squats:10" --fast --audio-out data/tmp/squats.wav
```

**Your own phone video must be 10 fps first.** The replay treats every file as 10 fps and loads every frame into RAM (a raw
30 fps 1080p clip plays as 3x slow motion, counts land late, sets time out, and two minutes of it is 20 GB of frames). Convert
once; the pipeline resizes to short side 288 itself, this just keeps the file small and the clock right:
```
ffmpeg -i IMG_1234.mov -vf "fps=10,scale='if(gt(iw,ih),-2,288)':'if(gt(iw,ih),288,-2)'" -an -c:v libx264 -preset fast -crf 23 data/tmp/mine.mp4
python -m facetime.replay_local --video data/tmp/mine.mp4 --plan "squats:10" --fast --events data/tmp/mine.events.jsonl
```
(A 30 fps 1080 copy of `run3_squats` replayed as 111.9 s and timed out at seven; after the one-liner it gave the
same nine reps at the same clip times as the original.) Film in portrait, phone propped at knee height 2 to 3 m away, whole body
in frame, stand still a second before the set (README section 7).

## Decision table

| I want to ... | change this file / use this flag | latency class |
|---|---|---|
| run a different workout | `starter/plan.txt` or `--plan "name:count,..."` (names in section 1) | fast path, 0.2 s |
| change what a correction says | the `cue` strings in `src/rtcoach/cues.json`, then `python -m facetime.cues_tts --backend gemini` | fast path |
| say something new at a moment of my choosing | a new entry in `cues.json` (any key) + re-render, then `say_now(text)` in `starter/handlers.py` | fast path |
| speak at halfway, after a correction streak, at the end | `starter/handlers.py` `on_event` with `say_now` (bank) or `ask_voice` (model) | 0.2 s / 1 s |
| change how Brodie talks between sets | `starter/persona.md`, `ask_voice(prompt)` in handlers, `--voice gemini` | model, 1 s |
| let the caller skip ahead with a gesture | `--gesture-next` (T-pose 1.5 s), a `gesture` event, 4 s of open floor | perception event |
| drop the plan and let the coach guess | `--auto` (`--auto-classes all` for every class) | perception, 3 s to declare |
| develop without calling | `python -m facetime.replay_local --events ...` on a crop, then the harness in section 3 | offline |
| coach something that is not reps (a recipe, a checklist) | `--perception off --vision --voice gemini --persona ... --plan-file ...` (section 8) | model, 1 s; video 1 fps |
| a different voice or accent for the bank | `cues_tts --voice NAME` / `--style "(direction) "` into `--out-dir`, then `CUE_AUDIO_DIR=...` | fast path |
| a longer rest, shorter sets, fewer corrections | `--rest-s`, `--set-timeout`, `--tempo`, `--params '{"gap": 6}'`, `--no-announce` | no code |
| count one exercise with no target | `--exercise squats` (single mode) | fast path |
| a live dashboard of events | `--publish-data` in a LiveKit room you own (not a Chert room) + your own listener | events |
| a laptop-only demo with no phone | `python -m facetime.webcam_local --camera 0 --plan ... --show` (no voice, no handlers) | fast path |
| hand in your call logs | leave `data/tmp/worker_events.jsonl` and `worker_cues.jsonl` on (default) | - |

## The catalogue

Each entry: what it is, the extension point, the command, the latency class, a first version you can have in 30 minutes, and
the limits as measured, not as hoped.

### 1. Your own session plan

**What.** A different list of exercises and targets, in `starter/plan.txt` (read whenever `--plan` and `PLAN` are absent) or on the
command line. Syntax `name:count,name:count`; a typo exits at start with the list of accepted names.

**Names the plan accepts** (27; `facetime/pipeline.py accepted_names()`), and what the counter does with each:
- Counted, validated on a device (the four demo exercises; masks and gates in `src/rtcoach/repcount2.py`): `squats` (vote over
  hip height, knee angle, knee width, shoulder height, shoulder to hip; upright gate; settle rule; walk veto), `pushups` (vote over
  elbow, elbow width, shoulder height and width, wrist height; prone gate), `lunges` or `walking lunges` (upright gate, settle
  rule, the first count must come from a leg or torso tracker; one count per leg), `jumping jacks` (full five-signal vote, no gate).
- Counted, QEVD-trained only, never on a device: `squat jumps`, `squat kicks`, `boxing squat punches`, `lunge jumps`,
  `good mornings`, `good morning beginner`, `plank taps`, `puddle jumps`, `floor touches`, `toe touchers`. These use the full
  top-5 vote with no body-state gate, so any repetitive movement counts (`squat jumps:5` on the jumping-jacks segment
  counts five).
- Accepted but not counted (corrections only; the set runs to `--set-timeout` and closes with "Nice work"): `high knees`,
  `butt kickers`, `quick feet`, `mountain climbers`, `moving plank`, `air jump rope`, `shoulder gators`, `standing kicks`,
  `quad stretch left`, `quad stretch right`, `armcrosschest left`, `armcrosschest right` (the four stretches get the side cues).
  There is no `name:30s` syntax; a timed block is `high knees:1 --set-timeout 30` and the timeout applies to every item.

**Corrections the bank has for the demo four** (`cues.json`, 556 QEVD fault labels over 50 exercises, each mapped to a short
string): squats "Sink lower", "Squat deeper", "Hips back", "Keep back straight"; pushups "Go lower", "Hips down", "Keep going",
"Keep knees up", "Stay in plank"; jumping jacks "Use your arms", "Bring arms up", "Reach arms higher", "Jump feet wider",
"Step feet wider", "Add the jump", "Move your legs", "Sync arms and legs", "Keep movements sharp", "Pick up the pace"; lunges
"Lunge lower", "Sink deeper", "Step deeper", "Keep chest up", "Bend back knee", "Alternate your legs", "Pick up pace"; plus the
tempo cues "Slow down" and "Pick up the pace" for all four.

**Command.**
```
python -m facetime.replay_local --video data/tmp/mac_handoff/crops/run2.mp4 --plan "pushups:8,jumping jacks:20,lunges:10" --fast --events data/tmp/plan.events.jsonl
python -m facetime.worker --plan "pushups:8,jumping jacks:20,lunges:10" dev          # on a call (README section 3)
```
**Latency.** Fast path throughout (counts 0.81 s p50 after the rep bottom on the phone page; announces and milestones 0.2 s).

**30 minutes.** Write the plan, replay it on `run2.mp4` or `live.mp4`, read the `session` events to see each set open, close and
rest, and set `--set-timeout 45` if a set you expect to be short keeps running to 75 s (pushups did on a test call).

**Limits.** Shallow reps are undercounted (exact 0.87 on MM-Fit, 0.59 against coaches' spoken numbers on home sessions); pushups
from a propped front camera can read +2; jacks +1 on hands to the head; a squat within a second of walking in is lost; the
first count needs 0.8 s of standing still; lunges count one per leg; the non-demo countable names have no device numbers at all.

### 2. New cue wording and new bank strings

**What.** Change what a correction says, or add a string of your own that `say_now` can play. The bank is keyed on the exact string:
nothing is synthesised at run time.

**Extension point.** `src/rtcoach/cues.json` (`{"<exercise> - <label>": {"cue": "Sink lower", "kind": "fault"}}`): edit a `cue` value to
reword a correction. For a brand-new line add an entry under any key of your own (the Coach only looks keys up by classifier
label, so `"brodie extras - halfway": {"cue": "Halfway there", "kind": "good"}` is harmless and puts "Halfway there" in the
vocabulary). Then render only what is missing (`GEMINI_API_KEY` exported; at the hackathon that is your team token
and `GEMINI_BASE_URL` must be exported too, so the render goes through Brodie's Gemini relay like the worker's Live session does):
```
python -m facetime.cues_tts --backend gemini --dry-run     # lists the strings to render, no API call
python -m facetime.cues_tts --backend gemini               # renders them into data/cues_audio_gemini, idempotent
```
Every start line then reports coverage: `[cues] bank .../data/cues_audio_gemini: 380 clips, covers 378/378`. A string that is not
rendered is skipped with a warning (`no cue audio for 'Halfway there'`) and `say_now` returns a record with `status: 'missing'`.

**Latency.** Fast path, 0.2 s.

**Cost and time.** About 55 audio tokens per clip; the whole 369-string bank cost $0.16 and the Australian re-render
$0.17 plus re-renders; a handful of new strings is cents. A full bank takes about 4 minutes.

**30 minutes.** Reword the four squat corrections to your coach's voice, add "Halfway there", render, replay `run3_squats.mp4`
and hear it with `--audio-out`.

**Limits.** Counts ("One" to "Twenty", digits to 30), "That's N, nice work", "Let's do X", "Rest for a moment", the praise list
("Nice work", "Looking strong", "Good pace", "That's it", "Keep it up", "Great rhythm"), the visibility lines and "Okay, moving
on" are generated in code (`facetime/cues_tts.py vocabulary()`, `src/rtcoach/coach.py PRAISE`, `facetime/session.py`), not in
`cues.json`: rewording them means editing those strings in code and re-rendering. Keep the plain Puck voice for the strings
you add so the bank stays one voice (an edited `cues.json` key set still carries the QEVD label names; section 11 of the README).

### 3. Handlers: bank lines at moments the shipped coach is silent

**What.** `starter/handlers.py` `on_event(event, say_now, ask_voice)` is called for every event, in order, on the worker's event
loop (return quickly, never block; exceptions are logged and ignored). It loads by default whenever the file exists, in every
mode (`--handlers PATH` for another file, `--handlers none` for nothing). Two verbs: `say_now(text, intent='other')` plays a bank
string through the fast path at priority 'other' (never over a count; `intent='announce'` replaces a pending cue), about 0.2 s;
`ask_voice(prompt)` asks the model (about 1 s to first audio), spoken at once while the floor is open, else queued for the next
opening (dropped when a set starts first or when older than 15 s; at most 3 queued); returns `'spoken'`, `'queued'`, or `None`
without `--voice gemini`.

**Moments the shipped coach does not use** (the default handler asks for one sentence at `rest` and a close at `done`):
the halfway rep (`rep.n == target // 2`, the target from the `session` event at `wait`); a pep line at `rest` after two or more
`cue` events with `intent: correction` in the set; a cool-down ask at `done` (the floor stays open after "All done" until the
caller hangs up, so a timed `later(seconds, prompt, ask_voice)` as in `starter/examples/meal_prep/handlers.py` works there too);
a bank countdown in a timed block (`say_now('Ten')` ... from a timer, in a non-counted item nothing competes with it).

**Develop against a recording, no call.** `replay_local` does not load handlers; it writes the events, and this feeds them to
your file (paste as is; the handler is the one below, tested on `run3`):
```
python - data/tmp/run3.events.jsonl starter/handlers.py <<'EOF'
import json, sys, importlib.util
events, path = sys.argv[1], sys.argv[2]
spec = importlib.util.spec_from_file_location('h', path); h = importlib.util.module_from_spec(spec); spec.loader.exec_module(h)
say = lambda text, intent='other': print(f'          -> say_now {text!r}') or {'status': 'ok'}
ask = lambda prompt: print(f'          -> ask_voice {prompt[:60]!r}...') or 'queued'
for line in open(events):
    ev = json.loads(line)
    if ev['type'] == 'session': print(f"{ev['t']:7.1f}  session {ev['state']:6} {ev.get('exercise') or '-':14} target {ev.get('target')} count {ev.get('count')}")
    elif ev['type'] == 'cue': print(f"{ev['t']:7.1f}  cue     {ev['intent']:10} {ev['text']!r}")
    h.on_event(ev, say, ask)
EOF
```
A 30-minute handler (halfway marker, pep line after a correction streak, cool-down ask):
```python
STATE = {'target': 0, 'corrections': 0}

def on_event(event, say_now, ask_voice):
    t = event['type']
    if t == 'session':
        st = event.get('state')
        if st == 'wait':                                   # a new set is announced: remember its target, reset the streak
            STATE['target'] = event.get('target') or 0; STATE['corrections'] = 0
        elif st == 'rest':
            if STATE['corrections'] >= 2: say_now('Keep it up')          # bank string, about 0.2 s, never over a count
            ask_voice('The set just ended: in ONE short sentence, say how it went and name what comes next.')
        elif st == 'done':
            ask_voice('The session is over: close it warmly in one short sentence and suggest one 30 second stretch.')
    elif t == 'rep' and STATE['target'] >= 6 and event['n'] == STATE['target'] // 2:
        say_now('Looking strong')                                        # the halfway rep
    elif t == 'cue' and event.get('intent') == 'correction':
        STATE['corrections'] += 1
```
On `run3` this says "Looking strong" right after "Five" of the squat set and asks the voice at the 78.0 s rest. On a call the
default command picks the file up: `python -m facetime.worker --voice gemini dev` (without `--voice gemini` only `say_now` works).

**Latency.** `say_now` 0.2 s; `ask_voice` 1 s plus whatever the floor rules impose.

**Limits.** Only bank strings (section 2 to add one). `cue` events are the transcript of what the fast path already said, do not
repeat them. Never ask the voice during `active`: the first count of a set cuts a reply still playing and drops queued asks.
A `visibility` pause is not a slot. `say_now` during a set is legal but shares the track with the counts: keep it to one short
string, the player plays it right after the count in flight.

### 4. A between-sets conversational persona

**What.** Who Brodie is when it talks: `starter/persona.md` is the system prompt of the Gemini Live session (`--persona PATH` for
another file). At every `rest` and at `done` the model also receives a context line it did not ask for, built from the events
(`facetime/voice.py SetSummary`): "Set finished: squats, 10 of 10 reps counted. Corrections spoken: Squat deeper, Sink lower.
Faults seen: squats - shallow x3. Next up: pushups x 8 after a five second rest. You may say one or two short sentences now.";
after a gesture "The caller held a T-pose (the "next" gesture). You have a few seconds; one short line at most."; at `done` the
whole plan's counts and "Close the session warmly in one or two sentences." Nothing is sent at `wait` (a line there can be heard
under the first counts). The model replies only when a handler calls `ask_voice` or the caller talks.

**Command.**
```
python -m facetime.worker --voice gemini dev                                             # starter/persona.md, starter/handlers.py
python -m facetime.worker --voice gemini --persona my_persona.md --handlers my_handlers.py dev
python scripts/test_voice.py --live                                                      # the model session for real, needs GEMINI_API_KEY (41 checks on the Mac)
```
`--voice-name` (env `GEMINI_VOICE`, default Puck) picks another Gemini voice; `GEMINI_LANGUAGE` (default `en-US`) and the persona's
"Always speak English" line set the language; `--log-caller` keeps the caller's transcripts in the `voice` events (off by default).

**Latency.** Model: 0.9 to 1.3 s to first audio with `gemini-3.8-live`; the model session starts 0.9 to 1.0 s after the caller is
bound; a rest is 5 s plus the wait before the next first rep, so ask for one sentence.

**30 minutes.** Rewrite `persona.md` (a drill sergeant, a physio, a parent coaching a child), keep the English line and the
"never count, never correct" rule, change the two `ask_voice` prompts in `handlers.py` to match, run `test_voice.py --live` to
hear a turn, then one call.

**Limits.** The floor is enforced in code, not by the prompt: the model cannot speak during a set whatever the persona says.
Use the plain Puck voice for both the bank and the conversation; mixing voices sounds like two speakers. Another
language only changes the conversation: the bank stays English (section 9 for a partial fix). The fixes for a reply over the
counts and a reply in German pass offline and live checks but have not been verified on a device yet.

### 5. Gesture-driven flows (the T-pose)

**What.** With `--gesture-next` the pipeline runs a T-pose detector (`facetime/probe.py TPose`: both arms straight out at shoulder
height, held 1.5 s, one missed frame allowed, then 5 s before it can fire again). On a fire the session acts: in `active` the set
closes early with "Okay, moving on", then the rest and the next item; in `rest` the next exercise starts at once; in `wait` the
Coach is armed; after `done` nothing. A `gesture {name: tpose}` event is emitted and, with `--voice gemini`, the floor opens for
4 s (not during a set), so a handler can `ask_voice` on it ("the caller wants the next thing").

**Command.**
```
python -m facetime.replay_local --video data/tmp/mine.mp4 --plan "squats:5,lunges:6" --fast --gesture-next --events data/tmp/g.events.jsonl
python -m facetime.worker --gesture-next --voice gemini dev
```
**Latency.** Perception event: 1.5 s of hold plus one frame; the line itself 0.2 s.

**30 minutes.** Record yourself: walk in, three squats, hold a T for two full seconds facing the camera, drop the arms, then the
lunges. Replay with `--gesture-next` and look for the `gesture` event and "Okay, moving on". None of the shipped recordings has a
T-pose (`live.mp4` with `--gesture-next` gives 0 gesture events), so this needs your own clip.

**Limits.** "Okay, moving on" has fired on a device only after "All done" (twice, nothing left to move on from); mid-set
it is unexercised on a call. The detector is ignored while the whole body is not visible, and alternating raises and T-poses
with no pause fires nothing. The arms-up probe (`--latency-probe`) is the other gesture: a beep and
a 4 s floor window, never during a set, off by default because a jumping jack is an arm raise.

### 6. Auto exercise detection

**What.** No plan, no exercise: the classifier's exercise head, smoothed and summed over families, declares an exercise after
3 s above 0.5 ("Looks like jumping jacks"), the counter and corrections switch to it, and a challenger can take over every 10 s.
`--auto-classes demo` (default) declares only the four demo exercises and folds look-alikes in; `all` keeps every class.

**Command.**
```
python -m facetime.replay_local --video data/tmp/mac_handoff/segments/run3_jacks.mp4 --fast --events data/tmp/auto.events.jsonl   # no --plan: auto
python -m facetime.worker --auto dev
```
On `run3_jacks.mp4`: "Looks like jumping jacks" at 4.5 s, then One to Seventeen.

**Latency.** 3 s to declare, then the fast path.

**30 minutes.** A "just start moving" coach: `--auto` plus a handler that asks the voice to name the exercise on each `announce` cue
with intent `announce` and text starting "Looks like".

**Limits.** The head is about 0.75 accurate on real sessions (`exercise_seen` said "floor touches" at 0.94 while the caller stood);
there is no target in auto mode, so no "That's ten" and no rest; trust the plan for anything you demo.

### 7. Replay-driven development

**What.** Any 10 fps recording goes through the same pipeline and session as a call and writes the same events (schema v1, `t` is
the clip time). The bundle has three sample calls cropped to the camera view: `data/tmp/mac_handoff/crops/run3.mp4` (clean
full plan, phone page), `run2.mp4` (end of squats, pushups with phone handling, 20 jacks, 10 lunges), `live.mp4` (the full plan
on a real FaceTime call), and per-exercise cuts in `data/tmp/mac_handoff/segments/` (`run3_squats`, `run3_pushups`, `run3_jacks`,
`run3_lunges`, `run2_*`, `live_*`).

**Commands.** The first block of this page, plus:
```
python -m facetime.replay_local --video data/tmp/mac_handoff/segments/run3_squats.mp4 --exercise squats --fast      # single mode: counts, no target
python -m facetime.replay_local --video data/tmp/mac_handoff/crops/live.mp4 --plan "squats:10,pushups:8,jumping jacks:20,lunges:10" --fast --set-timeout 45 --rest-s 8 --events data/tmp/live.events.jsonl
python scripts/e2e_from_recording.py film.mp4 --crop W:H:X:Y --t0 11 --t1 36 --exercise squats                     # rep bottom to count heard, from a film of the phone
```
**Latency.** None: `--fast` runs at about 28 fps here; without it the replay paces at wall clock like the live agent.

**30 minutes.** Record one set of each exercise you plan to demo with the phone propped as on the call, convert, replay, and read
the counts before you ever dial.

**Limits.** A replay has no transport timing (`status.fps_recv`, `dropped` and the delay are null; `probe` has no `on_box_ms`);
handlers are not run by `replay_local` (section 3 harness); the visibility rule is the same as live, so a laptop-framed clip
will pause.

### 8. A voice-first skill with vision, perception off

**What.** `--perception off` runs no pose model, no counter, no script: the worker joins, publishes the track, binds to the first
caller, starts the model session, and the floor is always open (`--floor open` is then the default): the caller's audio is always
forwarded, `ask_voice` is spoken at once, nothing is cut. `--vision` hands the caller's video to the model at 1 fps (`--vision-fps`,
at most 2; each frame a JPEG scaled to 1024x576). The plan file is appended to the persona as free text, not parsed. The handler
sees one synthetic `session {state: "open", mode: "voice"}` event when the session is up, a `status` every 5 s, and `voice` events.
The example is a cooking coach: `starter/examples/meal_prep/` (a persona that names what it sees before it instructs, a 15-minute
garlic butter pasta as timed steps in `recipe.md`, and a handler that greets on the open event with a `later(seconds, prompt,
ask_voice)` helper for timed nudges).

**Command** (reports `mode voice (perception off, vision on, floor open)` at start; the call itself needs the keys):
```
python -m facetime.worker --perception off --vision --voice gemini --persona starter/examples/meal_prep/persona.md --plan-file starter/examples/meal_prep/recipe.md --handlers starter/examples/meal_prep/handlers.py dev
```
**Latency.** Everything through the model: about 1 s to first audio; the model sees a change in the scene within a second or two.

**30 minutes.** Copy `starter/examples/meal_prep/` to `starter/examples/<yours>/`, replace the recipe with your checklist (a
stretching routine, a desk setup check, a packing list), and change the greeting prompt. Tell the caller how to prop the phone:
the model only sees what the camera sees.

**Limits.** No fast path: every word costs about 1 s. `say_now` still plays bank strings (counts, "Nice work", "Rest for a
moment") and nothing else. No probe, no T-pose, no `rep`, `fault` or `visibility` events. The model cannot read a thermometer
or taste. This mode has been tested offline only (fake room, fake frames, a no-room model session with one pushed frame); it has
not been on a call, phone page or FaceTime. Expect to tune the persona on the first call. `--vision` with perception on is wired in code but untested anywhere,
and the floor stays gated there.

### 9. The bank in another voice, accent or language

**What.** Re-render the 377 strings with a Gemini TTS voice or a stage direction into a new directory and select it with
`CUE_AUDIO_DIR`. The Australian-direction bank already exists in the bundle (`data/cues_audio_gemini_au`, 378 clips, "(Australian
accent, warm, upbeat trainer)"); by default the player prefers the plain Puck bank, then the Australian one.

**Command.**
```
CUE_AUDIO_DIR=data/cues_audio_gemini_au python -m facetime.worker dev                                                   # the Australian bank
python -m facetime.cues_tts --backend gemini --style "(Calm, slow, physiotherapist) " --out-dir data/cues_audio_gemini_calm --dry-run
python -m facetime.cues_tts --backend gemini --style "(Calm, slow, physiotherapist) " --out-dir data/cues_audio_gemini_calm
CUE_AUDIO_DIR=data/cues_audio_gemini_calm python -m facetime.replay_local --video data/tmp/mac_handoff/segments/run3_squats.mp4 --plan "squats:10" --fast --audio-out data/tmp/calm.wav
```
`--voice NAME` picks another prebuilt Gemini voice (default Puck). The bracketed direction is not spoken and not part of the
manifest key; a spoken prefix would be. Always a new `--out-dir`: banks never mix.

**Cost and time.** $0.16 to $0.21 per full bank, about 4 minutes. The
Australian bank needed a duration gate: 21 clips came out far longer than their Puck twins (the direction read aloud or drawn
out) and were re-rendered up to six times each; check clip durations against the Puck manifest before you trust a new bank.

**30 minutes.** A dry run, a render, a replay with `--audio-out`, listen.

**Limits.** Another language is partial: translate the `cue` strings in `cues.json` and the bank follows, but the counts,
"That's N, nice work", "Let's do X" and the praise list are English in code (section 2). A bank voice other than Puck and the
Gemini Live voice will sound like two people; change both (`--voice-name`) or neither. That is why the
default is the plain Puck bank, not the Australian one.

### 10. Logging and telemetry hand-in

**What.** Two local files by default, nothing else: `data/tmp/worker_events.jsonl` (every event plus `room`, `call_id`, `job`;
`--event-log` / `RTCOACH_EVENT_LOG`) and `data/tmp/worker_cues.jsonl` (cues played or dropped; `--cue-log` / `RTCOACH_CUE_LOG`).
No frames, no audio, no transcripts, no handles; the caller's words stay out unless `--log-caller`. Opt out with
`--event-log /dev/null --cue-log /dev/null`. Brodie asks you to leave them on and hand them in after the event, keyed on the room
name and the time of the call (`call_id` is null on the phone page, where the caller is plain `chert-bridge`; a Chert call also has
its `call_id`). A replay's `--events` file is the same schema, so a call log replays through the section 3 harness.

**30 minutes.** A set summary script over `worker_events.jsonl`: reps per set from `session` events at `rest`, corrections from
`cue` events with `intent: correction`, the dominant `fault` label; print one line per room (and per call id on a Chert line).

**Limits.** `t` is per process; `t_wall` is the only cross-process clock; `seq` restarts per job.

### 11. Your own LiveKit room: the phone page (the main path), a data channel, the laptop webcam

Most teams have no Chert line, and even the three that do get a dozen attempts an hour only in a room of their own: the phone
page over your LiveKit project is the main path (README section 3 has the two terminals and what prints; this is the short form).

- **The phone page** (`facetime/phone_test.html`, `facetime/mint_token.py`): a phone or a Mac joins a room of your project,
  publishes its camera as one track named `facetime-remote-video` (asking 1280x720 at 30 fps; a portrait phone delivers
  720x1280, the worker samples at 10 fps) and plays every audio track it subscribes to. Run the worker with automatic dispatch
  (`AGENT_NAME=` empty in `.env.local`, or `--agent-name ""`) so it joins every new room, start it first, and mint a fresh room
  name per attempt (dispatch fires when the room is created, which the phone's join does):
  ```
  python -m facetime.worker --agent-name "" --plan "squats:10" dev
  python -m facetime.mint_token --room rtcoach-team-1 --url      # three lines: the token, the ?url=...&token=...&room=... query string, and the https phone link
  ```
  `mint_token` needs `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET` exported (and `LIVEKIT_URL` for the link); nothing loads
  `.env.local` for you. The THIRD printed line is the phone link: `https://james-72765--rtcoach-webhook.modal.run/phone?url=...`
  served from our host, a static page whose `url` and `token` are your own project (`room` is informational, the token carries
  it). AirDrop it, Messages it or QR it; `&facing=user` for the front camera, `&autojoin=1` to skip the tap. On a Mac, open
  `facetime/phone_test.html` as a file in Safari with the same query string (file:// is a secure context for the camera; http
  from a laptop will not get a phone's camera, hence https). The worker binds to the first non-agent participant that publishes
  video and prints `bound to 'chert-bridge' (call id n/a): video 'facetime-remote-video' 720x1280`, then `first frame 720x1280`,
  then "Let's do squats" is heard. The page's "Coach present" indicator is green only for a participant whose identity equals
  `agent` exactly (default `brodie-agent`, `&agent=<identity>` to change it); a worker under `dev` joins as `agent-<job id>`, so
  it stays "no" on a working call and "Coach audio: playing from agent-..." is the real signal. `python -m facetime.worker
  connect --room <room> --participant-identity brodie-agent` joins one room directly under that identity and makes it green.
  The job ends 2 s after the phone leaves (`--leave-grace-s -1` keeps it). `call_id` is null on this path (section 10).
- **A data channel for a dashboard** (`--publish-data`, topic `--data-topic`, default `brodie.events`): every event as JSON on the
  room's reliable data channel. A third participant (a token from `mint_token.py --identity dashboard`) reads it with the
  LiveKit client's `data_received` event; no listener ships in the repo. Only in rooms you own (a Chert room has no third seat);
  untested on a call.
- **The laptop webcam** (`facetime/webcam_local.py`): the camera, the same pipeline and session, cues on the speaker through
  `sounddevice` (or a wav when there is no output device). Tested with a file as the camera:
  ```
  python -m facetime.webcam_local --camera 0 --plan "squats:10,pushups:8" --show
  python -m facetime.webcam_local --camera data/tmp/mac_handoff/segments/run3_squats.mp4 --plan "squats:10" --audio wav --audio-out data/tmp/webcam.wav --max-s 40
  ```
  No handlers, no voice, no events queue: it is for the perception and the bank only. A laptop camera cannot frame a standing
  adult top to bottom at 2 to 3 m (README section 7).
- **A fake bridge** (`facetime/fake_bridge.py`): publishes an mp4 into a room as the bridge and records the coach's audio with
  onset times, for latency numbers over your own LiveKit project; pair with `facetime/latency_report.py`. Used on the demo coach,
  not re-run with the worker.
- `python -m facetime.worker ... connect --room <room> [--participant-identity brodie-agent]` joins one room directly, no
  dispatch (a LiveKit Agents CLI subcommand; the identity is what the phone page's indicator looks for).

### 12. Tuning the shipped coach without code

All on `facetime.worker`, `facetime.replay_local` and `facetime.webcam_local` alike: `--rest-s 5` (the rest), `--set-timeout 75`
(close a set at this many seconds; 45 suits short sets), `--tempo '{"squats": 2.0}'` (seconds per rep; defaults squats 2.5,
pushups 2.0, jumping jacks 0.9, lunges 2.0; "Slow down" below 0.6x, "Pick up the pace" above 1.8x, at most once per 15 s),
`--no-announce` (no "Let's do X"), `--params '{"gap": 6, "praise_after": 20, "max_cues_per_label": 1}'` (the Coach policy, defaults
in `facetime/pipeline.py POLICY_DEFAULT`: gap 3 s between cues, cooldown 8 s per label, praise every 8 s, 3 cues per label per
set), `--trigger none` (the threshold rule instead of the learned trigger; fewer, later corrections), `--counter e1` (the older
pose-PCA counter), `--exercise squats` (single mode: one exercise for the whole call, no target, no rest), `--pose lite` (faster,
less accurate pose model). Replay first: the same flags give the same cues on a recording.

### 13. Bring your own mouth (advanced)

`--events-only` (or `EVENTS_ONLY=1`) publishes no audio at all: no cue player, no track. The events still flow to `job.events`
(register `@facetime.worker.consumer async def fn(job): ev = await job.events.get()` in a module you import before the worker
starts), to the JSONL log and, with `--publish-data`, to the room. You then own the only voice in the room and must publish
it from the same process. Honest cost: you lose the 0.2 s fast path and must design counts around 1 s (the Gemini-in-the-loop
baseline matched 0 of 16 counts at jacks cadence). A day's work with no shipped example; the shipped `--voice gemini` path is the
same idea with the fast path kept.

## What Brodie cannot do today

- **No new fault detectors.** The classifier heads are fixed: the 50 QEVD exercises and their 556 labels, trained under a licence
  whose data you do not have. A new exercise or a new fault needs training data; today you can only reword what exists, change
  when it fires (`--params`, `--trigger`) and which labels count (`--thr`).
- **No model-driven counts.** Counts come from the pose counter on the fast path; a realtime model answers in 0.8 to 1.2 s, one to
  two reps late at jacks cadence. Keep counts and form cues on the fast path, conversation on the model.
- **Vision for the model is about 1 fps** (at most 2), JPEG, and only tested offline; it is a scene reader, not a rep watcher.
- **Two seats per room on a Chert line.** The caller's bridge and your worker. No sidecar, no dashboard participant, no second audio
  publisher on a Chert call; `--publish-data` has no reader there. In your own LiveKit room there is no such limit (keep one audio
  publisher).
- **A Chert line is Offline for a few minutes after each call** (Chert's worker refreshes on its cron cycle); calls in that window ring
  out. The phone page has no such wait (a fresh room name per attempt); recordings are the inner loop on both.
- **No Windows.** macOS or Linux, Python 3.12; Node 22.23.2 or 24.x for the Chert CLI (the three Chert teams only).
- **The voice cannot speak during a set** whatever the persona says, and a reply still playing is cut by the first count.
- **Only bank strings are spoken on the fast path**; nothing is synthesised at run time. A new string is a render step (section 2).
- **Counting is validated on a device for four exercises only**; the other nine countable names have QEVD numbers at best.
- **No time-based plan items** (`name:count` only); a timed block is `name:1 --set-timeout N`, and the timeout is global.
- **The T-pose mid-set and the latest voice fixes have not been verified on a device**; the meal-prep mode has not been on a call.
- **The avatar face (`--avatar`) and the mirror (`--mirror`: the caller's own frame with the pose, the count and the last cue drawn
  on it as the coach's video) are on the demo coach (`facetime/coach_agent.py`), not on the starter worker.**
- **No hot reload**: restart Terminal A after every code change (the `dev mode is deprecated, use lk agent dev` notice from
  livekit-agents 1.8.5 is expected; ignore it); on a Chert line also restart `chert dev` so no prepared room runs old code.

## Latency classes, in one place

| path | measured | where |
|---|---|---|
| bank line, decision to heard | 145 to 225 ms over the LiveKit SFU | PROGRESS.md loopbacks |
| rep bottom to count heard | 0.81 s p50 (0.72 to 0.88) on the phone page; about 0.3 s is the counter's reversal detection | `scripts/e2e_from_recording.py`, a sample recording |
| receive to decision, perception | 27 to 40 ms p50 at 10 fps on the Mac over the CLI path | reports/MAC_TEST_2026-10-08.md sections 5 and 6 |
| model, prompt to first audio | 0.9 to 1.3 s (`gemini-3.8-live`), 1.2 to 1.3 s in the live tests | `facetime/voice.py`, `scripts/test_voice.py --live` |
| model session up after the caller is bound | 0.9 to 1.0 s | Mac section 6 |
| T-pose | 1.5 s hold, then 5 s refractory | `facetime/probe.py` |
| auto-detect | 3 s above 0.5 to declare | `facetime/session.py` |
| ring to answer on a Chert line | 8.6 to 8.7 s (Chert's side), then a 0.1 s warm-room accept, first cue about 7 s after the accept | Mac sections 5 and 6 |

# Events and run modes

The perception event schema and the worker's run modes: the contract between the coach's eyes and its voice.

The realtime coach's eyes as a stream of events. One process, one participant, one audio track: `facetime/worker.py` is a
LiveKit Agents worker that runs `facetime/perception.py` (FramePipeline + Session exactly as the demo coach runs them, with
`Session.say` replaced by an event sink and no audio of its own). By default the worker also attaches the demo coach's cue
player (`facetime/audio_out.py`, unchanged) so counts and corrections are spoken from the pre-rendered bank 0.2 s after the
decision; a conversational layer consumes the events and speaks only when the session opens a slot. The demo path
(`coach_agent.py`, `pipeline.py`, `session.py`, `audio_out.py`, `modal/webhook.py`) is untouched; replays of the three demo
clips are byte-identical with and without the new code.

Files: `facetime/perception.py` (service, schema, JSONL writer), `facetime/worker.py` (LiveKit Agents worker),
`facetime/replay_local.py --events PATH` (the same stream from a recording), `scripts/test_perception.py` (contract test).

## 1. Schema

One JSON object per event. Every event has:

| key | meaning |
|---|---|
| `v` | schema version, `"1"` (`perception.EVENT_SCHEMA_VERSION`) |
| `t` | seconds since the service started, on the producer's monotonic clock; for frame events the frame's receive time. A replay passes `t0=0` and the clip time, so `t` is the clip time there. Not comparable across processes: use `t_wall` for that. |
| `t_wall` | epoch seconds when the event was emitted |
| `seq` | 0, 1, 2 ... per service |
| `type` | one of `rep`, `exercise_seen`, `fault`, `cue`, `visibility`, `gesture`, `probe`, `session`, `status`; with `--voice gemini` also `voice` {text, role, producer} (section 2c), on the same queue and log, with its own `seq` and no `frame` |
| `frame` | index of the processed frame the event came from (all types except `status`) |

The worker's JSONL log adds `room`, `call_id` (from a Chert bridge identity `chert-bridge-<call_id>`, null until that bridge is
bound and null for the whole of a phone-page call, whose participant is plain `chert-bridge`: key those logs on `room` and
`t_wall`) and `job` to every line. Lines below are from a sample recording, `data/tmp/mac_handoff/crops/run3.mp4`
(portrait 288 px, 10 fps), plan `squats:10,pushups:8,jumping jacks:20,lunges:10`, through `replay_local --events`; with the current policy the 7.5 s fault below is no longer spoken as a `cue` (section 5), the other lines are unchanged.

**rep** {exercise, n, confidence, turning_t, lag_s}: the counter confirmed rep `n` of the current set. `confidence` is from
RepCounter2: `agree` = share of the vote's slots (3 for the masked squat and pushup votes, 5 otherwise; unfired trackers vote 0)
whose count reached `n`, `amp` = median reference rep amplitude of the voting trackers (torso lengths or normalised joint
angle), `trackers` = voting trackers; null with the E1 counter. `turning_t` is the rep's turning point (the bottom) on the event
clock, `lag_s = t - turning_t` is how long after it the count landed (0.1 to 0.3 s offline; the audio adds 0.2 s on the fast path).
```
{"v": "1", "t": 11.9, "t_wall": 1791526458.6, "seq": 25, "type": "rep", "frame": 119, "exercise": "squats", "n": 1, "confidence": {"agree": 1.0, "amp": 1.816, "trackers": 3}, "turning_t": 11.7, "lag_s": 0.2}
```
**exercise_seen** {name, p}: the classifier's exercise head, top class and probability, at most once per second while the top
class is unchanged, whenever the head ran (every armed frame with the trigger on; not during waits and rests).
```
{"v": "1", "t": 4.5, "t_wall": 1791526456.235, "seq": 6, "type": "exercise_seen", "frame": 45, "name": "floor touches", "p": 0.942}
```
**fault** {label, cue_text, p}: the Coach decided a correction on this frame (the trigger fired and a label won), before the
session's hold/drop scheduling. `p` is the Coach's EMA probability of the label (null for a side-rule pseudo-label). The
spoken version, if it was spoken, is a later `cue` with `intent: correction`.
```
{"v": "1", "t": 7.5, "t_wall": 1791526457.219, "seq": 12, "type": "fault", "frame": 75, "label": "squats - shallow", "cue_text": "Sink lower", "p": 0.545}
```
**cue** {text, intent, why, state, exercise, count, t_session, held_s?, t_decided?}: what the demo coach says, in the order its
player queues it: `intent` is `count`, `correction`, `positive` (praise, "Rest for a moment"), `announce` ("Let's do X",
"Moving on to X", "Looks like X", "All done, great session"), `milestone` ("That's nine, nice work", "Okay, moving on"),
`visibility` ("Step back so I can see your whole body", "Great, I can see you", "I will wait here ..."). A correction or praise
held for the count carries `held_s` and `t_decided`. In events-only mode nothing speaks these; they are the script.
```
{"v": "1", "t": 11.9, "t_wall": 1791526458.601, "seq": 26, "type": "cue", "frame": 119, "text": "One", "intent": "count", "why": "", "state": "active", "exercise": "squats", "count": 1, "t_session": 11.9}
{"v": "1", "t": 33.6, "t_wall": 1791526465.613, "seq": 73, "type": "cue", "frame": 336, "text": "Nice work", "intent": "positive", "why": "idle", "state": "active", "exercise": "squats", "count": 9, "t_session": 33.6, "held_s": 0.9, "t_decided": 32.7}
```
**visibility** {state, paused, no_pose_s}: `state` is `full` (nose and both ankles in frame), `partial` (a pose, not the whole
body) or `none` (no pose), reported after it has held 0.5 s; `paused` is the pipeline's hysteretic pause (3 s of partial or no
pose: counting, corrections and the session clock stop; "Step back ..." is the matching cue) and is reported at once.
```
{"v": "1", "t": 0.4, "t_wall": 1791526455.096, "seq": 2, "type": "visibility", "frame": 4, "state": "partial", "paused": false, "no_pose_s": 0.0}
```
**gesture** {name}: a T-pose held 1.5 s (`facetime/probe.py TPose`), at most once per 5 s. Emitted only when the pipeline runs with
`--gesture-next` (without the flag no T-pose detector runs); the session then acts on it ("Okay, moving on"). On a device it has fired
only after "All done", with nothing to move on from. No T-pose in this recording; the shape is:
```
{"v": "1", "t": 61.2, "t_wall": 1791526473.1, "seq": 118, "type": "gesture", "frame": 612, "name": "tpose"}
```
**probe** {arms_up, on_box_ms?, span_ms?, t_first_up?, beep?}: both wrists above the head after 1 s below (the latency probe). Live,
`on_box_ms` is the time from the firing frame's receipt to this event (the receive-to-decision leg, as coach_agent measures it, no
audio leg); `span_ms` counts from the first up frame's receipt (the probe confirms over two frames, so this includes a frame interval:
the 106 to 165 ms the Mac saw). The worker adds `beep`: true only when it runs with `--latency-probe` (env `RTCOACH_LATENCY_PROBE=1`;
off by default because a jumping jack is an arm raise and beeps mid-set), the cue player is attached, no set is active and
`--no-probe-beep` is not given; the cue log then shows first-up-frame receipt to beep push. The event is emitted and logged either
way (`[event] probe ... (probe off ...)` or `(during a set: no beep, no floor window)`). A replay has no timing.
```
{"v": "1", "t": 52.0, "t_wall": 1791526471.628, "seq": 104, "type": "probe", "frame": 520, "arms_up": true}
```
**session** {state, exercise, target, count, item, done, mode}: on every state change of the session script. Plan mode:
`init` -> `wait` (announced, counter only, up to 3 s or first movement) -> `active` (counting, correcting) -> `rest` (5 s)
-> `wait` of the next item ... -> `done`. Single mode: `init` -> `active`; auto mode: `init` -> `detect` -> `active`.
```
{"v": "1", "t": 78.0, "t_wall": 1791526480.102, "seq": 149, "type": "session", "frame": 780, "state": "rest", "exercise": null, "target": 10, "count": 9, "item": 0, "done": false, "mode": "plan"}
```
With `--perception off` (section 2d) there is no session script: the worker emits ONE synthetic session event once the model
session is up (the caller is bound, the model listens and, with `--vision`, watches), so a handler can greet there. `exercise`,
`target` and `item` are null, `count` 0, `done` false, `mode` is `"voice"`, `frame` is the number of video frames received so
far. No `rep`, `exercise_seen`, `fault`, `cue`, `visibility`, `gesture` or `probe` event is ever emitted in that mode.
```
{"v": "1", "t": 3.2, "t_wall": 1791654028.1, "seq": 0, "type": "session", "frame": 31, "state": "open", "exercise": null, "target": null, "count": 0, "item": null, "done": false, "mode": "voice"}
```
**status** {fps_recv, fps_processed, pose_ms, clf_ms, pose_model, recv_to_decision_p50_ms?, dropped?, underruns?,
cues_played?, cues_dropped?, note?, state, exercise, count}: every 5 s. `fps_recv`, `dropped` and the delay are set by the
transport (null in a replay); `underruns` and the cue counts only when the player is attached; `note` when the pose model
dropped to lite.
```
{"v": "1", "t": 15.0, "t_wall": 1791526459.617, "seq": 34, "type": "status", "fps_recv": null, "fps_processed": 10.0, "pose_ms": 26.6, "clf_ms": 4.9, "pose_model": "full", "state": "active", "exercise": "squats", "count": 2}
```

## 2. The run modes of the worker

```
.venv/bin/python -m facetime.worker --help
.venv/bin/python -m facetime.worker [--plan "squats:10,pushups:8"] dev                    # (a) default: eyes + fast mouth
EVENTS_ONLY=1 .venv/bin/python -m facetime.worker dev                                     # (b) events only, no audio at all
.venv/bin/python -m facetime.worker --events-only --publish-data start                   # production, events also on the data channel
GEMINI_API_KEY=... .venv/bin/python -m facetime.worker --voice gemini dev                 # (c) eyes + fast mouth + Gemini Live between sets
GEMINI_API_KEY=... .venv/bin/python -m facetime.worker --perception off --vision --voice gemini dev   # (d) no perception: voice always on, video to the model
```
At the hackathon `GEMINI_API_KEY` is the team token and `GEMINI_BASE_URL=https://james-72765--rtcoach-gemini.modal.run` must be
exported with it (Brodie's relay, [/hackathon/start](/hackathon/start) section 4); a Google key of your own works with `GEMINI_BASE_URL` unset.
(a) **Default (fast mouth attached).** The worker publishes ONE audio track (`coach-audio`, 24 kHz mono) right after joining
and feeds it with the demo coach's CuePlayer and CueBank, unchanged: counts, closed-vocabulary corrections, praise, announces,
milestones and visibility lines, through the same priority slots, 145 to 225 ms frame-to-heard. The conversational layer gets
the events and must not speak over the fast path during a set (section 4).
(b) **Events only** (`--events-only` or `EVENTS_ONLY=1`). No CuePlayer, no AudioSource, no track: the audio track belongs to
whoever owns the conversation. `cue` events are still emitted as the script of what the fast path would have said.
(c) **Voice** (`--voice gemini`, needs `GEMINI_API_KEY`; `facetime/voice.py`): (a) plus Gemini Live (voice Puck, model
`GEMINI_LIVE_MODEL`, default `gemini-3.8-live`, language `en-US` via `GEMINI_LANGUAGE`) in the same participant. Its audio goes
through the cue player's one AudioSource (`CuePlayer.attach_mixer()`: pushed between clips, ducked about 10 dB under a count, delaying
a count by at most 60 ms: one 20 ms frame beyond the 40 ms lead) and it may only speak while the floor is open: session state `wait`,
`rest` or `done`, four seconds after a `gesture` or an armed `probe` (never during an active set: a gesture or probe there is logged,
not acted on), or when the caller talks during a rest (the caller's audio is not forwarded during a set). The cut when a set starts
is hard (otherwise the voice is heard under the counts): the reply is interrupted, the sink's buffer and the mixer queue are
cleared, and any frame that still arrives while the floor is closed is dropped. At the `rest` and `done` openings the model gets a
text summary of the set (exercise, reps, corrections spoken, the next item), never per rep and nothing at `wait` (a line there was
answered under the first counts). `ask_voice` prompts queued while a set runs are dropped when they are older than 15 s at the
opening or when the next set starts first. The model session starts when a caller is bound, never on a warm job. Model turns are
logged as `voice` events {text, t, role coach}; the caller's transcribed turns are NOT emitted or logged unless the worker runs with
`--log-caller` (then role caller), so by default no caller words leave the model session.
Logging lines to read a call by: `[voice] speaking (floor open|closed, state X, N ms after the ask)` when the first audio frame of a
reply reaches the sink (the transcript line `[voice] coach: ...` can arrive tens of seconds later, so it is not the time the voice
was heard), `[voice] cut: N frames dropped (floor closed), M ms of buffered audio cleared; S s of the reply was heard`,
`[voice] dropped queued ask (why): ...`, `[voice] context: ...` at each opening, `[voice] caller spoke (N words; not logged)`.
Extension points in `starter/`: `persona.md` (the prompt), `plan.txt` (the plan when `--plan`/`PLAN` are not given), `handlers.py`
with `on_event(event, say_now, ask_voice)`: `say_now(text)` is a bank string through the fast path (about 0.2 s, never over a
count), `ask_voice(prompt)` asks the model (about 1 s to first audio; queued for the next opening while a set runs). `handlers.py`
is loaded in every mode whenever the file exists (`--handlers PATH` for another file, `--handlers none` for nothing); without the
voice `ask_voice` returns None. The default handler asks for one short sentence at each `rest` and at `done`, nothing else.

Events go, in both modes, to `job.events` (an `asyncio.Queue` in the job process, for a conversational layer: register
`@facetime.worker.consumer async def fn(job)`), to the JSONL log `--event-log` / `RTCOACH_EVENT_LOG` (default
`data/tmp/worker_events.jsonl`, appended, one line per event plus room, call_id, job), and with `--publish-data` to the room data
channel (topic `brodie.events`, reliable, JSON per event; off by default because on a Chert line the room holds exactly two
participants and nobody else is there to read it; in a room of your own LiveKit project a third participant can read it). Cues the player finished or dropped go to `--cue-log` /
`RTCOACH_CUE_LOG` (the coach_agent `--log` record shape). The plan comes from `--plan`, else env `PLAN`, else `starter/plan.txt`
(`--plan-file`), else the demo plan `squats:10,pushups:8,jumping jacks:20,lunges:10`; `--exercise X` for one exercise, `--auto` for
auto-detect. `--latency-probe` (env `RTCOACH_LATENCY_PROBE=1`) arms the arms-up probe (beep and a 4 s voice window, never during a
set; off by default); `--gesture-next` turns the T-pose detector on; `--log-caller` keeps the caller's transcripts in the voice events.

(d) **Voice plus vision, perception off** (`--perception off [--vision] --voice gemini`, [/hackathon/start](/hackathon/start) section 13):
```
.venv/bin/python -m facetime.worker --perception off --vision --voice gemini --persona starter/examples/meal_prep/persona.md --plan-file starter/examples/meal_prep/recipe.md --handlers starter/examples/meal_prep/handlers.py dev
```
No Perception at all: prewarm loads no pose model, no session script runs, no `rep`, `exercise_seen`, `fault`, `cue`,
`visibility`, `gesture` or `probe` event exists. The worker joins, publishes the cue track, binds to the first remote participant
that publishes video (or audio when there is no video), starts the model session on that bind (never on a warm job), emits the
synthetic `session` `{state: "open", mode: "voice"}` event (section 1) once the session is up, then a `status` event every 5 s
`{fps_recv, state, exercise: null, count: 0, mode: "voice", voice: on|off, vision_frames?, vision_fps?, vision_replaced?, underruns?,
cues_played?, cues_dropped?}` (no pose fields), and ends on the same paths as (a) to (c): the caller's leave after `--leave-grace-s`,
the room's disconnect, `--max-s` after the first frame, a connect failure. Flags:
- `--perception {on,off}` (env `PERCEPTION`): `off` is this mode. With `off` the plan file (`--plan-file`, `--plan`, `PLAN`) is
  appended to the persona as free text ("The plan you are coaching, step by step: ..."), not parsed as exercises.
- `--floor {gated,open}` (env `FLOOR`): `open` is the default with `--perception off` (`gated` with perception on). Open means the
  Floor never closes: the caller's audio is always forwarded, `ask_voice` is spoken at once (never queued, never dropped),
  nothing is ever interrupted or cut; a `session`, `gesture` or `probe` event changes nothing. `--floor open` with perception
  on is allowed and warned about (the voice would talk over the counts).
- `--vision` (env `VISION=1`), `--vision-fps N` (env `VISION_FPS`, default 1, at most 2): the caller's video track reaches the
  model session. The worker already opens one `rtc.VideoStream` on the bound track (that is how a caller is detected); with
  `--vision` that read loop also offers each frame to `facetime/voice.py VideoTee`, a `livekit.agents` `VideoInput` set as
  `AgentSession.input.video` before `start()` (the session is built with `video_sampler=None`, so the tee's one frame per 1/N s
  is the only sampling; the newest frame replaces an unread one). The framework's own forward task pulls from the tee and
  hands every frame to the realtime session's `push_video`, which JPEG-encodes it (quality 75, scaled to fit 1024x1024: a
  1280x720 frame goes out as 1024x576). The worker's stream is RGB24 (Perception's format) and the plugin's encoder cannot
  convert that (`rgb24 to Rgba is not supported` in the rtc FFI), so the tee hands each
  sampled frame over as RGBA (`voice.rgba_frame`, under 10 ms). RoomIO's own video input stays off: it selects tracks by source (camera, screenshare)
  and the bridge's track source is not relied on. The tee is enabled only after the model session started: frames offered
  before that are counted (`vision_early`) and never sent; a warm job has no video and offers nothing. Logged once:
  `[vision] forwarding video to the model at 1 fps (1280x720)`; the `[status]` line and event carry `vision_frames` (sent).
  `--vision` without `--voice gemini` forwards nothing (said at start).
`say_now` still plays bank strings; `ask_voice` returns `'spoken'` at once. The probe and the T-pose detector do not run.
Tested offline only (scripts/test_worker_shutdown.py E and F, scripts/test_voice.py 9 to 11): a fake room with a fake
10 fps video track and a no-room model session that accepts one pushed frame. Not yet on a FaceTime call.

Models (pose, classifier, trigger, thresholds) and the cue bank load in the worker's prewarm hook (about 5 s here, 0.7 s of it
after imports); one preloaded job process is kept idle (`--idle-processes`, env `RTCOACH_IDLE_PROCS`), so a job start costs no
model load.

## 3. Dispatch: your own LiveKit rooms (automatic) first, the Chert CLI (named) second

- The worker registers with LiveKit under `AGENT_NAME` (env or `--agent-name`, default `brodie-coach` when the variable is
  missing). An EMPTY name (`AGENT_NAME=` in `.env.local`, or `--agent-name ""`) is automatic dispatch: LiveKit hands the worker a
  job for every new room of the project. That is the phone-page path (/hackathon/start section 3) and `fake_bridge.py`.
  Automatic dispatch fires when a room is CREATED, so start the worker first and mint a fresh room name per attempt; a room
  created while the worker was down never gets a job. The phone page joins as `chert-bridge` (the `--identity` default of
  `mint_token.py`) and publishes one video track `facetime-remote-video` (720x1280 from a portrait phone); the worker binds to it,
  logs `call id n/a`, and `call_id` is null on every line of that call. The job ends 2 s after the bound participant leaves
  (`--leave-grace-s -1` keeps it).
- A named worker receives only explicit dispatches and never joins a phone room. `chert dev --agent <name>` (the three Chert
  teams) creates a room in the team's LiveKit project with a RoomAgentDispatch for that name; LiveKit dispatches a job to the
  worker; the call is accepted once a participant of kind AGENT is in the room.
- Chert rooms: `chert-warm-<random>` (prewarm, `chert doctor --agent`) and `chert-<operation_id>` (a call). The bridge joins as
  `chert-bridge-<call_id>` and publishes the caller's camera (1280x720 at 15 fps); the worker binds to the first non-agent
  participant that publishes video, whoever it is, and logs the call id. Chert rooms hold two participants; the bridge binds to
  the one participant with an unmuted audio track, so the audio track is published at join time, before any video arrives. In
  a room of your own there is no two-seat limit; keep one audio publisher.
- **Warm jobs** (job metadata `{"chert_warm": true}`, or any job with no caller): the worker joins, publishes its (silent) audio
  track in mode (a), loads nothing more, says nothing, starts no session script, and stays joined until a participant publishes
  video or the room closes. Metadata keys `chert_call` and `media` are logged, never acted on.
- The worker ends the job itself (livekit-agents runs shutdown callbacks only after the entrypoint returns): 2 s after the bound
  participant leaves unless somebody rebinds (`--leave-grace-s`), on the room's disconnect (Chert deletes the room when the call
  ends), when connect() fails (a doctor or warm room torn down early), on `--max-s N` (N s after the first frame, tests) and on a
  perception error; cleanup is bounded (about 250 ms on test calls) and the process exits clean. A video track that goes away
  and comes back within the grace (camera switch, rejoin) is bound again and the session continues.
- Env: `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` (the team's LiveKit project; the worker needs nothing else but the
  bundle on disk), `AGENT_NAME`, `PLAN`, `EVENTS_ONLY`, `RTCOACH_EVENT_LOG`, `RTCOACH_CUE_LOG`, `CUE_AUDIO_DIR` (voice bank);
  with `--voice gemini` also `GEMINI_API_KEY` and, at the hackathon, `GEMINI_BASE_URL`. Requires `livekit-agents==1.8.5`
  (requirements.txt).

## 4. What a conversational layer should do with the events

- **Speak only between sets or when the session opens a slot.** Slots: `session.state` `rest` (5 s after a set), `done`, the
  `wait` state right after an `announce` cue (the caller is getting into position; the floor is open for a question, but do not
  volunteer a line: one can be heard under the first counts), a `gesture` (`tpose`: the caller is asking for the next thing;
  4 s), an armed `probe` outside a set (4 s), and a caller's question. A `visibility.paused` is NOT a slot: the fast path has
  said "Step back ..." and the floor does not open on a pause. Never during `active`: the fast path owns the set, the first
  count cuts a voice reply, and an LLM voice lands 1 to 2 s after a rep bottom, one to two reps late at jacks cadence (the
  Gemini-in-the-loop baseline matched 0 of 16 counts).
- **Gesture and probe rules for the caller** (`facetime/probe.py`): the arm raise fires only after both
  wrists have been below the nose for a full second, so drop the arms and wait a second between raises; hold the T-pose for a
  good two seconds (1.5 s of detection, one missed frame allowed) and drop the arms once the coach speaks, since it will not fire
  again for 5 s; neither is acted on during a set (a jumping jack is an arm raise), and after "All done" a T-pose has nothing
  to move on to.
- **Do not repeat the fast path.** `cue` events are what was just said; use them as the transcript of the coach's half. In
  events-only mode the layer owns the only voice; if it speaks the counts itself it must design around 1 s, not 0.1 s.
- **Shape the next rest from the set.** `fault` events (label, p) say which corrections were decided and `cue`/`correction` which
  were spoken; `rep.confidence.agree` below 0.67 marks a rep only two of three trackers agreed on; `rep.lag_s` is the detection
  delay. Summarise at `rest`: count reached, the dominant fault, the tempo ("Slow down" / "Pick up the pace" cues).
- **Trust the plan, not `exercise_seen`.** The head is about 0.75 accurate on real sessions (above: "floor touches" at 0.94 while
  the caller stands); in plan mode it is informational. Only auto mode acts on it (3 s above 0.5 to declare).
- **Latency classes.** Fast path about 0.8 s rep bottom to count heard on the phone (0.81 s p50, 0.72 to 0.88; about 0.3 s of it is the counter's reversal detection); LLM voice 1 to
  2 s. Perception adds about 40 ms receive-to-decision at 10 fps on 4 CPUs.
- **Iterate without a call.** `replay_local --events PATH --fast --video my.mp4 --plan ...` writes the same stream for any phone recording that is already 10 fps (the replay does not resample: convert with the
  ffmpeg line in [/hackathon/start](/hackathon/start) section 6; the pipeline resizes to short side 288 itself); a handler can be
  developed against the file with the harness in [/hackathon/build](/hackathon/build) section 3 (`replay_local` itself does not run handlers). The phone
  page costs nothing per attempt (a fresh room name each time); a Chert line is shared by time slot and Offline for a cron cycle
  after each call; on both, this is the inner loop.

## 5. Known limits

- Counter (current defaults validated offline on the sample recordings): pushups from a propped front camera
  can read +2 at a bad floor angle; jacks +1 on hands to the head after a set; a person who walks into place and squats within
  a second loses that first rep (walk veto); the first count needs 0.8 s of standing still. Above, run3's squat set reached 9
  of 10 and closed on the 75 s set timeout, and the first pushup landed at 137.8 s.
- A correction decided before a set's first count waits for "One" and is dropped if no rep comes, and no praise is spoken
  before the first rep (the `fault` at 7.5 s above is still emitted, its cue is not); cues decided
  before a visibility pause are dropped, never replayed.
- Visibility: whole-body rule with a 0.9 margin, pauses after 3 s; laptop framing and jumps at the top of the frame produce
  `partial` flicker (0.5 s hold) and can pause sets; no hysteresis on the per-frame state.
- `gesture` (T-pose) has fired on a device only after the plan was done, so "Okay, moving on" is unexercised mid-set on a
  call; `probe.on_box_ms` is on-box only (106 to 165 ms first-up-frame to beep on a FaceTime call).
- `exercise_seen` accuracy about 0.75; `status.fps_recv`, `dropped` and the delay are null in replays.
- `t` is per process and monotonic; `t_wall` is the only cross-process clock. `seq` restarts per job.
- Both paths are verified end to end. Phone page: the worker under automatic dispatch binds to the page's `facetime-remote-video`
  track at 720x1280, prints `first frame 720x1280` and "Let's do squats" is heard (the demo coach ran the full plan twice on the
  same page, every set on target). Chert: a FaceTime Video call through `chert dev --agent brodie-coach` ran the whole plan on
  a Mac (2 min 47 s, squats, jacks and lunges on target) and `--voice gemini` ran on a call too. `--publish-data` cannot be used in a two-participant Chert room (it can in your own).
- Voice: the hard cut above keeps the model off the counts, the persona and `language=en-US` keep it in English, and the
  default handler asks for a line at each rest. The voice holds until a clip ends instead of playing under it, the caller's
  audio reaches the model only when no Brodie audio has played for 0.3 s, the rest and wait clock pauses while either side
  talks (up to 25 s), and only one Brodie line plays at a time (an ask waits while any Brodie audio plays); prompts are never
  read aloud. These pass 61 offline and 63 live checks and have not been verified on a device yet.
- The voice seam: the bank is Gemini Puck; a conversational layer brings its own voice. Use the plain Puck voice for both the
  bank and the conversation, not the Australian bank; mixing voices sounds like two speakers.

# The Gemini relay

What the relay does for your coach, what it costs in latency, and how the worker talks to it.

## 4. The Gemini relay

Your card carries a team token, not a Google key. With `GEMINI_API_KEY=<team token>` and
`GEMINI_BASE_URL=https://james-72765--rtcoach-gemini.modal.run` exported, every Gemini byte from your laptop goes to our relay
(`modal/gemini_relay.py`), which swaps the token for our key and forwards to Google: the Live websocket (`/ws/...`, the
worker's `--voice gemini`) and plain HTTPS (`generateContent`, `models.list`, TTS for `cues_tts`, streaming `alt=sse`). The
token travels in the `x-goog-api-key` header; the google-genai SDK does that itself once the two variables are set, so the
worker and `cues_tts` need no change. An unknown token gets HTTP 401 (JSON `{"error": "unknown team token"}`) or a websocket
close 1008. Each team has 240 Live minutes; past the cap the relay answers 429 or close 1008 "team cap reached, ask the
organiser". `GET /health` is public and shows token counts and totals, never a key or token. A team's own Google key works
too: put it in `GEMINI_API_KEY` and leave `GEMINI_BASE_URL` unset. Smoke test (replace the token, never paste it anywhere):
```
curl -s -X POST "https://james-72765--rtcoach-gemini.modal.run/v1beta/models/gemini-2.5-flash:generateContent" -H "x-goog-api-key: <team token>" -H "content-type: application/json" -d '{"contents":[{"parts":[{"text":"hi"}]}]}'
```

# If your team has one of the three Chert lines

What is different for a team on one of the three shared Chert FaceTime lines, and what to do about it.

## 5. If your team has one of the three Chert lines

Everything above still applies; the line adds a Chert project (one FaceTime line), the Chert CLI, a dispatch name, and a
real FaceTime call in place of the phone page. The CLI facts are from docs.trychert.com/facetime-cli.

The Chert CLI (Node 22.23.2 or 24.x, the same Node in both terminals):
```
node --version                       # 22.23.2 or 24.x; same Node in both terminals
npm install -g @trychert/facetime@0.1.0-beta.4 && chert --version    # pin the version: the unpinned install can fail
chert login                          # browser sign-in for your project; `chert login --device` on a remote shell
chert whoami
chert doctor                         # run from the repo folder: it reads LIVEKIT_* from .env.local; read-only
```
In `.env.local` set `AGENT_NAME=<the dispatch name on your assignment card>` while you work the line: the CLI dispatches
to that name, and a worker with an empty name is not what it dispatches to. Put it back to empty for the phone page.

Terminal A, the coach (export `.env.local` as in section 3):
```
cd brodie-rt-coach && source .venv/bin/activate && set -a && source .env.local && set +a
python -m facetime.worker dev                                   # or --plan "...", or --voice gemini, as in section 3
```
Terminal B, the line (same folder, same Node):
```
cd brodie-rt-coach && set -a && source .env.local && set +a
chert doctor --agent "$AGENT_NAME"   # dispatches a real job into a temporary room; Terminal A must already be running
chert dev --agent "$AGENT_NAME"
```
Then place a FaceTime VIDEO call from your own phone to the address on your card, prop the phone (section 7) and step in.

What it prints when it works. Terminal A, after the start-up lines of section 3: for Chert's waiting room `[worker] job ...:
joined room 'chert-warm-...'`, `[worker] published audio track coach-audio`, `[worker] warm job (chert_warm): no caller;
staying joined, nothing is said until a participant publishes video`; on your call `[worker] participant joined:
chert-bridge-<call_id>`, `[worker] bound to 'chert-bridge-<call_id>' ... 1280x720`, `[worker] first frame 1280x720`,
`[cue] HH:MM:SS [announce] "Let's do squats"` about 7 s after the accept, then the rep, count and `[status]` lines as in
section 3. Terminal B: `Connected to Chert. Call readiness is reported separately.`, then
`"<name>" joined a waiting room. Media readiness is not verified.`, and after you hang up `Call ended: room removed.`

Caveat: Chert's documented starter is FaceTime Audio only and the docs say video "require[s] separate development and
validation". In testing, a FaceTime Video call through `chert dev` on our dev line reached exactly this worker at
1280x720, 15 fps, and ran the whole plan, so the CLI path passes video on a validated line; whether your workshop line is validated is the organiser's question to Chert. If your call connects
but Terminal A never prints `bound to`, the line is not passing video: tell the organiser and iterate on recordings
(section 6) and the phone page (section 3) meanwhile.

The rules of the line:

- One line = one connected CLI = one active call at a time; more agent names add no capacity. Start Terminal A before
  B: the call is only accepted once an agent participant is in the room, and the worker needs its models preloaded.
- `chert dev` prepares and refills waiting rooms with no caller, and `chert doctor --agent` dispatches a real job. The
  worker treats both as warm jobs: it joins, publishes its silent track, says nothing and starts no script until video
  arrives. Keep it that way if you touch `worker.py`: never greet or start the plan on job start.
- The line is Offline for a few minutes after each call: on our dev line Chert's worker showed "Offline, no
  recent successful command poll" after every call, and calls in that window ring out; the line refreshes on Chert's
  next cron cycle (`chert doctor` can also report the machine behind the line offline for a while). The
  CLI docs expect a second call right after cleanup, so try it, but do not plan on it: recordings are your inner loop
  (section 6) and the phone page costs nothing per attempt (section 3); the line is for checking.
- After changing code, restart both terminals so no prepared room runs old code: end the call, Ctrl+C in Terminal B
  and wait for `Stopped. The line no longer routes to this agent.`, then stop Terminal A, then start A, then B.
- Handoff at the end of your slot: the caller hangs up, you wait for `Call ended: room removed.`, Ctrl+C `chert dev`
  and wait for the release, then stop the agent. The next team runs `chert doctor` before starting. If cleanup or
  release stays pending, hold the handoff, keep `~/.chert` intact, do not delete state, and contact the organiser.
  `--takeover` replaces another team's session and is not yours to run.
- When `chert doctor` fails: `project_not_signed_in`, run `chert login` for the right project (and `chert setup --clear`
  if a stale saved setup is reported); `project_role_insufficient`, ask the organiser for owner/admin; no line or more
  than one line, ask the organiser for exactly one; busy line or offline worker, wait or ask, never force it; missing
  LiveKit keys, run from the folder with `.env.local` and check for stale shell `LIVEKIT_*` overrides. Line provisioning
  and media troubleshooting belong to the organiser and Chert, not to you.

# Framing, latency, limits, privacy, licence

Iterating from recordings, framing, latency, what the coach cannot do, telemetry, the licence, and getting help.

## 6. Iterate without calling

Any video of a person goes through the same perception and session code and writes the same event stream:
```
python -m facetime.replay_local --video my_session.mp4 --plan "squats:10,pushups:8" --fast --events events.jsonl
```
It prints every cue with its clip time, and `events.jsonl` is schema v1 (section 1 of [/hackathon/events](/hackathon/events)) with `t`
equal to the clip time; `--fast` skips wall-clock pacing; `--exercise squats` for one exercise, `--gesture-next` to
try the T-pose. Record with your phone in portrait, propped as in section 7, whole body, 2 to 3 m away; walk in,
stand still for a second, then do the set. Convert the clip to 10 fps first: the replay treats every file as 10 fps and loads every frame into memory (a raw
30 fps phone clip runs as 3x slow motion and two minutes of 1080p is 20 GB of frames):
`ffmpeg -i IMG_1234.mov -vf "fps=10,scale='if(gt(iw,ih),-2,288)':'if(gt(iw,ih),288,-2)'" -an -c:v libx264 -preset fast -crf 23 data/tmp/mine.mp4`
(the pipeline resizes to short side 288 itself, the scale only keeps the file small). `replay_local` does not run
`handlers.py`: feed the events file to your `on_event` with the ten-line harness in [/hackathon/build](/hackathon/build) section 3,
then check once on the phone page (section 3) or, for the three Chert teams, on the line (section 5).

Three sample recordings ship in the bundle, already cropped to the camera view
(portrait 288 px, 10 fps) at `data/tmp/mac_handoff/crops/`: `run3.mp4` (a clean full plan on the LiveKit phone page),
`run2.mp4` (end of squats, pushups with phone handling, 20 jacks, 10 lunges) and `live.mp4` (the full plan on a real
FaceTime call), plus per-exercise cuts in `data/tmp/mac_handoff/segments/`. Their event logs sit beside them as `<name>.events.jsonl`, written by an earlier policy version: the current code
writes fewer `cue` events on the same crop (run3: 18 instead of 27; the nine that went are a standing "Sink lower" before
the first count and idle praise during the uncounted pushup set; reps and session states are identical). Regenerate them
with the command above (`--video data/tmp/mac_handoff/crops/run3.mp4`, about 60 s) rather than diffing against the
shipped files.

Latency from a recording: film the demo phone's screen and speaker with a second phone (a FaceTime screen recording has
no audio), then `python scripts/e2e_from_recording.py film.mp4 --crop W:H:X:Y --t0 11 --t1 36 --exercise squats`
prints rep bottom to count heard per rep and the p50 for one set.

## 7. Framing and physics

- Phone propped at knee to waist height, 2 to 3 m from the floor spot, whole body in frame with the feet above the
  bottom edge and room above the head, a plain wall behind if you can. A laptop camera cannot frame a standing adult
  top to bottom at that distance; a phone in portrait can.
- On the phone page the camera reaches the coach as the phone holds it, 720x1280 portrait at up to 30 fps, sampled at
  10 fps. On a Chert line FaceTime video arrives as 1280x720 landscape at 15 fps whatever the phone's orientation (Fill
  framing can crop a portrait phone to 16:9 and cut the feet or head). The pose works on both; keep the body central.
- Step into place before the start line and stand still for a second; do not reach for the phone mid-set.
- Pushups need the phone low and side-on to the floor spot, about 2 m away. Lunges count one per leg (ten = five pairs).
- If the head or feet leave the frame the coach says "Step back so I can see your whole body" (at most every 20 s),
  pauses counting and the set clock, and resumes with "Great, I can see you".
- Arm raise (the latency probe, `--latency-probe`) and T-pose (`--gesture-next`): both are off by default. The probe
  never acts during a set; between sets, keep both hands below the nose for a full second, then raise both arms (one
  beep per raise). The T-pose closes the current set or skips a rest: hold both arms straight out at shoulder height
  for two seconds, facing the camera, and drop the arms once it says "Okay, moving on"; it cannot fire again for 5 s,
  and after "All done" there is nothing left to move on to. Alternating raises and Ts with no pause between them fires
  nothing.

## 8. Latency you can count on

Two classes. Fast path (bank): decision to heard about 0.2 s (145 to 225 ms over the SFU); rep bottom to count heard
about 0.8 s (0.81 s p50 measured on a phone-page recording, 0.72 to 0.88; about 0.3 s of it is the counter's
reversal detection; the two FaceTime legs add a little more by ear). Anything through the LLM: 0.8 to 1.2 s to first audio for every 2026
realtime model, so a count spoken by the LLM lands 1.4 to 1.9 s after the bottom, one to two reps late at jacks cadence
(the Gemini-in-the-loop baseline matched 0 of 16 counts). Put counts and form cues on the fast path, conversation on
the LLM. Perception itself costs about 40 ms receive-to-decision at 10 fps on 4 CPU cores.

## 9. What the coach is good and bad at

Good: counting squats, pushups, lunges and jumping jacks from a propped phone with the whole body in frame; a plan that
announces each exercise, counts to a target and rests; corrections for the common faults (depth, back, knees, pace);
side cues for one-sided stretches; the step-back line; a count never stuck behind a correction, and no correction after
the set has closed.
Bad: shallow reps can be undercounted (counter exact 0.87 on MM-Fit, 0.59 against coaches' spoken numbers on
home sessions); corrections agree with the human coach about 44 percent of the time when both spoke, are generic ("Sink
lower") rather than specific, and about 5 per minute are not matched by a coach's; auto exercise detection
(`exercise_seen`, `--auto`) is about 0.75 accurate and needs 3 s, so trust your plan and keep plan mode; the false reps
that remain: pushups from a propped front camera can read +2 at a bad floor angle, jacks +1 on hands to the head after
a set, a person who walks in and squats within a second loses that first rep, and the first count needs 0.8 s of
standing still; a correction decided before a set's first count waits for "One" and is dropped if no rep comes, and no praise is spoken before the first rep; jumps at the top of the frame
can flicker `partial` and pause a set; the T-pose ("Okay, moving on", `--gesture-next`) has fired on a device only after
"All done", with no set left to move on from, so mid-set use is untested on a call; it needs the 1.5 s hold and 5 s between fires (section 7)
and is ignored by design while the whole body is not visible.

## 10. Telemetry and privacy

By default the worker writes two local files and nothing else: `data/tmp/worker_events.jsonl` (every event plus `room`,
`call_id`, `job`; `--event-log` / `RTCOACH_EVENT_LOG`) and `data/tmp/worker_cues.jsonl` (cues played or dropped;
`--cue-log` / `RTCOACH_CUE_LOG`). No frames, no thumbnails, no caller audio, no transcripts, no FaceTime handles, no
tokens: the only caller identifier is the Chert bridge's `call_id`, and on the phone page there is none (`call_id` is
null, the participant is plain `chert-bridge`). Opt out: `--event-log /dev/null --cue-log /dev/null`.
Brodie asks you to leave the logs on and hand them in after the event (the same files, keyed on the room name and the
time of the call; on a Chert line also on `call_id`). The Gemini
Live layer (`--voice gemini`) logs its own turns as `voice` events and keeps the caller's words out of every sink unless
you opt in with `--log-caller`; caller audio is never written anywhere by this code. Tell your callers one
line before the set: "This call is coached by software: it reads your body pose from the video and keeps a log of rep
counts and cues, not the video or the audio."

## 11. Licence and attribution

- `clf_pose.pt`, `trigger_lr.json`, `thresholds_e3c.json` and `cue_prior.json` are results of a 30-day QEVD evaluation
  licence and ship with the starter under its terms: keep them private (no public GitHub, no LFS, no public URL), never
  identify a participant, no face recognition. The cue wording in `src/rtcoach/cues.json` and the audio bank are ours.
- What ships from the QEVD taxonomy is its fault label names, used verbatim as keys in `src/rtcoach/cues.json`, the
  thresholds files and `cue_prior.json` (for example `squats - knees over toes`), and nothing else. No QEVD clips, frames,
  annotations or coach phrases are included, and none may be requested; the sample recordings are our own, used with consent. The
  counter benchmark data (MM-Fit, CC BY 4.0) is not included either.
- MediaPipe (pose models and SDK): Apache License 2.0. LiveKit Agents SDK and client: Apache License 2.0. Your LiveKit
  Cloud project: your own account under LiveKit's terms. The Chert CLI and FaceTime line (three teams): your organiser's
  agreement with Chert.

## 12. Getting help

Paste, in this order: the Terminal A lines from `[worker] prewarm` through the failure and the last `[status]` line; the
room name you minted and the time of the attempt; what the phone page shows in its log panel and on its "Coach audio" and
"Coach present" lines (remember that "Coach present" stays "no" under `dev`, section 3); and whether `curl` against the
relay's `/health` answers (section 4). A Chert team adds the full `chert doctor` output, `node --version`, `chert --version`
and its `AGENT_NAME`. Never paste tokens, API keys, `.env.local`, the phone link (it carries your token), or an unsanitised
log. Your support contact is on the card.