Framing, latency, limits, privacy, licence

6. Iterate without calling

Any video of a person goes through the same perception and session code and writes the same event stream:

python -m facetime.replay_local --video my_session.mp4 --plan "squats:10,pushups:8" --fast --events events.jsonl

It prints every cue with its clip time, and events.jsonl is schema v1 (section 1 of /hackathon/events) with t equal to the clip time; --fast skips wall-clock pacing; --exercise squats for one exercise, --gesture-next to try the T-pose. Record with your phone in portrait, propped as in section 7, whole body, 2 to 3 m away; walk in, stand still for a second, then do the set. Convert the clip to 10 fps first: the replay treats every file as 10 fps and loads every frame into memory (a raw 30 fps phone clip runs as 3x slow motion and two minutes of 1080p is 20 GB of frames): ffmpeg -i IMG_1234.mov -vf "fps=10,scale='if(gt(iw,ih),-2,288)':'if(gt(iw,ih),288,-2)'" -an -c:v libx264 -preset fast -crf 23 data/tmp/mine.mp4 (the pipeline resizes to short side 288 itself, the scale only keeps the file small). replay_local does not run handlers.py: feed the events file to your on_event with the ten-line harness in /hackathon/build section 3, then check once on the phone page (section 3) or, for the three Chert teams, on the line (section 5).

Three sample recordings ship in the bundle, already cropped to the camera view (portrait 288 px, 10 fps) at data/tmp/mac_handoff/crops/: run3.mp4 (a clean full plan on the LiveKit phone page), run2.mp4 (end of squats, pushups with phone handling, 20 jacks, 10 lunges) and live.mp4 (the full plan on a real FaceTime call), plus per-exercise cuts in data/tmp/mac_handoff/segments/. Their event logs sit beside them as <name>.events.jsonl, written by an earlier policy version: the current code writes fewer cue events on the same crop (run3: 18 instead of 27; the nine that went are a standing "Sink lower" before the first count and idle praise during the uncounted pushup set; reps and session states are identical). Regenerate them with the command above (--video data/tmp/mac_handoff/crops/run3.mp4, about 60 s) rather than diffing against the shipped files.

Latency from a recording: film the demo phone's screen and speaker with a second phone (a FaceTime screen recording has no audio), then python scripts/e2e_from_recording.py film.mp4 --crop W:H:X:Y --t0 11 --t1 36 --exercise squats prints rep bottom to count heard per rep and the p50 for one set.

7. Framing and physics

• Phone propped at knee to waist height, 2 to 3 m from the floor spot, whole body in frame with the feet above the bottom edge and room above the head, a plain wall behind if you can. A laptop camera cannot frame a standing adult top to bottom at that distance; a phone in portrait can.
• On the phone page the camera reaches the coach as the phone holds it, 720x1280 portrait at up to 30 fps, sampled at 10 fps. On a Chert line FaceTime video arrives as 1280x720 landscape at 15 fps whatever the phone's orientation (Fill framing can crop a portrait phone to 16:9 and cut the feet or head). The pose works on both; keep the body central.
• Step into place before the start line and stand still for a second; do not reach for the phone mid-set.
• Pushups need the phone low and side-on to the floor spot, about 2 m away. Lunges count one per leg (ten = five pairs).
• If the head or feet leave the frame the coach says "Step back so I can see your whole body" (at most every 20 s), pauses counting and the set clock, and resumes with "Great, I can see you".
• Arm raise (the latency probe, --latency-probe) and T-pose (--gesture-next): both are off by default. The probe never acts during a set; between sets, keep both hands below the nose for a full second, then raise both arms (one beep per raise). The T-pose closes the current set or skips a rest: hold both arms straight out at shoulder height for two seconds, facing the camera, and drop the arms once it says "Okay, moving on"; it cannot fire again for 5 s, and after "All done" there is nothing left to move on to. Alternating raises and Ts with no pause between them fires nothing.

8. Latency you can count on

Two classes. Fast path (bank): decision to heard about 0.2 s (145 to 225 ms over the SFU); rep bottom to count heard about 0.8 s (0.81 s p50 measured on a phone-page recording, 0.72 to 0.88; about 0.3 s of it is the counter's reversal detection; the two FaceTime legs add a little more by ear). Anything through the LLM: 0.8 to 1.2 s to first audio for every 2026 realtime model, so a count spoken by the LLM lands 1.4 to 1.9 s after the bottom, one to two reps late at jacks cadence (the Gemini-in-the-loop baseline matched 0 of 16 counts). Put counts and form cues on the fast path, conversation on the LLM. Perception itself costs about 40 ms receive-to-decision at 10 fps on 4 CPU cores.

9. What the coach is good and bad at

Good: counting squats, pushups, lunges and jumping jacks from a propped phone with the whole body in frame; a plan that announces each exercise, counts to a target and rests; corrections for the common faults (depth, back, knees, pace); side cues for one-sided stretches; the step-back line; a count never stuck behind a correction, and no correction after the set has closed. Bad: shallow reps can be undercounted (counter exact 0.87 on MM-Fit, 0.59 against coaches' spoken numbers on home sessions); corrections agree with the human coach about 44 percent of the time when both spoke, are generic ("Sink lower") rather than specific, and about 5 per minute are not matched by a coach's; auto exercise detection (exercise_seen, --auto) is about 0.75 accurate and needs 3 s, so trust your plan and keep plan mode; the false reps that remain: pushups from a propped front camera can read +2 at a bad floor angle, jacks +1 on hands to the head after a set, a person who walks in and squats within a second loses that first rep, and the first count needs 0.8 s of standing still; a correction decided before a set's first count waits for "One" and is dropped if no rep comes, and no praise is spoken before the first rep; jumps at the top of the frame can flicker partial and pause a set; the T-pose ("Okay, moving on", --gesture-next) has fired on a device only after "All done", with no set left to move on from, so mid-set use is untested on a call; it needs the 1.5 s hold and 5 s between fires (section 7) and is ignored by design while the whole body is not visible.

10. Telemetry and privacy

By default the worker writes two local files and nothing else: data/tmp/worker_events.jsonl (every event plus room, call_id, job; --event-log / RTCOACH_EVENT_LOG) and data/tmp/worker_cues.jsonl (cues played or dropped; --cue-log / RTCOACH_CUE_LOG). No frames, no thumbnails, no caller audio, no transcripts, no FaceTime handles, no tokens: the only caller identifier is the Chert bridge's call_id, and on the phone page there is none (call_id is null, the participant is plain chert-bridge). Opt out: --event-log /dev/null --cue-log /dev/null. Brodie asks you to leave the logs on and hand them in after the event (the same files, keyed on the room name and the time of the call; on a Chert line also on call_id). The Gemini Live layer (--voice gemini) logs its own turns as voice events and keeps the caller's words out of every sink unless you opt in with --log-caller; caller audio is never written anywhere by this code. Tell your callers one line before the set: "This call is coached by software: it reads your body pose from the video and keeps a log of rep counts and cues, not the video or the audio."

11. Licence and attribution

• clf_pose.pt, trigger_lr.json, thresholds_e3c.json and cue_prior.json are results of a 30-day QEVD evaluation licence and ship with the starter under its terms: keep them private (no public GitHub, no LFS, no public URL), never identify a participant, no face recognition. The cue wording in src/rtcoach/cues.json and the audio bank are ours.
• What ships from the QEVD taxonomy is its fault label names, used verbatim as keys in src/rtcoach/cues.json, the thresholds files and cue_prior.json (for example squats - knees over toes), and nothing else. No QEVD clips, frames, annotations or coach phrases are included, and none may be requested; the sample recordings are our own, used with consent. The counter benchmark data (MM-Fit, CC BY 4.0) is not included either.
• MediaPipe (pose models and SDK): Apache License 2.0. LiveKit Agents SDK and client: Apache License 2.0. Your LiveKit Cloud project: your own account under LiveKit's terms. The Chert CLI and FaceTime line (three teams): your organiser's agreement with Chert.

12. Getting help

Paste, in this order: the Terminal A lines from [worker] prewarm through the failure and the last [status] line; the room name you minted and the time of the attempt; what the phone page shows in its log panel and on its "Coach audio" and "Coach present" lines (remember that "Coach present" stays "no" under dev, section 3); and whether curl against the relay's /health answers (section 4). A Chert team adds the full chert doctor output, node --version, chert --version and its AGENT_NAME. Never paste tokens, API keys, .env.local, the phone link (it carries your token), or an unsanitised log. Your support contact is on the card.