Files
CyberGril/docs/voice-subtitle-contract-v2.md
T
zonggongandmultica-agent 3228b1e0d7 HEL-879 remaining closure: auto-expiry cleanup ownership, exact SDK speech-turn attribution, and the versioned v2 subtitle contract for HEL-898
A. Room lifecycle (总管 review items A-1/A-2): the single cleanup owner now watches
   ACTIVE session deadlines as well as debts - a session nobody touches expires into
   the ledger and has its room really deleted with zero further requests (old code:
   _cleanup_loop only scanned debts, expired rooms were never deleted until some
   request ran _prune). Explicit end and the background sweep share ONE persisted
   attempt/age cap and abandoned state - _sessions no longer bypasses it (the
   manager measured 2 background attempts + 3 more interactive deletes = 5); an
   abandoned debt raises CleanupAbandoned -> 503 with room/attempts/reason instead
   of pretending done-404, success clears both _sessions and _debts, restart
   continues the same persisted cap, close/cancel leaves no dangling coroutine, and
   only this instance's own rooms are ever touched. New wake event makes a fresh
   debt retry immediately instead of waiting out a deadline sleep.

B. Queued speech turns bind EXACTLY (总管 interleaving note_handle(H1)->H2->
   note_stream(S1)->S2 gave H1=None, H2='turn-one'): CyberGrillLLM.chat now reads
   the SDK's own agents_speech_handle ContextVar (livekit-agents 1.8.8 sets it in
   _create_speech_task around every speech pipeline; source-verified on .12 and
   asserted at runtime) so each ChatApiLLMStream carries speech_handle/speech_id of
   the very handle whose generation created it. TurnLedger binds only on that exact
   attribution - a newer speech NEVER retires an older pending handle, say/proactive
   generations bind nobody, missing attribution fails closed (no ack, no subtitle
   turn), interrupt/failure discards only its own binding.

C. Versioned subtitle contract v2 for the visual author's wiring card (HEL-898):
   every cg.subtitle payload gains {v:2, seq} (session-monotonic), turn identity
   (speech_id, turn_id, same-source message_id for proactive rows) and explicit
   lifecycle events - turn_start/assistant_interim/assistant_final/turn_end/
   turn_cancel(reason) plus proactive_start/proactive_end/proactive_cancel that
   ride the say SpeechHandle instead of the receipt branch. Accumulators are
   isolated per speech_id (a late delta after cancel is dropped producer-side);
   seq + speech_id give the frontend two independent stale-rejection rules. Full
   executable contract: docs/voice-subtitle-contract-v2.md with recorded samples in
   docs/rework/hel879/subtitle-samples/ (scripts/gen_subtitle_samples_hel879.py).
   web/src untouched.

Verification: the .12 real-SDK check now LOADS THE ACTUAL WORKER MODULE and drives
the callbacks IT registered on a real AgentSession (no hand-copied functions) -
both-queued delayed generations bind exactly, a say sandwiched between user turns
steals nothing, unattributed generations fail closed, interrupted turns cancel
without final/receipt (11/11 ALL PASS, source-level proof of the ContextVar wiring
included; evidence docs/rework/hel879/sdk-check-hel879-remaining.log). The old r2
lifespan test flake is root-caused (clock freeze and block were two separate portal
calls; a 10ms life tick could slip between them and claim the frozen day unblocked)
and fixed atomically without raising timeouts or dropping assertions - 12/12 solo,
full suite 350/350 three times in a row on .venv311; ruff back to the single
pre-existing error; voice chain 8/8; release offline R1-R7 with zero failures;
requirement checks 4/4 (origin/main and c121c42, each with/without handoff).
Zero real LLM/embed/ASR/TTS calls, zero downloads/installs, nothing deployed or
merged, real caps stay 0.

Co-authored-by: multica-agent <github@multica.ai>
2026-10-27 05:10:52 +08:00

5.8 KiB
Raw Blame History

语音字幕契约 v2(cg.subtitle topic,HEL-879 收口交付)

面向对象:原视觉作者(接线卡 HEL-898)与后台。本文是可执行契约:每条 消息的字段、乱序/迟到保护规则、每种场景的完整录制样例 (docs/rework/hel879/subtitle-samples/*.jsonl,由 scripts/gen_subtitle_samples_hel879.py 离线生成)。后台实现: src/cybergril/voice/seam.py(协议与累积器)+ worker.py(事件接线, bind_voice_session 唯一注册点)。v1 的 kind/role/text 三字段消息全部 向下兼容:v2 只做增字段(v/seq/speech_id/turn_id/message_id/reason) 与增事件(起止/取消),旧 kind 语义不变。

1. 信封(所有消息共用)

{"v": 2, "seq": 7, "kind": "...", "role": "user|assistant", "text": "...",
 "speech_id": "speech_xxx", "turn_id": "uuid?", "message_id": 9?, "reason": "..."?}
  • v:契约版本,恒为 2。前端见 v !== 2 整条丢弃(fail-closed)。
  • seq:会话内单调递增(一个 worker job 一个序列)。任何 seq <= 已应用的最大 seq 的消息直接丢弃——重放、迟到旧轮、跨回调乱序全被这一条 挡住。
  • speech_id:SDK SpeechHandle.id,一次「她开口」的唯一标识(用户话轮的 回复、主动播报各是独立 speech)。后台把它作为字幕轮次键;前端以它隔离 累积状态。空 speech_id 的 turn 类消息不存在(后台不会发)。
  • turn_id:服务端话轮 id(仅 assistant_final/turn_end,来自 /api/chat 回执归属);message_id:主动播报行的同源消息 id(仅 proactive_*)。
  • reason:turn_cancel/proactive_cancel 的取消原因 (interrupted|failed|no-generation|error_phi)。

2. 事件种类与生命周期

用户侧(不变,仅加信封)

  • interim:ASR 草稿(替换语义,text 为截至当前全句)。
  • user_final:纠错终稿。

她的回复轮(新:显式起止 + 确切归属 + 乱序保护)

turn_start{speech_id}
  → assistant_interim{speech_id, text=累计全句}   # 0..n 次,替换语义
  → assistant_final{speech_id, turn_id, text}     # 已落库的完整回复
  → turn_end{speech_id, turn_id}                  # 轮关闭
  • 打断/失败/无回执:没有 final,直接 turn_cancel{speech_id, reason=interrupted|failed|no-generation}。 前端必须把无 final 收尾的 interim 按「这轮没有成为她的正式回复」处理 (残影不越过新用户轮)。
  • φ 失败:error{speech_id, text=φ原句} 后跟 turn_cancel{reason=error_phi}; error 不落正式记录。
  • 迟到保护(前端规则):turn 类消息按 speech_id 只允许作用于「当前 打开的轮」;收到 turn_end/turn_cancel 后同 speech_id 的后续 interim/final 丢弃(seq 规则已再挡一层)。打断后晚到的旧增量被两条规则 共同拒绝——这正是 v1 kind/role/text 做不到的。

主动开口(CG-011 最小接缝,默认关)

proactive_start{speech_id, message_id, text=原文}
  → proactive_end{speech_id, message_id}          # 完整念完
  | proactive_cancel{speech_id, message_id, reason=interrupted|failed}
  • message_id 与文字收件箱同一行(同源可对齐);字幕生命周期挂在 say 的 SpeechHandle 上,不借用回执分支(无 LM 流也照发起止事件)。
  • 语义边界:这是「会话开时念已落库的旧 proactive 留言」(每会话 ≤1 条, CYBERGRIL_PROACTIVE_VOICE=1 才启用),不是会话中完整 CG-011 主动 表达——后者继续承接,不得冒充完成。未读/已读仍归页面,worker 不代标。

3. 前端实现要求(HEL-898,不改已认可视觉)

  1. web/src/voice/session.ts:按 §2 生命周期维护「当前 assistant 轮」 (speech_id 键 + seq 闸);turn_start 开轮、interim 替换渲染、 final 原子替换、turn_end/turn_cancel 关轮清残影;挂断全清。
  2. assistant_interim 与 assistant_final 同一气泡位(替换式过渡), 与用户 interim 视觉可区分;不进入正式聊天记录(只有 final 落库)。
  3. proactive_* 复用「她的回复」位或留言卡语义由视觉作者定; message_id 供未读联动,不代标已读。
  4. v1 兼容期:页面可以先按 v1 消费(忽略新字段),接线卡再切 v2 生命周期。

4. 录制样例(验收对照)

docs/rework/hel879/subtitle-samples/(离线生成,零真实调用):

  • normal-turn.jsonl:用户 interim→final→她的 turn_start→2×interim→ final→turn_end(seq 连续单调)。
  • interrupt.jsonl:interim 后用户打断→turn_cancel(interrupted);新用户轮 正常开始——旧轮无 final。
  • proactive.jsonl:proactive_start→proactive_end 与打断版 proactive_cancel,均带同源 message_id。

真 SDK 驱动(.12, livekit-agents 1.8.8, 假 SSE/假媒体):scripts/ gen_hel877_sdk_check.py 第 9 项加载真实 worker 模块、在真 AgentSession 上 触发其注册回调,覆盖「双 handle 先排队后生成」「say 夹在用户轮」「无归属 fail-closed」「打断取消」——与本文契约逐字段一致。

5. 后台保证(已实现并测)

  • 每个 assistant_interim/final 的 speech_id 来自 SDK 生成上下文的 确切归属(agents_speech_handle ContextVar),不按事件先后猜;无归属 的流不开轮、不绑回执(fail-closed)。
  • 累积器按 speech_id 隔离,轮关闭后同 id 增量在后台即被丢弃(生产端先挡 一层);seq 由单点 SubtitleBus 发号,会话内严格单调。
  • 回执确认仍只发生在「真实播完 + 未打断 + 有完整回执 + epoch 一致」 (should_ack),字幕事件与回执互相独立:主动播报永不走回执分支。