A. Room lifecycle (总管 review items A-1/A-2): the single cleanup owner now watches
ACTIVE session deadlines as well as debts - a session nobody touches expires into
the ledger and has its room really deleted with zero further requests (old code:
_cleanup_loop only scanned debts, expired rooms were never deleted until some
request ran _prune). Explicit end and the background sweep share ONE persisted
attempt/age cap and abandoned state - _sessions no longer bypasses it (the
manager measured 2 background attempts + 3 more interactive deletes = 5); an
abandoned debt raises CleanupAbandoned -> 503 with room/attempts/reason instead
of pretending done-404, success clears both _sessions and _debts, restart
continues the same persisted cap, close/cancel leaves no dangling coroutine, and
only this instance's own rooms are ever touched. New wake event makes a fresh
debt retry immediately instead of waiting out a deadline sleep.
B. Queued speech turns bind EXACTLY (总管 interleaving note_handle(H1)->H2->
note_stream(S1)->S2 gave H1=None, H2='turn-one'): CyberGrillLLM.chat now reads
the SDK's own agents_speech_handle ContextVar (livekit-agents 1.8.8 sets it in
_create_speech_task around every speech pipeline; source-verified on .12 and
asserted at runtime) so each ChatApiLLMStream carries speech_handle/speech_id of
the very handle whose generation created it. TurnLedger binds only on that exact
attribution - a newer speech NEVER retires an older pending handle, say/proactive
generations bind nobody, missing attribution fails closed (no ack, no subtitle
turn), interrupt/failure discards only its own binding.
C. Versioned subtitle contract v2 for the visual author's wiring card (HEL-898):
every cg.subtitle payload gains {v:2, seq} (session-monotonic), turn identity
(speech_id, turn_id, same-source message_id for proactive rows) and explicit
lifecycle events - turn_start/assistant_interim/assistant_final/turn_end/
turn_cancel(reason) plus proactive_start/proactive_end/proactive_cancel that
ride the say SpeechHandle instead of the receipt branch. Accumulators are
isolated per speech_id (a late delta after cancel is dropped producer-side);
seq + speech_id give the frontend two independent stale-rejection rules. Full
executable contract: docs/voice-subtitle-contract-v2.md with recorded samples in
docs/rework/hel879/subtitle-samples/ (scripts/gen_subtitle_samples_hel879.py).
web/src untouched.
Verification: the .12 real-SDK check now LOADS THE ACTUAL WORKER MODULE and drives
the callbacks IT registered on a real AgentSession (no hand-copied functions) -
both-queued delayed generations bind exactly, a say sandwiched between user turns
steals nothing, unattributed generations fail closed, interrupted turns cancel
without final/receipt (11/11 ALL PASS, source-level proof of the ContextVar wiring
included; evidence docs/rework/hel879/sdk-check-hel879-remaining.log). The old r2
lifespan test flake is root-caused (clock freeze and block were two separate portal
calls; a 10ms life tick could slip between them and claim the frozen day unblocked)
and fixed atomically without raising timeouts or dropping assertions - 12/12 solo,
full suite 350/350 three times in a row on .venv311; ruff back to the single
pre-existing error; voice chain 8/8; release offline R1-R7 with zero failures;
requirement checks 4/4 (origin/main and c121c42, each with/without handoff).
Zero real LLM/embed/ASR/TTS calls, zero downloads/installs, nothing deployed or
merged, real caps stay 0.
Co-authored-by: multica-agent <github@multica.ai>
5.8 KiB
5.8 KiB
语音字幕契约 v2(cg.subtitle topic,HEL-879 收口交付)
面向对象:原视觉作者(接线卡 HEL-898)与后台。本文是可执行契约:每条 消息的字段、乱序/迟到保护规则、每种场景的完整录制样例 (
docs/rework/hel879/subtitle-samples/*.jsonl,由scripts/gen_subtitle_samples_hel879.py离线生成)。后台实现:src/cybergril/voice/seam.py(协议与累积器)+worker.py(事件接线,bind_voice_session唯一注册点)。v1 的kind/role/text三字段消息全部 向下兼容:v2 只做增字段(v/seq/speech_id/turn_id/message_id/reason) 与增事件(起止/取消),旧 kind 语义不变。
1. 信封(所有消息共用)
{"v": 2, "seq": 7, "kind": "...", "role": "user|assistant", "text": "...",
"speech_id": "speech_xxx", "turn_id": "uuid?", "message_id": 9?, "reason": "..."?}
v:契约版本,恒为2。前端见v !== 2整条丢弃(fail-closed)。seq:会话内单调递增(一个 worker job 一个序列)。任何seq <= 已应用的最大 seq的消息直接丢弃——重放、迟到旧轮、跨回调乱序全被这一条 挡住。speech_id:SDKSpeechHandle.id,一次「她开口」的唯一标识(用户话轮的 回复、主动播报各是独立 speech)。后台把它作为字幕轮次键;前端以它隔离 累积状态。空speech_id的 turn 类消息不存在(后台不会发)。turn_id:服务端话轮 id(仅assistant_final/turn_end,来自 /api/chat 回执归属);message_id:主动播报行的同源消息 id(仅proactive_*)。reason:turn_cancel/proactive_cancel的取消原因 (interrupted|failed|no-generation|error_phi)。
2. 事件种类与生命周期
用户侧(不变,仅加信封)
interim:ASR 草稿(替换语义,text为截至当前全句)。user_final:纠错终稿。
她的回复轮(新:显式起止 + 确切归属 + 乱序保护)
turn_start{speech_id}
→ assistant_interim{speech_id, text=累计全句} # 0..n 次,替换语义
→ assistant_final{speech_id, turn_id, text} # 已落库的完整回复
→ turn_end{speech_id, turn_id} # 轮关闭
- 打断/失败/无回执:没有 final,直接
turn_cancel{speech_id, reason=interrupted|failed|no-generation}。 前端必须把无 final 收尾的 interim 按「这轮没有成为她的正式回复」处理 (残影不越过新用户轮)。 - φ 失败:
error{speech_id, text=φ原句}后跟turn_cancel{reason=error_phi}; error 不落正式记录。 - 迟到保护(前端规则):turn 类消息按
speech_id只允许作用于「当前 打开的轮」;收到turn_end/turn_cancel后同speech_id的后续 interim/final 丢弃(seq 规则已再挡一层)。打断后晚到的旧增量被两条规则 共同拒绝——这正是 v1 kind/role/text 做不到的。
主动开口(CG-011 最小接缝,默认关)
proactive_start{speech_id, message_id, text=原文}
→ proactive_end{speech_id, message_id} # 完整念完
| proactive_cancel{speech_id, message_id, reason=interrupted|failed}
message_id与文字收件箱同一行(同源可对齐);字幕生命周期挂在 say 的 SpeechHandle 上,不借用回执分支(无 LM 流也照发起止事件)。- 语义边界:这是「会话开时念已落库的旧 proactive 留言」(每会话 ≤1 条,
CYBERGRIL_PROACTIVE_VOICE=1才启用),不是会话中完整 CG-011 主动 表达——后者继续承接,不得冒充完成。未读/已读仍归页面,worker 不代标。
3. 前端实现要求(HEL-898,不改已认可视觉)
web/src/voice/session.ts:按 §2 生命周期维护「当前 assistant 轮」 (speech_id键 +seq闸);turn_start开轮、interim 替换渲染、 final 原子替换、turn_end/turn_cancel关轮清残影;挂断全清。assistant_interim与assistant_final同一气泡位(替换式过渡), 与用户interim视觉可区分;不进入正式聊天记录(只有 final 落库)。proactive_*复用「她的回复」位或留言卡语义由视觉作者定;message_id供未读联动,不代标已读。- v1 兼容期:页面可以先按 v1 消费(忽略新字段),接线卡再切 v2 生命周期。
4. 录制样例(验收对照)
docs/rework/hel879/subtitle-samples/(离线生成,零真实调用):
normal-turn.jsonl:用户 interim→final→她的 turn_start→2×interim→ final→turn_end(seq 连续单调)。interrupt.jsonl:interim 后用户打断→turn_cancel(interrupted);新用户轮 正常开始——旧轮无 final。proactive.jsonl:proactive_start→proactive_end 与打断版 proactive_cancel,均带同源 message_id。
真 SDK 驱动(.12, livekit-agents 1.8.8, 假 SSE/假媒体):scripts/ gen_hel877_sdk_check.py 第 9 项加载真实 worker 模块、在真 AgentSession 上
触发其注册回调,覆盖「双 handle 先排队后生成」「say 夹在用户轮」「无归属
fail-closed」「打断取消」——与本文契约逐字段一致。
5. 后台保证(已实现并测)
- 每个
assistant_interim/final的speech_id来自 SDK 生成上下文的 确切归属(agents_speech_handleContextVar),不按事件先后猜;无归属 的流不开轮、不绑回执(fail-closed)。 - 累积器按
speech_id隔离,轮关闭后同 id 增量在后台即被丢弃(生产端先挡 一层);seq由单点SubtitleBus发号,会话内严格单调。 - 回执确认仍只发生在「真实播完 + 未打断 + 有完整回执 + epoch 一致」
(
should_ack),字幕事件与回执互相独立:主动播报永不走回执分支。