HEL-906 rework: persistence fail-closed stop + conservative pre-record accounting

- asr_usage: ledger write/read failure, corruption or mid-window vanish now
  stop all window outbound (fail-closed), preserve known consumption
  (monotonic, no zero-budget default), log + record guard_stop (durable
  across same-window restarts)
- asr_usage: conservative reserve before send (failed/cancelled sends are
  NOT refunded) + separately tracked confirmed-delivered seconds; replay
  restores both
- stt_funasr: reserve_audio before ws.send, note_audio_delivered after
  successful await only
- tests: 24 deterministic offline cases incl. manager's budget-regression
  repro, read/corrupt/missing rejection, record-then-raise/cancel fake ws,
  guard_stop across restart

Co-authored-by: multica-agent <github@multica.ai>
This commit is contained in:
总工
2026-10-28 09:02:51 +08:00
co-authored by multica-agent
parent ccfc04edcd
commit cf395d4e06
3 changed files with 469 additions and 84 deletions
+178 -43
View File
@@ -6,12 +6,18 @@
worker 的真实送数路径上:
- FunASRSTT(voice/stt_funasr.py)在每次 websocket 建连前 check_connect、
建连结果 note_connect_result、每个真实发送帧发送前 precheck_audio、
发送成功后 note_audio_sent;worker(voice/worker.py)按 job 注入 guard
并把 on_exhausted 接到 ctx.shutdown。
建连结果 note_connect_result、每个真实发送帧发送前 reserve_audio
(保守预记+越限拒绝)、``await ws.send`` 成功返回后 note_audio_delivered
(确证送达量);worker(voice/worker.py)按 job 注入 guard 并把
on_exhausted 接到 ctx.shutdown。
- 音频按**实际发送帧**的 sample_rate 与样本数计时(重采样后的输出帧按其
自身采样率计),同一帧重复发送按重复计;墙钟、开房/删房流水不参与
音频计时,事后成功流水也不追溯改数。
- **保守计费 / 确证送达双口径**(HEL-906 复核返工):音频在发送**前**
按整帧预记入预算(conservative),发送失败或被取消**不退还**——帧可能
已经出站;只有 ``await ws.send`` 成功返回的帧才累计确证送达量
(confirmed)。保守量是硬闸依据,确证量只是更低的下界核对;不得把
预记量宣称全部确证送达。
- 连接事实逐条落账:每次建连尝试(含失败)、失败后的重试(retry)、
接收侧死亡(connection_died)。FunASR 一句一连的惰性建连里,正常说完
一轮后的下一次建连不是 retry,只有上一次尝试失败/连接死亡/流异常结束
@@ -20,9 +26,10 @@ worker 的真实送数路径上:
账本(追加写 JSONL,默认 /opt/voice-agent-cybergril/asr-usage.jsonl)独立
于实例侧会话账本 ``<db>-voice-sessions.json``:删房成功会把通话从会话账
本移除,但这里的逐通历史(call_start/audio/connect_attempt/
connection_died/call_end/block)永不改写、不随删房消失;窗口总量由重放
账本恢复,同 window_id 重启进程不能重置预算,旧窗口耗尽状态也被持久化
(不得把旧余额或新开账当作新许可)。
connection_died/call_end/block/guard_stop)永不改写、不随删房消失;窗口
总量由重放账本恢复,同 window_id 重启进程不能重置预算,旧窗口耗尽状态
与持久化停机状态(guard_stop)也被持久化(不得把旧余额或新开账当作
新许可)。
窗口语义沿用 think/budget.py 的 fail-closed 先例:
@@ -32,16 +39,24 @@ connection_died/call_end/block)永不改写、不随删房消失;窗口总
在发送前拒绝(AsrWindowClosed),worker 经 on_exhausted 结束本通 job;
上限只能收紧不能放宽,候选值见 docs/rework/hel902/r5-followup-package.md §4。
**持久化 fail-closed(HEL-906 复核返工)**:账本写失败、读失败、行损坏
或账本中途消失时,guard 停止本窗口一切出站(新连接/新音频一律拒绝)、
保全内存已知消耗(窗口计数只增不减,禁止默认零预算或按旧账回退)、
记 error 日志与停机原因,并把 ``guard_stop`` 落账——同窗口重启进程重放
到该记录仍保持停机。故障下不继续通话(不要求支持)。
程序能保证 / 剩余只能监看(如实边界):
- 保证:布防进程内逐帧发送前硬停(帧不越上限一个字节)、逐通与全窗
计数不因删房/重启丢失、同窗重启不能重置预算。
- 保证:布防进程内逐帧发送前硬停(帧不越上限一个字节;保守预记含
发送失败/取消的不确定出站)、逐通与全窗计数不因删房/重启丢失、
同窗重启不能重置预算、持久化故障即停不回退。
- 跨进程有界收敛:账本是共同事实,每次建连检查与 resync_seconds 间隔
重放账本;两进程并行通话时对端最近 ≤ max(resync, flush) 间隔的音频
可能未计入本轮判定(有界过冲)。
- 只能监看:进程崩溃前 ≤ flush_seconds 的已发音频可能少计;接收侧死亡
未伴随发送失败时按 connection_died 计连接事实、不改已计音频;真实
FunASR/LiveKit 行为仍以窗口前 .12 实测为准。
可能未计入本轮判定(有界过冲)。**不承诺全窗口跨进程绝不越限**。
- 只能监看:进程崩溃前 ≤ flush_seconds 的已发音频可能少计(确证量同样
只在 flush 后落账);接收侧死亡未伴随发送失败时按 connection_died
计连接事实、不改已计音频;真实 FunASR/LiveKit 行为仍以窗口前 .12
实测为准。
"""
from __future__ import annotations
@@ -49,6 +64,7 @@ from __future__ import annotations
import argparse
import contextlib
import json
import logging
import os
import secrets
import time
@@ -59,6 +75,8 @@ DEFAULT_ASR_USAGE_LEDGER = "/opt/voice-agent-cybergril/asr-usage.jsonl"
DEFAULT_FLUSH_SECONDS = 5.0
DEFAULT_RESYNC_SECONDS = 5.0
_logger = logging.getLogger("cybergril.voice.asr_usage")
# 允许浮点求和后恰好等于上限(“≤上限”而不是“<上限”)
_EPSILON = 1e-9
@@ -117,7 +135,8 @@ def policy_from_env(environ: dict | None = None) -> AsrWindowPolicy:
@dataclass
class _AudioBuf:
seconds: float = 0.0
seconds: float = 0.0 # 保守预记(发送前计入,失败/取消不退还)
confirmed_seconds: float = 0.0 # 确证送达(await ws.send 成功返回)
frames: int = 0
bytes: int = 0
sample_rate: int = 0
@@ -129,7 +148,8 @@ class CallUsage:
connect_attempts: int = 0
connect_failures: int = 0
connect_retries: int = 0
audio_seconds: float = 0.0
audio_seconds: float = 0.0 # 保守计费口径(含不确定出站)
confirmed_audio_seconds: float = 0.0 # 确证送达口径(≤ audio_seconds)
retry_pending: bool = False # 上一次连接事实是失败 → 下次建连是 retry
_would_retry: bool = False # check_connect 为本次在途尝试标记的 retry 身份
@@ -137,21 +157,17 @@ class CallUsage:
@dataclass
class _ReplayState:
window_audio_seconds: float = 0.0
window_confirmed_seconds: float = 0.0
stopped: bool = False
stop_reason: str = ""
calls: dict[str, CallUsage] = field(default_factory=dict)
def summarize_ledger(path: Path | str, window_id: str | None = None) -> dict:
"""重放账本成汇总(操作员核对口径;不修改账本).
window_id 为 None 时按账本里出现过的窗口分组汇总。
"""
def _fold_lines(lines: list[str], window_id: str | None) -> tuple[dict[str, _ReplayState], int, int]:
"""把 JSONL 行折叠为 {window_id: _ReplayState};返回 (windows, records, parse_errors)."""
windows: dict[str, _ReplayState] = {}
records = 0
parse_errors = 0
try:
lines = Path(path).read_text(encoding="utf-8").splitlines()
except OSError:
lines = []
for line in lines:
line = line.strip()
if not line:
@@ -183,16 +199,42 @@ def summarize_ledger(path: Path | str, window_id: str | None = None) -> dict:
cu.retry_pending = True
elif kind == "audio":
seconds = float(rec.get("seconds", 0.0) or 0.0)
# 旧记录(返工前)只在发送成功后计 seconds,即全部确证送达
confirmed = rec.get("confirmed_seconds")
confirmed_s = seconds if confirmed is None else float(confirmed or 0.0)
cu.audio_seconds = round(cu.audio_seconds + seconds, 6)
cu.confirmed_audio_seconds = round(cu.confirmed_audio_seconds + confirmed_s, 6)
state.window_audio_seconds = round(state.window_audio_seconds + seconds, 6)
state.window_confirmed_seconds = round(
state.window_confirmed_seconds + confirmed_s, 6)
elif kind == "call_end" and str(rec.get("reason")) == "error":
cu.retry_pending = True
elif kind == "guard_stop" and not state.stopped:
state.stopped = True
state.stop_reason = str(rec.get("reason") or "ledger-stopped")
return windows, records, parse_errors
def summarize_ledger(path: Path | str, window_id: str | None = None) -> dict:
"""重放账本成汇总(操作员核对口径;不修改账本).
window_id 为 None 时按账本里出现过的窗口分组汇总。宽容模式:读失败
按空、坏行只计数——只用于人工核对;guard 内部用严格读(见 _replay)。
"""
try:
lines = Path(path).read_text(encoding="utf-8").splitlines()
except OSError:
lines = []
windows, records, parse_errors = _fold_lines(lines, window_id)
return {
"records": records,
"parse_errors": parse_errors,
"windows": {
wid: {
"audio_seconds": round(st.window_audio_seconds, 6),
"confirmed_audio_seconds": round(st.window_confirmed_seconds, 6),
"stopped": st.stopped,
"stop_reason": st.stop_reason,
"calls": {
call: {
"streams": cu.streams,
@@ -200,6 +242,7 @@ def summarize_ledger(path: Path | str, window_id: str | None = None) -> dict:
"connect_failures": cu.connect_failures,
"connect_retries": cu.connect_retries,
"audio_seconds": round(cu.audio_seconds, 6),
"confirmed_audio_seconds": round(cu.confirmed_audio_seconds, 6),
"retry_pending": cu.retry_pending,
}
for call, cu in st.calls.items()
@@ -229,31 +272,70 @@ class AsrUsageGuard:
self._unlabeled = 0
self._pending: dict[str, _AudioBuf] = {}
self.calls: dict[str, CallUsage] = {}
self.window_audio_seconds = 0.0
self.window_audio_seconds = 0.0 # 保守计费(发送前预记,含不确定出站)
self.window_confirmed_seconds = 0.0 # 确证送达(await 成功返回)
self.ledger_errors = 0
self.persistence_stop_reason: str | None = None # 非 None = 本窗出站已停
self._ledger_seen = False # 见过账本存在(读成功或写成功过)
self._last_resync = -1e18
self._replay()
# -- 账本 ---------------------------------------------------------------
def _replay(self) -> None:
"""严格重放:读失败/损坏/账本中途消失 → fail-closed 停机,不套用数据."""
self._last_resync = self._clock()
if self.ledger_path is None or not self.ledger_path.is_file():
if self.ledger_path is None:
return
state = summarize_ledger(self.ledger_path, self.policy.window_id)["windows"].get(
self.policy.window_id
)
try:
exists = self.ledger_path.is_file()
except OSError as exc:
_logger.error("asr usage ledger stat failed: %s: %s", self.ledger_path, exc)
self._mark_persistence_stop("ledger-read-failed",
detail=f"stat: {type(exc).__name__}", fire=False)
return
if not exists:
if self._ledger_seen:
# 读过/写过的账本中途消失 = 已落账事实丢失,不许默认零预算
_logger.error("asr usage ledger vanished mid-window: %s", self.ledger_path)
self._mark_persistence_stop("ledger-missing", fire=False)
return
try:
lines = self.ledger_path.read_text(encoding="utf-8").splitlines()
except OSError as exc:
_logger.error("asr usage ledger read failed: %s: %s", self.ledger_path, exc)
self._mark_persistence_stop("ledger-read-failed",
detail=type(exc).__name__, fire=False)
return
self._ledger_seen = True
windows, _records, parse_errors = _fold_lines(lines, self.policy.window_id)
if parse_errors:
# 损坏账本不可信:不套用任何重放数据,内存已知消耗保全原值
_logger.error("asr usage ledger corrupt (%d bad lines): %s",
parse_errors, self.ledger_path)
self._mark_persistence_stop(
"ledger-corrupt", detail=f"{parse_errors} bad lines", fire=False)
return
state = windows.get(self.policy.window_id)
if state is None:
return
# 只重放计数事实;重放结果不覆盖本进程在途未落账的音频缓冲
self.window_audio_seconds = float(state["audio_seconds"])
for call, cu in state["calls"].items():
# 保全已知消耗:同窗计数只增不减(重放/内存取大,绝不倒退回旧账)
self.window_audio_seconds = max(
self.window_audio_seconds, state.window_audio_seconds)
self.window_confirmed_seconds = max(
self.window_confirmed_seconds, state.window_confirmed_seconds)
for call, cu in state.calls.items():
local = self.calls.setdefault(call, CallUsage())
local.streams = max(local.streams, cu["streams"])
local.connect_attempts = max(local.connect_attempts, cu["connect_attempts"])
local.connect_failures = max(local.connect_failures, cu["connect_failures"])
local.connect_retries = max(local.connect_retries, cu["connect_retries"])
local.audio_seconds = max(local.audio_seconds, cu["audio_seconds"])
local.retry_pending = bool(cu.get("retry_pending", False))
local.streams = max(local.streams, cu.streams)
local.connect_attempts = max(local.connect_attempts, cu.connect_attempts)
local.connect_failures = max(local.connect_failures, cu.connect_failures)
local.connect_retries = max(local.connect_retries, cu.connect_retries)
local.audio_seconds = max(local.audio_seconds, cu.audio_seconds)
local.confirmed_audio_seconds = max(
local.confirmed_audio_seconds, cu.confirmed_audio_seconds)
local.retry_pending = bool(local.retry_pending or cu.retry_pending)
if state.stopped:
# 其他/先前进程记录的持久化停机:同窗重启不恢复出站
self._mark_persistence_stop(state.stop_reason or "ledger-stopped", fire=False)
def _maybe_resync(self) -> None:
if self.policy.resync_seconds < 0:
@@ -274,9 +356,33 @@ class AsrUsageGuard:
self.ledger_path.parent.mkdir(parents=True, exist_ok=True)
with self.ledger_path.open("a", encoding="utf-8") as fh:
fh.write(line + "\n")
except OSError:
# 账本写失败不破坏通话,只降级可核算性;进程内闸仍按内存计数执行
self._ledger_seen = True
except OSError as exc:
# HEL-906 复核:写失败不再只降级可核算性——账本不可信即停本窗出站
self.ledger_errors += 1
_logger.error("asr usage ledger append failed: %s: %s", self.ledger_path, exc)
self._mark_persistence_stop(
"ledger-write-failed", detail=f"{type(exc).__name__}: {exc}", fire=True)
def _mark_persistence_stop(self, reason: str, *, detail: str = "", fire: bool) -> None:
"""持久化故障停机:保全已知消耗、留日志与停机原因、落账 guard_stop."""
if self.persistence_stop_reason is not None:
return # 保留首个停机原因,不重复触发
self.persistence_stop_reason = reason
_logger.error("asr usage ledger unusable — stopping window outbound: "
"reason=%s detail=%s ledger=%s", reason, detail, self.ledger_path)
self._append({"kind": "guard_stop", "reason": reason, "detail": (detail or "")[:200]})
if fire and not self._exhausted_fired:
self._exhausted_fired = True
if self._on_exhausted is not None:
with contextlib.suppress(Exception):
self._on_exhausted(reason)
def _gate_persistence_stop(self, call: str, what: str) -> None:
if self.persistence_stop_reason is None:
return
self._block(call, what, self.persistence_stop_reason,
detail="usage ledger unusable; window outbound stopped")
def _block(self, call: str, what: str, reason: str, detail: str = "") -> None:
self._append({"kind": "block", "call": call, "what": what, "reason": reason,
@@ -300,6 +406,7 @@ class AsrUsageGuard:
self._maybe_resync()
if not self.policy.armed:
self._block(call, "new_stream", "window-not-armed")
self._gate_persistence_stop(call, "new_stream")
if self.window_audio_seconds + self._pending_total() >= \
self.policy.max_audio_seconds - _EPSILON:
self._block(call, "new_stream", "window-audio-exhausted")
@@ -315,6 +422,7 @@ class AsrUsageGuard:
would_retry = cu.retry_pending
if not self.policy.armed:
self._block(call, "new_connection", "window-not-armed")
self._gate_persistence_stop(call, "new_connection")
if self.window_audio_seconds + self._pending_total() >= \
self.policy.max_audio_seconds - _EPSILON:
self._block(call, "new_connection", "window-audio-exhausted")
@@ -351,6 +459,7 @@ class AsrUsageGuard:
seconds = samples / sample_rate
if not self.policy.armed:
self._block(call, "audio", "window-not-armed")
self._gate_persistence_stop(call, "audio")
pending = self._pending.get(call)
pending_seconds = pending.seconds if pending else 0.0
total = self.window_audio_seconds + pending_seconds + seconds
@@ -361,8 +470,13 @@ class AsrUsageGuard:
f"frame={seconds:.3f}s cap={self.policy.max_audio_seconds:.3f}s"),
)
def note_audio_sent(self, call: str, sample_rate: int, samples: int, nbytes: int = 0) -> None:
"""真实发送成功后计数(重发按重复计);达到 flush 阈值聚合落账."""
def reserve_audio(self, call: str, sample_rate: int, samples: int, nbytes: int = 0) -> None:
"""发送前保守预记整帧(HEL-906 复核口径).
越限帧在此拒绝(不出站);计入预算后发送失败/取消**不退还**——帧
可能已出站。确证送达量另记 note_audio_delivered,二者不得混同。
"""
self.precheck_audio(call, sample_rate, samples)
cu = self.calls.setdefault(call, CallUsage())
seconds = round(samples / sample_rate, 6)
cu.audio_seconds = round(cu.audio_seconds + seconds, 6)
@@ -374,13 +488,25 @@ class AsrUsageGuard:
if buf.seconds >= self.policy.flush_seconds:
self.flush_audio(call)
def note_audio_delivered(self, call: str, sample_rate: int, samples: int,
nbytes: int = 0) -> None:
"""确证送达量单独累计(``await ws.send`` 成功返回后调用;不重复计费)."""
cu = self.calls.setdefault(call, CallUsage())
seconds = round(samples / sample_rate, 6)
cu.confirmed_audio_seconds = round(cu.confirmed_audio_seconds + seconds, 6)
buf = self._pending.setdefault(call, _AudioBuf())
buf.confirmed_seconds = round(buf.confirmed_seconds + seconds, 6)
def flush_audio(self, call: str) -> None:
"""把该通缓冲中的音频计数聚合落账(utterance 边界 / 流结束时调用)."""
buf = self._pending.pop(call, None)
if buf is None or buf.seconds <= 0:
if buf is None or (buf.seconds <= 0 and buf.confirmed_seconds <= 0):
return
self.window_audio_seconds = round(self.window_audio_seconds + buf.seconds, 6)
self.window_confirmed_seconds = round(
self.window_confirmed_seconds + buf.confirmed_seconds, 6)
self._append({"kind": "audio", "call": call, "seconds": buf.seconds,
"confirmed_seconds": buf.confirmed_seconds,
"frames": buf.frames, "bytes": buf.bytes, "sample_rate": buf.sample_rate})
def stream_end(self, call: str, reason: str) -> None:
@@ -392,6 +518,7 @@ class AsrUsageGuard:
self._append({
"kind": "call_end", "call": call, "reason": reason,
"audio_seconds": round(cu.audio_seconds, 6) if cu else 0.0,
"confirmed_audio_seconds": round(cu.confirmed_audio_seconds, 6) if cu else 0.0,
"connect_attempts": cu.connect_attempts if cu else 0,
"connect_failures": cu.connect_failures if cu else 0,
"connect_retries": cu.connect_retries if cu else 0,
@@ -404,15 +531,18 @@ class AsrUsageGuard:
def summary(self) -> dict:
pending = round(sum(b.seconds for b in self._pending.values()), 6)
pending_confirmed = round(sum(b.confirmed_seconds for b in self._pending.values()), 6)
return {
"window": self.policy.window_id,
"armed": self.policy.armed,
"max_audio_seconds": self.policy.max_audio_seconds,
"max_connect_retries_per_call": self.policy.max_connect_retries_per_call,
"audio_seconds_flushed": round(self.window_audio_seconds, 6),
"audio_seconds_confirmed": round(self.window_confirmed_seconds + pending_confirmed, 6),
"audio_seconds_with_pending": round(self.window_audio_seconds + pending, 6),
"ledger": str(self.ledger_path) if self.ledger_path else "",
"ledger_errors": self.ledger_errors,
"persistence_stop_reason": self.persistence_stop_reason,
"calls": {
call: {
"streams": cu.streams,
@@ -420,6 +550,7 @@ class AsrUsageGuard:
"connect_failures": cu.connect_failures,
"connect_retries": cu.connect_retries,
"audio_seconds": round(cu.audio_seconds, 6),
"audio_seconds_confirmed": round(cu.confirmed_audio_seconds, 6),
}
for call, cu in self.calls.items()
},
@@ -427,12 +558,16 @@ class AsrUsageGuard:
def summary_line(self) -> str:
s = self.summary()
return (
line = (
f"window={s['window'] or 'not-armed'} "
f"audio={s['audio_seconds_with_pending']:.3f}/{s['max_audio_seconds']:.3f}s "
f"confirmed={s['audio_seconds_confirmed']:.3f}s "
f"retries<= {s['max_connect_retries_per_call']}/call "
f"ledger={s['ledger']}"
)
if s["persistence_stop_reason"]:
line += f" STOPPED[{s['persistence_stop_reason']}]"
return line
def build_usage_from_env(
+12 -8
View File
@@ -14,10 +14,11 @@
HEL-906 用量接缝: FunASRSTT 可注入 AsrUsageGuard(voice/asr_usage.py, worker
按 job 构造). 每次建连前 check_connect、建连结果 note_connect_result、
每个真实发送帧发送前 precheck_audio(越限在发送前拒绝)、发送成功后
note_audio_sent; 流结束/utterance 结束 flush_audio, 流异常时 retry 标记
由 guard 落账. usage=None 时行为与本卡之前完全一致(计数只在布防窗口生效,
未布防 guard 本身即 fail-closed).
每个真实发送帧发送前 reserve_audio(保守预记+越限帧发送前拒绝)、
``await ws.send`` 成功返回后 note_audio_delivered(确证送达量);发送失败/
取消不退还预记——帧可能已出站(HEL-906 复核口径). 流结束/utterance 结束
flush_audio, 流异常时 retry 标记由 guard 落账. usage=None 时行为与本卡
之前完全一致(计数只在布防窗口生效, 未布防 guard 本身即 fail-closed).
"""
from __future__ import annotations
@@ -175,14 +176,17 @@ class FunASRRecognizeStream(RecognizeStream):
return u
async def send_frame(f) -> None:
# HEL-906: 发送前检查(越限帧一个字节都不出站), 发送成功才计数
# HEL-906 复核: 发送前保守预记(越限帧一个字节不出站、账本故障/
# 触线在此拒绝);await 成功返回才记确证送达。失败/取消不退还预记
# ——帧可能已出站,保守计费不退。
payload = bytes(f.data)
if usage is not None:
usage.precheck_audio(call_id, f.sample_rate, f.samples_per_channel)
usage.reserve_audio(call_id, f.sample_rate, f.samples_per_channel,
len(payload))
await utt.ws.send(payload) # type: ignore[union-attr]
if usage is not None:
usage.note_audio_sent(call_id, f.sample_rate,
f.samples_per_channel, len(payload))
usage.note_audio_delivered(call_id, f.sample_rate,
f.samples_per_channel, len(payload))
async def recv_loop(u: _Utterance) -> None:
assert u.ws is not None
+279 -33
View File
@@ -6,10 +6,14 @@
1. 纯计数/闸/账本恢复(guard 单元):fail-closed 默认、按实际帧样本计时、
触线前拒绝、重试口径、落盘恢复、跨窗口隔离、崩溃少计有界。
2. 真实接缝:用假 livekit/websockets 模块(隔离假 provider、假帧)导入
真实的 voice/stt_funasr.py 与 voice/worker.py,驱动真实 ``_run`` 送数
路径——建连检查、发送前拒绝、发送后计数、接收侧死亡、流异常重试、
utterance 聚合落账全部走产品代码;worker entrypoint 布线(guard 注入、
房间名标识、触线 ctx.shutdown、未布防拒绝)逐项断言。
真实的 voice/stt_funasr.py 与 voice/worker.py,驱动真实 ``_run`` 送数
路径——建连检查、发送前拒绝、保守预记/确证送达、接收侧死亡、流异常
重试、utterance 聚合落账全部走产品代码;worker entrypoint 布线(guard
注入、房间名标识、触线 ctx.shutdown、未布防拒绝)逐项断言。
3. 2026-10-28 总管定点复核返工:账本写失败→重放不得回退预算(复现脚本的
回归路径)、读失败/损坏/中途丢失 fail-closed 停机并保全已知消耗、
guard_stop 跨重启仍停、假 ws 先记 payload 再 raise/取消后预算仍包含
该帧(保守计费不退还、确证量单列)。
零真实 ASR/TTS/LiveKit 调用(fake 连接全部在进程内),零安装。
"""
@@ -82,19 +86,31 @@ class FakeResampler:
class FakeWS:
"""进程内 FunASR websocket 假 provider:记录每个出站字节."""
"""进程内 FunASR websocket 假 provider:记录每个出站字节.
fail_send 模拟「帧可能已出站」的不确定失败:先把 payload 记入 sent
再 raise/cancel(控制消息不受 fail_bytes_after 影响)。
"""
def __init__(self, server_msgs: list[str] | None = None, recv_fail: bool = False) -> None:
self.sent: list[bytes | str] = []
self.server_msgs = list(server_msgs or [])
self.recv_fail = recv_fail
self.closed = False
self.fail_send = False
self.fail_send = "" # "" | "raise" | "cancel"
self.fail_bytes_after: int | None = None # 第 N 个音频帧之后开始失败
async def send(self, payload: bytes | str) -> None:
if self.fail_send:
raise OSError("send refused")
self.sent.append(payload)
self.sent.append(payload) # 先记:即使随后失败,帧也可能已经出站
if isinstance(payload, (bytes, bytearray)) and self.fail_send:
if self.fail_bytes_after is not None and self.fail_bytes_after > 0:
self.fail_bytes_after -= 1
return # 前 N 个音频帧正常送达
self.fail_bytes_after = None # 计数耗尽:之后的音频帧都按失败语义
if self.fail_send == "raise":
raise OSError("send failed after outbound")
if self.fail_send == "cancel":
raise asyncio.CancelledError()
async def close(self) -> None:
self.closed = True
@@ -447,15 +463,32 @@ def test_guard_audio_counted_by_actual_frame_samples(tmp_path):
guard.note_connect_result(call, ok=True)
# 16k 直接路径:160 样本 = 0.01s;重采样路径按输出帧 16k 计
for _ in range(10):
guard.precheck_audio(call, 16000, 160)
guard.note_audio_sent(call, 16000, 160, 320)
guard.reserve_audio(call, 16000, 160, 320)
guard.note_audio_delivered(call, 16000, 160, 320)
for _ in range(5):
guard.precheck_audio(call, 16000, 160)
guard.note_audio_sent(call, 16000, 160, 320) # 重发按重复计
guard.reserve_audio(call, 16000, 160, 320)
guard.note_audio_delivered(call, 16000, 160, 320) # 重发按重复计
guard.stream_end(call, "closed")
assert guard.calls[call].audio_seconds == pytest.approx(0.15)
assert guard.calls[call].confirmed_audio_seconds == pytest.approx(0.15)
audio = [r for r in read_records(tmp_path) if r["kind"] == "audio"]
assert sum(r["seconds"] for r in audio) == pytest.approx(0.15)
assert sum(r["confirmed_seconds"] for r in audio) == pytest.approx(0.15)
def test_guard_uncertain_outbound_not_refunded(tmp_path):
"""发送前预记、失败/取消不退还(保守计费);确证量只记成功返回的帧."""
guard = make_guard(tmp_path)
call = guard.begin_stream("room-a")
guard.reserve_audio(call, 16000, 160) # 发送失败的帧:保守预记保留
guard.reserve_audio(call, 16000, 160)
guard.note_audio_delivered(call, 16000, 160) # 只有这帧确证送达
guard.flush_audio(call)
assert guard.calls[call].audio_seconds == pytest.approx(0.02)
assert guard.calls[call].confirmed_audio_seconds == pytest.approx(0.01)
audio = [r for r in read_records(tmp_path) if r["kind"] == "audio"][-1]
assert audio["seconds"] == pytest.approx(0.02)
assert audio["confirmed_seconds"] == pytest.approx(0.01)
def test_guard_blocks_crossing_frame_before_send(tmp_path):
@@ -464,14 +497,14 @@ def test_guard_blocks_crossing_frame_before_send(tmp_path):
call = guard.begin_stream("room-a")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
guard.precheck_audio(call, 16000, 16000) # 1.0s
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.reserve_audio(call, 16000, 16000, 32000) # 1.0s
guard.note_audio_delivered(call, 16000, 16000, 32000)
guard.flush_audio(call)
guard.precheck_audio(call, 16000, 16000) # 恰好等于上限:允许(≤上限)
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.reserve_audio(call, 16000, 16000) # 恰好等于上限:允许(≤上限)
guard.note_audio_delivered(call, 16000, 16000, 32000)
guard.flush_audio(call)
with pytest.raises(AsrWindowClosed): # 第 3 秒会在发送前拒绝
guard.precheck_audio(call, 16000, 16000)
guard.reserve_audio(call, 16000, 16000)
assert stops == ["window-audio-exhausted"] # 回调恰好一次
# 同窗新通也立即拒绝(新连接/新音频都不放行)
with pytest.raises(AsrWindowClosed):
@@ -526,16 +559,18 @@ def test_guard_ledger_recovery_same_window_no_reset(tmp_path):
guard1.check_connect(call)
guard1.note_connect_result(call, ok=False, error="x")
for _ in range(150):
guard1.precheck_audio(call, 16000, 160)
guard1.note_audio_sent(call, 16000, 160, 320)
guard1.reserve_audio(call, 16000, 160, 320)
guard1.note_audio_delivered(call, 16000, 160, 320)
guard1.flush_audio(call)
guard1.stream_end(call, "closed")
# 模拟进程重启:同窗口同账本重建 guard,预算与逐通历史都在
guard2 = make_guard(tmp_path, max_audio_seconds=3.0)
assert guard2.window_audio_seconds == pytest.approx(1.5)
assert guard2.window_confirmed_seconds == pytest.approx(1.5)
cu = guard2.calls["room-a"]
assert (cu.connect_attempts, cu.connect_failures, cu.connect_retries) == (2, 1, 0)
assert cu.audio_seconds == pytest.approx(1.5)
assert cu.confirmed_audio_seconds == pytest.approx(1.5)
assert cu.retry_pending is True # 最后一次连接事实是失败
# 耗尽同窗口剩余预算后拒绝;重开同窗口 id 不能重置
call = guard2.begin_stream("room-a")
@@ -543,8 +578,8 @@ def test_guard_ledger_recovery_same_window_no_reset(tmp_path):
guard2.note_connect_result(call, ok=True) # retry_pending=True → 这是 retry
with pytest.raises(AsrWindowClosed):
for _ in range(200):
guard2.precheck_audio(call, 16000, 160)
guard2.note_audio_sent(call, 16000, 160, 320)
guard2.reserve_audio(call, 16000, 160, 320)
guard2.note_audio_delivered(call, 16000, 160, 320)
guard2.flush_audio(call)
@@ -556,15 +591,15 @@ def test_guard_window_isolation_does_not_stop_others(tmp_path):
g1.check_connect(call)
g1.note_connect_result(call, ok=True)
for _ in range(10):
g1.precheck_audio(call, 16000, 160)
g1.note_audio_sent(call, 16000, 160, 320)
g1.reserve_audio(call, 16000, 160, 320)
g1.note_audio_delivered(call, 16000, 160, 320)
g2 = make_guard(tmp_path, window_id="w2", max_audio_seconds=100.0)
call = g2.begin_stream("room-a") # 同账本、不同窗口:完全不受 W1 影响
g2.check_connect(call)
g2.note_connect_result(call, ok=True)
for _ in range(10):
g2.precheck_audio(call, 16000, 160)
g2.note_audio_sent(call, 16000, 160, 320)
g2.reserve_audio(call, 16000, 160, 320)
g2.note_audio_delivered(call, 16000, 160, 320)
g2.stream_end(call, "closed")
assert g2.calls["room-a"].audio_seconds == pytest.approx(0.1)
@@ -576,15 +611,16 @@ def test_guard_crash_undercount_bounded_by_flush(tmp_path):
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
for _ in range(30):
guard.precheck_audio(call, 16000, 160)
guard.note_audio_sent(call, 16000, 160, 320) # 0.3s < flush 阈值
guard.reserve_audio(call, 16000, 160, 320)
guard.note_audio_delivered(call, 16000, 160, 320) # 0.3s < flush 阈值
# 模拟进程直接消失(没有 stream_end/flush)
replayed = summarize_ledger(tmp_path / "asr-usage.jsonl", "w-test")
assert replayed["windows"]["w-test"]["audio_seconds"] == 0.0
# 正常路径(flush/stream_end)计数完整落账
guard2 = make_guard(tmp_path, flush_seconds=5.0)
call = guard2.begin_stream("room-b")
guard2.note_audio_sent(call, 16000, 16000, 32000)
guard2.reserve_audio(call, 16000, 16000, 32000)
guard2.note_audio_delivered(call, 16000, 16000, 32000)
guard2.stream_end(call, "closed")
assert summarize_ledger(tmp_path / "asr-usage.jsonl", "w-test")["windows"][
"w-test"]["calls"]["room-b"]["audio_seconds"] == pytest.approx(1.0)
@@ -595,7 +631,8 @@ def test_summarize_cli_reproducible(tmp_path, monkeypatch):
call = guard.begin_stream("room-a")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.reserve_audio(call, 16000, 16000, 32000)
guard.note_audio_delivered(call, 16000, 16000, 32000)
guard.stream_end(call, "closed")
repo_root = Path(__file__).resolve().parents[1]
proc = subprocess.run(
@@ -631,11 +668,13 @@ async def test_seam_counts_fed_audio_both_sample_rates(fake_sdk, tmp_path):
cu = guard.calls["cg-voice-abc"]
# 16k 直接路径 0.3s + 48k→16k 重采样路径 0.3s(按实际输出帧计时)
assert cu.audio_seconds == pytest.approx(0.6)
assert cu.confirmed_audio_seconds == pytest.approx(0.6) # 全部送达成功
assert cu.connect_attempts == 2 # 两轮 utterance 两次建连
assert (cu.connect_failures, cu.connect_retries) == (0, 0)
records = read_records(tmp_path)
audio_records = [r for r in records if r["kind"] == "audio"]
assert [round(r["seconds"], 6) for r in audio_records] == [0.3, 0.3] # utterance 边界聚合
assert [round(r["confirmed_seconds"], 6) for r in audio_records] == [0.3, 0.3]
assert all(r["call"] == "cg-voice-abc" for r in records)
ends = [r for r in records if r["kind"] == "call_end"]
assert [r["reason"] for r in ends] == ["closed"]
@@ -719,9 +758,10 @@ async def test_seam_window_exhaustion_stops_before_crossing(fake_sdk, tmp_path):
stt = funasr.FunASRSTT(usage=guard, call_label="room-cap")
with pytest.raises(AsrWindowClosed):
await drive(funasr, stt, [frame_16k() for _ in range(6)] + [None])
# 0.05s = 5 帧放行,第 6 帧发送前拒绝:出站字节恰好 5 帧
# 0.05s = 5 帧放行,第 6 帧发送前拒绝:出站字节恰好 5 帧(provider 未被触线帧触碰)
assert len(fake_sdk.websockets.bytes_sent()) == 5
assert guard.calls["room-cap"].audio_seconds == pytest.approx(0.05)
assert guard.calls["room-cap"].confirmed_audio_seconds == pytest.approx(0.05)
assert stops == ["window-audio-exhausted"]
records = read_records(tmp_path)
assert records[-1]["kind"] == "call_end"
@@ -808,10 +848,10 @@ async def test_worker_entrypoint_wires_guard(fake_sdk, tmp_path, monkeypatch, ca
assert "asr_window=" in out and "hel902-w1" in out
# 触线回调真的接到 ctx.shutdown:预检越限帧 → 结束本通 job
call = guard.begin_stream("cg-voice-room1")
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.reserve_audio(call, 16000, 16000, 32000)
guard.flush_audio(call)
with pytest.raises(AsrWindowClosed):
guard.precheck_audio(call, 16000, 16000 * 2700) # 会越 2700s 上限的帧
guard.reserve_audio(call, 16000, 16000 * 2700) # 会越 2700s 上限的帧
assert ctx.shutdown_calls, "on_exhausted 必须真的调用 ctx.shutdown 结束本通"
@@ -829,3 +869,209 @@ async def test_worker_entrypoint_default_is_fail_closed(fake_sdk, monkeypatch, c
with pytest.raises(AsrWindowClosed):
guard.begin_stream("cg-voice-room2")
assert "asr_window=" in capsys.readouterr().out
# ---------------------------------------------------------------------------
# 4) 2026-10-28 总管定点复核返工:持久化 fail-closed 与不确定出站口径
# ---------------------------------------------------------------------------
class FakeClock:
def __init__(self, t: float = 0.0) -> None:
self.t = t
def __call__(self) -> float:
return self.t
def advance(self, dt: float) -> None:
self.t += dt
def _rework_guard(tmp_path: Path, clock: FakeClock, *, on_exhausted=None,
**overrides) -> AsrUsageGuard:
values = dict(window_id="w-rework", max_audio_seconds=10.0,
flush_seconds=1e9, resync_seconds=5.0)
values.update(overrides)
return AsrUsageGuard(AsrWindowPolicy(**values), tmp_path / "asr-usage.jsonl",
clock=clock, on_exhausted=on_exhausted)
def _feed(guard: AsrUsageGuard, call: str, seconds: float, *, deliver: bool = True) -> None:
frames = round(seconds * 100)
for _ in range(frames):
guard.reserve_audio(call, 16000, 160, 320)
if deliver:
guard.note_audio_delivered(call, 16000, 160, 320)
def test_ledger_write_failure_never_regresses_budget(tmp_path, monkeypatch):
"""总管复现的回归路径:写失败→flush 丢缓冲→重放按旧账回退,修复后不再发生."""
import pathlib
stops: list[str] = []
clock = FakeClock(0.0)
guard = _rework_guard(tmp_path, clock, max_audio_seconds=1.5,
on_exhausted=stops.append)
call = guard.begin_stream("room")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
_feed(guard, call, 0.8)
guard.flush_audio(call) # 0.8s 落账成功
assert guard.window_audio_seconds == pytest.approx(0.8)
real_open = pathlib.Path.open
def fail_append(self, mode="r", *args, **kwargs):
if "a" in (mode or ""):
raise OSError("append refused (injected)")
return real_open(self, mode, *args, **kwargs)
monkeypatch.setattr(pathlib.Path, "open", fail_append)
_feed(guard, call, 0.5)
guard.flush_audio(call) # 0.5s 落账失败:内存保全、立即停机、不许回退
assert guard.window_audio_seconds == pytest.approx(1.3)
assert guard.persistence_stop_reason == "ledger-write-failed"
assert stops == ["ledger-write-failed"] # 停机回调恰好一次并带原因
clock.advance(6.0) # 越过 resync:重放不得把内存拉回旧账本值 0.8
with pytest.raises(AsrWindowClosed) as ei:
_feed(guard, call, 0.6) # 复现脚本里的「再送 0.6s」:现在被拒
assert "ledger-write-failed" in str(ei.value)
assert guard.window_audio_seconds == pytest.approx(1.3) # 无预算倒退
# 写失败注入仍生效期间 block 记录同样写不落账——停机事实以日志与内存为准
assert not [r for r in read_records(tmp_path) if r["kind"] == "block"]
# 停机状态在 summary/summary_line 可见(操作员可核对)
assert guard.summary()["persistence_stop_reason"] == "ledger-write-failed"
assert "ledger-write-failed" in guard.summary_line()
def test_ledger_read_failure_stops_outbound_preserving_memory(tmp_path, monkeypatch):
import pathlib
clock = FakeClock(0.0)
guard = _rework_guard(tmp_path, clock)
call = guard.begin_stream("room")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
_feed(guard, call, 1.0)
guard.flush_audio(call)
def boom(self, *args, **kwargs):
raise OSError("read refused (injected)")
monkeypatch.setattr(pathlib.Path, "read_text", boom)
clock.advance(6.0) # resync 触发严格读:读失败 → fail-closed
with pytest.raises(AsrWindowClosed):
guard.reserve_audio(call, 16000, 160, 320)
assert guard.persistence_stop_reason == "ledger-read-failed"
assert guard.window_audio_seconds == pytest.approx(1.0) # 不清零、不倒退
with pytest.raises(AsrWindowClosed): # 新通也不放行
guard.begin_stream("room-2")
def test_ledger_corruption_stops_outbound(tmp_path):
clock = FakeClock(0.0)
guard = _rework_guard(tmp_path, clock)
call = guard.begin_stream("room")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
_feed(guard, call, 1.0)
guard.flush_audio(call)
with (tmp_path / "asr-usage.jsonl").open("a", encoding="utf-8") as fh:
fh.write("{corrupt-not-json\n") # 中途损坏一行
clock.advance(6.0)
with pytest.raises(AsrWindowClosed):
guard.reserve_audio(call, 16000, 160, 320)
assert guard.persistence_stop_reason == "ledger-corrupt"
assert guard.window_audio_seconds == pytest.approx(1.0) # 损坏不套用、内存保全
with pytest.raises(AsrWindowClosed):
guard.check_connect("room")
def test_ledger_missing_after_seen_stops_outbound(tmp_path):
clock = FakeClock(0.0)
guard = _rework_guard(tmp_path, clock)
call = guard.begin_stream("room")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
_feed(guard, call, 1.0)
guard.flush_audio(call) # 账本已存在(读过/写过)
(tmp_path / "asr-usage.jsonl").unlink() # 中途消失 = 已落账事实丢失
clock.advance(6.0)
with pytest.raises(AsrWindowClosed):
guard.reserve_audio(call, 16000, 160, 320)
assert guard.persistence_stop_reason == "ledger-missing"
assert guard.window_audio_seconds == pytest.approx(1.0)
# 停机事实已落回重建的账本(guard_stop):汇总 CLI 可见
assert summarize_ledger(tmp_path / "asr-usage.jsonl", "w-rework")[
"windows"]["w-rework"]["stopped"] is True
def test_guard_stop_record_keeps_window_stopped_across_restart(tmp_path):
ledger = tmp_path / "asr-usage.jsonl"
lines = [
{"kind": "audio", "window": "w", "call": "room", "seconds": 2.0,
"confirmed_seconds": 2.0, "t": 1},
{"kind": "guard_stop", "window": "w", "reason": "ledger-corrupt", "t": 2},
]
ledger.write_text("".join(json.dumps(x) + "\n" for x in lines), encoding="utf-8")
guard = AsrUsageGuard(
AsrWindowPolicy(window_id="w", max_audio_seconds=10.0), ledger)
# 同窗重启不恢复出站;已知消耗(2.0s)仍被重放保全
assert guard.persistence_stop_reason == "ledger-corrupt"
assert guard.window_audio_seconds == pytest.approx(2.0)
with pytest.raises(AsrWindowClosed):
guard.begin_stream("room-new")
async def test_seam_uncertain_outbound_counted_conservatively(fake_sdk, tmp_path):
"""假 ws 先记下 payload 再 raise/取消:预算仍包含该帧,确证量不包含."""
import cybergril.voice.stt_funasr as funasr
guard = make_guard(tmp_path)
def flaky(mode: str, ok_frames: int) -> FakeWS:
ws = FakeWS(server_msgs=[])
ws.fail_send = mode
ws.fail_bytes_after = ok_frames
return ws
# 场景 1:第 3 帧送出后 raise(帧可能已出站)
fake_sdk.websockets.ws_factory = lambda: flaky("raise", 2)
stt = funasr.FunASRSTT(usage=guard, call_label="room-raise")
with pytest.raises(OSError):
await drive(funasr, stt, [frame_16k() for _ in range(3)] + [None])
cu = guard.calls["room-raise"]
assert cu.audio_seconds == pytest.approx(0.03) # 保守计费含失败帧
assert cu.confirmed_audio_seconds == pytest.approx(0.02) # 确证只有前两帧
ws_raise = fake_sdk.websockets.sockets[-1]
assert len([p for p in ws_raise.sent if isinstance(p, bytes)]) == 3 # 帧确实已出站
rec = [r for r in read_records(tmp_path)
if r["kind"] == "audio" and r["call"] == "room-raise"][-1]
assert rec["seconds"] == pytest.approx(0.03)
assert rec["confirmed_seconds"] == pytest.approx(0.02)
assert [r for r in read_records(tmp_path)
if r["kind"] == "call_end" and r["call"] == "room-raise"][-1]["reason"] == "error"
# 场景 2:第 2 帧送出后被取消:同样不退还预记
fake_sdk.websockets.ws_factory = lambda: flaky("cancel", 1)
stt2 = funasr.FunASRSTT(usage=guard, call_label="room-cancel")
with pytest.raises(asyncio.CancelledError):
await drive(funasr, stt2, [frame_16k() for _ in range(2)] + [None])
cu2 = guard.calls["room-cancel"]
assert cu2.audio_seconds == pytest.approx(0.02)
assert cu2.confirmed_audio_seconds == pytest.approx(0.01)
ws_cancel = fake_sdk.websockets.sockets[-1]
assert len([p for p in ws_cancel.sent if isinstance(p, bytes)]) == 2
# 触线拒绝仍然不碰 provider:越限帧在 reserve 阶段就拒绝,ws.send 不执行
guard_cap = AsrUsageGuard(
make_policy(max_audio_seconds=0.01), tmp_path / "cap.jsonl")
fake_sdk.websockets.ws_factory = None
stt3 = funasr.FunASRSTT(usage=guard_cap, call_label="room-cap2")
with pytest.raises(AsrWindowClosed):
await drive(funasr, stt3, [frame_16k() for _ in range(3)] + [None])
ws_cap = fake_sdk.websockets.sockets[-1]
assert len([p for p in ws_cap.sent if isinstance(p, bytes)]) == 1 # 只有第 1 帧出站