HEL-906: continuous-call ASR usage metering + one-shot window guard in dedicated worker seam

- asr_usage.py (stdlib-only): append-only JSONL ledger, per-call/window audio
  seconds counted from actually-sent frames (sample-rate aware, resend counted),
  connect attempt/failure/retry semantics (normal per-utterance reconnects are
  not retries), fail-closed default (real cap=0 unless a window is armed),
  pre-send blocking so no byte crosses the cap, ledger replay for restart
  recovery and cross-window isolation, resumable summarize CLI
- stt_funasr.py: guard hooks on the real send path (check_connect before
  open_utt, precheck before send, count after send, recv-side death recorded,
  per-utterance flush aggregation); usage=None keeps prior behavior
- worker.py: entrypoint builds guard from env per job, room name as call id,
  on_exhausted -> ctx.shutdown ends the call, window summary in start log
- tests: 17 deterministic offline tests incl. fake livekit/websockets SDK
  driving the real _run seam, sample-rate variants, silence/cancel/failure/
  reconnect/repeated-end, cap rejection before crossing, ledger persistence
  across restart, room deletion keeps history, other windows unaffected

Co-authored-by: multica-agent <github@multica.ai>
This commit is contained in:
总工
2026-10-28 08:44:20 +08:00
co-authored by multica-agent
parent 0b804a49ef
commit ecec9df8e8
4 changed files with 1392 additions and 10 deletions
+464
View File
@@ -0,0 +1,464 @@
"""连续通话 ASR 实际用量计数与一次性窗口闸(HEL-906).
补齐 HEL-905 复审留下的 §4 待补阻塞:连续通话内 FunASR 的**实际送入音频
量**与**连接尝试/重连/失败**没有本仓库可执行的程序计数。本模块只依赖
标准库(API 进程与离线测试环境都能直接导入),计数全部发生在专属
worker 的真实送数路径上:
- FunASRSTT(voice/stt_funasr.py)在每次 websocket 建连前 check_connect、
建连结果 note_connect_result、每个真实发送帧发送前 precheck_audio、
发送成功后 note_audio_sent;worker(voice/worker.py)按 job 注入 guard
并把 on_exhausted 接到 ctx.shutdown。
- 音频按**实际发送帧**的 sample_rate 与样本数计时(重采样后的输出帧按其
自身采样率计),同一帧重复发送按重复计;墙钟、开房/删房流水不参与
音频计时,事后成功流水也不追溯改数。
- 连接事实逐条落账:每次建连尝试(含失败)、失败后的重试(retry)、
接收侧死亡(connection_died)。FunASR 一句一连的惰性建连里,正常说完
一轮后的下一次建连不是 retry,只有上一次尝试失败/连接死亡/流异常结束
之后的建连才计 retry。
账本(追加写 JSONL,默认 /opt/voice-agent-cybergril/asr-usage.jsonl)独立
于实例侧会话账本 ``<db>-voice-sessions.json``:删房成功会把通话从会话账
本移除,但这里的逐通历史(call_start/audio/connect_attempt/
connection_died/call_end/block)永不改写、不随删房消失;窗口总量由重放
账本恢复,同 window_id 重启进程不能重置预算,旧窗口耗尽状态也被持久化
(不得把旧余额或新开账当作新许可)。
窗口语义沿用 think/budget.py 的 fail-closed 先例:
- 未布防(CYBERGRIL_ASR_WINDOW_ID 未设,产品默认):一切新连接与新音频
在发送前拒绝(真实 cap=0),拒绝本身落账;
- 布防(获批的一次性窗口启动时由窗口负责人显式设置环境变量):触线即
在发送前拒绝(AsrWindowClosed),worker 经 on_exhausted 结束本通 job;
上限只能收紧不能放宽,候选值见 docs/rework/hel902/r5-followup-package.md §4。
程序能保证 / 剩余只能监看(如实边界):
- 保证:布防进程内逐帧发送前硬停(帧不越上限一个字节)、逐通与全窗
计数不因删房/重启丢失、同窗重启不能重置预算。
- 跨进程有界收敛:账本是共同事实,每次建连检查与 resync_seconds 间隔
重放账本;两进程并行通话时对端最近 ≤ max(resync, flush) 间隔的音频
可能未计入本轮判定(有界过冲)。
- 只能监看:进程崩溃前 ≤ flush_seconds 的已发音频可能少计;接收侧死亡
未伴随发送失败时按 connection_died 计连接事实、不改已计音频;真实
FunASR/LiveKit 行为仍以窗口前 .12 实测为准。
"""
from __future__ import annotations
import argparse
import contextlib
import json
import os
import secrets
import time
from dataclasses import dataclass, field
from pathlib import Path
DEFAULT_ASR_USAGE_LEDGER = "/opt/voice-agent-cybergril/asr-usage.jsonl"
DEFAULT_FLUSH_SECONDS = 5.0
DEFAULT_RESYNC_SECONDS = 5.0
# 允许浮点求和后恰好等于上限(“≤上限”而不是“<上限”)
_EPSILON = 1e-9
class AsrWindowClosed(Exception):
"""The armed window (or the fail-closed default) refuses this connection/audio."""
@dataclass(frozen=True)
class AsrWindowPolicy:
window_id: str = "" # 空 = 未布防:真实 cap=0,拒绝一切 ASR 连接/音频
max_audio_seconds: float = 0.0 # 全窗口实际送入 ASR 音频上限
max_connect_retries_per_call: int = 0 # 每通连接重试上限(失败后的再建连)
flush_seconds: float = DEFAULT_FLUSH_SECONDS # 音频计数聚合落账阈值
resync_seconds: float = DEFAULT_RESYNC_SECONDS # 跨进程账本重放间隔
@property
def armed(self) -> bool:
return bool(self.window_id)
def _env_float(env: dict, name: str, default: float) -> float:
raw = (env.get(name) or "").strip()
if not raw:
return default
try:
value = float(raw)
except ValueError:
return 0.0 # 配置损坏按 fail-closed 处理
return value if value > 0 else 0.0
def _env_int(env: dict, name: str, default: int) -> int:
raw = (env.get(name) or "").strip()
if not raw:
return default
try:
value = int(raw)
except ValueError:
return 0
return value if value > 0 else 0
def policy_from_env(environ: dict | None = None) -> AsrWindowPolicy:
env = dict(os.environ) if environ is None else environ
return AsrWindowPolicy(
window_id=(env.get("CYBERGRIL_ASR_WINDOW_ID") or "").strip(),
max_audio_seconds=_env_float(env, "CYBERGRIL_ASR_WINDOW_MAX_AUDIO_SECONDS", 0.0),
max_connect_retries_per_call=_env_int(
env, "CYBERGRIL_ASR_MAX_CONNECT_RETRIES_PER_CALL", 0
),
flush_seconds=_env_float(env, "CYBERGRIL_ASR_USAGE_FLUSH_SECONDS", DEFAULT_FLUSH_SECONDS),
resync_seconds=_env_float(env, "CYBERGRIL_ASR_USAGE_RESYNC_SECONDS", DEFAULT_RESYNC_SECONDS),
)
@dataclass
class _AudioBuf:
seconds: float = 0.0
frames: int = 0
bytes: int = 0
sample_rate: int = 0
@dataclass
class CallUsage:
streams: int = 0
connect_attempts: int = 0
connect_failures: int = 0
connect_retries: int = 0
audio_seconds: float = 0.0
retry_pending: bool = False # 上一次连接事实是失败 → 下次建连是 retry
_would_retry: bool = False # check_connect 为本次在途尝试标记的 retry 身份
@dataclass
class _ReplayState:
window_audio_seconds: float = 0.0
calls: dict[str, CallUsage] = field(default_factory=dict)
def summarize_ledger(path: Path | str, window_id: str | None = None) -> dict:
"""重放账本成汇总(操作员核对口径;不修改账本).
window_id 为 None 时按账本里出现过的窗口分组汇总。
"""
windows: dict[str, _ReplayState] = {}
records = 0
parse_errors = 0
try:
lines = Path(path).read_text(encoding="utf-8").splitlines()
except OSError:
lines = []
for line in lines:
line = line.strip()
if not line:
continue
records += 1
try:
rec = json.loads(line)
except ValueError:
parse_errors += 1
continue
wid = str(rec.get("window", ""))
if window_id is not None and wid != window_id:
continue
state = windows.setdefault(wid, _ReplayState())
call = str(rec.get("call", ""))
cu = state.calls.setdefault(call, CallUsage())
kind = rec.get("kind")
if kind == "call_start":
cu.streams += 1
elif kind == "connect_attempt":
cu.connect_attempts += 1
if rec.get("retry"):
cu.connect_retries += 1
if not bool(rec.get("ok")):
cu.connect_failures += 1
cu.retry_pending = not bool(rec.get("ok"))
elif kind == "connection_died":
cu.connect_failures += 1
cu.retry_pending = True
elif kind == "audio":
seconds = float(rec.get("seconds", 0.0) or 0.0)
cu.audio_seconds = round(cu.audio_seconds + seconds, 6)
state.window_audio_seconds = round(state.window_audio_seconds + seconds, 6)
elif kind == "call_end" and str(rec.get("reason")) == "error":
cu.retry_pending = True
return {
"records": records,
"parse_errors": parse_errors,
"windows": {
wid: {
"audio_seconds": round(st.window_audio_seconds, 6),
"calls": {
call: {
"streams": cu.streams,
"connect_attempts": cu.connect_attempts,
"connect_failures": cu.connect_failures,
"connect_retries": cu.connect_retries,
"audio_seconds": round(cu.audio_seconds, 6),
"retry_pending": cu.retry_pending,
}
for call, cu in st.calls.items()
},
}
for wid, st in windows.items()
},
}
class AsrUsageGuard:
"""worker 侧计数与闸(每个 job 一个实例,注入 FunASRSTT)."""
def __init__(
self,
policy: AsrWindowPolicy,
ledger_path: Path | str | None = None,
*,
clock=time.time,
on_exhausted=None,
) -> None:
self.policy = policy
self.ledger_path = Path(ledger_path) if ledger_path else None
self._clock = clock
self._on_exhausted = on_exhausted
self._exhausted_fired = False
self._unlabeled = 0
self._pending: dict[str, _AudioBuf] = {}
self.calls: dict[str, CallUsage] = {}
self.window_audio_seconds = 0.0
self.ledger_errors = 0
self._last_resync = -1e18
self._replay()
# -- 账本 ---------------------------------------------------------------
def _replay(self) -> None:
self._last_resync = self._clock()
if self.ledger_path is None or not self.ledger_path.is_file():
return
state = summarize_ledger(self.ledger_path, self.policy.window_id)["windows"].get(
self.policy.window_id
)
if state is None:
return
# 只重放计数事实;重放结果不覆盖本进程在途未落账的音频缓冲
self.window_audio_seconds = float(state["audio_seconds"])
for call, cu in state["calls"].items():
local = self.calls.setdefault(call, CallUsage())
local.streams = max(local.streams, cu["streams"])
local.connect_attempts = max(local.connect_attempts, cu["connect_attempts"])
local.connect_failures = max(local.connect_failures, cu["connect_failures"])
local.connect_retries = max(local.connect_retries, cu["connect_retries"])
local.audio_seconds = max(local.audio_seconds, cu["audio_seconds"])
local.retry_pending = bool(cu.get("retry_pending", False))
def _maybe_resync(self) -> None:
if self.policy.resync_seconds < 0:
return
if self._clock() - self._last_resync < self.policy.resync_seconds:
return
self._replay()
def _append(self, record: dict) -> None:
if self.ledger_path is None:
return
line = json.dumps(
{**record, "window": self.policy.window_id, "t": round(self._clock(), 3)},
ensure_ascii=False,
separators=(",", ":"),
)
try:
self.ledger_path.parent.mkdir(parents=True, exist_ok=True)
with self.ledger_path.open("a", encoding="utf-8") as fh:
fh.write(line + "\n")
except OSError:
# 账本写失败不破坏通话,只降级可核算性;进程内闸仍按内存计数执行
self.ledger_errors += 1
def _block(self, call: str, what: str, reason: str, detail: str = "") -> None:
self._append({"kind": "block", "call": call, "what": what, "reason": reason,
"detail": detail})
if not self._exhausted_fired:
self._exhausted_fired = True
if self._on_exhausted is not None:
with contextlib.suppress(Exception):
self._on_exhausted(reason)
raise AsrWindowClosed(f"asr window stop [{reason}] {what} call={call} {detail}".strip())
# -- worker 接缝(stt_funasr / worker 真实调用) -------------------------
def _pending_total(self) -> float:
return round(sum(b.seconds for b in self._pending.values()), 6)
def begin_stream(self, call: str = "") -> str:
"""一个 RecognizeStream 开始(worker 按房间名标识;返回生效 call id)."""
if not call:
self._unlabeled += 1
call = f"call-{self._unlabeled}-{secrets.token_hex(3)}"
self._maybe_resync()
if not self.policy.armed:
self._block(call, "new_stream", "window-not-armed")
if self.window_audio_seconds + self._pending_total() >= \
self.policy.max_audio_seconds - _EPSILON:
self._block(call, "new_stream", "window-audio-exhausted")
cu = self.calls.setdefault(call, CallUsage())
cu.streams += 1
self._append({"kind": "call_start", "call": call, "stream_seq": cu.streams})
return call
def check_connect(self, call: str) -> None:
"""建连前检查(可抛 AsrWindowClosed;之后必须 note_connect_result)."""
self._maybe_resync()
cu = self.calls.setdefault(call, CallUsage())
would_retry = cu.retry_pending
if not self.policy.armed:
self._block(call, "new_connection", "window-not-armed")
if self.window_audio_seconds + self._pending_total() >= \
self.policy.max_audio_seconds - _EPSILON:
self._block(call, "new_connection", "window-audio-exhausted")
if would_retry and cu.connect_retries >= self.policy.max_connect_retries_per_call:
self._block(
call, "new_connection", "connect-retries-exhausted",
detail=f"retries={cu.connect_retries}",
)
cu._would_retry = would_retry
def note_connect_result(self, call: str, *, ok: bool, error: str = "") -> None:
cu = self.calls.setdefault(call, CallUsage())
cu.connect_attempts += 1
retry = cu._would_retry
if retry:
cu.connect_retries += 1
if not ok:
cu.connect_failures += 1
cu.retry_pending = not ok
cu._would_retry = False
self._append({"kind": "connect_attempt", "call": call, "seq": cu.connect_attempts,
"ok": ok, "retry": retry, "error": (error or "")[:200]})
def note_connection_died(self, call: str, error: str = "") -> None:
"""连接接收侧死亡(未伴随建连失败):计连接失败并置 retry 标记."""
cu = self.calls.setdefault(call, CallUsage())
cu.connect_failures += 1
cu.retry_pending = True
self._append({"kind": "connection_died", "call": call, "error": (error or "")[:200]})
def precheck_audio(self, call: str, sample_rate: int, samples: int) -> None:
"""发送前检查:这一帧会让窗口越限就在发送前拒绝(帧不越限一个字节)."""
self._maybe_resync()
seconds = samples / sample_rate
if not self.policy.armed:
self._block(call, "audio", "window-not-armed")
pending = self._pending.get(call)
pending_seconds = pending.seconds if pending else 0.0
total = self.window_audio_seconds + pending_seconds + seconds
if total > self.policy.max_audio_seconds + _EPSILON:
self._block(
call, "audio", "window-audio-exhausted",
detail=(f"window={self.window_audio_seconds:.3f}s pending={pending_seconds:.3f}s "
f"frame={seconds:.3f}s cap={self.policy.max_audio_seconds:.3f}s"),
)
def note_audio_sent(self, call: str, sample_rate: int, samples: int, nbytes: int = 0) -> None:
"""真实发送成功后计数(重发按重复计);达到 flush 阈值聚合落账."""
cu = self.calls.setdefault(call, CallUsage())
seconds = round(samples / sample_rate, 6)
cu.audio_seconds = round(cu.audio_seconds + seconds, 6)
buf = self._pending.setdefault(call, _AudioBuf())
buf.seconds = round(buf.seconds + seconds, 6)
buf.frames += 1
buf.bytes += nbytes
buf.sample_rate = sample_rate
if buf.seconds >= self.policy.flush_seconds:
self.flush_audio(call)
def flush_audio(self, call: str) -> None:
"""把该通缓冲中的音频计数聚合落账(utterance 边界 / 流结束时调用)."""
buf = self._pending.pop(call, None)
if buf is None or buf.seconds <= 0:
return
self.window_audio_seconds = round(self.window_audio_seconds + buf.seconds, 6)
self._append({"kind": "audio", "call": call, "seconds": buf.seconds,
"frames": buf.frames, "bytes": buf.bytes, "sample_rate": buf.sample_rate})
def stream_end(self, call: str, reason: str) -> None:
"""一个 RecognizeStream 结束(closed/error/window_exhausted)."""
self.flush_audio(call)
cu = self.calls.get(call)
if cu is not None and reason == "error":
cu.retry_pending = True
self._append({
"kind": "call_end", "call": call, "reason": reason,
"audio_seconds": round(cu.audio_seconds, 6) if cu else 0.0,
"connect_attempts": cu.connect_attempts if cu else 0,
"connect_failures": cu.connect_failures if cu else 0,
"connect_retries": cu.connect_retries if cu else 0,
})
# -- 观测 ---------------------------------------------------------------
def pending_audio_seconds(self, call: str) -> float:
buf = self._pending.get(call)
return buf.seconds if buf else 0.0
def summary(self) -> dict:
pending = round(sum(b.seconds for b in self._pending.values()), 6)
return {
"window": self.policy.window_id,
"armed": self.policy.armed,
"max_audio_seconds": self.policy.max_audio_seconds,
"max_connect_retries_per_call": self.policy.max_connect_retries_per_call,
"audio_seconds_flushed": round(self.window_audio_seconds, 6),
"audio_seconds_with_pending": round(self.window_audio_seconds + pending, 6),
"ledger": str(self.ledger_path) if self.ledger_path else "",
"ledger_errors": self.ledger_errors,
"calls": {
call: {
"streams": cu.streams,
"connect_attempts": cu.connect_attempts,
"connect_failures": cu.connect_failures,
"connect_retries": cu.connect_retries,
"audio_seconds": round(cu.audio_seconds, 6),
}
for call, cu in self.calls.items()
},
}
def summary_line(self) -> str:
s = self.summary()
return (
f"window={s['window'] or 'not-armed'} "
f"audio={s['audio_seconds_with_pending']:.3f}/{s['max_audio_seconds']:.3f}s "
f"retries<= {s['max_connect_retries_per_call']}/call "
f"ledger={s['ledger']}"
)
def build_usage_from_env(
environ: dict | None = None,
*,
clock=time.time,
on_exhausted=None,
ledger_path: Path | str | None = None,
) -> AsrUsageGuard:
"""worker entrypoint 的唯一构造入口(env 只在窗口启动时由负责人显式设置)."""
env = os.environ if environ is None else environ
path = ledger_path or env.get("CYBERGRIL_ASR_USAGE_LEDGER") or DEFAULT_ASR_USAGE_LEDGER
return AsrUsageGuard(policy_from_env(env), path, clock=clock, on_exhausted=on_exhausted)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m cybergril.voice.asr_usage",
description="重放 ASR 用量账本为汇总 JSON(只读,不修改账本)",
)
parser.add_argument("ledger", help="asr-usage.jsonl 路径")
parser.add_argument("--window", default=None, help="只汇总指定 window id")
args = parser.parse_args(argv)
print(json.dumps(summarize_ledger(args.ledger, args.window), ensure_ascii=False, indent=1))
return 0
if __name__ == "__main__":
raise SystemExit(main())
+74 -8
View File
@@ -11,6 +11,13 @@
-> FINAL_TRANSCRIPT, 之后为本轮重建 websocket(FunASR 一次会话一句话) -> FINAL_TRANSCRIPT, 之后为本轮重建 websocket(FunASR 一次会话一句话)
一个 RecognizeStream 对应一个房间会话, websocket 按轮次惰性重建. 一个 RecognizeStream 对应一个房间会话, websocket 按轮次惰性重建.
HEL-906 用量接缝: FunASRSTT 可注入 AsrUsageGuard(voice/asr_usage.py, worker
按 job 构造). 每次建连前 check_connect、建连结果 note_connect_result、
每个真实发送帧发送前 precheck_audio(越限在发送前拒绝)、发送成功后
note_audio_sent; 流结束/utterance 结束 flush_audio, 流异常时 retry 标记
由 guard 落账. usage=None 时行为与本卡之前完全一致(计数只在布防窗口生效,
未布防 guard 本身即 fail-closed).
""" """
from __future__ import annotations from __future__ import annotations
@@ -31,15 +38,21 @@ from livekit.agents.stt import (
) )
from livekit.agents.types import DEFAULT_API_CONNECT_OPTIONS, APIConnectOptions from livekit.agents.types import DEFAULT_API_CONNECT_OPTIONS, APIConnectOptions
from .asr_usage import AsrUsageGuard, AsrWindowClosed
FUNASR_WS = "ws://127.0.0.1:10095" FUNASR_WS = "ws://127.0.0.1:10095"
class FunASRSTT(stt.STT): class FunASRSTT(stt.STT):
def __init__(self, endpoint: str = FUNASR_WS) -> None: def __init__(self, endpoint: str = FUNASR_WS, *,
usage: AsrUsageGuard | None = None,
call_label: str = "") -> None:
super().__init__( super().__init__(
capabilities=stt.STTCapabilities(streaming=True, interim_results=True) capabilities=stt.STTCapabilities(streaming=True, interim_results=True)
) )
self._endpoint = endpoint self._endpoint = endpoint
self._usage = usage
self._call_label = call_label
# 回声抑制(worker 运行时注入): agent 说话期间, 短文本或与刚说内容 # 回声抑制(worker 运行时注入): agent 说话期间, 短文本或与刚说内容
# 高度相似的识别结果视为音箱回声, 不作为新轮次提交 # 高度相似的识别结果视为音箱回声, 不作为新轮次提交
self.agent_speaking = False self.agent_speaking = False
@@ -75,7 +88,9 @@ class FunASRSTT(stt.STT):
def stream( def stream(
self, *, language=None, conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS self, *, language=None, conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS
) -> RecognizeStream: ) -> RecognizeStream:
return _LoggedRecognizeStream(stt=self, conn_options=conn_options, endpoint=self._endpoint) return _LoggedRecognizeStream(stt=self, conn_options=conn_options,
endpoint=self._endpoint,
usage=self._usage, call_label=self._call_label)
class _Utterance: class _Utterance:
@@ -86,12 +101,16 @@ class _Utterance:
self.ws: websockets.WebSocketClientProtocol | None = None self.ws: websockets.WebSocketClientProtocol | None = None
self.closed = False self.closed = False
self.last_text = "" self.last_text = ""
self.recv_failed = False # 接收侧死亡(未伴随建连失败), HEL-906 计数用
class FunASRRecognizeStream(RecognizeStream): class FunASRRecognizeStream(RecognizeStream):
def __init__(self, *, stt: FunASRSTT, conn_options: APIConnectOptions, endpoint: str) -> None: def __init__(self, *, stt: FunASRSTT, conn_options: APIConnectOptions, endpoint: str,
usage: AsrUsageGuard | None = None, call_label: str = "") -> None:
super().__init__(stt=stt, conn_options=conn_options) super().__init__(stt=stt, conn_options=conn_options)
self._endpoint = endpoint self._endpoint = endpoint
self._usage = usage
self._call_label = call_label
def _ev(self, ev_type: SpeechEventType, text: str) -> SpeechEvent: def _ev(self, ev_type: SpeechEventType, text: str) -> SpeechEvent:
return SpeechEvent( return SpeechEvent(
@@ -100,6 +119,23 @@ class FunASRRecognizeStream(RecognizeStream):
) )
async def _run(self) -> None: async def _run(self) -> None:
if self._usage is None:
await self._pump(None, "")
return
call_id = self._usage.begin_stream(self._call_label)
end_reason = "closed"
try:
await self._pump(self._usage, call_id)
except AsrWindowClosed:
end_reason = "window_exhausted"
raise
except BaseException:
end_reason = "error"
raise
finally:
self._usage.stream_end(call_id, end_reason)
async def _pump(self, usage: AsrUsageGuard | None, call_id: str) -> None:
utt: _Utterance | None = None utt: _Utterance | None = None
recv_task: asyncio.Task | None = None recv_task: asyncio.Task | None = None
finals: asyncio.Queue[str] = asyncio.Queue() finals: asyncio.Queue[str] = asyncio.Queue()
@@ -107,7 +143,17 @@ class FunASRRecognizeStream(RecognizeStream):
n_frames = 0 n_frames = 0
async def open_utt() -> _Utterance: async def open_utt() -> _Utterance:
ws = await websockets.connect(self._endpoint, max_size=2**24) if usage is not None:
usage.check_connect(call_id)
try:
ws = await websockets.connect(self._endpoint, max_size=2**24)
except Exception as exc:
if usage is not None:
usage.note_connect_result(
call_id, ok=False, error=f"{type(exc).__name__}: {exc}")
raise
if usage is not None:
usage.note_connect_result(call_id, ok=True)
cfg = { cfg = {
"mode": "2pass", "mode": "2pass",
"wav_name": "livekit", "wav_name": "livekit",
@@ -117,11 +163,27 @@ class FunASRRecognizeStream(RecognizeStream):
"audio_fs": 16000, "audio_fs": 16000,
"itn": True, "itn": True,
} }
await ws.send(json.dumps(cfg)) try:
await ws.send(json.dumps(cfg))
except Exception as exc:
if usage is not None:
usage.note_connection_died(
call_id, error=f"{type(exc).__name__}: {exc}")
raise
u = _Utterance(self._endpoint) u = _Utterance(self._endpoint)
u.ws = ws u.ws = ws
return u return u
async def send_frame(f) -> None:
# HEL-906: 发送前检查(越限帧一个字节都不出站), 发送成功才计数
payload = bytes(f.data)
if usage is not None:
usage.precheck_audio(call_id, f.sample_rate, f.samples_per_channel)
await utt.ws.send(payload) # type: ignore[union-attr]
if usage is not None:
usage.note_audio_sent(call_id, f.sample_rate,
f.samples_per_channel, len(payload))
async def recv_loop(u: _Utterance) -> None: async def recv_loop(u: _Utterance) -> None:
assert u.ws is not None assert u.ws is not None
pending: list[str] = [] # 去抖合并中的 final 片段 pending: list[str] = [] # 去抖合并中的 final 片段
@@ -165,7 +227,7 @@ class FunASRRecognizeStream(RecognizeStream):
self._ev(SpeechEventType.INTERIM_TRANSCRIPT, text) self._ev(SpeechEventType.INTERIM_TRANSCRIPT, text)
) )
except Exception: except Exception:
pass u.recv_failed = True # 接收侧死亡: HEL-906 计入连接失败事实
try: try:
async for ev in self._input_ch: async for ev in self._input_ch:
@@ -195,6 +257,10 @@ class FunASRRecognizeStream(RecognizeStream):
await recv_task await recv_task
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
await utt.ws.close() await utt.ws.close()
if usage is not None:
if utt.recv_failed:
usage.note_connection_died(call_id)
usage.flush_audio(call_id) # utterance 边界聚合落账
utt = None utt = None
continue continue
if not isinstance(ev, rtc.AudioFrame): if not isinstance(ev, rtc.AudioFrame):
@@ -210,9 +276,9 @@ class FunASRRecognizeStream(RecognizeStream):
if resampler is None: if resampler is None:
resampler = rtc.AudioResampler(frame.sample_rate, 16000) resampler = rtc.AudioResampler(frame.sample_rate, 16000)
for f in resampler.push(frame): for f in resampler.push(frame):
await utt.ws.send(bytes(f.data)) await send_frame(f)
else: else:
await utt.ws.send(bytes(frame.data)) await send_frame(frame)
finally: finally:
if recv_task is not None: if recv_task is not None:
recv_task.cancel() recv_task.cancel()
+23 -2
View File
@@ -20,6 +20,14 @@
见 docs/voice-subtitle-contract-v2.md)。 见 docs/voice-subtitle-contract-v2.md)。
音色: 独立 voice_state_cybergril.json;D1 未拍板前回落 default,不替用户定。 音色: 独立 voice_state_cybergril.json;D1 未拍板前回落 default,不替用户定。
HEL-906 连续通话 ASR 用量: entrypoint 按 job 构造 AsrUsageGuard 注入
FunASRSTT(房间名作为每通标识)。未布防(无 CYBERGRIL_ASR_WINDOW_ID)
即 fail-closed 拒绝一切 ASR 连接/音频(真实 cap=0);获批的一次性窗口
由窗口负责人启动时显式设置环境变量(见 voice/asr_usage.py 与
docs/rework/hel902/r5-followup-package.md §4)。触线时 guard 在发送前
拒绝并经 on_exhausted 回调 ctx.shutdown 结束本通 job;逐通/全窗计数落
asr-usage.jsonl,不随删房丢失。
运行位置: /opt/voice-agent-cybergril/(独立目录,复用 .12 现有 agent venv 与 运行位置: /opt/voice-agent-cybergril/(独立目录,复用 .12 现有 agent venv 与
LiveKit/FunASR/CosyVoice;专属健康端口 CYBERGRIL_WORKER_HEALTH_PORT—— LiveKit/FunASR/CosyVoice;专属健康端口 CYBERGRIL_WORKER_HEALTH_PORT——
livekit-agents worker 内建 HTTP 健康检查,GET / 返回 OK/503)。 livekit-agents worker 内建 HTTP 健康检查,GET / 返回 OK/503)。
@@ -40,6 +48,7 @@ from pathlib import Path
from livekit.agents import Agent, AgentSession, JobContext, JobProcess, WorkerOptions, cli from livekit.agents import Agent, AgentSession, JobContext, JobProcess, WorkerOptions, cli
from livekit.plugins.silero import VAD from livekit.plugins.silero import VAD
from .asr_usage import build_usage_from_env
from .llm_chat_api import CyberGrillLLM from .llm_chat_api import CyberGrillLLM
from .receipt import ReceiptSender, TurnLedger, should_ack from .receipt import ReceiptSender, TurnLedger, should_ack
from .seam import AssistantInterimPublisher, SubtitleBus, pending_proactive from .seam import AssistantInterimPublisher, SubtitleBus, pending_proactive
@@ -266,8 +275,19 @@ async def _maybe_open_proactive(session: AgentSession, room, bus: SubtitleBus,
return spoken_ids return spoken_ids
def _shutdown_job(ctx, reason: str) -> None:
"""AsrUsageGuard 触线回调: 结束本通 job(异步 shutdown 兼容为任务)."""
print(f"[cybergril-worker] asr window stop: {reason}; ending this call", flush=True)
with contextlib.suppress(Exception):
result = ctx.shutdown()
if inspect.isawaitable(result):
asyncio.get_running_loop().create_task(result)
async def entrypoint(ctx: JobContext) -> None: async def entrypoint(ctx: JobContext) -> None:
stt = FunASRSTT() usage = build_usage_from_env(
on_exhausted=lambda reason: _shutdown_job(ctx, reason))
stt = FunASRSTT(usage=usage, call_label=getattr(ctx.room, "name", ""))
# agent 说话期间短文本/相似文本视为回声丢弃(同通用 worker 的实测做法) # agent 说话期间短文本/相似文本视为回声丢弃(同通用 worker 的实测做法)
stt.agent_speaking = False stt.agent_speaking = False
stt.agent_recent_text = "" stt.agent_recent_text = ""
@@ -305,7 +325,8 @@ async def entrypoint(ctx: JobContext) -> None:
flush=True) flush=True)
print( print(
f"[cybergril-worker] session started (api={API_BASE}, " f"[cybergril-worker] session started (api={API_BASE}, "
f"voice={read_default_voice()}, agent_name={AGENT_NAME})", f"voice={read_default_voice()}, agent_name={AGENT_NAME}, "
f"asr_window=({usage.summary_line()}))",
flush=True, flush=True,
) )
+831
View File
@@ -0,0 +1,831 @@
"""HEL-906 连续通话 ASR 实际用量计数与窗口闸的确定性离线测试.
两层验证(复审要求"接缝必须真的被专属 worker 调用,不能只有独立计数类
测试"):
1. 纯计数/闸/账本恢复(guard 单元):fail-closed 默认、按实际帧样本计时、
触线前拒绝、重试口径、落盘恢复、跨窗口隔离、崩溃少计有界。
2. 真实接缝:用假 livekit/websockets 模块(隔离假 provider、假帧)导入
真实的 voice/stt_funasr.py 与 voice/worker.py,驱动真实 ``_run`` 送数
路径——建连检查、发送前拒绝、发送后计数、接收侧死亡、流异常重试、
utterance 聚合落账全部走产品代码;worker entrypoint 布线(guard 注入、
房间名标识、触线 ctx.shutdown、未布防拒绝)逐项断言。
零真实 ASR/TTS/LiveKit 调用(fake 连接全部在进程内),零安装。
"""
from __future__ import annotations
import asyncio
import enum
import json
import subprocess
import sys
import types
from pathlib import Path
from types import SimpleNamespace
import httpx
import pytest
from cybergril.voice.asr_usage import (
AsrUsageGuard,
AsrWindowClosed,
AsrWindowPolicy,
build_usage_from_env,
policy_from_env,
summarize_ledger,
)
# ---------------------------------------------------------------------------
# fake livekit / websockets SDK(只满足本仓库代码用到的表面,进程内假 provider)
# ---------------------------------------------------------------------------
_FAKE_MODULES = (
"livekit",
"livekit.rtc",
"livekit.agents",
"livekit.agents.stt",
"livekit.agents.llm",
"livekit.agents.tts",
"livekit.agents.types",
"livekit.plugins",
"livekit.plugins.silero",
"websockets",
)
class FakeFrame:
def __init__(self, data: bytes, sample_rate: int, samples_per_channel: int) -> None:
self.data = data
self.sample_rate = sample_rate
self.samples_per_channel = samples_per_channel
class FakeResampler:
"""线性比例重采样替身:输出帧数与输入样本数严格成比例(16bit/单声道)."""
def __init__(self, in_rate: int, out_rate: int) -> None:
self.out_rate = out_rate
self.ratio = out_rate / in_rate
self._in_samples = 0
self._out_samples = 0
def push(self, frame: FakeFrame) -> list[FakeFrame]:
self._in_samples += frame.samples_per_channel
target = round(self._in_samples * self.ratio)
n = target - self._out_samples
if n <= 0:
return []
self._out_samples = target
return [FakeFrame(b"\x00" * (n * 2), self.out_rate, n)]
class FakeWS:
"""进程内 FunASR websocket 假 provider:记录每个出站字节."""
def __init__(self, server_msgs: list[str] | None = None, recv_fail: bool = False) -> None:
self.sent: list[bytes | str] = []
self.server_msgs = list(server_msgs or [])
self.recv_fail = recv_fail
self.closed = False
self.fail_send = False
async def send(self, payload: bytes | str) -> None:
if self.fail_send:
raise OSError("send refused")
self.sent.append(payload)
async def close(self) -> None:
self.closed = True
def __aiter__(self):
return self
async def __anext__(self):
while True:
if self.closed:
raise StopAsyncIteration
if self.recv_fail:
raise ConnectionError("recv died")
if self.server_msgs:
return self.server_msgs.pop(0)
await asyncio.sleep(0.005)
class FakeWebsockets:
"""websockets.connect 假 provider:可编程失败与假服务端响应."""
def __init__(self) -> None:
self.attempts = 0
self.fail_attempts: set[int] = set()
self.sockets: list[FakeWS] = []
self.ws_factory = None
def connect(self, url: str, max_size: int | None = None):
async def _connect() -> FakeWS:
self.attempts += 1
if self.attempts in self.fail_attempts:
raise OSError(f"connect refused (attempt {self.attempts})")
if self.ws_factory is not None:
ws = self.ws_factory()
else:
ws = FakeWS(server_msgs=[json.dumps(
{"mode": "2pass-offline", "text": "测试终稿"})])
self.sockets.append(ws)
return ws
return _connect()
def bytes_sent(self) -> list[bytes]:
return [p for ws in self.sockets for p in ws.sent if isinstance(p, bytes)]
class _InCh:
def __init__(self) -> None:
self.q: asyncio.Queue = asyncio.Queue()
def push(self, item) -> None:
self.q.put_nowait(item)
def end(self) -> None:
self.q.put_nowait(None)
def __aiter__(self):
return self
async def __anext__(self):
return await self.q.get()
class _OutCh:
def __init__(self) -> None:
self.q: asyncio.Queue = asyncio.Queue()
def send_nowait(self, item) -> None:
self.q.put_nowait(item)
def empty(self) -> bool:
return self.q.empty()
def get_nowait(self):
return self.q.get_nowait()
class SpeechEventType(enum.Enum):
START_OF_SPEECH = "start_of_speech"
INTERIM_TRANSCRIPT = "interim_transcript"
FINAL_TRANSCRIPT = "final_transcript"
END_OF_SPEECH = "end_of_speech"
class SpeechData:
def __init__(self, *, language: str, text: str) -> None:
self.language = language
self.text = text
class SpeechEvent:
def __init__(self, *, type, alternatives) -> None:
self.type = type
self.alternatives = alternatives
def _kwargs_ns(**kwargs):
return SimpleNamespace(**kwargs)
class FakeRecognizeStream:
class _FlushSentinel:
pass
def __init__(self, *, stt=None, conn_options=None) -> None:
self._stt = stt
self._conn_options = conn_options
self._input_ch = _InCh()
self._event_ch = _OutCh()
self._run_task: asyncio.Task | None = None
def _ensure_run(self) -> asyncio.Task:
if self._run_task is None:
self._run_task = asyncio.get_running_loop().create_task(self._run())
return self._run_task
def __aiter__(self):
self._ensure_run()
return self
async def __anext__(self):
self._ensure_run()
while True:
if not self._event_ch.empty():
return self._event_ch.get_nowait()
task = self._run_task
if task.done():
exc = task.exception()
if exc is not None:
raise exc
raise StopAsyncIteration
await asyncio.sleep(0.005)
class FakeAgentSession:
last: FakeAgentSession | None = None # 测试替身的类级记录
def __init__(self, **kwargs) -> None:
self.kwargs = kwargs
self.handlers: dict[str, list] = {}
self.started = None
FakeAgentSession.last = self
def on(self, ev: str):
def deco(fn):
self.handlers.setdefault(ev, []).append(fn)
return fn
return deco
async def start(self, *, room=None, agent=None) -> None:
self.started = (room, agent)
class FakeCtx:
"""JobContext 替身:记录 shutdown 调用."""
def __init__(self, room_name: str) -> None:
self.room = SimpleNamespace(
name=room_name,
local_participant=SimpleNamespace(
publish_data=self._publish_data),
)
self.proc = SimpleNamespace(userdata={"vad": object()})
self.shutdown_calls: list[str] = []
async def _publish_data(self, data, topic=None, reliable=False) -> None:
return None
async def connect(self) -> None:
return None
def shutdown(self, reason: str = "", message: str = "") -> None:
self.shutdown_calls.append(reason or message or "shutdown")
class _VAD:
@classmethod
def load(cls):
return cls()
def _install_fake_sdk(monkeypatch) -> SimpleNamespace:
rtc_mod = types.ModuleType("livekit.rtc")
rtc_mod.AudioFrame = FakeFrame
rtc_mod.AudioResampler = FakeResampler
stt_mod = types.ModuleType("livekit.agents.stt")
stt_mod.STT = type("STT", (), {"__init__": lambda self, *, capabilities=None: setattr(
self, "capabilities", capabilities)})
stt_mod.STTCapabilities = _kwargs_ns
stt_mod.SpeechData = SpeechData
stt_mod.SpeechEvent = SpeechEvent
stt_mod.SpeechEventType = SpeechEventType
stt_mod.RecognizeStream = FakeRecognizeStream
llm_mod = types.ModuleType("livekit.agents.llm")
for name in ("LLM", "ChatChunk", "ChatContext", "ChoiceDelta", "LLMStream", "Tool"):
setattr(llm_mod, name, type(name, (), {"__init__": lambda self, *a, **k: None}))
tts_mod = types.ModuleType("livekit.agents.tts")
tts_mod.TTS = type("TTS", (), {"__init__": lambda self, *, capabilities=None,
sample_rate=None, num_channels=None: None})
tts_mod.TTSCapabilities = _kwargs_ns
for name in ("AudioEmitter", "ChunkedStream"):
setattr(tts_mod, name, type(name, (), {}))
types_mod = types.ModuleType("livekit.agents.types")
types_mod.APIConnectOptions = type("APIConnectOptions", (), {})
types_mod.DEFAULT_API_CONNECT_OPTIONS = types_mod.APIConnectOptions()
types_mod.NOT_GIVEN = object()
types_mod.NotGivenOr = object
agents_mod = types.ModuleType("livekit.agents")
agents_mod.Agent = type("Agent", (), {"__init__": lambda self, *a, **k: None})
agents_mod.AgentSession = FakeAgentSession
for name in ("JobContext", "JobProcess"):
setattr(agents_mod, name, type(name, (), {}))
agents_mod.WorkerOptions = type("WorkerOptions", (), {"__init__": lambda self, **k: None})
agents_mod.cli = type("cli", (), {"run_app": staticmethod(lambda *a, **k: None)})
agents_mod.APIConnectionError = type("APIConnectionError", (Exception,), {})
agents_mod.APIStatusError = type("APIStatusError", (Exception,), {})
agents_mod.stt = stt_mod
agents_mod.llm = llm_mod
agents_mod.tts = tts_mod
agents_mod.types = types_mod
silero_mod = types.ModuleType("livekit.plugins.silero")
silero_mod.VAD = _VAD
plugins_mod = types.ModuleType("livekit.plugins")
plugins_mod.silero = silero_mod
livekit_mod = types.ModuleType("livekit")
livekit_mod.rtc = rtc_mod
livekit_mod.agents = agents_mod
livekit_mod.plugins = plugins_mod
websockets_mod = types.ModuleType("websockets")
fake_ws_module = FakeWebsockets()
websockets_mod.connect = fake_ws_module.connect
for name, mod in (
("livekit", livekit_mod),
("livekit.rtc", rtc_mod),
("livekit.agents", agents_mod),
("livekit.agents.stt", stt_mod),
("livekit.agents.llm", llm_mod),
("livekit.agents.tts", tts_mod),
("livekit.agents.types", types_mod),
("livekit.plugins", plugins_mod),
("livekit.plugins.silero", silero_mod),
("websockets", websockets_mod),
):
monkeypatch.setitem(sys.modules, name, mod)
for cached in ("cybergril.voice.stt_funasr", "cybergril.voice.worker",
"cybergril.voice.tts_cosyvoice", "cybergril.voice.llm_chat_api"):
monkeypatch.delitem(sys.modules, cached, raising=False)
return SimpleNamespace(websockets=fake_ws_module, rtc=rtc_mod, stt=stt_mod)
# ---------------------------------------------------------------------------
# helpers
# ---------------------------------------------------------------------------
def make_policy(**overrides) -> AsrWindowPolicy:
values = dict(window_id="w-test", max_audio_seconds=1000.0,
max_connect_retries_per_call=6, flush_seconds=5.0,
resync_seconds=1e9)
values.update(overrides)
return AsrWindowPolicy(**values)
def make_guard(tmp_path: Path, *, on_exhausted=None, **overrides) -> AsrUsageGuard:
return AsrUsageGuard(make_policy(**overrides), tmp_path / "asr-usage.jsonl",
on_exhausted=on_exhausted)
def frame_16k(n_samples: int = 160) -> FakeFrame:
return FakeFrame(b"\x00" * (n_samples * 2), 16000, n_samples)
def frame_48k(n_samples: int = 480) -> FakeFrame:
return FakeFrame(b"\x00" * (n_samples * 2), 48000, n_samples)
def read_records(tmp_path: Path) -> list[dict]:
path = tmp_path / "asr-usage.jsonl"
if not path.exists():
return []
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()
if line.strip()]
async def drive(module, stt, script, *, timeout: float = 10.0):
"""按脚本驱动真实 RecognizeStream(帧/flush/None=结束),等待跑完."""
stream = stt.stream()
task = stream._ensure_run()
for item in script:
if item is None:
stream._input_ch.end()
elif isinstance(item, str) and item == "flush":
stream._input_ch.push(module.RecognizeStream._FlushSentinel())
else:
stream._input_ch.push(item)
await asyncio.sleep(0)
await asyncio.wait_for(task, timeout=timeout)
return stream
# ---------------------------------------------------------------------------
# 1) guard 单元:计数口径、fail-closed、触线、恢复、隔离
# ---------------------------------------------------------------------------
def test_policy_env_armed_and_fail_closed_default():
assert policy_from_env({}).armed is False # 未布防 = 真实 cap=0
policy = policy_from_env({
"CYBERGRIL_ASR_WINDOW_ID": "hel902-w1",
"CYBERGRIL_ASR_WINDOW_MAX_AUDIO_SECONDS": "2700",
"CYBERGRIL_ASR_MAX_CONNECT_RETRIES_PER_CALL": "6",
})
assert policy.armed
assert policy.max_audio_seconds == 2700.0
assert policy.max_connect_retries_per_call == 6
# 配置损坏按 fail-closed(0)处理,不放宽
bad = policy_from_env({"CYBERGRIL_ASR_WINDOW_ID": "w",
"CYBERGRIL_ASR_WINDOW_MAX_AUDIO_SECONDS": "abc"})
assert bad.max_audio_seconds == 0.0
def test_unarmed_guard_blocks_everything_and_records(tmp_path):
guard = make_guard(tmp_path, window_id="")
with pytest.raises(AsrWindowClosed):
guard.begin_stream("room-a")
with pytest.raises(AsrWindowClosed):
guard.check_connect("room-a")
with pytest.raises(AsrWindowClosed):
guard.precheck_audio("room-a", 16000, 160)
blocks = [r for r in read_records(tmp_path) if r["kind"] == "block"]
assert {(b["what"], b["reason"]) for b in blocks} == {
("new_stream", "window-not-armed"),
("new_connection", "window-not-armed"),
("audio", "window-not-armed"),
}
def test_guard_audio_counted_by_actual_frame_samples(tmp_path):
guard = make_guard(tmp_path)
call = guard.begin_stream("room-a")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
# 16k 直接路径:160 样本 = 0.01s;重采样路径按输出帧 16k 计
for _ in range(10):
guard.precheck_audio(call, 16000, 160)
guard.note_audio_sent(call, 16000, 160, 320)
for _ in range(5):
guard.precheck_audio(call, 16000, 160)
guard.note_audio_sent(call, 16000, 160, 320) # 重发按重复计
guard.stream_end(call, "closed")
assert guard.calls[call].audio_seconds == pytest.approx(0.15)
audio = [r for r in read_records(tmp_path) if r["kind"] == "audio"]
assert sum(r["seconds"] for r in audio) == pytest.approx(0.15)
def test_guard_blocks_crossing_frame_before_send(tmp_path):
stops: list[str] = []
guard = make_guard(tmp_path, max_audio_seconds=2.0, on_exhausted=stops.append)
call = guard.begin_stream("room-a")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
guard.precheck_audio(call, 16000, 16000) # 1.0s
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.flush_audio(call)
guard.precheck_audio(call, 16000, 16000) # 恰好等于上限:允许(≤上限)
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.flush_audio(call)
with pytest.raises(AsrWindowClosed): # 第 3 秒会在发送前拒绝
guard.precheck_audio(call, 16000, 16000)
assert stops == ["window-audio-exhausted"] # 回调恰好一次
# 同窗新通也立即拒绝(新连接/新音频都不放行)
with pytest.raises(AsrWindowClosed):
guard.begin_stream("room-b")
blocks = [r for r in read_records(tmp_path) if r["kind"] == "block"]
assert {b["reason"] for b in blocks} == {"window-audio-exhausted"}
assert guard.window_audio_seconds == pytest.approx(2.0) # 一个字节没多
def test_guard_retry_semantics_only_after_failure(tmp_path):
guard = make_guard(tmp_path, max_connect_retries_per_call=2)
call = guard.begin_stream("room-a")
guard.check_connect(call) # 初次建连不是 retry
guard.note_connect_result(call, ok=True)
guard.check_connect(call) # 正常说完一轮后的下一次建连也不是 retry
guard.note_connect_result(call, ok=True)
assert guard.calls[call].connect_retries == 0
guard.check_connect(call) # 失败 → retry_pending
guard.note_connect_result(call, ok=False, error="OSError: refused")
guard.check_connect(call) # 失败后的再建连 = retry #1
guard.note_connect_result(call, ok=True)
guard.check_connect(call) # 再次失败 → retry_pending
guard.note_connect_result(call, ok=False, error="OSError: refused")
guard.check_connect(call) # retry #2(用尽上限)
guard.note_connect_result(call, ok=True)
cu = guard.calls[call]
assert (cu.connect_attempts, cu.connect_failures, cu.connect_retries) == (6, 2, 2)
guard.check_connect(call) # 上次成功 → 非 retry:不受 retry 上限约束
guard.note_connect_result(call, ok=True)
guard.check_connect(call) # 失败 → 置 retry_pending
guard.note_connect_result(call, ok=False, error="OSError: refused")
with pytest.raises(AsrWindowClosed): # retry #3 在建连前被拒
guard.check_connect(call)
cu = guard.calls[call]
assert (cu.connect_attempts, cu.connect_failures, cu.connect_retries) == (8, 3, 2)
# 新的一通有独立 retry 预算(每通口径)
call2 = guard.begin_stream("room-b")
guard.check_connect(call2)
guard.note_connect_result(call2, ok=True)
assert guard.calls["room-b"].connect_attempts == 1
def test_guard_ledger_recovery_same_window_no_reset(tmp_path):
guard1 = make_guard(tmp_path, max_audio_seconds=3.0)
call = guard1.begin_stream("room-a")
guard1.check_connect(call)
guard1.note_connect_result(call, ok=True)
guard1.check_connect(call)
guard1.note_connect_result(call, ok=False, error="x")
for _ in range(150):
guard1.precheck_audio(call, 16000, 160)
guard1.note_audio_sent(call, 16000, 160, 320)
guard1.flush_audio(call)
guard1.stream_end(call, "closed")
# 模拟进程重启:同窗口同账本重建 guard,预算与逐通历史都在
guard2 = make_guard(tmp_path, max_audio_seconds=3.0)
assert guard2.window_audio_seconds == pytest.approx(1.5)
cu = guard2.calls["room-a"]
assert (cu.connect_attempts, cu.connect_failures, cu.connect_retries) == (2, 1, 0)
assert cu.audio_seconds == pytest.approx(1.5)
assert cu.retry_pending is True # 最后一次连接事实是失败
# 耗尽同窗口剩余预算后拒绝;重开同窗口 id 不能重置
call = guard2.begin_stream("room-a")
guard2.check_connect(call)
guard2.note_connect_result(call, ok=True) # retry_pending=True → 这是 retry
with pytest.raises(AsrWindowClosed):
for _ in range(200):
guard2.precheck_audio(call, 16000, 160)
guard2.note_audio_sent(call, 16000, 160, 320)
guard2.flush_audio(call)
def test_guard_window_isolation_does_not_stop_others(tmp_path):
"""W1 耗尽不误停别的窗口/会话(一次性窗口各有独立预算)."""
g1 = make_guard(tmp_path, window_id="w1", max_audio_seconds=0.05)
with pytest.raises(AsrWindowClosed):
call = g1.begin_stream("room-a")
g1.check_connect(call)
g1.note_connect_result(call, ok=True)
for _ in range(10):
g1.precheck_audio(call, 16000, 160)
g1.note_audio_sent(call, 16000, 160, 320)
g2 = make_guard(tmp_path, window_id="w2", max_audio_seconds=100.0)
call = g2.begin_stream("room-a") # 同账本、不同窗口:完全不受 W1 影响
g2.check_connect(call)
g2.note_connect_result(call, ok=True)
for _ in range(10):
g2.precheck_audio(call, 16000, 160)
g2.note_audio_sent(call, 16000, 160, 320)
g2.stream_end(call, "closed")
assert g2.calls["room-a"].audio_seconds == pytest.approx(0.1)
def test_guard_crash_undercount_bounded_by_flush(tmp_path):
"""如实边界:崩溃前 ≤flush 阈值的已发音频未落账(有界少计)."""
guard = make_guard(tmp_path, flush_seconds=5.0)
call = guard.begin_stream("room-a")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
for _ in range(30):
guard.precheck_audio(call, 16000, 160)
guard.note_audio_sent(call, 16000, 160, 320) # 0.3s < flush 阈值
# 模拟进程直接消失(没有 stream_end/flush)
replayed = summarize_ledger(tmp_path / "asr-usage.jsonl", "w-test")
assert replayed["windows"]["w-test"]["audio_seconds"] == 0.0
# 正常路径(flush/stream_end)计数完整落账
guard2 = make_guard(tmp_path, flush_seconds=5.0)
call = guard2.begin_stream("room-b")
guard2.note_audio_sent(call, 16000, 16000, 32000)
guard2.stream_end(call, "closed")
assert summarize_ledger(tmp_path / "asr-usage.jsonl", "w-test")["windows"][
"w-test"]["calls"]["room-b"]["audio_seconds"] == pytest.approx(1.0)
def test_summarize_cli_reproducible(tmp_path, monkeypatch):
guard = make_guard(tmp_path)
call = guard.begin_stream("room-a")
guard.check_connect(call)
guard.note_connect_result(call, ok=True)
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.stream_end(call, "closed")
repo_root = Path(__file__).resolve().parents[1]
proc = subprocess.run(
[sys.executable, "-m", "cybergril.voice.asr_usage",
str(tmp_path / "asr-usage.jsonl"), "--window", "w-test"],
capture_output=True, text=True, timeout=60,
env={"PYTHONPATH": str(repo_root / "src"), "PATH": "/usr/bin:/bin"},
check=True,
)
out = json.loads(proc.stdout)
call_summary = out["windows"]["w-test"]["calls"]["room-a"]
assert call_summary["audio_seconds"] == pytest.approx(1.0)
assert call_summary["connect_attempts"] == 1
assert out["parse_errors"] == 0
# ---------------------------------------------------------------------------
# 2) 真实接缝:stt_funasr._run 在假 provider/假帧上走完整送数路径
# ---------------------------------------------------------------------------
@pytest.fixture
def fake_sdk(monkeypatch):
return _install_fake_sdk(monkeypatch)
async def test_seam_counts_fed_audio_both_sample_rates(fake_sdk, tmp_path):
import cybergril.voice.stt_funasr as funasr
guard = make_guard(tmp_path)
stt = funasr.FunASRSTT(usage=guard, call_label="cg-voice-abc")
await drive(funasr, stt, [frame_16k() for _ in range(30)] + ["flush"]
+ [frame_48k() for _ in range(30)] + [None])
cu = guard.calls["cg-voice-abc"]
# 16k 直接路径 0.3s + 48k→16k 重采样路径 0.3s(按实际输出帧计时)
assert cu.audio_seconds == pytest.approx(0.6)
assert cu.connect_attempts == 2 # 两轮 utterance 两次建连
assert (cu.connect_failures, cu.connect_retries) == (0, 0)
records = read_records(tmp_path)
audio_records = [r for r in records if r["kind"] == "audio"]
assert [round(r["seconds"], 6) for r in audio_records] == [0.3, 0.3] # utterance 边界聚合
assert all(r["call"] == "cg-voice-abc" for r in records)
ends = [r for r in records if r["kind"] == "call_end"]
assert [r["reason"] for r in ends] == ["closed"]
# 假 provider 侧:每帧恰好一次出站,cfg/is_speaking 各一条文本
audio_payloads = fake_sdk.websockets.bytes_sent()
assert len(audio_payloads) == 60
assert all(len(p) == 320 for p in audio_payloads)
async def test_seam_silence_and_cancel_counted(fake_sdk, tmp_path):
import cybergril.voice.stt_funasr as funasr
guard = make_guard(tmp_path)
stt = funasr.FunASRSTT(usage=guard, call_label="room-silent")
# 静音数据帧只要真实送出就计数;不 flush 直接结束(用户取消/挂断)
silence = FakeFrame(b"\x00" * 320, 16000, 160)
await drive(funasr, stt, [silence] * 20 + [None])
cu = guard.calls["room-silent"]
assert cu.audio_seconds == pytest.approx(0.2)
records = read_records(tmp_path)
assert [r["reason"] for r in records if r["kind"] == "call_end"] == ["closed"]
# 反复结束:第二通历史不覆盖第一通
stt2 = funasr.FunASRSTT(usage=guard, call_label="room-silent-2")
await drive(funasr, stt2, [frame_16k(1600)] + [None]) # 0.1s
assert guard.calls["room-silent-2"].audio_seconds == pytest.approx(0.1)
assert len([r for r in read_records(tmp_path)
if r["kind"] == "call_start"]) == 2
async def test_seam_connect_failure_marks_error_and_retry_denied(fake_sdk, tmp_path):
import cybergril.voice.stt_funasr as funasr
guard = make_guard(tmp_path, max_connect_retries_per_call=2)
stt = funasr.FunASRSTT(usage=guard, call_label="room-flaky")
fake_sdk.websockets.fail_attempts = {1, 3, 4}
with pytest.raises(OSError): # 初次建连失败 → 流异常结束
await drive(funasr, stt, [frame_16k(), None])
records = read_records(tmp_path)
assert records[-1]["kind"] == "call_end" and records[-1]["reason"] == "error"
stt2 = funasr.FunASRSTT(usage=guard, call_label="room-flaky")
await drive(funasr, stt2, [frame_16k(), "flush", None]) # retry#1 成功
assert guard.calls["room-flaky"].connect_retries == 1
stt3 = funasr.FunASRSTT(usage=guard, call_label="room-flaky")
with pytest.raises(OSError): # 上次成功后这次是普通建连,但连接失败
await drive(funasr, stt3, [frame_16k(), None])
stt4 = funasr.FunASRSTT(usage=guard, call_label="room-flaky")
with pytest.raises(OSError): # retry#2(用尽上限)且仍失败
await drive(funasr, stt4, [frame_16k(), None])
stt5 = funasr.FunASRSTT(usage=guard, call_label="room-flaky")
attempts_before = fake_sdk.websockets.attempts
with pytest.raises(AsrWindowClosed): # retry#3 建连前被拒:连假 provider 都不碰
await drive(funasr, stt5, [frame_16k(), None])
assert fake_sdk.websockets.attempts == attempts_before
blocks = [r for r in read_records(tmp_path) if r["kind"] == "block"]
assert blocks[-1]["reason"] == "connect-retries-exhausted"
cu = guard.calls["room-flaky"]
assert (cu.connect_attempts, cu.connect_failures, cu.connect_retries) == (4, 3, 2)
stt5 = funasr.FunASRSTT(usage=guard, call_label="room-flaky")
with pytest.raises(AsrWindowClosed): # 该通 retry 预算已用尽:再建连被拒
await drive(funasr, stt5, [frame_16k(), None])
# 新的一通有独立 retry 预算:两次正常 utterance 建连都放行
stt6 = funasr.FunASRSTT(usage=guard, call_label="room-flaky-2")
await drive(funasr, stt6, [frame_16k(), "flush", frame_16k(), "flush", None])
assert guard.calls["room-flaky-2"].connect_attempts == 2
assert guard.calls["room-flaky-2"].connect_retries == 0
async def test_seam_window_exhaustion_stops_before_crossing(fake_sdk, tmp_path):
import cybergril.voice.stt_funasr as funasr
stops: list[str] = []
guard = AsrUsageGuard(
make_policy(max_audio_seconds=0.05),
tmp_path / "asr-usage.jsonl", on_exhausted=stops.append)
stt = funasr.FunASRSTT(usage=guard, call_label="room-cap")
with pytest.raises(AsrWindowClosed):
await drive(funasr, stt, [frame_16k() for _ in range(6)] + [None])
# 0.05s = 5 帧放行,第 6 帧发送前拒绝:出站字节恰好 5 帧
assert len(fake_sdk.websockets.bytes_sent()) == 5
assert guard.calls["room-cap"].audio_seconds == pytest.approx(0.05)
assert stops == ["window-audio-exhausted"]
records = read_records(tmp_path)
assert records[-1]["kind"] == "call_end"
assert records[-1]["reason"] == "window_exhausted"
assert any(r["kind"] == "block" for r in records)
# 本窗口后续会话:新流在建流即被拒
stt2 = funasr.FunASRSTT(usage=guard, call_label="room-cap-2")
with pytest.raises(AsrWindowClosed):
await drive(funasr, stt2, [frame_16k(), None])
async def test_seam_recv_death_recorded_as_connection_died(fake_sdk, tmp_path):
import cybergril.voice.stt_funasr as funasr
guard = make_guard(tmp_path)
fake_sdk.websockets.ws_factory = lambda: FakeWS(recv_fail=True)
stt = funasr.FunASRSTT(usage=guard, call_label="room-dead")
await drive(funasr, stt, [frame_16k(), "flush", None], timeout=6)
records = read_records(tmp_path)
died = [r for r in records if r["kind"] == "connection_died"]
assert len(died) == 1 and died[0]["call"] == "room-dead"
assert guard.calls["room-dead"].connect_failures == 1
assert guard.calls["room-dead"].retry_pending is True
# 死亡后的下一次建连计为 retry(接收侧死亡未伴随建连失败也如实计数)
fake_sdk.websockets.ws_factory = None
stt2 = funasr.FunASRSTT(usage=guard, call_label="room-dead")
await drive(funasr, stt2, [frame_16k(), "flush", None])
assert guard.calls["room-dead"].connect_retries == 1
async def test_ledger_survives_room_deletion(fake_sdk, tmp_path):
"""删房成功会话账本移除该通;ASR 账本逐通历史原样保留."""
import cybergril.voice.stt_funasr as funasr
from cybergril.voice.session import VoiceConfig, VoiceSessionService
guard = make_guard(tmp_path)
stt = funasr.FunASRSTT(usage=guard, call_label="cg-voice-keep")
await drive(funasr, stt, [frame_16k(1600), "flush", None])
before = read_records(tmp_path)
service = VoiceSessionService(
config=VoiceConfig(livekit_url="ws://v.test", livekit_internal="http://v.test",
api_key="devkey", api_secret="s"),
http=httpx.AsyncClient(transport=httpx.MockTransport(_fake_twirp)))
session = await service.open_session(epoch=1)
ended = await service.end_session(session.session_id)
assert ended is not None
assert service.get(session.session_id) is None # 会话账本已移除
assert read_records(tmp_path) == before # ASR 账本一字未动
await service.aclose()
def _fake_twirp(request: httpx.Request) -> httpx.Response:
if request.url.path.endswith("CreateDispatch"):
return httpx.Response(200, json={"id": "AD_1"})
return httpx.Response(200, content=b"")
# ---------------------------------------------------------------------------
# 3) worker 接线:entrypoint 真的构造并注入 guard,触线真的关 job
# ---------------------------------------------------------------------------
async def test_worker_entrypoint_wires_guard(fake_sdk, tmp_path, monkeypatch, capsys):
import cybergril.voice.worker as worker
ledger = tmp_path / "worker-asr.jsonl"
env = {"CYBERGRIL_ASR_WINDOW_ID": "hel902-w1",
"CYBERGRIL_ASR_WINDOW_MAX_AUDIO_SECONDS": "2700",
"CYBERGRIL_ASR_MAX_CONNECT_RETRIES_PER_CALL": "6",
"CYBERGRIL_ASR_USAGE_LEDGER": str(ledger)}
monkeypatch.setattr(worker, "build_usage_from_env",
lambda on_exhausted=None: build_usage_from_env(
env, on_exhausted=on_exhausted, ledger_path=ledger))
ctx = FakeCtx("cg-voice-room1")
await worker.entrypoint(ctx)
captured = FakeAgentSession.last
assert captured is not None and captured.started is not None
stt_obj = captured.kwargs["stt"]
assert stt_obj._call_label == "cg-voice-room1" # 房间名 = 每通标识
guard = stt_obj._usage
assert isinstance(guard, AsrUsageGuard)
assert guard.policy.window_id == "hel902-w1"
out = capsys.readouterr().out
assert "asr_window=" in out and "hel902-w1" in out
# 触线回调真的接到 ctx.shutdown:预检越限帧 → 结束本通 job
call = guard.begin_stream("cg-voice-room1")
guard.note_audio_sent(call, 16000, 16000, 32000)
guard.flush_audio(call)
with pytest.raises(AsrWindowClosed):
guard.precheck_audio(call, 16000, 16000 * 2700) # 会越 2700s 上限的帧
assert ctx.shutdown_calls, "on_exhausted 必须真的调用 ctx.shutdown 结束本通"
async def test_worker_entrypoint_default_is_fail_closed(fake_sdk, monkeypatch, capsys):
import cybergril.voice.worker as worker
monkeypatch.setattr(
worker, "build_usage_from_env",
lambda on_exhausted=None: build_usage_from_env({}, on_exhausted=on_exhausted,
ledger_path=None))
ctx = FakeCtx("cg-voice-room2")
await worker.entrypoint(ctx)
guard = FakeAgentSession.last.kwargs["stt"]._usage
assert guard.policy.armed is False # 未布防 = 真实 cap=0
with pytest.raises(AsrWindowClosed):
guard.begin_stream("cg-voice-room2")
assert "asr_window=" in capsys.readouterr().out