Files
CyberGril/tests/test_hard_gates_hel882.py
T
zonggongandmultica-agent 78b6267d3a HEL-885 rework: per-row progress matches what was really sent, cross-day and unknown-outcome accounting stay conservative, positive caps require the ledger authority, and the release dependency-fill command actually exists - every Memory.add now carries an anchor for exactly its own row (the HEL-882 whole-batch anchor let the first stored vector mark unsent rows complete; the manager's real-SDK partial reproduction showed the user's original words never leaving the process), a row finishing with a legitimately EMPTY extraction gets a durable sem_row_done record (r4.sql additive table) written inside the worker thread the moment its add completes, resume skips = vector-store anchors ∪ sem_row_done so no row is lost, none falsely complete, and finished work is never re-sent across budget-refusal/crash/restart/lease/concurrency, entry_ids merge across partial attempts, and the swallowed-failure detection is per-row; PersistentBudget.settle/release book on hold.day (the old write to state.day manufactured -80 used on the new day and re-opened headroom the reservation day had already spent; token_usage/request_usage refund the paying day, budget_total stays global) and the empty-budget_total TypeError crash in total_requests_used/total_tokens_used is fixed - the old one-shot-total test passed only because that crash surfaced as chat.error, it now books the total honestly; _atomic_reserve no longer purges aged reserve: rows (a crashed worker's hold is an UNKNOWN outcome - an hour-old timeout is not proof the call never happened; unknown consumption stays billed across restarts, only a provably-unsent release() ever removes a hold, embed_ops and llm_ops share the fix); Mem0Backend refuses construction before importing mem0 when any cap>0 lacks a (non-empty dict) authority block - ledger_authority_required - or when the prefix check fails, closing the fresh-empty-ledger bypass the manager reproduced live, config/semantic.example.yaml ships the authority block as REQUIRED with the explicit cumulative formula (embed_cap=500+approved-increment e.g. 680, embed_reserve=2 frozen untouched, min_used=498; never write 180 as the cumulative cap of a 498-used account) mirrored into docs/budget-plan-hel874.md §2 with the HEL-885 accounting rules; deploy/hel874-release.md §4 replaces the nonexistent uv pip download (rc=2, re-verified) with python3 -m pip download --only-binary :all: --platform manylinux2014_x86_64 --python-version 3.11 --implementation cp --abi cp311 + sha256 manifest + uv pip install --no-index --find-links offline merge + same-process co-import verdict, honest missing-materials list unchanged (no 3.11 wheels cached, zero downloads this card, no version swaps), and tools/check_release_offline_hel882.py gains R7 probing both command surfaces offline; verification: 281 python tests pass on 3.11.16 venv + requirements.lock offline (9 new tests/test_rework_hel885.py cases: partial batch claims only the sent row / empty-extraction row completes and is never resent / process restart sends only the missing row / concurrent ticks never double-send / cross-day settle and release book on the reservation day / aged unknown llm+embed reservations never refunded / missing-empty-malformed-wrong authority all fail closed; test_resume_never_loses_or_duplicates_rows rewritten to drive the REAL product path instead of hand-planting single-row anchors the product never wrote), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 install on the deterministic boundary: contract re-run 12/12 (harness now hands a synthetic authority prefix - production defaults not relaxed), new tools/verify_semantic_recovery_hel885.py 7/7 (S0 authority fail-closed, S1 per-row partial with exactly 1 HTTP, S2 restart sends only the missing row proven by HTTP count since mem0 prompts legitimately carry old rows in rolling history, S3 lease recovery + concurrent ticks single-send, S4a/S4b unknown reservations kept), both manager review scripts re-run VERBATIM against this head (partial: completed=[2] only, resume done with 2 entries, HTTP=2, user words actually sent; extra: new-day used=0, over-cap request refused, aged reservation second_allowed=false, uv probe still rc=2), release offline check R1-R7 green, dual-target requirement checks 4/4 with handoff-HEL-885.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
Co-authored-by: multica-agent <github@multica.ai>
2026-10-27 00:01:44 +08:00

197 lines
9.3 KiB
Python

"""HEL-882 hard gates: the execution table's caps stop calls at the wire.
Reproduces the manager's independent review failure first (api review:
daily_tokens=1 with 2962 already used still fired a provider request and
returned chat.final), then proves the gate: every real provider exit — main
dialogue, structured background scenes, and the mem0 SDK's internal extraction
LLM — pre-allocates persistently before the send and refuses at the tightest
cap with ZERO provider calls. Counters survive restarts.
"""
import pytest
from conftest import engineering_budget
from fastapi.testclient import TestClient
from test_api import CREATE, events, turn
from cybergril.api.app import create_app
from cybergril.contracts import LLMDelta
from cybergril.life.runtime import LifeConfig
from cybergril.memory.store import Store
from cybergril.think.budget import BudgetConfig, BudgetGuard, BudgetStop
from cybergril.think.router import build_router
from cybergril.think.scenes import Scene, Tier
class GateLLM:
"""Fake provider that records every stream call."""
def __init__(self):
self.calls = []
async def stream(self, messages, **kwargs):
self.calls.append(messages)
if messages and "只返回JSON" in messages[0].content:
yield LLMDelta(text='{"ok": true}')
yield LLMDelta(done=True, finish_reason="stop", usage={"total_tokens": 12})
return
yield LLMDelta(text="好")
yield LLMDelta(text="的")
yield LLMDelta(done=True, finish_reason="stop", usage={"total_tokens": 12})
async def aclose(self):
return None
def gated_app(tmp_path, *, budget=None, db="gates.db"):
provider = GateLLM()
router = build_router(
providers={t: provider for t in (Tier.HIGH, Tier.STANDARD, Tier.LOW)},
budget=budget or engineering_budget(),
)
app = create_app(db_path=tmp_path / db, router=router,
life_config=LifeConfig(enabled=True, poll_seconds=3600),
semantic_mode="off")
return app, provider
def create_companion(client):
assert client.post("/api/companion", json=CREATE).status_code == 201
def test_daily_token_wall_refuses_before_any_provider_call(tmp_path):
"""The review's exact reproduction: over the daily token top, a turn must
not reach the provider — honest chat.error, counters unchanged."""
app, provider = gated_app(tmp_path)
with TestClient(app) as client:
create_companion(client)
guard = app.state.life.router.guard
guard.config.daily_tokens = 1
guard.record(2962, "review_budget_already_used")
used_before, calls_before = guard.used, len(provider.calls)
result = [e["t"] for e in events(turn(client, "合成测试:额度用完后仍能发出请求吗"))]
assert result == ["chat.error"] # honest refusal, never a fake reply
assert len(provider.calls) == calls_before # ZERO provider requests
assert guard.used == used_before # nothing further pre-allocated
def test_dialogue_request_cap_persists_across_restart(tmp_path):
"""Category daily request caps are persistent: 1 dialogue request/day lets
exactly one turn through; a second turn — in the SAME process or after a
full restart on the same database — refuses with zero provider calls."""
budget = engineering_budget(daily_requests={"dialogue": 1})
app, provider = gated_app(tmp_path, budget=budget)
with TestClient(app) as client:
create_companion(client)
assert [e["t"] for e in events(turn(client, "第一句"))][-1] == "chat.final"
calls_after_first = len(provider.calls)
assert [e["t"] for e in events(turn(client, "第二句"))] == ["chat.error"]
assert len(provider.calls) == calls_after_first
# restart: fresh app, fresh router, SAME database — the cap still holds
app2, _provider2 = gated_app(tmp_path, budget=budget)
with TestClient(app2) as client:
assert [e["t"] for e in events(turn(client, "重启后"))] == ["chat.error"]
assert app2.state.life.router.guard.requests_used("dialogue") == 1
@pytest.mark.asyncio
async def test_background_category_caps_stop_structured_calls(tmp_path):
"""structured() exits (life events, gists, judgments) count into their own
category: a 1/day life cap refuses the second generation before the wire."""
app, provider = gated_app(tmp_path, budget=engineering_budget(
daily_requests={"dialogue": 100, "life": 1, "background": 100}))
with TestClient(app) as client:
create_companion(client)
guard = app.state.life.router.guard
assert guard.requests_used("background") >= 0 # persistent guard wired
calls_before = len(provider.calls)
from cybergril.life.model import structured
scene = Scene.LIFE_EVENT
payload = {"time": "2026-10-26T10:00:00", "persona": "p", "activity_state": {}}
raw, _ = await structured(app.state.life.router, app.state.life.models, scene,
'返回 {"ok": true}', payload)
assert raw == {"ok": True}
assert len(provider.calls) == calls_before + 1
with pytest.raises(BudgetStop, match="budget_requests_exhausted:life"):
await structured(app.state.life.router, app.state.life.models, scene,
'返回 {"ok": true}', payload)
assert len(provider.calls) == calls_before + 1 # refused before the wire
def test_default_config_is_fail_closed_for_every_category():
guard = BudgetGuard(BudgetConfig())
for scene in ("dialogue_main", "life_event", "memory_gist", "mem0_extract"):
with pytest.raises(BudgetStop):
guard.acquire(scene, requests=1, tokens=10)
def test_in_memory_guard_tightest_constraint_and_settlement():
guard = BudgetGuard(BudgetConfig(daily_tokens=1000, daily_requests={"dialogue": 2},
total_requests=3), today="2026-10-26")
hold1 = guard.acquire("dialogue_main", requests=1, tokens=300)
assert guard.requests_used("dialogue") == 1
# daily token wall: the tightest cap decides, not the request count
with pytest.raises(BudgetStop, match="budget_tokens_exhausted"):
guard.acquire("dialogue_main", requests=1, tokens=800)
hold2 = guard.acquire("dialogue_main", requests=1, tokens=100)
# category daily top reached
with pytest.raises(BudgetStop, match="budget_requests_exhausted:dialogue"):
guard.acquire("proactive_message", requests=1, tokens=10)
# one-shot total: third request would pass the daily top of 2? no — category
# top already 2/2; use another category for the total check
guard3 = BudgetGuard(BudgetConfig(daily_tokens=1000, daily_requests={"dialogue": 10},
total_requests=2), today="2026-10-26")
guard3.acquire("dialogue_main", requests=1, tokens=10)
guard3.acquire("dialogue_main", requests=1, tokens=10)
with pytest.raises(BudgetStop, match="budget_total_requests_exhausted"):
guard3.acquire("dialogue_main", requests=1, tokens=10)
# settle: known usage adjusts, unknown keeps the whole reservation
guard.settle(hold1, actual_tokens=250)
assert guard.used == 350 # 250 + hold2's 100
guard.settle(hold2, actual_tokens=None) # disconnect: keep full reserve
assert guard.used == 350
# release: a provably-unsent hold refunds requests and tokens
guard4 = BudgetGuard(BudgetConfig(daily_tokens=1000,
daily_requests={"dialogue": 1}), today="2026-10-26")
hold = guard4.acquire("dialogue_main", requests=1, tokens=100)
guard4.release(hold)
assert guard4.used == 0 and guard4.requests_used("dialogue") == 0
def test_persistent_budget_counters_survive_restart(tmp_path):
from cybergril.life.model import PersistentBudget
store = Store(tmp_path / "persist.db")
guard = PersistentBudget(store, BudgetGuard(engineering_budget(), today="2026-10-26"))
guard.acquire("dialogue_main", requests=1, tokens=111)
guard.acquire("mem0_extract", requests=2, tokens=22)
store.close()
store2 = Store(tmp_path / "persist.db")
guard2 = PersistentBudget(store2, BudgetGuard(engineering_budget(), today="2026-10-26"))
assert guard2.requests_used("dialogue") == 1
assert guard2.requests_used("semantic") == 2
assert guard2.used == 133 # restart did not reset the day's tokens
store2.close()
def test_total_tokens_cap_blocks_before_daily_wall(tmp_path):
app, provider = gated_app(tmp_path, budget=engineering_budget(
daily_requests={"dialogue": 100}, total_tokens=5000))
with TestClient(app) as client:
create_companion(client)
guard = app.state.life.router.guard
# HEL-885: the one-shot TOTAL must be booked in budget_total — the old
# setup used record(), which only writes the DAILY table, and passed
# merely because total_tokens_used() CRASHED on the then-empty
# budget_total (TypeError surfaced as chat.error). Book the total
# honestly now that the read is fixed.
with guard.store.conn:
guard.store.execute(
"INSERT INTO budget_total VALUES('total_tokens',4999) "
"ON CONFLICT(key) DO UPDATE SET value=4999")
calls_before = len(provider.calls)
assert [e["t"] for e in events(turn(client, "还会发吗"))] == ["chat.error"]
assert len(provider.calls) == calls_before