Files
CyberGril/docs/rework
zonggongandmultica-agent 3228b1e0d7 HEL-879 remaining closure: auto-expiry cleanup ownership, exact SDK speech-turn attribution, and the versioned v2 subtitle contract for HEL-898
A. Room lifecycle (总管 review items A-1/A-2): the single cleanup owner now watches
   ACTIVE session deadlines as well as debts - a session nobody touches expires into
   the ledger and has its room really deleted with zero further requests (old code:
   _cleanup_loop only scanned debts, expired rooms were never deleted until some
   request ran _prune). Explicit end and the background sweep share ONE persisted
   attempt/age cap and abandoned state - _sessions no longer bypasses it (the
   manager measured 2 background attempts + 3 more interactive deletes = 5); an
   abandoned debt raises CleanupAbandoned -> 503 with room/attempts/reason instead
   of pretending done-404, success clears both _sessions and _debts, restart
   continues the same persisted cap, close/cancel leaves no dangling coroutine, and
   only this instance's own rooms are ever touched. New wake event makes a fresh
   debt retry immediately instead of waiting out a deadline sleep.

B. Queued speech turns bind EXACTLY (总管 interleaving note_handle(H1)->H2->
   note_stream(S1)->S2 gave H1=None, H2='turn-one'): CyberGrillLLM.chat now reads
   the SDK's own agents_speech_handle ContextVar (livekit-agents 1.8.8 sets it in
   _create_speech_task around every speech pipeline; source-verified on .12 and
   asserted at runtime) so each ChatApiLLMStream carries speech_handle/speech_id of
   the very handle whose generation created it. TurnLedger binds only on that exact
   attribution - a newer speech NEVER retires an older pending handle, say/proactive
   generations bind nobody, missing attribution fails closed (no ack, no subtitle
   turn), interrupt/failure discards only its own binding.

C. Versioned subtitle contract v2 for the visual author's wiring card (HEL-898):
   every cg.subtitle payload gains {v:2, seq} (session-monotonic), turn identity
   (speech_id, turn_id, same-source message_id for proactive rows) and explicit
   lifecycle events - turn_start/assistant_interim/assistant_final/turn_end/
   turn_cancel(reason) plus proactive_start/proactive_end/proactive_cancel that
   ride the say SpeechHandle instead of the receipt branch. Accumulators are
   isolated per speech_id (a late delta after cancel is dropped producer-side);
   seq + speech_id give the frontend two independent stale-rejection rules. Full
   executable contract: docs/voice-subtitle-contract-v2.md with recorded samples in
   docs/rework/hel879/subtitle-samples/ (scripts/gen_subtitle_samples_hel879.py).
   web/src untouched.

Verification: the .12 real-SDK check now LOADS THE ACTUAL WORKER MODULE and drives
the callbacks IT registered on a real AgentSession (no hand-copied functions) -
both-queued delayed generations bind exactly, a say sandwiched between user turns
steals nothing, unattributed generations fail closed, interrupted turns cancel
without final/receipt (11/11 ALL PASS, source-level proof of the ContextVar wiring
included; evidence docs/rework/hel879/sdk-check-hel879-remaining.log). The old r2
lifespan test flake is root-caused (clock freeze and block were two separate portal
calls; a 10ms life tick could slip between them and claim the frozen day unblocked)
and fixed atomically without raising timeouts or dropping assertions - 12/12 solo,
full suite 350/350 three times in a row on .venv311; ruff back to the single
pre-existing error; voice chain 8/8; release offline R1-R7 with zero failures;
requirement checks 4/4 (origin/main and c121c42, each with/without handoff).
Zero real LLM/embed/ASR/TTS calls, zero downloads/installs, nothing deployed or
merged, real caps stay 0.

Co-authored-by: multica-agent <github@multica.ai>
2026-10-27 05:10:52 +08:00
..
HEL-882 rework: execution-table caps become real hard gates at the wire, SDK hidden retries and mid-add failures recover without loss, the 8360 release package runs offline, and the HEL-874 embedding evidence is corrected - every real provider exit (create/dialogue/life/gist/judgment/growth/proactive phrasing + mem0's internal extraction LLM) now persistently pre-allocates request count (per-category daily + one-shot totals in request_usage/budget_total, BEGIN IMMEDIATE) and a conservative token estimate before sending, refuses at the tightest cap (BudgetStop, no more downgrade-and-keep-calling) and survives restarts; known usage settles to actual, unknown/disconnect/failure keeps the full reservation; ledger guards gain per-day tops (embed_daily/llm_daily) and verify_ledger_authority rejects a fresh file bypassing the frozen 498/500 prefix; the mem0 SDK client's hidden OpenAI retries are disabled (max_retries=0) so every actual send is gated and answered failures (even 5xx) settle as used with only httpx.ConnectError-grade unsent requests released; embed refusals the SDK swallows into empty adds are re-raised as budget errors so the batch re-queues without burning an attempt and resumes when the cap returns, add_group resumes at vector-store level (src_group anchors: only missing rows resent, crash/restart/concurrency never duplicate), sem coverage is per-backend (stub batches no longer block a real rebuild, real done batches never resend) with backend-scoped batch keys; the release package is offline-executable: fresh-dir git clone, cybergril service account, PYTHONPATH in the unit so the lock-only venv imports the product, shipped caps explicitly 0 (=refuse by design until approved), deploy/mem0-stack.lock + accurate 3.11 wheel gap with a no-download path (hel857-venv is 3.10; never present stub/FTS as the real memory experience), and tools/check_release_offline_hel882.py verifies checkout/import/health/fail-closed/approved-run/data-preservation on loopback with echo stand-ins; evidence corrected per manager review: docs/rework/hel874/erratum-hel882.json + README appendix record that e978e87's verification did real local embeddings from cache (provable lower bound 12, exact total unrecoverable, temp sandbox gone) while zero-real-LLM/zero-download/zero-outbound stand, old JSONs and authoritative ledgers untouched, and tools/verify_mem0_real_hel874.py now installs the deterministic 512-dim stand-in before TextEmbedding construction plus non-loopback socket rejection (zero model loads) with final_embed_rows persisted before temp cleanup; verification: 272 python tests pass on 3.11.16 venv + requirements.lock offline (16 new: manager api repro blocked with 0 provider calls, sdk repro 1-HTTP/max_retries=0/llm_ops=1, mid-add swallow requeued-then-resumed, resume-no-duplicate, stub-to-mem0 rebuild, daily embed cap, unified mem0-LLM gate, authority refusal, restart-persistent counters, tightest-constraint settle/release), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned-install deterministic re-run 12/12, manager review scripts re-run evidence attached in delivery, dual-target requirement checks 4/4 with handoff-HEL-882.json; zero real calls, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-26 23:20:18 +08:00
HEL-885 rework: per-row progress matches what was really sent, cross-day and unknown-outcome accounting stay conservative, positive caps require the ledger authority, and the release dependency-fill command actually exists - every Memory.add now carries an anchor for exactly its own row (the HEL-882 whole-batch anchor let the first stored vector mark unsent rows complete; the manager's real-SDK partial reproduction showed the user's original words never leaving the process), a row finishing with a legitimately EMPTY extraction gets a durable sem_row_done record (r4.sql additive table) written inside the worker thread the moment its add completes, resume skips = vector-store anchors ∪ sem_row_done so no row is lost, none falsely complete, and finished work is never re-sent across budget-refusal/crash/restart/lease/concurrency, entry_ids merge across partial attempts, and the swallowed-failure detection is per-row; PersistentBudget.settle/release book on hold.day (the old write to state.day manufactured -80 used on the new day and re-opened headroom the reservation day had already spent; token_usage/request_usage refund the paying day, budget_total stays global) and the empty-budget_total TypeError crash in total_requests_used/total_tokens_used is fixed - the old one-shot-total test passed only because that crash surfaced as chat.error, it now books the total honestly; _atomic_reserve no longer purges aged reserve: rows (a crashed worker's hold is an UNKNOWN outcome - an hour-old timeout is not proof the call never happened; unknown consumption stays billed across restarts, only a provably-unsent release() ever removes a hold, embed_ops and llm_ops share the fix); Mem0Backend refuses construction before importing mem0 when any cap>0 lacks a (non-empty dict) authority block - ledger_authority_required - or when the prefix check fails, closing the fresh-empty-ledger bypass the manager reproduced live, config/semantic.example.yaml ships the authority block as REQUIRED with the explicit cumulative formula (embed_cap=500+approved-increment e.g. 680, embed_reserve=2 frozen untouched, min_used=498; never write 180 as the cumulative cap of a 498-used account) mirrored into docs/budget-plan-hel874.md §2 with the HEL-885 accounting rules; deploy/hel874-release.md §4 replaces the nonexistent uv pip download (rc=2, re-verified) with python3 -m pip download --only-binary :all: --platform manylinux2014_x86_64 --python-version 3.11 --implementation cp --abi cp311 + sha256 manifest + uv pip install --no-index --find-links offline merge + same-process co-import verdict, honest missing-materials list unchanged (no 3.11 wheels cached, zero downloads this card, no version swaps), and tools/check_release_offline_hel882.py gains R7 probing both command surfaces offline; verification: 281 python tests pass on 3.11.16 venv + requirements.lock offline (9 new tests/test_rework_hel885.py cases: partial batch claims only the sent row / empty-extraction row completes and is never resent / process restart sends only the missing row / concurrent ticks never double-send / cross-day settle and release book on the reservation day / aged unknown llm+embed reservations never refunded / missing-empty-malformed-wrong authority all fail closed; test_resume_never_loses_or_duplicates_rows rewritten to drive the REAL product path instead of hand-planting single-row anchors the product never wrote), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 install on the deterministic boundary: contract re-run 12/12 (harness now hands a synthetic authority prefix - production defaults not relaxed), new tools/verify_semantic_recovery_hel885.py 7/7 (S0 authority fail-closed, S1 per-row partial with exactly 1 HTTP, S2 restart sends only the missing row proven by HTTP count since mem0 prompts legitimately carry old rows in rolling history, S3 lease recovery + concurrent ticks single-send, S4a/S4b unknown reservations kept), both manager review scripts re-run VERBATIM against this head (partial: completed=[2] only, resume done with 2 entries, HTTP=2, user words actually sent; extra: new-day used=0, over-cap request refused, aged reservation second_allowed=false, uv probe still rc=2), release offline check R1-R7 green, dual-target requirement checks 4/4 with handoff-HEL-885.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:01:44 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-879 rework materials: hatchling cache correction with the offline mem0ai wheel actually built from the fixed source (sha256 698284b6..., zero downloads/installs, no 3.10 mixing, recorded in release-artifacts.json), hel895 release doc §0 now separates the historical HEL-895/896 candidate from the current hel-879-assembly candidate and §4/§5 carry the v3 snapshot semantics, budget §6.2.4 amended - a sealed library is never reopened by raising total_requests (contradiction removed), runbook §3.3/3.5/3.6/4.4/4.5 corrected (bounded cleanup ledger replaces the unproven rooms-die-naturally claim; assistant_interim kind; real-but-default-off proactive seam; single-owner dual launcher), one comment-only correction in web/src/voice/session.ts removing the unproven server-auto-cleanup claim (no logic/style change; CSS and JS bundle hashes byte-stable), CG-011 + her incremental subtitles now have a real minimal backend seam (voice/seam.py + worker/llm_chat_api wiring, proactive default off) with the precise frontend interface and acceptance list in docs/rework/hel879/voice-seam-interfaces.md, real-SDK check extended to 10 items (worker callback path on real SpeechHandle/SpeechCreatedEvent; interim seam through real LLMStream) and re-run on .12 ALL PASS, handoff-HEL-879.json appended with the rework scope; regression at this head: pytest 341/341, ruff clean except 1 pre-existing error identical at 7f0a739, web typecheck 0 errors / lint 0 errors 1 pre-existing warning / vitest 88/88 / build passes (CSS index-BpZaybfD.css unchanged), voice chain 8/8, budget split 10/10, release offline R1-R7, http pagination 7/7, real-service browser 9/9 with zero non-loopback outbound, pinned-SDK 16/16+12/12+6/6+7/7+12/12, requirements checks 4/4 (origin/main and c121c42, each with/without handoff)
2026-10-27 04:27:04 +08:00