Files
CyberGril/deploy
zonggongandmultica-agent 78a5cfd8d7 HEL-879 rework: fix the four manager-review edges - (1) dual-port launcher owns the app lifecycle exactly once: uvicorn listeners run with lifespan=off, either listener ending stops the other BEFORE the shared store closes, cancellation cleans up listeners first, and create_app's lifespan gains a fail-closed double-entry guard (the old shape silently re-initialized Store/epoch/life/chat_lock on the second entry); (2) room ownership survives delete failures: end_session keeps the session in a bounded persisted cleanup ledger until the remote delete lands (retry/idempotent, LiveKit 404 = already gone = settled), expiry moves rooms into that ledger instead of silently dropping them (the manager measured zero DELETE attempts on expired ends), graceful shutdown does not hang up live calls, restart compensates only this instance's own recorded rooms bounded by attempts/TTL+grace with abandonment recorded, and the router answers 503 (not 404) while a room is still owned so the browser keeps its owed DELETE; (3) TurnLedger binds each speech handle to its own generation stream explicitly (oldest pending handle gets the stream the SDK starts; a newer speech retires an older generation-less handle) - a handle without a generation can no longer steal the next turn's stream/receipt, the worker discards interrupted/failed bindings fail-closed before consuming; (4) release snapshot v3: verify requires three-way count equality (manifest source = manifest snapshot = restored actual; rows_match is an extra constraint, never an OR bypass), the rollback drill actually restores by source path into a persistent isolated tree and opens the restored DBs there, same-basename DBs get collision-free archive names, duplicate sources are refused; new regression file tests/test_voice_rework_hel879.py 14/14 reproducing every edge first, full pytest 338/338
Co-authored-by: multica-agent <github@multica.ai>
2026-10-27 04:10:23 +08:00
..
HEL-879 rework: fix the four manager-review edges - (1) dual-port launcher owns the app lifecycle exactly once: uvicorn listeners run with lifespan=off, either listener ending stops the other BEFORE the shared store closes, cancellation cleans up listeners first, and create_app's lifespan gains a fail-closed double-entry guard (the old shape silently re-initialized Store/epoch/life/chat_lock on the second entry); (2) room ownership survives delete failures: end_session keeps the session in a bounded persisted cleanup ledger until the remote delete lands (retry/idempotent, LiveKit 404 = already gone = settled), expiry moves rooms into that ledger instead of silently dropping them (the manager measured zero DELETE attempts on expired ends), graceful shutdown does not hang up live calls, restart compensates only this instance's own recorded rooms bounded by attempts/TTL+grace with abandonment recorded, and the router answers 503 (not 404) while a room is still owned so the browser keeps its owed DELETE; (3) TurnLedger binds each speech handle to its own generation stream explicitly (oldest pending handle gets the stream the SDK starts; a newer speech retires an older generation-less handle) - a handle without a generation can no longer steal the next turn's stream/receipt, the worker discards interrupted/failed bindings fail-closed before consuming; (4) release snapshot v3: verify requires three-way count equality (manifest source = manifest snapshot = restored actual; rows_match is an extra constraint, never an OR bypass), the rollback drill actually restores by source path into a persistent isolated tree and opens the restored DBs there, same-basename DBs get collision-free archive names, duplicate sources are refused; new regression file tests/test_voice_rework_hel879.py 14/14 reproducing every edge first, full pytest 338/338
2026-10-27 04:10:23 +08:00
HEL-882 rework: execution-table caps become real hard gates at the wire, SDK hidden retries and mid-add failures recover without loss, the 8360 release package runs offline, and the HEL-874 embedding evidence is corrected - every real provider exit (create/dialogue/life/gist/judgment/growth/proactive phrasing + mem0's internal extraction LLM) now persistently pre-allocates request count (per-category daily + one-shot totals in request_usage/budget_total, BEGIN IMMEDIATE) and a conservative token estimate before sending, refuses at the tightest cap (BudgetStop, no more downgrade-and-keep-calling) and survives restarts; known usage settles to actual, unknown/disconnect/failure keeps the full reservation; ledger guards gain per-day tops (embed_daily/llm_daily) and verify_ledger_authority rejects a fresh file bypassing the frozen 498/500 prefix; the mem0 SDK client's hidden OpenAI retries are disabled (max_retries=0) so every actual send is gated and answered failures (even 5xx) settle as used with only httpx.ConnectError-grade unsent requests released; embed refusals the SDK swallows into empty adds are re-raised as budget errors so the batch re-queues without burning an attempt and resumes when the cap returns, add_group resumes at vector-store level (src_group anchors: only missing rows resent, crash/restart/concurrency never duplicate), sem coverage is per-backend (stub batches no longer block a real rebuild, real done batches never resend) with backend-scoped batch keys; the release package is offline-executable: fresh-dir git clone, cybergril service account, PYTHONPATH in the unit so the lock-only venv imports the product, shipped caps explicitly 0 (=refuse by design until approved), deploy/mem0-stack.lock + accurate 3.11 wheel gap with a no-download path (hel857-venv is 3.10; never present stub/FTS as the real memory experience), and tools/check_release_offline_hel882.py verifies checkout/import/health/fail-closed/approved-run/data-preservation on loopback with echo stand-ins; evidence corrected per manager review: docs/rework/hel874/erratum-hel882.json + README appendix record that e978e87's verification did real local embeddings from cache (provable lower bound 12, exact total unrecoverable, temp sandbox gone) while zero-real-LLM/zero-download/zero-outbound stand, old JSONs and authoritative ledgers untouched, and tools/verify_mem0_real_hel874.py now installs the deterministic 512-dim stand-in before TextEmbedding construction plus non-loopback socket rejection (zero model loads) with final_embed_rows persisted before temp cleanup; verification: 272 python tests pass on 3.11.16 venv + requirements.lock offline (16 new: manager api repro blocked with 0 provider calls, sdk repro 1-HTTP/max_retries=0/llm_ops=1, mid-add swallow requeued-then-resumed, resume-no-duplicate, stub-to-mem0 rebuild, daily embed cap, unified mem0-LLM gate, authority refusal, restart-persistent counters, tightest-constraint settle/release), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned-install deterministic re-run 12/12, manager review scripts re-run evidence attached in delivery, dual-target requirement checks 4/4 with handoff-HEL-882.json; zero real calls, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-26 23:20:18 +08:00
HEL-879 release closure: the four HEL-896 audit findings become enforced behavior - (1) archive consistency: tools/release_snapshot_hel896.py v2 checks the wal_checkpoint(TRUNCATE) return (busy!=0 = hard failure, no deliverable tar, half-built dir removed - 'stopped service first' is a precondition, never a reason to ignore an actual failed result), aborts packaging on any source-vs-snapshot table-count mismatch instead of recording rows_match=False and shipping anyway, and verify_archive now rejects legacy packages that recorded rows_match=False while mapping a real rollback drill (unpack to an isolated dir, open every DB, check rows AND hashes against both recorded source and snapshot counts, emit a source-path rollback_plan - no more tar-internal self-attestation); selftest 9/9 reproduces every old failure shape first (naive WAL copy loses companion, held-reader checkpoint=(1,9,8) leaves the copied image malformed/missing the events row, the manager's rows_match=False legacy package verifying ok) then proves each fix (busy fails with zero artifacts, stale-snapshot mismatch fails with zero artifacts, online mode refuses --file dirs containing SQLite by magic-header scan, stop mode recursively takes embedded SQLite like history_root/mem0_public_history.db through checkpoint+per-table verification, verify maps source paths and passes); (2) early-finish seal: PersistentBudget gains sealed()/seal() writing sealed/sealed_requests/sealed_tokens/sealed_at atomically into budget_total beside the real usage (never faking 55, never zeroing), acquire checks the seal FIRST so every category and every remaining allowance rejects budget_sealed:51/55, the marker survives restart and backup-API archive/restore, reseal is idempotent, and there is deliberately no unseal path (re-test = new approval + new instance per budget-package 6.2.4); tools/seal_budget_hel879.py makes the closure an explicit auditable operator command with a post-seal rejection self-check; verify_budget_split_hel896.py gains S6 (51/55 experience chain completes -> seal keeps usage 51 -> 52nd and cross-category rejected with dialogue daily still 19/60 proving it is the seal not the caps, mistaken restart rejected, archive restore rejected, marker requests=51 intact) now 10/10, HEAD made dynamic instead of hardcoded 04eff61; budget-package-hel895.md 6.2 rewritten to the executable seal order (stop -> seal -> consistency snapshot); (3) single install entry: deploy/hel895-release-candidate.md 2 is now the only copy-paste install (uv --offline, npm --offline, cache-miss = stop and record, network only inside an authorized release window with per-item sha256) and 4 is the only fixed-SDK entry as a five-step chain (verify tarball sha256 9dcef930...dda23 against mem0-stack.lock -> build the wheel from the fixed source with hatchling honestly listed as the missing build backend -> cp311 dependency prep with greenlet/pillow/truststore confirmed cache-reusable on re-run -> local-wheels-only --no-index install where mem0ai must come from the fixed-source wheel, the lock line is version-only -> same-process co-import plus the 92MB fastembed cache placement); deploy/hel874-release.md 2/4 carry superseded banners pointing here, mem0-stack.lock no longer routes installs through the index mem0ai==2.2.1 wheel; (4) exact-commit honesty: audit_sdk_offline_hel896.py derives head from git rev-parse and accepts an output dir, fresh evidence written to docs/rework/hel879/sdk-audit/ (hel896 evidence left untouched) recording closure 55 / missing 41 (28 pure + 13 native) / cache-reusable [greenlet, pillow, truststore] / tarball-matches-lock at c121c42; regression on this head: pytest 299/299, ruff clean (src/tests/tools), release-snapshot selftest 9/9, budget-split 10/10 including the new S6 negatives; zero provider calls, zero downloads, authoritative ledgers/freezes/HEL-835 gate untouched, not deployed, PR#11 untouched
2026-10-27 03:26:30 +08:00
HEL-879 release closure: the four HEL-896 audit findings become enforced behavior - (1) archive consistency: tools/release_snapshot_hel896.py v2 checks the wal_checkpoint(TRUNCATE) return (busy!=0 = hard failure, no deliverable tar, half-built dir removed - 'stopped service first' is a precondition, never a reason to ignore an actual failed result), aborts packaging on any source-vs-snapshot table-count mismatch instead of recording rows_match=False and shipping anyway, and verify_archive now rejects legacy packages that recorded rows_match=False while mapping a real rollback drill (unpack to an isolated dir, open every DB, check rows AND hashes against both recorded source and snapshot counts, emit a source-path rollback_plan - no more tar-internal self-attestation); selftest 9/9 reproduces every old failure shape first (naive WAL copy loses companion, held-reader checkpoint=(1,9,8) leaves the copied image malformed/missing the events row, the manager's rows_match=False legacy package verifying ok) then proves each fix (busy fails with zero artifacts, stale-snapshot mismatch fails with zero artifacts, online mode refuses --file dirs containing SQLite by magic-header scan, stop mode recursively takes embedded SQLite like history_root/mem0_public_history.db through checkpoint+per-table verification, verify maps source paths and passes); (2) early-finish seal: PersistentBudget gains sealed()/seal() writing sealed/sealed_requests/sealed_tokens/sealed_at atomically into budget_total beside the real usage (never faking 55, never zeroing), acquire checks the seal FIRST so every category and every remaining allowance rejects budget_sealed:51/55, the marker survives restart and backup-API archive/restore, reseal is idempotent, and there is deliberately no unseal path (re-test = new approval + new instance per budget-package 6.2.4); tools/seal_budget_hel879.py makes the closure an explicit auditable operator command with a post-seal rejection self-check; verify_budget_split_hel896.py gains S6 (51/55 experience chain completes -> seal keeps usage 51 -> 52nd and cross-category rejected with dialogue daily still 19/60 proving it is the seal not the caps, mistaken restart rejected, archive restore rejected, marker requests=51 intact) now 10/10, HEAD made dynamic instead of hardcoded 04eff61; budget-package-hel895.md 6.2 rewritten to the executable seal order (stop -> seal -> consistency snapshot); (3) single install entry: deploy/hel895-release-candidate.md 2 is now the only copy-paste install (uv --offline, npm --offline, cache-miss = stop and record, network only inside an authorized release window with per-item sha256) and 4 is the only fixed-SDK entry as a five-step chain (verify tarball sha256 9dcef930...dda23 against mem0-stack.lock -> build the wheel from the fixed source with hatchling honestly listed as the missing build backend -> cp311 dependency prep with greenlet/pillow/truststore confirmed cache-reusable on re-run -> local-wheels-only --no-index install where mem0ai must come from the fixed-source wheel, the lock line is version-only -> same-process co-import plus the 92MB fastembed cache placement); deploy/hel874-release.md 2/4 carry superseded banners pointing here, mem0-stack.lock no longer routes installs through the index mem0ai==2.2.1 wheel; (4) exact-commit honesty: audit_sdk_offline_hel896.py derives head from git rev-parse and accepts an output dir, fresh evidence written to docs/rework/hel879/sdk-audit/ (hel896 evidence left untouched) recording closure 55 / missing 41 (28 pure + 13 native) / cache-reusable [greenlet, pillow, truststore] / tarball-matches-lock at c121c42; regression on this head: pytest 299/299, ruff clean (src/tests/tools), release-snapshot selftest 9/9, budget-split 10/10 including the new S6 negatives; zero provider calls, zero downloads, authoritative ledgers/freezes/HEL-835 gate untouched, not deployed, PR#11 untouched
2026-10-27 03:26:30 +08:00
HEL-879 release closure: the four HEL-896 audit findings become enforced behavior - (1) archive consistency: tools/release_snapshot_hel896.py v2 checks the wal_checkpoint(TRUNCATE) return (busy!=0 = hard failure, no deliverable tar, half-built dir removed - 'stopped service first' is a precondition, never a reason to ignore an actual failed result), aborts packaging on any source-vs-snapshot table-count mismatch instead of recording rows_match=False and shipping anyway, and verify_archive now rejects legacy packages that recorded rows_match=False while mapping a real rollback drill (unpack to an isolated dir, open every DB, check rows AND hashes against both recorded source and snapshot counts, emit a source-path rollback_plan - no more tar-internal self-attestation); selftest 9/9 reproduces every old failure shape first (naive WAL copy loses companion, held-reader checkpoint=(1,9,8) leaves the copied image malformed/missing the events row, the manager's rows_match=False legacy package verifying ok) then proves each fix (busy fails with zero artifacts, stale-snapshot mismatch fails with zero artifacts, online mode refuses --file dirs containing SQLite by magic-header scan, stop mode recursively takes embedded SQLite like history_root/mem0_public_history.db through checkpoint+per-table verification, verify maps source paths and passes); (2) early-finish seal: PersistentBudget gains sealed()/seal() writing sealed/sealed_requests/sealed_tokens/sealed_at atomically into budget_total beside the real usage (never faking 55, never zeroing), acquire checks the seal FIRST so every category and every remaining allowance rejects budget_sealed:51/55, the marker survives restart and backup-API archive/restore, reseal is idempotent, and there is deliberately no unseal path (re-test = new approval + new instance per budget-package 6.2.4); tools/seal_budget_hel879.py makes the closure an explicit auditable operator command with a post-seal rejection self-check; verify_budget_split_hel896.py gains S6 (51/55 experience chain completes -> seal keeps usage 51 -> 52nd and cross-category rejected with dialogue daily still 19/60 proving it is the seal not the caps, mistaken restart rejected, archive restore rejected, marker requests=51 intact) now 10/10, HEAD made dynamic instead of hardcoded 04eff61; budget-package-hel895.md 6.2 rewritten to the executable seal order (stop -> seal -> consistency snapshot); (3) single install entry: deploy/hel895-release-candidate.md 2 is now the only copy-paste install (uv --offline, npm --offline, cache-miss = stop and record, network only inside an authorized release window with per-item sha256) and 4 is the only fixed-SDK entry as a five-step chain (verify tarball sha256 9dcef930...dda23 against mem0-stack.lock -> build the wheel from the fixed source with hatchling honestly listed as the missing build backend -> cp311 dependency prep with greenlet/pillow/truststore confirmed cache-reusable on re-run -> local-wheels-only --no-index install where mem0ai must come from the fixed-source wheel, the lock line is version-only -> same-process co-import plus the 92MB fastembed cache placement); deploy/hel874-release.md 2/4 carry superseded banners pointing here, mem0-stack.lock no longer routes installs through the index mem0ai==2.2.1 wheel; (4) exact-commit honesty: audit_sdk_offline_hel896.py derives head from git rev-parse and accepts an output dir, fresh evidence written to docs/rework/hel879/sdk-audit/ (hel896 evidence left untouched) recording closure 55 / missing 41 (28 pure + 13 native) / cache-reusable [greenlet, pillow, truststore] / tarball-matches-lock at c121c42; regression on this head: pytest 299/299, ruff clean (src/tests/tools), release-snapshot selftest 9/9, budget-split 10/10 including the new S6 negatives; zero provider calls, zero downloads, authoritative ledgers/freezes/HEL-835 gate untouched, not deployed, PR#11 untouched
2026-10-27 03:26:30 +08:00