Files
zonggongandmultica-agent
ceb0f35418
HEL-889 evidence: add the same-shape rerun JSON referenced by docs/rework/hel889/README.md
Co-authored-by: multica-agent <github@multica.ai>
HEL-889 rework: shutdown never closes under a live worker, and a queued-cancelled worker is released by the authoritative future state — SemanticMemory now runs batch workers on its OWN dedicated ThreadPoolExecutor and keeps each work item's concurrent.futures.Future: cancelling the awaiting tick propagates to that future, and its post-cancel state is the deterministic queued/started oracle (the manager's review_hel886_edges.py reproduced the old false-liveness on the shared default executor: register _live, await to_thread, executor busy -> cancel tick cancels the QUEUED item, _run and its finally never run, _live=1 forever with the batch pending and recovery permanently skipping a worker that does not exist, restart the only way out; now a CANCELLED future proves nothing ever ran: _release_never_started removes the registry entry idempotently and returns the batch to queued with its attempt restored and zero sends, while a running/finished future keeps HEL-886 semantics — the worker thread owns the batch fate, single send, no reclaim, no resend), and aclose becomes a real ownership protocol instead of a bounded promise it could break: a closing flag makes tick refuse new claims and makes both backends' _add_group_sync check should_stop BEFORE each row's wire work (a _ShutdownAbort at a boundary keeps finished rows' sem_row_done proofs and requeues only the unsent remainder — never a mid-row cut, and a single-row batch still completes done); never-started workers are cancelled out of the executor queue under the same authoritative CANCELLED proof; started workers finish their current row and flush, and aclose WAITS for every live worker to actually exit before backend/executor close — drain_timeout only bounds the polite phase (after expiry: honest warning, keep waiting; a live worker is never treated as dead), bounded in practice because Mem0Backend now sets the gateway OpenAI client's timeout (gateway_timeout config, default 60s) instead of the 600s default that made the drain effectively open-ended; a cancelled aclose closes NOTHING (shielded finish, closing signal reset, layer stays usable), concurrent aclose calls serialize on an asyncio.Lock and are idempotent, and the app lifespan comment now states the ownership contract — the manager's fixed-SDK reproduction of the OLD timeout branch (loopback gateway accepts the first HTTP then withholds the answer, aclose(drain_timeout=0.05) returns with the worker alive, app closes the store, late worker dies on 'QdrantLocal instance is closed'/'Cannot operate on a closed database', reopen shows pending/row_done=0/llm_used=1) is closed: aclose does not return while the worker is alive, the worker's writes land BEFORE the app-order store.close(), and the reopened store shows done + row_done=1 + full entry_ids with llm_used=1 and zero late errors; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); verification: 294 python tests pass on 3.11.16 venv + requirements.lock offline (7 new tests/test_rework_hel889.py cases, all barrier-driven real product paths: queued-cancel release + in-process recovery via injected single-worker executor / shutdown-timeout never closes under the live worker and flushes before app-order store close / between-rows abort requeues and the next process sends ONLY the missing row / shutdown-before-start zero sends then resumes / cancelled aclose leaves everything open / concurrent aclose serializes / closed layer refuses new claims), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_shutdown_queued_hel889.py 12/12 (Q1 queued-cancel phantom released and recovered in-process with the manager's 41s lease probe, S1 fixed-SDK shutdown-timeout branch, S2 between-rows abort + missing-row-only resume proven by gateway POST counts, S3 cancelled aclose, S4 concurrent aclose), the manager's review_hel886_edges.py re-run VERBATIM on the pristine
f1b50ad first reproduced both defects exactly (docs/rework/hel889/edges-verbatim-before-fix-f1b50ad.json matches the manager's independent JSON), the same-shape rerun review_hel886_edges_rerun_hel889.py now passes both scenarios with every original result key (queued: done/attempts=1/registered 0/entries 1/row_done 1; timeout: aclose_returned_while_worker_alive=false, late_worker_error null, persisted done+entry_ids+row_done=1, llm_used=1) while the verbatim original stops at its own premise assert executor._work_queue.qsize()==1 — batch workers no longer ride the loop default executor, the encoded defect is structurally gone (original negative and baseline kept); regressions all green: verify_inflight_lifecycle_hel886.py 6/6 (single-send preserved), verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass, review_hel885_inflight.py still fails at its 2-add barrier exactly as documented in HEL-886; runbook §4 gains the HEL-889 ownership/queued-cancel semantics, config/semantic.example.yaml documents gateway_timeout, docs/requirements/handoff-HEL-889.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
HEL-889 rework: shutdown never closes under a live worker, and a queued-cancelled worker is released by the authoritative future state — SemanticMemory now runs batch workers on its OWN dedicated ThreadPoolExecutor and keeps each work item's concurrent.futures.Future: cancelling the awaiting tick propagates to that future, and its post-cancel state is the deterministic queued/started oracle (the manager's review_hel886_edges.py reproduced the old false-liveness on the shared default executor: register _live, await to_thread, executor busy -> cancel tick cancels the QUEUED item, _run and its finally never run, _live=1 forever with the batch pending and recovery permanently skipping a worker that does not exist, restart the only way out; now a CANCELLED future proves nothing ever ran: _release_never_started removes the registry entry idempotently and returns the batch to queued with its attempt restored and zero sends, while a running/finished future keeps HEL-886 semantics — the worker thread owns the batch fate, single send, no reclaim, no resend), and aclose becomes a real ownership protocol instead of a bounded promise it could break: a closing flag makes tick refuse new claims and makes both backends' _add_group_sync check should_stop BEFORE each row's wire work (a _ShutdownAbort at a boundary keeps finished rows' sem_row_done proofs and requeues only the unsent remainder — never a mid-row cut, and a single-row batch still completes done); never-started workers are cancelled out of the executor queue under the same authoritative CANCELLED proof; started workers finish their current row and flush, and aclose WAITS for every live worker to actually exit before backend/executor close — drain_timeout only bounds the polite phase (after expiry: honest warning, keep waiting; a live worker is never treated as dead), bounded in practice because Mem0Backend now sets the gateway OpenAI client's timeout (gateway_timeout config, default 60s) instead of the 600s default that made the drain effectively open-ended; a cancelled aclose closes NOTHING (shielded finish, closing signal reset, layer stays usable), concurrent aclose calls serialize on an asyncio.Lock and are idempotent, and the app lifespan comment now states the ownership contract — the manager's fixed-SDK reproduction of the OLD timeout branch (loopback gateway accepts the first HTTP then withholds the answer, aclose(drain_timeout=0.05) returns with the worker alive, app closes the store, late worker dies on 'QdrantLocal instance is closed'/'Cannot operate on a closed database', reopen shows pending/row_done=0/llm_used=1) is closed: aclose does not return while the worker is alive, the worker's writes land BEFORE the app-order store.close(), and the reopened store shows done + row_done=1 + full entry_ids with llm_used=1 and zero late errors; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); verification: 294 python tests pass on 3.11.16 venv + requirements.lock offline (7 new tests/test_rework_hel889.py cases, all barrier-driven real product paths: queued-cancel release + in-process recovery via injected single-worker executor / shutdown-timeout never closes under the live worker and flushes before app-order store close / between-rows abort requeues and the next process sends ONLY the missing row / shutdown-before-start zero sends then resumes / cancelled aclose leaves everything open / concurrent aclose serializes / closed layer refuses new claims), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_shutdown_queued_hel889.py 12/12 (Q1 queued-cancel phantom released and recovered in-process with the manager's 41s lease probe, S1 fixed-SDK shutdown-timeout branch, S2 between-rows abort + missing-row-only resume proven by gateway POST counts, S3 cancelled aclose, S4 concurrent aclose), the manager's review_hel886_edges.py re-run VERBATIM on the pristine
f1b50ad first reproduced both defects exactly (docs/rework/hel889/edges-verbatim-before-fix-f1b50ad.json matches the manager's independent JSON), the same-shape rerun review_hel886_edges_rerun_hel889.py now passes both scenarios with every original result key (queued: done/attempts=1/registered 0/entries 1/row_done 1; timeout: aclose_returned_while_worker_alive=false, late_worker_error null, persisted done+entry_ids+row_done=1, llm_used=1) while the verbatim original stops at its own premise assert executor._work_queue.qsize()==1 — batch workers no longer ride the loop default executor, the encoded defect is structurally gone (original negative and baseline kept); regressions all green: verify_inflight_lifecycle_hel886.py 6/6 (single-send preserved), verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass, review_hel885_inflight.py still fails at its 2-add barrier exactly as documented in HEL-886; runbook §4 gains the HEL-889 ownership/queued-cancel semantics, config/semantic.example.yaml documents gateway_timeout, docs/requirements/handoff-HEL-889.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes