Files
CyberGril/docs/rework/hel894
zonggongandmultica-agent 152479b350 HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
Co-authored-by: multica-agent <github@multica.ai>
2026-10-27 01:43:54 +08:00
..
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00
HEL-894 rework: a cancelled close owns nothing it did not start — aclose cancellation is now a two-phase ownership protocol - the manager's independent pinned-SDK review of ceb0f35 reproduced two combo holes in the HEL-889 close path: (1) cancelling aclose AFTER backend.aclose began made the shielded finish keep running unowned in the background while the outer except already released the lock and cleared closing — the layer was reported reusable (closing=false/closed=false/client usable), a new tick claimed a batch onto real HTTP, then the orphaned finish closed Qdrant under it (SDK-only branch: batch failed on 'QdrantLocal instance is closed' with row_done=0/llm_used=1; with the caller closing the store afterwards the late worker died on ProgrammingError 'Cannot operate on a closed database' and the reopened batch sat pending), and (2) cancelling a SECOND aclose that was still WAITING for the close lock ran the same except and cleared the FIRST closer's flag (first close still pending but signal=false, a new batch ran to done while the original worker was still on the wire). SemanticMemory.aclose now states the revocation boundary explicitly: the DRAIN phase (nothing closed yet) stays revocable — a cancelled caller leaves everything open, resets the closing signal and the layer remains usable (HEL-889 semantics preserved); the FINISH phase (backend.aclose begun) is irreversible — a cancelled caller keeps the closing markers (a half-closed layer is never reported usable, ticks keep refusing), keeps the lock, and waits — repeated cancels only re-shield — for the layer-owned close task it started (self._close_task, created exactly once with asyncio.create_task and stored on the layer, never dropped to the background) to land before propagating CancelledError, so resources close exactly once; _finish_close also joins this layer's OWN executor via asyncio.to_thread so a cancelled caller can never block the event loop on a synchronous shutdown wait (caller-injected executors stay caller-owned), and signal reset is ownership-gated by owns_close: only the call that actually acquired the lock and set the signal may clear it, and only while no close task exists — a lock waiter cancelled on acquisition touches nothing; module docstring, aclose docstring and the app lifespan comment state the same contract; verification: 299 python tests pass on 3.11.16 venv + requirements.lock offline (5 new tests/test_rework_hel894.py cases, all barrier-driven real product paths: finish-phase cancel keeps ownership and refuses new work with zero sends and zero budget / repeated cancels during finish land the close exactly once / a same-phase second aclose waits then returns idempotently while a third cancelled lock-waiter touches nothing / waiter-cancel + owner draining + refused new tick + original worker completing combo / drain-phase cancel still re-arms for reuse), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_close_ownership_hel894.py 16/16 reruns the manager's two negatives in the same shape with assertions restated for the new protocol (F1 SDK-only: after the cancel closing=true/closed=false/client usable and the cancelled caller does not return before its close lands; the new tick is refused with zero gateway arrivals, batch queued/owner NULL/attempts 0 and llm_used=0; after the gate the close lands exactly once with live_workers=0 and the reopened batch is intact queued/row_done=0/llm_used=0 — before the fix the same run gave closing=false, a claimed batch, QdrantLocal instance is closed, failed/pending with row_done=0/llm_used=1; F2 + store-close branch: same shape and no late error — before: ProgrammingError; W waiter cancel: closing stays set, the new batch stays queued, the original worker completes done/row_done=1 after release and the owner's close then lands once); the manager's review_hel889_cancel.py re-run VERBATIM on the pristine ceb0f35 first reproduced both defects exactly (docs/rework/hel894/before-ceb0f35.json matches the manager's independent JSON), and on the fixed tree stops honestly at its own premise assert server.arrived — 'a finish-phase-cancelled layer still starts new real HTTP' is structurally gone because tick refuses while closing is held (verbatim-after-fix.log; original negative and baseline kept); regressions all green: verify_shutdown_queued_hel889.py 12/12 (queued cancel, timeout drain, between-rows resume, drain-cancel re-arm, concurrent close all preserved), verify_inflight_lifecycle_hel886.py 6/6, verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass (extra's uv rc=2 is its own expected negative), HEL-889 same-shape rerun review_hel886_edges_rerun_hel889.py passes; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); runbook §4 gains the HEL-894 cancellation-ownership paragraph, docs/requirements/handoff-HEL-894.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 01:43:54 +08:00

HEL-894 返工证据 · 关闭被取消时的资源所有权

总管对 HEL-889 head ceb0f35418f0d0b28b4e2a359615194db8eb4567 独立复审 (review_hel889_cancel.py,附件与本目录 before-ceb0f35.*)确认的两个组合 漏洞,在本 head 修复并复验。零真实出站、零真实嵌入、零下载;全部回环 + 确定性 512 维替身(TextEmbedding 构造前安装);未写权威三账、未改冻结余额 与真实 cap、未部署、未合并、未改视觉。

修复前基线(本卡在未改动的 ceb0f35 上原样复跑总管脚本)

before-ceb0f35.json 与总管独立结果一致——

  • finish_cancel(SDK-only 与 +Store 关闭两分支):取消返回时 closing=false/closed=false/client 可用(宣称可续用)→ 新批被领、HTTP 已 接受未答 → 后台旧 _finish 关 Qdrant(closed=true、live_workers=1)→ 不关 Store 也 failed/Failed to insert any of the 1 extracted memories into the vector store/row_done=0/llm_used=1(QdrantLocal instance is closed);+Store 分支晚到 worker 另报 ProgrammingError: Cannot operate on a closed database.,重开 pending/row_done=0/llm_used=1。
  • waiter_cancel:第一关闭排空中、第二等锁即取消后 closing 被清 (signal=false),原 worker 未放行时新批已 done。

修复(src/cybergril/memory/semantic.py)

两阶段取消语义(HEL-894):

  1. 排空阶段(可撤销):尚未关任何资源。此阶段取消维持 HEL-889 语义—— 什么都不关、closing 复位、层次可续用,之后 aclose 再正常收尾。
  2. 提交阶段(不可撤销):backend.aclose() 一旦开始,资源只能关完。 被取消的调用方不清 closing/不放行 tick(半关闭状态绝不宣称可用)、不释放 锁,并等待(重复取消也只重新 shield)自己启动的层持有关闭任务 (_close_task,创建恰一次、从不丢到后台)落成后才传播 CancelledError; 资源恰关一次。本层自有执行器的 shutdown 也走 to_thread,取消方不可能 用同步 join 阻塞事件循环(调用方注入执行器仍由调用方持有,与总管脚本 口径一致)。
  3. 所有权最小化:只有真正取得锁并置位信号的调用(owns_close)才可在 仍可撤销时复位信号;等锁者被取消不改变持锁关闭者的任何状态(修复负例 二)。app lifespan 注释同步为该口径。

修复后验证

  • close-ownership-real-sdk.json(tools/verify_close_ownership_hel894.py, 固定 mem0 2.2.1 + 真实 Qdrant/OpenAI client 实跑,总管两负例同形、断言按 新协议改写并逐条注明)16/16:F1 提交阶段取消(SDK-only)——取消后 closing=true/closed=false/client 可用(不再宣称可续用),取消方未返回先 等自己启动的关闭落成;新批 tick 被拒(网关零到达、批次 queued/owner NULL/attempts 0、llm_used=0);放行后关闭恰一次(closed=true、client 不可 用、live_workers=0);重开批次 queued 完好、row_done=0、llm_used=0(修复 前为 failed/pending + llm_used=1)。F2(+调用方随后关 Store)同形且无任何 晚到错误(修复前 ProgrammingError)。W 等锁者取消——closing 保持置位、新 批 queued 不被领、原 worker 放行后 done/row_done=1、第一关闭一次收尾。
  • verbatim-after-fix.log:总管原脚本原样复跑在修复后树上如实停在其自 身前提断言 assert await asyncio.to_thread(server.arrived.wait, 5)—— 「取消后新 tick 发出真实 HTTP」这一前提被新协议结构性移除(tick 在 closing 持有期间拒新领);原负例与基线保留不删。
  • 回归(固定安装实跑):shutdown-queued-hel889-rerun.json 12/12(原排队 取消/超时排空/行间续跑/排空取消可续用/并发关闭全部保持)、 inflight-hel886-rerun.json 6/6(处理中取消单发/全完成零预算收尾保持)、 recovery-hel885-rerun.json 7/7、contract-hel874-rerun.json 12/12、 release-offline-hel894.json R1–R7 15/15、总管 review_hel882_{partial,extra}.py 原样复跑通过 (manager-review-*-rerun.json;extra 内 uv rc=2 为其自身预期负例)、 edges-rerun-hel889.json 同形复跑通过。review_hel885_inflight.py 本轮 未重跑,其「仍在 2-add 屏障如预期失败」的存档口径(HEL-886/889)不变。
  • 产品测试:python 3.11.16 venv + requirements.lock 离线 299 pytest 全过 (新增 tests/test_rework_hel894.py 5 例,全部屏障驱动真实产品路径:提交 阶段取消保持所有权并拒新工作/重复取消仍一次落成/同阶段第二调用等待后幂 等且等锁者取消不动状态/等锁者取消不改持锁关闭者+tick+原 worker 完成组合/ 排空阶段取消仍可续用)、ruff 0 错(src/tests/tools)。本卡未改前端,未重 跑前端检查,不冒充历史数字。