
zonggongandmultica-agent
f1b50ad91a
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
Co-authored-by: multica-agent <github@multica.ai>
2026-10-27 00:30:47 +08:00
..
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00
HEL-886 rework: a live worker is never treated as dead, and a fully-completed batch finalizes without new budget - SemanticMemory gains an in-process live-worker registry (batch_id -> _LiveWorker, registered BEFORE the thread starts, so the batch is provably active from the moment the claim commits) and the batch's whole execution (row building, anchor verify, resume set, real adds, terminal sem_batch write) moves INTO the worker thread function, because asyncio.to_thread cannot interrupt a running call: cancelling the tick cancels only the waiter, and the old finally that marked the still-running batch failed and cleared its owner was the exact double-send the manager reproduced live with the pinned SDK (first HTTP accepted but unanswered -> cancel tick -> failed written while the add was on the wire -> second tick re-claimed and sent the same sentence again: 2 HTTP, 2 vectors, 1 row_done entry, the late worker's result dropped); recover() now fails only expired pending batches WITHOUT a live worker, so a lease past 40s with the thread still inside its add is never reclaimed and concurrent ticks cannot re-send an active row (same 1-HTTP/1-vector/1-record outcome on the manager's clock-injection lease variant), while a genuinely dead owner (crash/restart, no registered worker) keeps the honest conservative recovery: unknown wire outcome, ledger holds stay billed (HEL-885), the retry sends only rows without a completion proof, and strict-once is NOT claimed for arbitrary crashes without remote idempotency (module docstring + runbook say so in words); new SemanticMemory.aclose() drains live workers (bounded wait, honest warning on timeout) BEFORE backend/store close so shutdown never orphans durable writes, and the app lifespan now drains through it; Mem0Backend/StubBackend extract the sync core _add_group_sync (async add_group contract unchanged) where the completion set (vector-store anchors + sem_row_done + skip) is proven BEFORE any budget gate: a batch whose every row is already complete returns done with ZERO new HTTP/embed/LLM after restart (the manager's third example: llm_cap=1 fully spent, row really completed, 'row persisted, final status did not' crash window -> was queued/semantic_llm_budget_exhausted, now done with entry_ids intact, cap unchanged, nothing refunded), while any missing row keeps the hard fail-closed gate - finalization never buys headroom; verification: 287 python tests pass on 3.11.16 venv + requirements.lock offline (6 new tests/test_rework_hel886.py cases: in-flight cancel never double-sends / expired lease with live worker never reclaims + concurrent ticks single-send / dead-owner lease still recovers / shutdown drains the live worker before close proven by post-aclose durable writes / all-rows-done restart finalizes with zero new HTTP / partially-done keeps refusing), ruff clean (src/tests/tools), web tsc + 42 vitest + build via npm ci --offline, pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in installed before TextEmbedding construction, loopback-only): the manager's review_hel885_inflight.py re-run VERBATIM now fails exactly at its worker_done 2-add barrier (the assertion that a second send must happen; before the fix the same script passed with 2 HTTP/2 vectors - baseline JSON kept), the same-shape rerun with the barrier at ONE completed add passes both scenarios with 1 HTTP/1 vector/1 sem_row_done, batch pending while in flight, done with full entry_ids, and the all-done restart finalizes done with zero new HTTP and llm_used still 1; new tools/verify_inflight_lifecycle_hel886.py 6/6 (L1 cancel single-send full record, L2 lease-expired-live no reclaim, L3 shutdown drain before close, L4a all-done finalize without budget, L4b partial keeps the hard gate, L5 dead-owner recovery still works); regressions all green: verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass; dual-target requirement checks 4/4 with handoff-HEL-886.json; zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
2026-10-27 00:30:47 +08:00