Files
CyberGril/docs/requirements/handoff-HEL-889.json
T
zonggongandmultica-agent 440d95cad8 HEL-889 rework: shutdown never closes under a live worker, and a queued-cancelled worker is released by the authoritative future state — SemanticMemory now runs batch workers on its OWN dedicated ThreadPoolExecutor and keeps each work item's concurrent.futures.Future: cancelling the awaiting tick propagates to that future, and its post-cancel state is the deterministic queued/started oracle (the manager's review_hel886_edges.py reproduced the old false-liveness on the shared default executor: register _live, await to_thread, executor busy -> cancel tick cancels the QUEUED item, _run and its finally never run, _live=1 forever with the batch pending and recovery permanently skipping a worker that does not exist, restart the only way out; now a CANCELLED future proves nothing ever ran: _release_never_started removes the registry entry idempotently and returns the batch to queued with its attempt restored and zero sends, while a running/finished future keeps HEL-886 semantics — the worker thread owns the batch fate, single send, no reclaim, no resend), and aclose becomes a real ownership protocol instead of a bounded promise it could break: a closing flag makes tick refuse new claims and makes both backends' _add_group_sync check should_stop BEFORE each row's wire work (a _ShutdownAbort at a boundary keeps finished rows' sem_row_done proofs and requeues only the unsent remainder — never a mid-row cut, and a single-row batch still completes done); never-started workers are cancelled out of the executor queue under the same authoritative CANCELLED proof; started workers finish their current row and flush, and aclose WAITS for every live worker to actually exit before backend/executor close — drain_timeout only bounds the polite phase (after expiry: honest warning, keep waiting; a live worker is never treated as dead), bounded in practice because Mem0Backend now sets the gateway OpenAI client's timeout (gateway_timeout config, default 60s) instead of the 600s default that made the drain effectively open-ended; a cancelled aclose closes NOTHING (shielded finish, closing signal reset, layer stays usable), concurrent aclose calls serialize on an asyncio.Lock and are idempotent, and the app lifespan comment now states the ownership contract — the manager's fixed-SDK reproduction of the OLD timeout branch (loopback gateway accepts the first HTTP then withholds the answer, aclose(drain_timeout=0.05) returns with the worker alive, app closes the store, late worker dies on 'QdrantLocal instance is closed'/'Cannot operate on a closed database', reopen shows pending/row_done=0/llm_used=1) is closed: aclose does not return while the worker is alive, the worker's writes land BEFORE the app-order store.close(), and the reopened store shows done + row_done=1 + full entry_ids with llm_used=1 and zero late errors; crash semantics unchanged and still honest (unknown wire outcomes billed, proof-gated retries, strict-once not claimed); verification: 294 python tests pass on 3.11.16 venv + requirements.lock offline (7 new tests/test_rework_hel889.py cases, all barrier-driven real product paths: queued-cancel release + in-process recovery via injected single-worker executor / shutdown-timeout never closes under the live worker and flushes before app-order store close / between-rows abort requeues and the next process sends ONLY the missing row / shutdown-before-start zero sends then resumes / cancelled aclose leaves everything open / concurrent aclose serializes / closed layer refuses new claims), ruff clean (src/tests/tools), no frontend changes (frontend checks not re-run, stated honestly); pinned mem0 2.2.1 on the deterministic boundary (512-dim stand-in before TextEmbedding construction, loopback-only): new tools/verify_shutdown_queued_hel889.py 12/12 (Q1 queued-cancel phantom released and recovered in-process with the manager's 41s lease probe, S1 fixed-SDK shutdown-timeout branch, S2 between-rows abort + missing-row-only resume proven by gateway POST counts, S3 cancelled aclose, S4 concurrent aclose), the manager's review_hel886_edges.py re-run VERBATIM on the pristine f1b50ad first reproduced both defects exactly (docs/rework/hel889/edges-verbatim-before-fix-f1b50ad.json matches the manager's independent JSON), the same-shape rerun review_hel886_edges_rerun_hel889.py now passes both scenarios with every original result key (queued: done/attempts=1/registered 0/entries 1/row_done 1; timeout: aclose_returned_while_worker_alive=false, late_worker_error null, persisted done+entry_ids+row_done=1, llm_used=1) while the verbatim original stops at its own premise assert executor._work_queue.qsize()==1 — batch workers no longer ride the loop default executor, the encoded defect is structurally gone (original negative and baseline kept); regressions all green: verify_inflight_lifecycle_hel886.py 6/6 (single-send preserved), verify_semantic_recovery_hel885.py 7/7, verify_mem0_real_hel874.py 12/12, check_release_offline_hel882.py R1-R7 15/15, both manager review_hel882_{partial,extra}.py scripts re-run verbatim pass, review_hel885_inflight.py still fails at its 2-add barrier exactly as documented in HEL-886; runbook §4 gains the HEL-889 ownership/queued-cancel semantics, config/semantic.example.yaml documents gateway_timeout, docs/requirements/handoff-HEL-889.json written, dual-target requirement checks 4/4 (origin/main and origin/agent/agent/de8874a3803b, with and without handoff); zero real LLM, zero real embeddings, zero non-loopback outbound, authoritative ledgers and freezes untouched, not deployed, PR not merged, no visual changes
Co-authored-by: multica-agent <github@multica.ai>
2026-10-27 01:24:11 +08:00

54 lines
9.6 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"baseline_version": "1.11",
"read_commit": "f1b50ad91ad8c856f28383ee59eddb6547d9c586",
"task": "HEL-889 返工(总管对 HEL-886 候选 f1b50ad 独立复审确认的两个生命周期边界缺口一次修完):关闭超时不得把仍可能写入的活 worker 当死而关掉其依赖(总管实际固定 SDK 复现:回环网关先接受第一 HTTP 未答,aclose(drain_timeout=0.05) 返回时 worker 仍活,按 app 顺序关 Store 后放行,Qdrant 报 QdrantLocal instance is closed、worker 报 ProgrammingError: Cannot operate on a closed database,重开批次仍 pending/row_done=0/llm_used=1——仍活进程主动关资源造成的丢落盘窗口,不能归入真崩溃结果未知)、执行器排队阶段取消不得永久假活跃(先登记 _live 再 await to_thread,执行器忙时取消 tick 会取消排队 future,_run/finally 从未运行,注册表留下不存在的 worker,批次 pending 卡死且之后恢复一直跳过,只能重启;单 worker 执行器+41 秒时钟前进+recover+tick 复现 _live=1/entries=0/row_done=0/队列空);零真实调用、零真实嵌入、零下载、不部署、不合并、不改视觉、不碰权威三账、不新增任何 cap、不改运行时/模型/认证",
"requirement_ids": [
"CG-006",
"CG-008",
"CG-011",
"CG-013",
"CG-014",
"CG-015",
"CG-016",
"CG-018",
"CG-021",
"AM-001",
"AM-002",
"AM-003",
"UX-004",
"UX-006",
"GOV-001",
"GOV-002",
"GOV-003",
"GOV-004",
"GOV-005",
"GOV-006",
"DEL-001"
],
"deliverable": "在统一候选母线 hel-870-assembly(基于 f1b50ad 追加,PR#11 只追加不强推不合并)完成两项修正:1) 关闭所有权协议——批 worker 移入 SemanticMemory 专用执行器(本层持有 concurrent.futures.Future,即权威的排队/已启动判定,且长批次不再挤占 loop 共享默认执行器);aclose 先置 closing 信号:tick 拒新领、后端 _add_group_sync 在每行发送前查 should_stop 只在行边界中止(已完行保留 sem_row_done 证明,未发行回 queued,绝无在飞中断)、未启动 worker 直接按 CANCELLED 权威取消(从未运行=零出站,安全回队并复原 attempts);随后等待所有活跃 worker 真实退出才关 backend 与执行器——drain_timeout 只界定礼貌阶段(超时如实告警并继续等,绝不把活 worker 当死),Mem0Backend 构造时给网关 OpenAI 客户端设 gateway_timeout(默认 60s,可配)保证排空有界;aclose 被取消时什么都不关且 closing 复位可续用,并发 aclose 用 asyncio.Lock 串行化且幂等,完成端关闭 asyncio.shield 保护;真崩溃语义不变(保守留账、只发无证明行、不宣称严格一次)。2) 排队取消恢复——_LiveWorker 增加 started 事件与 future 槽位区分排队/已启动/已完成;取消等待协程时以 future.cancelled() 权威判定:从未运行则立即释放假活跃注册项并回队(attempts 复原、零发送),释放执行器、时钟前进 41 秒后 recover+tick 进程内完成,无需重启;已开始/已完成则维持 HEL-886 语义(worker 线程写终态、并发 tick 不重发,不回到双发)。保留全部已通过行为(回读、隐私、单行恢复、预算硬闸、未知预占不退款、全完成零预算收尾)。app lifespan 注释同步为 HEL-889 所有权口径。验证:python 3.11.16 venv + requirements.lock 离线 294 pytest 全过(新增 tests/test_rework_hel889.py 7 例,全部屏障驱动真实产品路径:排队取消恢复/关闭超时不关活 worker 且落盘先于关库/行间中止下一进程只发缺行/未启动即关闭零发送并可续跑/关闭被取消什么都不关/并发关闭串行化/关闭后拒新领),ruff 0 错(src/tests/tools);固定 mem0 2.2.1 确定性替身实跑 tools/verify_shutdown_queued_hel889.py 12/12(Q1 排队取消、S1 关闭超时分支固定 SDK 实跑、S2 行间中止、S3 关闭被取消、S4 并发关闭);总管 review_hel886_edges.py 原负例与基线保留(edges-verbatim-before-fix-f1b50ad.json 与总管独立结果一致),同形复跑 review_hel886_edges_rerun_hel889.py 全过(queued:done/attempts=1/registered 0/entries 1/row_done 1;timeout:aclose_returned_while_worker_alive=false/late_worker_error=null/persisted done+entry_ids+row_done=1/llm_used=1),原样复跑如实停在其自身前提断言(批 worker 不再走 loop 默认执行器,docs/rework/hel889/original-verbatim-after-fix.log);回归 verify_inflight_lifecycle_hel886.py 6/6、verify_semantic_recovery_hel885.py 7/7、verify_mem0_real_hel874.py 12/12、check_release_offline_hel882.py R1–R7 15/15、总管 review_hel882_{partial,extra}.py 原样复跑通过、review_hel885_inflight.py 仍在 2-add 屏障如预期失败(单发保持);零真实调用、零真实嵌入、零非回环出站,权威三账与冻结余额未写,未部署未合并未交体验地址,不改视觉",
"acceptance": [
"P1 修复(固定 SDK 实跑关闭超时分支):第一 HTTP 已接受未答时 aclose(drain_timeout=0.05) 在 worker 仍活时绝不返回;应答放行后 worker 完成落盘,aclose 才返回、才关 client;按 app 顺序关库重开:批 done、row_done=1、entry_ids 完整、llm_used=1、无任何晚到错误(docs/rework/hel889/edges-real-sdk-hel889.json S1 + HEL-889-edges-rerun-fixed.json shutdown_timeout;修复前基线 aclose_returned_while_worker_alive=true/ProgrammingError/pending/row_done=0 保留对照)",
"P1 关闭取消/并发:排空中的 aclose 被取消什么都不关(client 可用、批次仍 pending、closing 复位可续用),worker 正常完成后再次 aclose 干净收尾;两个 aclose 并发串行化、排空一次、资源只关一次、第三次调用幂等即返(S3/S4 + tests/test_rework_hel889.py)",
"P2 修复(排队取消):单 worker 执行器占住、批次仅入队未启动即取消 tick——假活跃注册项立即释放、批次回 queued 且 attempts 复原、零出站;释放执行器、推进 41 秒后 recover+tick 进程内完成:done、1 条目、1 sem_row_done、注册表清零,无需重启(Q1 + 同形复跑 queued_cancel;修复前基线 pending/_live=1/entries=0/row_done=0 保留对照)",
"已开始取消仍单发:处理中取消/跨租约仍活跃不重发不回收(HEL-886 L1/L2 回归 6/6),review_hel885_inflight.py 仍在 2-add 屏障如预期失败,未回退双发",
"行间中止与恢复:两行批次关闭中止保留第 1 行完成证明、批次回 queued/shutdown_between_rows、关闭期间网关 POST 不新增;下一进程同账同库只发缺行(POST 仅 +1)后 done 且两行 entry_ids 完整(S2 + pytest)",
"原有能力全部保留:回读/隐私/单行恢复/预算硬闸/未知预占不退款/全完成零预算收尾/真死租约回收(recovery-hel885-rerun 7/7、inflight-hel886-rerun 6/6、contract-hel874-rerun 12/12、release-offline R1–R7 15/15、总管 882 两脚本原样复跑通过)",
"工程检查:294 pytest(含 7 新例)全过、ruff 0 错(src/tests/tools);本卡不改前端,未重跑前端 42 项/构建,不以历史数字冒充本轮实测;双目标需求检查 4/4(含本 handoff)"
],
"deferred": [
"发布与体验地址:由总管独立复审本 head 后交发布卡按 deploy/hel874-release.md 执行(8360 隔离实例);本卡未部署未合并 PR#11 未交体验地址",
"真实语义激活(CYBERGRIL_SEMANTIC=mem0):需 B 部分预算批准 + 3.11 mem0 栈 wheel 落位;未激活前 stub/FTS 不冒充真实记忆体验",
"A/B/C 执行表数值仍待批建议:批准前所有 cap 保持 0=拒绝;新增预算由总管统一处理,不作为本卡阻塞",
"真死进程『严格一次』不在本卡宣称范围:远端无幂等支持时崩溃结果未知,恢复保守留账、只发无证明行(runbook §4 HEL-889 段已如实写明)",
"R5 语音 HEL-878/879、R6 天气/素材/完整备份、R7 有条件观察承接不变;视觉仍交原作者,本卡未改视觉;真实体验效果待体验地址交付后由白栖知判断"
],
"requirement_changes": [],
"verification": [
"分支 hel-870-assembly 追加提交(最终 head 见交付评论),PR#11 只追加未强推未合并;视觉总监分支未覆盖",
"python 3.11.16 venv + requirements.lock 离线:294 项测试全过(新增 tests/test_rework_hel889.py 7 例,全部屏障驱动真实产品路径,无预置死租约冒充);ruff 0 错(src/tests/tools);本卡未改前端故未重跑前端检查,如实注明",
"固定 mem0 2.2.1 安装(hel857-venv 3.10.12)确定性替身实跑(docs/rework/hel889/):tools/verify_shutdown_queued_hel889.py 12/12(edges-real-sdk-hel889.json);总管脚本基线原样复跑 edges-verbatim-before-fix-f1b50ad.json 与总管独立结果一致;同形复跑 review_hel886_edges_rerun_hel889.py 全过(HEL-889-edges-rerun-fixed.json);原样复跑停在其自身前提断言(original-verbatim-after-fix.log,原负例与基线不删)",
"回归:verify_inflight_lifecycle_hel886.py 6/6、verify_semantic_recovery_hel885.py 7/7、verify_mem0_real_hel874.py 12/12、check_release_offline_hel882.py R1–R7 15/15、总管 review_hel882_{partial,extra}.py 原样复跑通过;review_hel885_inflight.py 原样复跑仍在 2-add 屏障如预期失败(单发保持,与 HEL-886 存档口径一致)",
"python3 scripts/check_requirements.py --base-ref origin/main [--handoff docs/requirements/handoff-HEL-889.json] 与 --base-ref origin/agent/agent/de8874a3803b [--handoff …] 四项通过"
]
}