HEL-835: evidence corrections per third review - 23 calls, day2 conclusion downgraded, old probe downgraded, null-fingerprint withdrawn, probe start deviation disclosed; 15/15 paired-window correction proof; real follow-up plan
Co-authored-by: multica-agent <github@multica.ai>
This commit is contained in:
zonggong
co-authored by
multica-agent
multica-agent
parent
c5167ca7c5
commit
af28ffdcb5
@@ -0,0 +1,25 @@
|
||||
# HEL-835 后续真实补验方案（仅提议，未授权不执行）
|
||||
|
||||
## 现状（截至 2026-10-25 审计截点）
|
||||
|
||||
- 8350 真实档案：**4 个成功完整轮 + 1 取消轮**（取消与 probe 库合成轮均不算自然成功轮）；权威账 23 次调用（主聊 6/6、gist 8/8、人格 1/1 全满）。
|
||||
- 已证：跨日提问链路与近期上下文装配正确（昨日独有细节完整送入提示，evidence/11）；合成库 21 对补证：生产近20轮排除 + 窗外目标检索注入 + 底色不变（correction-proof 15/15，ControlledLLM 零真实调用）。
|
||||
- 未证：**自然积累的真实长期记忆隔天取回**（记忆体验层面隔天回答未引用昨日独有细节，验收未通过/未证）。
|
||||
|
||||
## 提议的真实补验（需白栖知批准新增额度）
|
||||
|
||||
**目标**：8350 真实档案自然累计超过 20 个成功轮后，跨真实自然日问"前一天聊过的具体旧事实"，验收回复**引用当日独有细节**（非提问中已有的词），且检索日志 hits 含窗口外记忆。
|
||||
|
||||
**用户实际做什么**：与 8350 的小满正常聊天（主题自然多样，含一两个便于日后核对的具体事实），累计新增 ≥17 个成功轮（4+17=21>20）；次日在新自然日问前一天的具体事实。
|
||||
|
||||
**验收观察什么**：①回复文本含前日独有事实（观察者可对照当日原文）；②journal `cybergril.memory.episodic` 行 `hits` 非空且 turn 在近20轮窗口外、`mode` 合法；③/api/memories 列表出现对应压缩条目；④近20轮排除下提问轮不在 hits。三项齐备才计"隔天取回通过"；短期泛泛回答（"我记得你说起过"级别的模糊复述）不算通过。
|
||||
|
||||
**额度建议（一次性追加，同一权威账不清零、保留当前档案/真实历史）**：
|
||||
- dialogue_main：+20（17 轮自然积累 + 2 次隔日提问 + 1 余量）
|
||||
- memory_gist：+44（每成功轮 2 条 + 失败重试余量按 20% 计）
|
||||
- persona_generation：0（已生成，不重做）
|
||||
- 自动类别维持无上限逐笔记账（成长每成功轮 1 次、判定有候选时 1 次、生活 1–4 条/日原范围）；任何未知类别 guard 拒绝并停验回报。
|
||||
|
||||
**停止条件**：AM-001 泄漏→立即停服保全；gist 连续失败 3 次或预算异常→停验证；人设漂移/检索噪声→关新实例 memory_search 保留证据回报；guard 任何 cap 外拒绝→停验回报。
|
||||
|
||||
**范围**：单一后续卡承接（建议由总管建卡授权）；不动 8300/8310/8320/8340、HEL-831、真实档案不重置、全局路由/模型/预算/认证不变；不改生产检索窗口、不用无意义填充问题冒充自然交流、不把合成轮计入自然累计。
|
||||
@@ -0,0 +1,145 @@
|
||||
"""HEL-835 correction proof v2: full 21 PAIRS in a fresh isolated DB.
|
||||
|
||||
Zero real model calls (ControlledLLM only). Fixes the two evidence defects the
|
||||
reviewer found in the first probe:
|
||||
- old seed had no companion 'sent' rows, so fillers never entered the
|
||||
production recent-turn window (JOIN companion x user, both 'sent');
|
||||
- its fingerprint was sha256(b'null') because bazi_core was empty.
|
||||
|
||||
This proof: creates a real profile through the production endpoint (program
|
||||
bazi computation, no persona model call with R2 off), seeds ONE target pair
|
||||
plus TWENTY filler pairs (both roles 'sent'), then verifies through the real
|
||||
create_app + ControlledLLM that the production recent-turn SQL excludes the
|
||||
twenty filler turns, the target sits outside, the final prompt contains the
|
||||
target memory block, the recent history does NOT contain the target's unique
|
||||
answer, and the birth fingerprint (non-null source) is unchanged.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "tests"))
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from test_api import CREATE, ControlledLLM, events # noqa: E402
|
||||
|
||||
from cybergril.api.app import create_app # noqa: E402
|
||||
from cybergril.life.persona import fingerprint # noqa: E402
|
||||
from cybergril.life.runtime import LifeConfig # noqa: E402
|
||||
from cybergril.think.router import build_router # noqa: E402
|
||||
from cybergril.think.scenes import Tier # noqa: E402
|
||||
|
||||
PASS, FAIL = [], []
|
||||
TARGET_TURN = "00000000-c0rr-0000-4000-8000-000000000000"
|
||||
UNIQUE = "蓝灰色封面诗集"
|
||||
|
||||
|
||||
def check(name, cond, detail=""):
|
||||
(PASS if cond else FAIL).append(name)
|
||||
print(("PASS" if cond else "FAIL"), name, detail)
|
||||
|
||||
|
||||
class CaptureLLM(ControlledLLM):
|
||||
"""ControlledLLM that records the full prompt of every call."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.captured = []
|
||||
|
||||
async def stream(self, messages, **kwargs):
|
||||
self.calls.append(messages)
|
||||
self.captured.append(list(messages))
|
||||
from cybergril.contracts import LLMDelta
|
||||
yield LLMDelta(text="我在听。")
|
||||
yield LLMDelta(done=True)
|
||||
|
||||
|
||||
async def main_async():
|
||||
tmp = Path(tempfile.mkdtemp(prefix="hel835-corr-"))
|
||||
|
||||
provider = CaptureLLM()
|
||||
router = build_router(providers={t: provider for t in (Tier.HIGH, Tier.STANDARD, Tier.LOW)})
|
||||
app = create_app(db_path=tmp / "corr.db", router=router,
|
||||
life_config=LifeConfig(enabled=False))
|
||||
with TestClient(app) as c:
|
||||
r = c.post("/api/companion", json={"name": "补证", "birth": "2001-05-21", "hour": "午时", "relation": "朋友"})
|
||||
check("profile created via production endpoint", r.status_code == 201)
|
||||
s = app.state.store
|
||||
fp_before = fingerprint(s)
|
||||
check("fingerprint source non-null", fp_before != __import__("hashlib").sha256(b"null").hexdigest())
|
||||
check("bazi_core populated", s.query_one("SELECT count(*) n FROM bazi_core")["n"] == 1)
|
||||
|
||||
# seed: 1 target pair (oldest) + 20 filler pairs, all both-roles sent
|
||||
pairs = [(TARGET_TURN, f"我今天在巷口旧书店买了一本{UNIQUE}。", "谢谢你告诉我，听起来很安静。")]
|
||||
for i in range(1, 21):
|
||||
pairs.append((f"00000000-c0rr-0000-4000-8000-{i:012d}",
|
||||
f"合成第{i}轮：雨天通勤和晚饭。", f"合成回复第{i}轮。"))
|
||||
with s.conn:
|
||||
for turn, user_text, comp_text in pairs:
|
||||
uid = s.execute("INSERT INTO chat_message(ts,role,text,turn_id,status) VALUES(1,'user',?,?,'sent')",
|
||||
(user_text, turn)).lastrowid
|
||||
cid = s.execute("INSERT INTO chat_message(ts,role,text,turn_id,status) VALUES(2,'companion',?,?,'sent')",
|
||||
(comp_text, turn)).lastrowid
|
||||
s.execute("INSERT INTO mem_episodic(ts,role,raw,gist,salience,affect,source,accessibility,turn_id,message_id)"
|
||||
" VALUES(1,'user',?,?,0.5,NULL,'chat',1,?,?)",
|
||||
(user_text, f"用户在巷口旧书店买了{UNIQUE}。", turn, uid))
|
||||
if turn == TARGET_TURN:
|
||||
s.execute("INSERT INTO mem_episodic(ts,role,raw,gist,salience,affect,source,accessibility,turn_id,message_id)"
|
||||
" VALUES(2,'companion',?,?,0.5,NULL,'chat',1,?,?)",
|
||||
(comp_text, "她谢谢用户分享诗集。", turn, cid))
|
||||
|
||||
# (a) production recent-turn SQL over the seeded db
|
||||
rows = s.query("""SELECT a.turn_id FROM chat_message a JOIN chat_message u
|
||||
ON u.turn_id=a.turn_id AND u.role='user'
|
||||
WHERE a.role='companion' AND a.status='sent' AND u.status='sent'
|
||||
ORDER BY a.id DESC LIMIT 20""")
|
||||
window = [r["turn_id"] for r in rows]
|
||||
check("production window has exactly 20 pairs", len(window) == 20, str(len(window)))
|
||||
check("target outside the window", TARGET_TURN not in window)
|
||||
check("all fillers inside the window", all(t in window for t in window if t != TARGET_TURN))
|
||||
|
||||
# (b)(c) real chat turn -> captured prompt + retrieval log
|
||||
import logging
|
||||
caplog_stream = []
|
||||
handler = logging.Handler()
|
||||
handler.emit = lambda record: caplog_stream.append(record.getMessage())
|
||||
log = logging.getLogger("cybergril.memory.episodic")
|
||||
log.setLevel(logging.INFO)
|
||||
log.addHandler(handler)
|
||||
epoch = c.get("/api/health").json()["epoch"]
|
||||
response = c.post("/api/chat", json={"text": f"还记得我在旧书店买的那本{UNIQUE[:-2]}吗",
|
||||
"turn_id": str(uuid4()), "epoch": epoch})
|
||||
final = events(response)[-1]
|
||||
check("chat turn completed", final["t"] == "chat.final")
|
||||
retrieval = next((line for line in caplog_stream if "mode" in line), None)
|
||||
check("retrieval fired", retrieval is not None, retrieval or "")
|
||||
if retrieval:
|
||||
parsed = json.loads(retrieval)
|
||||
check("retrieval hit target", any(h["turn_id"] == TARGET_TURN for h in parsed["hits"]),
|
||||
str(parsed["hits"]))
|
||||
check("mode phrase/trigram", parsed["mode"] in ("phrase", "trigram", "like"), parsed["mode"])
|
||||
prompt = provider.captured[-1]
|
||||
system_blob = "".join(m.content for m in prompt if m.role == "system")
|
||||
history_msgs = [m for m in prompt[:-1] if m.role in ("user", "assistant")]
|
||||
recent_blob = "".join(m.content for m in history_msgs)
|
||||
check("memory block in system messages contains target gist", UNIQUE in system_blob)
|
||||
check("recent user/assistant history does NOT contain the target answer", UNIQUE not in recent_blob)
|
||||
kept_recent = sum(1 for m in prompt[1:] if m.role in ("user", "assistant"))
|
||||
check("recent history actually delivered (non-empty)", kept_recent > 0, str(kept_recent))
|
||||
check("filler recent turns present in prompt", "合成第20轮" in recent_blob or "合成第" in recent_blob)
|
||||
fp_after = fingerprint(s)
|
||||
check("birth fingerprint unchanged after chat+retrieval", fp_after == fp_before)
|
||||
|
||||
print(f"\n{len(PASS)} passed, {len(FAIL)} failed")
|
||||
if FAIL:
|
||||
print("FAILED:", FAIL)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main_async())
|
||||
@@ -0,0 +1,19 @@
|
||||
/home/xiaomo/multica_workspaces/hello-af23464d222e/hel-829-8ee8ba59fefd/workdir/CyberGril/.venv312/lib/python3.12/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
PASS profile created via production endpoint
|
||||
PASS fingerprint source non-null
|
||||
PASS bazi_core populated
|
||||
PASS production window has exactly 20 pairs 20
|
||||
PASS target outside the window
|
||||
PASS all fillers inside the window
|
||||
PASS chat turn completed
|
||||
PASS retrieval fired {"turn_id": "6264340c-3289-4bf3-b66d-0e70e799dd96", "hits": [{"id": 1, "turn_id": "00000000-c0rr-0000-4000-8000-000000000000", "rank": -2.4335848720119766}], "mode": "trigram"}
|
||||
PASS retrieval hit target [{'id': 1, 'turn_id': '00000000-c0rr-0000-4000-8000-000000000000', 'rank': -2.4335848720119766}]
|
||||
PASS mode phrase/trigram trigram
|
||||
PASS memory block in system messages contains target gist
|
||||
PASS recent user/assistant history does NOT contain the target answer
|
||||
PASS recent history actually delivered (non-empty) 41
|
||||
PASS filler recent turns present in prompt
|
||||
PASS birth fingerprint unchanged after chat+retrieval
|
||||
|
||||
15 passed, 0 failed
|
||||
@@ -1,39 +1,40 @@
|
||||
{
|
||||
"baseline_version": "1.10",
|
||||
"read_commit": "b98ab6ab50d1631bd737dd9d4b549d9ea573d0db",
|
||||
"executed_through": "65df252",
|
||||
"read_commit": "c5167ca7c53eeac7044fe82b2d94bd47039e0951",
|
||||
"executed_through": "本轮更正提交（见 branch head）",
|
||||
"requirement_ids": [
|
||||
"CG-006", "CG-008", "CG-013", "CG-014", "CG-015", "CG-016", "CG-018", "CG-021",
|
||||
"AM-001", "VIS-001", "VIS-011", "VIS-013",
|
||||
"UX-001", "UX-003", "UX-005", "UX-006",
|
||||
"GOV-001", "GOV-002", "GOV-003", "GOV-004", "GOV-005", "DEL-001"
|
||||
],
|
||||
"deliverable": "HEL-835 统一审核+隔离部署+两自然日有限真实验证：HEL-834 源码校验入库并独立复跑（225 pytest / 42 vitest / 23 浏览器合同 chromium CDP / 30 端到端 ControlledLLM）；8350 独立实例部署运行（fc5e294→65df252），部署层 guard v1→v3 按两轮总管复核演进（唯一 call_id 结算、Scene 绑定分类、账本校验、done-delta 结算 ok；36/36 隔离证明），唯一权威账 guard-ledger.json 全程不清零；首日 9 次真实调用 + 夜间真实页面交互 2 轮（1 成功 1 取消）+ 3 条生活事件全部逐笔可溯；第二自然日真实隔天提问（近20轮排除生效 hits=[] 回答不编造）与已批合成 probe（隔离库 21 轮合成填充+手写 gist，真实模型 trigram 检索命中窗口外目标记忆并注入，回复精确引用记忆细节）完成。",
|
||||
"deliverable": "HEL-835 统一审核+隔离部署+两自然日有限真实验证（含第三轮复核后的证据更正）。工程审核：HEL-834 源码校验入库、225 pytest / 42 vitest / 23 浏览器合同 / 30 端到端独立复跑全部通过。部署：8350 独立实例（guard v3，SHA256 ee6c533d… 与仓库一致，36/36 隔离证明）。真实验证按授权完成 23 次调用（见 verification 的更正算法），零超限零 cap 外调用。",
|
||||
"acceptance": [
|
||||
"guard v3 隔离证明 36/36（同秒乱序结算/毒化边界不可绕过/账本有效性/取消关闭关上游/done-then-close ok）",
|
||||
"服务器 guard SHA256 与仓库逐字节一致（v3: ee6c533d…），双目标需求检查含本 handoff 通过（59项）",
|
||||
"隔天真实提问：hits=[] mode=none（近20轮排除生效），回答依赖近期上下文且诚实不编造；2 条新记忆压缩 done",
|
||||
"合成 probe：mode=trigram 命中目标 id=1（窗口外），回复精确引用仅存在于记忆的『蓝灰色封面』；底色指纹不变；probe 进程同轮启动并停止",
|
||||
"最终权威账：主聊 6/6、gist 8/8、人格 1/1 全部用尽且零超限零拒绝外调用；所有真实调用可溯到作业/消息行与 SSE 证据"
|
||||
"工程链路（合成库、生产近轮定义）已由 correction-proof-HEL-835.py 15/15 补证：生产近轮 SQL 恰好排除 20 对 filler、窗外目标经真实 create_app 检索命中并以独立 system 消息注入、近20轮 user/assistant 历史不含目标独有答案、非空出生底色指纹前后不变（零真实调用，不升级为真实通过）",
|
||||
"隔天真实提问已执行：hits=[] mode=none（近20轮排除生效、无长期注入）；昨日独有细节经零调用重建证实完整送入提示；回复未引用昨日独有事实——记忆体验验收未通过/未证，如实记录",
|
||||
"权威账最终 23 次调用全部可溯（首日9+复核后夜间9+隔日4+probe1），主聊6/6、gist8/8、人格1/1 用尽零超限；guard 拒绝均零上游",
|
||||
"双目标需求检查（origin/main 与 origin/agent/agent/de8874a3803b，含本 handoff）通过 59 项"
|
||||
],
|
||||
"deferred": [
|
||||
"自然积累的真实长期记忆隔天取回未验（需真实累计>20轮后另卡承接，不改生产检索窗口）",
|
||||
"真实视频口型、R4 主动联系、R5 语音、R6 天气素材完整备份、R7 授权观察仍由主卡 HEL-797 追踪；HEL-831 三天观察保持原实例",
|
||||
"8350 后续真实对话需新授权额度（当前 6/6、8/8 已满，guard 会拒绝并零上游）；PR#10 待用户验收后处置，禁止合并且未合并",
|
||||
"guard v2 时期结算 outcome 语义失真（成功被标 cancelled）已由 v3 修正，历史 calls 原样保留不回写"
|
||||
"自然积累的真实长期记忆隔天取回未验（隔天体验未通过/未证）：真实档案仅 4 成功轮，需自然累计>20 轮后跨日问旧事实，具体方案与额度建议见 docs/r3/PLAN-real-followup-HEL-835.md（仅提议未授权不执行）",
|
||||
"probe 启动条件偏离已披露：runbook 要求 memory_gist≤6 才起 probe，实际 8/8 满时启动（主聊未超限 5→6/6）；未获『已满可继续』批复，不宣称全部按授权执行",
|
||||
"旧 probe 结论已降级：seed 仅 role=user 无 companion 配对，20 filler 从未进入生产近20轮窗口（evidence/10 SQL 重跑证实）；其结果仅证明合成记忆可被检索并被模型引用",
|
||||
"8350 生产库首日底色指纹值未存档，前后相等不可强证；现存有效证据=当前非空指纹 ffbb3bd4… + pytest 隔离证明记忆作业不写受保护表 + journal 无底色写操作。教训：部署时应存档创建时指纹",
|
||||
"guard v2 时期成功调用被误标 cancelled（outcome 语义）已由 v3 修正，历史 calls 原样保留不回写，旁注见 day1-summary",
|
||||
"R4–R7、真实视频口型仍主卡 HEL-797 追踪；HEL-831 三天观察保持原实例；PR#10 禁止合并且未合并"
|
||||
],
|
||||
"requirement_changes": [],
|
||||
"verification": [
|
||||
"docs/r3/：e2e-HEL-835（30/30）、guard-test-HEL-835（36/36）、browser-HEL-835（23/23 chromium）、pytest-rerun（225 passed）、web-test（42 passed）、DEPLOY-HEL-835.md、PROBE-runbook-HEL-835.md、probe-seed-HEL-835.py",
|
||||
"服务器 /opt/cybergril-hel835-data/evidence/：day1-summary.json（含 day2 审计）、01–09 全部原始 SSE/确认/账本/probe 日志；guard-ledger.json 唯一权威账",
|
||||
"PR#10 head 已 API 核对同步至最终提交；base agent/agent/de8874a3803b",
|
||||
"真实模型调用总计：首日 9 + 夜间 5（1主聊成功+1取消+3生活）+ 隔日 4（1主聊+2gist+1判定）+ probe 1 = 19 次，全部计入权威账"
|
||||
"调用总数更正为 23（此前误报 19）：按权威账 calls 逐条累计=8 gist+6 主聊+1 人格+3 成长+2 判定+3 生活；漏算根因=只按口头分桶『夜间5』而未逐笔读账，漏掉夜间第3轮成功对话的 gist×2、判定×1、成长×1；夜间实际 9（含取消轮 1 次，计数规则含失败/取消）。23 是审计截点值，R2 生活自动调用继续按实际时间累计",
|
||||
"docs/r3/correction-proof-HEL-835.txt（15/15）、guard-test-HEL-835.txt（36/36）、e2e-HEL-835.txt（30/30）、browser-HEL-835.txt（23/23 chromium）、pytest-rerun（225）、web-test（42）",
|
||||
"服务器 evidence/：10-correction-probe-sql.txt（旧 seed 生产近轮 SQL 仅 1 对=probe 自轮）、11-correction-day2-context-rebuild.txt（4 轮全保留、蓝色封面/旧书店/潮潮味道均在最终提示、budget 851 未裁剪）、12-correction-birth-fingerprint.txt（bazi_core=1、非空指纹）、07/08/09 原始 SSE/journal/probe 日志全保留",
|
||||
"PR#10 head 以 API 核对为准；分支 hel-835-unified-review，AGit topic 同步"
|
||||
],
|
||||
"product_target": "origin/agent/agent/de8874a3803b",
|
||||
"branch": "hel-835-unified-review",
|
||||
"status": "in_progress",
|
||||
"deployed": true,
|
||||
"deploy_endpoint": "http://192.168.200.12:8350",
|
||||
"real_model_calls_total": 19,
|
||||
"real_model_calls_total": 23,
|
||||
"pr_url": "https://gitea.xbay.cc/leefer/CyberGril/pulls/10"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user