Compare commits

...
Author SHA1 Message Date
Multica 施工员andmultica-agent dab536987a Fix bedroom loop return and stabilize hand mattes
Co-authored-by: multica-agent <github@multica.ai>
2026-10-20 16:14:20 +08:00
Multica 施工员andmultica-agent 444ee56c6a feat: add layered bedroom motion review sample
Co-authored-by: multica-agent <github@multica.ai>
2026-10-20 15:35:25 +08:00
19 changed files with 2626 additions and 0 deletions
+7
View File
@@ -0,0 +1,7 @@
node_modules/
dist/
public/assets/*
!public/assets/manifest.json
evidence/*
!evidence/.gitkeep
__pycache__/
+55
View File
@@ -0,0 +1,55 @@
HEL-711 最小动态校对小样,基于 main `1e9b08a`。仅在 `samples/bedroom` 内工作;不连接人格、记忆、天气或语音。采用 HEL-708 的环境/人物/交谈分层边界,以及 CG-001~026 的当前需求。旧演示不变。
## 本地复核
1. 将本卡附件 `HEL-711-review-bundle.zip` 中的 `public/assets` 解压到此目录对应位置。
2. `npm ci --ignore-scripts --no-audit --no-fund`,沿用旧前端分支的 package.json/package-lock.json/tsconfig.json,无新依赖。
3. `npm run build`(包含严格 TypeScript 检查)。
4. `npm run dev -- --port 5173`,本机浏览器打开终端输出的地址。
5. `python3 tools/browser_check.py` 使用已安装的 Playwright/Chromium,自动启停本机测试服务并保存证据。此运行机需要给该命令设置 `LD_LIBRARY_PATH=/home/xiaomo/pwlibs/root/usr/lib/x86_64-linux-gnu`,复用了已有浏览器库,未安装系统依赖。
素材不进 Git,缺失时页面明确提示;manifest 在 Git 中保存来源、原图 SHA256、坐标、编码和问题。附件内含完整 assets、已构建 dist、浏览器连续录屏、原始试片、透明试片、分层校对图及生成工作流/日志。
## 代码与边界
- `src/main.ts`:一个持久 video 元素驱动透明 canvas;环境按钮只改环境亮度,不重建/seek 视频。人物、前景遮挡、接触阴影独立开关;静态近景不替代真实交谈。
- `src/tokens.css`:原样复用旧前端唯一 token 表。`style.css` 为独立工程校对台,无第二套同类组件。
- `tools/extract_layers.py`:读取现有 SAM vit-b 模型的原始 modeling API,避免可选 predictor 的 OpenCV 依赖;不伪造 cv2,不下载。原图 RGB 保留,只生成 alpha;封闭孔洞填充、0.65px 羽化。
- `tools/contact_layers.py`:从原入景图提取毯子前景,另建可关闭的暂定接触阴影;未宣称物理阴影校准完成。
- `tools/generate_once.py`:原 ComfyUI venv + 原生 Wan 节点,单片段、离线、自定义节点禁用。临时服务器仅监听回环 18189,finally 终止自己启动的精确 PID。不修改现有服务、系统 Python、驱动或模型路径配置。
- `tools/matte_video.py`:49 帧逐帧 SAM 分割,原有 PyAV 编码 VP9 alpha。原始视频未抠像的背景不用于浏览器人物合成。
- `tools/browser_check.py`:真实 Chromium 检查、三种视口截图和可下载连续录屏。页面隐藏的事件处理做了合成 visibilitychange 检查;真实系统切后台/锁屏仍待体验环境手测。
## 素材处理命令
处理目录:`.12:/media/lucian/Downloads/ComfyUI/output/hel711-task`,输入复制到其 `assets`,脚本放此目录;没有改变现有项目输入。原图由 Multica 鉴权 attachment download 获取。运行时使用原 `/media/lucian/Downloads/ComfyUI/venv/bin/python`。
```bash
python extract_layers.py assets \
--sam-source /media/lucian/Downloads/ComfyUI/custom_nodes/was-node-suite-comfyui/repos/SAM/segment_anything \
--checkpoint /media/lucian/Downloads/ComfyUI/models/sams/sam_vit_b_01ec64.pth
python generate_once.py
python matte_video.py
```
回传处理后的素材到本地 `public/assets`,运行 `python3 tools/contact_layers.py`。手工蒙版和坐标均保留在脚本中。`workflow.json` 是本次实际运行的请求,最终脚本仅补了目录创建/日志关闭等清理逻辑;日志和 history 可交叉核对。执行新试片前必须重新检查显存和端口;禁止停止已有服务腾资源。
## 当前实测与未通过项
当前结果见 `REPORT.md`,属于可审核候选,不代表 D1–D6 全部通过。图层是真实独立合成;没有用重生成整幅图代替分层。近景仍用已确认 v2 静态方向图;没有真人语音或口型。
## HEL-712 返工复现
本轮只补原动作的真实回程,并处理书本/手部蒙版;环境、机位、比例、UI token 与静态近景不变。完整返工包中的 `public/assets` 替换旧素材后执行同一构建/浏览器检查。
已有 ComfyUI 环境中,保留原片在 `output/hel711-task/reading_00001_.webm`、原透明片在同目录 `reading-alpha.webm`:
```bash
python generate_return.py
python stabilize_loop.py
python measure_loop.py
```
脚本放在 `output/hel712-task` 并附同目录 `extract_layers.py`。生成器只用已有原生 `WanFirstLastFrameToVideo` 节点与原有权重,以原片 f48、f0 为两端,seed 712,只生成一条49帧回程。服务器仍仅临时监听回环18189;主程序收齐结果后停止精确子进程。`stabilize_loop.py` 读取原49帧 alpha,新增49帧按书本/衣袖补充分割提示;只在原512坐标 y175–330 对动作对齐的5帧有符号距离去掉最大/最小值后取均值稳定,RGB不做时域融合,原发丝区域不重新分割。原始两段按顺序拼成98帧,接点48→49、97→0,无倒放/淡化/冻结/短窗口裁切。
`measure_loop.py` 输出固定坐标口径的逐对差异、蒙版统计、动作补偿残差、接头与手部证据图;数值不替代目验。`browser_check.py` 先在真实Chromium完整播放至少3周期,再做原交互与三视口复测。真实后台/锁屏及Safari/iOS仍未测。
+52
View File
@@ -0,0 +1,52 @@
HEL-712:原阅读动作循环与书本/手部蒙版返工候选,交总工复审,不部署。
## 实现
沿用原人物、米色开衫、深灰长裤、卧室、机位与552px人物落点。现有 Wan2.2 原生首尾帧条件节点,使用原片 f48→f0,seed 712,实际生成一条49帧抬头回程。原49帧向前播放,随后49帧真实回程,98帧/16fps/6.125秒,内部接点48→49,跨周期97→0。没有倒放、冻结、静态摆动、遮脸淡化或短窗口裁切。
原49帧 alpha 不重新分割;新增49帧补书本/衣袖正点,保留连通人物,去掉分离的沙发碎点。两段仅在原512px坐标 y175:330(输出552px坐标189:356)稳定蒙版:TV-L1对齐前后各2帧,有符号距离去掉最高/最低值,再取均值,限制到当前轮廓6px不确定区域,首尾8行回接原蒙版。只处理alpha,RGB不做时间平均;原发丝区域保持原alpha。原始源片与透明片逐帧映射见manifest/stabilization.json。
第一次新增回程直接套原分割提示时,会漏掉书本/衣袖;该版已拒绝,`before-crops.png` 保留实证。补充分割提示后才进入最终蒙版稳定。共生成一条视频,没有批量换装或动作试片。第一次稳定使用中值共识,接头alpha变化仍偏集中;最终使用去极值均值,缓和了两个分段蒙版的多数票切换。
## 同口径差异与边缘统计
本报告独立固定口径:RGB头部框为512px源坐标 x230:340、y50:180;蒙版缩到512px、alpha>127,边界用二值侵蚀差,书本/手部带为全部x、y175:330。与HEL-710采用的框/缩放细节不同,所以这里原接头22.30不直接与审核卡29.5混算;下列前后全部由同一脚本重算。完整逐对数据见 `evidence/metrics.json`,覆盖全部98个相邻对(包括末→首)。
| 头部RGB绝对差 /255 | 最小 | 中位 | P95 | 最大 |
|---|---:|---:|---:|---:|
| 原49帧普通相邻 | 0.36 | 1.73 | 4.97 | 5.75 |
| 新98帧普通相邻(排除两接头) | 0.24 | 1.65 | 5.47 | 8.30 |
原f48→f0:22.30;新内部f48→49:3.66;新跨周期f97→0:5.22。两个接头落在普通相邻变化范围内。接头前后整帧、头部放大与98帧序列均附;连续录屏为实际浏览器播放,未用离线合成冒充。
| 蒙版指标 | 稳定前 | 最终稳定后 |
|---|---:|---:|
| 原49帧带内边界像素范围 | 510–794 | 509–604 |
| 原49帧最差相邻边界变化 | 255 | 75 |
| 原49帧相邻alpha开关最大像素 | 771 | 300 |
| 全98帧边界像素范围 | 504–794 | 503–604 |
| 全98对最差边界变化(含循环) | 261 | 75 |
| 全98对alpha开关P95 / 最大像素 | 505.05 / 988 | 142.55 / 300 |
| 内部 / 跨周期接头alpha开关像素 | 988 / 528 | 195 / 193 |
| 动作补偿后alpha开关P95 / 最大 | 503.25 / 988 | 152.75 / 300 |
动作补偿使用同一源RGB光流对齐前后蒙版;可见减少的是分割开关,不靠抹掉真实运动。人物内部透明片与源RGB编码误差逐帧中位1.89、最大2.10/255;无RGB时间混合。带外alpha逐数组一致为true。重点帧可见书本/指尖与袖口保持,未见新增吞手或RGB拖影;仍需总工以连续视频独立确认。没有把数值降低当作人体解剖或身份认证。
## 验证
- 来源包SHA256:60/60核对一致。
- `npm ci --ignore-scripts --no-audit --no-fund`:沿用锁文件,无新依赖。
- `npm run build`:严格TypeScript检查和Vite构建通过;原JS/CSS逻辑保持,返工通过素材和工具落地。
- `ruff check src tests samples/bedroom/tools`:通过。
- `PYTHONPATH=src python3 -m pytest -q`:78通过。
- `LD_LIBRARY_PATH=/home/xiaomo/pwlibs/root/usr/lib/x86_64-linux-gnu python3 tools/browser_check.py`:真实Chromium通过。最终视频先完整播放3周期,记录循环时间约6.28、12.42、18.56秒;此段全部图层显示,之后再执行交互测试。
- VP9 alpha背景0/人物255;环境切换currentTime 0.905816→0.905816,source不变;图层开关、静态近景进退且播放继续、手动暂停、合成visibility暂停/恢复、减少动效切静态、缺片回退通过;1600×900、1280×800、390×844无横溢,人脸在画面内;pageerror=0。
- 真实后台/锁屏、Safari/iOS未测;近景仅静态,未增加口型、语音或后台生活功能。手机构图、沙发几何、重打光与近景润色保持原记录。
## 文件与运行边界
修改 `tools/extract_layers.py`(可选补充提示)、新增 `tools/generate_return.py`、`stabilize_loop.py`、`measure_loop.py`,扩展 `tools/browser_check.py` 的连续三周期验证,更新manifest/README/本报告。没有改CSS、主题、核心服务或部署配置。
分支继续 `hel-711-layered-bedroom`,基于原444ee56;最终提交与PR head核对写入附件 `DELIVERY.json` 及任务完成评论。素材不进Git。完整包包含源码、构建产物、manifest、SHA256SUMS、原始前程/回程、整周期源片、透明片、逐帧前后alpha、生成请求/节点/history/server日志、浏览器连续录屏、三视口及接头/手部证据、复现说明。
严格校验了.12已确认SSH指纹。临时ComfyUI仅回环18189、精确PID600329,生成请求成功后已停止。CPU蒙版任务均前台收齐,没有遗留生成任务、没有新增依赖/模型/付费接口;ASR/TTS及18080原演示保持运行。未推main、未合并、未部署。
+38
View File
@@ -0,0 +1,38 @@
<!doctype html>
<html lang="zh-CN">
<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>卧室分层校对 · HEL-711</title></head>
<body>
<main>
<header><h1>卧室分层校对</h1><p>工程小样 · 手动光照演示 · 近景仅静态参考,未接语音或口型</p></header>
<div id="stage" role="img" aria-label="固定机位卧室人物合成">
<div id="frame">
<img id="environment" src="./assets/environment.png" alt="无人高层卧室">
<img id="shadow" src="./assets/contact-shadow.png" alt="" class="layer">
<img id="person" src="./assets/person.png" alt="阅读人物透明层" class="layer">
<canvas id="motion" width="552" height="552" hidden></canvas>
<img id="occlusion" src="./assets/occlusion.png" alt="" class="layer">
<img id="mask" src="./assets/person-mask.png" alt="人物蒙版" class="layer" hidden>
<img id="talk" src="./assets/talk-reference.png" alt="交谈静态近景参考" class="layer" hidden>
</div>
</div>
<video id="source" muted playsinline loop preload="metadata" hidden></video>
<section aria-label="校对控制">
<fieldset><legend>图层开关</legend>
<label><input id="show-person" type="checkbox" checked>人物</label>
<label><input id="show-shadow" type="checkbox" checked>接触阴影</label>
<label><input id="show-occlusion" type="checkbox" checked>前景毯子</label>
<label><input id="show-mask" type="checkbox">仅看人物蒙版</label>
</fieldset>
<div class="controls">
<button id="lighting" type="button" aria-pressed="false">光照演示:原始</button>
<button id="play" type="button" disabled>读取动作素材</button>
<button id="closeup" type="button" aria-pressed="false">进入静态近景</button>
</div>
<p id="status" role="status">读取素材清单…</p>
<output id="time">动作位置:未载入</output>
<p id="fallback" role="alert" hidden></p>
<p>人物、房间和近景来自已确认的 v2 参考。图层独立不等于几何、发丝与循环已经通过验收。</p>
</section>
</main>
<script type="module" src="./src/main.ts"></script>
</body></html>
+1166
View File
File diff suppressed because it is too large Load Diff
+19
View File
@@ -0,0 +1,19 @@
{
"name": "cybergril-rain-room",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite --host 127.0.0.1",
"typecheck": "tsc --noEmit",
"build": "tsc --noEmit && vite build",
"preview": "vite preview --host 127.0.0.1"
},
"dependencies": {
"pixi.js": "^8.0.0"
},
"devDependencies": {
"typescript": "~5.9.3",
"vite": "^5.4.0"
}
}
+162
View File
@@ -0,0 +1,162 @@
{
"schema": "hel712-review-1",
"status": "review_candidate",
"reference_sha256": {
"reference.png": "226d7922e042e22e4b8a8241a7cf6f7aea50c301029978e4ca26cad9b3d56359",
"environment.png": "b5d4771eb010dc1d0a41d1b698e45edaa45d2f90bb441378bc32cf5c812e76a2",
"talk-reference.png": "0ae9b1a1cb9c590ab89a81ce4efd25de843a600ff227c184663db23e9e9c16b3"
},
"camera": {
"canvas": [
1672,
941
],
"generation_crop": [
1120,
200,
552,
552
],
"person_anchor": [
1390,
530
],
"coordinate_units": "source pixels, top left origin",
"transform": "none; crop reinserted into original position",
"clothing": "v2 cream cardigan, charcoal trousers"
},
"layers": [
{
"file": "environment.png",
"source": "approved 02-v2 original",
"alpha": false
},
{
"file": "contact-shadow.png",
"source": "engineered ellipse 65/255, 12px blur, not physically calibrated",
"alpha": true
},
{
"file": "person.png",
"source": "original 03-v2 RGB + existing SAM vit-b alpha; enclosed mask holes filled",
"alpha": true
},
{
"file": "occlusion.png",
"source": "03-v2 blanket pixels, hand-authored polygon matte",
"alpha": true
}
],
"motion": {
"status": "真实回程与书本手部蒙版返工候选,待复审",
"file": "reading-alpha.webm",
"alpha": true,
"maskFile": null,
"original": "loop-source.webm",
"generator": "installed ComfyUI native Wan2.2 high/low 14B fp8 + installed LightX2V 4-step LoRAs",
"seed": 711,
"dimensions": [
552,
552
],
"generation_dimensions": [
512,
512
],
"frames": 98,
"fps": 16,
"duration_seconds": 6.125,
"codec": "VP9 yuva420p / WebM",
"matting": "original 49 alpha retained outside hand/book band; added return frames use book/sleeve foreground points + connected subject; y175:330 (512 source) uses TV-L1 aligned 5-frame trimmed signed-distance mean within 6px uncertainty band",
"start_pose": "seated holding open book, head slightly tilted toward camera",
"end_pose": "generated return to the original relaxed camera-facing pose",
"loop": "forward 0–48 + real generated return 49–97; joins 48→49 and 97→0, no reverse/fade/freeze",
"cut_point": "no subwindow trimming",
"return_seed": 712,
"source_map": [
{
"frames": [
0,
48
],
"file": "reading_00001_.webm",
"source_frames": [
0,
48
]
},
{
"frames": [
49,
97
],
"file": "return_00001_.webm",
"source_frames": [
0,
48
]
}
]
},
"closeup": {
"file": "talk-reference.png",
"type": "static_reference_only",
"identity_verified": false,
"lipsync": false,
"source": "approved v2 static direction reference"
},
"known_issues": [
"Original empty/occupied sofa and pillow geometry differs; residual contact geometry needs art review.",
"Book and fingers are small; hand anatomy/identity not certified.",
"Manual environment dimming changes background only; subject relighting not implemented.",
"Static near-shot is a generated reference; cross-shot face continuity not certified.",
"No real weather, speech, lipsync or life-state integration.",
"Loop joins and stabilized hand/book edges are a HEL-712 candidate pending independent re-review."
],
"files": {
"talk-reference.png": {
"bytes": 2246098,
"sha256": "0ae9b1a1cb9c590ab89a81ce4efd25de843a600ff227c184663db23e9e9c16b3"
},
"reading-alpha.webm": {
"bytes": 1971817,
"sha256": "9b9b95fff6c047c68ea66d14eba1ca896a80107b7c5b531c326bfac8bdc90b76"
},
"person.png": {
"bytes": 2522355,
"sha256": "a904f4081069ba97019d3c679d8dd59102118560471f8d6337f7b29d60380f24"
},
"composite.png": {
"bytes": 2576324,
"sha256": "4354716558858d0e9ac9a4c4a025d77f4042a0e106f10fc37e500c7157e77456"
},
"person-mask.png": {
"bytes": 7264,
"sha256": "7569160b3c87ebf1d5bb66792ed6d2db6b41fde006fad0473e7db5337d18b0f7"
},
"environment.png": {
"bytes": 2442707,
"sha256": "b5d4771eb010dc1d0a41d1b698e45edaa45d2f90bb441378bc32cf5c812e76a2"
},
"occlusion-mask.png": {
"bytes": 4838,
"sha256": "dc22d1823a02df4fa5f3bf484f157c49b270f100185c0a35b3c31f28880dd3f0"
},
"generation-input.png": {
"bytes": 508812,
"sha256": "f3922903fb28c5940a2e272b70be6e33dc094039b4e275724cf3ff1aa4af1584"
},
"occlusion.png": {
"bytes": 2522991,
"sha256": "5f0f4aa7ae90ad863f694365eaa93d7d7558841ecca08c95d415731450226c4f"
},
"reference.png": {
"bytes": 2384917,
"sha256": "226d7922e042e22e4b8a8241a7cf6f7aea50c301029978e4ca26cad9b3d56359"
},
"contact-shadow.png": {
"bytes": 16144,
"sha256": "948f88c1298638158f9a21ffe107bf403106dcf6e58c72a3d6e738fe0c7cc003"
}
}
}
+141
View File
@@ -0,0 +1,141 @@
import './style.css';
type Manifest = {
motion: { status: string; file: string | null; alpha: boolean; maskFile: string | null };
};
const get = <T extends HTMLElement>(id: string) => document.getElementById(id) as T;
const video = get<HTMLVideoElement>('source');
const canvas = get<HTMLCanvasElement>('motion');
const context = canvas.getContext('2d')!;
const play = get<HTMLButtonElement>('play');
const person = get<HTMLImageElement>('person');
const status = get('status');
const reduced = matchMedia('(prefers-reduced-motion: reduce)');
const missingImages = new Set<string>();
let ready = false;
let wantsPlay = false;
let near = false;
let failed = false;
let frameHandle = 0;
function fallback(message: string) {
const node = get('fallback');
node.hidden = false;
node.textContent = message;
}
for (const image of document.querySelectorAll<HTMLImageElement>('#frame img')) {
image.addEventListener('error', () => {
missingImages.add(image.id);
image.hidden = true;
fallback(`素材缺失:${image.id}。请解压任务素材包到 public/assets;当前内容不能作为验收结果。`);
});
}
function visibility() {
const visible = get<HTMLInputElement>('show-person').checked;
const maskOnly = get<HTMLInputElement>('show-mask').checked;
const activeMotion = ready && !failed && !reduced.matches;
person.hidden = !visible || maskOnly || near || activeMotion;
canvas.hidden = !visible || maskOnly || near || !activeMotion;
get('environment').hidden = maskOnly;
get('mask').hidden = !maskOnly || near;
get('shadow').hidden = !visible || maskOnly || near || !get<HTMLInputElement>('show-shadow').checked;
get('occlusion').hidden = !visible || maskOnly || near || !get<HTMLInputElement>('show-occlusion').checked;
get('talk').hidden = !near;
for (const id of missingImages) get(id).hidden = true;
}
function paint() {
if (video.readyState >= 2) {
context.clearRect(0, 0, canvas.width, canvas.height);
context.globalCompositeOperation = 'source-over';
context.drawImage(video, 0, 0, canvas.width, canvas.height);
}
get('time').textContent = ready
? `动作位置:${video.currentTime.toFixed(2)} / ${Number.isFinite(video.duration) ? video.duration.toFixed(2) : '…'} 秒`
: '动作位置:未载入';
}
function draw() {
paint();
if (!video.paused && !document.hidden) frameHandle = requestAnimationFrame(draw);
}
async function syncPlayback() {
cancelAnimationFrame(frameHandle);
const run = wantsPlay && ready && !failed && !document.hidden && !reduced.matches;
if (run) {
try {
await video.play();
draw();
} catch {
wantsPlay = false;
video.pause();
fallback('浏览器未允许播放;请点击播放重试。');
}
} else { video.pause(); paint(); }
play.textContent = ready ? (video.paused ? '播放真实试片' : '暂停真实试片') : '真实动作尚不可用';
visibility();
}
function layout() {
const stage = get('frame');
const scale = Math.min(stage.clientWidth / 1672, stage.clientHeight / 941);
const left = (stage.clientWidth - 1672 * scale) / 2;
const top = (stage.clientHeight - 941 * scale) / 2;
Object.assign(canvas.style, { left: `${left + 1120 * scale}px`, top: `${top + 200 * scale}px`,
width: `${552 * scale}px`, height: `${552 * scale}px` });
}
new ResizeObserver(layout).observe(get('frame'));
for (const input of document.querySelectorAll('input')) input.addEventListener('change', visibility);
get('lighting').addEventListener('click', () => {
const button = get<HTMLButtonElement>('lighting');
const dim = button.getAttribute('aria-pressed') !== 'true';
button.setAttribute('aria-pressed', String(dim));
button.textContent = `光照演示:${dim ? '环境减光' : '原始'}`;
get('environment').style.setProperty('--demo-environment-brightness', dim ? '0.65' : '1');
// No source assignment, load(), currentTime mutation or player recreation here.
});
get('closeup').addEventListener('click', () => {
near = !near;
const button = get<HTMLButtonElement>('closeup');
button.setAttribute('aria-pressed', String(near));
button.textContent = near ? '退出静态近景' : '进入静态近景';
visibility();
});
play.addEventListener('click', () => { wantsPlay = !wantsPlay; void syncPlayback(); });
document.addEventListener('visibilitychange', () => { void syncPlayback(); });
reduced.addEventListener('change', () => {
if (reduced.matches) wantsPlay = false;
void syncPlayback();
});
video.addEventListener('error', () => {
failed = true; ready = false; wantsPlay = false; play.disabled = true;
fallback('动作素材缺失或解码失败,回退静态人物;未通过动态验证。');
void syncPlayback();
});
async function init() {
try {
const response = await fetch('./assets/manifest.json');
if (!response.ok) throw new Error('manifest');
const manifest: Manifest = await response.json();
const motion = manifest.motion;
if (!motion.file || !motion.alpha) {
status.textContent = '静态分层可校对;无合格的透明动作素材,动态验收未通过。';
play.disabled = true; play.textContent = '真实动作尚不可用'; return;
}
video.src = `./assets/${motion.file}`;
video.addEventListener('loadeddata', () => {
ready = true; play.disabled = false;
status.textContent = `真实生成试片 · ${motion.status} · 循环切点与身份一致性以检查报告为准`;
paint(); void syncPlayback();
}, { once: true });
} catch {
fallback('素材清单不可用;保留静态分层校对。');
play.textContent = '真实动作尚不可用';
}
}
visibility(); layout(); void init();
+22
View File
@@ -0,0 +1,22 @@
@import './tokens.css';
* { box-sizing: border-box; }
[hidden] { display: none !important; }
body { margin: 0; background: var(--scene-night-deep); color: var(--ink-primary); font: var(--fs-body) var(--font-sans); }
main { max-width: 1600px; margin: auto; }
header, section { padding: var(--space-4) var(--space-6); }
h1 { font-size: var(--fs-title); margin: 0; }
p { color: var(--ink-secondary); line-height: 1.5; }
#stage { position: relative; overflow: hidden; height: min(60vh, 640px); background: var(--glass-bg-strong); }
#frame { position: absolute; width: 100%; height: 100%; }
#environment, .layer { position: absolute; width: 100%; height: 100%; object-fit: contain; }
#environment { filter: brightness(var(--demo-environment-brightness, 1)); }
#motion { position: absolute; }
fieldset { border: 1px solid var(--glass-border); border-radius: var(--r-control); padding: var(--space-3); display: flex; flex-wrap: wrap; gap: var(--space-4); }
label, button { cursor: pointer; }
.controls { display: flex; flex-wrap: wrap; gap: var(--space-2); margin-top: var(--space-3); }
button { color: var(--ink-primary); background: var(--glass-bg-strong); border: 1px solid var(--glass-highlight); border-radius: var(--r-control); padding: var(--space-3) var(--space-4); font: inherit; }
button[aria-pressed=true] { border-color: var(--accent); }
button:disabled { color: var(--ink-disabled); cursor: default; }
button:focus-visible, input:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
output { font-variant-numeric: tabular-nums; color: var(--accent); }
@media (max-width: 600px) { header, section { padding: var(--space-3); } #stage { height: 38vh; } }
+73
View File
@@ -0,0 +1,73 @@
/* Reused verbatim from hel-700-rain-room; no alternative product theme. */
:root {
--scene-night-deep:#16233c;
--scene-city-blue:#305481;
--scene-city-mid:#5a76a2;
--scene-tower:#2b456b;
--scene-glass-blue:#718fc2;
--scene-amber:#e8a45c;
--scene-amber-soft:#f2c894;
--scene-wood:#59361d;
--scene-cream:#e8d9c8;
--scene-blanket:#362928;
--ink-primary:#f2efe9;
--ink-secondary:rgba(242,239,233,.64);
--ink-tertiary:rgba(242,239,233,.4);
--ink-disabled:rgba(242,239,233,.28);
--accent:#e8a45c;
--accent-hover:#f2b879;
--accent-pressed:#c9853f;
--accent-ink:#1a1208;
--glass-bg:rgba(14,18,30,.55);
--glass-bg-strong:rgba(14,18,30,.72);
--glass-border:rgba(255,255,255,.08);
--glass-highlight:rgba(255,255,255,.14);
--bubble-user:rgba(120,144,180,.2);
--bubble-user-border:rgba(255,255,255,.07);
--bubble-her:rgba(255,255,255,.06);
--bubble-failed:rgba(200,90,80,.14);
--state-danger:#e07a6a;
--font-sans:"PingFang SC","HarmonyOS Sans SC","MiSans","Source Han Sans SC",system-ui,-apple-system,sans-serif;
--fs-caption:12px;
--fs-small:13px;
--fs-body:14px;
--fs-chat:15px;
--fs-nav:15px;
--fs-title:20px;
--fs-brand:16px;
--fs-time:13px;
--space-1:4px;
--space-2:8px;
--space-3:12px;
--space-4:16px;
--space-5:20px;
--space-6:24px;
--space-8:32px;
--space-10:40px;
--r-panel:24px;
--r-bubble:16px;
--r-card:16px;
--r-control:12px;
--r-pill:999px;
--shadow-panel:0 24px 80px rgba(0,0,0,.45);
--shadow-pop:0 8px 24px rgba(0,0,0,.35);
--glow-accent:0 0 12px rgba(232,164,92,.45);
--glass-blur:blur(28px) saturate(1.2);
--z-scene-far:0;
--z-scene-rain:1;
--z-scene-near:2;
--z-caption:10;
--z-nav:10;
--z-topbar:20;
--z-panel:30;
--z-toast:40;
--ease-out:cubic-bezier(.23,1,.32,1);
--ease-in-out:cubic-bezier(.77,0,.175,1);
--ease-drawer:cubic-bezier(.32,.72,0,1);
--lamp-opacity:.12;
--panel-width:clamp(360px,30vw,460px);
--panel-top:64px;
--panel-inset:24px;
--scene-height:330px;
--drawer-height:30dvh;
}
+158
View File
@@ -0,0 +1,158 @@
"""Foreground Chromium interaction checks, screenshots and downloadable recording."""
import functools
import http.server
import json
import threading
from pathlib import Path
from playwright.sync_api import sync_playwright
ROOT = Path(__file__).resolve().parents[1]
EVIDENCE = ROOT / 'evidence'
class Quiet(http.server.SimpleHTTPRequestHandler):
def log_message(self, *_args):
pass
def main():
EVIDENCE.mkdir(exist_ok=True)
handler = functools.partial(Quiet, directory=str(ROOT / 'dist'))
server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), handler)
thread = threading.Thread(target=server.serve_forever)
thread.start()
results = {}
try:
with sync_playwright() as p:
browser = p.chromium.launch()
url = f'http://127.0.0.1:{server.server_port}'
context = browser.new_context(viewport={'width': 1280, 'height': 800},
record_video_dir=str(EVIDENCE),
record_video_size={'width': 1280, 'height': 800})
page = context.new_page()
errors = []
page.on('pageerror', lambda error: errors.append(str(error)))
page.goto(url)
page.wait_for_function('!document.querySelector("#play").disabled')
assert page.locator('#source').evaluate('(v) => v.paused')
page.click('#play')
# Keep every layer visible for three full cycles before interaction checks.
duration = page.locator('#source').evaluate('(v) => v.duration')
page.evaluate("""() => {
window.observedLoops = []; window.lastVideoTime = 0;
const v = document.querySelector('#source');
v.addEventListener('timeupdate', () => {
if (v.currentTime < window.lastVideoTime) window.observedLoops.push(performance.now());
window.lastVideoTime = v.currentTime;
});
}""")
page.wait_for_timeout((duration * 3 + 0.8) * 1000)
observed = page.evaluate('window.observedLoops')
assert len(observed) >= 3, observed
results['continuous_three_cycles'] = {'duration': duration, 'wraps_ms': observed}
page.wait_for_timeout(800)
alpha = page.locator('#motion').evaluate('(c) => {const x=c.getContext("2d"); return [x.getImageData(0,0,1,1).data[3], x.getImageData(300,220,1,1).data[3]]}')
assert alpha[0] == 0 and alpha[1] > 240, alpha
results['vp9_alpha_decoded_by_chromium'] = alpha
page.wait_for_function('document.querySelector("#source").currentTime > 0.9 && document.querySelector("#source").currentTime < 1.4')
positions = page.evaluate('''() => {
const video=document.querySelector('#source'); const before=video.currentTime;
const source=video.currentSrc; document.querySelector('#lighting').click();
return {before, after:video.currentTime, sourcePreserved: source===video.currentSrc};
}''')
assert positions['before'] == positions['after'] and positions['sourcePreserved']
results['environment_toggle_no_seek'] = positions
page.wait_for_timeout(650)
assert page.locator('#source').evaluate('(v) => !v.paused')
page.click('#show-person')
assert page.locator('#motion').is_hidden() and page.locator('#person').is_hidden()
page.wait_for_timeout(600)
page.screenshot(path=str(EVIDENCE / 'environment-only.png'))
page.click('#show-person')
page.click('#show-occlusion')
assert page.locator('#occlusion').is_hidden()
page.click('#show-occlusion')
page.click('#show-shadow')
assert page.locator('#shadow').is_hidden()
page.click('#show-shadow')
page.click('#show-mask')
assert page.locator('#mask').is_visible() and page.locator('#environment').is_hidden()
page.wait_for_timeout(500)
page.click('#show-mask')
results['independent_layers'] = True
page.click('#closeup')
assert page.locator('#talk').is_visible()
assert page.locator('#source').evaluate('(v) => !v.paused')
page.wait_for_timeout(700)
page.screenshot(path=str(EVIDENCE / 'static-near.png'))
page.click('#closeup')
assert page.locator('#talk').is_hidden()
page.wait_for_timeout(6400) # Includes the forward and generated return.
results['near_is_static_player_continues'] = True
page.click('#play')
t = page.locator('#source').evaluate('(v) => v.currentTime')
page.wait_for_timeout(250)
assert page.locator('#source').evaluate('(v) => v.currentTime') == t
results['manual_pause'] = True
# Chromium lifecycle API freezes the page after making it hidden.
session = context.new_cdp_session(page)
session.send('Page.setWebLifecycleState', {'state': 'active'})
# Visibility handler unit probe. Browser screenshot/recording checks above are real.
page.click('#play')
page.evaluate('''() => {
Object.defineProperty(document, 'hidden', {configurable:true, get:()=>true});
document.dispatchEvent(new Event('visibilitychange'));
}''')
assert page.locator('#source').evaluate('(v) => v.paused')
page.evaluate('''() => {
delete document.hidden; document.dispatchEvent(new Event('visibilitychange'));
}''')
page.wait_for_function('!document.querySelector("#source").paused')
results['visibility_pause_resume_handler_synthetic_event'] = True
for width, height in [(1600, 900), (1280, 800), (390, 844)]:
page.set_viewport_size({'width': width, 'height': height})
page.wait_for_timeout(180)
bounds = page.evaluate('''() => {
const r=document.querySelector('#motion').getBoundingClientRect();
const s=document.querySelector('#stage').getBoundingClientRect();
return {faceX:r.x+305/552*r.width, faceY:r.y+135/552*r.height,
stage:{left:s.left,right:s.right,top:s.top,bottom:s.bottom},
scroll:document.documentElement.scrollWidth, width:innerWidth};
}''')
assert bounds['scroll'] <= width
assert bounds['stage']['left'] <= bounds['faceX'] <= bounds['stage']['right']
assert bounds['stage']['top'] <= bounds['faceY'] <= bounds['stage']['bottom']
page.screenshot(path=str(EVIDENCE / f'viewport-{width}x{height}.png'))
results[f'viewport_{width}x{height}'] = bounds
page.emulate_media(reduced_motion='reduce')
page.wait_for_timeout(100)
assert page.locator('#source').evaluate('(v) => v.paused')
assert page.locator('#person').is_visible() and page.locator('#motion').is_hidden()
results['reduced_motion_static_fallback'] = True
page.emulate_media(reduced_motion='no-preference')
assert page.locator('#source').evaluate('(v) => v.paused')
video_path = Path(page.video.path())
context.close()
video_path.rename(EVIDENCE / 'browser-continuous.webm')
missing = browser.new_page()
missing.route('**/reading-alpha.webm', lambda route: route.abort())
missing.goto(url)
missing.wait_for_function('!document.querySelector("#fallback").hidden')
assert missing.locator('#person').is_visible()
assert missing.locator('#play').is_disabled()
results['missing_motion_static_fallback'] = True
missing.close()
assert not errors, errors
results['page_errors'] = errors
browser.close()
finally:
server.shutdown()
thread.join()
server.server_close()
(EVIDENCE / 'browser-results.json').write_text(json.dumps(results, ensure_ascii=False, indent=2))
print(json.dumps(results, ensure_ascii=False, indent=2))
if __name__ == '__main__':
main()
+33
View File
@@ -0,0 +1,33 @@
"""Reproduce the separately switchable foreground and provisional contact shadow."""
from pathlib import Path
from PIL import Image, ImageDraw, ImageFilter
def main():
assets = Path(__file__).resolve().parents[1] / 'public/assets'
source = Image.open(assets / 'reference.png').convert('RGBA')
mask = Image.new('L', source.size)
ImageDraw.Draw(mask).polygon([
(1240, 582), (1282, 566), (1316, 556), (1360, 552), (1403, 532),
(1438, 526), (1486, 514), (1517, 521), (1539, 544), (1559, 556),
(1565, 634), (1520, 707), (1407, 747), (1260, 703), (1227, 650),
], fill=255)
mask = mask.filter(ImageFilter.GaussianBlur(1))
source.putalpha(mask)
source.save(assets / 'occlusion.png')
mask.save(assets / 'occlusion-mask.png')
mask = Image.new('L', source.size)
ImageDraw.Draw(mask).ellipse((1230, 511, 1512, 595), fill=65)
mask = mask.filter(ImageFilter.GaussianBlur(12))
shadow = Image.new('RGBA', source.size, (44, 28, 20, 0))
shadow.putalpha(mask)
shadow.save(assets / 'contact-shadow.png')
composite = Image.open(assets / 'environment.png').convert('RGBA')
for name in ['contact-shadow.png', 'person.png', 'occlusion.png']:
composite = Image.alpha_composite(composite, Image.open(assets / name))
composite.save(assets / 'composite.png')
if __name__ == '__main__':
main()
+79
View File
@@ -0,0 +1,79 @@
"""Use installed SAM modeling modules directly; no predictor/OpenCV dependency.
Run in the existing ComfyUI venv. No downloads or dependency changes.
"""
import argparse
import importlib
import sys
import types
from pathlib import Path
import numpy as np
import torch
from PIL import Image, ImageChops, ImageDraw, ImageFilter
def load_sam(source: Path, checkpoint: Path):
# The optional predictor package imports cv2. We only use the original model API.
package = types.ModuleType("hel711_sam")
package.__path__ = [str(source)]
sys.modules[package.__name__] = package
builder = importlib.import_module("hel711_sam.build_sam")
return builder.build_sam_vit_b(str(checkpoint)).eval()
def segment(model, image, extra_positive=()):
width, height = image.size
scale = 1024 / max(width, height)
size = (round(width * scale), round(height * scale))
tensor = torch.from_numpy(np.array(image.resize(size))).permute(2, 0, 1).to(model.device)
# Crop coordinate system: fixed right-hand seat, 552 x 552.
points = torch.tensor([[[305, 140], [300, 245], [200, 300], [307, 301],
[80, 100], [450, 400], [330, 450]]], dtype=torch.float)
if extra_positive:
extra = torch.tensor([extra_positive], dtype=torch.float)
points = torch.cat((points, extra), dim=1)
points = (points * scale).to(model.device)
labels = torch.tensor([[1, 1, 1, 1, 0, 0, 0]], device=model.device)
if extra_positive:
labels = torch.cat((labels, torch.ones((1, len(extra_positive)), device=model.device)), dim=1)
box = torch.tensor([[100, 60, 420, 380]], dtype=torch.float, device=model.device) * scale
with torch.inference_mode():
result = model([dict(image=tensor, original_size=(height, width),
point_coords=points, point_labels=labels, boxes=box)],
multimask_output=False)[0]
mask = result['masks'][0, 0].cpu().numpy().astype(np.uint8) * 255
mask = Image.fromarray(mask)
# The book/knit texture produces enclosed SAM holes. Fill enclosed holes only.
exterior = mask.copy()
ImageDraw.floodfill(exterior, (0, 0), 255)
mask = ImageChops.lighter(mask, ImageChops.invert(exterior))
return mask.filter(ImageFilter.GaussianBlur(0.65))
def main():
parser = argparse.ArgumentParser()
parser.add_argument('assets', type=Path)
parser.add_argument('--sam-source', type=Path, required=True)
parser.add_argument('--checkpoint', type=Path, required=True)
args = parser.parse_args()
torch.set_num_threads(8)
model = load_sam(args.sam_source, args.checkpoint)
source = Image.open(args.assets / 'reference.png').convert('RGB')
region = (1120, 200, 1672, 752)
crop = source.crop(region)
mask = segment(model, crop)
full_mask = Image.new('L', source.size)
full_mask.paste(mask, region[:2])
full_mask.save(args.assets / 'person-mask.png')
layer = source.convert('RGBA')
layer.putalpha(full_mask)
layer.save(args.assets / 'person.png')
crop.save(args.assets / 'generation-input.png')
environment = Image.open(args.assets / 'environment.png').convert('RGBA')
Image.alpha_composite(environment, layer).save(args.assets / 'composite.png')
print('Saved original-pixel person layer, SAM mask, crop and real composite.', flush=True)
if __name__ == '__main__':
main()
+141
View File
@@ -0,0 +1,141 @@
"""One foreground job using the installed ComfyUI native Wan nodes.
Owns and stops its exact server PID in finally. Does not modify the installation.
"""
import json
import os
import subprocess
import time
import urllib.error
import urllib.request
from pathlib import Path
ROOT = Path('/media/lucian/Downloads/ComfyUI')
TASK = ROOT / 'output/hel711-task'
URL = 'http://127.0.0.1:18189'
def request(path, payload=None):
data = None if payload is None else json.dumps(payload).encode()
req = urllib.request.Request(URL + path, data=data,
headers={'Content-Type': 'application/json'})
with urllib.request.urlopen(req, timeout=20) as response:
return json.load(response)
def graph():
nodes = {}
def add(key, kind, **inputs):
nodes[str(key)] = dict(class_type=kind, inputs=inputs)
for key, mode in [(1, 'high'), (2, 'low')]:
add(key, 'UNETLoader', unet_name=f'wan2.2_i2v_{mode}_noise_14B_fp8_scaled.safetensors',
weight_dtype='default')
add(key + 2, 'LoraLoaderModelOnly', model=[str(key), 0],
lora_name=f'wan2.2_i2v_lightx2v_4steps_lora_v1_{mode}_noise.safetensors',
strength_model=1.0)
add(key + 4, 'ModelSamplingSD3', model=[str(key + 2), 0], shift=5.0)
add(7, 'CLIPLoader', clip_name='umt5_xxl_fp8_e4m3fn_scaled.safetensors',
type='wan', device='cpu')
add(8, 'VAELoader', vae_name='wan_2.1_vae.safetensors')
add(9, 'LoadImage', image='generation-input.png')
add(10, 'CLIPTextEncode', clip=['7', 0], text=(
'Locked off camera, photorealistic young adult woman with short dark hair, '
'cream cardigan and dark trousers seated on the same sofa in the same bedroom. '
'She quietly reads the book in her hands, slowly lowers her gaze to the page '
'then gently raises her head a little. Small natural eye blink and breathing. '
'Her hands keep holding the book. Fixed seat position, stationary room and furniture.'))
add(11, 'CLIPTextEncode', clip=['7', 0],
text='camera movement, zoom, changing face, changing clothes, standing up, '
'extra fingers, distorted hands, deformed face, moving furniture, subtitles')
add(12, 'WanImageToVideo', positive=['10', 0], negative=['11', 0], vae=['8', 0],
width=512, height=512, length=49, batch_size=1, start_image=['9', 0])
add(13, 'KSamplerAdvanced', model=['5', 0], add_noise='enable', noise_seed=711,
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
positive=['12', 0], negative=['12', 1], latent_image=['12', 2],
start_at_step=0, end_at_step=2, return_with_leftover_noise='enable')
add(14, 'KSamplerAdvanced', model=['6', 0], add_noise='disable', noise_seed=711,
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
positive=['12', 0], negative=['12', 1], latent_image=['13', 0],
start_at_step=2, end_at_step=4, return_with_leftover_noise='disable')
add(15, 'VAEDecode', samples=['14', 0], vae=['8', 0])
add(16, 'SaveWEBM', images=['15', 0], codec='vp9', fps=16.0,
filename_prefix='reading', crf=20.0)
return nodes
def main():
for name in ['user', 'temp']:
(TASK / name).mkdir(exist_ok=True)
os.environ['HF_HUB_OFFLINE'] = '1'
os.environ['TRANSFORMERS_OFFLINE'] = '1'
command = [str(ROOT / 'venv/bin/python'), str(ROOT / 'main.py'),
'--listen', '127.0.0.1', '--port', '18189', '--disable-all-custom-nodes',
'--offline', '--disable-manager-ui', '--disable-assets',
'--reserve-vram', '8', '--disable-dynamic-vram',
'--input-directory', str(TASK / 'assets'),
'--output-directory', str(TASK), '--user-directory', str(TASK / 'user'),
'--temp-directory', str(TASK / 'temp'), '--database-url', 'sqlite:///:memory:']
# Refuse to attach to an unrelated listener.
import socket
with socket.socket() as sock:
if sock.connect_ex(('127.0.0.1', 18189)) == 0:
raise RuntimeError('Task port already occupied; no process started')
with open(TASK / 'server.log', 'w') as log:
server = subprocess.Popen(command, cwd=ROOT, stdout=log, stderr=subprocess.STDOUT)
(TASK / 'process.json').write_text(json.dumps(dict(pid=server.pid, command=command)))
try:
deadline = time.monotonic() + 90
while True:
if server.poll() is not None:
raise RuntimeError(f'ComfyUI exited: {server.returncode}; see server.log')
try:
info = request('/object_info')
break
except (OSError, urllib.error.URLError):
if time.monotonic() > deadline:
raise TimeoutError('Startup exceeded 90 seconds') from None
time.sleep(1)
workflow = graph()
(TASK / 'workflow.json').write_text(json.dumps(workflow, indent=2))
required = {n['class_type']: info.get(n['class_type']) for n in workflow.values()}
(TASK / 'nodes.json').write_text(json.dumps(required, indent=2))
if any(v is None for v in required.values()):
raise RuntimeError('Required native node unavailable; see nodes.json')
result = request('/prompt', {'prompt': workflow})
(TASK / 'submission.json').write_text(json.dumps(result))
prompt_id = result['prompt_id']
print(f'Generating one clip: {prompt_id}; owned server PID {server.pid}', flush=True)
deadline = time.monotonic() + 2400
last_report = 0
while time.monotonic() < deadline:
history = request('/history/' + prompt_id)
if prompt_id in history:
(TASK / 'history.json').write_text(json.dumps(history, indent=2))
print(json.dumps(history[prompt_id]['status']), flush=True)
if history[prompt_id]['status']['status_str'] != 'success':
raise RuntimeError('Generation failed; see history.json')
break
if time.monotonic() - last_report > 30:
print('Single generation still active; existing services untouched.', flush=True)
last_report = time.monotonic()
time.sleep(3)
else:
request('/interrupt', {})
raise TimeoutError('Single generation exceeded 40 minutes')
except urllib.error.HTTPError as error:
(TASK / 'api-error.json').write_bytes(error.read())
raise
finally:
server.terminate()
try:
server.wait(timeout=20)
except subprocess.TimeoutExpired:
server.kill()
server.wait()
print(f'Stopped owned server PID {server.pid}', flush=True)
if __name__ == '__main__':
main()
+149
View File
@@ -0,0 +1,149 @@
"""Generate one conditioned return, from original frame 48 back to frame 0.
Owns and stops its exact server PID in finally. Does not modify the installation.
"""
import json
import os
import subprocess
import time
import urllib.error
import urllib.request
from pathlib import Path
ROOT = Path('/media/lucian/Downloads/ComfyUI')
TASK = ROOT / 'output/hel712-task'
URL = 'http://127.0.0.1:18189'
def request(path, payload=None):
data = None if payload is None else json.dumps(payload).encode()
req = urllib.request.Request(URL + path, data=data,
headers={'Content-Type': 'application/json'})
with urllib.request.urlopen(req, timeout=20) as response:
return json.load(response)
def graph():
nodes = {}
def add(key, kind, **inputs):
nodes[str(key)] = dict(class_type=kind, inputs=inputs)
for key, mode in [(1, 'high'), (2, 'low')]:
add(key, 'UNETLoader', unet_name=f'wan2.2_i2v_{mode}_noise_14B_fp8_scaled.safetensors',
weight_dtype='default')
add(key + 2, 'LoraLoaderModelOnly', model=[str(key), 0],
lora_name=f'wan2.2_i2v_lightx2v_4steps_lora_v1_{mode}_noise.safetensors',
strength_model=1.0)
add(key + 4, 'ModelSamplingSD3', model=[str(key + 2), 0], shift=5.0)
add(7, 'CLIPLoader', clip_name='umt5_xxl_fp8_e4m3fn_scaled.safetensors',
type='wan', device='cpu')
add(8, 'VAELoader', vae_name='wan_2.1_vae.safetensors')
add(9, 'LoadImage', image='return-start.png')
add(17, 'LoadImage', image='return-end.png')
add(10, 'CLIPTextEncode', clip=['7', 0], text=(
'Locked off camera, photorealistic young adult woman with short dark hair, '
'cream cardigan and dark trousers seated on the same sofa in the same bedroom. '
'She slowly and smoothly raises her head from looking down at the book, '
'returning to the exact relaxed head pose and eye direction in the final reference. '
'Her hands keep holding the open book without moving it. Natural continuous breathing. '
'Fixed seat position, stationary room and furniture, identical face and outfit.'))
add(11, 'CLIPTextEncode', clip=['7', 0],
text='camera movement, zoom, changing face, changing clothes, standing up, '
'extra fingers, distorted hands, deformed face, moving furniture, subtitles')
add(12, 'WanFirstLastFrameToVideo', positive=['10', 0], negative=['11', 0], vae=['8', 0],
width=512, height=512, length=49, batch_size=1, start_image=['9', 0], end_image=['17', 0])
add(13, 'KSamplerAdvanced', model=['5', 0], add_noise='enable', noise_seed=712,
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
positive=['12', 0], negative=['12', 1], latent_image=['12', 2],
start_at_step=0, end_at_step=2, return_with_leftover_noise='enable')
add(14, 'KSamplerAdvanced', model=['6', 0], add_noise='disable', noise_seed=712,
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
positive=['12', 0], negative=['12', 1], latent_image=['13', 0],
start_at_step=2, end_at_step=4, return_with_leftover_noise='disable')
add(15, 'VAEDecode', samples=['14', 0], vae=['8', 0])
add(16, 'SaveWEBM', images=['15', 0], codec='vp9', fps=16.0,
filename_prefix='return', crf=20.0)
return nodes
def main():
import av
(TASK / 'assets').mkdir(parents=True, exist_ok=True)
with av.open(str(ROOT / 'output/hel711-task/reading_00001_.webm')) as source:
frames = [frame.to_image() for frame in source.decode(video=0)]
frames[-1].save(TASK / 'assets/return-start.png')
frames[0].save(TASK / 'assets/return-end.png')
for name in ['user', 'temp']:
(TASK / name).mkdir(exist_ok=True)
os.environ['HF_HUB_OFFLINE'] = '1'
os.environ['TRANSFORMERS_OFFLINE'] = '1'
command = [str(ROOT / 'venv/bin/python'), str(ROOT / 'main.py'),
'--listen', '127.0.0.1', '--port', '18189', '--disable-all-custom-nodes',
'--offline', '--disable-manager-ui', '--disable-assets',
'--reserve-vram', '8', '--disable-dynamic-vram',
'--input-directory', str(TASK / 'assets'),
'--output-directory', str(TASK), '--user-directory', str(TASK / 'user'),
'--temp-directory', str(TASK / 'temp'), '--database-url', 'sqlite:///:memory:']
# Refuse to attach to an unrelated listener.
import socket
with socket.socket() as sock:
if sock.connect_ex(('127.0.0.1', 18189)) == 0:
raise RuntimeError('Task port already occupied; no process started')
with open(TASK / 'server.log', 'w') as log:
server = subprocess.Popen(command, cwd=ROOT, stdout=log, stderr=subprocess.STDOUT)
(TASK / 'process.json').write_text(json.dumps(dict(pid=server.pid, command=command)))
try:
deadline = time.monotonic() + 90
while True:
if server.poll() is not None:
raise RuntimeError(f'ComfyUI exited: {server.returncode}; see server.log')
try:
info = request('/object_info')
break
except (OSError, urllib.error.URLError):
if time.monotonic() > deadline:
raise TimeoutError('Startup exceeded 90 seconds') from None
time.sleep(1)
workflow = graph()
(TASK / 'workflow.json').write_text(json.dumps(workflow, indent=2))
required = {n['class_type']: info.get(n['class_type']) for n in workflow.values()}
(TASK / 'nodes.json').write_text(json.dumps(required, indent=2))
if any(v is None for v in required.values()):
raise RuntimeError('Required native node unavailable; see nodes.json')
result = request('/prompt', {'prompt': workflow})
(TASK / 'submission.json').write_text(json.dumps(result))
prompt_id = result['prompt_id']
print(f'Generating one clip: {prompt_id}; owned server PID {server.pid}', flush=True)
deadline = time.monotonic() + 2400
last_report = 0
while time.monotonic() < deadline:
history = request('/history/' + prompt_id)
if prompt_id in history:
(TASK / 'history.json').write_text(json.dumps(history, indent=2))
print(json.dumps(history[prompt_id]['status']), flush=True)
if history[prompt_id]['status']['status_str'] != 'success':
raise RuntimeError('Generation failed; see history.json')
break
if time.monotonic() - last_report > 30:
print('Single generation still active; existing services untouched.', flush=True)
last_report = time.monotonic()
time.sleep(3)
else:
request('/interrupt', {})
raise TimeoutError('Single generation exceeded 40 minutes')
except urllib.error.HTTPError as error:
(TASK / 'api-error.json').write_bytes(error.read())
raise
finally:
server.terminate()
try:
server.wait(timeout=20)
except subprocess.TimeoutExpired:
server.kill()
server.wait()
print(f'Stopped owned server PID {server.pid}', flush=True)
if __name__ == '__main__':
main()
+59
View File
@@ -0,0 +1,59 @@
"""Per-frame real SAM segmentation, then VP9 alpha encoding with existing PyAV."""
import json
from pathlib import Path
import av
import numpy as np
import torch
from extract_layers import load_sam, segment
from PIL import Image
ROOT = Path('/media/lucian/Downloads/ComfyUI')
TASK = ROOT / 'output/hel711-task'
def main():
torch.set_num_threads(8)
model = load_sam(
ROOT / 'custom_nodes/was-node-suite-comfyui/repos/SAM/segment_anything',
ROOT / 'models/sams/sam_vit_b_01ec64.pth').to('cuda')
output = av.open(str(TASK / 'reading-alpha.webm'), 'w')
stream = output.add_stream('libvpx-vp9', rate=16)
stream.width = 552
stream.height = 552
stream.pix_fmt = 'yuva420p'
stream.options = {'crf': '20', 'b': '0', 'auto-alt-ref': '0'}
metrics = []
previews = []
source = av.open(str(TASK / 'reading_00001_.webm'))
previous = None
first = None
for index, decoded in enumerate(source.decode(video=0)):
image = decoded.to_image().resize((552, 552), Image.Resampling.LANCZOS)
mask = segment(model, image)
frame = image.convert('RGBA')
frame.putalpha(mask)
pixels = np.array(image).astype(float)
if first is None:
first = pixels
metrics.append(dict(frame=index, mask_pixels=int((np.array(mask) > 127).sum()),
mean_absolute_rgb_change=0 if previous is None else float(np.abs(pixels-previous).mean())))
previous = pixels
if index in [0, 16, 32, 48]:
previews.append(frame)
frame.save(TASK / f'matted-{index:02}.png')
for packet in stream.encode(av.VideoFrame.from_ndarray(np.array(frame), format='rgba')):
output.mux(packet)
if index % 16 == 0:
print(f'Matted frame {index}', flush=True)
for packet in stream.encode():
output.mux(packet)
output.close()
source.close()
(TASK / 'frame-metrics.json').write_text(json.dumps(dict(
frames=metrics, loop_rgb_difference=float(np.abs(previous-first).mean()),
interpretation='Real generated frames; RGB differences alone are not proof of valid anatomy or identity.'), indent=2))
if __name__ == '__main__':
main()
+106
View File
@@ -0,0 +1,106 @@
"""Measure all joins and before/after hand mattes; export reviewable frame evidence."""
import json
from pathlib import Path
import av
import numpy as np
from PIL import Image, ImageDraw
from scipy.ndimage import binary_erosion, map_coordinates
from skimage.color import rgb2gray
from skimage.registration import optical_flow_tvl1
TASK = Path('/media/lucian/Downloads/ComfyUI/output/hel712-task')
def summary(values):
a = np.array(values)
return dict(min=float(a.min()), median=float(np.median(a)),
p95=float(np.percentile(a, 95)), max=float(a.max()))
def panel(images, labels, path, size=(256, 256), columns=6):
width, height = size
board = Image.new('RGB', (columns*width, ((len(images)+columns-1)//columns)*(height+22)), '#202020')
draw = ImageDraw.Draw(board)
for i, (image, label) in enumerate(zip(images, labels, strict=True)):
x, y = i % columns*width, i // columns*(height+22)
board.paste(image.resize(size), (x, y))
draw.text((x+3, y+height+3), label, fill='white')
board.save(path)
def main():
data = np.load(TASK/'segmentation-v2.npz')
mattes = np.load(TASK/'mattes.npz')
rgb = data['images']
# All measurement uses 512px source coordinates. The head box is fixed for both clips.
images = [np.array(Image.fromarray(x).resize((512,512))) for x in rgb]
before, after = [[np.array(Image.fromarray(x).resize((512,512))) > 127
for x in mattes[key]] for key in ['before','after']]
head = [x[50:180,230:340].astype(float) for x in images]
pairs = [(i, i+1) for i in range(97)] + [(97,0)]
head_diff = [float(np.abs(head[a]-head[b]).mean()) for a,b in pairs]
ordinary = [d for i,d in enumerate(head_diff) if i not in [48,97]]
report = {'coordinates':'512x512, head x230:340 y50:180, book/hand all x y175:330',
'original_head_adjacent':summary(head_diff[:48]),
'original_head_seam':float(np.abs(head[48]-head[0]).mean()),
'loop_ordinary_head':summary(ordinary),
'joins':[{'pair':[48,49],'head_mae':head_diff[48]},
{'pair':[97,0],'head_mae':head_diff[97]}],
'head_mae_all_pairs':head_diff}
for name, masks in [('before',before),('after',after)]:
edge = [int((m & ~binary_erosion(m))[175:330].sum()) for m in masks]
switches = [int((masks[a]^masks[b])[175:330].sum()) for a,b in pairs]
report[name] = {'boundary_pixels': edge, 'switch_pixels':switches,
'original49_boundary':summary(edge[:49]),
'original49_boundary_jump':summary(np.abs(np.diff(edge[:49]))),
'original49_switch':summary(switches[:48]),
'all_boundary':summary(edge), 'all_boundary_jump':summary(np.abs(np.diff(edge + [edge[0]]))),
'all_switch':summary(switches),
'joins_switch':[switches[48],switches[97]]}
# Warp both alpha candidates with the same image motion. Residuals separate
# mask switching from unaligned silhouette motion (not an anatomy metric).
compensated = {'before':[], 'after':[]}
rows, cols = np.meshgrid(np.arange(155), np.arange(512), indexing='ij')
for a,b in pairs:
flow = optical_flow_tvl1(rgb2gray(images[a][175:330]).astype(np.float32),
rgb2gray(images[b][175:330]).astype(np.float32),
num_warp=3, num_iter=5)
for name,masks in [('before',before),('after',after)]:
warped = map_coordinates(masks[b][175:330].astype(float),
[rows+flow[0],cols+flow[1]], order=1, mode='nearest')>0.5
compensated[name].append(int((masks[a][175:330]^warped).sum()))
report['flow_compensated_switch']={k:{'all':summary(v), 'original49':summary(v[:48]),'per_pair':v}
for k,v in compensated.items()}
# Raw interior stays paired with alpha output; encoding error is reported separately.
with av.open(str(TASK/'reading-alpha.webm')) as src:
decoder=av.CodecContext.create('libvpx-vp9','r')
decoded=[f.to_ndarray(format='rgba') for p in src.demux(video=0) for f in decoder.decode(p)]
report['alpha_rgb_interior_mae']=summary([
float(np.abs(out[:,:,:3].astype(float)-im).mean(axis=2)[m>250].mean())
for out,im,m in zip(decoded,rgb,mattes['after'],strict=True)])
(TASK/'metrics.json').write_text(json.dumps(report,indent=2))
print(json.dumps({k:v for k,v in report.items() if k not in ['before','after','head_mae_all_pairs','flow_compensated_switch']},indent=2))
print(json.dumps({k:{x:y for x,y in v.items() if not isinstance(y,list)} for k,v in report.items() if k in ['before','after']},indent=2))
ids=[46,47,48,49,50,51,94,95,96,97,0,1]
panel([Image.fromarray(images[i]) for i in ids], [f'frame {i}' for i in ids],TASK/'join-frames.png')
panel([Image.fromarray(images[i][50:180,230:340]) for i in ids], [f'head {i}' for i in ids],TASK/'join-heads.png',size=(220,260))
ids=[0,8,16,24,32,40,47,48,49,56,64,72,80,88,96,97]
crops=[]
labels=[]
for i in ids:
for name,masks in [('before',mattes['before']),('after',mattes['after'])]:
rgba=np.dstack((rgb[i],masks[i]))
bg=Image.new('RGBA',(552,552),'#557788')
bg.alpha_composite(Image.fromarray(rgba))
crops.append(bg.convert('RGB').crop((90,189,430,356)))
labels.append(f'{name} {i}')
panel(crops,labels,TASK/'hand-before-after.png',size=(340,167),columns=4)
panel([Image.fromarray(mattes['after'][i]).convert('RGB').crop((90,189,430,356)) for i in ids],
[f'alpha {i}' for i in ids],TASK/'hand-mattes.png',size=(340,167),columns=4)
panel([Image.fromarray(images[i]) for i in range(98)], [str(i) for i in range(98)],
TASK/'all-frames.png',size=(170,170),columns=14)
if __name__=='__main__':
main()
+146
View File
@@ -0,0 +1,146 @@
"""Assemble the generated forward/return clips and stabilize only the hand/book matte.
Run with the existing ComfyUI environment; no models or dependencies are fetched.
Optical flow aligns neighbouring masks; only alpha changes, never source RGB.
"""
import json
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import av
import numpy as np
import torch
from extract_layers import load_sam, segment
from PIL import Image
from scipy.ndimage import distance_transform_edt, gaussian_filter, label, map_coordinates
from skimage.color import rgb2gray
from skimage.registration import optical_flow_tvl1
ROOT = Path('/media/lucian/Downloads/ComfyUI')
TASK = ROOT / 'output/hel712-task'
BASE = ROOT / 'output/hel711-task'
Y0, Y1 = round(175 * 552 / 512), round(330 * 552 / 512)
def decode_alpha(path):
with av.open(str(path)) as source:
decoder = av.CodecContext.create('libvpx-vp9', 'r')
return [f.to_ndarray(format='rgba') for p in source.demux(video=0)
for f in decoder.decode(p)]
def encode(path, frames, alpha):
with av.open(str(path), 'w') as output:
stream = output.add_stream('libvpx-vp9', rate=16)
stream.width = stream.height = frames[0].shape[0]
stream.pix_fmt = 'yuva420p' if alpha else 'yuv420p'
stream.options = {'crf': '10', 'b': '0', 'auto-alt-ref': '0'}
for frame in frames:
for packet in stream.encode(av.VideoFrame.from_ndarray(
frame, format='rgba' if alpha else 'rgb24')):
output.mux(packet)
for packet in stream.encode():
output.mux(packet)
def signed(mask):
binary = mask > 127
return distance_transform_edt(binary) - distance_transform_edt(~binary)
def stabilize(images, masks):
# Padding supplies flow context, but edits are restricted to the specified band.
top, bottom = Y0 - 24, Y1 + 24
gray = [rgb2gray(im[top:bottom]).astype(np.float32) for im in images]
distances = [signed(m[top:bottom]) for m in masks]
rows, cols = np.meshgrid(np.arange(bottom-top), np.arange(552), indexing='ij')
def one(i):
mask = masks[i]
aligned = [distances[i]]
for offset in [-2, -1, 1, 2]:
j = (i + offset) % len(masks)
flow = optical_flow_tvl1(gray[i], gray[j], attachment=15,
num_warp=5, num_iter=10)
aligned.append(map_coordinates(distances[j], [rows+flow[0], cols+flow[1]],
order=1, mode='nearest'))
# Trim the most extreme masks; averaging the remaining three signed
# distances also avoids a majority-vote jump at the two clip joins.
consensus = np.sort(aligned, axis=0)[1:-1].mean(axis=0)
# A narrow spatial regularization removes single-pixel segmentation teeth.
stable = gaussian_filter(consensus, 0.65)
current = distances[i]
stable = np.where(np.abs(current) <= 6, stable, current)
alpha = np.clip(127.5 + stable * 127.5, 0, 255)
# Blend the first/last 8 rows back to the unchanged matte.
weight = np.ones(Y1-Y0)
weight[:8] = np.linspace(0, 1, 8)
weight[-8:] = np.linspace(1, 0, 8)
new = mask.copy()
roi = alpha[Y0-top:Y1-top]
new[Y0:Y1] = np.rint(roi*weight[:, None] + mask[Y0:Y1]*(1-weight[:, None])).astype(np.uint8)
return new, int(np.count_nonzero((new>127) != (mask>127)))
output, changes = [], []
with ThreadPoolExecutor(max_workers=4) as executor:
for i, (mask, change) in enumerate(executor.map(one, range(len(masks)))):
output.append(mask)
changes.append(change)
if i % 8 == 0:
print(f'Flow-aligned matte {i}/{len(masks)}', flush=True)
return output, changes
def main():
torch.set_num_threads(8)
cache = TASK / 'segmentation-v2.npz'
if cache.exists():
data = np.load(cache)
images, masks = list(data['images']), list(data['masks'])
else:
# Preserve original matte outside the hand/book band without rerunning it.
original = decode_alpha(BASE / 'reading-alpha.webm')
with av.open(str(BASE / 'reading_00001_.webm')) as source:
images = [np.array(f.to_image().resize((552, 552), Image.Resampling.LANCZOS))
for f in source.decode(video=0)]
masks = [f[:, :, 3] for f in original]
model = load_sam(ROOT / 'custom_nodes/was-node-suite-comfyui/repos/SAM/segment_anything',
ROOT / 'models/sams/sam_vit_b_01ec64.pth').to('cuda')
with av.open(str(TASK / 'return_00001_.webm')) as source:
for i, frame in enumerate(source.decode(video=0)):
image = frame.to_image().resize((552, 552), Image.Resampling.LANCZOS)
images.append(np.array(image))
alpha = np.array(segment(model, image, extra_positive=(
(245, 258), (280, 275), (360, 245), (365, 295))))
components, count = label(alpha > 127)
sizes = np.bincount(components.ravel())
sizes[0] = 0
keep = components == sizes.argmax()
# Retain feather next to the connected subject; discard isolated sofa specks.
from scipy.ndimage import binary_dilation
alpha[~binary_dilation(keep, iterations=3)] = 0
masks.append(alpha)
if i % 8 == 0:
print(f'Segmented return frame {i}', flush=True)
np.savez_compressed(cache, images=np.array(images), masks=np.array(masks))
del model
torch.cuda.empty_cache()
stable, changes = stabilize(images, masks)
np.savez_compressed(TASK / 'mattes.npz', before=np.array(masks), after=np.array(stable))
encode(TASK / 'loop-source.webm', images, False)
frames = [np.dstack((rgb, a)) for rgb, a in zip(images, stable, strict=True)]
encode(TASK / 'reading-alpha.webm', frames, True)
(TASK / 'stabilization.json').write_text(json.dumps({
'frames': len(images), 'fps': 16, 'join_pairs': [[48,49],[97,0]],
'source_map': [{'frames':[0,48], 'source':'reading_00001_.webm'},
{'frames':[49,97], 'source':'return_00001_.webm'}],
'band_512': [175,330], 'band_552': [Y0,Y1],
'flow': 'TV-L1, four neighbours, cyclic trimmed signed-distance mean, 6px uncertainty band',
'changed_binary_pixels_per_frame': changes,
'rgb': 'Original RGB retained; no temporal RGB blending or frame replacement',
'outside_band_unchanged': all(np.array_equal(a[:Y0],b[:Y0]) and np.array_equal(a[Y1:],b[Y1:])
for a,b in zip(masks,stable,strict=True))}, indent=2))
print('Encoded 98-frame forward/return candidate, not yet accepted.', flush=True)
if __name__ == '__main__':
main()
+20
View File
@@ -0,0 +1,20 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "Bundler",
"lib": [
"ES2022",
"DOM",
"DOM.Iterable"
],
"strict": true,
"noEmit": true,
"skipLibCheck": true,
"noUnusedLocals": true,
"noUnusedParameters": true
},
"include": [
"src"
]
}