Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dab536987a | ||
|
|
444ee56c6a |
@@ -0,0 +1,7 @@
|
||||
node_modules/
|
||||
dist/
|
||||
public/assets/*
|
||||
!public/assets/manifest.json
|
||||
evidence/*
|
||||
!evidence/.gitkeep
|
||||
__pycache__/
|
||||
@@ -0,0 +1,55 @@
|
||||
HEL-711 最小动态校对小样,基于 main `1e9b08a`。仅在 `samples/bedroom` 内工作;不连接人格、记忆、天气或语音。采用 HEL-708 的环境/人物/交谈分层边界,以及 CG-001~026 的当前需求。旧演示不变。
|
||||
|
||||
## 本地复核
|
||||
|
||||
1. 将本卡附件 `HEL-711-review-bundle.zip` 中的 `public/assets` 解压到此目录对应位置。
|
||||
2. `npm ci --ignore-scripts --no-audit --no-fund`,沿用旧前端分支的 package.json/package-lock.json/tsconfig.json,无新依赖。
|
||||
3. `npm run build`(包含严格 TypeScript 检查)。
|
||||
4. `npm run dev -- --port 5173`,本机浏览器打开终端输出的地址。
|
||||
5. `python3 tools/browser_check.py` 使用已安装的 Playwright/Chromium,自动启停本机测试服务并保存证据。此运行机需要给该命令设置 `LD_LIBRARY_PATH=/home/xiaomo/pwlibs/root/usr/lib/x86_64-linux-gnu`,复用了已有浏览器库,未安装系统依赖。
|
||||
|
||||
素材不进 Git,缺失时页面明确提示;manifest 在 Git 中保存来源、原图 SHA256、坐标、编码和问题。附件内含完整 assets、已构建 dist、浏览器连续录屏、原始试片、透明试片、分层校对图及生成工作流/日志。
|
||||
|
||||
## 代码与边界
|
||||
|
||||
- `src/main.ts`:一个持久 video 元素驱动透明 canvas;环境按钮只改环境亮度,不重建/seek 视频。人物、前景遮挡、接触阴影独立开关;静态近景不替代真实交谈。
|
||||
- `src/tokens.css`:原样复用旧前端唯一 token 表。`style.css` 为独立工程校对台,无第二套同类组件。
|
||||
- `tools/extract_layers.py`:读取现有 SAM vit-b 模型的原始 modeling API,避免可选 predictor 的 OpenCV 依赖;不伪造 cv2,不下载。原图 RGB 保留,只生成 alpha;封闭孔洞填充、0.65px 羽化。
|
||||
- `tools/contact_layers.py`:从原入景图提取毯子前景,另建可关闭的暂定接触阴影;未宣称物理阴影校准完成。
|
||||
- `tools/generate_once.py`:原 ComfyUI venv + 原生 Wan 节点,单片段、离线、自定义节点禁用。临时服务器仅监听回环 18189,finally 终止自己启动的精确 PID。不修改现有服务、系统 Python、驱动或模型路径配置。
|
||||
- `tools/matte_video.py`:49 帧逐帧 SAM 分割,原有 PyAV 编码 VP9 alpha。原始视频未抠像的背景不用于浏览器人物合成。
|
||||
- `tools/browser_check.py`:真实 Chromium 检查、三种视口截图和可下载连续录屏。页面隐藏的事件处理做了合成 visibilitychange 检查;真实系统切后台/锁屏仍待体验环境手测。
|
||||
|
||||
## 素材处理命令
|
||||
|
||||
处理目录:`.12:/media/lucian/Downloads/ComfyUI/output/hel711-task`,输入复制到其 `assets`,脚本放此目录;没有改变现有项目输入。原图由 Multica 鉴权 attachment download 获取。运行时使用原 `/media/lucian/Downloads/ComfyUI/venv/bin/python`。
|
||||
|
||||
```bash
|
||||
python extract_layers.py assets \
|
||||
--sam-source /media/lucian/Downloads/ComfyUI/custom_nodes/was-node-suite-comfyui/repos/SAM/segment_anything \
|
||||
--checkpoint /media/lucian/Downloads/ComfyUI/models/sams/sam_vit_b_01ec64.pth
|
||||
python generate_once.py
|
||||
python matte_video.py
|
||||
```
|
||||
|
||||
回传处理后的素材到本地 `public/assets`,运行 `python3 tools/contact_layers.py`。手工蒙版和坐标均保留在脚本中。`workflow.json` 是本次实际运行的请求,最终脚本仅补了目录创建/日志关闭等清理逻辑;日志和 history 可交叉核对。执行新试片前必须重新检查显存和端口;禁止停止已有服务腾资源。
|
||||
|
||||
## 当前实测与未通过项
|
||||
|
||||
当前结果见 `REPORT.md`,属于可审核候选,不代表 D1–D6 全部通过。图层是真实独立合成;没有用重生成整幅图代替分层。近景仍用已确认 v2 静态方向图;没有真人语音或口型。
|
||||
|
||||
## HEL-712 返工复现
|
||||
|
||||
本轮只补原动作的真实回程,并处理书本/手部蒙版;环境、机位、比例、UI token 与静态近景不变。完整返工包中的 `public/assets` 替换旧素材后执行同一构建/浏览器检查。
|
||||
|
||||
已有 ComfyUI 环境中,保留原片在 `output/hel711-task/reading_00001_.webm`、原透明片在同目录 `reading-alpha.webm`:
|
||||
|
||||
```bash
|
||||
python generate_return.py
|
||||
python stabilize_loop.py
|
||||
python measure_loop.py
|
||||
```
|
||||
|
||||
脚本放在 `output/hel712-task` 并附同目录 `extract_layers.py`。生成器只用已有原生 `WanFirstLastFrameToVideo` 节点与原有权重,以原片 f48、f0 为两端,seed 712,只生成一条49帧回程。服务器仍仅临时监听回环18189;主程序收齐结果后停止精确子进程。`stabilize_loop.py` 读取原49帧 alpha,新增49帧按书本/衣袖补充分割提示;只在原512坐标 y175–330 对动作对齐的5帧有符号距离去掉最大/最小值后取均值稳定,RGB不做时域融合,原发丝区域不重新分割。原始两段按顺序拼成98帧,接点48→49、97→0,无倒放/淡化/冻结/短窗口裁切。
|
||||
|
||||
`measure_loop.py` 输出固定坐标口径的逐对差异、蒙版统计、动作补偿残差、接头与手部证据图;数值不替代目验。`browser_check.py` 先在真实Chromium完整播放至少3周期,再做原交互与三视口复测。真实后台/锁屏及Safari/iOS仍未测。
|
||||
@@ -0,0 +1,52 @@
|
||||
HEL-712:原阅读动作循环与书本/手部蒙版返工候选,交总工复审,不部署。
|
||||
|
||||
## 实现
|
||||
|
||||
沿用原人物、米色开衫、深灰长裤、卧室、机位与552px人物落点。现有 Wan2.2 原生首尾帧条件节点,使用原片 f48→f0,seed 712,实际生成一条49帧抬头回程。原49帧向前播放,随后49帧真实回程,98帧/16fps/6.125秒,内部接点48→49,跨周期97→0。没有倒放、冻结、静态摆动、遮脸淡化或短窗口裁切。
|
||||
|
||||
原49帧 alpha 不重新分割;新增49帧补书本/衣袖正点,保留连通人物,去掉分离的沙发碎点。两段仅在原512px坐标 y175:330(输出552px坐标189:356)稳定蒙版:TV-L1对齐前后各2帧,有符号距离去掉最高/最低值,再取均值,限制到当前轮廓6px不确定区域,首尾8行回接原蒙版。只处理alpha,RGB不做时间平均;原发丝区域保持原alpha。原始源片与透明片逐帧映射见manifest/stabilization.json。
|
||||
|
||||
第一次新增回程直接套原分割提示时,会漏掉书本/衣袖;该版已拒绝,`before-crops.png` 保留实证。补充分割提示后才进入最终蒙版稳定。共生成一条视频,没有批量换装或动作试片。第一次稳定使用中值共识,接头alpha变化仍偏集中;最终使用去极值均值,缓和了两个分段蒙版的多数票切换。
|
||||
|
||||
## 同口径差异与边缘统计
|
||||
|
||||
本报告独立固定口径:RGB头部框为512px源坐标 x230:340、y50:180;蒙版缩到512px、alpha>127,边界用二值侵蚀差,书本/手部带为全部x、y175:330。与HEL-710采用的框/缩放细节不同,所以这里原接头22.30不直接与审核卡29.5混算;下列前后全部由同一脚本重算。完整逐对数据见 `evidence/metrics.json`,覆盖全部98个相邻对(包括末→首)。
|
||||
|
||||
| 头部RGB绝对差 /255 | 最小 | 中位 | P95 | 最大 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 原49帧普通相邻 | 0.36 | 1.73 | 4.97 | 5.75 |
|
||||
| 新98帧普通相邻(排除两接头) | 0.24 | 1.65 | 5.47 | 8.30 |
|
||||
|
||||
原f48→f0:22.30;新内部f48→49:3.66;新跨周期f97→0:5.22。两个接头落在普通相邻变化范围内。接头前后整帧、头部放大与98帧序列均附;连续录屏为实际浏览器播放,未用离线合成冒充。
|
||||
|
||||
| 蒙版指标 | 稳定前 | 最终稳定后 |
|
||||
|---|---:|---:|
|
||||
| 原49帧带内边界像素范围 | 510–794 | 509–604 |
|
||||
| 原49帧最差相邻边界变化 | 255 | 75 |
|
||||
| 原49帧相邻alpha开关最大像素 | 771 | 300 |
|
||||
| 全98帧边界像素范围 | 504–794 | 503–604 |
|
||||
| 全98对最差边界变化(含循环) | 261 | 75 |
|
||||
| 全98对alpha开关P95 / 最大像素 | 505.05 / 988 | 142.55 / 300 |
|
||||
| 内部 / 跨周期接头alpha开关像素 | 988 / 528 | 195 / 193 |
|
||||
| 动作补偿后alpha开关P95 / 最大 | 503.25 / 988 | 152.75 / 300 |
|
||||
|
||||
动作补偿使用同一源RGB光流对齐前后蒙版;可见减少的是分割开关,不靠抹掉真实运动。人物内部透明片与源RGB编码误差逐帧中位1.89、最大2.10/255;无RGB时间混合。带外alpha逐数组一致为true。重点帧可见书本/指尖与袖口保持,未见新增吞手或RGB拖影;仍需总工以连续视频独立确认。没有把数值降低当作人体解剖或身份认证。
|
||||
|
||||
## 验证
|
||||
|
||||
- 来源包SHA256:60/60核对一致。
|
||||
- `npm ci --ignore-scripts --no-audit --no-fund`:沿用锁文件,无新依赖。
|
||||
- `npm run build`:严格TypeScript检查和Vite构建通过;原JS/CSS逻辑保持,返工通过素材和工具落地。
|
||||
- `ruff check src tests samples/bedroom/tools`:通过。
|
||||
- `PYTHONPATH=src python3 -m pytest -q`:78通过。
|
||||
- `LD_LIBRARY_PATH=/home/xiaomo/pwlibs/root/usr/lib/x86_64-linux-gnu python3 tools/browser_check.py`:真实Chromium通过。最终视频先完整播放3周期,记录循环时间约6.28、12.42、18.56秒;此段全部图层显示,之后再执行交互测试。
|
||||
- VP9 alpha背景0/人物255;环境切换currentTime 0.905816→0.905816,source不变;图层开关、静态近景进退且播放继续、手动暂停、合成visibility暂停/恢复、减少动效切静态、缺片回退通过;1600×900、1280×800、390×844无横溢,人脸在画面内;pageerror=0。
|
||||
- 真实后台/锁屏、Safari/iOS未测;近景仅静态,未增加口型、语音或后台生活功能。手机构图、沙发几何、重打光与近景润色保持原记录。
|
||||
|
||||
## 文件与运行边界
|
||||
|
||||
修改 `tools/extract_layers.py`(可选补充提示)、新增 `tools/generate_return.py`、`stabilize_loop.py`、`measure_loop.py`,扩展 `tools/browser_check.py` 的连续三周期验证,更新manifest/README/本报告。没有改CSS、主题、核心服务或部署配置。
|
||||
|
||||
分支继续 `hel-711-layered-bedroom`,基于原444ee56;最终提交与PR head核对写入附件 `DELIVERY.json` 及任务完成评论。素材不进Git。完整包包含源码、构建产物、manifest、SHA256SUMS、原始前程/回程、整周期源片、透明片、逐帧前后alpha、生成请求/节点/history/server日志、浏览器连续录屏、三视口及接头/手部证据、复现说明。
|
||||
|
||||
严格校验了.12已确认SSH指纹。临时ComfyUI仅回环18189、精确PID600329,生成请求成功后已停止。CPU蒙版任务均前台收齐,没有遗留生成任务、没有新增依赖/模型/付费接口;ASR/TTS及18080原演示保持运行。未推main、未合并、未部署。
|
||||
@@ -0,0 +1,38 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>卧室分层校对 · HEL-711</title></head>
|
||||
<body>
|
||||
<main>
|
||||
<header><h1>卧室分层校对</h1><p>工程小样 · 手动光照演示 · 近景仅静态参考,未接语音或口型</p></header>
|
||||
<div id="stage" role="img" aria-label="固定机位卧室人物合成">
|
||||
<div id="frame">
|
||||
<img id="environment" src="./assets/environment.png" alt="无人高层卧室">
|
||||
<img id="shadow" src="./assets/contact-shadow.png" alt="" class="layer">
|
||||
<img id="person" src="./assets/person.png" alt="阅读人物透明层" class="layer">
|
||||
<canvas id="motion" width="552" height="552" hidden></canvas>
|
||||
<img id="occlusion" src="./assets/occlusion.png" alt="" class="layer">
|
||||
<img id="mask" src="./assets/person-mask.png" alt="人物蒙版" class="layer" hidden>
|
||||
<img id="talk" src="./assets/talk-reference.png" alt="交谈静态近景参考" class="layer" hidden>
|
||||
</div>
|
||||
</div>
|
||||
<video id="source" muted playsinline loop preload="metadata" hidden></video>
|
||||
<section aria-label="校对控制">
|
||||
<fieldset><legend>图层开关</legend>
|
||||
<label><input id="show-person" type="checkbox" checked>人物</label>
|
||||
<label><input id="show-shadow" type="checkbox" checked>接触阴影</label>
|
||||
<label><input id="show-occlusion" type="checkbox" checked>前景毯子</label>
|
||||
<label><input id="show-mask" type="checkbox">仅看人物蒙版</label>
|
||||
</fieldset>
|
||||
<div class="controls">
|
||||
<button id="lighting" type="button" aria-pressed="false">光照演示:原始</button>
|
||||
<button id="play" type="button" disabled>读取动作素材</button>
|
||||
<button id="closeup" type="button" aria-pressed="false">进入静态近景</button>
|
||||
</div>
|
||||
<p id="status" role="status">读取素材清单…</p>
|
||||
<output id="time">动作位置:未载入</output>
|
||||
<p id="fallback" role="alert" hidden></p>
|
||||
<p>人物、房间和近景来自已确认的 v2 参考。图层独立不等于几何、发丝与循环已经通过验收。</p>
|
||||
</section>
|
||||
</main>
|
||||
<script type="module" src="./src/main.ts"></script>
|
||||
</body></html>
|
||||
Generated
+1166
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"name": "cybergril-rain-room",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite --host 127.0.0.1",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"build": "tsc --noEmit && vite build",
|
||||
"preview": "vite preview --host 127.0.0.1"
|
||||
},
|
||||
"dependencies": {
|
||||
"pixi.js": "^8.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "~5.9.3",
|
||||
"vite": "^5.4.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
{
|
||||
"schema": "hel712-review-1",
|
||||
"status": "review_candidate",
|
||||
"reference_sha256": {
|
||||
"reference.png": "226d7922e042e22e4b8a8241a7cf6f7aea50c301029978e4ca26cad9b3d56359",
|
||||
"environment.png": "b5d4771eb010dc1d0a41d1b698e45edaa45d2f90bb441378bc32cf5c812e76a2",
|
||||
"talk-reference.png": "0ae9b1a1cb9c590ab89a81ce4efd25de843a600ff227c184663db23e9e9c16b3"
|
||||
},
|
||||
"camera": {
|
||||
"canvas": [
|
||||
1672,
|
||||
941
|
||||
],
|
||||
"generation_crop": [
|
||||
1120,
|
||||
200,
|
||||
552,
|
||||
552
|
||||
],
|
||||
"person_anchor": [
|
||||
1390,
|
||||
530
|
||||
],
|
||||
"coordinate_units": "source pixels, top left origin",
|
||||
"transform": "none; crop reinserted into original position",
|
||||
"clothing": "v2 cream cardigan, charcoal trousers"
|
||||
},
|
||||
"layers": [
|
||||
{
|
||||
"file": "environment.png",
|
||||
"source": "approved 02-v2 original",
|
||||
"alpha": false
|
||||
},
|
||||
{
|
||||
"file": "contact-shadow.png",
|
||||
"source": "engineered ellipse 65/255, 12px blur, not physically calibrated",
|
||||
"alpha": true
|
||||
},
|
||||
{
|
||||
"file": "person.png",
|
||||
"source": "original 03-v2 RGB + existing SAM vit-b alpha; enclosed mask holes filled",
|
||||
"alpha": true
|
||||
},
|
||||
{
|
||||
"file": "occlusion.png",
|
||||
"source": "03-v2 blanket pixels, hand-authored polygon matte",
|
||||
"alpha": true
|
||||
}
|
||||
],
|
||||
"motion": {
|
||||
"status": "真实回程与书本手部蒙版返工候选,待复审",
|
||||
"file": "reading-alpha.webm",
|
||||
"alpha": true,
|
||||
"maskFile": null,
|
||||
"original": "loop-source.webm",
|
||||
"generator": "installed ComfyUI native Wan2.2 high/low 14B fp8 + installed LightX2V 4-step LoRAs",
|
||||
"seed": 711,
|
||||
"dimensions": [
|
||||
552,
|
||||
552
|
||||
],
|
||||
"generation_dimensions": [
|
||||
512,
|
||||
512
|
||||
],
|
||||
"frames": 98,
|
||||
"fps": 16,
|
||||
"duration_seconds": 6.125,
|
||||
"codec": "VP9 yuva420p / WebM",
|
||||
"matting": "original 49 alpha retained outside hand/book band; added return frames use book/sleeve foreground points + connected subject; y175:330 (512 source) uses TV-L1 aligned 5-frame trimmed signed-distance mean within 6px uncertainty band",
|
||||
"start_pose": "seated holding open book, head slightly tilted toward camera",
|
||||
"end_pose": "generated return to the original relaxed camera-facing pose",
|
||||
"loop": "forward 0–48 + real generated return 49–97; joins 48→49 and 97→0, no reverse/fade/freeze",
|
||||
"cut_point": "no subwindow trimming",
|
||||
"return_seed": 712,
|
||||
"source_map": [
|
||||
{
|
||||
"frames": [
|
||||
0,
|
||||
48
|
||||
],
|
||||
"file": "reading_00001_.webm",
|
||||
"source_frames": [
|
||||
0,
|
||||
48
|
||||
]
|
||||
},
|
||||
{
|
||||
"frames": [
|
||||
49,
|
||||
97
|
||||
],
|
||||
"file": "return_00001_.webm",
|
||||
"source_frames": [
|
||||
0,
|
||||
48
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"closeup": {
|
||||
"file": "talk-reference.png",
|
||||
"type": "static_reference_only",
|
||||
"identity_verified": false,
|
||||
"lipsync": false,
|
||||
"source": "approved v2 static direction reference"
|
||||
},
|
||||
"known_issues": [
|
||||
"Original empty/occupied sofa and pillow geometry differs; residual contact geometry needs art review.",
|
||||
"Book and fingers are small; hand anatomy/identity not certified.",
|
||||
"Manual environment dimming changes background only; subject relighting not implemented.",
|
||||
"Static near-shot is a generated reference; cross-shot face continuity not certified.",
|
||||
"No real weather, speech, lipsync or life-state integration.",
|
||||
"Loop joins and stabilized hand/book edges are a HEL-712 candidate pending independent re-review."
|
||||
],
|
||||
"files": {
|
||||
"talk-reference.png": {
|
||||
"bytes": 2246098,
|
||||
"sha256": "0ae9b1a1cb9c590ab89a81ce4efd25de843a600ff227c184663db23e9e9c16b3"
|
||||
},
|
||||
"reading-alpha.webm": {
|
||||
"bytes": 1971817,
|
||||
"sha256": "9b9b95fff6c047c68ea66d14eba1ca896a80107b7c5b531c326bfac8bdc90b76"
|
||||
},
|
||||
"person.png": {
|
||||
"bytes": 2522355,
|
||||
"sha256": "a904f4081069ba97019d3c679d8dd59102118560471f8d6337f7b29d60380f24"
|
||||
},
|
||||
"composite.png": {
|
||||
"bytes": 2576324,
|
||||
"sha256": "4354716558858d0e9ac9a4c4a025d77f4042a0e106f10fc37e500c7157e77456"
|
||||
},
|
||||
"person-mask.png": {
|
||||
"bytes": 7264,
|
||||
"sha256": "7569160b3c87ebf1d5bb66792ed6d2db6b41fde006fad0473e7db5337d18b0f7"
|
||||
},
|
||||
"environment.png": {
|
||||
"bytes": 2442707,
|
||||
"sha256": "b5d4771eb010dc1d0a41d1b698e45edaa45d2f90bb441378bc32cf5c812e76a2"
|
||||
},
|
||||
"occlusion-mask.png": {
|
||||
"bytes": 4838,
|
||||
"sha256": "dc22d1823a02df4fa5f3bf484f157c49b270f100185c0a35b3c31f28880dd3f0"
|
||||
},
|
||||
"generation-input.png": {
|
||||
"bytes": 508812,
|
||||
"sha256": "f3922903fb28c5940a2e272b70be6e33dc094039b4e275724cf3ff1aa4af1584"
|
||||
},
|
||||
"occlusion.png": {
|
||||
"bytes": 2522991,
|
||||
"sha256": "5f0f4aa7ae90ad863f694365eaa93d7d7558841ecca08c95d415731450226c4f"
|
||||
},
|
||||
"reference.png": {
|
||||
"bytes": 2384917,
|
||||
"sha256": "226d7922e042e22e4b8a8241a7cf6f7aea50c301029978e4ca26cad9b3d56359"
|
||||
},
|
||||
"contact-shadow.png": {
|
||||
"bytes": 16144,
|
||||
"sha256": "948f88c1298638158f9a21ffe107bf403106dcf6e58c72a3d6e738fe0c7cc003"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import './style.css';
|
||||
|
||||
type Manifest = {
|
||||
motion: { status: string; file: string | null; alpha: boolean; maskFile: string | null };
|
||||
};
|
||||
const get = <T extends HTMLElement>(id: string) => document.getElementById(id) as T;
|
||||
const video = get<HTMLVideoElement>('source');
|
||||
const canvas = get<HTMLCanvasElement>('motion');
|
||||
const context = canvas.getContext('2d')!;
|
||||
const play = get<HTMLButtonElement>('play');
|
||||
const person = get<HTMLImageElement>('person');
|
||||
const status = get('status');
|
||||
const reduced = matchMedia('(prefers-reduced-motion: reduce)');
|
||||
const missingImages = new Set<string>();
|
||||
let ready = false;
|
||||
let wantsPlay = false;
|
||||
let near = false;
|
||||
let failed = false;
|
||||
let frameHandle = 0;
|
||||
|
||||
function fallback(message: string) {
|
||||
const node = get('fallback');
|
||||
node.hidden = false;
|
||||
node.textContent = message;
|
||||
}
|
||||
|
||||
for (const image of document.querySelectorAll<HTMLImageElement>('#frame img')) {
|
||||
image.addEventListener('error', () => {
|
||||
missingImages.add(image.id);
|
||||
image.hidden = true;
|
||||
fallback(`素材缺失:${image.id}。请解压任务素材包到 public/assets;当前内容不能作为验收结果。`);
|
||||
});
|
||||
}
|
||||
|
||||
function visibility() {
|
||||
const visible = get<HTMLInputElement>('show-person').checked;
|
||||
const maskOnly = get<HTMLInputElement>('show-mask').checked;
|
||||
const activeMotion = ready && !failed && !reduced.matches;
|
||||
person.hidden = !visible || maskOnly || near || activeMotion;
|
||||
canvas.hidden = !visible || maskOnly || near || !activeMotion;
|
||||
get('environment').hidden = maskOnly;
|
||||
get('mask').hidden = !maskOnly || near;
|
||||
get('shadow').hidden = !visible || maskOnly || near || !get<HTMLInputElement>('show-shadow').checked;
|
||||
get('occlusion').hidden = !visible || maskOnly || near || !get<HTMLInputElement>('show-occlusion').checked;
|
||||
get('talk').hidden = !near;
|
||||
for (const id of missingImages) get(id).hidden = true;
|
||||
}
|
||||
|
||||
function paint() {
|
||||
if (video.readyState >= 2) {
|
||||
context.clearRect(0, 0, canvas.width, canvas.height);
|
||||
context.globalCompositeOperation = 'source-over';
|
||||
context.drawImage(video, 0, 0, canvas.width, canvas.height);
|
||||
|
||||
}
|
||||
get('time').textContent = ready
|
||||
? `动作位置:${video.currentTime.toFixed(2)} / ${Number.isFinite(video.duration) ? video.duration.toFixed(2) : '…'} 秒`
|
||||
: '动作位置:未载入';
|
||||
}
|
||||
|
||||
function draw() {
|
||||
paint();
|
||||
if (!video.paused && !document.hidden) frameHandle = requestAnimationFrame(draw);
|
||||
}
|
||||
|
||||
async function syncPlayback() {
|
||||
cancelAnimationFrame(frameHandle);
|
||||
const run = wantsPlay && ready && !failed && !document.hidden && !reduced.matches;
|
||||
if (run) {
|
||||
try {
|
||||
await video.play();
|
||||
draw();
|
||||
} catch {
|
||||
wantsPlay = false;
|
||||
video.pause();
|
||||
fallback('浏览器未允许播放;请点击播放重试。');
|
||||
}
|
||||
} else { video.pause(); paint(); }
|
||||
play.textContent = ready ? (video.paused ? '播放真实试片' : '暂停真实试片') : '真实动作尚不可用';
|
||||
visibility();
|
||||
}
|
||||
|
||||
function layout() {
|
||||
const stage = get('frame');
|
||||
const scale = Math.min(stage.clientWidth / 1672, stage.clientHeight / 941);
|
||||
const left = (stage.clientWidth - 1672 * scale) / 2;
|
||||
const top = (stage.clientHeight - 941 * scale) / 2;
|
||||
Object.assign(canvas.style, { left: `${left + 1120 * scale}px`, top: `${top + 200 * scale}px`,
|
||||
width: `${552 * scale}px`, height: `${552 * scale}px` });
|
||||
}
|
||||
new ResizeObserver(layout).observe(get('frame'));
|
||||
for (const input of document.querySelectorAll('input')) input.addEventListener('change', visibility);
|
||||
get('lighting').addEventListener('click', () => {
|
||||
const button = get<HTMLButtonElement>('lighting');
|
||||
const dim = button.getAttribute('aria-pressed') !== 'true';
|
||||
button.setAttribute('aria-pressed', String(dim));
|
||||
button.textContent = `光照演示:${dim ? '环境减光' : '原始'}`;
|
||||
get('environment').style.setProperty('--demo-environment-brightness', dim ? '0.65' : '1');
|
||||
// No source assignment, load(), currentTime mutation or player recreation here.
|
||||
});
|
||||
get('closeup').addEventListener('click', () => {
|
||||
near = !near;
|
||||
const button = get<HTMLButtonElement>('closeup');
|
||||
button.setAttribute('aria-pressed', String(near));
|
||||
button.textContent = near ? '退出静态近景' : '进入静态近景';
|
||||
visibility();
|
||||
});
|
||||
play.addEventListener('click', () => { wantsPlay = !wantsPlay; void syncPlayback(); });
|
||||
document.addEventListener('visibilitychange', () => { void syncPlayback(); });
|
||||
reduced.addEventListener('change', () => {
|
||||
if (reduced.matches) wantsPlay = false;
|
||||
void syncPlayback();
|
||||
});
|
||||
video.addEventListener('error', () => {
|
||||
failed = true; ready = false; wantsPlay = false; play.disabled = true;
|
||||
fallback('动作素材缺失或解码失败,回退静态人物;未通过动态验证。');
|
||||
void syncPlayback();
|
||||
});
|
||||
|
||||
async function init() {
|
||||
try {
|
||||
const response = await fetch('./assets/manifest.json');
|
||||
if (!response.ok) throw new Error('manifest');
|
||||
const manifest: Manifest = await response.json();
|
||||
const motion = manifest.motion;
|
||||
if (!motion.file || !motion.alpha) {
|
||||
status.textContent = '静态分层可校对;无合格的透明动作素材,动态验收未通过。';
|
||||
play.disabled = true; play.textContent = '真实动作尚不可用'; return;
|
||||
}
|
||||
video.src = `./assets/${motion.file}`;
|
||||
video.addEventListener('loadeddata', () => {
|
||||
ready = true; play.disabled = false;
|
||||
status.textContent = `真实生成试片 · ${motion.status} · 循环切点与身份一致性以检查报告为准`;
|
||||
paint(); void syncPlayback();
|
||||
}, { once: true });
|
||||
} catch {
|
||||
fallback('素材清单不可用;保留静态分层校对。');
|
||||
play.textContent = '真实动作尚不可用';
|
||||
}
|
||||
}
|
||||
visibility(); layout(); void init();
|
||||
@@ -0,0 +1,22 @@
|
||||
@import './tokens.css';
|
||||
* { box-sizing: border-box; }
|
||||
[hidden] { display: none !important; }
|
||||
body { margin: 0; background: var(--scene-night-deep); color: var(--ink-primary); font: var(--fs-body) var(--font-sans); }
|
||||
main { max-width: 1600px; margin: auto; }
|
||||
header, section { padding: var(--space-4) var(--space-6); }
|
||||
h1 { font-size: var(--fs-title); margin: 0; }
|
||||
p { color: var(--ink-secondary); line-height: 1.5; }
|
||||
#stage { position: relative; overflow: hidden; height: min(60vh, 640px); background: var(--glass-bg-strong); }
|
||||
#frame { position: absolute; width: 100%; height: 100%; }
|
||||
#environment, .layer { position: absolute; width: 100%; height: 100%; object-fit: contain; }
|
||||
#environment { filter: brightness(var(--demo-environment-brightness, 1)); }
|
||||
#motion { position: absolute; }
|
||||
fieldset { border: 1px solid var(--glass-border); border-radius: var(--r-control); padding: var(--space-3); display: flex; flex-wrap: wrap; gap: var(--space-4); }
|
||||
label, button { cursor: pointer; }
|
||||
.controls { display: flex; flex-wrap: wrap; gap: var(--space-2); margin-top: var(--space-3); }
|
||||
button { color: var(--ink-primary); background: var(--glass-bg-strong); border: 1px solid var(--glass-highlight); border-radius: var(--r-control); padding: var(--space-3) var(--space-4); font: inherit; }
|
||||
button[aria-pressed=true] { border-color: var(--accent); }
|
||||
button:disabled { color: var(--ink-disabled); cursor: default; }
|
||||
button:focus-visible, input:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
|
||||
output { font-variant-numeric: tabular-nums; color: var(--accent); }
|
||||
@media (max-width: 600px) { header, section { padding: var(--space-3); } #stage { height: 38vh; } }
|
||||
@@ -0,0 +1,73 @@
|
||||
/* Reused verbatim from hel-700-rain-room; no alternative product theme. */
|
||||
:root {
|
||||
--scene-night-deep:#16233c;
|
||||
--scene-city-blue:#305481;
|
||||
--scene-city-mid:#5a76a2;
|
||||
--scene-tower:#2b456b;
|
||||
--scene-glass-blue:#718fc2;
|
||||
--scene-amber:#e8a45c;
|
||||
--scene-amber-soft:#f2c894;
|
||||
--scene-wood:#59361d;
|
||||
--scene-cream:#e8d9c8;
|
||||
--scene-blanket:#362928;
|
||||
--ink-primary:#f2efe9;
|
||||
--ink-secondary:rgba(242,239,233,.64);
|
||||
--ink-tertiary:rgba(242,239,233,.4);
|
||||
--ink-disabled:rgba(242,239,233,.28);
|
||||
--accent:#e8a45c;
|
||||
--accent-hover:#f2b879;
|
||||
--accent-pressed:#c9853f;
|
||||
--accent-ink:#1a1208;
|
||||
--glass-bg:rgba(14,18,30,.55);
|
||||
--glass-bg-strong:rgba(14,18,30,.72);
|
||||
--glass-border:rgba(255,255,255,.08);
|
||||
--glass-highlight:rgba(255,255,255,.14);
|
||||
--bubble-user:rgba(120,144,180,.2);
|
||||
--bubble-user-border:rgba(255,255,255,.07);
|
||||
--bubble-her:rgba(255,255,255,.06);
|
||||
--bubble-failed:rgba(200,90,80,.14);
|
||||
--state-danger:#e07a6a;
|
||||
--font-sans:"PingFang SC","HarmonyOS Sans SC","MiSans","Source Han Sans SC",system-ui,-apple-system,sans-serif;
|
||||
--fs-caption:12px;
|
||||
--fs-small:13px;
|
||||
--fs-body:14px;
|
||||
--fs-chat:15px;
|
||||
--fs-nav:15px;
|
||||
--fs-title:20px;
|
||||
--fs-brand:16px;
|
||||
--fs-time:13px;
|
||||
--space-1:4px;
|
||||
--space-2:8px;
|
||||
--space-3:12px;
|
||||
--space-4:16px;
|
||||
--space-5:20px;
|
||||
--space-6:24px;
|
||||
--space-8:32px;
|
||||
--space-10:40px;
|
||||
--r-panel:24px;
|
||||
--r-bubble:16px;
|
||||
--r-card:16px;
|
||||
--r-control:12px;
|
||||
--r-pill:999px;
|
||||
--shadow-panel:0 24px 80px rgba(0,0,0,.45);
|
||||
--shadow-pop:0 8px 24px rgba(0,0,0,.35);
|
||||
--glow-accent:0 0 12px rgba(232,164,92,.45);
|
||||
--glass-blur:blur(28px) saturate(1.2);
|
||||
--z-scene-far:0;
|
||||
--z-scene-rain:1;
|
||||
--z-scene-near:2;
|
||||
--z-caption:10;
|
||||
--z-nav:10;
|
||||
--z-topbar:20;
|
||||
--z-panel:30;
|
||||
--z-toast:40;
|
||||
--ease-out:cubic-bezier(.23,1,.32,1);
|
||||
--ease-in-out:cubic-bezier(.77,0,.175,1);
|
||||
--ease-drawer:cubic-bezier(.32,.72,0,1);
|
||||
--lamp-opacity:.12;
|
||||
--panel-width:clamp(360px,30vw,460px);
|
||||
--panel-top:64px;
|
||||
--panel-inset:24px;
|
||||
--scene-height:330px;
|
||||
--drawer-height:30dvh;
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
"""Foreground Chromium interaction checks, screenshots and downloadable recording."""
|
||||
import functools
|
||||
import http.server
|
||||
import json
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
EVIDENCE = ROOT / 'evidence'
|
||||
|
||||
|
||||
class Quiet(http.server.SimpleHTTPRequestHandler):
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
EVIDENCE.mkdir(exist_ok=True)
|
||||
handler = functools.partial(Quiet, directory=str(ROOT / 'dist'))
|
||||
server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), handler)
|
||||
thread = threading.Thread(target=server.serve_forever)
|
||||
thread.start()
|
||||
results = {}
|
||||
try:
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch()
|
||||
url = f'http://127.0.0.1:{server.server_port}'
|
||||
context = browser.new_context(viewport={'width': 1280, 'height': 800},
|
||||
record_video_dir=str(EVIDENCE),
|
||||
record_video_size={'width': 1280, 'height': 800})
|
||||
page = context.new_page()
|
||||
errors = []
|
||||
page.on('pageerror', lambda error: errors.append(str(error)))
|
||||
page.goto(url)
|
||||
page.wait_for_function('!document.querySelector("#play").disabled')
|
||||
assert page.locator('#source').evaluate('(v) => v.paused')
|
||||
page.click('#play')
|
||||
# Keep every layer visible for three full cycles before interaction checks.
|
||||
duration = page.locator('#source').evaluate('(v) => v.duration')
|
||||
page.evaluate("""() => {
|
||||
window.observedLoops = []; window.lastVideoTime = 0;
|
||||
const v = document.querySelector('#source');
|
||||
v.addEventListener('timeupdate', () => {
|
||||
if (v.currentTime < window.lastVideoTime) window.observedLoops.push(performance.now());
|
||||
window.lastVideoTime = v.currentTime;
|
||||
});
|
||||
}""")
|
||||
page.wait_for_timeout((duration * 3 + 0.8) * 1000)
|
||||
observed = page.evaluate('window.observedLoops')
|
||||
assert len(observed) >= 3, observed
|
||||
results['continuous_three_cycles'] = {'duration': duration, 'wraps_ms': observed}
|
||||
page.wait_for_timeout(800)
|
||||
alpha = page.locator('#motion').evaluate('(c) => {const x=c.getContext("2d"); return [x.getImageData(0,0,1,1).data[3], x.getImageData(300,220,1,1).data[3]]}')
|
||||
assert alpha[0] == 0 and alpha[1] > 240, alpha
|
||||
results['vp9_alpha_decoded_by_chromium'] = alpha
|
||||
page.wait_for_function('document.querySelector("#source").currentTime > 0.9 && document.querySelector("#source").currentTime < 1.4')
|
||||
positions = page.evaluate('''() => {
|
||||
const video=document.querySelector('#source'); const before=video.currentTime;
|
||||
const source=video.currentSrc; document.querySelector('#lighting').click();
|
||||
return {before, after:video.currentTime, sourcePreserved: source===video.currentSrc};
|
||||
}''')
|
||||
assert positions['before'] == positions['after'] and positions['sourcePreserved']
|
||||
results['environment_toggle_no_seek'] = positions
|
||||
page.wait_for_timeout(650)
|
||||
assert page.locator('#source').evaluate('(v) => !v.paused')
|
||||
page.click('#show-person')
|
||||
assert page.locator('#motion').is_hidden() and page.locator('#person').is_hidden()
|
||||
page.wait_for_timeout(600)
|
||||
page.screenshot(path=str(EVIDENCE / 'environment-only.png'))
|
||||
page.click('#show-person')
|
||||
page.click('#show-occlusion')
|
||||
assert page.locator('#occlusion').is_hidden()
|
||||
page.click('#show-occlusion')
|
||||
page.click('#show-shadow')
|
||||
assert page.locator('#shadow').is_hidden()
|
||||
page.click('#show-shadow')
|
||||
page.click('#show-mask')
|
||||
assert page.locator('#mask').is_visible() and page.locator('#environment').is_hidden()
|
||||
page.wait_for_timeout(500)
|
||||
page.click('#show-mask')
|
||||
results['independent_layers'] = True
|
||||
page.click('#closeup')
|
||||
assert page.locator('#talk').is_visible()
|
||||
assert page.locator('#source').evaluate('(v) => !v.paused')
|
||||
page.wait_for_timeout(700)
|
||||
page.screenshot(path=str(EVIDENCE / 'static-near.png'))
|
||||
page.click('#closeup')
|
||||
assert page.locator('#talk').is_hidden()
|
||||
page.wait_for_timeout(6400) # Includes the forward and generated return.
|
||||
results['near_is_static_player_continues'] = True
|
||||
page.click('#play')
|
||||
t = page.locator('#source').evaluate('(v) => v.currentTime')
|
||||
page.wait_for_timeout(250)
|
||||
assert page.locator('#source').evaluate('(v) => v.currentTime') == t
|
||||
results['manual_pause'] = True
|
||||
# Chromium lifecycle API freezes the page after making it hidden.
|
||||
session = context.new_cdp_session(page)
|
||||
session.send('Page.setWebLifecycleState', {'state': 'active'})
|
||||
# Visibility handler unit probe. Browser screenshot/recording checks above are real.
|
||||
page.click('#play')
|
||||
page.evaluate('''() => {
|
||||
Object.defineProperty(document, 'hidden', {configurable:true, get:()=>true});
|
||||
document.dispatchEvent(new Event('visibilitychange'));
|
||||
}''')
|
||||
assert page.locator('#source').evaluate('(v) => v.paused')
|
||||
page.evaluate('''() => {
|
||||
delete document.hidden; document.dispatchEvent(new Event('visibilitychange'));
|
||||
}''')
|
||||
page.wait_for_function('!document.querySelector("#source").paused')
|
||||
results['visibility_pause_resume_handler_synthetic_event'] = True
|
||||
for width, height in [(1600, 900), (1280, 800), (390, 844)]:
|
||||
page.set_viewport_size({'width': width, 'height': height})
|
||||
page.wait_for_timeout(180)
|
||||
bounds = page.evaluate('''() => {
|
||||
const r=document.querySelector('#motion').getBoundingClientRect();
|
||||
const s=document.querySelector('#stage').getBoundingClientRect();
|
||||
return {faceX:r.x+305/552*r.width, faceY:r.y+135/552*r.height,
|
||||
stage:{left:s.left,right:s.right,top:s.top,bottom:s.bottom},
|
||||
scroll:document.documentElement.scrollWidth, width:innerWidth};
|
||||
}''')
|
||||
assert bounds['scroll'] <= width
|
||||
assert bounds['stage']['left'] <= bounds['faceX'] <= bounds['stage']['right']
|
||||
assert bounds['stage']['top'] <= bounds['faceY'] <= bounds['stage']['bottom']
|
||||
page.screenshot(path=str(EVIDENCE / f'viewport-{width}x{height}.png'))
|
||||
results[f'viewport_{width}x{height}'] = bounds
|
||||
page.emulate_media(reduced_motion='reduce')
|
||||
page.wait_for_timeout(100)
|
||||
assert page.locator('#source').evaluate('(v) => v.paused')
|
||||
assert page.locator('#person').is_visible() and page.locator('#motion').is_hidden()
|
||||
results['reduced_motion_static_fallback'] = True
|
||||
page.emulate_media(reduced_motion='no-preference')
|
||||
assert page.locator('#source').evaluate('(v) => v.paused')
|
||||
video_path = Path(page.video.path())
|
||||
context.close()
|
||||
video_path.rename(EVIDENCE / 'browser-continuous.webm')
|
||||
missing = browser.new_page()
|
||||
missing.route('**/reading-alpha.webm', lambda route: route.abort())
|
||||
missing.goto(url)
|
||||
missing.wait_for_function('!document.querySelector("#fallback").hidden')
|
||||
assert missing.locator('#person').is_visible()
|
||||
assert missing.locator('#play').is_disabled()
|
||||
results['missing_motion_static_fallback'] = True
|
||||
missing.close()
|
||||
assert not errors, errors
|
||||
results['page_errors'] = errors
|
||||
browser.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join()
|
||||
server.server_close()
|
||||
(EVIDENCE / 'browser-results.json').write_text(json.dumps(results, ensure_ascii=False, indent=2))
|
||||
print(json.dumps(results, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,33 @@
|
||||
"""Reproduce the separately switchable foreground and provisional contact shadow."""
|
||||
from pathlib import Path
|
||||
|
||||
from PIL import Image, ImageDraw, ImageFilter
|
||||
|
||||
|
||||
def main():
|
||||
assets = Path(__file__).resolve().parents[1] / 'public/assets'
|
||||
source = Image.open(assets / 'reference.png').convert('RGBA')
|
||||
mask = Image.new('L', source.size)
|
||||
ImageDraw.Draw(mask).polygon([
|
||||
(1240, 582), (1282, 566), (1316, 556), (1360, 552), (1403, 532),
|
||||
(1438, 526), (1486, 514), (1517, 521), (1539, 544), (1559, 556),
|
||||
(1565, 634), (1520, 707), (1407, 747), (1260, 703), (1227, 650),
|
||||
], fill=255)
|
||||
mask = mask.filter(ImageFilter.GaussianBlur(1))
|
||||
source.putalpha(mask)
|
||||
source.save(assets / 'occlusion.png')
|
||||
mask.save(assets / 'occlusion-mask.png')
|
||||
mask = Image.new('L', source.size)
|
||||
ImageDraw.Draw(mask).ellipse((1230, 511, 1512, 595), fill=65)
|
||||
mask = mask.filter(ImageFilter.GaussianBlur(12))
|
||||
shadow = Image.new('RGBA', source.size, (44, 28, 20, 0))
|
||||
shadow.putalpha(mask)
|
||||
shadow.save(assets / 'contact-shadow.png')
|
||||
composite = Image.open(assets / 'environment.png').convert('RGBA')
|
||||
for name in ['contact-shadow.png', 'person.png', 'occlusion.png']:
|
||||
composite = Image.alpha_composite(composite, Image.open(assets / name))
|
||||
composite.save(assets / 'composite.png')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Use installed SAM modeling modules directly; no predictor/OpenCV dependency.
|
||||
|
||||
Run in the existing ComfyUI venv. No downloads or dependency changes.
|
||||
"""
|
||||
import argparse
|
||||
import importlib
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from PIL import Image, ImageChops, ImageDraw, ImageFilter
|
||||
|
||||
|
||||
def load_sam(source: Path, checkpoint: Path):
|
||||
# The optional predictor package imports cv2. We only use the original model API.
|
||||
package = types.ModuleType("hel711_sam")
|
||||
package.__path__ = [str(source)]
|
||||
sys.modules[package.__name__] = package
|
||||
builder = importlib.import_module("hel711_sam.build_sam")
|
||||
return builder.build_sam_vit_b(str(checkpoint)).eval()
|
||||
|
||||
|
||||
def segment(model, image, extra_positive=()):
|
||||
width, height = image.size
|
||||
scale = 1024 / max(width, height)
|
||||
size = (round(width * scale), round(height * scale))
|
||||
tensor = torch.from_numpy(np.array(image.resize(size))).permute(2, 0, 1).to(model.device)
|
||||
# Crop coordinate system: fixed right-hand seat, 552 x 552.
|
||||
points = torch.tensor([[[305, 140], [300, 245], [200, 300], [307, 301],
|
||||
[80, 100], [450, 400], [330, 450]]], dtype=torch.float)
|
||||
if extra_positive:
|
||||
extra = torch.tensor([extra_positive], dtype=torch.float)
|
||||
points = torch.cat((points, extra), dim=1)
|
||||
points = (points * scale).to(model.device)
|
||||
labels = torch.tensor([[1, 1, 1, 1, 0, 0, 0]], device=model.device)
|
||||
if extra_positive:
|
||||
labels = torch.cat((labels, torch.ones((1, len(extra_positive)), device=model.device)), dim=1)
|
||||
box = torch.tensor([[100, 60, 420, 380]], dtype=torch.float, device=model.device) * scale
|
||||
with torch.inference_mode():
|
||||
result = model([dict(image=tensor, original_size=(height, width),
|
||||
point_coords=points, point_labels=labels, boxes=box)],
|
||||
multimask_output=False)[0]
|
||||
mask = result['masks'][0, 0].cpu().numpy().astype(np.uint8) * 255
|
||||
mask = Image.fromarray(mask)
|
||||
# The book/knit texture produces enclosed SAM holes. Fill enclosed holes only.
|
||||
exterior = mask.copy()
|
||||
ImageDraw.floodfill(exterior, (0, 0), 255)
|
||||
mask = ImageChops.lighter(mask, ImageChops.invert(exterior))
|
||||
return mask.filter(ImageFilter.GaussianBlur(0.65))
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('assets', type=Path)
|
||||
parser.add_argument('--sam-source', type=Path, required=True)
|
||||
parser.add_argument('--checkpoint', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
torch.set_num_threads(8)
|
||||
model = load_sam(args.sam_source, args.checkpoint)
|
||||
source = Image.open(args.assets / 'reference.png').convert('RGB')
|
||||
region = (1120, 200, 1672, 752)
|
||||
crop = source.crop(region)
|
||||
mask = segment(model, crop)
|
||||
full_mask = Image.new('L', source.size)
|
||||
full_mask.paste(mask, region[:2])
|
||||
full_mask.save(args.assets / 'person-mask.png')
|
||||
layer = source.convert('RGBA')
|
||||
layer.putalpha(full_mask)
|
||||
layer.save(args.assets / 'person.png')
|
||||
crop.save(args.assets / 'generation-input.png')
|
||||
environment = Image.open(args.assets / 'environment.png').convert('RGBA')
|
||||
Image.alpha_composite(environment, layer).save(args.assets / 'composite.png')
|
||||
print('Saved original-pixel person layer, SAM mask, crop and real composite.', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,141 @@
|
||||
"""One foreground job using the installed ComfyUI native Wan nodes.
|
||||
|
||||
Owns and stops its exact server PID in finally. Does not modify the installation.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path('/media/lucian/Downloads/ComfyUI')
|
||||
TASK = ROOT / 'output/hel711-task'
|
||||
URL = 'http://127.0.0.1:18189'
|
||||
|
||||
|
||||
def request(path, payload=None):
|
||||
data = None if payload is None else json.dumps(payload).encode()
|
||||
req = urllib.request.Request(URL + path, data=data,
|
||||
headers={'Content-Type': 'application/json'})
|
||||
with urllib.request.urlopen(req, timeout=20) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def graph():
|
||||
nodes = {}
|
||||
|
||||
def add(key, kind, **inputs):
|
||||
nodes[str(key)] = dict(class_type=kind, inputs=inputs)
|
||||
|
||||
for key, mode in [(1, 'high'), (2, 'low')]:
|
||||
add(key, 'UNETLoader', unet_name=f'wan2.2_i2v_{mode}_noise_14B_fp8_scaled.safetensors',
|
||||
weight_dtype='default')
|
||||
add(key + 2, 'LoraLoaderModelOnly', model=[str(key), 0],
|
||||
lora_name=f'wan2.2_i2v_lightx2v_4steps_lora_v1_{mode}_noise.safetensors',
|
||||
strength_model=1.0)
|
||||
add(key + 4, 'ModelSamplingSD3', model=[str(key + 2), 0], shift=5.0)
|
||||
add(7, 'CLIPLoader', clip_name='umt5_xxl_fp8_e4m3fn_scaled.safetensors',
|
||||
type='wan', device='cpu')
|
||||
add(8, 'VAELoader', vae_name='wan_2.1_vae.safetensors')
|
||||
add(9, 'LoadImage', image='generation-input.png')
|
||||
add(10, 'CLIPTextEncode', clip=['7', 0], text=(
|
||||
'Locked off camera, photorealistic young adult woman with short dark hair, '
|
||||
'cream cardigan and dark trousers seated on the same sofa in the same bedroom. '
|
||||
'She quietly reads the book in her hands, slowly lowers her gaze to the page '
|
||||
'then gently raises her head a little. Small natural eye blink and breathing. '
|
||||
'Her hands keep holding the book. Fixed seat position, stationary room and furniture.'))
|
||||
add(11, 'CLIPTextEncode', clip=['7', 0],
|
||||
text='camera movement, zoom, changing face, changing clothes, standing up, '
|
||||
'extra fingers, distorted hands, deformed face, moving furniture, subtitles')
|
||||
add(12, 'WanImageToVideo', positive=['10', 0], negative=['11', 0], vae=['8', 0],
|
||||
width=512, height=512, length=49, batch_size=1, start_image=['9', 0])
|
||||
add(13, 'KSamplerAdvanced', model=['5', 0], add_noise='enable', noise_seed=711,
|
||||
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
|
||||
positive=['12', 0], negative=['12', 1], latent_image=['12', 2],
|
||||
start_at_step=0, end_at_step=2, return_with_leftover_noise='enable')
|
||||
add(14, 'KSamplerAdvanced', model=['6', 0], add_noise='disable', noise_seed=711,
|
||||
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
|
||||
positive=['12', 0], negative=['12', 1], latent_image=['13', 0],
|
||||
start_at_step=2, end_at_step=4, return_with_leftover_noise='disable')
|
||||
add(15, 'VAEDecode', samples=['14', 0], vae=['8', 0])
|
||||
add(16, 'SaveWEBM', images=['15', 0], codec='vp9', fps=16.0,
|
||||
filename_prefix='reading', crf=20.0)
|
||||
return nodes
|
||||
|
||||
|
||||
def main():
|
||||
for name in ['user', 'temp']:
|
||||
(TASK / name).mkdir(exist_ok=True)
|
||||
os.environ['HF_HUB_OFFLINE'] = '1'
|
||||
os.environ['TRANSFORMERS_OFFLINE'] = '1'
|
||||
command = [str(ROOT / 'venv/bin/python'), str(ROOT / 'main.py'),
|
||||
'--listen', '127.0.0.1', '--port', '18189', '--disable-all-custom-nodes',
|
||||
'--offline', '--disable-manager-ui', '--disable-assets',
|
||||
'--reserve-vram', '8', '--disable-dynamic-vram',
|
||||
'--input-directory', str(TASK / 'assets'),
|
||||
'--output-directory', str(TASK), '--user-directory', str(TASK / 'user'),
|
||||
'--temp-directory', str(TASK / 'temp'), '--database-url', 'sqlite:///:memory:']
|
||||
# Refuse to attach to an unrelated listener.
|
||||
import socket
|
||||
with socket.socket() as sock:
|
||||
if sock.connect_ex(('127.0.0.1', 18189)) == 0:
|
||||
raise RuntimeError('Task port already occupied; no process started')
|
||||
with open(TASK / 'server.log', 'w') as log:
|
||||
server = subprocess.Popen(command, cwd=ROOT, stdout=log, stderr=subprocess.STDOUT)
|
||||
(TASK / 'process.json').write_text(json.dumps(dict(pid=server.pid, command=command)))
|
||||
try:
|
||||
deadline = time.monotonic() + 90
|
||||
while True:
|
||||
if server.poll() is not None:
|
||||
raise RuntimeError(f'ComfyUI exited: {server.returncode}; see server.log')
|
||||
try:
|
||||
info = request('/object_info')
|
||||
break
|
||||
except (OSError, urllib.error.URLError):
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError('Startup exceeded 90 seconds') from None
|
||||
time.sleep(1)
|
||||
workflow = graph()
|
||||
(TASK / 'workflow.json').write_text(json.dumps(workflow, indent=2))
|
||||
required = {n['class_type']: info.get(n['class_type']) for n in workflow.values()}
|
||||
(TASK / 'nodes.json').write_text(json.dumps(required, indent=2))
|
||||
if any(v is None for v in required.values()):
|
||||
raise RuntimeError('Required native node unavailable; see nodes.json')
|
||||
result = request('/prompt', {'prompt': workflow})
|
||||
(TASK / 'submission.json').write_text(json.dumps(result))
|
||||
prompt_id = result['prompt_id']
|
||||
print(f'Generating one clip: {prompt_id}; owned server PID {server.pid}', flush=True)
|
||||
deadline = time.monotonic() + 2400
|
||||
last_report = 0
|
||||
while time.monotonic() < deadline:
|
||||
history = request('/history/' + prompt_id)
|
||||
if prompt_id in history:
|
||||
(TASK / 'history.json').write_text(json.dumps(history, indent=2))
|
||||
print(json.dumps(history[prompt_id]['status']), flush=True)
|
||||
if history[prompt_id]['status']['status_str'] != 'success':
|
||||
raise RuntimeError('Generation failed; see history.json')
|
||||
break
|
||||
if time.monotonic() - last_report > 30:
|
||||
print('Single generation still active; existing services untouched.', flush=True)
|
||||
last_report = time.monotonic()
|
||||
time.sleep(3)
|
||||
else:
|
||||
request('/interrupt', {})
|
||||
raise TimeoutError('Single generation exceeded 40 minutes')
|
||||
except urllib.error.HTTPError as error:
|
||||
(TASK / 'api-error.json').write_bytes(error.read())
|
||||
raise
|
||||
finally:
|
||||
server.terminate()
|
||||
try:
|
||||
server.wait(timeout=20)
|
||||
except subprocess.TimeoutExpired:
|
||||
server.kill()
|
||||
server.wait()
|
||||
print(f'Stopped owned server PID {server.pid}', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Generate one conditioned return, from original frame 48 back to frame 0.
|
||||
|
||||
Owns and stops its exact server PID in finally. Does not modify the installation.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path('/media/lucian/Downloads/ComfyUI')
|
||||
TASK = ROOT / 'output/hel712-task'
|
||||
URL = 'http://127.0.0.1:18189'
|
||||
|
||||
|
||||
def request(path, payload=None):
|
||||
data = None if payload is None else json.dumps(payload).encode()
|
||||
req = urllib.request.Request(URL + path, data=data,
|
||||
headers={'Content-Type': 'application/json'})
|
||||
with urllib.request.urlopen(req, timeout=20) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def graph():
|
||||
nodes = {}
|
||||
|
||||
def add(key, kind, **inputs):
|
||||
nodes[str(key)] = dict(class_type=kind, inputs=inputs)
|
||||
|
||||
for key, mode in [(1, 'high'), (2, 'low')]:
|
||||
add(key, 'UNETLoader', unet_name=f'wan2.2_i2v_{mode}_noise_14B_fp8_scaled.safetensors',
|
||||
weight_dtype='default')
|
||||
add(key + 2, 'LoraLoaderModelOnly', model=[str(key), 0],
|
||||
lora_name=f'wan2.2_i2v_lightx2v_4steps_lora_v1_{mode}_noise.safetensors',
|
||||
strength_model=1.0)
|
||||
add(key + 4, 'ModelSamplingSD3', model=[str(key + 2), 0], shift=5.0)
|
||||
add(7, 'CLIPLoader', clip_name='umt5_xxl_fp8_e4m3fn_scaled.safetensors',
|
||||
type='wan', device='cpu')
|
||||
add(8, 'VAELoader', vae_name='wan_2.1_vae.safetensors')
|
||||
add(9, 'LoadImage', image='return-start.png')
|
||||
add(17, 'LoadImage', image='return-end.png')
|
||||
add(10, 'CLIPTextEncode', clip=['7', 0], text=(
|
||||
'Locked off camera, photorealistic young adult woman with short dark hair, '
|
||||
'cream cardigan and dark trousers seated on the same sofa in the same bedroom. '
|
||||
'She slowly and smoothly raises her head from looking down at the book, '
|
||||
'returning to the exact relaxed head pose and eye direction in the final reference. '
|
||||
'Her hands keep holding the open book without moving it. Natural continuous breathing. '
|
||||
'Fixed seat position, stationary room and furniture, identical face and outfit.'))
|
||||
add(11, 'CLIPTextEncode', clip=['7', 0],
|
||||
text='camera movement, zoom, changing face, changing clothes, standing up, '
|
||||
'extra fingers, distorted hands, deformed face, moving furniture, subtitles')
|
||||
add(12, 'WanFirstLastFrameToVideo', positive=['10', 0], negative=['11', 0], vae=['8', 0],
|
||||
width=512, height=512, length=49, batch_size=1, start_image=['9', 0], end_image=['17', 0])
|
||||
add(13, 'KSamplerAdvanced', model=['5', 0], add_noise='enable', noise_seed=712,
|
||||
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
|
||||
positive=['12', 0], negative=['12', 1], latent_image=['12', 2],
|
||||
start_at_step=0, end_at_step=2, return_with_leftover_noise='enable')
|
||||
add(14, 'KSamplerAdvanced', model=['6', 0], add_noise='disable', noise_seed=712,
|
||||
steps=4, cfg=1.0, sampler_name='euler', scheduler='simple',
|
||||
positive=['12', 0], negative=['12', 1], latent_image=['13', 0],
|
||||
start_at_step=2, end_at_step=4, return_with_leftover_noise='disable')
|
||||
add(15, 'VAEDecode', samples=['14', 0], vae=['8', 0])
|
||||
add(16, 'SaveWEBM', images=['15', 0], codec='vp9', fps=16.0,
|
||||
filename_prefix='return', crf=20.0)
|
||||
return nodes
|
||||
|
||||
|
||||
def main():
|
||||
import av
|
||||
(TASK / 'assets').mkdir(parents=True, exist_ok=True)
|
||||
with av.open(str(ROOT / 'output/hel711-task/reading_00001_.webm')) as source:
|
||||
frames = [frame.to_image() for frame in source.decode(video=0)]
|
||||
frames[-1].save(TASK / 'assets/return-start.png')
|
||||
frames[0].save(TASK / 'assets/return-end.png')
|
||||
for name in ['user', 'temp']:
|
||||
(TASK / name).mkdir(exist_ok=True)
|
||||
os.environ['HF_HUB_OFFLINE'] = '1'
|
||||
os.environ['TRANSFORMERS_OFFLINE'] = '1'
|
||||
command = [str(ROOT / 'venv/bin/python'), str(ROOT / 'main.py'),
|
||||
'--listen', '127.0.0.1', '--port', '18189', '--disable-all-custom-nodes',
|
||||
'--offline', '--disable-manager-ui', '--disable-assets',
|
||||
'--reserve-vram', '8', '--disable-dynamic-vram',
|
||||
'--input-directory', str(TASK / 'assets'),
|
||||
'--output-directory', str(TASK), '--user-directory', str(TASK / 'user'),
|
||||
'--temp-directory', str(TASK / 'temp'), '--database-url', 'sqlite:///:memory:']
|
||||
# Refuse to attach to an unrelated listener.
|
||||
import socket
|
||||
with socket.socket() as sock:
|
||||
if sock.connect_ex(('127.0.0.1', 18189)) == 0:
|
||||
raise RuntimeError('Task port already occupied; no process started')
|
||||
with open(TASK / 'server.log', 'w') as log:
|
||||
server = subprocess.Popen(command, cwd=ROOT, stdout=log, stderr=subprocess.STDOUT)
|
||||
(TASK / 'process.json').write_text(json.dumps(dict(pid=server.pid, command=command)))
|
||||
try:
|
||||
deadline = time.monotonic() + 90
|
||||
while True:
|
||||
if server.poll() is not None:
|
||||
raise RuntimeError(f'ComfyUI exited: {server.returncode}; see server.log')
|
||||
try:
|
||||
info = request('/object_info')
|
||||
break
|
||||
except (OSError, urllib.error.URLError):
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError('Startup exceeded 90 seconds') from None
|
||||
time.sleep(1)
|
||||
workflow = graph()
|
||||
(TASK / 'workflow.json').write_text(json.dumps(workflow, indent=2))
|
||||
required = {n['class_type']: info.get(n['class_type']) for n in workflow.values()}
|
||||
(TASK / 'nodes.json').write_text(json.dumps(required, indent=2))
|
||||
if any(v is None for v in required.values()):
|
||||
raise RuntimeError('Required native node unavailable; see nodes.json')
|
||||
result = request('/prompt', {'prompt': workflow})
|
||||
(TASK / 'submission.json').write_text(json.dumps(result))
|
||||
prompt_id = result['prompt_id']
|
||||
print(f'Generating one clip: {prompt_id}; owned server PID {server.pid}', flush=True)
|
||||
deadline = time.monotonic() + 2400
|
||||
last_report = 0
|
||||
while time.monotonic() < deadline:
|
||||
history = request('/history/' + prompt_id)
|
||||
if prompt_id in history:
|
||||
(TASK / 'history.json').write_text(json.dumps(history, indent=2))
|
||||
print(json.dumps(history[prompt_id]['status']), flush=True)
|
||||
if history[prompt_id]['status']['status_str'] != 'success':
|
||||
raise RuntimeError('Generation failed; see history.json')
|
||||
break
|
||||
if time.monotonic() - last_report > 30:
|
||||
print('Single generation still active; existing services untouched.', flush=True)
|
||||
last_report = time.monotonic()
|
||||
time.sleep(3)
|
||||
else:
|
||||
request('/interrupt', {})
|
||||
raise TimeoutError('Single generation exceeded 40 minutes')
|
||||
except urllib.error.HTTPError as error:
|
||||
(TASK / 'api-error.json').write_bytes(error.read())
|
||||
raise
|
||||
finally:
|
||||
server.terminate()
|
||||
try:
|
||||
server.wait(timeout=20)
|
||||
except subprocess.TimeoutExpired:
|
||||
server.kill()
|
||||
server.wait()
|
||||
print(f'Stopped owned server PID {server.pid}', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Per-frame real SAM segmentation, then VP9 alpha encoding with existing PyAV."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import av
|
||||
import numpy as np
|
||||
import torch
|
||||
from extract_layers import load_sam, segment
|
||||
from PIL import Image
|
||||
|
||||
ROOT = Path('/media/lucian/Downloads/ComfyUI')
|
||||
TASK = ROOT / 'output/hel711-task'
|
||||
|
||||
|
||||
def main():
|
||||
torch.set_num_threads(8)
|
||||
model = load_sam(
|
||||
ROOT / 'custom_nodes/was-node-suite-comfyui/repos/SAM/segment_anything',
|
||||
ROOT / 'models/sams/sam_vit_b_01ec64.pth').to('cuda')
|
||||
output = av.open(str(TASK / 'reading-alpha.webm'), 'w')
|
||||
stream = output.add_stream('libvpx-vp9', rate=16)
|
||||
stream.width = 552
|
||||
stream.height = 552
|
||||
stream.pix_fmt = 'yuva420p'
|
||||
stream.options = {'crf': '20', 'b': '0', 'auto-alt-ref': '0'}
|
||||
metrics = []
|
||||
previews = []
|
||||
source = av.open(str(TASK / 'reading_00001_.webm'))
|
||||
previous = None
|
||||
first = None
|
||||
for index, decoded in enumerate(source.decode(video=0)):
|
||||
image = decoded.to_image().resize((552, 552), Image.Resampling.LANCZOS)
|
||||
mask = segment(model, image)
|
||||
frame = image.convert('RGBA')
|
||||
frame.putalpha(mask)
|
||||
pixels = np.array(image).astype(float)
|
||||
if first is None:
|
||||
first = pixels
|
||||
metrics.append(dict(frame=index, mask_pixels=int((np.array(mask) > 127).sum()),
|
||||
mean_absolute_rgb_change=0 if previous is None else float(np.abs(pixels-previous).mean())))
|
||||
previous = pixels
|
||||
if index in [0, 16, 32, 48]:
|
||||
previews.append(frame)
|
||||
frame.save(TASK / f'matted-{index:02}.png')
|
||||
for packet in stream.encode(av.VideoFrame.from_ndarray(np.array(frame), format='rgba')):
|
||||
output.mux(packet)
|
||||
if index % 16 == 0:
|
||||
print(f'Matted frame {index}', flush=True)
|
||||
for packet in stream.encode():
|
||||
output.mux(packet)
|
||||
output.close()
|
||||
source.close()
|
||||
(TASK / 'frame-metrics.json').write_text(json.dumps(dict(
|
||||
frames=metrics, loop_rgb_difference=float(np.abs(previous-first).mean()),
|
||||
interpretation='Real generated frames; RGB differences alone are not proof of valid anatomy or identity.'), indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Measure all joins and before/after hand mattes; export reviewable frame evidence."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import av
|
||||
import numpy as np
|
||||
from PIL import Image, ImageDraw
|
||||
from scipy.ndimage import binary_erosion, map_coordinates
|
||||
from skimage.color import rgb2gray
|
||||
from skimage.registration import optical_flow_tvl1
|
||||
|
||||
TASK = Path('/media/lucian/Downloads/ComfyUI/output/hel712-task')
|
||||
|
||||
|
||||
def summary(values):
|
||||
a = np.array(values)
|
||||
return dict(min=float(a.min()), median=float(np.median(a)),
|
||||
p95=float(np.percentile(a, 95)), max=float(a.max()))
|
||||
|
||||
|
||||
def panel(images, labels, path, size=(256, 256), columns=6):
|
||||
width, height = size
|
||||
board = Image.new('RGB', (columns*width, ((len(images)+columns-1)//columns)*(height+22)), '#202020')
|
||||
draw = ImageDraw.Draw(board)
|
||||
for i, (image, label) in enumerate(zip(images, labels, strict=True)):
|
||||
x, y = i % columns*width, i // columns*(height+22)
|
||||
board.paste(image.resize(size), (x, y))
|
||||
draw.text((x+3, y+height+3), label, fill='white')
|
||||
board.save(path)
|
||||
|
||||
|
||||
def main():
|
||||
data = np.load(TASK/'segmentation-v2.npz')
|
||||
mattes = np.load(TASK/'mattes.npz')
|
||||
rgb = data['images']
|
||||
# All measurement uses 512px source coordinates. The head box is fixed for both clips.
|
||||
images = [np.array(Image.fromarray(x).resize((512,512))) for x in rgb]
|
||||
before, after = [[np.array(Image.fromarray(x).resize((512,512))) > 127
|
||||
for x in mattes[key]] for key in ['before','after']]
|
||||
head = [x[50:180,230:340].astype(float) for x in images]
|
||||
pairs = [(i, i+1) for i in range(97)] + [(97,0)]
|
||||
head_diff = [float(np.abs(head[a]-head[b]).mean()) for a,b in pairs]
|
||||
ordinary = [d for i,d in enumerate(head_diff) if i not in [48,97]]
|
||||
report = {'coordinates':'512x512, head x230:340 y50:180, book/hand all x y175:330',
|
||||
'original_head_adjacent':summary(head_diff[:48]),
|
||||
'original_head_seam':float(np.abs(head[48]-head[0]).mean()),
|
||||
'loop_ordinary_head':summary(ordinary),
|
||||
'joins':[{'pair':[48,49],'head_mae':head_diff[48]},
|
||||
{'pair':[97,0],'head_mae':head_diff[97]}],
|
||||
'head_mae_all_pairs':head_diff}
|
||||
for name, masks in [('before',before),('after',after)]:
|
||||
edge = [int((m & ~binary_erosion(m))[175:330].sum()) for m in masks]
|
||||
switches = [int((masks[a]^masks[b])[175:330].sum()) for a,b in pairs]
|
||||
report[name] = {'boundary_pixels': edge, 'switch_pixels':switches,
|
||||
'original49_boundary':summary(edge[:49]),
|
||||
'original49_boundary_jump':summary(np.abs(np.diff(edge[:49]))),
|
||||
'original49_switch':summary(switches[:48]),
|
||||
'all_boundary':summary(edge), 'all_boundary_jump':summary(np.abs(np.diff(edge + [edge[0]]))),
|
||||
'all_switch':summary(switches),
|
||||
'joins_switch':[switches[48],switches[97]]}
|
||||
# Warp both alpha candidates with the same image motion. Residuals separate
|
||||
# mask switching from unaligned silhouette motion (not an anatomy metric).
|
||||
compensated = {'before':[], 'after':[]}
|
||||
rows, cols = np.meshgrid(np.arange(155), np.arange(512), indexing='ij')
|
||||
for a,b in pairs:
|
||||
flow = optical_flow_tvl1(rgb2gray(images[a][175:330]).astype(np.float32),
|
||||
rgb2gray(images[b][175:330]).astype(np.float32),
|
||||
num_warp=3, num_iter=5)
|
||||
for name,masks in [('before',before),('after',after)]:
|
||||
warped = map_coordinates(masks[b][175:330].astype(float),
|
||||
[rows+flow[0],cols+flow[1]], order=1, mode='nearest')>0.5
|
||||
compensated[name].append(int((masks[a][175:330]^warped).sum()))
|
||||
report['flow_compensated_switch']={k:{'all':summary(v), 'original49':summary(v[:48]),'per_pair':v}
|
||||
for k,v in compensated.items()}
|
||||
# Raw interior stays paired with alpha output; encoding error is reported separately.
|
||||
with av.open(str(TASK/'reading-alpha.webm')) as src:
|
||||
decoder=av.CodecContext.create('libvpx-vp9','r')
|
||||
decoded=[f.to_ndarray(format='rgba') for p in src.demux(video=0) for f in decoder.decode(p)]
|
||||
report['alpha_rgb_interior_mae']=summary([
|
||||
float(np.abs(out[:,:,:3].astype(float)-im).mean(axis=2)[m>250].mean())
|
||||
for out,im,m in zip(decoded,rgb,mattes['after'],strict=True)])
|
||||
(TASK/'metrics.json').write_text(json.dumps(report,indent=2))
|
||||
print(json.dumps({k:v for k,v in report.items() if k not in ['before','after','head_mae_all_pairs','flow_compensated_switch']},indent=2))
|
||||
print(json.dumps({k:{x:y for x,y in v.items() if not isinstance(y,list)} for k,v in report.items() if k in ['before','after']},indent=2))
|
||||
ids=[46,47,48,49,50,51,94,95,96,97,0,1]
|
||||
panel([Image.fromarray(images[i]) for i in ids], [f'frame {i}' for i in ids],TASK/'join-frames.png')
|
||||
panel([Image.fromarray(images[i][50:180,230:340]) for i in ids], [f'head {i}' for i in ids],TASK/'join-heads.png',size=(220,260))
|
||||
ids=[0,8,16,24,32,40,47,48,49,56,64,72,80,88,96,97]
|
||||
crops=[]
|
||||
labels=[]
|
||||
for i in ids:
|
||||
for name,masks in [('before',mattes['before']),('after',mattes['after'])]:
|
||||
rgba=np.dstack((rgb[i],masks[i]))
|
||||
bg=Image.new('RGBA',(552,552),'#557788')
|
||||
bg.alpha_composite(Image.fromarray(rgba))
|
||||
crops.append(bg.convert('RGB').crop((90,189,430,356)))
|
||||
labels.append(f'{name} {i}')
|
||||
panel(crops,labels,TASK/'hand-before-after.png',size=(340,167),columns=4)
|
||||
panel([Image.fromarray(mattes['after'][i]).convert('RGB').crop((90,189,430,356)) for i in ids],
|
||||
[f'alpha {i}' for i in ids],TASK/'hand-mattes.png',size=(340,167),columns=4)
|
||||
panel([Image.fromarray(images[i]) for i in range(98)], [str(i) for i in range(98)],
|
||||
TASK/'all-frames.png',size=(170,170),columns=14)
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
main()
|
||||
@@ -0,0 +1,146 @@
|
||||
"""Assemble the generated forward/return clips and stabilize only the hand/book matte.
|
||||
|
||||
Run with the existing ComfyUI environment; no models or dependencies are fetched.
|
||||
Optical flow aligns neighbouring masks; only alpha changes, never source RGB.
|
||||
"""
|
||||
import json
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
import av
|
||||
import numpy as np
|
||||
import torch
|
||||
from extract_layers import load_sam, segment
|
||||
from PIL import Image
|
||||
from scipy.ndimage import distance_transform_edt, gaussian_filter, label, map_coordinates
|
||||
from skimage.color import rgb2gray
|
||||
from skimage.registration import optical_flow_tvl1
|
||||
|
||||
ROOT = Path('/media/lucian/Downloads/ComfyUI')
|
||||
TASK = ROOT / 'output/hel712-task'
|
||||
BASE = ROOT / 'output/hel711-task'
|
||||
Y0, Y1 = round(175 * 552 / 512), round(330 * 552 / 512)
|
||||
|
||||
|
||||
def decode_alpha(path):
|
||||
with av.open(str(path)) as source:
|
||||
decoder = av.CodecContext.create('libvpx-vp9', 'r')
|
||||
return [f.to_ndarray(format='rgba') for p in source.demux(video=0)
|
||||
for f in decoder.decode(p)]
|
||||
|
||||
|
||||
def encode(path, frames, alpha):
|
||||
with av.open(str(path), 'w') as output:
|
||||
stream = output.add_stream('libvpx-vp9', rate=16)
|
||||
stream.width = stream.height = frames[0].shape[0]
|
||||
stream.pix_fmt = 'yuva420p' if alpha else 'yuv420p'
|
||||
stream.options = {'crf': '10', 'b': '0', 'auto-alt-ref': '0'}
|
||||
for frame in frames:
|
||||
for packet in stream.encode(av.VideoFrame.from_ndarray(
|
||||
frame, format='rgba' if alpha else 'rgb24')):
|
||||
output.mux(packet)
|
||||
for packet in stream.encode():
|
||||
output.mux(packet)
|
||||
|
||||
|
||||
def signed(mask):
|
||||
binary = mask > 127
|
||||
return distance_transform_edt(binary) - distance_transform_edt(~binary)
|
||||
|
||||
|
||||
def stabilize(images, masks):
|
||||
# Padding supplies flow context, but edits are restricted to the specified band.
|
||||
top, bottom = Y0 - 24, Y1 + 24
|
||||
gray = [rgb2gray(im[top:bottom]).astype(np.float32) for im in images]
|
||||
distances = [signed(m[top:bottom]) for m in masks]
|
||||
rows, cols = np.meshgrid(np.arange(bottom-top), np.arange(552), indexing='ij')
|
||||
def one(i):
|
||||
mask = masks[i]
|
||||
aligned = [distances[i]]
|
||||
for offset in [-2, -1, 1, 2]:
|
||||
j = (i + offset) % len(masks)
|
||||
flow = optical_flow_tvl1(gray[i], gray[j], attachment=15,
|
||||
num_warp=5, num_iter=10)
|
||||
aligned.append(map_coordinates(distances[j], [rows+flow[0], cols+flow[1]],
|
||||
order=1, mode='nearest'))
|
||||
# Trim the most extreme masks; averaging the remaining three signed
|
||||
# distances also avoids a majority-vote jump at the two clip joins.
|
||||
consensus = np.sort(aligned, axis=0)[1:-1].mean(axis=0)
|
||||
# A narrow spatial regularization removes single-pixel segmentation teeth.
|
||||
stable = gaussian_filter(consensus, 0.65)
|
||||
current = distances[i]
|
||||
stable = np.where(np.abs(current) <= 6, stable, current)
|
||||
alpha = np.clip(127.5 + stable * 127.5, 0, 255)
|
||||
# Blend the first/last 8 rows back to the unchanged matte.
|
||||
weight = np.ones(Y1-Y0)
|
||||
weight[:8] = np.linspace(0, 1, 8)
|
||||
weight[-8:] = np.linspace(1, 0, 8)
|
||||
new = mask.copy()
|
||||
roi = alpha[Y0-top:Y1-top]
|
||||
new[Y0:Y1] = np.rint(roi*weight[:, None] + mask[Y0:Y1]*(1-weight[:, None])).astype(np.uint8)
|
||||
return new, int(np.count_nonzero((new>127) != (mask>127)))
|
||||
|
||||
output, changes = [], []
|
||||
with ThreadPoolExecutor(max_workers=4) as executor:
|
||||
for i, (mask, change) in enumerate(executor.map(one, range(len(masks)))):
|
||||
output.append(mask)
|
||||
changes.append(change)
|
||||
if i % 8 == 0:
|
||||
print(f'Flow-aligned matte {i}/{len(masks)}', flush=True)
|
||||
return output, changes
|
||||
|
||||
|
||||
def main():
|
||||
torch.set_num_threads(8)
|
||||
cache = TASK / 'segmentation-v2.npz'
|
||||
if cache.exists():
|
||||
data = np.load(cache)
|
||||
images, masks = list(data['images']), list(data['masks'])
|
||||
else:
|
||||
# Preserve original matte outside the hand/book band without rerunning it.
|
||||
original = decode_alpha(BASE / 'reading-alpha.webm')
|
||||
with av.open(str(BASE / 'reading_00001_.webm')) as source:
|
||||
images = [np.array(f.to_image().resize((552, 552), Image.Resampling.LANCZOS))
|
||||
for f in source.decode(video=0)]
|
||||
masks = [f[:, :, 3] for f in original]
|
||||
model = load_sam(ROOT / 'custom_nodes/was-node-suite-comfyui/repos/SAM/segment_anything',
|
||||
ROOT / 'models/sams/sam_vit_b_01ec64.pth').to('cuda')
|
||||
with av.open(str(TASK / 'return_00001_.webm')) as source:
|
||||
for i, frame in enumerate(source.decode(video=0)):
|
||||
image = frame.to_image().resize((552, 552), Image.Resampling.LANCZOS)
|
||||
images.append(np.array(image))
|
||||
alpha = np.array(segment(model, image, extra_positive=(
|
||||
(245, 258), (280, 275), (360, 245), (365, 295))))
|
||||
components, count = label(alpha > 127)
|
||||
sizes = np.bincount(components.ravel())
|
||||
sizes[0] = 0
|
||||
keep = components == sizes.argmax()
|
||||
# Retain feather next to the connected subject; discard isolated sofa specks.
|
||||
from scipy.ndimage import binary_dilation
|
||||
alpha[~binary_dilation(keep, iterations=3)] = 0
|
||||
masks.append(alpha)
|
||||
if i % 8 == 0:
|
||||
print(f'Segmented return frame {i}', flush=True)
|
||||
np.savez_compressed(cache, images=np.array(images), masks=np.array(masks))
|
||||
del model
|
||||
torch.cuda.empty_cache()
|
||||
stable, changes = stabilize(images, masks)
|
||||
np.savez_compressed(TASK / 'mattes.npz', before=np.array(masks), after=np.array(stable))
|
||||
encode(TASK / 'loop-source.webm', images, False)
|
||||
frames = [np.dstack((rgb, a)) for rgb, a in zip(images, stable, strict=True)]
|
||||
encode(TASK / 'reading-alpha.webm', frames, True)
|
||||
(TASK / 'stabilization.json').write_text(json.dumps({
|
||||
'frames': len(images), 'fps': 16, 'join_pairs': [[48,49],[97,0]],
|
||||
'source_map': [{'frames':[0,48], 'source':'reading_00001_.webm'},
|
||||
{'frames':[49,97], 'source':'return_00001_.webm'}],
|
||||
'band_512': [175,330], 'band_552': [Y0,Y1],
|
||||
'flow': 'TV-L1, four neighbours, cyclic trimmed signed-distance mean, 6px uncertainty band',
|
||||
'changed_binary_pixels_per_frame': changes,
|
||||
'rgb': 'Original RGB retained; no temporal RGB blending or frame replacement',
|
||||
'outside_band_unchanged': all(np.array_equal(a[:Y0],b[:Y0]) and np.array_equal(a[Y1:],b[Y1:])
|
||||
for a,b in zip(masks,stable,strict=True))}, indent=2))
|
||||
print('Encoded 98-frame forward/return candidate, not yet accepted.', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"lib": [
|
||||
"ES2022",
|
||||
"DOM",
|
||||
"DOM.Iterable"
|
||||
],
|
||||
"strict": true,
|
||||
"noEmit": true,
|
||||
"skipLibCheck": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user