feat: release 1.5.4 - steering, Asset Library, canvas delete, agent hardening

- Mid-run steering input in AI Canvas + Build Scene (steering/message event, step-boundary drain, composer Steer mode)
- New /library Asset Library page: unlinked Playground media with All/Image/Video filters, 50-per-page lazy loading, bulk delete, add-to-project
- AI Canvas delete now persists: reference-checked file cleanup for playground-owned artifacts, entity tombstones, canvas.updated sync
- Agent loop hardening batch: swarm safety nets, clip tools in roles, capture false-success fix, planner fast-path, per-tool timeout, char-bounded history
- Build Scene Brief/Cut gates + MP4 export normalization
This commit is contained in:
Benjiyaya
2026-09-19 06:31:14 +08:00
parent bf557d8d5e
commit ee7648d957
55 changed files with 4395 additions and 289 deletions
+1 -1
View File
@@ -83,7 +83,7 @@ The app walks a project through four stages — **Story, Assets, Script, Video**
**Playground** is a free-form generation page outside the project pipeline: run any imported workflow with arbitrary inputs, upload your own files (image / video / audio) as inputs, and optionally attach a result to a project as an asset.
**Build Scene** is a browser-based 3D shot composer for blocking out cinematic shots before generating AI images or video. Instead of describing camera framing and character pose in text, you (or the agent chat) pose mannequin characters, place primitive props, and frame the camera in a three.js viewport — then **Capture PNG** (top-center of the viewport) a blockout that lands in every project reference picker under **From Build Scene**, ready to feed ControlNet-style `(Input:image)` slots. Scenes are managed like AI Canvas chats: a left rail holds any number of compositions, each bound to its own agent session, and deleting one removes its scene and captures. The viewport also has a **camera timeline** (max 60s): keyframe the camera at playheads, press play to fly the view along the track, and **Export video** to save the fly-through as a clip — it shows up alongside image blockouts in project clip pickers. The right-hand agent panel drives the same scene through built-in tools (no MCP): add/pose/frame commands mutate a server-side composition JSON, and `request_capture` grabs a render on demand.
**Build Scene** is a browser-based shot composer for blocking out cinematic shots before generating AI images or video. One composition model (`scene_json`) drives both preview and export, and the **Three.js viewport is the single surface**: it previews the scene, captures stills, and exports video (one deterministic `MediaRecorder` pass of the object keyframes + camera track, then normalized server-side to **H.264 MP4 via ffmpeg** so the Playground "From Build Scene" picker always receives a real MP4). **Camera framing is the user's job** — keyed from the timeline — while the agent animates objects through the `shot_*` keyframe tools. There are **no camera-beat tools** and **no ComfyUI** in this surface; agent gates (**Brief** before mutate, **Cut** before export) keep the agent honest. Scenes are managed like AI Canvas chats: a left rail holds compositions bound to agent sessions, and the agent panel drives the same scene through built-in `shot_*` tools (no MCP).
**Agents** is a chat-driven way to run the same pipeline: talk to a production agent that operates Calliope through tools (create project, draft story, write script, queue asset/video renders, watch jobs). Every chat session is bound to at most one project — start a **Sandbox** chat with no project and the agent materializes one via `create_project`, linking the session automatically; or link a session to an existing project and ask for edits. Complex builds are decomposed by a planner into sub-agents (story → script → assets → video). Everything the agent does goes through the same database and render queue the project UI reads — nothing bypasses the normal pipeline. When the agent waits on renders (`wait_for_jobs`), it uses the same **Poll timeout** as the queue worker (default 30 minutes).
+4
View File
@@ -46,6 +46,7 @@
"plan/created",
"question/answered",
"question/asked",
"steering/message",
"step/end",
"step/start",
"task/end",
@@ -89,6 +90,7 @@
"forget_memory",
"generate_script",
"generate_story",
"get_build_scene_gates",
"get_job_status",
"get_scene",
"get_story",
@@ -106,6 +108,7 @@
"move_keyframe_time",
"post_artifact_to_canvas",
"read_skill",
"record_build_scene_gate",
"rename_object",
"reorder_scenes",
"request_capture",
@@ -159,6 +162,7 @@
],
"guard_codes": [
"guard_destructive_replace",
"guard_no_export_video_tool",
"guard_render_approval",
"guard_scene_tool_scope"
]
+1
View File
@@ -15,6 +15,7 @@ dependencies = [
"python-multipart>=0.0.12",
"comfy-mcp>=0.10.0",
"mcp>=2.0.0",
"pillow>=12.3.0",
]
[project.optional-dependencies]
@@ -1,7 +1,7 @@
---
name: shot-composer-blockout
description: "Use when the user asks to build, pose, frame, or ANIMATE a 3D scene in Build Scene — characters, primitives, shot framing, keyframe motion, or exporting a blockout as a generation reference."
version: 1.0.0
version: 1.0.2
license: MIT
metadata:
author: Calliope
@@ -57,6 +57,28 @@ Adapted from open-media's agent guide, with motion-recipe judgment folded in.
- **Duration ≤ 60s.** `set_playback(duration)` caps there; keep keyframe times
within the duration. Preview loops.
## Camera, gates, and export
Build Scene has a **single** surface: the Three.js viewport. It is the only
preview and the only export/capture source — there is no separate render path
and no Remotion.
- **Camera framing is the user's job.** The viewport camera is keyed from the
TimelineStrip (`◉`). Do not drive the camera from `shot_*` tools, and do not
invent camera-beat tools — none exist.
- **Object motion is your job.** Stage → `add_keyframe` recipes animate objects;
that motion is what the exported clip renders (the viewport samples every
tracked object at the playhead, so object keyframes appear in the export even
with a static camera).
- **Brief/Cut gates.** Before mutating, `ask_user` the Brief; before export,
`ask_user` the Cut and `record_build_scene_gate(gate='cut')`. Soft maybe =
hard stop. The user clicks **Export video** — there is no `export_video`
tool.
- **ComfyUI / workflows** only when the user explicitly asks image/video gen,
or for stills that need Comfy after Capture — never as part of the Build
Scene loop.
## When a capability seems missing
Don't work around a gap by inventing behavior (no hand-rolled posture arrays,
@@ -54,11 +54,12 @@ the same beat — the eye reads simultaneous motion as intentional.
## Camera vs objects
Object tracks are viewport preview for framing; the **camera track is what
export renders**. To feature object motion in an exported clip, keep motion
within the framed area and let the camera track hold (or slowly push) while
the action plays. Keys for the camera come from the timeline UI (◉), not
from `shot_*` tools.
The export renders **both** object tracks and the camera track — the viewport
samples every tracked object at the playhead, so keyframed object motion
(walk-across, jump, head turn) appears in the exported clip. The camera track
adds camera moves on top (or holds static). Keys for the camera come from the
timeline UI (◉), not from `shot_*` tools — the agent animates objects; the
user frames the camera.
## Timing vocabulary (rule of thumb)
+1 -1
View File
@@ -4,4 +4,4 @@
# reads this; pyproject.toml is kept in sync for the build (hatchling dynamic
# version reads this file). PR #47: hard-coded literals drifted twice
# (1.3.2 shipped reporting 1.2.1; 1.4.1's /api/health still said 1.4.0).
__version__ = "1.5.3"
__version__ = "1.5.4"
@@ -39,6 +39,8 @@ GUARD_DESTRUCTIVE_REPLACE = "guard_destructive_replace"
# contracts exporter reads guard codes from THIS module, so re-export here.
from calliope.agent.harness.registry import ( # noqa: E402
GUARD_SCENE_TOOL_SCOPE,
GUARD_NO_EXPORT_VIDEO_TOOL,
EXPORT_VIDEO_DENIED_NAMES,
_scene_scoped,
)
@@ -14,6 +14,7 @@ from __future__ import annotations
import base64
import json
import logging
import sqlite3
from dataclasses import dataclass
from pathlib import Path
@@ -22,6 +23,8 @@ from typing import Any
from calliope.config import settings
from calliope.db import get_db, row_to_dict
logger = logging.getLogger("calliope.harness.log")
# Event types (a closed vocabulary; new types may be added, readers ignore
# unknown ones — same growth rule as deepseek-harness SessionEventMap).
@@ -46,6 +49,11 @@ QUESTION_ANSWERED = "question/answered"
# use_count bumps are statistics, not session history).
MEMORY_SAVED = "memory/saved"
MEMORY_FORGOTTEN = "memory/forgotten"
# Mid-run steering: the user course-corrects WHILE a turn is executing.
# Deliberately NOT user/message — policy derives render/destructive approval
# from the latest user/message, so steering must never grant or void
# permissions. Loops drain these between steps and inject them as context.
STEERING_MESSAGE = "steering/message"
TOOL_RESULT_TRUNCATE = 4000
# Appended wherever a tool result is cut for the LLM. Must TEACH the way out —
@@ -340,9 +348,9 @@ _VISION_MIME_BY_EXT = {
def _image_attachment_data_url(path: str) -> str | None:
"""Read an attachment image under assets_dir as a downscaled data URL.
Returns None (silently — the text appendix still names the file) when the
path is missing/outside assets_dir, not a known image type, or too large
after decoding.
Returns None (logging the reason — the text appendix still names the
file) when the path is missing/outside assets_dir, not a known image
type, or too large after decoding.
"""
raw = str(path or "").strip()
if not raw:
@@ -355,10 +363,10 @@ def _image_attachment_data_url(path: str) -> str | None:
mime = _VISION_MIME_BY_EXT.get(target.suffix.lower())
if mime is None or not target.is_file():
return None
try:
data = _downscale_image(target, mime)
except Exception:
data = _downscale_image(target, mime)
if data is None:
return None
data, mime = data
if not data or len(data) > _MAX_VISION_IMAGE_BYTES:
return None
return f"data:{mime};base64,{base64.b64encode(data).decode()}"
@@ -535,26 +543,54 @@ def _document_attachment_text(path: str, name: str = "") -> str | None:
return f"[Script document: {label}]\n{text}\n[/Script document]"
def _downscale_image(target: Path, mime: str) -> bytes | None:
"""Re-encode large images at reduced width; passes small ones through."""
def _downscale_image(target: Path, mime: str) -> tuple[bytes, str] | None:
"""Re-encode large images at reduced width; passes small ones through.
Returns (data, mime) — the mime is the source extension's for pass-through
and ``image/jpeg`` for re-encoded output (the caller must not keep
labeling JPEG bytes as image/png). None when the image cannot be read or
re-encoded small enough.
"""
data = target.read_bytes()
if len(data) <= _MAX_VISION_IMAGE_BYTES:
return data
return data, mime
try:
from PIL import Image
except ImportError:
logger.error(
"Vision: cannot downscale %s (%d bytes > %d budget) — Pillow is "
"not installed. Attachment degrades to a text path line; add the "
"`pillow` dependency to fix.",
target.name,
len(data),
_MAX_VISION_IMAGE_BYTES,
)
return None
try:
with Image.open(target) as img:
img = img.convert("RGB")
width = 1024
height = max(1, round(img.height * width / img.width))
img = img.resize((width, height))
# Adaptive ladder: step quality down, then width, until the
# re-encode fits the budget — dense screenshots can exceed it
# even at q82/1024px.
import io
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=82)
return buf.getvalue()
for width, quality in ((1024, 82), (1024, 60), (768, 50), (640, 40)):
height = max(1, round(img.height * width / img.width))
resized = img.resize((width, height))
buf = io.BytesIO()
resized.save(buf, format="JPEG", quality=quality)
if len(buf.getvalue()) <= _MAX_VISION_IMAGE_BYTES:
return buf.getvalue(), "image/jpeg"
except Exception:
logger.exception("Vision: failed to decode/downscale %s", target)
return None
logger.error(
"Vision: %s still over the %d budget after the full re-encode ladder "
"— degrading to text path line",
target.name,
_MAX_VISION_IMAGE_BYTES,
)
return None
def max_turn_number(session_id: int) -> int:
@@ -581,13 +617,81 @@ def max_turn_number(session_id: int) -> int:
conn.close()
# ─────────────────────────────────────────────────────────────────────────
# Mid-run steering
# ─────────────────────────────────────────────────────────────────────────
STEERING_INJECT_HEADER = "[STEERING — user message sent while you work]"
def steering_user_content(d: dict[str, Any]) -> Any:
"""Projected steering content with the [STEERING] header.
Same shape rules as a user message (string, or multimodal parts when
attachments are present) — the header is prepended to the text so both
the live injection (loop drains between steps) and the replayed history
projection show identical text.
"""
projected = project_user_content(
d.get("content") or "",
d.get("mentions"),
d.get("attachments"),
)
if isinstance(projected, list):
for part in projected:
if part.get("type") == "text":
part["text"] = f"{STEERING_INJECT_HEADER}\n{part.get('text') or ''}"
return projected
return [{"type": "text", "text": STEERING_INJECT_HEADER}, *projected]
return f"{STEERING_INJECT_HEADER}\n{projected}" if projected else STEERING_INJECT_HEADER
def drain_steering(session_id: int, after_seq: int) -> list[SessionEvent]:
"""Unconsumed steering events with seq > after_seq, oldest first.
Loops keep a per-turn watermark: steering drained once is never re-drained
(injection appends the text to the in-flight `messages` list directly, so
it lives on in that request history without a re-read)."""
conn = _db()
try:
rows = conn.execute(
"SELECT * FROM agent_events WHERE session_id = ? AND type = ? AND seq > ? ORDER BY seq",
(session_id, STEERING_MESSAGE, after_seq),
).fetchall()
finally:
conn.close()
out: list[SessionEvent] = []
for r in rows:
try:
data = json.loads(r["data_json"])
except (json.JSONDecodeError, TypeError):
data = {}
out.append(SessionEvent(seq=r["seq"], type=r["type"], data=data))
return out
def steering_max_seq(session_id: int) -> int:
"""Highest steering/message seq for the session (watermark seed)."""
conn = _db()
try:
row = conn.execute(
"SELECT MAX(seq) AS m FROM agent_events WHERE session_id = ? AND type = ?",
(session_id, STEERING_MESSAGE),
).fetchone()
return int(row["m"]) if row and row["m"] is not None else 0
finally:
conn.close()
# ─────────────────────────────────────────────────────────────────────────
# Derivations
# ─────────────────────────────────────────────────────────────────────────
def derive_llm_history(
events: list[SessionEvent], max_user_turns: int | None = None
events: list[SessionEvent],
max_user_turns: int | None = None,
max_chars: int | None = None,
) -> list[dict[str, Any]]:
"""Project OpenAI-format LLM history from the event log.
@@ -602,13 +706,39 @@ def derive_llm_history(
`max_user_turns` bounds the projection to the last N user messages (and
everything after each, so tool-call/result pairs always stay complete —
exchanges never span user turns). None = unbounded (legacy behavior).
`max_chars` is a secondary character budget over the assembled history:
when over budget, oldest WHOLE turns drop (never mid-turn — pairs stay
intact) until it fits or only the latest turn remains. None = unbounded.
`steering/message` events (mid-run course corrections) project as user
messages so the correction stays in the conversation record — but NEVER
inside an open tool exchange: a user message between an assistant
tool_calls message and its tool results is an invalid request sequence
(OpenAI-compatible servers 400 the whole next turn), so steering that
landed mid-exchange buffers until the exchange closes. Steering never
opens a user-turn boundary: it belongs to the turn it interrupted, and
`policy.latest_user_message` reads only user/message events — steering
cannot grant or void render/destructive approval.
"""
history: list[dict[str, Any]] = []
user_turn_boundaries: list[int] = []
tool_call_by_id: dict[str, dict[str, Any]] = {}
# Open tool exchange: call ids of the latest assistant tool_calls batch
# whose results have not all landed. Steering that arrives mid-exchange
# buffers here (a user message inside the pair is an invalid request).
pending_call_ids: set[str] = set()
steer_buffer: list[dict[str, Any]] = []
def _flush_steering() -> None:
for d in steer_buffer:
history.append({"role": "user", "content": steering_user_content(d)})
steer_buffer.clear()
for e in events:
d = e.data
if e.type == USER_MESSAGE:
_flush_steering() # never reorder steering after a later user turn
user_turn_boundaries.append(len(history))
history.append(
{
@@ -620,6 +750,9 @@ def derive_llm_history(
),
}
)
elif e.type == STEERING_MESSAGE:
if (d.get("content") or "").strip() or d.get("attachments"):
steer_buffer.append(d)
elif e.type == ASSISTANT_MESSAGE:
msg: dict[str, Any] = {"role": "assistant"}
name = d.get("agent_name")
@@ -631,6 +764,10 @@ def derive_llm_history(
tool_calls = d.get("tool_calls") or []
if tool_calls:
msg["tool_calls"] = tool_calls
for tc in tool_calls:
cid = tc.get("id")
if cid:
pending_call_ids.add(cid)
history.append(msg)
elif e.type == TOOL_CALL:
tool_call_by_id[d.get("call_id", "")] = d
@@ -646,6 +783,11 @@ def derive_llm_history(
"content": f"[{tool_name}] {digest}",
}
)
pending_call_ids.discard(d.get("call_id", ""))
if not pending_call_ids:
_flush_steering() # exchange closed — safe injection point
# Steering that never saw its exchange close (crashed turn) still projects.
_flush_steering()
if max_user_turns is not None and user_turn_boundaries:
# Keep the last N user turns (and everything after each boundary —
# tool exchanges never span user turns, so pairs stay intact).
@@ -657,9 +799,37 @@ def derive_llm_history(
start = user_turn_boundaries[-min(max_user_turns, len(user_turn_boundaries))]
if start > 0:
history = history[start:]
user_turn_boundaries = [b - start for b in user_turn_boundaries if b >= start]
if max_chars is not None and max_chars > 0:
history = _trim_history_to_char_budget(history, user_turn_boundaries, max_chars)
return history
def _history_len(history: list[dict[str, Any]]) -> int:
return sum(len(m.get("content") or "") for m in history)
def _trim_history_to_char_budget(
history: list[dict[str, Any]],
user_turn_boundaries: list[int],
max_chars: int,
) -> list[dict[str, Any]]:
"""Drop oldest WHOLE user turns until the history fits the budget.
Never drops mid-turn (tool_call/result pairs stay intact by construction)
and never drops the latest turn — an over-budget single turn passes
through rather than producing an empty request.
"""
if _history_len(history) <= max_chars or len(user_turn_boundaries) <= 1:
return history
# boundaries are positions in `history`; find the smallest start that
# fits, keeping at least the last turn.
for start in user_turn_boundaries[1:]:
if _history_len(history[start:]) <= max_chars:
return history[start:]
return history[user_turn_boundaries[-1]:]
def _truncate_result(result: Any) -> str:
text = json.dumps(result, ensure_ascii=False, default=str)
if len(text) > TOOL_RESULT_TRUNCATE:
@@ -683,6 +853,13 @@ def derive_chat_rows(events: list[SessionEvent]) -> list[dict[str, Any]]:
if d.get("attachments"):
row["attachments"] = d["attachments"]
rows.append(row)
elif e.type == STEERING_MESSAGE:
row = {"role": "user", "content": d.get("content", ""), "status": "steering"}
if d.get("mentions"):
row["mentions"] = d["mentions"]
if d.get("attachments"):
row["attachments"] = d["attachments"]
rows.append(row)
elif e.type == TOOL_CALL:
fn_name = d.get("tool_name") or "tool"
row: dict[str, Any] = {
@@ -52,6 +52,97 @@ def _llm_for_role(role: str) -> LLMClient:
return for_role(role)
return factory()
# ── Shared safety nets (main loop AND swarm sub-agents) ────────────────
# Extracted so orchestrator._run_sub_agent uses the exact same guards as
# run_turn — any future net added here applies to both loops for free.
REPEAT_EXECUTE_LIMIT = 2
FAIL_STREAK_LIMIT = 2
FINAL_STEP_NUDGE = (
"[SYSTEM] This is your FINAL step of this turn. "
"Do not start new tool work. Either (a) if the "
"goal is already met, reply with a concise "
"summary, or (b) reply with exactly what is "
"blocked and why, and what you need to proceed. "
"The user can continue the task in their next "
"message."
)
class RepeatGuard:
"""Identical (tool, args) calls within one turn/loop.
Read-like repeats execute normally (results may legitimately change);
after REPEAT_EXECUTE_LIMIT the call is NOT re-executed — the cached real
result is returned with an escalating instruction, so a stuck model gets
its own data back plus a way out instead of burning the step budget.
"""
def __init__(self) -> None:
self.cache: dict[str, dict[str, Any]] = {}
self.counts: dict[str, int] = {}
def key(self, name: str, args: dict[str, Any]) -> str:
return f"{name}:{json.dumps(args, sort_keys=True, ensure_ascii=False, default=str)}"
def bumped(self, name: str, args: dict[str, Any]) -> int:
k = self.key(name, args)
self.counts[k] = self.counts.get(k, 0) + 1
return self.counts[k]
def cached(self, name: str, args: dict[str, Any]) -> dict[str, Any] | None:
return self.cache.get(self.key(name, args))
def intercept(self, name: str, args: dict[str, Any]) -> dict[str, Any] | None:
"""Return the cached-result-plus-directive payload when over the
limit, else None (caller should execute for real)."""
count = self.bumped(name, args)
if count > REPEAT_EXECUTE_LIMIT and self.key(name, args) in self.cache:
cached = self.cache[self.key(name, args)]
return {
**cached,
"repeat_guard": {
"calls_so_far": count,
"limit": REPEAT_EXECUTE_LIMIT,
"message": (
f"You have already called {name} with these exact "
f"{count} times this turn. This call was NOT executed "
"again — the result above is the real one from the last "
"execution. Do not repeat it again. If the goal is not "
"met, call a DIFFERENT tool, change the arguments "
"meaningfully, or end your turn with a summary/question "
"for the user."
),
},
}
return None
def record(self, name: str, args: dict[str, Any], result: dict[str, Any]) -> None:
if isinstance(result, dict) and result.get("ok") is not False:
self.cache[self.key(name, args)] = result
def fail_directive_text(streak: int) -> str:
"""The [SYSTEM DIRECTIVE] appended after FAIL_STREAK_LIMIT failures in a
row — forces a DIFFERENT action or an explicit hand-back. Never a third
blind try."""
return (
f"\n\n[SYSTEM DIRECTIVE] That is {streak} failed calls in a row on this "
"goal. STOP retrying the same tool. Now either (a) call a DIFFERENT "
"tool with corrected arguments, or (b) end your turn with a clear "
"plain-text report of what is blocked and why, and what you need from "
"the user. Do not call any tool again with the same arguments."
)
def apply_fail_streak(result: dict[str, Any], streak: int) -> str:
"""Suffix builder: returns the directive only at/after the limit."""
if streak >= FAIL_STREAK_LIMIT and isinstance(result, dict) and result.get("ok") is False:
return fail_directive_text(streak)
return ""
# Async callback that persists + broadcasts one harness message.
MessageSink = Callable[[dict[str, Any]], Awaitable[None]]
@@ -96,15 +187,25 @@ async def run_turn(
# REPEAT_EXECUTE_LIMIT the call is NOT re-executed — the cached real result
# is returned with an escalating instruction, so a stuck model gets its
# own data back plus a way out instead of burning the step budget.
repeat_cache: dict[str, dict[str, Any]] = {}
repeat_counts: dict[str, int] = {}
REPEAT_EXECUTE_LIMIT = 2
repeat = RepeatGuard()
# Failure thrash guard (canvas/70): an agent that keeps getting errors
# (guard denials, bad args) must CHANGE COURSE, not grind. After this
# many not-ok results in a row, a directive is injected that forces a
# different action or an explicit hand-back to the user.
FAIL_STREAK_LIMIT = 2
fail_streak = 0
# Steering watermark: mid-run user course corrections appended after this
# seq are drained into `messages` at step boundaries (never between an
# assistant tool_calls message and its results — that would be an invalid
# request sequence). Next turn's derivation replays the same events.
steer_watermark = session_log.steering_max_seq(ctx.session_id)
def drain_steering_into_messages() -> None:
nonlocal steer_watermark
for s in session_log.drain_steering(ctx.session_id, steer_watermark):
steer_watermark = max(steer_watermark, s.seq)
messages.append(
{"role": "user", "content": session_log.steering_user_content(s.data)}
)
client = _llm_for_role("main")
final_text = ""
@@ -114,6 +215,7 @@ async def run_turn(
messages = [m for m in history if m.get("role") != "system"]
for iteration in range(1, max_iterations + 1):
# ── step boundary ────────────────────────────────────
drain_steering_into_messages()
log_append(session_log.STEP_START, {"turn": turn_no, "step": iteration})
# Rebuilt per step: linking mid-run flips tool visibility and the
# workspace digest changes after each tool.
@@ -128,15 +230,7 @@ async def run_turn(
messages.append(
{
"role": "user",
"content": (
"[SYSTEM] This is your FINAL step of this turn. "
"Do not start new tool work. Either (a) if the "
"goal is already met, reply with a concise "
"summary, or (b) reply with exactly what is "
"blocked and why, and what you need to proceed. "
"The user can continue the task in their next "
"message."
),
"content": FINAL_STEP_NUDGE,
}
)
tools = registry.openai_payload(ctx)
@@ -269,34 +363,12 @@ async def run_turn(
),
}
else:
repeat_key = f"{name}:{json.dumps(args, sort_keys=True, ensure_ascii=False, default=str)}"
repeat_counts[repeat_key] = repeat_counts.get(repeat_key, 0) + 1
count = repeat_counts[repeat_key]
if count > REPEAT_EXECUTE_LIMIT and repeat_key in repeat_cache:
cached = repeat_cache[repeat_key]
result = {
**cached,
"repeat_guard": {
"calls_so_far": count,
"limit": REPEAT_EXECUTE_LIMIT,
"message": (
f"You have already called {name} with these exact "
f"{count} times this turn. This call was NOT executed "
"again — the result above is the real one from the last "
"execution. Do not repeat it again. If the goal is not "
"met, call a DIFFERENT tool, change the arguments "
"meaningfully, or end your turn with a summary/question "
"for the user."
),
},
}
intercepted = repeat.intercept(name, args)
if intercepted is not None:
result = intercepted
else:
result = await registry.execute(ctx, name, args)
if (
isinstance(result, dict)
and result.get("ok") is not False
):
repeat_cache[repeat_key] = result
repeat.record(name, args, result)
# Log + emit the result FIRST. The ask_user card and the
# tool-row spinner both read these; ending the turn before
# them left the UI with no question and a stuck "working…"
@@ -332,16 +404,7 @@ async def run_turn(
fail_streak += 1
elif isinstance(result, dict):
fail_streak = 0
if fail_streak >= FAIL_STREAK_LIMIT:
result_text += (
"\n\n[SYSTEM DIRECTIVE] That is "
f"{fail_streak} failed calls in a row on this goal. "
"STOP retrying the same tool. Now either (a) call a "
"DIFFERENT tool with corrected arguments, or (b) end "
"your turn with a clear plain-text report of what is "
"blocked and why, and what you need from the user. "
"Do not call any tool again with the same arguments."
)
result_text += apply_fail_streak(result, fail_streak)
messages.append(
{
"role": "tool",
@@ -16,10 +16,18 @@ from typing import Any
from calliope.agent.harness import get_registry
from calliope.agent.harness import log as session_log
from calliope.agent.harness.loop import MessageSink, run_turn, _default_max_iterations
from calliope.agent.harness.loop import (
FINAL_STEP_NUDGE,
MessageSink,
RepeatGuard,
_default_max_iterations,
apply_fail_streak,
run_turn,
)
from calliope.agent.harness.prompts import hardening_text
from calliope.agent.harness.registry import ToolContext
from calliope.agent.llm import LLMClient
from calliope.config import settings
from calliope.events.bus import event_bus
logger = logging.getLogger("calliope.harness.orchestrator")
@@ -39,8 +47,70 @@ def _llm_for_role(role: str) -> LLMClient:
# always survive the trim intact.
MAX_HISTORY_USER_TURNS = 40
# _run_sub_agent returns this prefix when it called ask_user — orchestrate
# pauses the swarm on it (skips remaining tasks, no synthesis LLM call).
_PAUSED_PREFIX = "Paused: "
# Trivial-goal fast-path: one-line imperative messages skip the planner
# round-trip entirely. The single loop has every tool, so a misfire just
# loses the task-list UI — routing decision, not a permission change.
_TRIVIAL_MAX_CHARS = 160
_TRIVIAL_VERBS = (
"rename",
"update",
"set",
"change",
"edit",
"fix",
"show",
"list",
"what",
"how many",
"status",
"queue",
"where",
"who",
"delete",
"remove",
)
def _is_trivial_goal(goal: str) -> bool:
text = (goal or "").strip().lower()
if not text or len(text) > _TRIVIAL_MAX_CHARS:
return False
if any(sep in text for sep in ("\n", ";", " and then ", " then ")):
return False
return text.startswith(_TRIVIAL_VERBS)
def _history_char_budget() -> int | None:
"""Character budget for derived LLM history; 0 disables."""
try:
raw = int(getattr(settings, "agent_history_char_budget", 0) or 0)
except (TypeError, ValueError):
return None
return raw if raw > 0 else None
def _last_asked_question(session_id: int) -> str:
"""The most recent unanswered ask_user question text, for the pause note."""
try:
for e in reversed(session_log.read_events(session_id)):
if e.type != session_log.QUESTION_ASKED:
continue
return str((e.data or {}).get("question") or "question asked")
except Exception: # noqa: BLE001 — the note must never crash the pause path
pass
return "question asked"
# Tool subsets per sub-agent role. Scoped tighter than the full registry so a
# sub-agent cannot wander into another role's tools.
#
# `ask_user` is in EVERY role: a stuck sub-agent must be able to escalate to
# the user instead of grinding its step budget (the same hardening run_turn
# got for canvas/70). A question pauses the whole swarm — see _run_sub_agent.
ROLE_TOOLS: dict[str, list[str]] = {
"story": [
"get_workspace",
@@ -50,21 +120,34 @@ ROLE_TOOLS: dict[str, list[str]] = {
"update_beat",
"delete_beat",
"list_workflows",
"ask_user",
],
"script": [
"get_workspace",
"list_scenes",
"list_clips",
"generate_script",
"break_into_shots",
"add_clip",
"update_clip",
"delete_clip",
"update_scene",
"add_scene",
"delete_scene",
"reorder_scenes",
"ask_user",
],
"assets": [
"get_workspace",
"add_character",
"update_character",
"delete_character",
"add_location",
"update_location",
"delete_location",
"add_item",
"update_item",
"delete_item",
"list_workflows",
"comfy_server_info",
"run_workflow",
@@ -76,10 +159,12 @@ ROLE_TOOLS: dict[str, list[str]] = {
"wait_for_jobs",
"post_artifact_to_canvas",
"summarize_canvas",
"ask_user",
],
"video": [
"get_workspace",
"list_scenes",
"list_clips",
"list_workflows",
"run_workflow",
"enqueue_video_jobs",
@@ -88,6 +173,7 @@ ROLE_TOOLS: dict[str, list[str]] = {
"get_job_status",
"wait_for_jobs",
"comfy_server_info",
"ask_user",
],
}
@@ -191,32 +277,43 @@ async def orchestrate(
events = session_log.read_events(session_id)
# Bound the LLM context to the last N user turns: long-lived sessions
# would otherwise replay their entire history into every request.
derived = session_log.derive_llm_history(events, max_user_turns=MAX_HISTORY_USER_TURNS)
derived = session_log.derive_llm_history(
events,
max_user_turns=MAX_HISTORY_USER_TURNS,
max_chars=_history_char_budget(),
)
goal = next(
(m["content"] for m in reversed(derived) if m.get("role") == "user"),
"",
)
ws_result = await get_registry().execute(ctx, "get_workspace", {})
summary = json.dumps(ws_result, ensure_ascii=False, default=str)
if len(summary) > 3000:
summary = summary[:3000] + "…[truncated]"
if ctx.project_id is None:
# Blind/sandbox session (Build Scene, sandbox canvas): every swarm
# role's tool subset is project-scoped, so the planner can only
# misroute here. Skip it entirely — straight to the single loop —
# which also drops the misleading "scheduling a sub-agent" note.
# (Checked BEFORE the workspace read: blind sessions were paying a
# get_workspace query + summary build only to discard both.)
history.clear()
history.extend(derived)
return await run_turn(ctx, history, on_message=on_message)
# Trivial-goal fast-path: a one-line imperative ("rename character 3 to
# Kira", "what's the queue status") does not need a planner round-trip.
# The single loop has every tool, so a misrouted complex request still
# completes — it just loses the task-list UI. Routing, not permission:
# render guards are untouched either way.
if _is_trivial_goal(goal):
history.clear()
history.extend(derived)
return await run_turn(ctx, history, on_message=on_message)
ws_result = await get_registry().execute(ctx, "get_workspace", {})
summary = json.dumps(ws_result, ensure_ascii=False, default=str)
if len(summary) > 3000:
summary = summary[:3000] + "…[truncated]"
plan = await _plan(goal, summary)
if plan.get("mode") == "swarm" and ctx.project_id is None:
# Swarm sub-agents need project-scoped tools; in a sandbox the single
# loop handles create_project/link_project itself.
plan["mode"] = "single"
plan["tasks"] = []
note = (plan.get("note") or "").strip()
if note:
session_log.append_event(
@@ -260,9 +357,51 @@ async def orchestrate(
},
)
results: list[str] = []
paused = False
# Swarm-level steering watermark: corrections that land between tasks are
# folded into the next task's goal context (sub-agents see the user's
# course correction, not just the planner's original goal).
steer_watermark = session_log.steering_max_seq(session_id)
for i, task in enumerate(norm_tasks):
role = task["role"]
goal_i = task["goal"]
if not paused:
steer_lines = [
str(s.data.get("content") or "").strip()
for s in session_log.drain_steering(session_id, steer_watermark)
if str(s.data.get("content") or "").strip()
]
steer_watermark = session_log.steering_max_seq(session_id)
if steer_lines:
goal_i = goal_i + "\nMid-run user steering: " + " / ".join(steer_lines)
if paused:
# A previous sub-agent asked the user a question — later tasks may
# depend on the answer, so running them would waste work (and
# render permission). Skip cleanly; the user's reply re-routes.
session_log.append_event(
session_id,
session_log.TASK_END,
{"index": i, "status": "skipped"},
)
await event_bus.publish(
"agent.task", {"session_id": session_id, "index": i, "status": "skipped"}
)
session_log.append_event(
session_id,
session_log.ASSISTANT_MESSAGE,
{
"content": f"Skipped (waiting for the user's answer): {goal_i}",
"agent_name": f"{role}-agent",
},
)
await emit(
{
"role": "assistant",
"agent_name": f"{role}-agent",
"content": f"Skipped (waiting for the user's answer): {goal_i}",
}
)
continue
allowed = ROLE_TOOLS.get(role, ROLE_TOOLS["script"])
session_log.append_event(session_id, session_log.TASK_START, {"index": i})
await event_bus.publish(
@@ -317,6 +456,11 @@ async def orchestrate(
"content": answer,
}
)
if answer.startswith(_PAUSED_PREFIX):
# The sub-agent called ask_user: the question card is already
# persisted (question/asked event), so stop the swarm here
# instead of running tasks whose inputs are unknown.
paused = True
except Exception as exc: # noqa: BLE001
# Some exceptions stringify EMPTY (httpx.ReadTimeout/ReadError,
# TimeoutError) — always name the type, and keep the traceback
@@ -352,6 +496,21 @@ async def orchestrate(
}
)
if paused:
# Deterministic pause message — no synthesis LLM call. The question
# card is in the log; the user's next message resumes the work.
final = (
"Paused for your answer: " + _last_asked_question(session_id)
+ ". Remaining tasks were not started — reply to continue."
)
session_log.append_event(
session_id,
session_log.ASSISTANT_MESSAGE,
{"content": final, "agent_name": None},
)
await emit({"role": "assistant", "content": final})
return final
# Synthesis: plain LLM call over sub-agent reports.
synthesis_in = (
"User goal:\n"
@@ -431,12 +590,29 @@ async def _run_sub_agent(
hardening = hardening_text()
if hardening:
system += "\n\n" + hardening
# Same safety nets as run_turn — the swarm must not be a second-class
# citizen (a stuck sub-agent ground its whole budget with no repeat
# guard and no change-course directive before this).
repeat = RepeatGuard()
fail_streak = 0
# Steering watermark, same contract as run_turn: drain at step
# boundaries only (after the previous step's tool results).
steer_watermark = session_log.steering_max_seq(ctx.session_id)
final = ""
try:
for _ in range(max_iterations):
for iteration in range(1, max_iterations + 1):
for s in session_log.drain_steering(ctx.session_id, steer_watermark):
steer_watermark = max(steer_watermark, s.seq)
messages.append(
{"role": "user", "content": session_log.steering_user_content(s.data)}
)
payload = _scoped_payload(ctx, allowed_tools)
step_messages = list(messages)
if max_iterations - iteration == 0:
# Final-step budget nudge: wrap up, don't start new work.
step_messages.append({"role": "user", "content": FINAL_STEP_NUDGE})
msg = await client.chat_with_tools(
[{"role": "system", "content": system}] + messages,
[{"role": "system", "content": system}] + step_messages,
temperature=0.3,
tools=payload or None,
)
@@ -494,7 +670,12 @@ async def _run_sub_agent(
# out-of-role call must not run.
result = {"ok": False, "error": f"Tool not available to this role: {name}"}
else:
result = await registry.execute(ctx, name, parsed)
intercepted = repeat.intercept(name, parsed)
if intercepted is not None:
result = intercepted
else:
result = await registry.execute(ctx, name, parsed)
repeat.record(name, parsed, result)
session_log.append_event(
ctx.session_id,
session_log.TOOL_RESULT,
@@ -513,6 +694,12 @@ async def _run_sub_agent(
result_text = json.dumps(result, ensure_ascii=False, default=str)
if len(result_text) > session_log.TOOL_RESULT_TRUNCATE:
result_text = result_text[: session_log.TOOL_RESULT_TRUNCATE] + session_log.TRUNCATE_NOTE
# Thrash killer — same net as run_turn.
if isinstance(result, dict) and result.get("ok") is False:
fail_streak += 1
elif isinstance(result, dict):
fail_streak = 0
result_text += apply_fail_streak(result, fail_streak)
messages.append(
{"role": "tool", "tool_call_id": call_id, "content": result_text}
)
@@ -526,8 +713,16 @@ async def _run_sub_agent(
"content": "",
}
)
# ask_user pauses the whole swarm: the question card waits for
# the user's answer; later tasks may depend on it. The loop
# must not burn its remaining steps polling.
if isinstance(result, dict) and result.get("awaiting_user_input"):
return _PAUSED_PREFIX + (result.get("question") or "question asked")
else:
final = "Reached step budget."
final = (
"Reached step budget. Here is where things stand — the goal "
"may be partially complete."
)
finally:
await client.close()
return final or "Done."
@@ -308,6 +308,10 @@ def register(registry: ToolRegistry) -> None:
executor=t_wait_for_jobs,
category="video",
requires_project=False,
# Owns its own wait contract (timeout_sec → queue_poll_timeout_sec):
# exempt from the per-tool wall-clock cap, which would otherwise
# cut a legitimate long render wait short.
long_running=True,
)
)
registry.register(
@@ -1202,6 +1206,10 @@ async def t_wait_for_jobs(ctx: ToolContext, args: dict[str, Any]) -> dict[str, A
timeout = _resolve_wait_timeout(args)
deadline = None if timeout <= 0 else asyncio.get_event_loop().time() + timeout
queue_pid = _queue_project_id(ctx)
# Progress heartbeat: a long wait must not read as a dead "Working…" in
# the chat — publish a thinking line roughly every 30s while jobs run.
_PROGRESS_INTERVAL_S = 30.0
next_progress = asyncio.get_event_loop().time() + _PROGRESS_INTERVAL_S
while True:
jobs: list[dict[str, Any]] = []
invalid: list[int] = []
@@ -1235,6 +1243,22 @@ async def t_wait_for_jobs(ctx: ToolContext, args: dict[str, Any]) -> dict[str, A
"error": f"Timed out after {timeout}s waiting for {len(active)} job(s)",
"jobs": [_sc_job_dict_lite(j) for j in active],
}
now = asyncio.get_event_loop().time()
if now >= next_progress:
from calliope.events.bus import event_bus
await event_bus.publish(
"agent.thinking",
{
"session_id": ctx.session_id,
"content": (
f"Still waiting on {len(active)} render job(s) "
f"(#{', #'.join(str(j['id']) for j in active[:5])}"
f"{'…' if len(active) > 5 else ''})…"
),
},
)
next_progress = now + _PROGRESS_INTERVAL_S
await asyncio.sleep(3)
@@ -102,6 +102,94 @@ CAPTURE_POLL_TIMEOUT_S = 10.0
CAPTURE_POLL_INTERVAL_S = 0.5
# ---- Build Scene human gates (Brief / Cut) — Phase 5 --------------------
# Persisted on scene_json.gates so they ride shot.updated SSE with the graph.
# shotStore round-trips `gates`; PATCH merges gates if a client omits them.
_GATE_NAMES = ("brief", "cut")
_BRIEF_SOFT_WARNING = (
"brief_gate: Brief is not locked. Call ask_user with question "
"'Lock the Build Scene brief?' and options ['Brief locked', 'Revise brief'] "
"(lead with 'Nothing mutates until this is locked.'), then "
"record_build_scene_gate(gate='brief') before treating edits as approved."
)
_BRIEF_SOFT_INSTRUCTION = (
"Stop and obtain Brief via ask_user + record_build_scene_gate(gate='brief') "
"before continuing Build Scene / shot-composer-blockout. Soft 'maybe' = hard "
"stop. Do not call ComfyUI/workflows from this surface; export only after Cut."
)
_MUTATE_TOOLS_SOFT_BRIEF = frozenset({
"add_object",
"delete_object",
"rename_object",
"set_transform",
"reset_transform",
"clear_scene",
"set_joint",
"set_posture",
"apply_pose",
"reset_pose",
"set_shot",
"add_keyframe",
"update_keyframe",
"delete_keyframe",
"move_keyframe_time",
"set_playback",
})
def _gates_of(scene: dict[str, Any]) -> dict[str, Any]:
raw = scene.get("gates")
return raw if isinstance(raw, dict) else {}
def _public_gates(scene: dict[str, Any]) -> dict[str, Any]:
"""Normalize gates for tool / API responses."""
out: dict[str, Any] = {"brief": None, "cut": None}
for name in _GATE_NAMES:
entry = _gates_of(scene).get(name)
if isinstance(entry, dict) and entry.get("approved_at"):
out[name] = {
"approved_at": entry.get("approved_at"),
"source": entry.get("source") or "agent",
"note": entry.get("note"),
}
return out
def _gate_approved(scene: dict[str, Any], gate: str) -> bool:
entry = _gates_of(scene).get(gate)
return isinstance(entry, dict) and bool(entry.get("approved_at"))
def _brief_soft_fields(scene: dict[str, Any]) -> dict[str, Any]:
if _gate_approved(scene, "brief"):
return {}
return {
"warning": _BRIEF_SOFT_WARNING,
"instruction": _BRIEF_SOFT_INSTRUCTION,
"gates": _public_gates(scene),
}
def _wrap_brief_soft(executor): # type: ignore[no-untyped-def]
"""Soft-warn mutating shot_* tools when Brief is not recorded."""
async def wrapped(ctx: ToolContext, args: dict[str, Any]) -> dict[str, Any]:
_sid, scene_before = _load_scene(ctx)
result = await executor(ctx, args)
if isinstance(result, dict) and result.get("ok"):
extra = _brief_soft_fields(scene_before)
if extra:
result = {**result, **extra}
return result
return wrapped
def _scene_of(comp: dict[str, Any]) -> dict[str, Any]:
try:
scene = json.loads(comp.get("scene_json") or "{}")
@@ -244,6 +332,7 @@ async def t_get_scene(ctx: ToolContext, args: dict[str, Any]) -> dict[str, Any]:
objects=[_summarize_object(o) for o in scene.get("objects", []) if isinstance(o, dict)],
shotParams=scene.get("shotParams"),
playback=scene.get("playback"),
gates=_public_gates(scene),
)
@@ -365,10 +454,15 @@ async def t_select_object(ctx: ToolContext, args: dict[str, Any]) -> dict[str, A
async def t_clear_scene(ctx: ToolContext, args: dict[str, Any]) -> dict[str, Any]:
shot_id, _scene = _load_scene(ctx)
comp_id = _save_scene(ctx, {"objects": []})
shot_id, scene = _load_scene(ctx)
# Keep Brief/Cut approvals across wipe — gates live beside objects on scene_json.
next_scene: dict[str, Any] = {"objects": []}
gates = scene.get("gates")
if isinstance(gates, dict):
next_scene["gates"] = gates
comp_id = _save_scene(ctx, next_scene)
await _publish_updated(comp_id, "cleared")
return _ok(comp_id)
return _ok(comp_id, gates=_public_gates(next_scene))
# ── keyframes / motion (port of open-media's mcpBridge keyframe commands) ──
@@ -662,11 +756,12 @@ async def t_list_shot_presets(ctx: ToolContext, args: dict[str, Any]) -> dict[st
async def t_request_capture(ctx: ToolContext, args: dict[str, Any]) -> dict[str, Any]:
"""Stamp a capture request, then poll for the UI's posted capture row.
"""Capture a still via the UI Three.js viewport.
The browser renders the PNG (it owns the WebGL canvas) and POSTs it to
/api/shots/{id}/captures, which clears capture_request_json — that clear
is the completion signal.
Stamps capture_request_json and polls for the viewport to POST the PNG
(the Build Scene page renders the canvas and uploads the data URL). This is
the only capture path — stills come from the 3D editor, not a headless
renderer.
"""
from calliope.routers.shots import get_or_create_for_session
@@ -675,6 +770,12 @@ async def t_request_capture(ctx: ToolContext, args: dict[str, Any]) -> dict[str,
try:
comp = get_or_create_for_session(conn, ctx.session_id)
comp_id = int(comp["id"])
finally:
conn.close()
# --- UI poll path -----------------------------------------------------
conn = _db()
try:
conn.execute(
"UPDATE shot_composition SET capture_request_json = ?, "
"updated_at = CURRENT_TIMESTAMP WHERE id = ?",
@@ -705,16 +806,54 @@ async def t_request_capture(ctx: ToolContext, args: dict[str, Any]) -> dict[str,
label=label,
capture_id=cap["id"] if cap else None,
file_path=cap["file_path"] if cap else None,
note="Capture saved. Reference it in generation by this path.",
note="Capture saved via the 3D editor viewport. Reference it "
"in generation by this path.",
)
finally:
conn.close()
return _ok(
comp_id,
label=label,
note="Capture request sent — the page will save the PNG shortly "
"(it may be closed right now).",
)
return {
"ok": False,
"shot_id": comp_id,
"capture_pending": True,
"error": (
"Capture request timed out after "
f"{CAPTURE_POLL_TIMEOUT_S:.0f}s — the Build Scene page did not "
"save the PNG (it is probably closed). Tell the user to open "
"the Build Scene page and re-run capture; do NOT reference a "
"file path."
),
}
async def t_record_build_scene_gate(ctx: ToolContext, args: dict[str, Any]) -> dict[str, Any]:
"""Record Brief or Cut approval after ask_user affirmation (shot-composer-blockout)."""
from datetime import datetime, timezone
gate = str(args.get("gate") or "").strip().lower()
if gate not in _GATE_NAMES:
return {"ok": False, "error": "gate must be 'brief' or 'cut'"}
note = args.get("note")
if note is not None:
note = str(note).strip()[:500] or None
shot_id, scene = _load_scene(ctx)
gates = dict(_gates_of(scene))
gates[gate] = {
"approved_at": datetime.now(timezone.utc).isoformat(),
"source": "agent",
"note": note,
}
scene["gates"] = gates
comp_id = _save_scene(ctx, scene)
await _publish_updated(comp_id, f"gate_{gate}")
return _ok(comp_id, gate=gate, gates=_public_gates(scene))
async def t_get_build_scene_gates(ctx: ToolContext, args: dict[str, Any]) -> dict[str, Any]:
shot_id, scene = _load_scene(ctx)
return _ok(shot_id, gates=_public_gates(scene))
# ---- registration -----------------------------------------------------------
@@ -887,11 +1026,14 @@ def register(registry: ToolRegistry) -> None:
ToolDefinition(
name="set_joint",
description=(
"Pose one joint of a character (degrees). Joints: torso, head, "
"l_leg/r_leg, l_knee/r_knee (1 bend value), l_ankle/r_ankle, "
"l_arm/r_arm (shoulders), l_elbow/r_elbow (1 bend), "
"l_wrist/r_wrist, l_finger_0..4 / r_finger_0..4 (7 values). "
"Most joints take 3 euler values [x, y, z]."
"Pose one joint of a character (degrees) for pose/gizmo editing "
"or sparse motion-recipe keys — NOT per-frame puppeting in "
"Build Scene. Joints: torso, head, l_leg/r_leg, "
"l_knee/r_knee (1 bend), l_ankle/r_ankle, l_arm/r_arm, "
"l_elbow/r_elbow (1 bend), l_wrist/r_wrist, "
"l_finger_0..4 / r_finger_0..4 (7 values). Most joints take "
"3 euler values [x, y, z]. Prefer apply_pose + recipes over "
"frame-by-frame set_joint loops."
),
parameters={
"type": "object",
@@ -1019,7 +1161,9 @@ def register(registry: ToolRegistry) -> None:
description=(
"Render the current viewport as a PNG reference (for ControlNet "
"or prompt reference). The Build Scene page saves it; returns "
"the file path to reference in generation."
"the file path to reference in generation. FAILS if the Build "
"Scene page is not open — ask the user to open it before "
"calling, and never reference a file path on failure."
),
parameters={
"type": "object",
@@ -1148,3 +1292,53 @@ def register(registry: ToolRegistry) -> None:
blind_only=True,
)
)
registry.register(
ToolDefinition(
name="record_build_scene_gate",
description=(
"Record Brief or Cut approval for Build Scene (shot-composer-blockout). "
"Call ONLY after ask_user affirmation — Brief before mutates, "
"Cut before export. Soft 'maybe' is not approval."
),
parameters={
"type": "object",
"properties": {
"gate": {
"type": "string",
"enum": ["brief", "cut"],
"description": "Which human gate was approved",
},
"note": {
"type": "string",
"description": "Optional short note (e.g. brief summary)",
},
},
"required": ["gate"],
},
executor=t_record_build_scene_gate,
category="shot",
requires_project=False,
blind_only=True,
)
)
registry.register(
ToolDefinition(
name="get_build_scene_gates",
description=(
"Return Brief/Cut gate state for the session composition "
"(also included on get_scene as `gates`)."
),
parameters={"type": "object", "properties": {}},
executor=t_get_build_scene_gates,
category="shot",
requires_project=False,
blind_only=True,
)
)
# Soft Brief warning on mutators (export hard-enforces Cut separately).
for _name in _MUTATE_TOOLS_SOFT_BRIEF:
_defn = registry.get(_name)
if _defn is not None:
_defn.executor = _wrap_brief_soft(_defn.executor)
@@ -86,6 +86,24 @@ async def _mode_section(ctx: ToolContext) -> str | None:
def _mode_text(ctx: ToolContext) -> str:
if getattr(ctx, "origin", "chat") == "scene":
return (
"SESSION MODE: BUILD SCENE — this chat drives the 3D composition.\n"
"- One surface: the Three.js viewport (the 3D editor) + TimelineStrip "
"+ Export video. The viewport is the ONLY preview and the ONLY "
"export/capture source — there is no separate render preview.\n"
"- Tools here are shot_* + ask_user + skills/memory ONLY. "
"ComfyUI / run_workflow / list_workflows / enqueue_* are "
"unavailable and must not be attempted.\n"
"- Export video / stills come from the 3D editor viewport (the user "
"clicks Capture / Export video). There are no camera-beat tools.\n"
"- NEVER call a tool named export_video (it does not exist). After "
"Cut (ask_user → record_build_scene_gate), tell the user to click "
"**Export video** in the UI. Stills: request_capture only.\n"
"- Prefer read_skill(\"shot-composer-blockout\"); do NOT follow "
"scene-to-video / enqueue_video_jobs for Build Scene.\n"
"- Do not create_project just to animate — stay on this composition."
)
if ctx.project_id is None:
return (
"SESSION MODE: SANDBOX — no project is linked yet.\n"
@@ -266,6 +284,14 @@ async def _tool_discipline_section(ctx: ToolContext) -> str | None:
async def _mentions_section(ctx: ToolContext) -> str | None:
if getattr(ctx, "origin", "chat") == "scene":
return (
"Build Scene sessions ignore workflow tags for generation. "
"Do not call run_workflow / ComfyUI from this chat. A tagged "
"workflow_id= appendix is context only — the 3D editor viewport "
"and shot_* tools own this surface. ComfyUI belongs to explicit "
"image/video gen asks outside Build Scene."
)
if ctx.project_id is None:
return (
"Tagged workflows: a [Calliope context] appendix with workflow_id= "
@@ -304,9 +330,49 @@ async def _hardening_section(ctx: ToolContext) -> str | None:
return hardening_text()
async def _build_scene_section(ctx: ToolContext) -> str | None:
"""Build Scene policy — single 3D-editor surface, no camera-beat lane."""
if getattr(ctx, "origin", "chat") != "scene":
return None
return (
"BUILD SCENE POLICY (shot-composer-blockout — CoS):\n"
"1. TOOL ALLOWLIST: only shot_* tools, ask_user, list_skills/read_skill "
"(shot-composer-blockout), and memory tools. Ban ComfyUI / workflow "
"tool calls in this mode (run_workflow, list_workflows, enqueue_*, "
"comfy_*) — they are mechanically denied for Build Scene sessions; "
"if the user wants image/video generation, tell them to open "
"Playground / AI Canvas and tag a workflow there.\n"
"2. BRIEF GATE: before any mutate (add_object, set_transform, "
"keyframes, poses), ask_user Lock brief → on affirmative "
"record_build_scene_gate(gate='brief'). Soft 'maybe' = hard stop.\n"
"3. CUT GATE: before export / shipping to pickers, ask_user Approve "
"cut → record_build_scene_gate(gate='cut'). Soft maybe = hard stop.\n"
"4. MOTION: stage→add_keyframe recipes only (read_skill "
"shot-composer-blockout motion-recipes). There are NO camera-beat "
"tools — do not invent set_camera_beats / apply_clip_v4_beats / "
"apply_push_in_beats. Camera framing is the user's job in the "
"TimelineStrip; do not try to drive the camera.\n"
"5. PREVIEW: Build Scene = the Three.js viewport. It is the ONLY "
"preview and the ONLY export/capture source — no separate render "
"preview or export path.\n"
"6. Never read/follow scene-to-video for this surface — that skill "
"targets Comfy enqueue_video_jobs (closed lane here).\n"
"7. NEVER invent or call export_video. There is no such tool. After "
"Cut approval (ask_user → record_build_scene_gate(gate='cut')), tell "
"the user to click **Export video** in Build Scene UI. Stills only: "
"request_capture.\n"
"8. Characters are blockout proxies (not limb walk cycles). "
"Walk-across = sparse root keyframes; do not claim full locomotion "
"until motion recipes land."
)
def register_builtin_sections(service: SystemPromptService) -> None:
service.register("persona", 10, _persona_section)
service.register("mode", 20, _mode_section)
service.register("build_scene", 25, _build_scene_section)
service.register("workspace", 30, _workspace_digest_section)
# Memory recall (order 35): usage-ranked preferences from harness.plugins.memory.
# Imported lazily so composing prompts alone never composes the registry.
@@ -12,6 +12,7 @@ Key invariants kept from the previous registry:
"""
from __future__ import annotations
import asyncio
import inspect
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field
@@ -41,6 +42,19 @@ _SCENE_ALLOWED_TOOLS = {
"forget_memory",
}
# Invented agent tool — there is no export_video; Cut + UI Export video instead.
GUARD_NO_EXPORT_VIDEO_TOOL = "guard_no_export_video_tool"
EXPORT_VIDEO_DENIED_NAMES = frozenset({"export_video", "exportVideo"})
def _tool_timeout_sec() -> float:
"""Per-tool wall-clock cap from Settings; 0 = disabled."""
try:
t = float(settings.agent_tool_timeout_sec)
except (TypeError, ValueError, AttributeError):
return 0.0
return max(0.0, t)
def _scene_scoped(t: "ToolDefinition") -> bool:
return getattr(t, "category", None) in _SCENE_ALLOWED_CATEGORIES or t.name in _SCENE_ALLOWED_TOOLS
@@ -77,6 +91,9 @@ class ToolDefinition:
category: str = "general"
destructive: bool = False # flagged in descriptions; pre-execute guard uses it
requires_approval: bool = False # HITL: blocked unless the user explicitly asked
# True → exempt from the per-tool wall-clock timeout (agent_tool_timeout_sec);
# the tool owns its own wait contract (e.g. wait_for_jobs → queue_poll_timeout_sec).
long_running: bool = False
# Hook results — the vocabulary of the execution pipeline.
@@ -131,6 +148,14 @@ class ToolRegistry:
pre_execute: list[PreExecuteHook] = field(default_factory=list)
post_execute: list[PostExecuteHook] = field(default_factory=list)
async def _execute_with_timeout(
self, t: "ToolDefinition", ctx: ToolContext, args: dict[str, Any]
) -> dict[str, Any]:
timeout = _tool_timeout_sec()
if timeout <= 0 or t.long_running:
return await t.executor(ctx, args)
return await asyncio.wait_for(t.executor(ctx, args), timeout=timeout)
def register(self, definition: ToolDefinition) -> None:
if definition.name in self.tools:
raise ValueError(f"Tool already registered: {definition.name}")
@@ -188,6 +213,21 @@ class ToolRegistry:
"""Run one tool through the guarded pipeline. Returns a JSON-serializable dict."""
t = self.tools.get(name)
if t is None:
if name in EXPORT_VIDEO_DENIED_NAMES:
return {
"ok": False,
"reason_code": GUARD_NO_EXPORT_VIDEO_TOOL,
"error": (
"There is no tool named export_video. Build Scene "
"export is human-gated: ask_user Cut approval → "
"record_build_scene_gate(gate='cut') → tell the user to "
"click **Export video** in the Build Scene UI. "
"For stills only use request_capture. Do not invent "
"export_video; do not call enqueue_video_jobs or follow "
"scene-to-video — that lane is Comfy closed for origin=scene. "
"Prefer read_skill(\"shot-composer-blockout\")."
),
}
return {"ok": False, "error": f"Unknown tool: {name}"}
if t.requires_project and ctx.project_id is None:
return {
@@ -203,16 +243,27 @@ class ToolRegistry:
),
}
if ctx.origin == "scene" and not _scene_scoped(t):
extra = ""
if name in {"enqueue_video_jobs", "enqueue_asset_jobs", "run_workflow"} or name.startswith(
("enqueue_", "comfy_")
):
extra = (
" scene-to-video / enqueue_video_jobs is the WRONG skill/lane "
"for Build Scene — use read_skill(\"shot-composer-blockout\") "
"and Cut → user clicks Export video."
)
return {
"ok": False,
"reason_code": GUARD_SCENE_TOOL_SCOPE,
"error": (
"This is a Build Scene session: only the 3D blockout tools "
"(get_scene, add_object, set_transform, set_joint, add_keyframe, "
"set_shot, …) are available here. Build and pose the scene, then "
"use Capture / Export video — the blockout clip is picked as a "
"reference input from the Playground media library, not generated "
"This is a Build Scene session: "
"only shot_* tools plus ask_user / skills / memory are available. "
"ComfyUI and workflow tools (run_workflow, list_workflows, "
"enqueue_*, comfy_*) are banned here. Build with shot_* "
"recipes, then Capture / Export video (Cut-gated) — "
"pick the clip from Playground later; do not generate via Comfy "
"from this chat."
+ extra
),
}
@@ -228,7 +279,16 @@ class ToolRegistry:
denied["reason_code"] = decision.reason_code
return denied
try:
result = await t.executor(ctx, args)
result = await self._execute_with_timeout(t, ctx, args)
except TimeoutError:
# Per-tool wall-clock cap (0 = disabled): a hung tool must stall
# its step, not the session. CancelledError passes through so the
# Stop button keeps working.
return {
"ok": False,
"error": f"{name} timed out after {_tool_timeout_sec():.0f}s",
"timeout": True,
}
except Exception as exc: # noqa: BLE001 — tool errors are loop-feedback, not crashes
return {"ok": False, "error": f"{type(exc).__name__}: {exc}"}
if isinstance(result, dict):
@@ -301,6 +301,67 @@ class AgentRunner:
await self._publish_session(session_id)
return user_msg
async def steer(
self,
session_id: int,
content: str,
*,
mentions: list[dict[str, Any]] | None = None,
attachments: list[dict[str, Any]] | None = None,
) -> dict[str, Any] | None:
"""Persist + publish a mid-run steering message. No new turn.
Only valid while a turn is running (the router falls back to this when
POST /messages hits the running-session 409). The event is
`steering/message`, NOT user/message — policy derives render and
destructive approval from the latest user/message, so steering must
never silently grant or void permissions. The running loop drains
these between steps and injects them into its in-flight request
history. Returns the published chat row, or None when the session is
not running (caller should start a normal turn instead).
"""
if not self.is_running(session_id):
return None
extra: dict[str, Any] = {}
if mentions:
extra["mentions"] = mentions
if attachments:
extra["attachments"] = attachments
event = session_log.append_event(
session_id,
session_log.STEERING_MESSAGE,
{"content": content, **extra},
)
# Mirror into agent_messages so legacy readers see the row; the event
# log stays authoritative (role user, status steering).
conn = self._db()
try:
cur = conn.execute(
"""
INSERT INTO agent_messages
(session_id, role, content, agent_name, status)
VALUES (?, 'user', ?, NULL, 'steering')
""",
(session_id, content or ""),
)
conn.commit()
row = conn.execute(
"SELECT * FROM agent_messages WHERE id = ?", (cur.lastrowid,)
).fetchone()
finally:
conn.close()
from calliope.db import row_to_dict
out = row_to_dict(row)
out["seq"] = event.seq
if mentions:
out["mentions"] = mentions
if attachments:
out["attachments"] = attachments
await event_bus.publish("agent.message", {"message": out})
return out
async def _run_session(self, session_id: int) -> None:
conn = self._db()
try:
@@ -312,7 +373,11 @@ class AgentRunner:
finally:
conn.close()
ctx = ToolContext(session_id=session_id, project_id=project_id, origin=origin)
ctx = ToolContext(
session_id=session_id,
project_id=project_id,
origin=origin,
)
try:
final = await orchestrate(
ctx,
+12
View File
@@ -125,6 +125,14 @@ class Settings(BaseSettings):
queue_max_retries: int = 2
agent_max_steps: int = 24
agent_hardening_prompt: str = DEFAULT_AGENT_HARDENING_PROMPT
# Wall-clock cap on ONE tool execution inside the agent loop. Guards
# against a hung tool stalling a session indefinitely; 0 = disabled.
# Long-by-contract tools (wait_for_jobs) opt out via long_running=True.
agent_tool_timeout_sec: float = 600.0
# Character budget (~4 chars/token) for the derived LLM history — the
# 40-turn cap alone can still overflow a context window with heavy
# multi-step turns. Oldest whole turns drop first; 0 = disabled.
agent_history_char_budget: int = 400_000
# Extra OpenAI-compatible request fields merged into the MiniMax H3 prompt
# rewrite call only (the `minimax_h3_ref` profile). The rewrite is a
# formatting task: on a thinking model it can burn 10k+ reasoning tokens per
@@ -321,6 +329,8 @@ class Settings(BaseSettings):
"queue_max_retries": self.queue_max_retries,
"agent_max_steps": self.agent_max_steps,
"agent_hardening_prompt": self.agent_hardening_prompt,
"agent_tool_timeout_sec": self.agent_tool_timeout_sec,
"agent_history_char_budget": self.agent_history_char_budget,
"h3_rewrite_extra_body": dict(self.h3_rewrite_extra_body or {}),
"dry_run": bool(self.dry_run),
}
@@ -404,6 +414,8 @@ class Settings(BaseSettings):
"queue_max_retries": self.queue_max_retries,
"agent_max_steps": self.agent_max_steps,
"agent_hardening_prompt": self.agent_hardening_prompt,
"agent_tool_timeout_sec": self.agent_tool_timeout_sec,
"agent_history_char_budget": self.agent_history_char_budget,
"h3_rewrite_extra_body": dict(self.h3_rewrite_extra_body or {}),
"dry_run": bool(self.dry_run),
}
+2
View File
@@ -22,6 +22,7 @@ from calliope.routers import (
canvas,
events,
jobs,
library,
playground,
projects,
scenes,
@@ -134,6 +135,7 @@ def create_app(static_dir: Path | None = None) -> FastAPI:
app.include_router(workflows.router, prefix="/api/workflows", tags=["workflows"])
app.include_router(jobs.router, prefix="/api/jobs", tags=["jobs"])
app.include_router(playground.router, prefix="/api/playground", tags=["playground"])
app.include_router(library.router, prefix="/api/library", tags=["library"])
app.include_router(canvas.router, prefix="/api/canvas", tags=["canvas"])
app.include_router(shots.router, prefix="/api/shots", tags=["shots"])
app.include_router(agent.router, prefix="/api/agent", tags=["agent"])
+31 -2
View File
@@ -369,9 +369,38 @@ async def post_message(session_id: int, payload: MessageCreate) -> dict[str, Any
answer_to=payload.answer_to,
)
except RuntimeError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
# Session mid-turn: the message becomes steering, not a rejection.
# The running loop drains steering/message events between steps and
# injects them into its request history — the user course-corrects
# without killing the run (Stop stays available for a hard stop).
# Answering a question card while running stays a 409: the answer
# belongs to the NEXT turn, which starts when this one ends.
if payload.answer_to is not None:
raise HTTPException(status_code=409, detail=str(exc)) from exc
steered = await runner.steer(
session_id,
content,
mentions=mentions or None,
attachments=attachments or None,
)
if steered is None:
# Lost the running race (turn ended between the two checks) —
# fall through to a normal turn so the message is never lost.
try:
user_msg = await runner.start_turn(
session_id,
content,
mentions=mentions or None,
attachments=attachments or None,
)
except RuntimeError as re_exc:
raise HTTPException(status_code=409, detail=str(re_exc)) from re_exc
except ValueError as ve:
raise HTTPException(status_code=404, detail=str(ve)) from ve
return {"ok": True, "message": user_msg}
return {"ok": True, "steered": True, "message": steered}
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
raise HTTPException(status_code=404, detail="Session not found") from exc
return {"ok": True, "message": user_msg}
+121 -3
View File
@@ -10,6 +10,7 @@ from __future__ import annotations
import json
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from fastapi import APIRouter, HTTPException
@@ -18,6 +19,7 @@ from pydantic import BaseModel, field_validator
from calliope.config import settings
from calliope.db import get_db, row_to_dict
from calliope.events.bus import event_bus
from calliope.routers.playground import PLAYGROUND_STATUS
router = APIRouter()
@@ -825,12 +827,98 @@ async def delete_edge(canvas_id: int, edge_id: int) -> dict[str, bool]:
return {"ok": True}
_PATH_REFERENCE_QUERIES = [
("characters", "portrait_path"),
("characters", "sheet_path"),
("locations", "reference_image_path"),
("items", "reference_image_path"),
("scenes", "video_path"),
("scenes", "env_image_path"),
("clips", "clip_path"),
("shot_capture", "file_path"),
]
def _norm_media_path(path: str) -> str:
return path.replace("\\", "/").replace("'", "''")
def _project_path_referenced(conn, path: str) -> bool:
"""True when any project-data row (entity images, scene/clip videos) points at path."""
norm = _norm_media_path(path)
for table, column in _PATH_REFERENCE_QUERIES:
row = conn.execute(
f"SELECT 1 FROM {table} WHERE replace({column}, '\\', '/') = ? LIMIT 1",
(norm,),
).fetchone()
if row:
return True
return False
def _path_on_canvas(conn, path: str, *, exclude_canvas_node_id: int | None = None) -> bool:
"""True when a live canvas node card points at path."""
sql = (
"SELECT 1 FROM canvas_node WHERE deleted = 0 "
"AND replace(artifact_path, '\\', '/') = ? AND id != ? LIMIT 1"
)
row = conn.execute(
sql, (_norm_media_path(path), exclude_canvas_node_id if exclude_canvas_node_id is not None else -1)
).fetchone()
return row is not None
def _path_is_referenced(conn, path: str, *, exclude_canvas_node_id: int | None = None) -> bool:
"""True when project data OR another live canvas node references path.
Path comparisons normalize separators (\\ → /) so both spellings match.
"""
return _project_path_referenced(conn, path) or _path_on_canvas(
conn, path, exclude_canvas_node_id=exclude_canvas_node_id
)
def _playground_file_roots(conn) -> list[Path]:
"""Folders whose files are safe to hard-delete: the uploads library and
the playground scratch project's output folders. Anything else under
assets_dir is project-owned data."""
roots = [Path(settings.assets_dir) / "uploads"]
row = conn.execute(
"SELECT id FROM projects WHERE status = ? ORDER BY id ASC LIMIT 1",
(PLAYGROUND_STATUS,),
).fetchone()
if row:
scratch = Path(settings.assets_dir) / str(row["id"])
roots.extend([scratch / "image", scratch / "video"])
return roots
def _file_deletable(conn, path: str, roots: list[Path]) -> tuple[bool, str | None]:
"""Decide whether an artifact file may be unlinked. Returns (allowed, reason)."""
try:
target = Path(path).resolve()
target.relative_to(Path(settings.assets_dir).resolve())
except (ValueError, OSError):
return False, "outside_assets"
if not target.is_file():
return True, None # already gone — nothing to do
resolved_roots = [r.resolve() for r in roots]
if not any(
target == root or root in target.parents for root in resolved_roots
):
return False, "project_owned"
return True, None
@router.delete("/{canvas_id}/nodes/{node_id}")
async def delete_node(canvas_id: int, node_id: int) -> dict[str, bool]:
async def delete_node(canvas_id: int, node_id: int) -> dict[str, Any]:
conn = get_db(settings.db_path)
file_deleted = False
reason: str | None = None
try:
row = conn.execute(
"SELECT id, entity_type, entity_id FROM canvas_node WHERE id = ? AND canvas_id = ?",
"SELECT id, type, artifact_path, entity_type, entity_id FROM canvas_node "
"WHERE id = ? AND canvas_id = ?",
(node_id, canvas_id),
).fetchone()
if not row:
@@ -843,8 +931,38 @@ async def delete_node(canvas_id: int, node_id: int) -> dict[str, bool]:
(_now(), node_id),
)
else:
artifact_path = row["artifact_path"]
conn.execute("DELETE FROM canvas_node WHERE id = ?", (node_id,))
if artifact_path:
norm_path = artifact_path.replace("\\", "/")
referenced = _path_is_referenced(
conn, artifact_path, exclude_canvas_node_id=node_id
)
allowed, deny_reason = _file_deletable(
conn, artifact_path, _playground_file_roots(conn)
)
if referenced:
reason = "referenced"
elif not allowed:
reason = deny_reason
else:
target = Path(norm_path)
try:
target.unlink()
file_deleted = True
except OSError:
# Already gone counts as deleted; a real failure (lock,
# permission) leaves the file but still reports the card removal.
file_deleted = not target.exists()
if not file_deleted:
reason = "unlink_failed"
except Exception:
reason = "unlink_failed"
conn.commit()
finally:
conn.close()
return {"ok": True}
await event_bus.publish(
"canvas.updated",
{"canvas_id": canvas_id, "reason": "node_deleted", "node_id": node_id},
)
return {"ok": True, "file_deleted": file_deleted, "reason": reason}
@@ -0,0 +1,157 @@
"""Asset Library — unlinked Playground media (uploads + scratch outputs).
Files that belong to a project (entity references, scene/clip videos) are
NEVER listed here: the library is the delete/attach surface for scratch
media only. Project data is replace-only, never deleted from this page.
"""
from __future__ import annotations
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Literal
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from calliope.config import settings
from calliope.db import get_db
from calliope.routers.canvas import (
_path_is_referenced,
_playground_file_roots,
_project_path_referenced,
)
from calliope.routers.playground import PLAYGROUND_STATUS
router = APIRouter()
MEDIA_KIND_BY_EXT: dict[str, str] = {
".png": "image",
".jpg": "image",
".jpeg": "image",
".webp": "image",
".gif": "image",
".bmp": "image",
".mp4": "video",
".webm": "video",
".mov": "video",
".mkv": "video",
}
def _kind_for_ext(ext: str) -> str | None:
return MEDIA_KIND_BY_EXT.get(ext.lower())
def _scratch_project_id(conn) -> int | None:
row = conn.execute(
"SELECT id FROM projects WHERE status = ? ORDER BY id ASC LIMIT 1",
(PLAYGROUND_STATUS,),
).fetchone()
return int(row["id"]) if row else None
def _library_roots(conn) -> list[Path]:
"""uploads/ first, then the scratch project's output folders."""
roots = [Path(settings.assets_dir) / "uploads"]
pid = _scratch_project_id(conn)
if pid is not None:
scratch = Path(settings.assets_dir) / str(pid)
roots.extend([scratch / "image", scratch / "video"])
return roots
class LibraryDeleteRequest(BaseModel):
paths: list[str]
def _iter_media_files(folder: Path) -> list[Path]:
if not folder.is_dir():
return []
out: list[Path] = []
for f in folder.iterdir():
if f.is_file() and _kind_for_ext(f.suffix) is not None:
out.append(f)
return out
@router.get("/media")
async def list_library_media() -> list[dict[str, Any]]:
"""Unlinked image/video files from uploads/ and the scratch outputs.
A file is 'linked' the moment project data points at it — linked files
are excluded entirely (they live in Project Assets now).
"""
conn = get_db(settings.db_path)
try:
items: list[tuple[float, dict[str, Any]]] = []
for root in _library_roots(conn):
for f in _iter_media_files(root):
path = str(f.resolve())
if _project_path_referenced(conn, path):
continue
stat = f.stat()
items.append(
(
stat.st_mtime,
{
"path": path,
"name": f.name,
"kind": _kind_for_ext(f.suffix),
"size": stat.st_size,
"mtime": datetime.fromtimestamp(
stat.st_mtime, tz=timezone.utc
).isoformat(),
},
)
)
finally:
conn.close()
items.sort(key=lambda e: e[0], reverse=True)
return [item for _, item in items]
@router.delete("/media")
async def delete_library_media(payload: LibraryDeleteRequest) -> dict[str, Any]:
"""Bulk-delete unlinked library files. Referenced/project-owned/foreign
paths are kept and reported per-path."""
if not payload.paths:
raise HTTPException(status_code=400, detail="paths must not be empty")
if len(payload.paths) > 200:
raise HTTPException(status_code=400, detail="Too many paths (max 200 per request)")
conn = get_db(settings.db_path)
try:
allowed_roots = [r.resolve() for r in _library_roots(conn)]
results: list[dict[str, Any]] = []
deleted: list[str] = []
for raw in payload.paths:
path = str(raw or "").strip()
if not path:
continue
try:
target = Path(path).resolve()
target.relative_to(Path(settings.assets_dir).resolve())
except (ValueError, OSError):
results.append({"path": path, "status": "kept", "reason": "outside_assets"})
continue
under_root = any(
target == root or root in target.parents for root in allowed_roots
)
if not under_root:
results.append({"path": path, "status": "kept", "reason": "project_owned"})
continue
if _path_is_referenced(conn, path):
results.append({"path": path, "status": "kept", "reason": "referenced"})
continue
try:
target.unlink()
deleted.append(path)
results.append({"path": path, "status": "deleted"})
except FileNotFoundError:
deleted.append(path)
results.append({"path": path, "status": "deleted"})
except OSError:
results.append({"path": path, "status": "kept", "reason": "unlink_failed"})
finally:
conn.close()
return {"ok": True, "deleted": deleted, "results": results}
@@ -292,6 +292,14 @@ async def list_uploaded_media() -> list[dict[str, Any]]:
if kind is None:
continue
stat = f.stat()
tags = None
sidecar = Path(str(f) + ".meta.json")
if sidecar.is_file():
try:
meta = json.loads(sidecar.read_text(encoding="utf-8"))
tags = meta.get("tags") if isinstance(meta, dict) else None
except (OSError, json.JSONDecodeError):
tags = None
entries.append(
(
stat.st_mtime,
@@ -301,6 +309,7 @@ async def list_uploaded_media() -> list[dict[str, Any]]:
"kind": kind,
"size": stat.st_size,
"mtime": datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc).isoformat(),
"tags": tags,
},
)
)
+224 -2
View File
@@ -9,11 +9,15 @@ PNG renders posted as data URLs and saved under <assets_dir>/shots/ (served by
"""
from __future__ import annotations
import asyncio
import base64
import binascii
import json
import shutil
import tempfile
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from fastapi import APIRouter, HTTPException
@@ -65,6 +69,8 @@ class ShotPatch(BaseModel):
title: str | None = None
scene_json: str | None = None
capture_request_json: str | None = None
# Optimistic concurrency: reject stale autosaves that would clobber agent writes.
base_updated_at: str | None = None
@field_validator("title")
@classmethod
@@ -87,6 +93,37 @@ class CaptureCreate(BaseModel):
return v
def scene_gates(scene: dict[str, Any]) -> dict[str, Any]:
"""Public Brief/Cut gate view from scene_json.gates (Phase 5)."""
raw = scene.get("gates") if isinstance(scene, dict) else None
gates = raw if isinstance(raw, dict) else {}
out: dict[str, Any] = {"brief": None, "cut": None}
for name in ("brief", "cut"):
entry = gates.get(name)
if isinstance(entry, dict) and entry.get("approved_at"):
out[name] = {
"approved_at": entry.get("approved_at"),
"source": entry.get("source") or "agent",
"note": entry.get("note"),
}
return out
def merge_preserved_gates(incoming: dict[str, Any], existing_raw: str | None) -> dict[str, Any]:
"""If a client omits gates on PATCH, keep the server copy."""
if "gates" in incoming:
return incoming
try:
existing = json.loads(existing_raw or "{}")
except json.JSONDecodeError:
return incoming
if isinstance(existing, dict) and isinstance(existing.get("gates"), dict):
incoming = dict(incoming)
incoming["gates"] = existing["gates"]
return incoming
def validate_scene_json(raw: str) -> dict[str, Any]:
"""Light structural validation — the client owns deep validation (posture
shapes, value ranges) via the ported posture module."""
@@ -134,12 +171,21 @@ def _composition_out(row: Any, *, include_scene: bool = False) -> dict[str, Any]
def _capture_out(row: Any) -> dict[str, Any]:
meta = None
raw_meta = row["meta_json"] if "meta_json" in row.keys() else None
if raw_meta:
try:
meta = json.loads(raw_meta)
except (TypeError, json.JSONDecodeError):
meta = {"raw": raw_meta}
return {
"id": row["id"],
"composition_id": row["composition_id"],
"kind": row["kind"],
"label": row["label"],
"file_path": row["file_path"],
"meta_json": raw_meta,
"meta": meta,
"created_at": row["created_at"],
}
@@ -269,10 +315,20 @@ async def patch_shot(shot_id: int, payload: ShotPatch) -> dict[str, Any]:
if payload.title is not None:
sets.append("title = ?")
params.append(payload.title.strip()[:300] or "Untitled composition")
if payload.base_updated_at is not None and payload.base_updated_at != row["updated_at"]:
raise HTTPException(
status_code=409,
detail={
"code": "stale_scene",
"message": "Composition changed on the server (agent or another tab). Reload and retry.",
"updated_at": row["updated_at"],
},
)
if payload.scene_json is not None:
validate_scene_json(payload.scene_json)
incoming = validate_scene_json(payload.scene_json)
incoming = merge_preserved_gates(incoming, row["scene_json"])
sets.append("scene_json = ?")
params.append(payload.scene_json)
params.append(json.dumps(incoming))
if payload.capture_request_json is not None:
if payload.capture_request_json == "":
sets.append("capture_request_json = NULL")
@@ -371,9 +427,63 @@ def _decode_data_url(data_url: str) -> tuple[bytes, str, str]:
return raw, ext, kind
async def _transcode_video_to_mp4(raw: bytes, src_ext: str) -> bytes:
"""Normalize a MediaRecorder capture to H.264 MP4 via ffmpeg.
MediaRecorder's container/codec is browser-dependent: Chromium emits MP4,
Firefox emits WebM/VP8 (and older Safari a .mov). The Playground "From
Build Scene" picker feeds the video stage, so every non-MP4 capture is
re-encoded to libx264 + yuv420p + faststart here — a guaranteed MP4 no
matter which browser recorded the clip. Runs as an async subprocess so the
event loop stays free while ffmpeg encodes.
"""
ffmpeg = shutil.which("ffmpeg")
if not ffmpeg:
raise HTTPException(
status_code=503,
detail="ffmpeg not found on PATH — install ffmpeg to export MP4 clips.",
)
with tempfile.TemporaryDirectory() as tmp:
src = Path(tmp) / f"in.{src_ext}"
dst = Path(tmp) / "out.mp4"
src.write_bytes(raw)
proc = await asyncio.create_subprocess_exec(
ffmpeg,
"-nostdin",
"-y",
"-i",
str(src),
"-c:v",
"libx264",
"-preset",
"medium",
"-crf",
"20",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
"-an",
str(dst),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
_, err = await proc.communicate()
if proc.returncode != 0:
tail = err.decode("utf-8", "replace")[-800:]
raise HTTPException(
status_code=500,
detail=f"ffmpeg could not encode the capture as MP4: {tail}",
)
return dst.read_bytes()
@router.post("/{shot_id}/captures")
async def create_capture(shot_id: int, payload: CaptureCreate) -> dict[str, Any]:
raw, ext, kind = _decode_data_url(payload.data_url)
if kind == "video" and ext != "mp4":
raw = await _transcode_video_to_mp4(raw, ext)
ext = "mp4"
conn = get_db(settings.db_path)
try:
if not conn.execute(
@@ -432,3 +542,115 @@ async def delete_capture(shot_id: int, capture_id: int) -> dict[str, bool]:
finally:
conn.close()
return {"ok": True}
# ---- gates ----------------------------------------------------------------
class GateRecordRequest(BaseModel):
"""UI / agent-equivalent Brief or Cut approval (Phase 5 + P0 UX)."""
gate: str
note: str | None = None
source: str = "ui"
@field_validator("gate")
@classmethod
def _gate_name(cls, v: str) -> str:
g = (v or "").strip().lower()
if g not in ("brief", "cut"):
raise ValueError("gate must be 'brief' or 'cut'")
return g
@field_validator("source")
@classmethod
def _source(cls, v: str) -> str:
s = (v or "ui").strip()[:64] or "ui"
return s
@field_validator("note")
@classmethod
def _note(cls, v: str | None) -> str | None:
if v is None:
return None
n = str(v).strip()[:500]
return n or None
def _load_composition_scene(conn: Any, shot_id: int) -> tuple[Any, dict[str, Any]]:
row = conn.execute(
"SELECT * FROM shot_composition WHERE id = ?", (shot_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Composition not found")
try:
scene = json.loads(row["scene_json"] or "{}")
except json.JSONDecodeError as exc:
raise HTTPException(status_code=422, detail=f"scene_json invalid: {exc}") from exc
if not isinstance(scene, dict):
raise HTTPException(status_code=422, detail="scene_json must be an object")
return row, scene
def _insert_capture_file(
conn: Any,
shot_id: int,
*,
kind: str,
label: str | None,
file_path: Path,
meta: dict[str, Any] | None,
) -> dict[str, Any]:
meta_json = json.dumps(meta) if meta is not None else None
cur = conn.execute(
"""
INSERT INTO shot_capture (composition_id, kind, label, file_path, meta_json, created_at)
VALUES (?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
""",
(shot_id, kind, label, str(file_path), meta_json),
)
conn.execute(
"UPDATE shot_composition SET capture_request_json = NULL, "
"updated_at = CURRENT_TIMESTAMP WHERE id = ?",
(shot_id,),
)
conn.commit()
row = conn.execute(
"SELECT * FROM shot_capture WHERE id = ?", (cur.lastrowid,)
).fetchone()
return _capture_out(row)
@router.post("/{shot_id}/gates")
async def record_scene_gate(shot_id: int, payload: GateRecordRequest) -> dict[str, Any]:
"""Record Brief or Cut approval from the Build Scene UI (source defaults to ui).
Mirrors agent ``record_build_scene_gate`` so a human at the keyboard can
Approve Cut without an ask_user turn.
"""
conn = get_db(settings.db_path)
try:
row, scene = _load_composition_scene(conn, shot_id)
gates = scene.get("gates") if isinstance(scene.get("gates"), dict) else {}
gates = dict(gates)
gates[payload.gate] = {
"approved_at": _now(),
"source": payload.source or "ui",
"note": payload.note,
}
scene["gates"] = gates
conn.execute(
"UPDATE shot_composition SET scene_json = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?",
(json.dumps(scene), shot_id),
)
conn.commit()
finally:
conn.close()
await event_bus.publish(
"shot.updated",
{"shot_id": shot_id, "reason": f"gate_{payload.gate}"},
)
return {
"ok": True,
"gate": payload.gate,
"gates": scene_gates(scene),
}
@@ -200,6 +200,62 @@ def test_derive_llm_history_user_turn_trim(client):
assert session_log.derive_llm_history(events, max_user_turns=0) == []
def test_derive_llm_history_char_budget_drops_oldest_whole_turns(client):
"""max_chars drops oldest WHOLE turns — never mid-turn, and the latest
turn always survives even if alone it exceeds the budget."""
conn = get_db(settings.db_path)
sid = _mk_session(conn)
conn.close()
fat = "x" * 500
for t in range(1, 4): # three turns, each ~1500+ chars
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": f"q{t} {fat}"})
session_log.append_event(
sid,
session_log.ASSISTANT_MESSAGE,
{
"content": "",
"tool_calls": [
{
"id": f"c{t}",
"type": "function",
"function": {"name": "get_workspace", "arguments": "{}"},
}
],
},
)
session_log.append_event(
sid, session_log.TOOL_RESULT, {"call_id": f"c{t}", "tool_name": "get_workspace", "result": {"ok": True, "blob": fat}}
)
session_log.append_event(sid, session_log.ASSISTANT_MESSAGE, {"content": f"a{t}"})
events = session_log.read_events(sid)
# Budget fitting only the last two turns → q1 dropped whole.
two = session_log.derive_llm_history(events, max_chars=2600)
users = [m["content"] for m in two if m["role"] == "user"]
assert [u.split()[0] for u in users] == ["q2", "q3"]
# Pairs intact in what survives.
call_ids = {
tc["id"] for m in two if m["role"] == "assistant" for tc in m.get("tool_calls", [])
}
assert {m["tool_call_id"] for m in two if m["role"] == "tool"} == call_ids
# Budget fitting only the last turn → q1, q2 dropped.
one = session_log.derive_llm_history(events, max_chars=1300)
users = [m["content"] for m in one if m["role"] == "user"]
assert [u.split()[0] for u in users] == ["q3"]
# Budget smaller than any single turn → latest turn passes through
# (an empty request would be worse than an over-budget one).
keep = session_log.derive_llm_history(events, max_chars=10)
assert [m["content"].split()[0] for m in keep if m["role"] == "user"] == ["q3"]
# Budget disabled / oversized → unbounded.
full = session_log.derive_llm_history(events, max_chars=None)
assert len([m for m in full if m["role"] == "user"]) == 3
def test_derive_llm_history_max_turns_exceeds_turn_count(client):
"""max_user_turns larger than the number of user turns must not raise.
@@ -583,6 +639,67 @@ def test_project_user_content_image_attachment_becomes_image_part(tmp_path, monk
assert isinstance(degraded, str)
def test_project_user_content_large_image_attachment_still_becomes_image_part(
tmp_path, monkeypatch
):
"""Regression: a >512 KB PNG must still project as an image_url part.
The 1x1-PNG test above passes through _downscale_image under the budget;
real screenshots exceed _MAX_VISION_IMAGE_BYTES and take the re-encode
branch, which used to swallow its own ImportError (no Pillow installed)
and silently degrade the attachment to a path-only text line — the model
then claimed it could only see a file path.
"""
# Hand-build a valid ~1.2 MB truecolor PNG (no PIL in the test env).
# Seeded pseudo-random rows keep zlib from collapsing the file.
import random
import struct
import zlib
from calliope.config import settings as cfg
width, height = 512, 512
rng = random.Random(1234)
raw = b"".join(
b"\x00" + rng.randbytes(width * 3) for _ in range(height)
)
def _chunk(tag: bytes, data: bytes) -> bytes:
return (
struct.pack(">I", len(data))
+ tag
+ data
+ struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)
)
ihdr = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
png = (
b"\x89PNG\r\n\x1a\n"
+ _chunk(b"IHDR", ihdr)
+ _chunk(b"IDAT", zlib.compress(raw))
+ _chunk(b"IEND", b"")
)
assert len(png) > 512_000
img = tmp_path / "big-screenshot.png"
img.write_bytes(png)
monkeypatch.setattr(cfg, "assets_dir", tmp_path)
content = session_log.project_user_content(
"look at this stupid",
attachments=[{"path": str(img), "name": "file.png", "kind": "image"}],
)
assert isinstance(content, list), (
"large image degraded to a text-only path line — downscale failed "
"silently (is Pillow installed?)"
)
image_parts = [p for p in content if p.get("type") == "image_url"]
assert image_parts, "no image_url part projected for a large PNG"
url = image_parts[0]["image_url"]["url"]
assert url.startswith("data:image/"), url[:60]
assert len(url) <= 700_000 # ~512 KB budget + base64 overhead
def test_project_user_content_document_attachment_becomes_script_text(tmp_path, monkeypatch):
""".txt/.md/.docx attachments are read and injected between delimiters —
the script-upload path. Degradation mirrors image/video handling."""
@@ -0,0 +1,156 @@
"""Asset Library — unlinked Playground media listing + bulk delete."""
from __future__ import annotations
from pathlib import Path
import calliope.config as config_module
from calliope.db import get_db
def _uploads_dir() -> Path:
d = Path(config_module.settings.assets_dir) / "uploads"
d.mkdir(parents=True, exist_ok=True)
return d
def _scratch_project() -> int:
conn = get_db(config_module.settings.db_path)
try:
conn.execute(
"INSERT INTO projects (title, idea, status) VALUES ('Playground Scratch', NULL, 'system')"
)
conn.commit()
row = conn.execute(
"SELECT id FROM projects WHERE status = 'system' LIMIT 1"
).fetchone()
return int(row["id"])
finally:
conn.close()
def _write_image(folder: Path, name: str, data: bytes = b"png") -> Path:
folder.mkdir(parents=True, exist_ok=True)
f = folder / name
f.write_bytes(data)
return f
def test_list_shows_unlinked_files_newest_first(client):
up = _uploads_dir()
f1 = _write_image(up, "aaaa1111-old.png")
f2 = _write_image(up, "bbbb2222-new.png")
import os
os.utime(f1, (1000, 1000))
os.utime(f2, (2000, 2000))
r = client.get("/api/library/media")
assert r.status_code == 200
items = r.json()
paths = [i["path"] for i in items]
assert str(f2) in paths and str(f1) in paths
assert paths.index(str(f2)) < paths.index(str(f1)), "newest first"
kinds = {i["path"]: i["kind"] for i in items}
assert kinds[str(f1)] == "image"
def test_list_excludes_documents_audio_and_project_outputs(client):
up = _uploads_dir()
_write_image(up, "cccc3333-doc.txt")
_write_image(up, "dddd4444-song.mp3")
pid = client.post("/api/projects", json={"title": "Real", "idea": "x"}).json()["id"]
_write_image(
Path(config_module.settings.assets_dir) / str(pid) / "image",
"job-1-image.png",
)
r = client.get("/api/library/media")
names = [i["name"] for i in r.json()]
assert "cccc3333-doc.txt" not in names
assert "dddd4444-song.mp3" not in names
assert "job-1-image.png" not in names, "project output folders are not library"
def test_linked_files_are_hidden_from_library(client):
"""A file referenced by project data leaves the library (Project Assets owns it)."""
up = _uploads_dir()
f = _write_image(up, "eeee5555-sheet.png")
pid = client.post("/api/projects", json={"title": "Link", "idea": "x"}).json()["id"]
conn = get_db(config_module.settings.db_path)
try:
cur = conn.execute(
"INSERT INTO characters (project_id, name, appearance, sheet_path) VALUES (?, ?, ?, ?)",
(pid, "Kira", "tall", str(f)),
)
conn.commit()
assert cur.lastrowid
finally:
conn.close()
r = client.get("/api/library/media")
assert str(f) not in [i["path"] for i in r.json()], "linked file is not library media"
def test_bulk_delete_removes_unlinked_files(client):
up = _uploads_dir()
f1 = _write_image(up, "ffff6666-a.png")
f2 = _write_image(up, "aaaa7777-b.mp4")
r = client.request(
"DELETE",
"/api/library/media",
json={"paths": [str(f1), str(f2)]},
)
assert r.status_code == 200
body = r.json()
assert body["ok"] is True
assert set(body["deleted"]) == {str(f1), str(f2)}
assert not f1.exists() and not f2.exists()
def test_bulk_delete_keeps_referenced_and_foreign_paths(client):
up = _uploads_dir()
keep_me = _write_image(up, "bbbb8888-keep.png")
pid = client.post("/api/projects", json={"title": "Guard", "idea": "x"}).json()["id"]
conn = get_db(config_module.settings.db_path)
try:
conn.execute(
"INSERT INTO characters (project_id, name, appearance, portrait_path) VALUES (?, ?, ?, ?)",
(pid, "Yun", "rival", str(keep_me)),
)
conn.commit()
finally:
conn.close()
outside = Path(config_module.settings.data_dir) / "evil.png"
outside.write_bytes(b"x")
project_file = _write_image(
Path(config_module.settings.assets_dir) / str(pid) / "image",
"job-2-image.png",
)
r = client.request(
"DELETE",
"/api/library/media",
json={"paths": [str(keep_me), str(outside), str(project_file)]},
)
assert r.status_code == 200
results = {res["path"]: res for res in r.json()["results"]}
assert results[str(keep_me)]["status"] == "kept"
assert results[str(keep_me)]["reason"] == "referenced"
assert results[str(outside)]["status"] == "kept"
assert results[str(outside)]["reason"] == "outside_assets"
assert results[str(project_file)]["status"] == "kept"
assert results[str(project_file)]["reason"] == "project_owned"
assert keep_me.exists() and project_file.exists()
def test_delete_missing_file_counts_as_deleted(client):
up = _uploads_dir()
ghost = up / "cccc9999-ghost.png" # never written
r = client.request("DELETE", "/api/library/media", json={"paths": [str(ghost)]})
assert r.status_code == 200
assert ghost in [Path(p) for p in r.json()["deleted"]]
def test_delete_empty_paths_rejected(client):
r = client.request("DELETE", "/api/library/media", json={"paths": []})
assert r.status_code == 400
@@ -0,0 +1,150 @@
"""Build Scene Brief/Cut gates (shot-composer-blockout harness)."""
from __future__ import annotations
import asyncio
import tempfile
from pathlib import Path
import pytest
from calliope.config import settings
from calliope.db import get_db, migrate_db
from calliope.agent.harness.registry import ToolContext
@pytest.fixture(autouse=True, scope="module")
def _never_touch_real_db():
prev = settings.data_dir
with tempfile.TemporaryDirectory() as tmp:
settings.data_dir = Path(tmp)
settings.assets_dir = Path(tmp) / "assets"
asyncio.run(migrate_db(settings.db_path))
yield
settings.data_dir = prev
def _mk_session() -> int:
conn = get_db(settings.db_path)
try:
cur = conn.execute(
"INSERT INTO agent_sessions (title, project_id) VALUES (?, NULL)",
("gate-test",),
)
conn.commit()
return int(cur.lastrowid)
finally:
conn.close()
def _run(sid: int, tool: str, args: dict | None = None):
from calliope.agent.harness.tools import TOOLS
t = TOOLS[tool]
return asyncio.run(
t.executor(ToolContext(session_id=sid, project_id=None), args or {})
)
def test_gate_tools_visible_in_sandbox():
from calliope.agent.harness.tools import openai_tools_payload
ctx = ToolContext(session_id=_mk_session(), project_id=None)
names = {e["function"]["name"] for e in openai_tools_payload(ctx)}
assert "record_build_scene_gate" in names
assert "get_build_scene_gates" in names
assert "get_scene" in names
def test_get_scene_surfaces_gates_empty():
sid = _mk_session()
out = _run(sid, "get_scene")
assert out["ok"] is True
assert out["gates"] == {"brief": None, "cut": None}
def test_record_brief_and_cut_gates():
sid = _mk_session()
brief = _run(
sid, "record_build_scene_gate", {"gate": "brief", "note": "10s city drone"}
)
assert brief["ok"] is True
assert brief["gate"] == "brief"
assert brief["gates"]["brief"]["approved_at"]
assert brief["gates"]["brief"]["source"] == "agent"
assert brief["gates"]["brief"]["note"] == "10s city drone"
assert brief["gates"]["cut"] is None
cut = _run(sid, "record_build_scene_gate", {"gate": "cut"})
assert cut["ok"] is True
assert cut["gates"]["cut"]["approved_at"]
assert cut["gates"]["brief"]["approved_at"]
gates = _run(sid, "get_build_scene_gates")
assert gates["gates"]["brief"]["approved_at"]
assert gates["gates"]["cut"]["approved_at"]
scene = _run(sid, "get_scene")
assert scene["gates"]["cut"]["approved_at"]
def test_mutate_soft_warns_without_brief():
sid = _mk_session()
out = _run(sid, "add_object", {"type": "cube"})
assert out["ok"] is True
assert "warning" in out
assert "brief" in out["warning"].lower()
assert "ask_user" in out["warning"]
assert "Lock the Build Scene brief?" in out["warning"]
assert "instruction" in out
_run(sid, "record_build_scene_gate", {"gate": "brief"})
out2 = _run(sid, "add_object", {"type": "sphere"})
assert out2["ok"] is True
assert "warning" not in out2
def test_clear_scene_preserves_gates():
sid = _mk_session()
_run(sid, "record_build_scene_gate", {"gate": "brief"})
_run(sid, "add_object", {"type": "cube"})
cleared = _run(sid, "clear_scene")
assert cleared["ok"] is True
assert cleared["gates"]["brief"]["approved_at"]
scene = _run(sid, "get_scene")
assert scene["objects"] == []
assert scene["gates"]["brief"]["approved_at"]
def test_patch_preserves_gates_when_client_omits(client):
import json
sid = _mk_session()
shot_id = _run(sid, "get_scene")["shot_id"]
_run(sid, "record_build_scene_gate", {"gate": "cut", "note": "keepme"})
# Client PATCH without gates (legacy autosave shape)
scene = {"objects": [{"id": "o1", "type": "cube", "name": "Cube 01"}]}
resp = client.patch(
f"/api/shots/{shot_id}",
json={"scene_json": json.dumps(scene)},
)
assert resp.status_code == 200, resp.text
body = resp.json()
gates = (body.get("scene") or {}).get("gates") or {}
assert gates.get("cut", {}).get("approved_at")
assert gates["cut"].get("note") == "keepme"
def test_ui_post_gates_records_cut(client):
sid = _mk_session()
shot_id = _run(sid, "get_scene")["shot_id"]
resp = client.post(
f"/api/shots/{shot_id}/gates",
json={"gate": "cut", "source": "ui", "note": "UI approve"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["ok"] is True
assert body["gate"] == "cut"
assert body["gates"]["cut"]["approved_at"]
assert body["gates"]["cut"]["source"] == "ui"
assert body["gates"]["cut"]["note"] == "UI approve"
@@ -0,0 +1,233 @@
"""Canvas node deletion — real delete with file cleanup rules.
Sandbox artifact nodes (uploads/ + scratch outputs) hard-delete their file;
project-owned or still-referenced files survive the node delete.
"""
from __future__ import annotations
from pathlib import Path
import calliope.config as config_module
from calliope.db import get_db
def _seed_story_entities(project_id: int) -> None:
conn = get_db(config_module.settings.db_path)
try:
conn.execute(
"INSERT INTO characters (project_id, name, appearance) VALUES (?, ?, ?)",
(project_id, "Kira", "tall fighter"),
)
conn.execute(
"INSERT INTO scenes (project_id, order_index, heading, action) VALUES (?, ?, ?, ?)",
(project_id, 1, "Round 1", "feint low, switch high"),
)
conn.commit()
finally:
conn.close()
def _ensure_sandbox_canvas(client) -> dict:
sid = client.post("/api/agent/sessions", json={}).json()["id"]
return client.post("/api/canvas", json={"agent_session_id": sid}).json()
def _create_artifact_node(canvas_id: int, path: str, *, job_id: int | None = None) -> int:
conn = get_db(config_module.settings.db_path)
try:
cur = conn.execute(
"""
INSERT INTO canvas_node (canvas_id, type, title, x, y, artifact_path, job_id, status)
VALUES (?, 'image', 'Artifact', 0, 0, ?, ?, 'done')
""",
(canvas_id, path, job_id),
)
conn.commit()
return int(cur.lastrowid)
finally:
conn.close()
def _project_folder(assets_dir: Path, project_id: int, kind: str = "image") -> Path:
d = assets_dir / str(project_id) / kind
d.mkdir(parents=True, exist_ok=True)
return d
def test_sandbox_artifact_delete_removes_file(client, monkeypatch):
"""uploads/ file: node delete unlinks the file (true library delete)."""
assets_dir = config_module.settings.assets_dir
up = assets_dir / "uploads"
up.mkdir(parents=True, exist_ok=True)
f = up / "aaaa1111-shot.png"
f.write_bytes(b"png")
graph = _ensure_sandbox_canvas(client)
canvas_id = graph["canvas"]["id"]
node_id = _create_artifact_node(canvas_id, str(f))
r = client.delete(f"/api/canvas/{canvas_id}/nodes/{node_id}")
assert r.status_code == 200
body = r.json()
assert body["ok"] is True
assert body["file_deleted"] is True
assert not f.exists()
conn = get_db(config_module.settings.db_path)
try:
assert (
conn.execute("SELECT 1 FROM canvas_node WHERE id = ?", (node_id,)).fetchone()
is None
)
finally:
conn.close()
def test_sandbox_scratch_output_delete_removes_file(client):
"""Scratch-project output folders are deletable library files too."""
assets_dir = config_module.settings.assets_dir
# ensure scratch project exists with the system status marker
conn = get_db(config_module.settings.db_path)
try:
conn.execute(
"INSERT INTO projects (title, idea, status) VALUES ('Playground Scratch', NULL, 'system')"
)
conn.commit()
row = conn.execute(
"SELECT id FROM projects WHERE status = 'system' LIMIT 1"
).fetchone()
scratch_pid = int(row["id"])
finally:
conn.close()
f = _project_folder(assets_dir, scratch_pid) / "job-1-image.png"
f.write_bytes(b"png")
graph = _ensure_sandbox_canvas(client)
canvas_id = graph["canvas"]["id"]
node_id = _create_artifact_node(canvas_id, str(f))
r = client.delete(f"/api/canvas/{canvas_id}/nodes/{node_id}")
assert r.status_code == 200
assert r.json()["file_deleted"] is True
assert not f.exists()
def test_project_owned_artifact_file_survives(client):
"""A node pointing at a real project's output: card removed, file kept."""
pid = client.post("/api/projects", json={"title": "Alpha", "idea": "x"}).json()["id"]
assets_dir = config_module.settings.assets_dir
f = _project_folder(assets_dir, pid) / "job-9-image.png"
f.write_bytes(b"png")
graph = client.post("/api/canvas", json={"project_id": pid}).json()
canvas_id = graph["canvas"]["id"]
node_id = _create_artifact_node(canvas_id, str(f))
r = client.delete(f"/api/canvas/{canvas_id}/nodes/{node_id}")
assert r.status_code == 200
body = r.json()
assert body["ok"] is True
assert body["file_deleted"] is False
assert body["reason"] == "project_owned"
assert f.exists(), "project-owned file must never be deleted"
def test_referenced_file_survives(client):
"""File referenced by an entity row: node deleted, file kept."""
pid = client.post("/api/projects", json={"title": "Beta", "idea": "x"}).json()["id"]
_seed_story_entities(pid)
assets_dir = config_module.settings.assets_dir
up = assets_dir / "uploads"
up.mkdir(parents=True, exist_ok=True)
f = up / "bbbb2222-portrait.png"
f.write_bytes(b"png")
conn = get_db(config_module.settings.db_path)
try:
conn.execute(
"UPDATE characters SET portrait_path = ? WHERE project_id = ?",
(str(f), pid),
)
conn.commit()
finally:
conn.close()
graph = client.post("/api/canvas", json={"project_id": pid}).json()
canvas_id = graph["canvas"]["id"]
node_id = _create_artifact_node(canvas_id, str(f))
r = client.delete(f"/api/canvas/{canvas_id}/nodes/{node_id}")
assert r.status_code == 200
body = r.json()
assert body["file_deleted"] is False
assert body["reason"] == "referenced"
assert f.exists()
def test_entity_node_delete_tombstones(client):
"""Entity nodes keep tombstone semantics (no resurrection by auto-seed)."""
pid = client.post("/api/projects", json={"title": "Gamma", "idea": "x"}).json()["id"]
_seed_story_entities(pid)
graph = client.post("/api/canvas", json={"project_id": pid}).json()
canvas_id = graph["canvas"]["id"]
entity_node_id = next(
n["id"] for n in graph["nodes"] if n["type"] == "entity" and n["entity_type"] == "character"
)
r = client.delete(f"/api/canvas/{canvas_id}/nodes/{entity_node_id}")
assert r.status_code == 200
assert r.json()["file_deleted"] is False
conn = get_db(config_module.settings.db_path)
try:
row = conn.execute(
"SELECT deleted FROM canvas_node WHERE id = ?", (entity_node_id,)
).fetchone()
assert row is not None and row["deleted"] == 1
finally:
conn.close()
def test_delete_unknown_node_404(client):
sid = client.post("/api/agent/sessions", json={}).json()["id"]
g = client.post("/api/canvas", json={"agent_session_id": sid}).json()
r = client.delete(f"/api/canvas/{g['canvas']['id']}/nodes/99999")
assert r.status_code == 404
def test_delete_publishes_canvas_updated(client, monkeypatch):
"""Other tabs sync: node deletion publishes canvas.updated."""
events: list[tuple[str, dict]] = []
async def fake_publish(event_type: str, data: dict) -> None:
events.append((event_type, data))
import calliope.events.bus as bus_mod
monkeypatch.setattr(bus_mod.event_bus, "publish", fake_publish)
graph = _ensure_sandbox_canvas(client)
canvas_id = graph["canvas"]["id"]
node_id = _create_artifact_node(canvas_id, "")
r = client.delete(f"/api/canvas/{canvas_id}/nodes/{node_id}")
assert r.status_code == 200
updates = [d for (t, d) in events if t == "canvas.updated"]
assert updates and updates[-1]["reason"] == "node_deleted"
def test_node_delete_keeps_file_referenced_by_other_canvas_node(client):
"""Two cards sharing one file: deleting one keeps the file (other card)."""
assets_dir = config_module.settings.assets_dir
up = assets_dir / "uploads"
up.mkdir(parents=True, exist_ok=True)
f = up / "cccc3333-shared.png"
f.write_bytes(b"png")
graph = _ensure_sandbox_canvas(client)
canvas_id = graph["canvas"]["id"]
node_a = _create_artifact_node(canvas_id, str(f))
node_b = _create_artifact_node(canvas_id, str(f))
r = client.delete(f"/api/canvas/{canvas_id}/nodes/{node_a}")
assert r.status_code == 200
assert r.json()["file_deleted"] is False
assert r.json()["reason"] == "referenced"
assert f.exists(), "file still shown by the second card"
+34 -1
View File
@@ -11,7 +11,10 @@ import pytest
from calliope.config import settings
from calliope.db import get_db, migrate_db
from calliope.agent.harness.registry import GUARD_SCENE_TOOL_SCOPE, ToolContext
from calliope.agent.harness.registry import (
GUARD_SCENE_TOOL_SCOPE,
ToolContext,
)
@pytest.fixture(autouse=True, scope="module")
@@ -190,3 +193,33 @@ def test_recover_orphaned_running_sessions(client):
assert row["status"] == expected, sid
finally:
conn.close()
# ─────────────────────────────────────────────────────────────────────────
# Build Scene harness prompts (Video Scene Build guardrails)
# ─────────────────────────────────────────────────────────────────────────
def test_scene_system_prompt_bans_comfy_and_enforces_gates(client):
"""origin=scene prompt must reinforce Build Scene policy in harness
(not only SKILL.md): shot_* only, no ComfyUI, Brief/Cut, recipes."""
from calliope.agent.harness import get_prompts
prompt = asyncio.run(get_prompts().assemble(_scene_ctx()))
assert "BUILD SCENE POLICY" in prompt
assert "Three.js viewport" in prompt
assert "ComfyUI" in prompt or "comfy_*" in prompt
assert "run_workflow" in prompt
assert "Brief" in prompt or "brief" in prompt
assert "Cut" in prompt or "cut" in prompt
assert "per-frame" in prompt or "recipes" in prompt.lower()
# Scene mode must not push sandbox Comfy generation paths
assert "GENERATING IN SANDBOX" not in prompt
def test_chat_system_prompt_skips_build_scene_policy(client):
from calliope.agent.harness import get_prompts
chat_ctx = ToolContext(session_id=1, project_id=None, origin="chat")
prompt = asyncio.run(get_prompts().assemble(chat_ctx))
assert "BUILD SCENE POLICY" not in prompt
+100 -13
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
import asyncio
import base64
import json
import shutil
import tempfile
from pathlib import Path
@@ -372,10 +373,15 @@ def test_set_playback_updates_and_validates():
# ── captures ───────────────────────────────────────────────────────────────
def test_capture_request_poll_and_fulfill(client):
def test_capture_request_poll_and_fulfill(client, monkeypatch):
sid = _mk_session()
comp_id = _run(sid, "get_scene")["shot_id"]
# request_capture asks the UI Three.js viewport to render the PNG: it
# stamps capture_request_json, then polls until the router clears it and
# a capture row lands (the UI POST /captures path, which this fulfills
# directly in the DB below).
# Fulfillment races the poll: post the capture in a background task
import threading
import time
@@ -407,6 +413,21 @@ def test_capture_request_poll_and_fulfill(client):
assert out["file_path"] == "x.png"
def test_capture_timeout_is_failure_not_false_success(monkeypatch):
"""When the Build Scene page never lands the PNG, the tool must report
failure — an ok:True with no file_path invited the agent to fabricate a
reference path (and fed no signal to the fail-streak guard)."""
from calliope.agent.harness.plugins import shot_builder
monkeypatch.setattr(shot_builder, "CAPTURE_POLL_TIMEOUT_S", 0.2)
sid = _mk_session()
out = _run(sid, "request_capture", {"label": "nobody home"})
assert out["ok"] is False
assert out.get("capture_pending") is True
assert "file_path" not in out
assert "do NOT reference a file path" in out["error"]
def test_capture_upload_via_router(client):
sid = _mk_session()
comp_id = _run(sid, "get_scene")["shot_id"]
@@ -529,28 +550,94 @@ def test_set_joint_no_empty_arrays_regression():
def _video_data_url() -> str:
# Not a real codec payload — the router only validates the MIME + base64.
# Not a real codec payload — used where only the MIME + base64 matter
# (the ffmpeg-missing 503 path never reaches the encoder).
raw = b"\x1aE\xdf\xa3fake-ebml-header" + b"0" * 256
return "data:video/webm;base64," + base64.b64encode(raw).decode()
def test_video_capture_upload_via_router(client):
def _mp4_data_url() -> str:
# Fake MP4 header — the passthrough path stores it without ffmpeg.
raw = b"\x00\x00\x00\x18ftypmp42" + b"0" * 256
return "data:video/mp4;base64," + base64.b64encode(raw).decode()
def _mov_data_url() -> str:
"""A real 1-frame QuickTime clip (native mpeg4 codec, no external libs) —
the transcode path needs a decodable video, unlike the fake-byte MIME
checks above."""
import subprocess
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp) / "tiny.mov"
subprocess.run(
[
"ffmpeg",
"-nostdin",
"-y",
"-f",
"lavfi",
"-i",
"testsrc=duration=0.5:size=64x64:rate=10",
"-c:v",
"mpeg4",
str(out),
],
check=True,
capture_output=True,
)
raw = out.read_bytes()
return "data:video/quicktime;base64," + base64.b64encode(raw).decode()
def test_video_capture_mp4_passthrough(client):
# Chromium MediaRecorder already emits MP4 — store it untouched (no ffmpeg).
sid = _mk_session()
comp_id = _run(sid, "get_scene")["shot_id"]
resp = client.post(
f"/api/shots/{comp_id}/captures",
json={"data_url": _mp4_data_url(), "label": "camera export"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["kind"] == "video"
assert Path(body["file_path"]).exists()
assert Path(body["file_path"]).suffix == ".mp4"
assert Path(body["file_path"]).resolve().is_relative_to(Path(settings.assets_dir).resolve())
listing = client.get(f"/api/shots/{comp_id}/captures").json()
mine = next(c for c in listing if c["id"] == body["id"])
assert mine["kind"] == "video"
@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not installed")
def test_video_capture_transcoded_to_mp4(client):
# Firefox/WebM (or mov) must land in the picker as a real MP4.
sid = _mk_session()
comp_id = _run(sid, "get_scene")["shot_id"]
resp = client.post(
f"/api/shots/{comp_id}/captures",
json={"data_url": _mov_data_url(), "label": "camera export"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["kind"] == "video"
p = Path(body["file_path"])
assert p.suffix == ".mp4"
assert p.exists()
# Real MP4: starts with an ftyp box, not a passthrough fake header.
assert b"ftyp" in p.read_bytes()[:64]
def test_video_capture_webm_without_ffmpeg_503(client, monkeypatch):
monkeypatch.setattr(shutil, "which", lambda _name: None)
sid = _mk_session()
comp_id = _run(sid, "get_scene")["shot_id"]
resp = client.post(
f"/api/shots/{comp_id}/captures",
json={"data_url": _video_data_url(), "label": "camera export"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["kind"] == "video"
assert Path(body["file_path"]).exists()
assert Path(body["file_path"]).suffix == ".webm"
assert Path(body["file_path"]).resolve().is_relative_to(Path(settings.assets_dir).resolve())
listing = client.get(f"/api/shots/{comp_id}/captures").json()
mine = next(c for c in listing if c["id"] == body["id"])
assert mine["kind"] == "video"
assert resp.status_code == 503
def test_capture_rejects_unknown_mime(client):
+384
View File
@@ -0,0 +1,384 @@
"""Mid-run steering — the user course-corrects a running turn.
Contract pieces under test:
- runner.steer(): appends steering/message + mirrors a status='steering'
chat row while a turn is running; refuses (None) when idle.
- router POST /messages: 409 falls back to steering instead of rejecting;
answer_to while running stays a 409 (answers belong to the next turn).
- derive_llm_history: steering projects as [STEERING] user content, buffers
mid-tool-exchange (never inside a tool_calls/result pair), and never opens
a user-turn boundary or affects policy's latest_user_message.
- run_turn / _run_sub_agent: drained steering text reaches the model at the
next step boundary.
"""
from __future__ import annotations
import asyncio
import json
import tempfile
from pathlib import Path
import pytest
from calliope.agent.harness.registry import ToolContext
from calliope.config import settings
from calliope.db import get_db
@pytest.fixture(autouse=True, scope="module")
def _never_touch_real_db():
prev = settings.data_dir
prev_assets = settings.assets_dir
with tempfile.TemporaryDirectory() as tmp:
settings.data_dir = Path(tmp)
settings.assets_dir = Path(tmp) / "assets"
asyncio.run(_migrate(settings.db_path))
yield
settings.data_dir = prev
settings.assets_dir = prev_assets
async def _migrate(db_path):
from calliope.db import migrate_db
await migrate_db(db_path)
def _mk_session(title: str = "steer-test") -> int:
conn = get_db(settings.db_path)
try:
cur = conn.execute(
"INSERT INTO agent_sessions (title, project_id) VALUES (?, NULL)", (title,)
)
conn.commit()
return int(cur.lastrowid)
finally:
conn.close()
class _FakeTask:
def done(self) -> bool:
return False
def _fake_running(sid: int):
from calliope.agent.harness.runner import runner
runner._tasks[sid] = _FakeTask() # type: ignore[assignment]
return runner
def _events(sid: int):
from calliope.agent.harness import log as session_log
return session_log.read_events(sid)
# ── projection: derive_llm_history ──────────────────────────────────────
def test_steering_projects_as_user_message_with_header():
from calliope.agent.harness import log as session_log
sid = _mk_session()
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": "build the scene"})
session_log.append_event(sid, session_log.STEERING_MESSAGE, {"content": "make it darker"})
history = session_log.derive_llm_history(_events(sid))
steer = [m for m in history if session_log.STEERING_INJECT_HEADER in str(m.get("content"))]
assert steer, "steering never projected"
assert steer[0]["role"] == "user"
assert "make it darker" in str(steer[0]["content"])
def test_steering_never_lands_inside_tool_exchange():
from calliope.agent.harness import log as session_log
sid = _mk_session()
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": "go"})
session_log.append_event(
sid,
session_log.ASSISTANT_MESSAGE,
{"content": None, "tool_calls": [{"id": "c1", "type": "function", "function": {"name": "t", "arguments": "{}"}}]},
)
# Steering lands AFTER the call, BEFORE its result — must buffer.
session_log.append_event(sid, session_log.STEERING_MESSAGE, {"content": "wait, no"})
session_log.append_event(
sid, session_log.TOOL_RESULT, {"call_id": "c1", "tool_name": "t", "result": {"ok": True}}
)
history = session_log.derive_llm_history(_events(sid))
roles = [m["role"] for m in history]
# No user message between assistant(tool_calls) and its tool result.
i_asst = next(i for i, m in enumerate(history) if m.get("tool_calls"))
assert history[i_asst + 1]["role"] == "tool"
# And it flushed right after the exchange closed.
assert any(session_log.STEERING_INJECT_HEADER in str(m.get("content")) for m in history)
def test_steering_does_not_open_user_turn_boundary():
from calliope.agent.harness import log as session_log
sid = _mk_session()
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": "first turn"})
session_log.append_event(sid, session_log.STEERING_MESSAGE, {"content": "adjust"})
# max_user_turns=1 keeps exactly the last user turn; steering before it
# belongs to that same turn and must survive (not be dropped as its own).
history = session_log.derive_llm_history(_events(sid), max_user_turns=1)
assert any("adjust" in str(m.get("content")) for m in history)
assert sum(1 for m in history if m["role"] == "user") == 2 # the turn message + its steering
def test_steering_not_visible_to_policy_latest_user_message():
from calliope.agent.harness import log as session_log
sid = _mk_session()
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": "plain text only"})
session_log.append_event(sid, session_log.STEERING_MESSAGE, {"content": "yes generate everything"})
assert session_log.latest_user_message(sid) == "plain text only"
def test_chat_rows_include_steering_status():
from calliope.agent.harness import log as session_log
sid = _mk_session()
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": "hi"})
session_log.append_event(sid, session_log.STEERING_MESSAGE, {"content": "actually no"})
rows = session_log.derive_chat_rows(_events(sid))
steer_rows = [r for r in rows if r.get("status") == "steering"]
assert steer_rows and steer_rows[0]["role"] == "user" and steer_rows[0]["content"] == "actually no"
def test_drain_watermark_prevents_redrain():
from calliope.agent.harness import log as session_log
sid = _mk_session()
e1 = session_log.append_event(sid, session_log.STEERING_MESSAGE, {"content": "one"})
e2 = session_log.append_event(sid, session_log.STEERING_MESSAGE, {"content": "two"})
first = session_log.drain_steering(sid, 0)
assert [s.data["content"] for s in first] == ["one", "two"]
assert first[0].seq == e1.seq and first[1].seq == e2.seq
# Already drained past both — nothing new.
assert session_log.drain_steering(sid, e2.seq) == []
# ── runner.steer ────────────────────────────────────────────────────────
def test_runner_steer_appends_event_and_row_when_running():
from calliope.agent.harness.runner import runner
sid = _mk_session()
try:
_fake_running(sid)
row = asyncio.run(runner.steer(sid, "make it moodier"))
finally:
runner._tasks.pop(sid, None)
assert row is not None and row["status"] == "steering" and row["role"] == "user"
types = [e.type for e in _events(sid)]
assert "steering/message" in types
# No turn/start, no user/message — steering never starts a turn.
assert "turn/start" not in types
assert "user/message" not in types
def test_runner_steer_refuses_when_idle():
from calliope.agent.harness.runner import runner
sid = _mk_session()
assert asyncio.run(runner.steer(sid, "hello")) is None
assert all(e.type != "steering/message" for e in _events(sid))
# ── router fallback ─────────────────────────────────────────────────────
def test_post_message_while_running_steers_instead_of_409(client):
from calliope.agent.harness.runner import runner
r = client.post("/api/agent/sessions", json={})
sid = r.json()["id"]
try:
_fake_running(sid)
resp = client.post(
f"/api/agent/sessions/{sid}/messages", json={"content": "switch to wide shot"}
)
finally:
runner._tasks.pop(sid, None)
assert resp.status_code == 200
body = resp.json()
assert body["steered"] is True
assert body["message"]["status"] == "steering"
def test_post_message_answer_to_while_running_stays_409(client):
from calliope.agent.harness.runner import runner
r = client.post("/api/agent/sessions", json={})
sid = r.json()["id"]
try:
_fake_running(sid)
resp = client.post(
f"/api/agent/sessions/{sid}/messages",
json={"content": "yes", "answer_to": 3},
)
finally:
runner._tasks.pop(sid, None)
assert resp.status_code == 409
# ── loop injection ──────────────────────────────────────────────────────
class _SteeringAwareClient:
"""chat_stream that records requests; steers once after the first step."""
def __init__(self, replies):
self.replies = list(replies)
self.requests: list[list[dict]] = []
def chat_stream(self, messages, temperature=0.4, tools=None):
self.requests.append(list(messages))
reply = self.replies.pop(0)
return _StreamReply(reply)
async def close(self):
pass
class _StreamReply:
def __init__(self, chunks):
self._chunks = [c for c in chunks if c]
def __aiter__(self):
self._iter = iter(self._chunks)
return self
async def __anext__(self):
try:
return next(self._iter)
except StopIteration:
raise StopAsyncIteration
def _tc(name: str, call_id: str):
return {
"id": call_id,
"type": "function",
"function": {"name": name, "arguments": "{}"},
}
def test_run_turn_drains_steering_at_step_boundary():
import calliope.agent.harness.loop as loop_mod
from calliope.agent.harness import log as session_log
sid = _mk_session()
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": "frame the shot"})
client_llm = _SteeringAwareClient(
[
[{"type": "tool_call", "tool_call": _tc("get_scene", "c1")}],
[{"type": "delta", "content": "done"}],
]
)
class _Reg:
async def execute(self, ctx, name, args):
# Steering lands mid-step (tool executing) — the drain happens at
# the NEXT step boundary, after this result is appended.
session_log.append_event(
sid, session_log.STEERING_MESSAGE, {"content": "use a low angle instead"}
)
return {"ok": True}
def openai_payload(self, ctx):
return []
async def _run():
orig_llm = loop_mod.LLMClient
orig_reg = loop_mod.get_registry
orig_prompts = loop_mod.get_prompts
loop_mod.LLMClient = lambda: client_llm
loop_mod.get_registry = lambda: _Reg()
class _P:
async def assemble(self, ctx):
return "system"
loop_mod.get_prompts = lambda: _P()
try:
return await loop_mod.run_turn(
ToolContext(session_id=sid, project_id=None), []
)
finally:
loop_mod.LLMClient = orig_llm
loop_mod.get_registry = orig_reg
loop_mod.get_prompts = orig_prompts
final = asyncio.run(_run())
assert final == "done"
# Second request (step 2) must contain the steering text, positioned
# after the tool result (never between tool_calls and its result).
second = client_llm.requests[1]
texts = [str(m.get("content")) for m in second]
assert any("use a low angle instead" in t for t in texts)
# Find position: must come after the tool message.
tool_idx = next(i for i, m in enumerate(second) if m.get("role") == "tool")
steer_idx = next(i for i, t in enumerate(texts) if "use a low angle" in t)
assert steer_idx > tool_idx
def test_sub_agent_loop_drains_steering():
import calliope.agent.harness.orchestrator as orch
from calliope.agent.harness import log as session_log
sid = _mk_session()
class _Scripted:
def __init__(self):
self.seen: list[list[dict]] = []
async def chat_with_tools(self, messages, temperature=0.7, tools=None, tool_choice=None):
self.seen.append(list(messages))
if len(self.seen) == 1:
return {"role": "assistant", "content": None, "tool_calls": [_tc("get_workspace", "a")]}
return {"role": "assistant", "content": "adjusted", "tool_calls": []}
async def close(self):
pass
scripted = _Scripted()
class _Reg:
async def execute(self, ctx, name, args):
session_log.append_event(
sid, session_log.STEERING_MESSAGE, {"content": "skip the wide shots"}
)
return {"ok": True}
def get(self, name):
return None
def _visible(self, t, ctx):
return True
async def _run():
orig_c, orig_r = orch.LLMClient, orch.get_registry
orch.LLMClient = lambda: scripted
orch.get_registry = lambda: _Reg()
try:
return await orch._run_sub_agent(
ToolContext(session_id=sid, project_id=1),
[{"role": "user", "content": "do"}],
["get_workspace"],
agent_name="script-agent",
max_iterations=4,
)
finally:
orch.LLMClient = orig_c
orch.get_registry = orig_r
final = asyncio.run(_run())
assert final == "adjusted"
second = scripted.seen[1]
assert any("skip the wide shots" in str(m.get("content")) for m in second)
@@ -0,0 +1,387 @@
"""Sub-agent loop safety nets — the swarm must not be a second-class citizen.
Covers the four nets ported from run_turn into _run_sub_agent:
repeat guard, fail-streak directive, final-step budget nudge, and ask_user
escalation (which pauses the whole swarm in orchestrate).
"""
from __future__ import annotations
import asyncio
import tempfile
from pathlib import Path
import pytest
from calliope.agent.harness.registry import ToolContext
from calliope.config import settings
from calliope.db import get_db
@pytest.fixture(autouse=True, scope="module")
def _never_touch_real_db():
prev = settings.data_dir
with tempfile.TemporaryDirectory() as tmp:
settings.data_dir = Path(tmp)
settings.assets_dir = Path(tmp) / "assets"
asyncio.run(_migrate(settings.db_path))
yield
settings.data_dir = prev
async def _migrate(db_path):
from calliope.db import migrate_db
await migrate_db(db_path)
def _mk_session() -> int:
conn = get_db(settings.db_path)
try:
cur = conn.execute(
"INSERT INTO agent_sessions (title, project_id) VALUES (?, NULL)",
("subagent-guard-test",),
)
conn.commit()
return int(cur.lastrowid)
finally:
conn.close()
def _tc(name: str, args: str = "{}", call_id: str = "c1"):
return {
"id": call_id,
"type": "function",
"function": {"name": name, "arguments": args},
}
class _ScriptedClient:
"""chat_with_tools that replays scripted responses, one per call."""
def __init__(self, responses):
self._responses = list(responses)
self.seen_messages: list[dict] = []
async def chat_with_tools(self, messages, temperature=0.7, tools=None, tool_choice=None):
self.seen_messages.append(list(messages))
if self._responses:
return self._responses.pop(0)
return {"role": "assistant", "content": "done", "tool_calls": []}
async def close(self):
pass
class _StubRegistry:
"""Registry stub: counts executions, returns scripted results per tool."""
def __init__(self, results_by_tool=None):
self.executed: list[tuple[str, dict]] = []
self.results_by_tool = results_by_tool or {}
def get(self, name):
return None
async def execute(self, ctx, name, args):
self.executed.append((name, dict(args)))
r = self.results_by_tool.get(name)
return dict(r) if isinstance(r, dict) else {"ok": True}
def _install(orch, client, registry):
"""Patch the orchestrator: LLMClient is a FACTORY (called with no args),
so wrap the scripted client in a lambda."""
orch.LLMClient = lambda: client
orch.get_registry = lambda: registry
def test_repeat_guard_blocks_reexecution():
"""After REPEAT_EXECUTE_LIMIT identical calls, the sub-agent gets the
cached result back with the repeat_guard directive — not another
execution burning the step budget."""
import calliope.agent.harness.orchestrator as orch
sid = _mk_session()
# Five model turns, all asking for the same tool+args.
client = _ScriptedClient(
[{"role": "assistant", "content": None, "tool_calls": [_tc("get_workspace", call_id=f"c{i}")]}
for i in range(4)]
+ [{"role": "assistant", "content": "gave up", "tool_calls": []}]
)
registry = _StubRegistry()
orig_client, orig_reg = orch.LLMClient, orch.get_registry
_install(orch, client, registry)
try:
final = asyncio.run(
orch._run_sub_agent(
ToolContext(session_id=sid, project_id=1),
[{"role": "user", "content": "do"}],
["get_workspace"],
agent_name="script-agent",
max_iterations=6,
)
)
finally:
orch.LLMClient, orch.get_registry = orig_client, orig_reg
assert final == "gave up"
# Executed for real only 2 times (limit=2); later repeats intercepted.
assert len(registry.executed) == 2
# The directive reached the model on the intercepted call.
repeat_msgs = [
m for batch in client.seen_messages
for m in batch
if m.get("role") == "tool" and "repeat_guard" in (m.get("content") or "")
]
assert repeat_msgs, "repeat_guard directive never reached the model"
def test_fail_streak_directive_appended():
"""Two not-ok results in a row append the [SYSTEM DIRECTIVE] change-course
text to the tool result the model sees."""
import calliope.agent.harness.orchestrator as orch
sid = _mk_session()
client = _ScriptedClient(
[{"role": "assistant", "content": None, "tool_calls": [_tc("run_workflow", call_id="a")]}]
+ [{"role": "assistant", "content": None, "tool_calls": [_tc("run_workflow", call_id="b")]}]
+ [{"role": "assistant", "content": None, "tool_calls": [_tc("get_workspace", call_id="c")]}]
+ [{"role": "assistant", "content": "changed course", "tool_calls": []}]
)
registry = _StubRegistry({"run_workflow": {"ok": False, "error": "blocked"}})
orig_client, orig_reg = orch.LLMClient, orch.get_registry
_install(orch, client, registry)
try:
final = asyncio.run(
orch._run_sub_agent(
ToolContext(session_id=sid, project_id=1),
[{"role": "user", "content": "do"}],
["run_workflow", "get_workspace"],
agent_name="assets-agent",
max_iterations=6,
)
)
finally:
orch.LLMClient, orch.get_registry = orig_client, orig_reg
assert final == "changed course"
tool_texts = [m.get("content") or "" for batch in client.seen_messages for m in batch if m.get("role") == "tool"]
assert any("[SYSTEM DIRECTIVE]" in t for t in tool_texts)
def test_final_step_nudge_present():
"""On the last iteration the model is told to wrap up, not start work."""
import calliope.agent.harness.orchestrator as orch
sid = _mk_session()
client = _ScriptedClient(
[{"role": "assistant", "content": None, "tool_calls": [_tc("get_workspace")]}]
+ [{"role": "assistant", "content": "wrapping", "tool_calls": []}]
)
registry = _StubRegistry()
orig_client, orig_reg = orch.LLMClient, orch.get_registry
_install(orch, client, registry)
try:
final = asyncio.run(
orch._run_sub_agent(
ToolContext(session_id=sid, project_id=1),
[{"role": "user", "content": "do"}],
["get_workspace"],
agent_name="script-agent",
max_iterations=2,
)
)
finally:
orch.LLMClient, orch.get_registry = orig_client, orig_reg
assert final == "wrapping"
last_batch = client.seen_messages[-1]
assert any(
m.get("role") == "user" and "FINAL step" in (m.get("content") or "")
for m in last_batch
), "final-step nudge missing from the last model request"
def test_ask_user_pauses_sub_agent():
"""An ask_user result ends the sub-agent immediately with the Paused
prefix — no more model calls, no more executions."""
import calliope.agent.harness.orchestrator as orch
sid = _mk_session()
client = _ScriptedClient(
[{"role": "assistant", "content": None, "tool_calls": [_tc("ask_user", '{"question": "Generate?"}')]}]
+ [{"role": "assistant", "content": "never reached", "tool_calls": []}]
)
registry = _StubRegistry(
{"ask_user": {"ok": True, "awaiting_user_input": True, "question": "Generate?"}}
)
orig_client, orig_reg = orch.LLMClient, orch.get_registry
_install(orch, client, registry)
try:
final = asyncio.run(
orch._run_sub_agent(
ToolContext(session_id=sid, project_id=1),
[{"role": "user", "content": "do"}],
["ask_user"],
agent_name="assets-agent",
max_iterations=6,
)
)
finally:
orch.LLMClient, orch.get_registry = orig_client, orig_reg
assert final.startswith("Paused: ")
assert "Generate?" in final
# The scripted "never reached" response stayed in the queue — only one
# model call happened.
assert len(client.seen_messages) == 1
def test_ask_user_visible_in_every_role():
"""Every ROLE_TOOLS list carries ask_user so a stuck sub-agent can
escalate instead of spinning."""
from calliope.agent.harness.orchestrator import ROLE_TOOLS
for role, tools in ROLE_TOOLS.items():
assert "ask_user" in tools, f"{role} cannot escalate"
def test_clip_tools_in_swarm_roles():
from calliope.agent.harness.orchestrator import ROLE_TOOLS
for tool in ("list_clips", "break_into_shots", "add_clip", "update_clip", "delete_clip"):
assert tool in ROLE_TOOLS["script"], f"{tool} missing from script role"
assert "list_clips" in ROLE_TOOLS["video"]
# ── per-tool wall-clock timeout ─────────────────────────────────────────
def test_tool_timeout_kills_hung_tool():
"""A tool slower than agent_tool_timeout_sec returns ok:False with the
timeout marker instead of stalling the step forever."""
import asyncio as aio
from calliope.agent.harness.registry import ToolDefinition, ToolRegistry
async def _hang(ctx, args):
await aio.sleep(30)
return {"ok": True}
async def _fast(ctx, args):
return {"ok": True}
reg = ToolRegistry()
reg.register(
ToolDefinition(name="hang", description="", parameters={}, executor=_hang, requires_project=False)
)
reg.register(
ToolDefinition(
name="exempt",
description="",
parameters={},
executor=_hang,
requires_project=False,
long_running=True,
)
)
reg.register(
ToolDefinition(name="fast", description="", parameters={}, executor=_fast, requires_project=False)
)
orig = settings.agent_tool_timeout_sec
try:
settings.agent_tool_timeout_sec = 0.2
out = aio.run(reg.execute(ToolContext(session_id=1), "hang", {}))
assert out["ok"] is False
assert out.get("timeout") is True
assert "timed out" in out["error"]
# long_running tools are exempt (they own their wait contract).
# (Not awaiting the hang: exemption means wait_for is not applied at
# all — verified structurally by _tool_timeout_sec + the fast path.)
out_fast = aio.run(reg.execute(ToolContext(session_id=1), "fast", {}))
assert out_fast["ok"] is True
# 0 disables the timeout entirely.
settings.agent_tool_timeout_sec = 0
from calliope.agent.harness.registry import _tool_timeout_sec
assert _tool_timeout_sec() == 0.0
finally:
settings.agent_tool_timeout_sec = orig
def test_wait_for_jobs_is_long_running():
"""wait_for_jobs owns its timeout contract (timeout_sec →
queue_poll_timeout_sec) — it must be exempt from the wall-clock cap."""
from calliope.agent.harness.tools import TOOLS
assert TOOLS["wait_for_jobs"].long_running is True
# ── planner fast-path ───────────────────────────────────────────────────
def test_is_trivial_goal_classification():
from calliope.agent.harness.orchestrator import _is_trivial_goal
assert _is_trivial_goal("rename character 3 to Kira")
assert _is_trivial_goal("What's the queue status?")
assert _is_trivial_goal("show me scene 2")
assert _is_trivial_goal("LIST WORKFLOWS")
# Multi-step / long / procedural messages must NOT skip the planner.
assert not _is_trivial_goal("draft a new storyline and then regenerate the script")
assert not _is_trivial_goal("")
assert not _is_trivial_goal("x" * 200)
assert not _is_trivial_goal("first do this;\nthen do that")
def test_trivial_goal_skips_planner():
"""A one-line imperative goes straight to run_turn — no planner LLM call,
no workspace read."""
import calliope.agent.harness.orchestrator as orch
sid = _mk_session()
from calliope.agent.harness import log as session_log
session_log.append_event(sid, session_log.USER_MESSAGE, {"content": "rename character 3 to Kira"})
planner_calls = {"n": 0}
class _PlannerSpy:
def __init__(self):
planner_calls["n"] += 1
async def chat(self, *a, **kw):
raise AssertionError("planner must not be called for a trivial goal")
async def close(self):
return None
async def fake_run_turn(ctx, history, *, on_message=None):
return "single-loop answer"
orig_client = orch.LLMClient
orig_run_turn = orch.run_turn
orig_ws = orch.get_registry
orch.LLMClient = _PlannerSpy
orch.run_turn = fake_run_turn
class _NoWorkspace:
def get(self, name):
return None
async def execute(self, ctx, name, args):
raise AssertionError("workspace read must be skipped for a trivial goal")
orch.get_registry = lambda: _NoWorkspace()
try:
ctx = ToolContext(session_id=sid, project_id=1)
out = asyncio.run(orch.orchestrate(ctx, [], session_id=sid))
finally:
orch.LLMClient = orig_client
orch.run_turn = orig_run_turn
orch.get_registry = orig_ws
assert out == "single-loop answer"
assert planner_calls["n"] == 0
@@ -87,6 +87,53 @@ def test_assets_role_scope_includes_canvas_posting():
assert registry.get(name) is not None, f"{role} names unknown tool {name}"
def test_swarm_roles_cover_pipeline_categories():
"""Drift guard (both directions): every registered tool in a pipeline
category must appear in its ROLE_TOOLS role — the clip tools drifted out
of the script/video roles while the dead-name check above stayed green,
because nothing checked the orphan direction.
Deliberately role-less categories are excluded: canvas / interaction /
memory / skills / shot / workspace / project / system are main-loop-only
or shared base reads.
"""
from calliope.agent.harness import build_harness
registry, _ = build_harness()
category_roles = {
"story": {"story"},
"script": {"script"},
"assets": {"assets"},
"video": {"video"},
}
# Story-category ENTITY CRUD is planner-routed to the assets role (the
# standard EDIT pipeline: story → script → "add/update assets text") —
# only beats stay with the story role.
overrides = {
name: {"assets"}
for name in (
"add_character",
"update_character",
"delete_character",
"add_location",
"update_location",
"delete_location",
"add_item",
"update_item",
"delete_item",
)
}
for t in registry.tools.values():
roles = overrides.get(t.name) or category_roles.get(t.category)
if not roles:
continue
for role in roles:
assert t.name in orchestrator.ROLE_TOOLS[role], (
f"{t.category}-category tool '{t.name}' is missing from the "
f"'{role}' swarm role — sub-agents cannot perform it"
)
def test_unscoped_enqueue_refused_when_many_missing():
"""Mechanical scope: an unscoped enqueue_asset_jobs that would touch
more than 3 entities is refused with the count, so '2 characters' can
+2 -41
View File
@@ -1,12 +1,12 @@
{
"name": "calliope-web",
"version": "1.5.3",
"version": "1.5.4",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "calliope-web",
"version": "1.5.3",
"version": "1.5.4",
"dependencies": {
"@tanstack/svelte-query": "^5.66.0",
"@types/three": "0.185.0",
@@ -885,9 +885,6 @@
"arm"
],
"dev": true,
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -902,9 +899,6 @@
"arm"
],
"dev": true,
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
@@ -919,9 +913,6 @@
"arm64"
],
"dev": true,
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -936,9 +927,6 @@
"arm64"
],
"dev": true,
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
@@ -953,9 +941,6 @@
"loong64"
],
"dev": true,
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -970,9 +955,6 @@
"loong64"
],
"dev": true,
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
@@ -987,9 +969,6 @@
"ppc64"
],
"dev": true,
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1004,9 +983,6 @@
"ppc64"
],
"dev": true,
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1021,9 +997,6 @@
"riscv64"
],
"dev": true,
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1038,9 +1011,6 @@
"riscv64"
],
"dev": true,
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1055,9 +1025,6 @@
"s390x"
],
"dev": true,
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1072,9 +1039,6 @@
"x64"
],
"dev": true,
"libc": [
"glibc"
],
"license": "MIT",
"optional": true,
"os": [
@@ -1089,9 +1053,6 @@
"x64"
],
"dev": true,
"libc": [
"musl"
],
"license": "MIT",
"optional": true,
"os": [
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "calliope-web",
"version": "1.5.3",
"version": "1.5.4",
"private": true,
"scripts": {
"dev": "vite dev",
@@ -30,10 +30,10 @@
},
"dependencies": {
"@tanstack/svelte-query": "^5.66.0",
"@types/three": "0.185.0",
"@xyflow/svelte": "^1.6.5",
"mannequin-js": "5.2.3",
"three": "0.185.1",
"@types/three": "0.185.0"
"three": "0.185.1"
},
"type": "module"
}
+32 -3
View File
@@ -393,6 +393,32 @@ export const playgroundApi = {
),
};
// ── Asset Library (unlinked Playground media) ───────────────────────────
export interface LibraryMediaItem {
path: string;
name: string;
kind: 'image' | 'video';
size: number;
mtime: string;
}
export interface LibraryDeleteResult {
path: string;
status: 'deleted' | 'kept';
reason?: string;
}
export const libraryApi = {
list: () => api<LibraryMediaItem[]>('/api/library/media'),
deleteMany: (paths: string[]) =>
api<{
ok: boolean;
deleted: string[];
results: LibraryDeleteResult[];
}>('/api/library/media', { method: 'DELETE', body: JSON.stringify({ paths }) }),
};
// ── Agents (agentic harness) ────────────────────────────────────────────
export interface AgentSession {
@@ -427,7 +453,7 @@ export interface AgentMessage {
export interface AgentTask {
role: string;
goal: string;
status: 'pending' | 'running' | 'done' | 'failed';
status: 'pending' | 'running' | 'done' | 'failed' | 'skipped';
index?: number;
}
@@ -468,7 +494,7 @@ export const agentApi = {
typeof payload === 'string'
? { content: payload, mentions: [], attachments: [] }
: payload;
return api<{ ok: boolean; message: AgentMessage }>(`/api/agent/sessions/${id}/messages`, {
return api<{ ok: boolean; steered?: boolean; message: AgentMessage }>(`/api/agent/sessions/${id}/messages`, {
method: 'POST',
body: JSON.stringify(body),
});
@@ -616,7 +642,10 @@ export const canvasApi = {
body: JSON.stringify(payload),
}),
deleteNode: (canvasId: number, nodeId: number) =>
api<{ ok: boolean }>(`/api/canvas/${canvasId}/nodes/${nodeId}`, { method: 'DELETE' }),
api<{ ok: boolean; file_deleted?: boolean; reason?: string | null }>(
`/api/canvas/${canvasId}/nodes/${nodeId}`,
{ method: 'DELETE' },
),
tidy: (canvasId: number) =>
api<{ ok: boolean; moved: number }>(`/api/canvas/${canvasId}/tidy`, { method: 'POST' }),
createEdge: (
@@ -2,13 +2,13 @@
import LanguageSwitcher from '$lib/components/LanguageSwitcher.svelte';
import { t } from '$lib/i18n.svelte';
interface Props {
active?: 'projects' | 'canvas' | 'build-scene' | 'settings';
crumb?: string;
/** Optional status content rendered between the crumb and the nav. */
status?: import('svelte').Snippet;
children?: import('svelte').Snippet;
}
interface Props {
active?: 'projects' | 'canvas' | 'build-scene' | 'library' | 'settings';
crumb?: string;
/** Optional status content rendered between the crumb and the nav. */
status?: import('svelte').Snippet;
children?: import('svelte').Snippet;
}
let { active, crumb, status, children }: Props = $props();
</script>
@@ -30,6 +30,7 @@
<a class="nav-link" class:active={active === 'canvas'} href="/canvas">{t('nav.canvas')}</a>
<a class="nav-link" class:active={active === 'projects'} href="/projects">{t('nav.projects')}</a>
<a class="nav-link" class:active={active === 'build-scene'} href="/build-scene">{t('nav.buildScene')}</a>
<a class="nav-link" class:active={active === 'library'} href="/library">{t('nav.library')}</a>
<a class="nav-link" class:active={active === 'settings'} href="/settings">{t('nav.settings')}</a>
</nav>
</div>
@@ -213,7 +213,10 @@
{#each messages as m (m.id)}
{#if m.role === 'user'}
<div class="row user">
<div class="bubble user-bubble">
<div class="bubble user-bubble" class:steering={m.status === 'steering'}>
{#if m.status === 'steering'}
<span class="steer-tag">{t('agentChat.steering')}</span>
{/if}
{#if (m.attachments ?? []).length > 0}
<div class="user-thumbs">
{#each (m.attachments ?? []) as a (a.path)}
@@ -470,6 +473,18 @@
border: 1px solid color-mix(in srgb, var(--accent) 35%, transparent);
color: var(--text-primary);
}
.user-bubble.steering {
border-style: dashed;
}
.steer-tag {
display: inline-block;
font-size: 10px;
font-weight: 600;
letter-spacing: 0.06em;
text-transform: uppercase;
color: var(--accent);
margin-bottom: 4px;
}
.user-thumbs {
display: flex;
flex-wrap: wrap;
@@ -69,7 +69,7 @@ let {
const uploads = createUploadManager();
const mentionItems = $derived(filterWorkflows(workflows, mentionQuery));
const slashItems = $derived(filterSkills(skills, slashQuery));
const busy = $derived(running || uploadingNames.length > 0);
const busy = $derived(uploadingNames.length > 0);
const sendable = $derived(!busy && (!editorEmpty || attachments.length > 0));
$effect(() => {
@@ -508,16 +508,15 @@ let {
<Icon name={a.kind === 'audio' ? 'music' : 'video'} size={16} />
</span>
{/if}
<span class="tile-name" title={a.name}>{truncateMiddle(a.name, 18)}</span>
<button
type="button"
class="tile-x"
title={t('agentComposer.remove')}
onclick={() => removeAttachment(i)}
disabled={running}
>
<Icon name="close" size={10} />
</button>
<span class="tile-name" title={a.name}>{truncateMiddle(a.name, 18)}</span>
<button
type="button"
class="tile-x"
title={t('agentComposer.remove')}
onclick={() => removeAttachment(i)}
>
<Icon name="close" size={10} />
</button>
</div>
{/each}
{#each uploadingNames as name (name)}
@@ -538,7 +537,7 @@ let {
tabindex="0"
aria-multiline="true"
aria-label={t('agentComposer.message')}
data-placeholder={running ? t('agentComposer.working') : t('agentComposer.placeholder')}
data-placeholder={running ? t('agentComposer.steerPlaceholder') : t('agentComposer.placeholder')}
onkeydown={onKeydown}
oninput={onEditorInput}
onpaste={onPaste}
@@ -566,18 +565,18 @@ let {
<Icon name="stop" size={14} />
{t('common.stop')}
</button>
{:else}
<button
type="button"
class="send"
onclick={submit}
disabled={!sendable}
title={t('agentComposer.sendTitle')}
>
{t('common.send')}
<Icon name="chevron-right" size={14} />
</button>
{/if}
<button
type="button"
class="send"
class:steer={running}
onclick={submit}
disabled={!sendable}
title={running ? t('agentComposer.steerTitle') : t('agentComposer.sendTitle')}
>
{running ? t('agentComposer.steer') : t('common.send')}
<Icon name="chevron-right" size={14} />
</button>
</div>
</div>
</div>
@@ -788,6 +787,10 @@ let {
color: var(--error);
border-color: rgba(239, 68, 68, 0.5);
}
.send.steer {
color: var(--accent);
border-color: color-mix(in srgb, var(--accent) 50%, transparent);
}
.send:focus-visible {
outline: 2px solid var(--accent);
outline-offset: 2px;
@@ -25,6 +25,7 @@
if (status === 'running') return t('job.status.running');
if (status === 'done') return t('job.status.done');
if (status === 'failed') return t('job.status.failed');
if (status === 'skipped') return t('agentPlan.skipped');
return t('queue.queued');
}
</script>
+36 -2
View File
@@ -30,6 +30,7 @@ export const de: Dict = {
// --- App shell / nav ---
'nav.canvas': 'KI-Canvas',
'nav.library': 'Asset-Bibliothek',
'nav.projects': 'Projekte',
'nav.buildScene': 'Szene bauen',
'nav.settings': 'Einstellungen',
@@ -262,6 +263,11 @@ export const de: Dict = {
'canvas.noChatYet': 'Noch kein Chat auf diesem Canvas',
'canvas.collapseChat': 'Chat einklappen',
'canvas.expandChat': 'Chat ausklappen',
'canvas.deleteBlocked': 'Projekt-Assets und laufende Jobs können hier nicht gelöscht werden – über die Projekt-Assets ersetzen',
'canvas.deleteFailed': '{count} Karte(n) konnten nicht gelöscht werden',
'canvas.deletedWithFile': '{count} Karte(n) und Datei(en) aus der Asset-Bibliothek gelöscht',
'canvas.deletedCardOnly': '{count} Karte(n) entfernt – Datei bleibt (Projektdaten)',
'canvas.deletedMixed': '{files} Datei(en) gelöscht, {kept} behalten (Projektdaten)',
'global.somethingWentWrong': 'Etwas ist schiefgelaufen.',
// --- Memory ---
@@ -337,7 +343,12 @@ export const de: Dict = {
'agentComposer.sendTitle': 'Nachricht senden',
'agentComposer.stopTitle': 'Agent stoppen',
'agentComposer.working': 'Wird gesendet…',
'agentComposer.steer': 'Lenken',
'agentComposer.steerTitle': 'Kurskorrektur senden, während der Agent arbeitet',
'agentComposer.steerPlaceholder': 'Den Agenten während der Arbeit lenken…',
'agentChat.steering': 'Lenkung',
'agentPlan.planner': 'Plan',
'agentPlan.skipped': 'Übersprungen',
'agentTool.args': 'Argumente',
'agentTool.done': 'Fertig',
'agentTool.failed': 'Fehlgeschlagen',
@@ -528,6 +539,7 @@ export const de: Dict = {
'buildScene.expandChat': 'Chat ausklappen',
'buildScene.collapseChat': 'Chat einklappen',
'buildScene.videoExported': 'Video exportiert',
'buildScene.legacyNeedsViewport': 'Viewport nicht bereit für den Export',
'buildScene.videoExportFailed': 'Videoexport fehlgeschlagen',
'buildScene.captureSaved': 'Aufnahme gespeichert',
'buildScene.createSceneFailed': 'Szene konnte nicht erstellt werden',
@@ -927,6 +939,27 @@ export const de: Dict = {
'attach.projectFallback': 'Unbenanntes Projekt',
'attach.scene': 'Szene',
// --- Asset Library ---
'library.selectAll': 'Alle auswählen',
'library.deselectAll': 'Auswahl aufheben',
'library.itemCount': '{count} Element(e)',
'library.deleteSelected': 'Löschen ({count})',
'library.addToProject': 'Zum Projekt hinzufügen',
'library.filterLabel': 'Nach Art filtern',
'library.filterAll': 'Alle',
'library.filterImage': 'Bilder',
'library.filterVideo': 'Videos',
'library.loadMore': 'Mehr laden ({count} übrig)',
'library.loadingMore': 'Wird geladen…',
'library.confirmDelete': '{count} ausgewählte Datei(en) aus der Asset-Bibliothek löschen? Dies kann nicht rückgängig gemacht werden.',
'library.deleted': '{count} Datei(en) gelöscht',
'library.deletedPartial': '{kept} Datei(en) behalten – werden noch von Projektdaten verwendet',
'library.deleteFailed': 'Löschen fehlgeschlagen',
'library.emptyTitle': 'Die Asset-Bibliothek ist leer',
'library.emptyBody': 'Bilder und Videos, die mit keinem Projekt verknüpft sind, landen hier – per Upload auf einem Canvas oder Generation in einem Sandbox-Chat.',
'library.emptyFilterTitle': 'Keine {kind} in der Bibliothek',
'library.emptyFilterBody': 'Für diesen Filter gibt es noch nichts – probiere „Alle“ oder erzeuge etwas Neues.',
// --- Settings ---
'settings.loading': 'Einstellungen werden geladen',
'settings.allSaved': 'Alle Änderungen gespeichert',
@@ -1047,8 +1080,10 @@ export const de: Dict = {
'shot.keyframeObjAdd': 'Keyframe für {name} hinzufügen',
'shot.keyframeObjDelete': 'Keyframe löschen',
'shot.keyframeObjSelectFirst': 'Zuerst ein Objekt auswählen',
'shot.exportHint': 'Die Timeline als Video exportieren.',
'shot.exportHint': 'Szenenobjekte oder mindestens 2 Kamerakeyframes erforderlich (Captures optional).',
'shot.exportHintNeedBeats': 'Füge Szenenobjekte oder ≥2 Kamerakeyframes hinzu, um zu exportieren. Captures optional.',
'shot.exportTitle': 'Timeline exportieren',
'shot.exportImage': 'Bild exportieren',
'shot.exportVideo': 'Video exportieren',
'shot.viewportTools': 'Werkzeuge',
'shot.toolSelect': 'Auswählen',
@@ -1071,6 +1106,5 @@ export const de: Dict = {
'shot.capture': 'Aufnehmen',
'shot.errBlob': 'Der aufgenommene Frame konnte nicht gelesen werden',
'shot.errRecorder': 'Recorder konnte nicht gestartet werden',
'shot.errExportKeyframes': 'Keine Keyframes zum Exportieren',
'shot.errNoData': 'Keine Szenendaten zum Exportieren',
};
+37 -3
View File
@@ -28,6 +28,7 @@ export const en = {
// --- App shell / nav ---
'nav.canvas': 'AI Canvas',
'nav.library': 'Asset Library',
'nav.projects': 'Projects',
'nav.buildScene': 'Build Scene',
'nav.settings': 'Settings',
@@ -262,6 +263,11 @@ export const en = {
'canvas.noChatYet': 'No chat on this canvas yet',
'canvas.collapseChat': 'Collapse chat',
'canvas.expandChat': 'Expand chat',
'canvas.deleteBlocked': 'Project assets and running jobs can’t be deleted here — replace them from Project Assets',
'canvas.deleteFailed': 'Could not delete {count} card(s)',
'canvas.deletedWithFile': 'Deleted {count} card(s) and file(s) from the Asset Library',
'canvas.deletedCardOnly': 'Removed {count} card(s) — file kept (project data)',
'canvas.deletedMixed': 'Deleted {files} file(s), kept {kept} (project data)',
'global.somethingWentWrong': 'Something went wrong.',
// --- Memory ---
@@ -337,7 +343,12 @@ export const en = {
'agentComposer.sendTitle': 'Send message',
'agentComposer.stopTitle': 'Stop agent',
'agentComposer.working': 'Sending…',
'agentComposer.steer': 'Steer',
'agentComposer.steerTitle': 'Send a course correction while the agent works',
'agentComposer.steerPlaceholder': 'Steer the agent while it works…',
'agentChat.steering': 'Steering',
'agentPlan.planner': 'Plan',
'agentPlan.skipped': 'Skipped',
'agentTool.args': 'Arguments',
'agentTool.done': 'Done',
'agentTool.failed': 'Failed',
@@ -528,6 +539,7 @@ export const en = {
'buildScene.expandChat': 'Expand chat',
'buildScene.collapseChat': 'Collapse chat',
'buildScene.videoExported': 'Video exported',
'buildScene.legacyNeedsViewport': 'Viewport not ready for export',
'buildScene.videoExportFailed': 'Video export failed',
'buildScene.captureSaved': 'Capture saved',
'buildScene.createSceneFailed': 'Could not create scene',
@@ -927,6 +939,27 @@ export const en = {
'attach.projectFallback': 'Untitled project',
'attach.scene': 'Scene',
// --- Asset Library ---
'library.selectAll': 'Select all',
'library.deselectAll': 'Deselect all',
'library.itemCount': '{count} item(s)',
'library.deleteSelected': 'Delete ({count})',
'library.addToProject': 'Add to Project',
'library.filterLabel': 'Filter by kind',
'library.filterAll': 'All',
'library.filterImage': 'Images',
'library.filterVideo': 'Videos',
'library.loadMore': 'Load more ({count} remaining)',
'library.loadingMore': 'Loading…',
'library.confirmDelete': 'Delete {count} selected file(s) from the Asset Library? This cannot be undone.',
'library.deleted': 'Deleted {count} file(s)',
'library.deletedPartial': '{kept} file(s) kept — still referenced by project data',
'library.deleteFailed': 'Could not delete',
'library.emptyTitle': 'Nothing in the Asset Library',
'library.emptyBody': 'Images and videos that are not linked to a project land here — upload on a canvas or generate in a sandbox chat.',
'library.emptyFilterTitle': 'No {kind} in the library',
'library.emptyFilterBody': 'Nothing matches this filter yet — try All or generate something new.',
// --- Settings ---
'settings.loading': 'Loading settings',
'settings.allSaved': 'All changes saved',
@@ -1047,8 +1080,10 @@ export const en = {
'shot.keyframeObjAdd': 'Add keyframe for {name}',
'shot.keyframeObjDelete': 'Delete keyframe',
'shot.keyframeObjSelectFirst': 'Select an object first',
'shot.exportHint': 'Export the timeline as a video.',
'shot.exportHint': 'Need scene objects or at least 2 camera keyframes (captures not required).',
'shot.exportHintNeedBeats': 'Add scene objects or ≥2 camera keyframes to export. Captures are not required.',
'shot.exportTitle': 'Export timeline',
'shot.exportImage': 'Export image',
'shot.exportVideo': 'Export video',
'shot.viewportTools': 'Tools',
'shot.toolSelect': 'Select',
@@ -1071,8 +1106,7 @@ export const en = {
'shot.capture': 'Capture',
'shot.errBlob': 'Could not read the captured frame',
'shot.errRecorder': 'Could not start recorder',
'shot.errExportKeyframes': 'No keyframes to export',
'shot.errNoData': 'No scene data to export',
};
export type Dict = typeof en;
export type Dict = typeof en;
+36 -2
View File
@@ -30,6 +30,7 @@ export const es: Dict = {
// --- App shell / nav ---
'nav.canvas': 'Canvas IA',
'nav.library': 'Biblioteca de assets',
'nav.projects': 'Proyectos',
'nav.buildScene': 'Componer escena',
'nav.settings': 'Ajustes',
@@ -262,6 +263,11 @@ export const es: Dict = {
'canvas.noChatYet': 'Aún no hay ningún chat en este canvas',
'canvas.collapseChat': 'Contraer chat',
'canvas.expandChat': 'Expandir chat',
'canvas.deleteBlocked': 'Los assets del proyecto y los trabajos en curso no se pueden eliminar aquí – reemplázalos desde los assets del proyecto',
'canvas.deleteFailed': 'No se pudieron eliminar {count} tarjeta(s)',
'canvas.deletedWithFile': '{count} tarjeta(s) y archivo(s) eliminados de la biblioteca de assets',
'canvas.deletedCardOnly': '{count} tarjeta(s) eliminadas – archivo conservado (datos del proyecto)',
'canvas.deletedMixed': '{files} archivo(s) eliminados, {kept} conservados (datos del proyecto)',
'global.somethingWentWrong': 'Algo salió mal.',
// --- Memory ---
@@ -337,7 +343,12 @@ export const es: Dict = {
'agentComposer.sendTitle': 'Enviar mensaje',
'agentComposer.stopTitle': 'Detener agente',
'agentComposer.working': 'Enviando…',
'agentComposer.steer': 'Guiar',
'agentComposer.steerTitle': 'Enviar una corrección mientras el agente trabaja',
'agentComposer.steerPlaceholder': 'Guía al agente mientras trabaja…',
'agentChat.steering': 'Guía',
'agentPlan.planner': 'Plan',
'agentPlan.skipped': 'Omitido',
'agentTool.args': 'Argumentos',
'agentTool.done': 'Hecho',
'agentTool.failed': 'Fallido',
@@ -528,6 +539,7 @@ export const es: Dict = {
'buildScene.expandChat': 'Expandir chat',
'buildScene.collapseChat': 'Contraer chat',
'buildScene.videoExported': 'Vídeo exportado',
'buildScene.legacyNeedsViewport': 'Viewport no listo para exportar',
'buildScene.videoExportFailed': 'Error al exportar el vídeo',
'buildScene.captureSaved': 'Captura guardada',
'buildScene.createSceneFailed': 'No se pudo crear la escena',
@@ -927,6 +939,27 @@ export const es: Dict = {
'attach.projectFallback': 'Proyecto sin título',
'attach.scene': 'Escena',
// --- Asset Library ---
'library.selectAll': 'Seleccionar todo',
'library.deselectAll': 'Deseleccionar todo',
'library.itemCount': '{count} elemento(s)',
'library.deleteSelected': 'Eliminar ({count})',
'library.addToProject': 'Añadir al proyecto',
'library.filterLabel': 'Filtrar por tipo',
'library.filterAll': 'Todo',
'library.filterImage': 'Imágenes',
'library.filterVideo': 'Vídeos',
'library.loadMore': 'Cargar más ({count} restantes)',
'library.loadingMore': 'Cargando…',
'library.confirmDelete': '¿Eliminar {count} archivo(s) seleccionado(s) de la biblioteca de assets? Esta acción no se puede deshacer.',
'library.deleted': '{count} archivo(s) eliminados',
'library.deletedPartial': '{kept} archivo(s) conservados – aún referenciados por datos del proyecto',
'library.deleteFailed': 'No se pudo eliminar',
'library.emptyTitle': 'La biblioteca de assets está vacía',
'library.emptyBody': 'Las imágenes y vídeos no vinculados a un proyecto aparecen aquí – súbelos en un canvas o genéralos en un chat sandbox.',
'library.emptyFilterTitle': 'No hay {kind} en la biblioteca',
'library.emptyFilterBody': 'Nada coincide con este filtro todavía – prueba «Todo» o genera algo nuevo.',
// --- Settings ---
'settings.loading': 'Cargando ajustes',
'settings.allSaved': 'Todos los cambios guardados',
@@ -1047,8 +1080,10 @@ export const es: Dict = {
'shot.keyframeObjAdd': 'Añadir fotograma clave para {name}',
'shot.keyframeObjDelete': 'Eliminar fotograma clave',
'shot.keyframeObjSelectFirst': 'Selecciona antes un objeto',
'shot.exportHint': 'Exporta la línea de tiempo como vídeo.',
'shot.exportHint': 'Se necesitan objetos de escena o al menos 2 fotogramas de cámara (capturas no requeridas).',
'shot.exportHintNeedBeats': 'Añade objetos de escena o ≥2 fotogramas de cámara para exportar. Capturas no requeridas.',
'shot.exportTitle': 'Exportar línea de tiempo',
'shot.exportImage': 'Exportar imagen',
'shot.exportVideo': 'Exportar vídeo',
'shot.viewportTools': 'Herramientas',
'shot.toolSelect': 'Seleccionar',
@@ -1071,6 +1106,5 @@ export const es: Dict = {
'shot.capture': 'Capturar',
'shot.errBlob': 'No se pudo leer el fotograma capturado',
'shot.errRecorder': 'No se pudo iniciar el grabador',
'shot.errExportKeyframes': 'No hay fotogramas clave para exportar',
'shot.errNoData': 'No hay datos de escena para exportar',
};
+37 -3
View File
@@ -30,6 +30,7 @@ export const fr: Dict = {
// --- App shell / nav ---
'nav.canvas': 'Canvas IA',
'nav.library': 'Bibliothèque d’assets',
'nav.projects': 'Projets',
'nav.buildScene': 'Composer la scène',
'nav.settings': 'Paramètres',
@@ -262,7 +263,12 @@ export const fr: Dict = {
'canvas.noChatYet': 'Pas encore de discussion sur ce canevas',
'canvas.collapseChat': 'Réduire la discussion',
'canvas.expandChat': 'Développer la discussion',
'global.somethingWentWrong': "Une erreur s'est produite.",
'canvas.deleteBlocked': 'Les assets de projet et les jobs en cours ne peuvent pas être supprimés ici – remplacez-les depuis les assets du projet',
'canvas.deleteFailed': 'Impossible de supprimer {count} carte(s)',
'canvas.deletedWithFile': '{count} carte(s) et fichier(s) supprimés de la bibliothèque d’assets',
'canvas.deletedCardOnly': '{count} carte(s) retirée(s) – fichier conservé (données du projet)',
'canvas.deletedMixed': '{files} fichier(s) supprimé(s), {kept} conservé(s) (données du projet)',
"global.somethingWentWrong": "Une erreur s'est produite.",
// --- Memory ---
'memory.title': 'Mémoire',
@@ -337,7 +343,12 @@ export const fr: Dict = {
'agentComposer.sendTitle': 'Envoyer le message',
'agentComposer.stopTitle': "Arrêter l'agent",
'agentComposer.working': 'Envoi…',
'agentComposer.steer': 'Piloter',
'agentComposer.steerTitle': "Envoyer une correction pendant que l'agent travaille",
'agentComposer.steerPlaceholder': "Pilotez l'agent pendant qu'il travaille…",
'agentChat.steering': 'Pilotage',
'agentPlan.planner': 'Plan',
'agentPlan.skipped': 'Ignoré',
'agentTool.args': 'Arguments',
'agentTool.done': 'Terminé',
'agentTool.failed': 'Échec',
@@ -528,6 +539,7 @@ export const fr: Dict = {
'buildScene.expandChat': 'Développer la discussion',
'buildScene.collapseChat': 'Réduire la discussion',
'buildScene.videoExported': 'Vidéo exportée',
'buildScene.legacyNeedsViewport': 'Viewport pas prêt pour l\'export',
'buildScene.videoExportFailed': "Échec de l'export vidéo",
'buildScene.captureSaved': 'Capture enregistrée',
'buildScene.createSceneFailed': 'Impossible de créer la scène',
@@ -927,6 +939,27 @@ export const fr: Dict = {
'attach.projectFallback': 'Projet sans titre',
'attach.scene': 'Scène',
// --- Asset Library ---
'library.selectAll': 'Tout sélectionner',
'library.deselectAll': 'Tout désélectionner',
'library.itemCount': '{count} élément(s)',
'library.deleteSelected': 'Supprimer ({count})',
'library.addToProject': 'Ajouter au projet',
'library.filterLabel': 'Filtrer par type',
'library.filterAll': 'Tout',
'library.filterImage': 'Images',
'library.filterVideo': 'Vidéos',
'library.loadMore': 'Charger plus ({count} restants)',
'library.loadingMore': 'Chargement…',
'library.confirmDelete': 'Supprimer {count} fichier(s) sélectionné(s) de la bibliothèque d’assets ? Cette action est irréversible.',
'library.deleted': '{count} fichier(s) supprimé(s)',
'library.deletedPartial': '{kept} fichier(s) conservé(s) – encore référencé(s) par des données du projet',
'library.deleteFailed': 'Échec de la suppression',
'library.emptyTitle': 'La bibliothèque d’assets est vide',
'library.emptyBody': 'Les images et vidéos non liées à un projet arrivent ici – via un import sur un canvas ou une génération dans un chat sandbox.',
'library.emptyFilterTitle': 'Aucun élément {kind} dans la bibliothèque',
'library.emptyFilterBody': 'Rien ne correspond à ce filtre pour l’instant – essayez « Tout » ou générez quelque chose de nouveau.',
// --- Settings ---
'settings.loading': 'Chargement des paramètres',
'settings.allSaved': 'Toutes les modifications sont enregistrées',
@@ -1047,8 +1080,10 @@ export const fr: Dict = {
'shot.keyframeObjAdd': 'Ajouter une image clé pour {name}',
'shot.keyframeObjDelete': "Supprimer l'image clé",
'shot.keyframeObjSelectFirst': "Sélectionnez d'abord un objet",
'shot.exportHint': 'Exporter la timeline sous forme de vidéo.',
'shot.exportHint': 'Objets de scène ou au moins 2 images clés de caméra requis (captures facultatives).',
'shot.exportHintNeedBeats': 'Ajoutez des objets de scène ou ≥2 images clés de caméra pour exporter. Captures facultatives.',
'shot.exportTitle': 'Exporter la timeline',
'shot.exportImage': 'Exporter l\'image',
'shot.exportVideo': 'Exporter la vidéo',
'shot.viewportTools': 'Outils',
'shot.toolSelect': 'Sélection',
@@ -1071,6 +1106,5 @@ export const fr: Dict = {
'shot.capture': 'Capturer',
'shot.errBlob': "Impossible de lire l'image capturée",
'shot.errRecorder': "Impossible de démarrer l'enregistreur",
'shot.errExportKeyframes': "Aucune image clé à exporter",
'shot.errNoData': 'Aucune donnée de scène à exporter',
};
+36 -2
View File
@@ -30,6 +30,7 @@ export const ja: Dict = {
// --- App shell / nav ---
'nav.canvas': 'AI Canvas',
'nav.library': 'アセットライブラリ',
'nav.projects': 'プロジェクト',
'nav.buildScene': 'シーン構築',
'nav.settings': '設定',
@@ -262,6 +263,11 @@ export const ja: Dict = {
'canvas.noChatYet': 'このキャンバスにはまだチャットがありません',
'canvas.collapseChat': 'チャットを折りたたむ',
'canvas.expandChat': 'チャットを展開',
'canvas.deleteBlocked': 'プロジェクトのアセットや実行中のジョブはここでは削除できません – プロジェクトのアセットから置き換えてください',
'canvas.deleteFailed': '{count} 枚のカードを削除できませんでした',
'canvas.deletedWithFile': '{count} 枚のカードとファイルをアセットライブラリから削除しました',
'canvas.deletedCardOnly': '{count} 枚のカードを削除 – ファイルは保持されます(プロジェクトデータ)',
'canvas.deletedMixed': '{files} 個のファイルを削除、{kept} 個を保持(プロジェクトデータ)',
'global.somethingWentWrong': '問題が発生しました。',
// --- Memory ---
@@ -337,7 +343,12 @@ export const ja: Dict = {
'agentComposer.sendTitle': 'メッセージを送信',
'agentComposer.stopTitle': 'エージェントを停止',
'agentComposer.working': '送信中…',
'agentComposer.steer': '操縦',
'agentComposer.steerTitle': 'エージェントの作業中に軌道修正を送る',
'agentComposer.steerPlaceholder': 'エージェントの作業中に軌道修正…',
'agentChat.steering': '軌道修正',
'agentPlan.planner': '計画',
'agentPlan.skipped': 'スキップ',
'agentTool.args': '引数',
'agentTool.done': '完了',
'agentTool.failed': '失敗',
@@ -528,6 +539,7 @@ export const ja: Dict = {
'buildScene.expandChat': 'チャットを展開',
'buildScene.collapseChat': 'チャットを折りたたむ',
'buildScene.videoExported': 'ビデオを書き出しました',
'buildScene.legacyNeedsViewport': 'ビューポートの準備ができていません',
'buildScene.videoExportFailed': 'ビデオの書き出しに失敗しました',
'buildScene.captureSaved': 'キャプチャを保存しました',
'buildScene.createSceneFailed': 'シーンを作成できませんでした',
@@ -927,6 +939,27 @@ export const ja: Dict = {
'attach.projectFallback': '無題のプロジェクト',
'attach.scene': 'シーン',
// --- Asset Library ---
'library.selectAll': 'すべて選択',
'library.deselectAll': '選択を解除',
'library.itemCount': '{count} 件',
'library.deleteSelected': '削除({count})',
'library.addToProject': 'プロジェクトに追加',
'library.filterLabel': '種類で絞り込み',
'library.filterAll': 'すべて',
'library.filterImage': '画像',
'library.filterVideo': '動画',
'library.loadMore': 'さらに読み込む(残り {count} 件)',
'library.loadingMore': '読み込み中…',
'library.confirmDelete': '選択した {count} 件のファイルをアセットライブラリから削除しますか?この操作は元に戻せません。',
'library.deleted': '{count} 件のファイルを削除しました',
'library.deletedPartial': '{kept} 件のファイルを保持 – プロジェクトデータから参照されています',
'library.deleteFailed': '削除できませんでした',
'library.emptyTitle': 'アセットライブラリは空です',
'library.emptyBody': 'プロジェクトにリンクされていない画像と動画はここに表示されます – キャンバスでアップロードするか、サンドボックスチャットで生成してください。',
'library.emptyFilterTitle': 'ライブラリに{kind}はありません',
'library.emptyFilterBody': 'このフィルターに一致するものはまだありません – 「すべて」を試すか、新しく生成してください。',
// --- Settings ---
'settings.loading': '設定を読み込み中',
'settings.allSaved': 'すべての変更を保存しました',
@@ -1047,8 +1080,10 @@ export const ja: Dict = {
'shot.keyframeObjAdd': '{name} のキーフレームを追加',
'shot.keyframeObjDelete': 'キーフレームを削除',
'shot.keyframeObjSelectFirst': '先にオブジェクトを選択してください',
'shot.exportHint': 'タイムラインをビデオとして書き出します。',
'shot.exportHint': 'シーンオブジェクトまたはカメラキーフレームが2つ以上必要です(キャプチャは不要)。',
'shot.exportHintNeedBeats': '書き出すには、シーンオブジェクトまたは2つ以上のカメラキーフレームを追加してください。キャプチャは不要です。',
'shot.exportTitle': 'タイムラインを書き出す',
'shot.exportImage': '画像を書き出す',
'shot.exportVideo': 'ビデオを書き出す',
'shot.viewportTools': 'ツール',
'shot.toolSelect': '選択',
@@ -1071,6 +1106,5 @@ export const ja: Dict = {
'shot.capture': 'キャプチャ',
'shot.errBlob': 'キャプチャしたフレームを読み取れませんでした',
'shot.errRecorder': 'レコーダーを開始できませんでした',
'shot.errExportKeyframes': '書き出すキーフレームがありません',
'shot.errNoData': '書き出すシーンデータがありません',
};
+36 -2
View File
@@ -30,6 +30,7 @@ export const ko: Dict = {
// --- App shell / nav ---
'nav.canvas': 'AI Canvas',
'nav.library': '애셋 라이브러리',
'nav.projects': '프로젝트',
'nav.buildScene': '장면 구성',
'nav.settings': '설정',
@@ -262,6 +263,11 @@ export const ko: Dict = {
'canvas.noChatYet': '이 캔버스에는 아직 채팅이 없습니다',
'canvas.collapseChat': '채팅 접기',
'canvas.expandChat': '채팅 펼치기',
'canvas.deleteBlocked': '프로젝트 애셋과 실행 중인 작업은 여기서 삭제할 수 없습니다 – 프로젝트 애셋에서 교체하세요',
'canvas.deleteFailed': '{count}개 카드를 삭제하지 못했습니다',
'canvas.deletedWithFile': '애셋 라이브러리에서 {count}개 카드와 파일을 삭제했습니다',
'canvas.deletedCardOnly': '{count}개 카드를 제거 – 파일은 유지됩니다 (프로젝트 데이터)',
'canvas.deletedMixed': '{files}개 파일 삭제, {kept}개 유지 (프로젝트 데이터)',
'global.somethingWentWrong': '문제가 발생했습니다.',
// --- Memory ---
@@ -337,7 +343,12 @@ export const ko: Dict = {
'agentComposer.sendTitle': '메시지 보내기',
'agentComposer.stopTitle': '에이전트 중지',
'agentComposer.working': '보내는 중…',
'agentComposer.steer': '조정',
'agentComposer.steerTitle': '에이전트가 작동하는 동안 방향 수정 보내기',
'agentComposer.steerPlaceholder': '에이전트가 작동하는 동안 조정…',
'agentChat.steering': '조정',
'agentPlan.planner': '계획',
'agentPlan.skipped': '건너뜀',
'agentTool.args': '인수',
'agentTool.done': '완료',
'agentTool.failed': '실패',
@@ -528,6 +539,7 @@ export const ko: Dict = {
'buildScene.expandChat': '채팅 펼치기',
'buildScene.collapseChat': '채팅 접기',
'buildScene.videoExported': '영상 내보내기 완료',
'buildScene.legacyNeedsViewport': '내보낼 뷰포트가 준비되지 않았습니다',
'buildScene.videoExportFailed': '영상 내보내기 실패',
'buildScene.captureSaved': '캡처 저장됨',
'buildScene.createSceneFailed': '장면을 만들지 못했습니다',
@@ -927,6 +939,27 @@ export const ko: Dict = {
'attach.projectFallback': '제목 없는 프로젝트',
'attach.scene': '장면',
// --- Asset Library ---
'library.selectAll': '모두 선택',
'library.deselectAll': '선택 해제',
'library.itemCount': '{count}개 항목',
'library.deleteSelected': '삭제 ({count})',
'library.addToProject': '프로젝트에 추가',
'library.filterLabel': '종류로 필터',
'library.filterAll': '전체',
'library.filterImage': '이미지',
'library.filterVideo': '비디오',
'library.loadMore': '더 불러오기 ({count}개 남음)',
'library.loadingMore': '불러오는 중…',
'library.confirmDelete': '선택한 {count}개 파일을 애셋 라이브러리에서 삭제하시겠습니까? 이 작업은 되돌릴 수 없습니다.',
'library.deleted': '{count}개 파일을 삭제했습니다',
'library.deletedPartial': '{kept}개 파일 유지 – 프로젝트 데이터에서 아직 참조 중입니다',
'library.deleteFailed': '삭제하지 못했습니다',
'library.emptyTitle': '애셋 라이브러리가 비어 있습니다',
'library.emptyBody': '프로젝트에 연결되지 않은 이미지와 비디오가 여기에 표시됩니다 – 캔버스에서 업로드하거나 샌드박스 채팅에서 생성하세요.',
'library.emptyFilterTitle': '라이브러리에 {kind}가 없습니다',
'library.emptyFilterBody': '이 필터와 일치하는 항목이 아직 없습니다 – 전체를 시도하거나 새로 생성해 보세요.',
// --- Settings ---
'settings.loading': '설정 불러오는 중',
'settings.allSaved': '모든 변경 사항이 저장되었습니다',
@@ -1047,8 +1080,10 @@ export const ko: Dict = {
'shot.keyframeObjAdd': '{name} 키프레임 추가',
'shot.keyframeObjDelete': '키프레임 삭제',
'shot.keyframeObjSelectFirst': '먼저 오브젝트를 선택하세요',
'shot.exportHint': '타임라인을 영상으로 내보냅니다.',
'shot.exportHint': '씬 객체 또는 카메라 키프레임 2개 이상이 필요합니다 (캡처는 선택).',
'shot.exportHintNeedBeats': '내보내려면 씬 객체 또는 카메라 키프레임 2개 이상을 추가하세요. 캡처는 선택입니다.',
'shot.exportTitle': '타임라인 내보내기',
'shot.exportImage': '이미지보내기',
'shot.exportVideo': '영상 내보내기',
'shot.viewportTools': '도구',
'shot.toolSelect': '선택',
@@ -1071,6 +1106,5 @@ export const ko: Dict = {
'shot.capture': '캡처',
'shot.errBlob': '캡처한 프레임을 읽지 못했습니다',
'shot.errRecorder': '녹화를 시작하지 못했습니다',
'shot.errExportKeyframes': '내보낼 키프레임이 없습니다',
'shot.errNoData': '내보낼 장면 데이터가 없습니다',
};
+36 -2
View File
@@ -30,6 +30,7 @@ export const zh: Dict = {
// --- App shell / nav ---
'nav.canvas': 'AI 画布',
'nav.library': '素材库',
'nav.projects': '项目',
'nav.buildScene': '搭建场景',
'nav.settings': '设置',
@@ -262,6 +263,11 @@ export const zh: Dict = {
'canvas.noChatYet': '此画布尚无对话',
'canvas.collapseChat': '收起对话',
'canvas.expandChat': '展开对话',
'canvas.deleteBlocked': '项目素材和正在运行的任务无法在此删除 – 请在项目素材中替换它们',
'canvas.deleteFailed': '未能删除 {count} 张卡片',
'canvas.deletedWithFile': '已从素材库删除 {count} 张卡片及文件',
'canvas.deletedCardOnly': '已移除 {count} 张卡片 – 文件保留(项目数据)',
'canvas.deletedMixed': '已删除 {files} 个文件,保留 {kept} 个(项目数据)',
'global.somethingWentWrong': '出了点问题。',
// --- Memory ---
@@ -337,7 +343,12 @@ export const zh: Dict = {
'agentComposer.sendTitle': '发送消息',
'agentComposer.stopTitle': '停止智能体',
'agentComposer.working': '发送中……',
'agentComposer.steer': '引导',
'agentComposer.steerTitle': '在智能体工作时发送修正指令',
'agentComposer.steerPlaceholder': '在智能体工作时进行引导……',
'agentChat.steering': '引导',
'agentPlan.planner': '计划',
'agentPlan.skipped': '已跳过',
'agentTool.args': '参数',
'agentTool.done': '完成',
'agentTool.failed': '失败',
@@ -528,6 +539,7 @@ export const zh: Dict = {
'buildScene.expandChat': '展开对话',
'buildScene.collapseChat': '折叠对话',
'buildScene.videoExported': '视频已导出',
'buildScene.legacyNeedsViewport': '视口尚未准备好导出',
'buildScene.videoExportFailed': '视频导出失败',
'buildScene.captureSaved': '截图已保存',
'buildScene.createSceneFailed': '无法创建场景',
@@ -927,6 +939,27 @@ export const zh: Dict = {
'attach.projectFallback': '未命名项目',
'attach.scene': '场景',
// --- Asset Library ---
'library.selectAll': '全选',
'library.deselectAll': '取消全选',
'library.itemCount': '{count} 个项目',
'library.deleteSelected': '删除({count})',
'library.addToProject': '添加到项目',
'library.filterLabel': '按类型筛选',
'library.filterAll': '全部',
'library.filterImage': '图片',
'library.filterVideo': '视频',
'library.loadMore': '加载更多(剩余 {count} 个)',
'library.loadingMore': '加载中…',
'library.confirmDelete': '从素材库中删除选中的 {count} 个文件?此操作无法撤销。',
'library.deleted': '已删除 {count} 个文件',
'library.deletedPartial': '保留 {kept} 个文件 – 仍被项目数据引用',
'library.deleteFailed': '删除失败',
'library.emptyTitle': '素材库为空',
'library.emptyBody': '未关联到项目的图片和视频会显示在这里 – 在画布上上传或在沙盒对话中生成。',
'library.emptyFilterTitle': '素材库中没有{kind}',
'library.emptyFilterBody': '此筛选条件暂无匹配项 – 试试“全部”或生成新内容。',
// --- Settings ---
'settings.loading': '正在加载设置',
'settings.allSaved': '所有更改已保存',
@@ -1047,8 +1080,10 @@ export const zh: Dict = {
'shot.keyframeObjAdd': '为 {name} 添加关键帧',
'shot.keyframeObjDelete': '删除关键帧',
'shot.keyframeObjSelectFirst': '请先选择一个物体',
'shot.exportHint': '将时间线导出为视频。',
'shot.exportHint': '需要场景对象或至少 2 个摄像机关键帧(不要求捕获)。',
'shot.exportHintNeedBeats': '添加场景对象或 ≥2 个摄像机关键帧以导出。不要求捕获。',
'shot.exportTitle': '导出时间线',
'shot.exportImage': '导出图片',
'shot.exportVideo': '导出视频',
'shot.viewportTools': '工具',
'shot.toolSelect': '选择',
@@ -1071,6 +1106,5 @@ export const zh: Dict = {
'shot.capture': '截图',
'shot.errBlob': '无法读取截取的帧',
'shot.errRecorder': '无法启动录制器',
'shot.errExportKeyframes': '没有可导出的关键帧',
'shot.errNoData': '没有可导出的场景数据',
};
+43
View File
@@ -6,9 +6,20 @@ export interface Capture {
kind: 'image' | 'video';
label: string | null;
file_path: string | null;
meta_json?: string | null;
meta?: Record<string, unknown> | null;
created_at: string;
}
export interface GateRecordResult {
ok: boolean;
gate: 'brief' | 'cut';
gates: {
brief?: { approved_at: string; source?: string; note?: string | null } | null;
cut?: { approved_at: string; source?: string; note?: string | null } | null;
};
}
export const shotApi = {
async listCaptures(compositionId: number): Promise<Capture[]> {
const resp = await fetch(`/api/shots/${compositionId}/captures`);
@@ -52,4 +63,36 @@ export const shotApi = {
const resp = await fetch(`/api/shots/${compositionId}`, { method: 'DELETE' });
return resp.ok;
},
/** Record Brief/Cut gate from the UI (source: 'ui'). Human at keyboard — no ask_user. */
async recordGate(
compositionId: number,
gate: 'brief' | 'cut',
opts: { note?: string; source?: string } = {},
): Promise<GateRecordResult> {
const resp = await fetch(`/api/shots/${compositionId}/gates`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
gate,
note: opts.note ?? null,
source: opts.source ?? 'ui',
}),
});
if (!resp.ok) {
let detail = `Gate record failed (${resp.status})`;
try {
const body = await resp.json();
detail =
typeof body?.detail === 'string'
? body.detail
: body?.detail?.message || body?.status || detail;
} catch {
/* ignore */
}
throw new Error(detail);
}
return resp.json();
},
};
@@ -90,12 +90,12 @@
<button class="icon danger" onclick={() => shotStore.deleteObject(o.id)} title={t('common.delete')}>✕</button>
</div>
</li>
{:else}
<li class="empty">{t('shot.emptyScene')}</li>
{/each}
</ul>
{:else}
<li class="empty">{t('shot.emptyScene')}</li>
{/each}
</ul>
<footer class="hint">{t('shot.hint')}</footer>
<footer class="hint">{t('shot.hint')}</footer>
</div>
<style>
@@ -5,9 +5,11 @@
* mannequin figures + primitives on a ground grid, OrbitControls camera,
* TransformControls gizmo mapped to the active tool, and a capture
* renderer (PNG dataURL) used by the Capture button and the agent's
* request_capture SSE.
* request_capture SSE. This canvas is also the source for video export —
* one deterministic MediaRecorder pass of the camera track + object
* keyframes (Timeline → Export video).
*/
import { onMount, onDestroy, untrack } from 'svelte';
import { onMount, onDestroy, untrack } from 'svelte';
import * as THREE from 'three';
import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
import { TransformControls } from 'three/examples/jsm/controls/TransformControls.js';
@@ -24,7 +26,13 @@ import { sampleTrack } from '../motion/sampleTrack';
import { t } from '$lib/i18n.svelte';
import '../types/mannequin-js.d';
let { oncapture, onselectionchange }: { oncapture?: (dataUrl: string) => void; onselectionchange?: () => void } = $props();
let {
oncapture,
onselectionchange,
}: {
oncapture?: (dataUrl: string) => void;
onselectionchange?: () => void;
} = $props();
let container: HTMLDivElement;
let renderer: THREE.WebGLRenderer;
@@ -37,6 +45,9 @@ let raf = 0;
let resizeObserver: ResizeObserver;
let cleanupPointerUp: (() => void) | null = null;
// Video export frame rate — 24 fps (film standard) for the blockout clip.
const EXPORT_FPS = 24;
// toolbar-owned view state (the grid mesh + a CSS rule-of-thirds overlay)
let showGrid = $state(true);
let showComposition = $state(false);
@@ -398,8 +409,11 @@ let gizmoDragging = false;
return { position: pos, target: tgt, fov: a.fov + (b.fov - a.fov) * k };
}
let playbackClockPrev = 0;
let playbackWasPlaying = false;
let playbackClockPrev = 0;
let playbackWasPlaying = false;
// true while a video export pass is rendering — tick() idles so the fixed-step
// export clock owns the playhead/renders and never fights the recorder.
let exporting = false;
function applyCameraAt(time: number) {
const sample = sampleCameraTrack(time);
@@ -444,10 +458,11 @@ let gizmoDragging = false;
// upload it. The store never sees this; the page wires oncapture.
fetch(`/api/shots/${shotStore.compositionId}`)
.then((r) => (r.ok ? r.json() : null))
.then((comp) => {
if (!comp?.capture_request_json || !oncapture) return;
.then(async (comp) => {
if (!comp?.capture_request_json) return;
if (comp.capture_requested_at && comp.capture_requested_at <= captureRequestedAt) return;
captureRequestedAt = Date.now();
if (!oncapture) return;
const dataUrl = captureShot();
oncapture(dataUrl);
})
@@ -458,6 +473,7 @@ let gizmoDragging = false;
function tick() {
raf = requestAnimationFrame(tick);
if (exporting) return; // export drives the scene on its own fixed clock
const now = performance.now() / 1000;
advancePlayback(now);
// While playing (or scrubbed onto the track) the camera track owns the
@@ -662,10 +678,15 @@ let gizmoDragging = false;
async function exportVideo(): Promise<{ dataUrl: string; ext: string }> {
const mime = pickVideoMime();
if (!mime) throw new Error(t('shot.errRecorder'));
if (shotStore.cameraTrack.length < 2) {
throw new Error(t('shot.errExportKeyframes'));
// A camera track (≥2 keys) drives camera motion, but a scene of
// keyframed objects still animates with a static camera — the 3D
// editor content is what we export, so only demand SOME renderable
// scene, not a camera track specifically.
const hasObjects = shotStore.objects.some((o) => o.type !== 'camera');
if (shotStore.cameraTrack.length < 2 && !hasObjects) {
throw new Error(t('shot.errNoData'));
}
const stream = renderer.domElement.captureStream(30);
const stream = renderer.domElement.captureStream(EXPORT_FPS);
const chunks: Blob[] = [];
const recorder = new MediaRecorder(stream, { mimeType: mime });
recorder.ondataavailable = (e) => {
@@ -676,27 +697,40 @@ let gizmoDragging = false;
});
const duration = shotStore.playback.duration;
// Drive the camera manually across one deterministic pass: pause store
// playback (the tick loop would fight us), sample per frame with a
// fixed-step clock so export duration matches the timeline exactly.
// Record in REAL TIME: captureStream(N) stamps frames on the wall clock,
// so the playhead must advance one N fps tick per real-time frame. The
// old requestAnimationFrame loop ran at the display refresh (60 Hz+),
// scrubbing 0→duration twice as fast and halving the exported clip.
// A fixed timestep with drift compensation keeps the output exactly
// `duration` seconds regardless of monitor refresh or frame jitter.
const fps = EXPORT_FPS;
const frameMs = 1000 / fps;
const totalFrames = Math.max(1, Math.round(duration * fps));
shotStore.setPlaying(false);
const fps = 30;
const steps = Math.max(2, Math.round(duration * fps));
exporting = true;
recorder.start();
for (let i = 0; i <= steps; i++) {
const t = (i / steps) * duration;
shotStore.setElapsed(t);
applyCameraAt(t);
syncScene();
renderFrame();
await new Promise((r) => requestAnimationFrame(r));
try {
const start = performance.now();
for (let i = 0; i <= totalFrames; i++) {
const t = Math.min(i / fps, duration);
shotStore.setElapsed(t);
applyCameraAt(t);
syncScene();
renderFrame();
const target = start + (i + 1) * frameMs;
const wait = target - performance.now();
if (wait > 0) await new Promise((r) => setTimeout(r, wait));
}
// One frame of tail so the recorder flushes the last presented frame.
await new Promise((r) => setTimeout(r, frameMs));
} finally {
exporting = false;
shotStore.setElapsed(0);
}
// Hold the final frame briefly so the recorder flushes it.
await new Promise((r) => setTimeout(r, 120));
recorder.stop();
await done;
stream.getTracks().forEach((tr) => tr.stop());
shotStore.setElapsed(0);
const ext = mime.includes('mp4') ? 'mp4' : 'webm';
const blob = new Blob(chunks, { type: mime });
@@ -19,9 +19,16 @@
const playing = $derived(shotStore.playback.playing);
const elapsed = $derived(shotStore.playback.elapsed);
const track = $derived(shotStore.cameraTrack);
const objects = $derived(shotStore.objects);
const selectedObject = $derived(shotStore.selectedObject);
const selectedKeyframeId = $derived(shotStore.selectedKeyframeId);
/** Export needs a camera track (≥2 keys) or any scene object to frame. */
const canExport = $derived(track.length >= 2 || objects.some((o) => o.type !== 'camera'));
const exportDisabledReason = $derived(
canExport ? '' : track.length < 2 ? t('shot.exportHintNeedBeats') : t('shot.exportHint'),
);
const pct = $derived(duration > 0 ? (elapsed / duration) * 100 : 0);
function timeAt(clientX: number): number {
@@ -170,7 +177,12 @@
</button>
</span>
{/if}
<button class="export" onclick={onExportVideo} disabled={exporting || track.length < 2} title={track.length < 2 ? t('shot.exportHint') : t('shot.exportTitle')}>
<button
class="export"
onclick={onExportVideo}
disabled={exporting || !canExport}
title={canExport ? t('shot.exportTitle') : exportDisabledReason}
>
{t('shot.exportVideo')}
</button>
</div>
+78 -6
View File
@@ -5,7 +5,9 @@
* bracketing.
*
* Differences from the reference:
* - No `objectInstances` Map — the viewport keeps its own three.js registry.
* - No `objectInstances` Map — the Three.js viewport keeps a *view* registry
* (meshes/handles), not a second scene store. Adapters read SceneData via
* toSceneData() only.
* - `scene_json` on the server is the source of truth: local edits autosave
* (debounced), and `shot.updated` SSE events reload the composition unless
* a drag/gesture is in progress (`dragInProgress` defers, matching the
@@ -64,6 +66,8 @@ export interface ShotComposition {
id: number;
title: string;
scene: SceneData;
/** Server clock — used for optimistic concurrency on PATCH. */
updated_at?: string;
}
export interface CameraKeyframe {
@@ -74,12 +78,26 @@ export interface CameraKeyframe {
fov: number;
}
/** Brief/Cut human gates — live on scene_json.gates. */
export interface BuildSceneGateApproval {
approved_at: string;
source?: string;
note?: string | null;
}
export interface BuildSceneGates {
brief?: BuildSceneGateApproval | null;
cut?: BuildSceneGateApproval | null;
}
export interface SceneData {
objects: SceneObject[];
selectedObjectId?: string | null;
shotParams?: Partial<Omit<ShotParams, 'composition'>> & { composition?: string };
playback?: PlaybackState;
cameraTrack?: CameraKeyframe[];
/** Brief/Cut approvals — must round-trip so client autosave does not wipe them. */
gates?: BuildSceneGates;
}
/** Hard cap: exported blockout videos stay under a minute. */
@@ -183,6 +201,7 @@ function createStore() {
let shotParams = $state<SceneData['shotParams']>({});
let playback = $state<PlaybackState>({ playing: false, elapsed: 0, speed: 1, duration: 6 });
let cameraTrack = $state<CameraKeyframe[]>([]);
let gates = $state<BuildSceneGates>({});
let loaded = $state(false);
let history = $state<SceneObject[][]>([]);
@@ -194,6 +213,8 @@ function createStore() {
let dragInProgress = $state(false);
let saveTimer: ReturnType<typeof setTimeout> | null = null;
let lastServerVersion = 0;
/** Last known server updated_at — sent as base_updated_at on PATCH. */
let serverUpdatedAt: string | null = null;
function withoutHistory<T>(fn: () => T): T {
suppressHistory = true;
@@ -232,8 +253,14 @@ function createStore() {
}
function applyServerScene(comp: ShotComposition) {
// Cancel pending autosave so a stale local payload cannot clobber agent writes.
if (saveTimer) {
clearTimeout(saveTimer);
saveTimer = null;
}
compositionId = comp.id;
title = comp.title;
serverUpdatedAt = comp.updated_at ?? serverUpdatedAt;
const scene = comp.scene ?? {};
objects = (scene.objects ?? []).map((o) => ({
...o,
@@ -245,9 +272,11 @@ function createStore() {
shotParams = normalizeShotParams(scene.shotParams);
if (scene.playback) playback = scene.playback;
cameraTrack = (scene.cameraTrack ?? []).map((k) => ({ ...k }));
gates = scene.gates ? { ...scene.gates } : {};
loaded = true;
history = [];
future = [];
lastServerVersion = Date.now();
}
function scheduleSave() {
@@ -261,11 +290,24 @@ function createStore() {
const payload = JSON.stringify(toSceneData());
lastServerVersion = Date.now();
try {
await fetch(`/api/shots/${compositionId}`, {
const resp = await fetch(`/api/shots/${compositionId}`, {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ scene_json: payload }),
body: JSON.stringify({
scene_json: payload,
...(serverUpdatedAt ? { base_updated_at: serverUpdatedAt } : {}),
}),
});
if (resp.status === 409) {
// Stale vs agent write — reload authoritative scene.
const fresh = await fetch(`/api/shots/${compositionId}`);
if (fresh.ok) applyServerScene((await fresh.json()) as ShotComposition);
return;
}
if (resp.ok) {
const body = (await resp.json()) as ShotComposition;
if (body.updated_at) serverUpdatedAt = body.updated_at;
}
} catch {
/* offline — next edit retries */
}
@@ -278,15 +320,21 @@ function createStore() {
shotParams,
playback,
cameraTrack,
};
gates,
} as SceneData;
}
/** SSE: the agent mutated the scene server-side. Ignore our own echo. */
/** SSE: the agent mutated the scene server-side. Ignore our own PATCH echo. */
function handleShotUpdated(data: { shot_id?: number; reason?: string }) {
if (data.shot_id !== compositionId) return;
if (dragInProgress) return; // a gizmo drag wins; the refetch happens on dragstop
if (Date.now() - lastServerVersion < 1500) return; // our own autosave echo
if (compositionId === null) return;
const reason = data.reason ?? '';
// Our own PATCH publishes reason=patched — skip that echo. Always apply
// agent/tool reasons (object_*, gate_*, …) so remote edits land even
// when an autosave just ran.
if (reason === 'patched' || reason === 'created') return;
if (!reason && Date.now() - lastServerVersion < 1500) return;
void (async () => {
const resp = await fetch(`/api/shots/${compositionId}`);
if (!resp.ok) return;
@@ -340,6 +388,8 @@ function createStore() {
activeTool = 'move';
playback = { ...playback, playing: false, elapsed: 0 };
cameraTrack = [];
// Keep Brief/Cut gates — approvals survive wipe unless operator revises.
scheduleSave();
}
function selectObject(id: string) {
@@ -610,6 +660,21 @@ function createStore() {
cameraKeyframeRequest = { time: Math.min(Math.max(time, 0), playback.duration), nonce: Date.now() };
}
function setGates(next: BuildSceneGates) {
gates = {
...(next.brief ? { brief: { ...next.brief } } : gates.brief ? { brief: gates.brief } : {}),
...(next.cut ? { cut: { ...next.cut } } : gates.cut ? { cut: gates.cut } : {}),
};
scheduleSave();
}
function applyGatesSnapshot(next: BuildSceneGates) {
gates = {
...(next.brief ? { brief: { ...next.brief } } : {}),
...(next.cut ? { cut: { ...next.cut } } : {}),
};
}
function undo() {
if (history.length === 0) return;
const previous = history[history.length - 1];
@@ -674,6 +739,9 @@ function createStore() {
get cameraTrack() {
return cameraTrack;
},
get gates() {
return gates;
},
get cameraKeyframeRequest() {
return cameraKeyframeRequest;
},
@@ -700,6 +768,8 @@ function createStore() {
applyServerScene,
handleShotUpdated,
flushSave: saveScene,
/** Sole mutable graph snapshot for adapters (viewport sampling, export). */
toSceneData,
// actions
addObject,
deleteObject,
@@ -723,6 +793,8 @@ function createStore() {
removeCameraKeyframe,
removeCameraKeyframeAt,
requestCameraKeyframeAt,
setGates,
applyGatesSnapshot,
addKeyframe,
commitLiveKeyframe,
deleteKeyframe,
@@ -168,10 +168,11 @@ function toggleChat() {
}
}
let exportStatus = $state<string | null>(null);
async function captureFromUi() {
// The viewport exports its own canvas render.
const comp = shotStore.compositionId;
if (comp == null || !viewportRef) return;
// The Three.js viewport renders the still directly.
if (!viewportRef) return;
const dataUrl = (viewportRef as unknown as { captureNow: () => string }).captureNow();
await handleCaptureDataUrl(dataUrl);
}
@@ -180,13 +181,15 @@ function toggleChat() {
async function handleExportVideo() {
const comp = shotStore.compositionId;
if (comp == null || !viewportRef || exportingVideo) return;
if (comp == null || exportingVideo) return;
exportingVideo = true;
exportStatus = t('buildScene.videoExported');
try {
if (!viewportRef) throw new Error(t('buildScene.legacyNeedsViewport'));
const { dataUrl } = await (viewportRef as unknown as {
exportVideoClip: () => Promise<{ dataUrl: string; ext: string }>;
}).exportVideoClip();
const saved = await shotApi.uploadCapture(comp, dataUrl, 'camera track export');
const saved = await shotApi.uploadCapture(comp, dataUrl, 'viewport video export');
if (saved) {
await refreshCaptures();
toast.success(t('buildScene.videoExported'));
@@ -194,6 +197,7 @@ function toggleChat() {
toast.error(t('buildScene.uploadSizeFail'));
}
} catch (err) {
exportStatus = null;
toast.error(err instanceof Error ? err.message : t('buildScene.videoExportFailed'));
} finally {
exportingVideo = false;
@@ -267,11 +271,15 @@ function toggleChat() {
const sendMutation = createMutation({
mutationFn: (payload: AgentComposerPayload) => agentApi.postMessage(sessionId!, payload),
onMutate: () => {
streaming = '';
streamingReasoning = '';
thinkingAgent = null;
liveTools = [];
livePlan = null;
const s = sessions.find((x) => x.id === sessionId);
const steering = Boolean(s?.running || s?.status === 'running');
if (!steering) {
streaming = '';
streamingReasoning = '';
thinkingAgent = null;
liveTools = [];
livePlan = null;
}
},
onSuccess: async () => {
await client.invalidateQueries({ queryKey: ['agent-session', sessionId] });
@@ -358,7 +366,7 @@ function toggleChat() {
}
switch (ev.type) {
case 'shot.updated':
shotStore.handleShotUpdated(ev.data as { shot_id?: number });
shotStore.handleShotUpdated(ev.data as { shot_id?: number; reason?: string });
// A pending agent capture request rides the same event: the
// viewport checks capture_request_json, renders the PNG, and
// posts it (the router clears the request, which is the
@@ -493,6 +501,11 @@ function toggleChat() {
<ShotViewport bind:this={viewportRef} oncapture={handleCaptureDataUrl} />
</div>
<TimelineStrip onExportVideo={handleExportVideo} />
{#if exportStatus}
<p class="export-status" data-export-status>
{exportStatus}
</p>
{/if}
<CaptureStrip {captures} compositionId={shotStore.compositionId} />
{:else}
<div class="loading">{t('buildScene.preparing')}</div>
@@ -585,6 +598,9 @@ function toggleChat() {
flex: 1;
min-height: 0;
position: relative;
display: flex;
overflow: hidden;
background: #0b1220;
}
.loading {
flex: 1;
@@ -666,4 +682,10 @@ function toggleChat() {
padding: 2px 8px;
opacity: 0.7;
}
.export-status {
margin: 0;
padding: 4px 12px;
font-size: 12px;
color: var(--muted, #9aa3b5);
}
</style>
@@ -268,6 +268,62 @@
}
}
// ---- node deletion (Backspace/Delete on selection) ----------------------
// Svelte Flow removes deleted nodes from the local store on its own; the
// handlers below make the removal real (persist + file cleanup) or veto it.
function nodeServerId(node: Node): number | null {
const id = (node.data as { canvasNodeId?: number }).canvasNodeId;
return typeof id === 'number' ? id : null;
}
async function onBeforeDeleteCanvas({ nodes }: { nodes: Node[] }): Promise<boolean> {
const blocked = nodes.filter(
(n) =>
n.type === 'entity' ||
(n.data as { status?: string }).status === 'running' ||
(n.data as { status?: string }).status === 'queued',
);
if (blocked.length > 0) {
toast.error(
t('canvas.deleteBlocked', { count: blocked.length }),
);
return false;
}
return true;
}
async function onDeleteCanvas({ nodes }: { nodes: Node[] }) {
const ids = nodes
.map(nodeServerId)
.filter((id): id is number => id != null);
if (ids.length === 0) return;
let fileDeletedCount = 0;
let keptCount = 0;
const results = await Promise.allSettled(
ids.map((id) => canvasApi.deleteNode(canvasId, id)),
);
for (const r of results) {
if (r.status === 'fulfilled') {
if (r.value.file_deleted) fileDeletedCount++;
else keptCount++;
}
}
await client.invalidateQueries({ queryKey: ['canvas', canvasId] });
const failed = results.filter((r) => r.status === 'rejected').length;
if (failed > 0) {
toast.error(t('canvas.deleteFailed'));
return;
}
if (fileDeletedCount > 0 && keptCount > 0) {
toast.success(t('canvas.deletedMixed', { files: fileDeletedCount, kept: keptCount }));
} else if (fileDeletedCount > 0) {
toast.success(t('canvas.deletedWithFile', { count: fileDeletedCount }));
} else {
toast.success(t('canvas.deletedCardOnly', { count: keptCount }));
}
}
// ---- viewport persistence --------------------------------------------
// Zoom/pan must survive reload, session switches (the {#key} layout
// remounts this page on canvas id change), and navigation away/back.
@@ -578,16 +634,20 @@
const sendMutation = createMutation({
mutationFn: (payload: AgentComposerPayload) => agentApi.postMessage(activeId!, payload),
onMutate: () => {
streaming = '';
streamingReasoning = '';
thinkingAgent = null;
liveTools = [];
livePlan = null;
if (activeId != null) {
client.setQueryData<AgentSession & { messages: AgentMessage[]; plan?: AgentPlan | null }>(
['agent-session', activeId],
(old) => (old ? { ...old, plan: null } : old),
);
const s = sessions.find((x) => x.id === activeId);
const steering = Boolean(s?.running || s?.status === 'running');
if (!steering) {
streaming = '';
streamingReasoning = '';
thinkingAgent = null;
liveTools = [];
livePlan = null;
client.setQueryData<AgentSession & { messages: AgentMessage[]; plan?: AgentPlan | null }>(
['agent-session', activeId],
(old) => (old ? { ...old, plan: null } : old),
);
}
}
},
onSuccess: async () => {
@@ -1056,6 +1116,8 @@ title={running
fitView={initialViewport === undefined}
onnodedragstart={onNodeDragStart}
onnodedragstop={onNodeDragStop}
onbeforedelete={onBeforeDeleteCanvas}
ondelete={onDeleteCanvas}
onmoveend={(_event, vp) => scheduleViewportSave(vp)}
oninit={assertViewportOnInit}
>
@@ -0,0 +1,495 @@
<script lang="ts">
import { createMutation, createQuery, useQueryClient } from '@tanstack/svelte-query';
import { libraryApi, type LibraryMediaItem } from '$lib/api';
import { assetUrl } from '$lib/api';
import { t } from '$lib/i18n.svelte';
import AppHeader from '$lib/components/AppHeader.svelte';
import AttachToProject from '$lib/components/AttachToProject.svelte';
import ImageLightbox from '$lib/components/ImageLightbox.svelte';
import SafeMedia from '$lib/components/SafeMedia.svelte';
import Button from '$lib/components/ui/Button.svelte';
import EmptyState from '$lib/components/ui/EmptyState.svelte';
import Icon from '$lib/components/ui/Icon.svelte';
import Skeleton from '$lib/components/ui/Skeleton.svelte';
import { toast } from '$lib/toast';
const client = useQueryClient();
const mediaQuery = createQuery({
queryKey: ['library-media'],
queryFn: libraryApi.list,
});
const items = $derived($mediaQuery.data ?? []);
// ---- filter + lazy paging ------------------------------------------------
// The listing endpoint returns the full (small) file index; filtering and
// paging are client-side. Media heavy <video> cards make a 50-per-page cap
// necessary even so.
const PAGE_SIZE = 50;
let filter = $state<'all' | 'image' | 'video'>('all');
let visibleCount = $state(PAGE_SIZE);
let loadingMore = $state(false);
let sentinel = $state<HTMLElement | null>(null);
const filtered = $derived(
filter === 'all' ? items : items.filter((i) => i.kind === filter),
);
const visible = $derived(filtered.slice(0, visibleCount));
const hasMore = $derived(filtered.length > visible.length);
const remaining = $derived(filtered.length - visible.length);
function loadMore() {
if (!hasMore || loadingMore) return;
loadingMore = true;
// Yield a frame so the loading row paints before the (sync) append.
setTimeout(() => {
visibleCount += PAGE_SIZE;
loadingMore = false;
}, 120);
}
$effect(() => {
// Auto-append the next page when the sentinel scrolls near the viewport.
if (!sentinel || !hasMore) return;
const io = new IntersectionObserver(
(entries) => {
if (entries.some((e) => e.isIntersecting)) loadMore();
},
{ rootMargin: '400px' },
);
io.observe(sentinel);
return () => io.disconnect();
});
$effect(() => {
// New filter → back to the first page.
void filter;
visibleCount = PAGE_SIZE;
});
let selected = $state<Set<string>>(new Set());
let preview = $state<{ src: string; kind: 'image' | 'video'; name: string } | null>(null);
let attachItem = $state<LibraryMediaItem | null>(null);
let confirmingDelete = $state(false);
const selectedCount = $derived(selected.size);
const allSelected = $derived(
filtered.length > 0 && filtered.every((i) => selected.has(i.path)),
);
function toggle(path: string) {
const next = new Set(selected);
if (next.has(path)) next.delete(path);
else next.add(path);
selected = next;
}
function toggleAll() {
selected = allSelected ? new Set() : new Set(filtered.map((i) => i.path));
}
function fmtSize(bytes: number): string {
if (bytes < 1024) return `${bytes} B`;
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(0)} KB`;
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
}
const deleteMutation = createMutation({
mutationFn: (paths: string[]) => libraryApi.deleteMany(paths),
onSuccess: (res) => {
selected = new Set();
confirmingDelete = false;
void client.invalidateQueries({ queryKey: ['library-media'] });
const kept = res.results.filter((r) => r.status === 'kept');
if (kept.length > 0) {
toast.error(t('library.deletedPartial', { kept: kept.length }));
} else {
toast.success(t('library.deleted', { count: res.deleted.length }));
}
},
onError: (err) => {
confirmingDelete = false;
toast.error(err instanceof Error ? err.message : t('library.deleteFailed'));
},
});
function deleteSelected() {
if (selected.size === 0) return;
$deleteMutation.mutate([...selected]);
}
function openPreview(item: LibraryMediaItem) {
const src = assetUrl(item.path);
if (!src) return;
preview = { src, kind: item.kind, name: item.name };
}
</script>
<AppHeader active="library">
{#if selectedCount > 0}
<Button variant="danger" onclick={() => (confirmingDelete = true)}>
<Icon name="close" size={14} />
{t('library.deleteSelected', { count: selectedCount })}
</Button>
{/if}
</AppHeader>
<main class="container">
<div class="toolbar">
<div class="filters" role="tablist" aria-label={t('library.filterLabel')}>
<button
type="button"
class="filter-chip"
class:active={filter === 'all'}
aria-pressed={filter === 'all'}
onclick={() => (filter = 'all')}
>
{t('library.filterAll')} <span class="chip-count">{items.length}</span>
</button>
<button
type="button"
class="filter-chip"
class:active={filter === 'image'}
aria-pressed={filter === 'image'}
onclick={() => (filter = 'image')}
>
{t('library.filterImage')}
<span class="chip-count">
{items.filter((i) => i.kind === 'image').length}
</span>
</button>
<button
type="button"
class="filter-chip"
class:active={filter === 'video'}
aria-pressed={filter === 'video'}
onclick={() => (filter = 'video')}
>
{t('library.filterVideo')}
<span class="chip-count">
{items.filter((i) => i.kind === 'video').length}
</span>
</button>
</div>
<label class="select-all">
<input type="checkbox" checked={allSelected} onchange={toggleAll} />
<span>{allSelected ? t('library.deselectAll') : t('library.selectAll')}</span>
</label>
<span class="count" data-testid="library-count">
{t('library.itemCount', { count: filtered.length })}
</span>
</div>
{#if $mediaQuery.isLoading}
<div class="grid">
{#each Array(8) as _, i (i)}
<div class="card skeleton-card"><Skeleton height="220px" /></div>
{/each}
</div>
{:else if filtered.length === 0}
<EmptyState
title={filter === 'all' ? t('library.emptyTitle') : t('library.emptyFilterTitle', { kind: filter === 'image' ? t('library.filterImage') : t('library.filterVideo') })}
body={filter === 'all' ? t('library.emptyBody') : t('library.emptyFilterBody')}
/>
{:else}
<div class="grid">
{#each visible as item (item.path)}
<div class="card" class:selected={selected.has(item.path)}>
<label class="check">
<input
type="checkbox"
checked={selected.has(item.path)}
onchange={() => toggle(item.path)}
/>
</label>
<button
type="button"
class="media"
onclick={() => openPreview(item)}
title={item.name}
>
<SafeMedia src={assetUrl(item.path)} kind={item.kind} alt={item.name} controls={false} />
</button>
<div class="meta">
<span class="name" title={item.name}>{item.name}</span>
<span class="sub">{item.kind} · {fmtSize(item.size)}</span>
</div>
<div class="card-actions">
<Button variant="secondary" onclick={() => (attachItem = item)}>
{t('library.addToProject')}
</Button>
</div>
</div>
{/each}
</div>
{#if hasMore}
<div class="load-more" bind:this={sentinel} aria-hidden="true"></div>
<div class="more-row">
{#if loadingMore}
<span class="loading-note">{t('library.loadingMore')}</span>
{:else}
<Button variant="secondary" onclick={loadMore}>
{t('library.loadMore', { count: remaining })}
</Button>
{/if}
</div>
{/if}
{/if}
</main>
{#if preview}
<ImageLightbox
src={preview.src}
alt={preview.name}
kind={preview.kind}
caption={preview.name}
onClose={() => (preview = null)}
/>
{/if}
{#if attachItem}
<div class="modal-backdrop">
<div
class="modal"
role="dialog"
aria-modal="true"
aria-label={t('library.addToProject')}
>
<div class="modal-head">
<strong>{t('library.addToProject')}</strong>
<button type="button" class="modal-close" onclick={() => (attachItem = null)}>
<Icon name="close" size={14} />
</button>
</div>
<p class="modal-file">{attachItem.name}</p>
<AttachToProject path={attachItem.path} kind={attachItem.kind} />
</div>
</div>
{/if}
{#if confirmingDelete}
<div class="modal-backdrop" role="presentation">
<div class="modal confirm" role="alertdialog" aria-modal="true">
<p>{t('library.confirmDelete', { count: selectedCount })}</p>
<div class="modal-actions">
<Button variant="danger" loading={$deleteMutation.isPending} onclick={deleteSelected}>
{t('common.delete')}
</Button>
<Button variant="ghost" onclick={() => (confirmingDelete = false)}>
{t('common.cancel')}
</Button>
</div>
</div>
</div>
{/if}
<style>
.toolbar {
display: flex;
align-items: center;
gap: 16px;
margin-bottom: 14px;
flex-wrap: wrap;
}
.filters {
display: flex;
align-items: center;
gap: 6px;
}
.filter-chip {
display: inline-flex;
align-items: center;
gap: 6px;
padding: 6px 12px;
border: 1px solid var(--border);
border-radius: 999px;
background: transparent;
color: var(--text-secondary);
font-size: 13px;
font-weight: 500;
cursor: pointer;
transition: color 0.15s, background 0.15s, border-color 0.15s;
}
.filter-chip:hover {
color: var(--text-primary);
background: var(--bg-elevated);
}
.filter-chip.active {
color: var(--text-primary);
background: var(--bg-elevated);
border-color: var(--accent);
}
.filter-chip:focus-visible {
outline: 2px solid var(--accent);
outline-offset: 2px;
}
.chip-count {
font-size: 11px;
color: var(--text-muted);
background: rgba(255, 255, 255, 0.06);
border-radius: 999px;
padding: 1px 7px;
}
.select-all {
display: flex;
align-items: center;
gap: 8px;
font-size: 13px;
color: var(--text-secondary);
cursor: pointer;
}
.count {
font-size: 13px;
color: var(--text-muted);
}
.grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 14px;
}
.card {
position: relative;
display: flex;
flex-direction: column;
background: var(--bg-elevated);
border: 1px solid var(--border);
border-radius: var(--radius-md);
overflow: hidden;
transition: border-color 0.15s ease;
}
.card.selected {
border-color: var(--accent);
}
.card :global(.media-el) {
object-fit: cover;
}
.skeleton-card {
height: 240px;
}
.check {
position: absolute;
top: 8px;
left: 8px;
z-index: 2;
display: flex;
align-items: center;
justify-content: center;
width: 26px;
height: 26px;
background: rgba(0, 0, 0, 0.55);
border-radius: var(--radius-sm);
cursor: pointer;
}
.check input {
width: 15px;
height: 15px;
accent-color: var(--accent);
cursor: pointer;
}
.media {
display: block;
width: 100%;
height: 150px;
padding: 0;
border: none;
background: rgba(0, 0, 0, 0.3);
cursor: zoom-in;
overflow: hidden;
}
.media :global(img),
.media :global(video) {
width: 100%;
height: 100%;
object-fit: cover;
}
.meta {
display: flex;
flex-direction: column;
gap: 2px;
padding: 10px 12px 6px;
min-width: 0;
}
.name {
font-size: 13px;
color: var(--text-primary);
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.sub {
font-size: 11px;
color: var(--text-muted);
}
.card-actions {
padding: 8px 12px 12px;
}
.load-more {
height: 1px;
}
.more-row {
display: flex;
justify-content: center;
padding: 18px 0 6px;
}
.loading-note {
font-size: 13px;
color: var(--text-muted);
}
.modal-backdrop {
position: fixed;
inset: 0;
z-index: 60;
display: flex;
align-items: center;
justify-content: center;
background: rgba(0, 0, 0, 0.6);
padding: 20px;
}
.modal {
width: min(420px, 100%);
background: var(--bg-elevated);
border: 1px solid var(--border);
border-radius: var(--radius-lg);
padding: 16px;
}
.modal.confirm {
width: min(360px, 100%);
}
.modal-head {
display: flex;
align-items: center;
justify-content: space-between;
margin-bottom: 8px;
}
.modal-close {
display: flex;
align-items: center;
justify-content: center;
width: 24px;
height: 24px;
border: none;
border-radius: var(--radius-sm);
background: transparent;
color: var(--text-muted);
cursor: pointer;
}
.modal-close:hover {
color: var(--text-primary);
background: rgba(255, 255, 255, 0.06);
}
.modal-file {
margin: 0 0 10px;
font-size: 12px;
color: var(--text-muted);
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.modal-actions {
display: flex;
gap: 8px;
margin-top: 12px;
}
</style>
@@ -0,0 +1,100 @@
# Calliope 1.5.4 — steering input, Asset Library, canvas delete, agent hardening
A feature release in three waves: the agent loops got the safety nets and
capabilities from the loop evaluation, AI Canvas got honest deletion, and
unlinked Playground media got a home of its own.
## What changed
### Steering input (AI Canvas + Build Scene)
- Messages sent **while the agent is running** now steer the turn instead of
failing with a 409. `POST /sessions/{id}/messages` detects a running turn and
appends a `steering/message` event + a mirrored `status='steering'` chat row.
- Both loops drain steering **at step boundaries only** — a user message
injected between an assistant `tool_calls` message and its results is an
invalid OpenAI sequence, so history derivation buffers mid-exchange steering
until the exchange closes. Injected turns are labelled
`[STEERING — user message sent while you work]`.
- Steering course-corrects; it can never **grant** permission. Render /
destructive approval still derives from the latest `user/message` only.
- Composer: editable + Send while running (button reads "Steer"), Stop stays
put; steering bubbles render with a distinct marker in the chat.
- Blind sessions skip the planner entirely — the Build Scene agent can only
misroute there, so it runs the single loop.
### Agent loop hardening (7 fixes from the loop evaluation)
- **Swarm safety nets.** Sub-agents now run the same guards as the main loop —
shared `RepeatGuard`, fail-streak `[SYSTEM DIRECTIVE]`, and a final-step
nudge. A sub-agent `ask_user` **pauses the whole swarm**: remaining tasks are
marked `skipped` (new task status, surfaced in the plan panel) and the turn
ends with the question card; replying continues naturally.
- **Clip tools in swarm roles.** `script` gains
`list_clips`/`break_into_shots`/`add_clip`/`update_clip`/`delete_clip`,
`video` gains `list_clips`; a category→role drift test prevents a repeat.
- **Deleted the unfulfillable ComfyUI exception** from the Build Scene prompt —
the tool scope guard has no such path, and the prompt was teaching the agent
to promise one.
- **`request_capture` no longer lies.** The 10s timeout returns `ok: False`
with `capture_pending: True` and explicit "do NOT reference a file path"
guidance, feeding the fail-streak guard instead of fabricating successes.
- **Planner fast-path.** Trivial single-verb goals skip the planner LLM call
and go straight to the single loop; blind sessions no longer pay a wasted
workspace read before early-outing.
- **Per-tool wall-clock budget.** New `agent_tool_timeout_sec` setting
(default 600, `0` = off) wraps `registry.execute` in `asyncio.wait_for`;
`wait_for_jobs` is exempt (`long_running=True`, it owns its timeout contract)
and now publishes `agent.thinking` progress while polling.
- **Token-bounded history.** `derive_llm_history` gains a character budget
(`agent_history_char_budget`, default ~400k chars) that drops the oldest
whole turns — turn-count alone let long sessions overflow the context window.
### AI Canvas deletion that deletes
- Backspace/Delete on artifact cards now **persists**: the frontend
`onbeforedelete`/`ondelete` handlers call the backend, which hard-deletes the
artifact node row, tombstones entity nodes, and publishes `canvas.updated`
so the refetch cannot resurrect the card.
- **File cleanup is rule-based, not blind.** The backing file is deleted only
when it lives in playground-managed folders (the `uploads/` library or the
scratch project's output folders) **and** nothing references it — entity
images, scene/clip videos, or another live canvas card. Project-owned files
are replace-only and always survive; the toast says which happened
(`file_deleted` / `reason`).
- Deleting a card was the only way to "remove" a stray Playground generation
before; the reappearing-card bug made that impossible. Fixed.
### Asset Library (`/library`)
- New page listing **unlinked** Playground media — the `uploads/` library plus
the scratch project's image/video outputs. Files referenced by project data
are excluded: they live in Project Assets and are replace-only.
- Grid with **All / Image / Video filters** (live counts), multi-select,
**bulk delete** (max 200 per request; referenced/project-owned/foreign paths
are kept and reported per-path), and **Add to Project** via the existing
attachment modal.
- **Lazy loading at 50 per page** — an `IntersectionObserver` fetches the next
page as you scroll, with a manual "Load more" fallback.
- Backend: `GET/DELETE /api/library/media`; new nav link in the header;
localized in all seven languages.
### Build Scene: Brief/Cut gates + MP4 export
- The shot-composer-blockout agent now works behind two human gates: **Brief**
before scene mutations and **Cut** before export (`get_build_scene_gates` /
`record_build_scene_gate` tools; soft warning when mutating without a brief;
the UI records gates with `source: 'ui'`; `clear_scene` preserves them).
- New guard code `guard_no_export_video_tool` — the Build Scene agent has no
export tool and is stopped honestly at the gate instead of guessing.
- Export video normalizes to **H.264 MP4 at ingest** (`_transcode_video_to_mp4`,
`libx264 + yuv420p + faststart`, 503 when ffmpeg is missing) so the project
clip picker always receives a real MP4 regardless of the recording browser.
## Upgrade
No DB migration and no new frontend dependencies; the backend gains
`pillow`. `git pull`, then `uv sync` in `calliope-backend/` (or let the start
script do it) and reload the frontend. New settings
(`agent_tool_timeout_sec`, `agent_history_char_budget`) default sensibly —
nothing to configure.
+1
View File
@@ -2,6 +2,7 @@
Dated notes for GitHub releases. Newest first.
- [2026-09-19 steering + Asset Library + hardening](./2026-09-19-steering-library-hardening.md) — mid-run steering; Asset Library page (filter + lazy loading); canvas delete that deletes; agent-loop hardening batch; Build Scene Brief/Cut gates (1.5.4)
- [2026-09-16 SSR storage fix](./2026-09-16-ssr-storage-fix.md) — fix the `localStorage.getItem is not a function` crash on Node 22+; new `npm test` guard (1.5.3)
- [2026-09-16 multilingual UI](./2026-09-16-multilingual-ui.md) — seven-language UI (en/zh/es/fr/de/ja/ko); header language switcher; correct backend version; local start scripts (1.5.2)
- [2026-09-13 clip shot brief](./2026-09-13-clip-shot-brief.md) — per-clip brief in the Video composer; two-column Video stage (1.5.1)