From 11ddf231e0dcfb3e5b26e23a60123e0f8ae69ce1 Mon Sep 17 00:00:00 2001 From: Martin Delophy <845046459@qq.com> Date: Wed, 22 Jul 2026 18:47:40 +0800 Subject: [PATCH] Publish Timeline Studio agent editing skill (#42) * publish timeline agent editing skill * install ffmpeg in CI --------- Co-authored-by: haixin.yang --- .github/workflows/ci.yml | 2 + README.md | 15 +- docs/community-post-zh.md | 190 ++++++ scripts/timeline-command.mjs | 178 +++++- .../.claude-plugin/plugin.json | 2 +- skills/edit-timeline-studio/README.md | 9 +- skills/edit-timeline-studio/SKILL.md | 7 +- .../docs/agent-video-editing.md | 9 +- .../docs/claude-code-video-editing.md | 7 +- .../docs/codex-video-editing.md | 7 +- .../docs/command-reference.md | 52 +- skills/edit-timeline-studio/docs/examples.md | 6 +- .../docs/gemini-cli-video-editing.md | 28 + .../docs/github-copilot-video-editing.md | 23 + .../references/command-contract.md | 26 +- .../references/current-capabilities.md | 23 +- src/hooks/useProjectFiles.js | 15 +- src/lib/projectArchive.js | 4 + src/lib/projectArchive.test.js | 17 + src/lib/projectCommandEngine.js | 543 +++++++++++++++++- src/lib/projectCommandEngine.test.js | 237 +++++++- src/lib/projectRenderPlan.js | 109 ++++ src/lib/projectRenderPlan.test.js | 40 ++ src/lib/timelineCommandCli.test.js | 131 ++++- 24 files changed, 1596 insertions(+), 84 deletions(-) create mode 100644 docs/community-post-zh.md create mode 100644 skills/edit-timeline-studio/docs/gemini-cli-video-editing.md create mode 100644 skills/edit-timeline-studio/docs/github-copilot-video-editing.md create mode 100644 src/lib/projectArchive.test.js create mode 100644 src/lib/projectRenderPlan.js create mode 100644 src/lib/projectRenderPlan.test.js diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b680b39..acfb59c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -18,6 +18,8 @@ jobs: with: node-version: 22 cache: npm + - name: Install media tooling + run: sudo apt-get update && sudo apt-get install --yes ffmpeg - run: npm ci - run: npm run lint - run: npm run typecheck diff --git a/README.md b/README.md index 98bb61f..bbc1686 100644 --- a/README.md +++ b/README.md @@ -45,14 +45,19 @@ It helps an agent: - verify track placement, transitions, captions, overlays, audible audio, and final export artifacts; - keep the editable `.timeline` project as the source of truth instead of returning only an opaque render. -The first versioned headless command runner is now available. It loads and inspects portable projects, validates revisioned JSON plans, applies supported operations transactionally, supports dry runs and idempotent operation IDs, and writes a new `.timeline` archive without rewriting its media files. Browser control remains the compatibility path for operations that are not in the command registry yet. +The versioned headless command runner loads and inspects portable projects, validates revisioned JSON plans, applies supported operations transactionally, supports dry runs and idempotent operation IDs, and writes a new `.timeline` archive without rewriting its media files. It also renders the documented portable Visuals + Voiceover + Music subset to a verified H.264/AAC MP4. Browser control remains the compatibility path for richer compositions and operations that are not in the command registry yet. ```bash -npm run agent -- inspect /absolute/path/project.timeline -npm run agent -- run /absolute/path/edit-plan.json +npm run agent -- project.inspect /absolute/path/project.timeline +npm run agent -- track.inspect /absolute/path/project.timeline visuals +npm run agent -- clip.inspect /absolute/path/project.timeline visual-123 +npm run agent -- transcript.inspect /absolute/path/project.timeline voice-123 +npm run agent -- project.diff /absolute/path/edit-plan.json +npm run agent -- project.run /absolute/path/edit-plan.json +npm run agent -- project.render /absolute/path/render-request.json ``` -The initial write registry supports `timed.move` for voiceover clips, `caption.update`, `caption.unlink_audio`, and `caption.link_audio`. See the [command contract](skills/edit-timeline-studio/references/command-contract.md) for the plan envelope. +The legacy `inspect` and `run` aliases remain available. The write registry supports ffprobe-backed, hashed visual/audio import to Visuals, Music, or the portable Voiceover slot, plus transactional timed edits, captions, source-accurate Visuals/Overlays, transitions, validated properties, track state, and ratio changes. Read commands return project, track, clip, transcript, media-inventory, and field-level predicted diffs. See the [command contract](skills/edit-timeline-studio/references/command-contract.md) for the plan envelope. Install through the public [skills.sh](https://skills.sh/MartinDelophy/ai-video-editor) directory (the current CLI requires Node.js 22.20.0 or later): @@ -79,7 +84,7 @@ gh skill preview MartinDelophy/ai-video-editor edit-timeline-studio ## Roadmap - **Now:** expand the versioned command registry, harden deterministic offline export, and improve timeline editing reliability. -- **Next:** add media probing/render commands and expose the shared command engine through MCP. +- **Next:** expand headless render parity and expose the shared command engine through MCP. - **Later:** add collaborative review workflows, a plugin extension surface, and more locally verified AI models. Roadmap priorities are shaped in [GitHub Discussions](https://github.com/MartinDelophy/ai-video-editor/discussions). Feature requests and real-world workflow feedback are welcome. diff --git a/docs/community-post-zh.md b/docs/community-post-zh.md new file mode 100644 index 0000000..ad81979 --- /dev/null +++ b/docs/community-post-zh.md @@ -0,0 +1,190 @@ +# 不用安装剪辑软件，AI 配音、字幕和视频导出都在浏览器完成：开源 Timeline Studio + +做短视频时，你是否也遇到过这些问题？ + +- 为了完成一次简单剪辑，要先安装体积庞大的桌面软件； +- 素材必须上传服务器，隐私和等待时间难以控制； +- AI 配音、自动字幕和高清导出分散在多个工具里； +- 手机与桌面项目来回迁移，工程很难继续编辑； +- AI Agent 可以生成文案，却很难稳定、可逆地修改时间线。 + +我们一直在尝试解决这些问题，于是有了 **Timeline Studio**：一款本地优先、开源、直接运行在现代浏览器中的 AI 视频编辑器。 + +项目地址：https://github.com/MartinDelophy/ai-video-editor + +在线体验：https://video-editor.ai-creator.top/ + +## 一、Timeline Studio 是什么？ + +Timeline Studio 把接近剪映/CapCut 的多轨时间线、浏览器端 AI 能力和离线视频渲染放进同一个工作区。 + +它的核心理念是： + +> 让视频项目尽可能留在用户设备上，让 AI 真正参与可继续编辑、可撤销、可验证的创作流程。 + +打开网页后，用户可以导入图片、视频和音频，在时间线上完成画面编排、字幕、贴纸、配音、音乐、画中画与属性调整，并直接在浏览器中预览和导出。 + +## 二、为什么强调“本地优先”？ + +Timeline Studio 的主要编辑状态、媒体处理和已支持的 AI 推理都尽可能在浏览器本地完成。 + +这带来几个直接好处： + +- 素材不必为了基础剪辑上传到远程编辑后端； +- 浏览器可按需缓存 AI 模型，重复使用时无需每次重新下载； +- 工程可以导出为可移植的 `.timeline` 文件； +- 用户可以保存工程、迁移设备，之后继续编辑； +- 支持安装为 PWA，获得更接近桌面应用的使用方式。 + +需要使用在线模型或第三方生成服务的能力，仍然取决于对应服务配置；本地优先并不意味着所有 AI 模型都能在每台设备上离线运行。 + +## 三、像专业剪辑软件一样组织时间线 + +Timeline Studio 不是只有一个上传按钮和“自动生成”按钮，而是一套可手工控制的多轨编辑工作区。 + +目前编辑器包含： + +- 连续的主画面轨； +- 图片和视频画中画/Overlay； +- 字幕轨与字幕样式； +- 可调整时长的贴纸轨； +- 配音、视频原声与背景音乐轨； +- 片段移动、裁剪、切分、复制和删除； +- 时间线磁吸、对齐线、缩放和撤销/重做； +- 画布内移动、缩放和旋转； +- 遮罩、滤镜、效果、动画、速度和关键帧； +- 针对触屏重新设计的移动端时间线与属性面板。 + +项目不是一次性生成的黑盒视频。字幕、声音、素材位置和时间关系都继续保留在时间线上，用户可以随时接手修改。 + +## 四、浏览器里的 AI 配音与自动字幕 + +Timeline Studio 将常见的音视频 AI 能力放进编辑工作流，而不是让用户频繁导入导出中间文件。 + +### 4.1 多语言 AI 配音 + +中文配音使用浏览器 Piper/VITS ONNX 路径，英文提供 Kokoro 82M，并为德语、西班牙语、法语、意大利语和巴西葡萄牙语提供浏览器 Piper 声音。 + +生成后的声音会成为时间线上的可编辑配音片段，而不是只能试听或下载的孤立结果。 + +### 4.2 Whisper 自动字幕 + +自动字幕默认使用 Whisper small q8 ONNX。识别结果会结合音频能量调整字幕边界，使字幕片段更接近实际语音节奏；中文流程还会进行克制的高置信上下文纠错。 + +### 4.3 人声分离与智能画面 + +编辑器支持在浏览器工作流中进行人声分离，将人声与伴奏放入对应音轨；智能画面能力则可以进行主体检测、人像抠图、智能裁切和字幕避让。 + +### 4.4 数字人工作流 + +项目还在探索 JoyVASA 音频驱动与 LivePortrait 神经渲染，让配音、角色画面和时间线继续保持在同一个可编辑项目里。 + +## 五、预览和最终导出为什么要分开？ + +视频编辑器的实时预览追求流畅，而最终导出追求确定性和画面一致性。Timeline Studio 因此采用两条路径： + +- 播放预览使用浏览器原生媒体能力，保证编辑时响应及时； +- 最终导出使用 WebCodecs 驱动的离线合成流程； +- 画面、字幕、贴纸、画中画、效果、变换和音频混音在同一个合成逻辑中处理； +- 不支持 WebCodecs 的环境提供 MediaRecorder 兼容回退。 + +这一设计的目标是让预览与最终文件尽量保持 WYSIWYG，同时允许程序化测试去解码并验证导出文件的尺寸、时长、画面和音轨。 + +## 六、`.timeline`：可以继续编辑的可移植工程 + +Timeline Studio 可以把项目导出为 `.timeline` 工程包。它不只是保存一个最终视频，还会保存项目结构和工程需要的媒体数据。 + +这意味着： + +1. 在浏览器中完成一个项目； +2. 导出 `.timeline`； +3. 在另一台设备或之后的会话中重新导入； +4. 恢复媒体和时间线，继续编辑。 + +可移植工程也是 Timeline Studio Agent 自动化能力的基础。 + +## 七、让 AI Agent 修改工程，而不是模拟鼠标 + +项目已经发布第一版 **versioned headless command runner**。它是一个面向 Agent 和脚本的本地命令执行层，可以： + +- 检查 `.timeline` 工程； +- 校验带版本和 revision 的 JSON 编辑计划； +- dry-run 并查看修改前后的差异； +- 事务化执行一组命令，失败时不留下“只改了一半”的工程； +- 使用幂等 operation ID，避免重复执行同一操作； +- 保存新的 `.timeline` 工程，同时保留归档中的媒体文件。 + +首版原生命令主要覆盖配音片段移动、关联字幕同步、字幕文本修改，以及字幕与音频的关联/解除。素材导入、AI 生成和最终渲染等尚未进入命令注册表的能力，目前仍通过浏览器兼容路径完成。 + +它不是远程控制网页的“鼠标机器人”。更准确地说，它让 AI 能够稳定修改可移植工程文件，再由用户将结果导回在线编辑器。随着命令范围扩大，它将成为自动剪辑、批量字幕处理和可复现视频工作流的基础。 + +## 八、桌面与手机都不是“附带支持” + +很多 Web 编辑器只是把桌面布局压缩到手机屏幕。Timeline Studio 将移动端视为触控优先的垂直剪辑工作区： + +- 预览画面保持醒目； +- 时间线紧邻预览； +- 工具和属性使用有边界的底部抽屉； +- 时间线围绕固定居中播放头滚动； +- 片段选择后出现类型对应的操作栏； +- 拖动、裁剪、缩放和纵向滚动按触控手势重新设计。 + +响应式一致性被当作发布要求，而不是项目完成后的补丁。 + +## 九、快速开始 + +在线使用： + +https://video-editor.ai-creator.top/ + +本地运行： + +```bash +git clone https://github.com/MartinDelophy/ai-video-editor.git +cd ai-video-editor +npm install +npm run dev +``` + +运行 Agent 工程检查： + +```bash +npm run agent -- inspect /absolute/path/project.timeline +``` + +执行经过校验的编辑计划： + +```bash +npm run agent -- run /absolute/path/edit-plan.json +``` + +首次使用某项 AI 能力时可能需要下载对应模型，之后会优先复用浏览器缓存。运行大型模型时，推荐使用支持 WebGPU 的现代 Chromium 浏览器。 + +## 十、技术栈与工程方向 + +- React + Vite； +- WebCodecs / Canvas / OfflineAudioContext； +- ONNX Runtime Web、Piper/VITS、Kokoro、Whisper； +- Service Worker 与 PWA； +- 可移植 `.timeline` ZIP 工程； +- Vitest 与浏览器端到端验证； +- MIT License。 + +当前开发重点包括提高离线导出的稳定性、完善桌面和移动端时间线交互、扩充 Agent 命令覆盖，以及为真实工程建立更完整的回归测试。 + +## 十一、欢迎体验和参与 + +Timeline Studio 仍在持续快速迭代。浏览器媒体、WebCodecs、WebGPU、ONNX、时间线交互和移动端剪辑都存在大量值得深入的问题。 + +如果你对这个方向感兴趣，欢迎： + +- 打开在线编辑器体验真实工作流； +- 在 GitHub 提交可复现的问题； +- 参与 Discussions，分享需求和作品； +- 贡献测试、翻译、文档或聚焦的小型修复； +- 如果项目对你有帮助，点亮一个 Star。 + +项目地址：https://github.com/MartinDelophy/ai-video-editor + +在线体验：https://video-editor.ai-creator.top/ + diff --git a/scripts/timeline-command.mjs b/scripts/timeline-command.mjs index 3068297..1f446f0 100644 --- a/scripts/timeline-command.mjs +++ b/scripts/timeline-command.mjs @@ -1,11 +1,17 @@ #!/usr/bin/env node -import { readFile, writeFile } from "node:fs/promises"; -import { resolve } from "node:path"; +import { createHash } from "node:crypto"; +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, readFile, rename, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { basename, dirname, extname, isAbsolute, join, resolve, sep } from "node:path"; +import { promisify } from "node:util"; import { strFromU8, strToU8, unzipSync, zipSync } from "fflate"; -import { applyCommandPlan, inspectProject } from "../src/lib/projectCommandEngine.js"; +import { applyCommandPlan, inspectClip, inspectProject, inspectTrack, inspectTranscript } from "../src/lib/projectCommandEngine.js"; +import { buildFfmpegRenderPlan } from "../src/lib/projectRenderPlan.js"; const PROJECT_FILE = "project.json"; +const executeFile = promisify(execFile); async function readArchive(path) { const files = unzipSync(new Uint8Array(await readFile(path))); @@ -19,17 +25,140 @@ function print(value) { process.stdout.write(`${JSON.stringify(value, null, 2)}\n`); } -async function main() { - const [command, argument] = process.argv.slice(2); - if (!command || !argument || !["inspect", "run"].includes(command)) { - throw Object.assign(new Error("Usage: npm run agent -- inspect | run "), { exitCode: 2 }); +const VISUAL_EXTENSIONS = Object.freeze({ + ".jpg": ["image", "image/jpeg"], ".jpeg": ["image", "image/jpeg"], ".png": ["image", "image/png"], + ".webp": ["image", "image/webp"], ".gif": ["image", "image/gif"], ".mp4": ["video", "video/mp4"], + ".webm": ["video", "video/webm"], ".mov": ["video", "video/quicktime"], + ".mp3": ["audio", "audio/mpeg"], ".wav": ["audio", "audio/wav"], ".m4a": ["audio", "audio/mp4"], + ".aac": ["audio", "audio/aac"], ".ogg": ["audio", "audio/ogg"], ".flac": ["audio", "audio/flac"], +}); + +function safeArchiveName(value) { + return String(value || "asset").replace(/[^a-zA-Z0-9._-]+/g, "-").slice(0, 96) || "asset"; +} + +async function prepareAssetImports(plan) { + const imports = new Map(); + const operations = []; + for (const operation of plan.operations || []) { + if (operation.type !== "asset.import") { operations.push(operation); continue; } + if (typeof operation.file !== "string" || !isAbsolute(operation.file)) { + throw Object.assign(new Error("asset.import file must be an absolute path"), { code: "INVALID_ARGUMENT" }); + } + const extension = extname(operation.file).toLowerCase(); + const inferred = VISUAL_EXTENSIONS[extension]; + if (!inferred) throw Object.assign(new Error(`Unsupported asset extension: ${extension || "none"}`), { code: "UNSUPPORTED_MEDIA_TYPE" }); + const bytes = new Uint8Array(await readFile(operation.file)); + if (!bytes.length) throw Object.assign(new Error("Imported asset is empty"), { code: "INVALID_ASSET" }); + const sha256 = createHash("sha256").update(bytes).digest("hex"); + const needsProbe = inferred[0] === "audio" || (inferred[0] === "video" && (!operation.duration || !operation.width || !operation.height)) || (inferred[0] === "image" && (!operation.width || !operation.height)); + let probe = {}; + if (needsProbe) { + try { + const { stdout } = await executeFile("ffprobe", ["-v", "error", "-show_entries", "format=duration:stream=codec_type,width,height,duration", "-of", "json", operation.file]); + const metadata = JSON.parse(stdout); + const video = metadata.streams?.find((stream) => stream.codec_type === "video") || {}; + probe = { duration: Number(metadata.format?.duration) || Number(video.duration) || 0, width: Number(video.width) || 0, height: Number(video.height) || 0 }; + } catch (error) { + throw Object.assign(new Error(`Unable to probe imported asset: ${basename(operation.file)}`), { code: "MEDIA_PROBE_FAILED", cause: error }); + } + } + const folder = inferred[0] === "audio" ? "audio" : "visuals"; + const archivePath = `media/${folder}/import-${sha256.slice(0, 12)}-${safeArchiveName(operation.name || basename(operation.file))}`; + const duration = Number(operation.duration) > 0 ? Number(operation.duration) : Number(probe.duration) > 0 ? Number(probe.duration) : inferred[0] === "image" ? 4 : undefined; + const prepared = { ...operation, prepared: true, mediaType: inferred[0], mimeType: operation.mimeType || inferred[1], name: operation.name || basename(operation.file), size: bytes.length, sha256, archivePath, duration, width: operation.width ?? probe.width, height: operation.height ?? probe.height }; + delete prepared.file; + operations.push(prepared); + imports.set(operation.id, { bytes, track: operation.track, manifest: { id: operation.clipId, path: archivePath, name: prepared.name, type: prepared.mimeType, size: bytes.length, sha256 } }); } - if (command === "inspect") { - const { payload } = await readArchive(resolve(argument)); - print({ ok: true, ...inspectProject(payload.project) }); + return { plan: { ...plan, operations }, imports }; +} + +function safeArchivePath(path) { + const normalized = String(path || "").replaceAll("\\", "/"); + if (!normalized || normalized.startsWith("/") || normalized.split("/").includes("..")) { + throw Object.assign(new Error(`Unsafe media path in archive: ${path}`), { code: "INVALID_ARCHIVE_PATH" }); + } + return normalized; +} + +async function renderProject(renderPath) { + const request = JSON.parse(await readFile(resolve(renderPath), "utf8")); + if (request?.schemaVersion !== 1) throw Object.assign(new Error("project.render schemaVersion must be 1"), { code: "UNSUPPORTED_SCHEMA_VERSION" }); + if (typeof request.project !== "string" || !request.project) throw Object.assign(new Error("project is required"), { code: "INVALID_ARGUMENT" }); + if (typeof request.output?.video !== "string" || !request.output.video) throw Object.assign(new Error("output.video is required"), { code: "INVALID_ARGUMENT" }); + const projectPath = resolve(request.project); + const outputPath = resolve(request.output.video); + if (extname(outputPath).toLowerCase() !== ".mp4") throw Object.assign(new Error("project.render currently outputs MP4 only"), { code: "UNSUPPORTED_OUTPUT_FORMAT" }); + const { files, payload } = await readArchive(projectPath); + const extractionRoot = await mkdtemp(join(tmpdir(), "timeline-render-")); + const extractedFiles = new Map(); + const temporaryOutput = join(dirname(outputPath), `.${basename(outputPath)}.${process.pid}.tmp.mp4`); + const abortController = new AbortController(); + const cancel = () => abortController.abort(); + process.once("SIGINT", cancel); process.once("SIGTERM", cancel); + try { + for (const [archivePath, bytes] of Object.entries(files)) { + if (archivePath === PROJECT_FILE) continue; + const safePath = safeArchivePath(archivePath); + const extractedPath = resolve(extractionRoot, ...safePath.split("/")); + if (!extractedPath.startsWith(`${extractionRoot}${sep}`)) throw Object.assign(new Error(`Unsafe media path in archive: ${archivePath}`), { code: "INVALID_ARCHIVE_PATH" }); + await mkdir(dirname(extractedPath), { recursive: true }); + await writeFile(extractedPath, bytes); + extractedFiles.set(archivePath, extractedPath); + } + const renderPlan = buildFfmpegRenderPlan({ project: payload.project, media: payload.media, extractedFiles, settings: request.render }); + await mkdir(dirname(outputPath), { recursive: true }); + await executeFile("ffmpeg", [...renderPlan.args, temporaryOutput], { signal: abortController.signal, maxBuffer: 10 * 1024 * 1024 }); + await rename(temporaryOutput, outputPath); + const { stdout } = await executeFile("ffprobe", ["-v", "error", "-show_entries", "format=duration:stream=codec_type,width,height", "-of", "json", outputPath]); + const verification = JSON.parse(stdout); + const video = verification.streams?.find((stream) => stream.codec_type === "video"); + const audio = verification.streams?.find((stream) => stream.codec_type === "audio"); + if (!video || video.width !== renderPlan.width || video.height !== renderPlan.height) { + throw Object.assign(new Error("Rendered video verification failed"), { code: "RENDER_VERIFICATION_FAILED" }); + } + return { + ok: true, + revision: inspectProject(payload.project).revision, + artifacts: { project: projectPath, video: outputPath }, + render: { ...renderPlan, args: undefined, codec: "h264", container: "mp4" }, + verification: { duration: Number(verification.format?.duration) || 0, width: video.width, height: video.height, hasAudio: Boolean(audio) }, + }; + } catch (error) { + await rm(temporaryOutput, { force: true }).catch(() => {}); + if (error?.name === "AbortError") throw Object.assign(new Error("Render canceled"), { code: "RENDER_CANCELED" }); + throw error; + } finally { + process.removeListener("SIGINT", cancel); process.removeListener("SIGTERM", cancel); + await rm(extractionRoot, { recursive: true, force: true }); + } +} + +async function main() { + const [rawCommand, argument, selector] = process.argv.slice(2); + const command = rawCommand === "inspect" ? "project.inspect" : rawCommand === "run" ? "project.run" : rawCommand; + const supported = ["project.inspect", "track.inspect", "clip.inspect", "transcript.inspect", "project.diff", "project.run", "project.render"]; + if (!command || !argument || !supported.includes(command)) { + throw Object.assign(new Error("Usage: npm run agent -- project.inspect | track.inspect | clip.inspect | transcript.inspect [audioClipId] | project.diff | project.run | project.render "), { exitCode: 2, code: "INVALID_COMMAND" }); + } + if (command === "project.render") { print(await renderProject(argument)); return; } + if (["project.inspect", "track.inspect", "clip.inspect", "transcript.inspect"].includes(command)) { + const { files, payload } = await readArchive(resolve(argument)); + if (command === "track.inspect" && !selector) throw Object.assign(new Error("track is required"), { code: "INVALID_ARGUMENT" }); + if (command === "clip.inspect" && !selector) throw Object.assign(new Error("clipId is required"), { code: "INVALID_ARGUMENT" }); + if (command === "track.inspect") print({ ok: true, ...inspectTrack(payload.project, selector) }); + else if (command === "clip.inspect") print({ ok: true, ...inspectClip(payload.project, selector) }); + else if (command === "transcript.inspect") print({ ok: true, ...inspectTranscript(payload.project, selector || "") }); + else { + const mediaPaths = Object.keys(files).filter((path) => path !== PROJECT_FILE).sort(); + print({ ok: true, archiveVersion: payload.version || 1, ...inspectProject(payload.project), mediaInventory: { count: mediaPaths.length, paths: mediaPaths } }); + } return; } - const plan = JSON.parse(await readFile(resolve(argument), "utf8")); + const rawPlan = JSON.parse(await readFile(resolve(argument), "utf8")); + const prepared = await prepareAssetImports(rawPlan); + const plan = prepared.plan; const projectPath = resolve(plan.project); const { files, payload } = await readArchive(projectPath); const result = applyCommandPlan(payload.project, plan); @@ -38,10 +167,27 @@ async function main() { process.exitCode = 1; return; } + const isDiff = command === "project.diff"; const outputPath = plan.output?.project ? resolve(plan.output.project) : ""; - if (!plan.dryRun) { + if (!isDiff && !plan.dryRun) { if (!outputPath) throw new Error("output.project is required unless dryRun is true"); - const nextPayload = { ...payload, exportedAt: new Date().toISOString(), project: result.project }; + const appliedImports = result.appliedOperationIds.flatMap((id) => prepared.imports.has(id) ? [prepared.imports.get(id)] : []); + for (const imported of appliedImports) files[imported.manifest.path] = imported.bytes; + const previousVisualMedia = Array.isArray(payload.media?.visuals) ? payload.media.visuals : []; + const visualImports = appliedImports.filter((item) => item.track === "visuals"); + const voiceImport = appliedImports.find((item) => item.track === "audio"); + const musicImport = appliedImports.find((item) => item.track === "music"); + const nextPayload = { + ...payload, + exportedAt: new Date().toISOString(), + project: result.project, + media: { + ...(payload.media || {}), + visuals: [...previousVisualMedia, ...visualImports.map((item) => item.manifest)], + ...(voiceImport ? { audio: voiceImport.manifest } : {}), + ...(musicImport ? { music: musicImport.manifest } : {}), + }, + }; files[PROJECT_FILE] = strToU8(JSON.stringify(nextPayload)); await writeFile(outputPath, zipSync(files, { level: 6 })); } @@ -50,12 +196,12 @@ async function main() { revision: result.revision, appliedOperationIds: result.appliedOperationIds, warnings: result.warnings, - diff: { before: result.before, after: result.after }, - artifacts: { ...(plan.dryRun ? {} : { project: outputPath }) }, + diff: { before: result.before, after: result.after, changes: result.changes }, + artifacts: { ...(isDiff || plan.dryRun ? {} : { project: outputPath }) }, }); } main().catch((error) => { - print({ ok: false, code: "COMMAND_FAILED", message: error instanceof Error ? error.message : String(error) }); + print({ ok: false, code: error?.code || "COMMAND_FAILED", message: error instanceof Error ? error.message : String(error) }); process.exitCode = error?.exitCode || 1; }); diff --git a/skills/edit-timeline-studio/.claude-plugin/plugin.json b/skills/edit-timeline-studio/.claude-plugin/plugin.json index 386e0bc..39dbe20 100644 --- a/skills/edit-timeline-studio/.claude-plugin/plugin.json +++ b/skills/edit-timeline-studio/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "timeline-studio", "displayName": "Timeline Studio", - "version": "0.1.0", + "version": "0.2.0", "description": "Create, edit, caption, voice, assemble, validate, and export editable Timeline Studio video projects for automatic editing and local rendering.", "author": { "name": "MartinDelophy", diff --git a/skills/edit-timeline-studio/README.md b/skills/edit-timeline-studio/README.md index 7d37860..3783c17 100644 --- a/skills/edit-timeline-studio/README.md +++ b/skills/edit-timeline-studio/README.md @@ -8,10 +8,11 @@ Use it when a user asks an Agent to make a vertical short from images, synchroni - Inspect, dry-run, and transactionally modify a portable `.timeline` archive through a versioned JSON command plan. - Move voiceover clips; update caption text and timing; unlink or relink caption/audio pairs. -- Use the browser compatibility path for visual import and assembly, AI speech, automatic captions, overlays, effects, and export while more commands move into the shared registry. +- Import local visual or audio assets through the command runner with probing, SHA-256 integrity metadata, and portable archive embedding; use archived media for Visuals assembly and overlays. +- Use the browser compatibility path for AI speech, automatic captions, effects, unsupported editor operations, and final video export while more commands move into the shared registry. - Preserve the editable project as the source of truth and verify the reopened result. -The command runner currently edits existing projects; it does not yet perform headless media import, AI generation, or rendering. Those workflows remain available through the local or hosted browser editor. +The command runner reads and writes `.timeline` projects, supports deterministic local media import, and can render its documented portable Visuals + Voiceover + Music subset to verified MP4. AI generation and richer composition rendering remain available through the local or hosted browser editor. ## Install @@ -32,7 +33,7 @@ For repository development: git clone https://github.com/MartinDelophy/ai-video-editor.git cd ai-video-editor npm install -npm run agent -- inspect /absolute/path/project.timeline +npm run agent -- project.inspect /absolute/path/project.timeline npm run dev ``` @@ -41,6 +42,8 @@ npm run dev - [What Timeline Studio is and what it automates](docs/agent-video-editing.md) - [Use it from Codex](docs/codex-video-editing.md) - [Use it from Claude Code](docs/claude-code-video-editing.md) +- [Use it from GitHub Copilot](docs/github-copilot-video-editing.md) +- [Use it from Gemini CLI](docs/gemini-cli-video-editing.md) - [Five reproducible workflows](docs/examples.md) - [Command reference](docs/command-reference.md) - [Comparison with FFmpeg, CapCut, and Remotion](docs/comparison.md) diff --git a/skills/edit-timeline-studio/SKILL.md b/skills/edit-timeline-studio/SKILL.md index 57fb87e..35d1e85 100644 --- a/skills/edit-timeline-studio/SKILL.md +++ b/skills/edit-timeline-studio/SKILL.md @@ -12,7 +12,7 @@ Turn the user's exact editorial request and media into reversible Timeline Studi 1. Treat `https://video-editor.ai-creator.top/` as the canonical hosted editor. When the user asks to use the website, provides no repository, or expects Browser Use, proactively open this URL and inspect the live editor before planning the edit. 2. When this repository is available and the task concerns local development, unpublished changes, or evaluation, start the local editor and use it instead of the hosted release. Read the actual server URL from the process output; never assume port 5173. 3. Inspect `package.json` for an Agent command script. Do not use `npm run ... --if-present` as capability detection because it can succeed silently. -4. If the command runner exists, read [references/command-contract.md](references/command-contract.md), build and validate a versioned plan, dry-run it, then apply it. +4. If the command runner exists, read [references/command-contract.md](references/command-contract.md), inspect the project, build a versioned plan, run the structural validator, and use `project.diff` as the authoritative semantic dry run before `project.run`. 5. Otherwise, read [references/browser-workflow.md](references/browser-workflow.md) and use the editor UI. Use a concise edit checklist rather than inventing stable IDs, revisions, transactions, or a JSON plan that the UI cannot honor. 6. Do not claim deterministic or idempotent execution when only UI automation was available. State the limitation and preserve an editable project archive when the UI supports it. @@ -28,7 +28,7 @@ Turn the user's exact editorial request and media into reversible Timeline Studi ### 2. Plan at the supported fidelity -- With the command runner, express edits as declarative operations with stable IDs, seconds, revisions, operation IDs, and preconditions. Validate them with `scripts/validate_edit_plan.mjs `. +- With the command runner, express edits as declarative operations with stable IDs, seconds, revisions, operation IDs, and preconditions. Run `scripts/validate_edit_plan.mjs ` for transport-shape errors, then run `npm run agent -- project.diff ` to reject unsupported operations and invalid project-specific edits before applying anything. - With browser UI only, write a short ordered checklist of visible user intents and expected UI outcomes. Prefer named controls and clip labels; use coordinates only as a last-resort fallback grounded in a current screenshot. - Keep main Visuals contiguous. Treat captions, stickers, source audio, voiceover, music, and overlays as timed clips. - Preserve media identity and source-time mapping when moving or trimming clips. @@ -40,6 +40,7 @@ Turn the user's exact editorial request and media into reversible Timeline Studi - Never silently substitute missing media, voices, models, fonts, or effects. - Keep every result undoable and editable in the normal UI. - Do not start a paid or remote generation job without a clear user request. +- Do not put `output.render` in a command plan or claim that `project.run` renders video. Use the separate versioned `project.render` request for its documented portable subset, and use the browser editor for AI generation or unsupported composition features. ### 4. Verify the result @@ -63,6 +64,6 @@ For editor evaluation, regression work, or any run that exposes friction, read [ Read [references/current-capabilities.md](references/current-capabilities.md) when deciding whether a request can be executed now. Read [references/command-contract.md](references/command-contract.md) only when implementing or invoking the Agent command layer. Read [references/browser-workflow.md](references/browser-workflow.md) for UI execution and [references/e2e-evaluation.md](references/e2e-evaluation.md) for repeated experience-driven testing. -For public explanations, route one question to one page: use [docs/agent-video-editing.md](docs/agent-video-editing.md) for what Timeline Studio is, [docs/codex-video-editing.md](docs/codex-video-editing.md) or [docs/claude-code-video-editing.md](docs/claude-code-video-editing.md) for installation and invocation, [docs/examples.md](docs/examples.md) for reproducible cases, [docs/command-reference.md](docs/command-reference.md) for runner syntax, and [docs/comparison.md](docs/comparison.md) for FFmpeg, CapCut, and Remotion comparisons. Do not load all public pages unless the user asks for a broad overview. +For public explanations, route one question to one page: use [docs/agent-video-editing.md](docs/agent-video-editing.md) for what Timeline Studio is; the platform guide for [Codex](docs/codex-video-editing.md), [Claude Code](docs/claude-code-video-editing.md), [GitHub Copilot](docs/github-copilot-video-editing.md), or [Gemini CLI](docs/gemini-cli-video-editing.md) for discovery and invocation; [docs/examples.md](docs/examples.md) for reproducible cases; [docs/command-reference.md](docs/command-reference.md) for exact runner syntax; and [docs/comparison.md](docs/comparison.md) for FFmpeg, CapCut, and Remotion comparisons. Do not load all public pages unless the user asks for a broad overview. If a requested operation is unsupported, keep the valid partial timeline unchanged and state the exact missing command or runtime capability. diff --git a/skills/edit-timeline-studio/docs/agent-video-editing.md b/skills/edit-timeline-studio/docs/agent-video-editing.md index 8242d4f..1d05b7c 100644 --- a/skills/edit-timeline-studio/docs/agent-video-editing.md +++ b/skills/edit-timeline-studio/docs/agent-video-editing.md @@ -6,14 +6,15 @@ It can assemble a contiguous Visuals track, timed overlays, captions, stickers, There are two automation paths: -1. The versioned command runner directly inspects and modifies existing `.timeline` archives. It is transactional, revision-checked, idempotent by operation ID, and supports dry runs. Its first registry covers voiceover movement and caption update/link operations. -2. Browser execution covers the rest of the current editor, including imports, image assembly, AI generation, and final rendering. The Skill requires visible-state verification and a reopened editable project. +1. The versioned command runner directly inspects and modifies existing `.timeline` archives. It is transactional, revision-checked, idempotent by operation ID, and supports non-writing project diffs. Its registry covers project/track/clip inspection, timed clip edits, captions, deletion, and source-accurate Visuals trim/split/reorder operations. +2. Browser execution covers AI generation, effects, unsupported editor operations, preview, and rich final rendering. The Skill requires visible-state verification and a reopened editable project. Supported local image, video, and audio imports—and the documented portable headless render subset—can stay on the command path. An Agent should use the command runner whenever the requested operations are registered and use the browser only for the remaining steps. It must never describe browser automation as deterministic headless execution. ```bash -npm run agent -- inspect /projects/product-demo.timeline -npm run agent -- run /projects/update-captions.json +npm run agent -- project.inspect /projects/product-demo.timeline +npm run agent -- project.diff /projects/update-captions.json +npm run agent -- project.run /projects/update-captions.json ``` See [command-reference.md](command-reference.md) for the exact JSON and [examples.md](examples.md) for complete workflows. diff --git a/skills/edit-timeline-studio/docs/claude-code-video-editing.md b/skills/edit-timeline-studio/docs/claude-code-video-editing.md index bad2e60..9397f96 100644 --- a/skills/edit-timeline-studio/docs/claude-code-video-editing.md +++ b/skills/edit-timeline-studio/docs/claude-code-video-editing.md @@ -17,8 +17,9 @@ export ./out/demo.timeline and ./out/demo.mp4, then reopen the project and verif Claude Code should inspect `package.json`, detect the `agent` script, and use it for registered operations: ```bash -npm run agent -- inspect /absolute/path/demo.timeline -npm run agent -- run /absolute/path/edit-plan.json +npm run agent -- project.inspect /absolute/path/demo.timeline +npm run agent -- project.diff /absolute/path/edit-plan.json +npm run agent -- project.run /absolute/path/edit-plan.json ``` -It should use the browser workflow for unsupported commands and state that boundary in its result. The output contract is the same on both paths: return the editable project, the render when requested, a concise timeline summary, and verification evidence. +It can import supported local image, video, and audio files through `asset.import` and use `project.render` for the documented portable Visuals + Voiceover + Music subset. It should use the browser workflow for AI generation, richer composition rendering, and unsupported commands, and state that boundary in its result. The output contract is the same on both paths: return the editable project, the render when requested, a concise timeline summary, and verification evidence. diff --git a/skills/edit-timeline-studio/docs/codex-video-editing.md b/skills/edit-timeline-studio/docs/codex-video-editing.md index 543c4ae..ef4ad6b 100644 --- a/skills/edit-timeline-studio/docs/codex-video-editing.md +++ b/skills/edit-timeline-studio/docs/codex-video-editing.md @@ -17,8 +17,9 @@ save /projects/launch-v2.timeline, reopen it, and report the revision and track For a supported headless edit, Codex creates a plan and executes: ```bash -npm run agent -- run /projects/launch-v2-plan.json -npm run agent -- inspect /projects/launch-v2.timeline +npm run agent -- project.diff /projects/launch-v2-plan.json +npm run agent -- project.run /projects/launch-v2-plan.json +npm run agent -- project.inspect /projects/launch-v2.timeline ``` -For image import, TTS, automatic captions, effects, or rendering, Codex starts the local server, uses the editor in the in-app browser, exports a `.timeline`, reopens it, and validates visible clips and console errors. The browser remains a compatibility path until those operations enter the shared command registry. +For supported local image, video, or audio import, Codex keeps the work on the command path and verifies the embedded archive media. It may use `project.render` for the documented portable Visuals + Voiceover + Music subset and must verify the resulting MP4. For TTS, automatic captions, effects, visible captions/stickers/overlays, separated source audio, or other unsupported render features, Codex starts the local server, uses the editor in the in-app browser, exports a `.timeline`, reopens it, and validates visible clips and console errors. diff --git a/skills/edit-timeline-studio/docs/command-reference.md b/skills/edit-timeline-studio/docs/command-reference.md index 430dbda..b0e9148 100644 --- a/skills/edit-timeline-studio/docs/command-reference.md +++ b/skills/edit-timeline-studio/docs/command-reference.md @@ -3,18 +3,24 @@ ## Inspect a project ```bash -npm run agent -- inspect /absolute/path/project.timeline +npm run agent -- project.inspect /absolute/path/project.timeline +npm run agent -- track.inspect /absolute/path/project.timeline visuals +npm run agent -- clip.inspect /absolute/path/project.timeline visual-123 +npm run agent -- transcript.inspect /absolute/path/project.timeline voice-123 ``` -The JSON result includes `revision`, duration, ratio, track counts, applied operation IDs, and warnings. +The project result includes `revision`, duration, ratio, track counts, archived media inventory, applied operation IDs, and warnings. Track inspection returns ordered timing summaries; clip inspection returns source mapping, links, transforms, effects, and other serialized properties. Transcript inspection returns ordered caption segments, speaker labels, linked audio IDs, and word timing/confidence when available; omit the audio ID to inspect all captions. ## Apply a plan ```bash node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs /absolute/path/plan.json -npm run agent -- run /absolute/path/plan.json +npm run agent -- project.diff /absolute/path/plan.json +npm run agent -- project.run /absolute/path/plan.json ``` +The standalone validator checks JSON transport shape, finite time fields, and duplicate operation IDs. It does not prove that an operation is implemented or valid for the selected project. Treat `project.diff` as the authoritative semantic dry run because it executes the same registry, revision checks, and reducers as `project.run` without writing an archive. + ```json { "schemaVersion": 1, @@ -33,11 +39,47 @@ Supported write operations: | Type | Required fields | Optional fields | Effect | |---|---|---|---| +| `asset.import` | `file`, `track`, `clipId` | `name`, `duration`, `width`, `height`, `start`, `volume`, `muted`, `replace` | Probes, hashes, and embeds an absolute local visual/audio path. Supports Visuals, Music, and an empty Voiceover media slot. | | `timed.move` | `track: "audio"`, `clipId`, `start` | — | Moves a voiceover and any still-linked captions by the same delta. | +| `timed.resize` | `track`, `clipId`, `duration` | `start` | Resizes audio, sticker, or overlay clips; linked captions follow audio movement and clamp to its end. | +| `visual.trim` | `clipId`, `sourceIn`, `sourceOut` | — | Trims a video within its serialized source range and remaps keyframes. | +| `visual.split` | `clipId`, `at`, `rightClipId` | — | Splits at clip-local seconds using an explicit stable ID for the right clip. | +| `visual.reorder` | `clipId`, `toIndex` | — | Reorders the contiguous main Visuals sequence. | +| `visual.append` | `sourceClipId`, `clipId` | `duration` | Appends a new stable clip reusing media already stored in the archive. | +| `visual.insert` | `sourceClipId`, `clipId`, `atIndex` | `duration` | Inserts archived media at a main-sequence boundary. | +| `overlay.add` | `sourceClipId`, `clipId`, `start` | `duration`, `layer`, `muted`, `transform` | Adds a timed Overlay that reuses archived visual media. | +| `transition.set` | `clipId`, `transitionId` | `duration` | Sets or clears a validated transition on an outgoing Visuals junction. | +| `caption.add` | `clipId`, `text`, `start`, `end` | `audioClipId` | Adds a timed caption, optionally linked to voiceover audio. | | `caption.update` | `clipId` | `text`, `start`, `end` | Updates caption content or its finite, non-negative range. | | `caption.unlink_audio` | `clipId` | — | Preserves the remembered audio ID but stops synchronization. | | `caption.link_audio` | `clipId` | `audioClipId`, `align` | Relinks remembered or explicit audio; `align: true` copies its range. | +| `clip.delete` | `track`, `clipId` | — | Deletes a caption or voiceover; deleting audio leaves caption relink metadata intact. | +| `clip.set_property` | `clipId`, `property`, `value` | — | Sets an allowlisted numeric transform/audio/layer property with range validation. | +| `clip.set_speed` | `clipId`, `speed` | — | Sets 0.25–4× video/audio speed while preserving source duration and remapping keyframes. | +| `clip.set_muted` | `clipId`, `muted` | — | Mutes or unmutes video, overlay-video, or audio clips. | +| `track.set_visibility` | `track`, `visible` | — | Changes serialized track visibility using the editor's canonical base-track key. | +| `track.set_locked` | `track`, `locked` | — | Changes serialized track locking using the editor's canonical base-track key. | +| `project.set_ratio` | `ratio` | — | Sets `16:9`, `9:16`, `1:1`, or `4:5`. | -Set `dryRun: true` to return the predicted before/after summary without writing output. A successful non-empty batch increments revision once. Reusing an applied operation ID is a no-op; a stale revision with new operations returns `REVISION_CONFLICT`. Failures return a stable code and operation ID and write no partial archive. +Set `dryRun: true` to return the predicted before/after summary without writing output. `project.diff` also includes project-field changes plus per-track added, removed, modified, and reordered clip IDs; modified clips identify their exact changed fields and before/after values. A successful non-empty batch increments revision once. Reusing an applied operation ID is a no-op; a stale revision with new operations returns `REVISION_CONFLICT`. Failures return a stable code and operation ID and write no partial archive. -Headless import, generation, render, and broader edit commands are not shipped yet. Use the browser workflow for those operations rather than inventing command types. +`asset.import` supports JPG, PNG, WebP, GIF, MP4, WebM, MOV, MP3, WAV, M4A, AAC, OGG, and FLAC from an explicit absolute path. Missing duration and visual dimensions are probed with ffprobe; images still default to four seconds when no media duration exists. It computes SHA-256, embeds bytes under `media/visuals/` or `media/audio/`, adds the correct manifest entry, and writes integrity metadata. Music preserves its stable timed segment when reopened in the browser. Voiceover currently has one portable binary slot: importing into a populated Voiceover track fails unless `replace: true` is explicit. Diff and dry-run probe and validate but never change the input archive. + +## Render a portable project + +```bash +npm run agent -- project.render /absolute/path/render.json +``` + +```json +{ + "schemaVersion": 1, + "project": "/projects/edited.timeline", + "output": { "video": "/renders/edited.mp4" }, + "render": { "width": 1280, "height": 720, "frameRate": 30, "crf": 18, "preset": "medium" } +} +``` + +The initial renderer uses local `ffmpeg`, writes a temporary output, atomically renames it only after encoding succeeds, and verifies the final dimensions, duration, and audio-track presence with `ffprobe`. It renders contiguous image/video Visuals with contain fitting plus portable Voiceover and Music clips into H.264/AAC MP4. Width and height are normalized to even values; defaults come from the project ratio. + +The command fails with `UNSUPPORTED_RENDER_FEATURE` instead of silently dropping visible captions, stickers, overlays, transitions, visual effects, or separated source audio. Use the browser renderer for those projects. `project.render` does not run AI generation and is intentionally separate from `project.run`; the editable `.timeline` remains the source of truth. diff --git a/skills/edit-timeline-studio/docs/examples.md b/skills/edit-timeline-studio/docs/examples.md index 3a27599..17711d7 100644 --- a/skills/edit-timeline-studio/docs/examples.md +++ b/skills/edit-timeline-studio/docs/examples.md @@ -24,7 +24,7 @@ Each case preserves an editable `.timeline`. Paths are examples; replace them wi - User prompt: `Move voice-main to 3s, keep linked captions synchronized, replace caption-hero text, and save a new revision.` - Input file: `/demo/product/product-intro.timeline`. -- Execution command: `npm run agent -- run /demo/product/revise-plan.json`, followed by `npm run agent -- inspect /demo/out/product-intro-v2.timeline`. +- Execution commands: `npm run agent -- project.diff /demo/product/revise-plan.json`, then `npm run agent -- project.run /demo/product/revise-plan.json`, followed by `npm run agent -- project.inspect /demo/out/product-intro-v2.timeline`. - Plan operations: `timed.move` for `voice-main`; `caption.update` for `caption-hero`. - Timeline summary: revision 1, voiceover and linked caption begin at 3 seconds, original media entries unchanged. - Final output: no opaque render is required for this metadata-only revision. @@ -34,7 +34,7 @@ Each case preserves an editable `.timeline`. Paths are examples; replace them wi - User prompt: `Create English, French, and German editable versions, preserving visuals and replacing narration and captions.` - Input files: `/demo/localize/master.timeline`, `/demo/localize/{en,fr,de}.txt`. -- Execution: use the browser Skill once per language for voice generation; export `master-en.timeline`, `master-fr.timeline`, and `master-de.timeline`. Use `npm run agent -- inspect` on every archive and verify the expected caption/audio counts. +- Execution: use the browser Skill once per language for voice generation; export `master-en.timeline`, `master-fr.timeline`, and `master-de.timeline`. Use `npm run agent -- project.inspect` on every archive and verify the expected caption/audio counts. - Timeline summary: identical Visuals timing across three projects; localized linked caption and voiceover tracks. - Final outputs: `/demo/out/master-{en,fr,de}.mp4`. - Editable project downloads: `/demo/out/master-{en,fr,de}.timeline`. @@ -43,7 +43,7 @@ Each case preserves an editable `.timeline`. Paths are examples; replace them wi - User prompt: `Unlink caption-7 from voice-2, set it to 8.2–10.0 seconds, change its text, and save without touching the source archive.` - Input file: `/demo/revise/source.timeline`. -- Execution command: `npm run agent -- run /demo/revise/retime-plan.json`. +- Execution commands: `npm run agent -- project.diff /demo/revise/retime-plan.json`, then `npm run agent -- project.run /demo/revise/retime-plan.json`. - Plan operations: `caption.unlink_audio`, then `caption.update` with `start`, `end`, and `text`; output points to a different archive. - Timeline summary: one detached caption at 8.2–10.0 seconds; voiceover timing and archived media are unchanged; revision increments once. - Final output: optionally reopen in the browser and export `/demo/out/revised.mp4`. diff --git a/skills/edit-timeline-studio/docs/gemini-cli-video-editing.md b/skills/edit-timeline-studio/docs/gemini-cli-video-editing.md new file mode 100644 index 0000000..776f023 --- /dev/null +++ b/skills/edit-timeline-studio/docs/gemini-cli-video-editing.md @@ -0,0 +1,28 @@ +# How does Gemini CLI edit video with Timeline Studio? + +Gemini CLI discovers Agent Skills from `.gemini/skills` in a workspace or `~/.gemini/skills` for a user. During development, link the complete skill directory and confirm discovery: + +```bash +gemini skills link ./skills/edit-timeline-studio --scope user +``` + +Inside Gemini CLI, use `/skills list` or `/skills reload` after updating the linked skill. Gemini activates a relevant skill on demand and requests activation consent. + +Give Gemini explicit local inputs and editable outputs: + +```text +Use the edit-timeline-studio skill. Inspect /projects/launch.timeline, update the +linked caption at 3 seconds, save /projects/launch-v2.timeline, reopen it, and +report the revision, changed fields, and track summary. +``` + +For registered operations, Gemini should use: + +```bash +npm run agent -- project.inspect /projects/launch.timeline +node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs /projects/launch-plan.json +npm run agent -- project.diff /projects/launch-plan.json +npm run agent -- project.run /projects/launch-plan.json +``` + +Treat `project.diff` as the authoritative semantic validation step. The command runner can import supported local media, save editable `.timeline` archives, and use `project.render` for the documented portable Visuals + Voiceover + Music MP4 subset. It cannot run AI generation or render richer compositions yet. Use the local or hosted browser editor for those remaining operations and verify the final media and reopened project. diff --git a/skills/edit-timeline-studio/docs/github-copilot-video-editing.md b/skills/edit-timeline-studio/docs/github-copilot-video-editing.md new file mode 100644 index 0000000..ed46c28 --- /dev/null +++ b/skills/edit-timeline-studio/docs/github-copilot-video-editing.md @@ -0,0 +1,23 @@ +# How does GitHub Copilot edit video with Timeline Studio? + +GitHub Copilot discovers Agent Skills from a repository `.github/skills`, `.claude/skills`, or `.agents/skills` directory, and personal skills from `~/.copilot/skills` or `~/.agents/skills`. Copy or link this complete skill directory so its `references/`, `docs/`, and validator script remain available alongside `SKILL.md`. + +Prompt Copilot with explicit absolute media paths, editorial intent, aspect ratio, and editable output requirements: + +```text +Use the edit-timeline-studio skill. Inspect /projects/demo.timeline, import +/assets/card.png to Visuals, add it after the current sequence, save +/projects/demo-v2.timeline, and verify the resulting revision and media inventory. +``` + +For registered operations, Copilot should run the versioned command workflow: + +```bash +npm run agent -- project.inspect /projects/demo.timeline +node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs /projects/demo-plan.json +npm run agent -- project.diff /projects/demo-plan.json +npm run agent -- project.run /projects/demo-plan.json +npm run agent -- project.inspect /projects/demo-v2.timeline +``` + +The structural validator is not sufficient approval by itself; `project.diff` is the authoritative semantic dry run. Use `project.render` for the documented portable Visuals + Voiceover + Music MP4 subset. Use the browser editor for AI generation, unsupported operations, preview, WebM, and richer compositions containing captions, stickers, overlays, transitions, effects, or separated source audio. Return the editable `.timeline` archive together with any requested render and verification evidence. diff --git a/skills/edit-timeline-studio/references/command-contract.md b/skills/edit-timeline-studio/references/command-contract.md index 991619c..9a09792 100644 --- a/skills/edit-timeline-studio/references/command-contract.md +++ b/skills/edit-timeline-studio/references/command-contract.md @@ -20,14 +20,15 @@ Use JSON as a transport-neutral plan. The CLI and a future MCP server should cal } ], "output": { - "project": "/absolute/path/project-edited.timeline", - "render": "/absolute/path/project-edited.mp4" + "project": "/absolute/path/project-edited.timeline" } } ``` All times are finite seconds. IDs are stable across reads and writes. A repeated operation ID must return its prior result without applying twice. +The standalone `validate_edit_plan.mjs` checks transport shape only. `project.diff` is the authoritative project-aware validation step: it uses the same registry and reducers as `project.run`, rejects unsupported operations, checks revision and clip/track preconditions, and never writes an archive. + ## Minimum read commands - `project.inspect`: format version, revision, duration, ratio, tracks, media inventory, warnings @@ -36,16 +37,26 @@ All times are finite seconds. IDs are stable across reads and writes. A repeated - `transcript.inspect`: timestamped words/segments and speakers - `project.diff`: predicted state changes, duration changes, and validation warnings -## Minimum write operations +## Implemented write operations - `asset.import` - `visual.append`, `visual.insert`, `visual.trim`, `visual.split`, `visual.reorder` - `overlay.add`, `timed.move`, `timed.resize` - `clip.delete`, `clip.set_property`, `clip.set_speed`, `clip.set_muted` -- `caption.add`, `caption.update`, `caption.generate` -- `audio.separate`, `voice.generate`, `music.add` +- `caption.add`, `caption.update`, `caption.link_audio`, `caption.unlink_audio` - `transition.set`, `track.set_visibility`, `track.set_locked` -- `project.set_ratio`, `project.save`, `project.render` +- `project.set_ratio` + +Use [../docs/command-reference.md](../docs/command-reference.md) for required fields and current media-import limits. Do not invent operation types not listed there. + +## Planned operations + +- `caption.generate` +- `audio.separate`, `voice.generate`, `music.add` +- multi-file Voiceover media storage +- richer `project.render` coverage, structured progress events, and cancellation reporting + +`project.render` is implemented as a separate versioned request rather than a write operation inside `project.run`. Its first deterministic local path renders the portable Visuals, Voiceover, and Music subset to H.264/AAC MP4 and rejects unsupported composition features. Until richer rendering and generation enter the shared services, use the browser workflow for captions, stickers, overlays, effects, ASR/TTS, and other AI generation. ## Result envelope @@ -56,8 +67,7 @@ All times are finite seconds. IDs are stable across reads and writes. A repeated "appliedOperationIds": ["op-001"], "warnings": [], "artifacts": { - "project": "/absolute/path/project-edited.timeline", - "render": "/absolute/path/project-edited.mp4" + "project": "/absolute/path/project-edited.timeline" } } ``` diff --git a/skills/edit-timeline-studio/references/current-capabilities.md b/skills/edit-timeline-studio/references/current-capabilities.md index 3030fca..2c5ba9c 100644 --- a/skills/edit-timeline-studio/references/current-capabilities.md +++ b/skills/edit-timeline-studio/references/current-capabilities.md @@ -16,9 +16,10 @@ - Browser-driven operation of the running editor - Import and export through visible file controls - Pure timeline helper functions in `src/lib/` -- A versioned JSON command plan runner through `npm run agent -- inspect ` and `npm run agent -- run ` -- Transactional, revision-checked, idempotent, dry-run-safe edits for moving voiceover clips, updating caption text/timing, and unlinking or relinking caption audio +- Versioned `project.inspect`, `track.inspect`, `clip.inspect`, `transcript.inspect`, field-level `project.diff`, and `project.run` commands, with legacy `inspect`/`run` aliases +- Transactional, revision-checked, idempotent edits for probed and hashed visual/audio import to Visuals, Music, or the portable Voiceover slot; timed edits, captions, Visuals/Overlays, transitions, validated properties, track state, and ratio - Portable `.timeline` output that preserves archived media entries while replacing only versioned project metadata +- Transactional local `project.render` for the portable Visuals + Voiceover + Music subset, with ffprobe verification and explicit rejection of unsupported composition features Browser-driven editing is a compatibility mechanism, not a stable public API. UI labels, selection state, drag thresholds, and file pickers make it unsuitable for unattended or idempotent jobs. @@ -38,20 +39,20 @@ Observed browser-path constraints: ## Missing for reliable Agent editing -1. Media probing and render support in the headless command runner (load, inspect, edit, and save are now available). -2. Broader command coverage beyond the initial voiceover/caption operations. +1. Full browser-renderer parity in the headless command runner; the first H.264/AAC Visuals + Voiceover + Music path and ffprobe-backed import probing are available. +2. Broader command coverage for multi-asset Voiceover storage, generation, and render. 3. A fully serializable editor core independent of React setters, DOM nodes, Blob URLs, and browser-only refs; the first shared reducers now live in `src/lib/projectCommandEngine.js`. -4. Persisted undo checkpoints and richer semantic diffs; transactions, revision preconditions, idempotency keys, structured errors, and summary dry-run diffs are available. -5. Read tools at three levels: project summary, track/clip detail, and transcript/analysis detail. +4. Persisted undo checkpoints; transactions, revision preconditions, idempotency keys, structured errors, and field-level dry-run diffs are available. +5. Richer non-caption analysis inspection; project, track, clip, and caption transcript reads are available. 6. Progress events and cancellation for ASR, TTS, vision, avatar generation, and export. -7. Deterministic media ingestion with content hashes and portable asset references. -8. Agent-focused integration tests that apply a command plan, reopen the project, render, decode, and verify the result. +7. Content-addressed deduplication and per-segment portable media references beyond the current hashed import paths and single Voiceover binary slot. +8. Agent-focused integration tests that apply a command plan, reopen the project, exercise both the supported headless render subset and browser-render parity cases, decode, and verify the result. ## Recommended delivery order -1. Extend the shared reducers with import, append, trim, split, delete, overlay, property, and track mute/visibility. -2. Add track/clip/transcript inspection and richer `project.diff` output. +1. Evolve the archive from one Voiceover binary slot to per-segment portable audio references. +2. Add vision/ASR analysis-record inspection beyond serialized caption transcript data. 3. Add persisted undo checkpoints around the existing command transaction. -4. Add media probing and `project.render` to the local Node/browser-worker runner. +4. Expand `project.render` with captions, stickers, overlays, transitions, effects, source audio, progress events, and cancellation diagnostics. 5. Add MCP as a thin transport adapter over the same registry. 6. Prefer the CLI from this Skill when an operation is supported, retaining browser control as the compatibility path. diff --git a/src/hooks/useProjectFiles.js b/src/hooks/useProjectFiles.js index be0f62e..26d5f75 100644 --- a/src/hooks/useProjectFiles.js +++ b/src/hooks/useProjectFiles.js @@ -1,7 +1,7 @@ import { useCallback, useRef } from "react"; import { DEFAULT_SCRIPT, DEFAULT_TIMELINE_DURATION_SECONDS, RATIO_OPTIONS, VOICES } from "../config/editor.js"; import { decodeWaveform, downloadBlob } from "../lib/media.js"; -import { createProjectArchive, readProjectArchive, readProjectFileAsText } from "../lib/projectArchive.js"; +import { createProjectArchive, readProjectArchive, readProjectFileAsText, resolveProjectVisualMedia } from "../lib/projectArchive.js"; import { createCaptionSegments, getImageThumbnailCount, getVisualSegmentsTotal } from "../lib/timeline.js"; export function useProjectFiles(deps) { @@ -14,7 +14,7 @@ export function useProjectFiles(deps) { script: deps.script, commandState: commandStateRef.current, selectedVoiceId: deps.selectedVoiceId, speed: deps.speed, volume: deps.volume, ratioId: deps.ratioId, fitMode: deps.fitMode, captionPosition: deps.captionPosition, captionPlacement: deps.captionPlacement, captionSize: deps.captionSize, captionStyle: deps.captionStyle, - captionsEnabled: deps.captionsEnabled, captionSegments: deps.captionSegments, audioSegments, visualSegments, visualOverlaySegments, + captionsEnabled: deps.captionsEnabled, captionSegments: deps.captionSegments, audioSegments, musicSegments: deps.musicSegments, visualSegments, visualOverlaySegments, stickerSegments: deps.stickerSegments, selectedFilterId: deps.selectedFilterId, selectedTransitionId: deps.selectedTransitionId, selectedStickerId: deps.selectedStickerId, trackVisibility: deps.trackVisibility, trackLocks: deps.trackLocks, timelineZoom: deps.timelineZoom, audioDuration: deps.audioDuration, @@ -80,13 +80,13 @@ export function useProjectFiles(deps) { deps.setSelectedFilterId(data.selectedFilterId || "none"); deps.setSelectedTransitionId(data.selectedTransitionId || "none"); deps.setSelectedStickerId(data.selectedStickerId || "none"); deps.setStickerSegments(Array.isArray(data.stickerSegments) ? data.stickerSegments : []); const visuals = Array.isArray(data.visualSegments) ? data.visualSegments.map((segment) => { - const media = visualMedia.get(segment.id); + const media = resolveProjectVisualMedia(visualMedia, segment); return media?.blob ? { ...segment, src: URL.createObjectURL(media.blob), blob: media.blob } : segment?.src ? segment : null; }).filter(Boolean) : []; visuals.filter((segment) => segment.src?.startsWith("blob:")).forEach((segment) => deps.imageUrlRefs.current.add(segment.src)); deps.setVisualSegments(visuals); deps.setImageDuration(getVisualSegmentsTotal(visuals)); const overlays = Array.isArray(data.visualOverlaySegments) ? data.visualOverlaySegments.map((segment) => { - const media = visualMedia.get(segment.assetId) || visualMedia.get(segment.id); + const media = resolveProjectVisualMedia(visualMedia, segment); return media?.blob ? { ...segment, src: URL.createObjectURL(media.blob), blob: media.blob } : segment?.src ? segment : null; }).filter(Boolean) : []; deps.setVisualOverlaySegments(overlays); deps.setSelectedVisualOverlayId(""); @@ -99,7 +99,12 @@ export function useProjectFiles(deps) { } else deps.replaceAudio(audio, Number(data.audioDuration) || decoded.duration, decoded.peaks, "已恢复工程配音"); } else deps.clearAudioTrack(""); if (sourceAudio) { const decoded = await decodeWaveform(sourceAudio); deps.replaceSourceAudio(sourceAudio, Number(data.sourceAudioDuration) || decoded.duration, decoded.peaks, data.sourceAudioName || "source-audio", "", Number(data.sourceAudioStart) || 0, data.sourceAudioAssetId || "", { focusAudio: false }); } else deps.clearSourceAudioTrack(""); - if (music) { const decoded = await decodeWaveform(music); deps.replaceMusic(music, Number(data.musicDuration) || decoded.duration, decoded.peaks, data.musicName || "background-music", ""); deps.setMusicStart(Math.max(0, Number(data.musicStart) || 0)); } else deps.clearMusicTrack(""); + if (music) { + const decoded = await decodeWaveform(music); + deps.replaceMusic(music, Number(data.musicDuration) || decoded.duration, decoded.peaks, data.musicName || "background-music", ""); + deps.setMusicStart(Math.max(0, Number(data.musicStart) || 0)); + if (Array.isArray(data.musicSegments) && data.musicSegments.length) deps.setMusicSegments(data.musicSegments.map((segment) => ({ ...segment, peaks: decoded.peaks }))); + } else deps.clearMusicTrack(""); deps.setMusicVolume(Number(data.musicVolume) || 0.35); deps.setSourceAudioVolume(Number(data.sourceAudioVolume) || 1); deps.setSourceAudioAssetId(data.sourceAudioAssetId || ""); deps.setSourceAudioLinked(data.sourceAudioLinked !== false); deps.setCurrentTime(0); deps.clearAllVisionState(); deps.setShowFileMenu(false); diff --git a/src/lib/projectArchive.js b/src/lib/projectArchive.js index df0c9d6..1ea4e41 100644 --- a/src/lib/projectArchive.js +++ b/src/lib/projectArchive.js @@ -4,6 +4,10 @@ export const PROJECT_ARCHIVE_FORMAT = "timeline-studio-archive"; export const PROJECT_ARCHIVE_VERSION = 2; const PROJECT_FILE = "project.json"; +export function resolveProjectVisualMedia(visualMedia, segment) { + return visualMedia.get(segment?.id) || visualMedia.get(segment?.archiveMediaId) || visualMedia.get(segment?.assetId) || null; +} + function readWithFileReader(file, mode) { return new Promise((resolve, reject) => { const reader = new FileReader(); diff --git a/src/lib/projectArchive.test.js b/src/lib/projectArchive.test.js new file mode 100644 index 0000000..952746f --- /dev/null +++ b/src/lib/projectArchive.test.js @@ -0,0 +1,17 @@ +import { describe, expect, it } from "vitest"; +import { resolveProjectVisualMedia } from "./projectArchive.js"; + +describe("project archive visual media resolution", () => { + it("restores a headless-cloned clip through its stable archive media reference", () => { + const original = { id: "visual-original", blob: new Blob(["media"]) }; + const media = new Map([[original.id, original]]); + expect(resolveProjectVisualMedia(media, { id: "visual-clone", archiveMediaId: "visual-original", assetId: "library-asset" })).toBe(original); + }); + + it("prefers a clip-specific media entry before archive and library fallbacks", () => { + const exact = { id: "visual-clone" }; + const archived = { id: "visual-original" }; + const media = new Map([[exact.id, exact], [archived.id, archived]]); + expect(resolveProjectVisualMedia(media, { id: exact.id, archiveMediaId: archived.id })).toBe(exact); + }); +}); diff --git a/src/lib/projectCommandEngine.js b/src/lib/projectCommandEngine.js index 19daf98..c56d108 100644 --- a/src/lib/projectCommandEngine.js +++ b/src/lib/projectCommandEngine.js @@ -13,6 +13,12 @@ function finiteNonNegative(value, name) { return value; } +function finitePositive(value, name) { + const result = finiteNonNegative(value, name); + if (result <= 0) throw Object.assign(new Error(`${name} must be greater than zero`), { code: "INVALID_ARGUMENT" }); + return result; +} + function findById(items, id, kind) { const item = (Array.isArray(items) ? items : []).find((entry) => entry.id === id); if (!item) throw Object.assign(new Error(`${kind} not found: ${id}`), { code: "CLIP_NOT_FOUND" }); @@ -42,6 +48,244 @@ function moveTimed(project, operation) { : caption); } +function resizeTimed(project, operation) { + const collections = { audio: "audioSegments", sticker: "stickerSegments", overlay: "visualOverlaySegments" }; + const key = collections[operation.track]; + if (!key) throw Object.assign(new Error(`Unsupported timed track: ${operation.track}`), { code: "UNSUPPORTED_TRACK" }); + const segment = findById(project[key], operation.clipId, "Timed clip"); + const previousStart = Number(segment.start) || 0; + const start = Object.hasOwn(operation, "start") ? finiteNonNegative(operation.start, "start") : previousStart; + const duration = finitePositive(operation.duration, "duration"); + segment.start = start; + segment.duration = duration; + if (operation.track === "audio") { + const delta = start - previousStart; + const clipEnd = start + duration; + project.captionSegments = (project.captionSegments || []).map((caption) => { + if (caption.audioSegmentId !== segment.id) return caption; + const captionStart = finiteNonNegative((Number(caption.start) || 0) + delta, "caption start"); + const captionEnd = Math.min(clipEnd, finiteNonNegative((Number(caption.end) || 0) + delta, "caption end")); + return { ...caption, start: Math.min(captionStart, captionEnd), end: captionEnd }; + }); + } +} + +function visualPlaybackRate(segment) { + const value = Number(segment?.playbackRate) || 1; + return Math.max(0.25, Math.min(4, value)); +} + +function remapKeyframes(keyframes, start, end) { + return (Array.isArray(keyframes) ? keyframes : []) + .filter((frame) => Number(frame?.time) >= start && Number(frame?.time) <= end) + .map((frame) => ({ ...frame, time: Number(frame.time) - start })); +} + +function trimVisual(project, operation) { + const segment = findById(project.visualSegments, operation.clipId, "Visual clip"); + if (segment.type !== "video") throw Object.assign(new Error("visual.trim currently supports video clips only"), { code: "UNSUPPORTED_MEDIA_TYPE" }); + const sourceIn = finiteNonNegative(operation.sourceIn, "sourceIn"); + const sourceOut = finiteNonNegative(operation.sourceOut, "sourceOut"); + if (sourceOut <= sourceIn) throw Object.assign(new Error("sourceOut must be after sourceIn"), { code: "INVALID_RANGE" }); + const previousSourceIn = Math.max(0, Number(segment.sourceStart) || 0); + const previousSourceDuration = Math.max(0, Number(segment.sourceDuration) || (Number(segment.duration) || 0) * visualPlaybackRate(segment)); + const previousSourceOut = previousSourceIn + previousSourceDuration; + if (sourceIn < previousSourceIn || sourceOut > previousSourceOut) { + throw Object.assign(new Error(`Trim range must stay within ${previousSourceIn}-${previousSourceOut}`), { code: "SOURCE_RANGE_EXCEEDED" }); + } + const rate = visualPlaybackRate(segment); + const removedLocalTime = (sourceIn - previousSourceIn) / rate; + const duration = (sourceOut - sourceIn) / rate; + segment.sourceStart = sourceIn; + segment.sourceDuration = sourceOut - sourceIn; + segment.duration = duration; + if (Array.isArray(segment.keyframes)) segment.keyframes = remapKeyframes(segment.keyframes, removedLocalTime, removedLocalTime + duration); +} + +function splitVisual(project, operation) { + const visuals = Array.isArray(project.visualSegments) ? project.visualSegments : []; + const index = visuals.findIndex((segment) => segment.id === operation.clipId); + if (index < 0) throw Object.assign(new Error(`Visual clip not found: ${operation.clipId}`), { code: "CLIP_NOT_FOUND" }); + const segment = visuals[index]; + const at = finitePositive(operation.at, "at"); + const duration = finitePositive(Number(segment.duration), "clip duration"); + if (at >= duration) throw Object.assign(new Error("at must be inside the visual clip"), { code: "INVALID_RANGE" }); + const rightClipId = typeof operation.rightClipId === "string" ? operation.rightClipId.trim() : ""; + if (!rightClipId) throw Object.assign(new Error("rightClipId is required"), { code: "INVALID_ARGUMENT" }); + if (visuals.some((item) => item.id === rightClipId)) throw Object.assign(new Error(`Visual clip already exists: ${rightClipId}`), { code: "CLIP_ALREADY_EXISTS" }); + const rate = segment.type === "video" ? visualPlaybackRate(segment) : 1; + const left = { ...segment, duration: at }; + const right = { ...segment, id: rightClipId, duration: duration - at }; + if (segment.type === "video") { + left.sourceDuration = at * rate; + right.sourceStart = Math.max(0, Number(segment.sourceStart) || 0) + at * rate; + right.sourceDuration = (duration - at) * rate; + } + if (Array.isArray(segment.keyframes)) { + left.keyframes = remapKeyframes(segment.keyframes, 0, at); + right.keyframes = remapKeyframes(segment.keyframes, at, duration); + } + project.visualSegments = [...visuals.slice(0, index), left, right, ...visuals.slice(index + 1)]; +} + +function reorderVisual(project, operation) { + const visuals = Array.isArray(project.visualSegments) ? project.visualSegments : []; + const from = visuals.findIndex((segment) => segment.id === operation.clipId); + if (from < 0) throw Object.assign(new Error(`Visual clip not found: ${operation.clipId}`), { code: "CLIP_NOT_FOUND" }); + if (!Number.isInteger(operation.toIndex) || operation.toIndex < 0 || operation.toIndex >= visuals.length) { + throw Object.assign(new Error("toIndex must identify an existing visual position"), { code: "INVALID_ARGUMENT" }); + } + const next = [...visuals]; + const [moved] = next.splice(from, 1); + next.splice(operation.toIndex, 0, moved); + project.visualSegments = next; +} + +function requireNewClipId(project, clipId) { + const id = typeof clipId === "string" ? clipId.trim() : ""; + if (!id) throw Object.assign(new Error("clipId is required"), { code: "INVALID_ARGUMENT" }); + const exists = Object.values(TRACK_COLLECTIONS).some((key) => (project?.[key] || []).some((clip) => clip.id === id)); + if (exists) throw Object.assign(new Error(`Clip already exists: ${id}`), { code: "CLIP_ALREADY_EXISTS" }); + return id; +} + +function findVisualSource(project, sourceClipId) { + const source = [...(project.visualSegments || []), ...(project.visualOverlaySegments || [])].find((clip) => clip.id === sourceClipId); + if (!source) throw Object.assign(new Error(`Visual source clip not found: ${sourceClipId}`), { code: "CLIP_NOT_FOUND" }); + return source; +} + +function cloneVisualForSequence(project, operation) { + const id = requireNewClipId(project, operation.clipId); + const source = findVisualSource(project, operation.sourceClipId); + const maximumDuration = finitePositive(Number(source.duration), "source clip duration"); + const duration = Object.hasOwn(operation, "duration") ? finitePositive(operation.duration, "duration") : maximumDuration; + if (source.type === "video" && duration > maximumDuration) { + throw Object.assign(new Error(`Video duration cannot exceed source clip duration ${maximumDuration}`), { code: "SOURCE_RANGE_EXCEEDED" }); + } + const next = { ...source, id, archiveMediaId: source.archiveMediaId || source.id, duration }; + delete next.start; + delete next.layer; + delete next.baseTransform; + delete next.transition; + if (source.type === "video") next.sourceDuration = duration * visualPlaybackRate(source); + if (Array.isArray(source.keyframes)) next.keyframes = remapKeyframes(source.keyframes, 0, duration); + return next; +} + +function appendVisual(project, operation) { + project.visualSegments = [...(project.visualSegments || []), cloneVisualForSequence(project, operation)]; +} + +function importAsset(project, operation) { + const id = requireNewClipId(project, operation.clipId); + if (!["visuals", "audio", "music"].includes(operation.track)) throw Object.assign(new Error("asset.import supports Visuals, Voiceover, or Music"), { code: "UNSUPPORTED_TRACK" }); + if (!operation.prepared || !["image", "video", "audio"].includes(operation.mediaType) || !operation.sha256 || !operation.archivePath) { + throw Object.assign(new Error("asset.import must be prepared by an archive media service"), { code: "ASSET_NOT_PREPARED" }); + } + const duration = finitePositive(operation.duration, "duration"); + const integrity = { sha256: operation.sha256, size: operation.size, mimeType: operation.mimeType, archivePath: operation.archivePath }; + if (operation.track === "audio" || operation.track === "music") { + if (operation.mediaType !== "audio") throw Object.assign(new Error(`${operation.track} import requires audio media`), { code: "UNSUPPORTED_MEDIA_TYPE" }); + const start = Object.hasOwn(operation, "start") ? finiteNonNegative(operation.start, "start") : 0; + const segment = { id, name: operation.name || id, start, duration, sourceStart: 0, sourceDuration: duration, playbackRate: 1, volume: operation.volume ?? (operation.track === "music" ? 0.35 : 1), fadeIn: 0, fadeOut: 0, muted: operation.muted === true, integrity }; + if (operation.track === "audio") { + if ((project.audioSegments || []).length && operation.replace !== true) throw Object.assign(new Error("Voiceover media already exists; set replace: true to replace it"), { code: "MEDIA_SLOT_OCCUPIED" }); + project.audioSegments = [segment]; + project.audioDuration = duration; + } else { + project.musicSegments = [segment]; + project.musicName = segment.name; + project.musicDuration = duration; + project.musicStart = start; + project.musicVolume = segment.volume; + } + return; + } + if (!["image", "video"].includes(operation.mediaType)) throw Object.assign(new Error("Visuals import requires image or video media"), { code: "UNSUPPORTED_MEDIA_TYPE" }); + const segment = { + id, + assetId: operation.assetId || `asset-${operation.sha256.slice(0, 16)}`, + archiveMediaId: id, + name: operation.name || id, + type: operation.mediaType, + duration, + width: Math.max(0, Number(operation.width) || 0), + height: Math.max(0, Number(operation.height) || 0), + sourceStart: 0, + sourceDuration: operation.mediaType === "video" ? duration : 0, + playbackRate: 1, + muted: operation.muted === true, + integrity, + }; + project.visualSegments = [...(project.visualSegments || []), segment]; +} + +function insertVisual(project, operation) { + const visuals = Array.isArray(project.visualSegments) ? project.visualSegments : []; + if (!Number.isInteger(operation.atIndex) || operation.atIndex < 0 || operation.atIndex > visuals.length) { + throw Object.assign(new Error("atIndex must be a valid visual insertion position"), { code: "INVALID_ARGUMENT" }); + } + const next = cloneVisualForSequence(project, operation); + project.visualSegments = [...visuals.slice(0, operation.atIndex), next, ...visuals.slice(operation.atIndex)]; +} + +function addOverlay(project, operation) { + const id = requireNewClipId(project, operation.clipId); + const source = findVisualSource(project, operation.sourceClipId); + const start = finiteNonNegative(operation.start, "start"); + const sourceDuration = finitePositive(Number(source.duration), "source clip duration"); + const duration = Object.hasOwn(operation, "duration") ? finitePositive(operation.duration, "duration") : Math.min(5, sourceDuration); + if (source.type === "video" && duration > sourceDuration) { + throw Object.assign(new Error(`Overlay duration cannot exceed source clip duration ${sourceDuration}`), { code: "SOURCE_RANGE_EXCEEDED" }); + } + const layer = Object.hasOwn(operation, "layer") ? finitePositive(operation.layer, "layer") + : (project.visualOverlaySegments || []).reduce((maximum, clip) => Math.max(maximum, Number(clip.layer) || 1), 0) + 1; + const transform = operation.transform || {}; + for (const key of ["x", "y", "scale", "rotation", "opacity"]) { + if (Object.hasOwn(transform, key) && (typeof transform[key] !== "number" || !Number.isFinite(transform[key]))) { + throw Object.assign(new Error(`transform.${key} must be finite`), { code: "INVALID_ARGUMENT" }); + } + } + const rate = visualPlaybackRate(source); + const overlay = { + id, + assetId: source.assetId || "", + archiveMediaId: source.archiveMediaId || source.id, + name: source.name || "Overlay", + type: source.type === "video" ? "video" : "image", + width: Number(source.width) || 0, + height: Number(source.height) || 0, + sourceStart: Math.max(0, Number(source.sourceStart) || 0), + sourceDuration: source.type === "video" ? duration * rate : Math.max(0, Number(source.sourceDuration) || 0), + playbackRate: rate, + start, + duration, + muted: operation.muted === true, + layer, + baseTransform: { x: 27, y: -24, scale: 0.34, rotation: 0, opacity: 1, ...transform }, + keyframes: [], + }; + project.visualOverlaySegments = [...(project.visualOverlaySegments || []), overlay]; +} + +function setTransition(project, operation) { + const transitions = new Set(["none", "fade", "zoom", "flash", "wipe-left", "wipe-up", "blur", "split", "glitch"]); + if (!transitions.has(operation.transitionId)) throw Object.assign(new Error(`Unknown transition: ${operation.transitionId}`), { code: "INVALID_TRANSITION" }); + const visuals = Array.isArray(project.visualSegments) ? project.visualSegments : []; + const index = visuals.findIndex((clip) => clip.id === operation.clipId); + if (index < 0) throw Object.assign(new Error(`Visual clip not found: ${operation.clipId}`), { code: "CLIP_NOT_FOUND" }); + if (operation.transitionId !== "none" && index >= visuals.length - 1) { + throw Object.assign(new Error("A transition requires a following visual clip"), { code: "INVALID_TRANSITION_TARGET" }); + } + const maximum = index < visuals.length - 1 + ? Math.max(0.1, Math.min(2, Number(visuals[index].duration) / 2, Number(visuals[index + 1].duration) / 2)) + : 0.5; + const duration = Object.hasOwn(operation, "duration") ? finitePositive(operation.duration, "duration") : 0.5; + if (duration > maximum) throw Object.assign(new Error(`Transition duration cannot exceed ${maximum}`), { code: "INVALID_RANGE" }); + visuals[index].transition = { id: operation.transitionId, duration: Math.min(duration, maximum) }; +} + function updateCaption(project, operation) { const caption = findById(project.captionSegments, operation.clipId, "Caption clip"); if (Object.hasOwn(operation, "text")) { @@ -56,6 +300,44 @@ function updateCaption(project, operation) { project.script = (project.captionSegments || []).map((item) => item.text).join("\n"); } +function addCaption(project, operation) { + const id = typeof operation.clipId === "string" ? operation.clipId.trim() : ""; + if (!id) throw Object.assign(new Error("clipId is required"), { code: "INVALID_ARGUMENT" }); + const captions = Array.isArray(project.captionSegments) ? project.captionSegments : []; + if (captions.some((caption) => caption.id === id)) { + throw Object.assign(new Error(`Caption clip already exists: ${id}`), { code: "CLIP_ALREADY_EXISTS" }); + } + if (typeof operation.text !== "string") throw Object.assign(new Error("text must be a string"), { code: "INVALID_ARGUMENT" }); + const start = finiteNonNegative(operation.start, "start"); + const end = finiteNonNegative(operation.end, "end"); + if (end < start) throw Object.assign(new Error("caption end must not be before start"), { code: "INVALID_RANGE" }); + let audioSegmentId = ""; + if (operation.audioClipId) audioSegmentId = findById(project.audioSegments, operation.audioClipId, "Audio clip").id; + project.captionSegments = [...captions, { + id, + text: operation.text, + start, + end, + ...(audioSegmentId ? { audioSegmentId } : {}), + }].sort((left, right) => (Number(left.start) || 0) - (Number(right.start) || 0)); + project.script = project.captionSegments.map((item) => item.text).join("\n"); +} + +function deleteClip(project, operation) { + const collections = { caption: "captionSegments", audio: "audioSegments" }; + const key = collections[operation.track]; + if (!key) throw Object.assign(new Error(`Unsupported clip track: ${operation.track}`), { code: "UNSUPPORTED_TRACK" }); + findById(project[key], operation.clipId, `${operation.track === "caption" ? "Caption" : "Audio"} clip`); + project[key] = project[key].filter((item) => item.id !== operation.clipId); + if (operation.track === "caption") { + project.script = project.captionSegments.map((item) => item.text).join("\n"); + } else { + project.captionSegments = (project.captionSegments || []).map((caption) => caption.audioSegmentId === operation.clipId + ? { ...caption, audioSegmentId: "", detachedAudioSegmentId: operation.clipId } + : caption); + } +} + function unlinkCaption(project, operation) { const caption = findById(project.captionSegments, operation.clipId, "Caption clip"); if (caption.audioSegmentId) caption.detachedAudioSegmentId = caption.audioSegmentId; @@ -75,11 +357,124 @@ function linkCaption(project, operation) { } } +function findClipMatch(project, clipId) { + const matches = Object.entries(TRACK_COLLECTIONS).flatMap(([track, key]) => { + const clips = Array.isArray(project?.[key]) ? project[key] : []; + return clips.flatMap((clip, index) => clip.id === clipId ? [{ track, key, clip, index }] : []); + }); + if (!matches.length) throw Object.assign(new Error(`Clip not found: ${clipId}`), { code: "CLIP_NOT_FOUND" }); + if (matches.length > 1) throw Object.assign(new Error(`Clip ID is not globally unique: ${clipId}`), { code: "CLIP_ID_AMBIGUOUS" }); + return matches[0]; +} + +const NUMERIC_CLIP_PROPERTIES = Object.freeze({ + x: { min: -1000, max: 1000 }, + y: { min: -1000, max: 1000 }, + scale: { min: 0.1, max: 20 }, + rotation: { min: -36000, max: 36000 }, + opacity: { min: 0, max: 1 }, + volume: { min: 0, max: 2 }, + fadeIn: { min: 0, max: 1800 }, + fadeOut: { min: 0, max: 1800 }, + layer: { min: 0, max: 1000 }, +}); + +function setClipProperty(project, operation) { + const { clip } = findClipMatch(project, operation.clipId); + const limits = NUMERIC_CLIP_PROPERTIES[operation.property]; + if (!limits) throw Object.assign(new Error(`Unsupported clip property: ${operation.property}`), { code: "UNSUPPORTED_PROPERTY" }); + if (typeof operation.value !== "number" || !Number.isFinite(operation.value) || operation.value < limits.min || operation.value > limits.max) { + throw Object.assign(new Error(`${operation.property} must be between ${limits.min} and ${limits.max}`), { code: "INVALID_ARGUMENT" }); + } + clip[operation.property] = operation.value; +} + +function setClipSpeed(project, operation) { + const match = findClipMatch(project, operation.clipId); + if (match.track === "captions" || match.track === "stickers" || (match.track === "visuals" && match.clip.type !== "video") || (match.track === "overlays" && match.clip.type !== "video")) { + throw Object.assign(new Error(`Speed is unsupported for ${match.track} clip ${operation.clipId}`), { code: "UNSUPPORTED_MEDIA_TYPE" }); + } + const speed = finitePositive(operation.speed, "speed"); + if (speed < 0.25 || speed > 4) throw Object.assign(new Error("speed must be between 0.25 and 4"), { code: "INVALID_ARGUMENT" }); + const clip = match.clip; + const previousDuration = finitePositive(Number(clip.duration), "clip duration"); + const previousRate = visualPlaybackRate(clip); + const sourceDuration = Math.max(0.001, Number(clip.sourceDuration) || previousDuration * previousRate); + const duration = sourceDuration / speed; + clip.playbackRate = speed; + clip.sourceDuration = sourceDuration; + clip.duration = duration; + if (Array.isArray(clip.keyframes)) { + const timeScale = duration / previousDuration; + clip.keyframes = clip.keyframes.map((frame) => ({ ...frame, time: Math.max(0, Number(frame.time) || 0) * timeScale })); + } + if (match.track === "audio") { + const clipEnd = (Number(clip.start) || 0) + duration; + project.captionSegments = (project.captionSegments || []).map((caption) => caption.audioSegmentId === clip.id + ? { ...caption, end: Math.max(Number(caption.start) || 0, Math.min(Number(caption.end) || 0, clipEnd)) } + : caption); + } +} + +function setClipMuted(project, operation) { + if (typeof operation.muted !== "boolean") throw Object.assign(new Error("muted must be a boolean"), { code: "INVALID_ARGUMENT" }); + const { track, clip } = findClipMatch(project, operation.clipId); + if (!["visuals", "audio", "overlays"].includes(track)) { + throw Object.assign(new Error(`Mute is unsupported for ${track} clips`), { code: "UNSUPPORTED_TRACK" }); + } + if ((track === "visuals" || track === "overlays") && clip.type !== "video") { + throw Object.assign(new Error("Mute is supported only for video or audio clips"), { code: "UNSUPPORTED_MEDIA_TYPE" }); + } + clip.muted = operation.muted; +} + +const TRACK_STATE_KEYS = Object.freeze({ + visuals: "image", image: "image", + captions: "caption", caption: "caption", + audio: "audio", + stickers: "sticker", sticker: "sticker", + overlays: "overlay", overlay: "overlay", + source: "source", + music: "music", +}); + +function setTrackState(project, operation, stateKey) { + const track = TRACK_STATE_KEYS[operation.track]; + if (!track) throw Object.assign(new Error(`Unknown track: ${operation.track}`), { code: "TRACK_NOT_FOUND" }); + const value = operation[stateKey]; + if (typeof value !== "boolean") throw Object.assign(new Error(`${stateKey} must be a boolean`), { code: "INVALID_ARGUMENT" }); + const projectKey = stateKey === "visible" ? "trackVisibility" : "trackLocks"; + project[projectKey] = { ...(project[projectKey] || {}), [track]: value }; +} + +function setProjectRatio(project, operation) { + const supportedRatios = new Set(["16:9", "9:16", "1:1", "4:5"]); + if (!supportedRatios.has(operation.ratio)) throw Object.assign(new Error(`Unsupported project ratio: ${operation.ratio}`), { code: "INVALID_RATIO" }); + project.ratioId = operation.ratio; +} + const reducers = { + "asset.import": importAsset, "timed.move": moveTimed, + "timed.resize": resizeTimed, + "visual.trim": trimVisual, + "visual.split": splitVisual, + "visual.reorder": reorderVisual, + "visual.append": appendVisual, + "visual.insert": insertVisual, + "overlay.add": addOverlay, + "transition.set": setTransition, + "caption.add": addCaption, "caption.update": updateCaption, "caption.unlink_audio": unlinkCaption, "caption.link_audio": linkCaption, + "clip.delete": deleteClip, + "clip.set_property": setClipProperty, + "clip.set_speed": setClipSpeed, + "clip.set_muted": setClipMuted, + "track.set_visibility": (project, operation) => setTrackState(project, operation, "visible"), + "track.set_locked": (project, operation) => setTrackState(project, operation, "locked"), + "project.set_ratio": setProjectRatio, }; export function validateCommandPlan(plan) { @@ -101,19 +496,161 @@ export function inspectProject(project) { const state = commandState(project); const captions = Array.isArray(project?.captionSegments) ? project.captionSegments : []; const audio = Array.isArray(project?.audioSegments) ? project.audioSegments : []; - const duration = [...captions.map((item) => Number(item.end) || 0), ...audio.map((item) => (Number(item.start) || 0) + (Number(item.duration) || 0))] + const visuals = Array.isArray(project?.visualSegments) ? project.visualSegments : []; + const stickers = Array.isArray(project?.stickerSegments) ? project.stickerSegments : []; + const overlays = Array.isArray(project?.visualOverlaySegments) ? project.visualOverlaySegments : []; + const music = Array.isArray(project?.musicSegments) ? project.musicSegments : []; + const visualDuration = visuals.reduce((total, item) => total + Math.max(0, Number(item.duration) || 0), 0); + const duration = [visualDuration, ...captions.map((item) => Number(item.end) || 0), ...audio.map((item) => (Number(item.start) || 0) + (Number(item.duration) || 0)), + ...stickers.map((item) => (Number(item.start) || 0) + (Number(item.duration) || 0)), + ...overlays.map((item) => (Number(item.start) || 0) + (Number(item.duration) || 0)), + ...music.map((item) => (Number(item.start) || 0) + (Number(item.duration) || 0))] .reduce((maximum, value) => Math.max(maximum, value), 0); return { schemaVersion: PROJECT_COMMAND_SCHEMA_VERSION, revision: state.revision, duration, ratio: project?.ratioId || "16:9", - tracks: { captions: captions.length, audio: audio.length, visuals: project?.visualSegments?.length || 0 }, + tracks: { captions: captions.length, audio: audio.length, visuals: visuals.length, stickers: stickers.length, overlays: overlays.length, music: music.length }, appliedOperationIds: state.appliedOperationIds, warnings: audio.length ? [] : ["Project has no serialized voiceover clips"], }; } +const TRACK_COLLECTIONS = Object.freeze({ + visuals: "visualSegments", + captions: "captionSegments", + audio: "audioSegments", + stickers: "stickerSegments", + overlays: "visualOverlaySegments", + music: "musicSegments", +}); + +function clipSummary(track, clip, index, visualStart = 0) { + if (track === "visuals") { + const duration = Math.max(0, Number(clip.duration) || 0); + return { id: clip.id, index, type: clip.type || "image", start: visualStart, end: visualStart + duration, duration, + assetId: clip.assetId || "", name: clip.name || "" }; + } + if (track === "captions") { + const start = Math.max(0, Number(clip.start) || 0); + const end = Math.max(start, Number(clip.end) || start); + return { id: clip.id, index, start, end, duration: end - start, text: clip.text || "", audioSegmentId: clip.audioSegmentId || "" }; + } + const start = Math.max(0, Number(clip.start) || 0); + const duration = Math.max(0, Number(clip.duration) || 0); + return { id: clip.id, index, start, end: start + duration, duration, name: clip.name || "" }; +} + +export function inspectTrack(project, track) { + const key = TRACK_COLLECTIONS[track]; + if (!key) throw Object.assign(new Error(`Unknown track: ${track}`), { code: "TRACK_NOT_FOUND" }); + const clips = Array.isArray(project?.[key]) ? project[key] : []; + let visualCursor = 0; + const summaries = clips.map((clip, index) => { + const summary = clipSummary(track, clip, index, visualCursor); + if (track === "visuals") visualCursor = summary.end; + return summary; + }).sort((left, right) => left.start - right.start || left.index - right.index); + return { + schemaVersion: PROJECT_COMMAND_SCHEMA_VERSION, + revision: commandState(project).revision, + track, + visible: project?.trackVisibility?.[TRACK_STATE_KEYS[track]] ?? true, + locked: project?.trackLocks?.[TRACK_STATE_KEYS[track]] ?? false, + clipCount: summaries.length, + duration: summaries.reduce((maximum, clip) => Math.max(maximum, clip.end), 0), + clips: summaries, + }; +} + +export function inspectClip(project, clipId) { + const { track, clip, index } = findClipMatch(project, clipId); + const visualStart = track === "visuals" + ? project.visualSegments.slice(0, index).reduce((total, item) => total + Math.max(0, Number(item.duration) || 0), 0) + : 0; + return { + schemaVersion: PROJECT_COMMAND_SCHEMA_VERSION, + revision: commandState(project).revision, + track, + summary: clipSummary(track, clip, index, visualStart), + source: { + assetId: clip.assetId || "", + archiveMediaId: clip.archiveMediaId || clip.id, + sourceStart: Math.max(0, Number(clip.sourceStart) || 0), + sourceDuration: Math.max(0, Number(clip.sourceDuration) || 0), + playbackRate: visualPlaybackRate(clip), + }, + links: { audioSegmentId: clip.audioSegmentId || "", detachedAudioSegmentId: clip.detachedAudioSegmentId || "" }, + properties: Object.fromEntries(Object.entries(clip).filter(([key]) => !["id", "assetId", "archiveMediaId", "sourceStart", "sourceDuration", "playbackRate", "audioSegmentId", "detachedAudioSegmentId"].includes(key))), + }; +} + +export function inspectTranscript(project, audioClipId = "") { + const captions = (Array.isArray(project?.captionSegments) ? project.captionSegments : []) + .filter((caption) => !audioClipId || caption.audioSegmentId === audioClipId) + .map((caption, index) => { + const start = Math.max(0, Number(caption.start) || 0); + const end = Math.max(start, Number(caption.end) || start); + const words = (Array.isArray(caption.words) ? caption.words : []).map((word) => ({ + text: String(word?.text ?? word?.word ?? ""), + start: Math.max(start, Number(word?.start) || start), + end: Math.max(start, Number(word?.end) || Number(word?.start) || start), + ...(Number.isFinite(Number(word?.confidence)) ? { confidence: Number(word.confidence) } : {}), + })); + return { + id: caption.id, + index, + text: caption.text || "", + start, + end, + duration: end - start, + speaker: caption.speaker || caption.speakerId || "", + audioClipId: caption.audioSegmentId || "", + words, + }; + }).sort((left, right) => left.start - right.start || left.index - right.index); + return { + schemaVersion: PROJECT_COMMAND_SCHEMA_VERSION, + revision: commandState(project).revision, + audioClipId, + segmentCount: captions.length, + wordCount: captions.reduce((total, caption) => total + caption.words.length, 0), + duration: captions.reduce((maximum, caption) => Math.max(maximum, caption.end), 0), + text: captions.map((caption) => caption.text).join("\n"), + segments: captions, + }; +} + +function sameValue(left, right) { + return JSON.stringify(left) === JSON.stringify(right); +} + +export function diffProjects(beforeProject, afterProject) { + const projectFields = ["ratioId", "fitMode", "trackVisibility", "trackLocks", "script"] + .flatMap((field) => sameValue(beforeProject?.[field], afterProject?.[field]) ? [] : [{ field, before: beforeProject?.[field] ?? null, after: afterProject?.[field] ?? null }]); + const tracks = Object.fromEntries(Object.entries(TRACK_COLLECTIONS).flatMap(([track, key]) => { + const before = Array.isArray(beforeProject?.[key]) ? beforeProject[key] : []; + const after = Array.isArray(afterProject?.[key]) ? afterProject[key] : []; + const beforeById = new Map(before.map((clip) => [clip.id, clip])); + const afterById = new Map(after.map((clip) => [clip.id, clip])); + const added = after.filter((clip) => !beforeById.has(clip.id)).map((clip) => clip.id); + const removed = before.filter((clip) => !afterById.has(clip.id)).map((clip) => clip.id); + const modified = after.flatMap((clip) => { + const previous = beforeById.get(clip.id); + if (!previous || sameValue(previous, clip)) return []; + const fields = [...new Set([...Object.keys(previous), ...Object.keys(clip)])] + .filter((field) => !sameValue(previous[field], clip[field])); + return [{ id: clip.id, fields, before: Object.fromEntries(fields.map((field) => [field, previous[field] ?? null])), after: Object.fromEntries(fields.map((field) => [field, clip[field] ?? null])) }]; + }); + const orderBefore = before.map((clip) => clip.id); + const orderAfter = after.map((clip) => clip.id); + if (!added.length && !removed.length && !modified.length && sameValue(orderBefore, orderAfter)) return []; + return [[track, { added, removed, modified, ...(sameValue(orderBefore, orderAfter) ? {} : { orderBefore, orderAfter }) }]]; + })); + return { projectFields, tracks }; +} + export function applyCommandPlan(project, plan) { const validity = validateCommandPlan(plan); if (!validity.ok) return validity; @@ -128,6 +665,7 @@ export function applyCommandPlan(project, plan) { project: structuredClone(project), before: inspectProject(project), after: inspectProject(project), + changes: { projectFields: [], tracks: {} }, }; } if (plan.baseRevision !== current.revision) { @@ -160,5 +698,6 @@ export function applyCommandPlan(project, plan) { project: next, before: inspectProject(project), after: inspectProject(next), + changes: diffProjects(project, next), }; } diff --git a/src/lib/projectCommandEngine.test.js b/src/lib/projectCommandEngine.test.js index c14f56f..7f7c3d0 100644 --- a/src/lib/projectCommandEngine.test.js +++ b/src/lib/projectCommandEngine.test.js @@ -1,13 +1,16 @@ import { describe, expect, it } from "vitest"; -import { applyCommandPlan, inspectProject } from "./projectCommandEngine.js"; +import { applyCommandPlan, diffProjects, inspectClip, inspectProject, inspectTrack, inspectTranscript } from "./projectCommandEngine.js"; function project() { return { ratioId: "16:9", script: "Hello", audioSegments: [{ id: "voice-1", start: 2, duration: 3 }], - captionSegments: [{ id: "caption-1", text: "Hello", start: 2, end: 5, audioSegmentId: "voice-1" }], - visualSegments: [{ id: "visual-1", duration: 8 }], + captionSegments: [{ id: "caption-1", text: "Hello", start: 2, end: 5, audioSegmentId: "voice-1", speaker: "Narrator", words: [{ word: "Hello", start: 2.1, end: 2.6, confidence: 0.98 }] }], + visualSegments: [ + { id: "visual-1", type: "video", duration: 8, sourceStart: 1, sourceDuration: 16, playbackRate: 2, keyframes: [{ time: 1, scale: 1 }, { time: 5, scale: 2 }] }, + { id: "visual-2", type: "image", duration: 2 }, + ], }; } @@ -26,6 +29,162 @@ describe("project command engine", () => { expect(result.revision).toBe(1); }); + it("resizes timed clips and clamps linked captions to the new audio end", () => { + const result = applyCommandPlan(project(), plan([ + { id: "resize-1", type: "timed.resize", track: "audio", clipId: "voice-1", start: 4, duration: 2 }, + ])); + expect(result.project.audioSegments[0]).toMatchObject({ start: 4, duration: 2 }); + expect(result.project.captionSegments[0]).toMatchObject({ start: 4, end: 6 }); + }); + + it("trims video source time and remaps clip-local keyframes", () => { + const result = applyCommandPlan(project(), plan([ + { id: "trim-1", type: "visual.trim", clipId: "visual-1", sourceIn: 3, sourceOut: 11 }, + ])); + expect(result.project.visualSegments[0]).toMatchObject({ sourceStart: 3, sourceDuration: 8, duration: 4 }); + expect(result.project.visualSegments[0].keyframes).toEqual([{ time: 0, scale: 1 }, { time: 4, scale: 2 }]); + }); + + it("splits visuals with an explicit stable ID and preserves source mapping", () => { + const result = applyCommandPlan(project(), plan([ + { id: "split-1", type: "visual.split", clipId: "visual-1", at: 3, rightClipId: "visual-1b" }, + ])); + expect(result.project.visualSegments.slice(0, 2)).toMatchObject([ + { id: "visual-1", duration: 3, sourceStart: 1, sourceDuration: 6 }, + { id: "visual-1b", duration: 5, sourceStart: 7, sourceDuration: 10 }, + ]); + expect(result.project.visualSegments[1].keyframes).toEqual([{ time: 2, scale: 2 }]); + }); + + it("reorders contiguous visual clips by stable ID", () => { + const result = applyCommandPlan(project(), plan([ + { id: "reorder-1", type: "visual.reorder", clipId: "visual-2", toIndex: 0 }, + ])); + expect(result.project.visualSegments.map((segment) => segment.id)).toEqual(["visual-2", "visual-1"]); + }); + + it("sets validated clip properties, speed, mute, track state, and project ratio", () => { + const result = applyCommandPlan(project(), plan([ + { id: "property-1", type: "clip.set_property", clipId: "visual-1", property: "opacity", value: 0.6 }, + { id: "speed-1", type: "clip.set_speed", clipId: "visual-1", speed: 4 }, + { id: "mute-1", type: "clip.set_muted", clipId: "visual-1", muted: true }, + { id: "visibility-1", type: "track.set_visibility", track: "visuals", visible: false }, + { id: "lock-1", type: "track.set_locked", track: "captions", locked: true }, + { id: "ratio-1", type: "project.set_ratio", ratio: "9:16" }, + ])); + expect(result.project.visualSegments[0]).toMatchObject({ opacity: 0.6, playbackRate: 4, sourceDuration: 16, duration: 4, muted: true }); + expect(result.project.visualSegments[0].keyframes).toEqual([{ time: 0.5, scale: 1 }, { time: 2.5, scale: 2 }]); + expect(result.project).toMatchObject({ ratioId: "9:16", trackVisibility: { image: false }, trackLocks: { caption: true } }); + expect(inspectTrack(result.project, "visuals")).toMatchObject({ visible: false, locked: false }); + expect(inspectTrack(result.project, "captions")).toMatchObject({ visible: true, locked: true }); + }); + + it("keeps source duration while speeding up audio and clamps linked captions", () => { + const result = applyCommandPlan(project(), plan([ + { id: "speed-1", type: "clip.set_speed", clipId: "voice-1", speed: 1.5 }, + ])); + expect(result.project.audioSegments[0]).toMatchObject({ sourceDuration: 3, playbackRate: 1.5, duration: 2 }); + expect(result.project.captionSegments[0]).toMatchObject({ start: 2, end: 4 }); + }); + + it("rejects unsafe arbitrary clip properties transactionally", () => { + const original = project(); + const result = applyCommandPlan(original, plan([ + { id: "ratio-1", type: "project.set_ratio", ratio: "1:1" }, + { id: "property-1", type: "clip.set_property", clipId: "visual-1", property: "sourceStart", value: 99 }, + ])); + expect(result).toMatchObject({ ok: false, code: "UNSUPPORTED_PROPERTY", operationId: "property-1" }); + expect(original.ratioId).toBe("16:9"); + expect(original.visualSegments[0].sourceStart).toBe(1); + }); + + it("appends and inserts archived visual media without breaking the contiguous sequence", () => { + const result = applyCommandPlan(project(), plan([ + { id: "append-1", type: "visual.append", sourceClipId: "visual-1", clipId: "visual-3", duration: 2 }, + { id: "insert-1", type: "visual.insert", sourceClipId: "visual-2", clipId: "visual-4", duration: 1, atIndex: 1 }, + ])); + expect(result.project.visualSegments.map((clip) => clip.id)).toEqual(["visual-1", "visual-4", "visual-2", "visual-3"]); + expect(result.project.visualSegments[1]).toMatchObject({ id: "visual-4", type: "image", duration: 1, archiveMediaId: "visual-2" }); + expect(result.project.visualSegments[3]).toMatchObject({ id: "visual-3", type: "video", duration: 2, sourceDuration: 4, archiveMediaId: "visual-1" }); + expect(inspectTrack(result.project, "visuals")).toMatchObject({ duration: 13 }); + }); + + it("adds a prepared binary asset to Visuals with integrity metadata", () => { + const result = applyCommandPlan(project(), plan([ + { + id: "import-1", type: "asset.import", prepared: true, track: "visuals", clipId: "visual-imported", + mediaType: "image", mimeType: "image/png", name: "card.png", duration: 3, + sha256: "a".repeat(64), size: 128, archivePath: "media/visuals/import-card.png", width: 800, height: 600, + }, + ])); + expect(result.project.visualSegments.at(-1)).toMatchObject({ + id: "visual-imported", type: "image", name: "card.png", duration: 3, archiveMediaId: "visual-imported", + integrity: { sha256: "a".repeat(64), size: 128, mimeType: "image/png", archivePath: "media/visuals/import-card.png" }, + }); + }); + + it("rejects unprepared asset imports without changing the project", () => { + const original = project(); + const result = applyCommandPlan(original, plan([ + { id: "import-1", type: "asset.import", track: "visuals", clipId: "visual-imported", file: "/tmp/card.png" }, + ])); + expect(result).toMatchObject({ ok: false, code: "ASSET_NOT_PREPARED", operationId: "import-1" }); + expect(original.visualSegments).toHaveLength(2); + }); + + it("imports prepared audio into Music and protects the occupied Voiceover media slot", () => { + const music = applyCommandPlan(project(), plan([{ + id: "music-1", type: "asset.import", prepared: true, track: "music", clipId: "music-1", mediaType: "audio", + mimeType: "audio/wav", name: "bed.wav", duration: 6, start: 1, sha256: "b".repeat(64), size: 256, archivePath: "media/audio/bed.wav", + }])); + expect(music.project).toMatchObject({ musicName: "bed.wav", musicDuration: 6, musicStart: 1, musicSegments: [{ id: "music-1", start: 1, duration: 6, volume: 0.35 }] }); + expect(inspectTrack(music.project, "music")).toMatchObject({ clipCount: 1, duration: 7 }); + + const voice = applyCommandPlan(project(), plan([{ + id: "voice-2", type: "asset.import", prepared: true, track: "audio", clipId: "voice-2", mediaType: "audio", + mimeType: "audio/wav", name: "voice.wav", duration: 1, sha256: "c".repeat(64), size: 64, archivePath: "media/audio/voice.wav", + }])); + expect(voice).toMatchObject({ ok: false, code: "MEDIA_SLOT_OCCUPIED", operationId: "voice-2" }); + }); + + it("adds a timed overlay from archived media with a validated transform", () => { + const result = applyCommandPlan(project(), plan([ + { id: "overlay-1", type: "overlay.add", sourceClipId: "visual-1", clipId: "overlay-1", start: 3, duration: 2, layer: 2, transform: { x: -20, y: 15, scale: 0.4 } }, + ])); + expect(result.project.visualOverlaySegments[0]).toMatchObject({ + id: "overlay-1", archiveMediaId: "visual-1", start: 3, duration: 2, layer: 2, + sourceStart: 1, sourceDuration: 4, playbackRate: 2, + baseTransform: { x: -20, y: 15, scale: 0.4, rotation: 0, opacity: 1 }, + }); + }); + + it("sets a bounded transition on an outgoing visual junction", () => { + const result = applyCommandPlan(project(), plan([ + { id: "transition-1", type: "transition.set", clipId: "visual-1", transitionId: "fade", duration: 0.8 }, + ])); + expect(result.project.visualSegments[0].transition).toEqual({ id: "fade", duration: 0.8 }); + }); + + it("rolls back when a transition targets the final visual", () => { + const original = project(); + const result = applyCommandPlan(original, plan([ + { id: "append-1", type: "visual.append", sourceClipId: "visual-2", clipId: "visual-3", duration: 1 }, + { id: "transition-1", type: "transition.set", clipId: "visual-3", transitionId: "fade", duration: 0.2 }, + ])); + expect(result).toMatchObject({ ok: false, code: "INVALID_TRANSITION_TARGET", operationId: "transition-1" }); + expect(original.visualSegments.map((clip) => clip.id)).toEqual(["visual-1", "visual-2"]); + }); + + it("rejects trims outside the serialized source range without partial edits", () => { + const original = project(); + const result = applyCommandPlan(original, plan([ + { id: "reorder-1", type: "visual.reorder", clipId: "visual-2", toIndex: 0 }, + { id: "trim-1", type: "visual.trim", clipId: "visual-1", sourceIn: 0, sourceOut: 20 }, + ])); + expect(result).toMatchObject({ ok: false, code: "SOURCE_RANGE_EXCEEDED", operationId: "trim-1" }); + expect(original.visualSegments.map((segment) => segment.id)).toEqual(["visual-1", "visual-2"]); + }); + it("unlinks, edits, and relinks a caption with optional alignment", () => { const result = applyCommandPlan(project(), plan([ { id: "unlink-1", type: "caption.unlink_audio", clipId: "caption-1" }, @@ -36,6 +195,33 @@ describe("project command engine", () => { expect(result.project.script).toBe("Updated"); }); + it("adds captions in timeline order and can link them to audio", () => { + const result = applyCommandPlan(project(), plan([ + { id: "add-1", type: "caption.add", clipId: "caption-2", text: "Earlier", start: 0, end: 1, audioClipId: "voice-1" }, + ])); + expect(result.project.captionSegments.map((caption) => caption.id)).toEqual(["caption-2", "caption-1"]); + expect(result.project.captionSegments[0]).toMatchObject({ text: "Earlier", audioSegmentId: "voice-1" }); + expect(result.project.script).toBe("Earlier\nHello"); + }); + + it("deletes clips and preserves a reversible caption-to-audio association", () => { + const result = applyCommandPlan(project(), plan([ + { id: "delete-1", type: "clip.delete", track: "audio", clipId: "voice-1" }, + ])); + expect(result.project.audioSegments).toEqual([]); + expect(result.project.captionSegments[0]).toMatchObject({ audioSegmentId: "", detachedAudioSegmentId: "voice-1" }); + }); + + it("rolls back caption additions with duplicate clip IDs", () => { + const original = project(); + const result = applyCommandPlan(original, plan([ + { id: "add-1", type: "caption.add", clipId: "caption-2", text: "New", start: 0, end: 1 }, + { id: "add-2", type: "caption.add", clipId: "caption-1", text: "Duplicate", start: 1, end: 2 }, + ])); + expect(result).toMatchObject({ ok: false, code: "CLIP_ALREADY_EXISTS", operationId: "add-2" }); + expect(original.captionSegments).toHaveLength(1); + }); + it("rejects a failing batch without changing the input", () => { const original = project(); const result = applyCommandPlan(original, plan([ @@ -57,6 +243,49 @@ describe("project command engine", () => { }); it("inspects stable project facts", () => { - expect(inspectProject(project())).toMatchObject({ revision: 0, duration: 5, ratio: "16:9", tracks: { captions: 1, audio: 1, visuals: 1 } }); + expect(inspectProject(project())).toMatchObject({ revision: 0, duration: 10, ratio: "16:9", tracks: { captions: 1, audio: 1, visuals: 2, stickers: 0, overlays: 0 } }); + }); + + it("inspects ordered tracks and detailed clips without mutating the project", () => { + const original = project(); + expect(inspectTrack(original, "visuals")).toMatchObject({ + track: "visuals", + clipCount: 2, + duration: 10, + clips: [{ id: "visual-1", start: 0, end: 8 }, { id: "visual-2", start: 8, end: 10 }], + }); + expect(inspectClip(original, "visual-1")).toMatchObject({ + track: "visuals", + summary: { id: "visual-1", index: 0, start: 0, duration: 8 }, + source: { sourceStart: 1, sourceDuration: 16, playbackRate: 2 }, + properties: { type: "video", keyframes: [{ time: 1, scale: 1 }, { time: 5, scale: 2 }] }, + }); + expect(original.visualSegments[0].duration).toBe(8); + }); + + it("inspects timestamped transcript segments, speakers, words, and audio scope", () => { + expect(inspectTranscript(project(), "voice-1")).toMatchObject({ + audioClipId: "voice-1", + segmentCount: 1, + wordCount: 1, + duration: 5, + text: "Hello", + segments: [{ id: "caption-1", speaker: "Narrator", start: 2, end: 5, words: [{ text: "Hello", start: 2.1, end: 2.6, confidence: 0.98 }] }], + }); + expect(inspectTranscript(project(), "missing")).toMatchObject({ segmentCount: 0, wordCount: 0, text: "" }); + }); + + it("produces field-level project and clip changes", () => { + const before = project(); + const result = applyCommandPlan(before, plan([ + { id: "ratio-1", type: "project.set_ratio", ratio: "1:1" }, + { id: "property-1", type: "clip.set_property", clipId: "visual-1", property: "opacity", value: 0.5 }, + { id: "append-1", type: "visual.append", sourceClipId: "visual-2", clipId: "visual-3", duration: 1 }, + ])); + expect(result.changes).toMatchObject({ + projectFields: [{ field: "ratioId", before: "16:9", after: "1:1" }], + tracks: { visuals: { added: ["visual-3"], removed: [], modified: [{ id: "visual-1", fields: ["opacity"] }] } }, + }); + expect(diffProjects(before, result.project)).toEqual(result.changes); }); }); diff --git a/src/lib/projectRenderPlan.js b/src/lib/projectRenderPlan.js new file mode 100644 index 0000000..bd087f3 --- /dev/null +++ b/src/lib/projectRenderPlan.js @@ -0,0 +1,109 @@ +const RATIO_SIZES = Object.freeze({ + "16:9": { width: 1280, height: 720 }, + "9:16": { width: 720, height: 1280 }, + "1:1": { width: 1080, height: 1080 }, + "4:5": { width: 864, height: 1080 }, +}); + +function renderError(code, message) { + return Object.assign(new Error(message), { code }); +} + +function evenDimension(value, fallback, name) { + const number = value == null ? fallback : Number(value); + if (!Number.isFinite(number) || number < 2) throw renderError("INVALID_RENDER_SETTINGS", `${name} must be at least 2`); + return Math.max(2, Math.round(number / 2) * 2); +} + +function finitePositive(value, name) { + const number = Number(value); + if (!Number.isFinite(number) || number <= 0) throw renderError("INVALID_PROJECT", `${name} must be greater than zero`); + return number; +} + +function atempoChain(rate) { + const filters = []; + let remaining = rate; + while (remaining > 2) { filters.push("atempo=2"); remaining /= 2; } + while (remaining < 0.5) { filters.push("atempo=0.5"); remaining /= 0.5; } + filters.push(`atempo=${remaining.toFixed(6)}`); + return filters.join(","); +} + +function assertSupportedProject(project) { + const unsupported = []; + if ((project.captionSegments || []).some((item) => !item.hidden)) unsupported.push("captions"); + if ((project.stickerSegments || []).length) unsupported.push("stickers"); + if ((project.visualOverlaySegments || []).length) unsupported.push("overlays"); + if (project.trackVisibility?.source !== false && (project.sourceAudioSegments || []).length) unsupported.push("source audio"); + if ((project.visualSegments || []).some((item) => item.transition?.id && item.transition.id !== "none")) unsupported.push("transitions"); + if ((project.visualSegments || []).some((item) => item.keyframes?.length || item.mask?.type || item.filter || item.effects?.length || item.vision)) unsupported.push("visual effects"); + if (unsupported.length) { + throw renderError("UNSUPPORTED_RENDER_FEATURE", `Headless render does not yet support: ${[...new Set(unsupported)].join(", ")}`); + } +} + +function resolveVisualPath(segment, media, extractedFiles) { + const entry = (media.visuals || []).find((item) => [segment.id, segment.archiveMediaId, segment.assetId].includes(item.id)); + const path = entry?.path ? extractedFiles.get(entry.path) : null; + if (!path) throw renderError("MISSING_MEDIA", `Portable media is missing for visual clip: ${segment.id}`); + return path; +} + +function addAudioTrack({ args, filters, inputs, segments, mediaEntry, extractedFiles, duration, prefix, defaultVolume = 1 }) { + if (!segments.length || !mediaEntry?.path) return []; + const path = extractedFiles.get(mediaEntry.path); + if (!path) throw renderError("MISSING_MEDIA", `Portable media is missing for ${prefix}`); + return segments.map((segment, index) => { + const inputIndex = inputs.count++; + args.push("-i", path); + const rate = Math.max(0.25, Math.min(4, Number(segment.playbackRate) || 1)); + const sourceStart = Math.max(0, Number(segment.sourceStart) || 0); + const sourceDuration = Math.max(0.001, Number(segment.sourceDuration) || finitePositive(segment.duration, `${prefix} duration`) * rate); + const start = Math.max(0, Number(segment.start) || 0); + const volume = Number.isFinite(Number(segment.volume)) ? Number(segment.volume) : defaultVolume; + const label = `${prefix}${index}`; + filters.push(`[${inputIndex}:a]atrim=start=${sourceStart}:duration=${sourceDuration},asetpts=PTS-STARTPTS,${atempoChain(rate)},volume=${volume},adelay=${Math.round(start * 1000)}|${Math.round(start * 1000)},apad,atrim=duration=${duration}[${label}]`); + return `[${label}]`; + }); +} + +export function buildFfmpegRenderPlan({ project, media = {}, extractedFiles, settings = {} }) { + if (!(extractedFiles instanceof Map)) throw renderError("INVALID_ARGUMENT", "extractedFiles must be a Map"); + assertSupportedProject(project || {}); + const visuals = project.visualSegments || []; + if (!visuals.length) throw renderError("EMPTY_TIMELINE", "Headless render requires at least one visual clip"); + const ratio = RATIO_SIZES[project.ratioId] || RATIO_SIZES["16:9"]; + const width = evenDimension(settings.width, ratio.width, "width"); + const height = evenDimension(settings.height, ratio.height, "height"); + const frameRate = Math.max(1, Math.min(60, Math.round(Number(settings.frameRate) || 30))); + const duration = visuals.reduce((sum, segment) => sum + finitePositive(segment.duration, `Visual clip ${segment.id} duration`), 0); + const args = ["-hide_banner", "-y"]; + const filters = []; + const inputs = { count: 0 }; + const videoLabels = visuals.map((segment, index) => { + const path = resolveVisualPath(segment, media, extractedFiles); + const clipDuration = Number(segment.duration); + const inputIndex = inputs.count++; + if (segment.type === "image") args.push("-loop", "1", "-t", String(clipDuration), "-i", path); + else args.push("-i", path); + const sourceStart = Math.max(0, Number(segment.sourceStart) || 0); + const rate = Math.max(0.25, Math.min(4, Number(segment.playbackRate) || 1)); + const trim = segment.type === "video" ? `trim=start=${sourceStart}:duration=${clipDuration * rate},setpts=(PTS-STARTPTS)/${rate},` : ""; + filters.push(`[${inputIndex}:v]${trim}scale=${width}:${height}:force_original_aspect_ratio=decrease,pad=${width}:${height}:(ow-iw)/2:(oh-ih)/2:black,fps=${frameRate},setsar=1,format=yuv420p[v${index}]`); + return `[v${index}]`; + }); + filters.push(`${videoLabels.join("")}concat=n=${videoLabels.length}:v=1:a=0[vout]`); + + const visible = project.trackVisibility || {}; + const audioLabels = []; + if (visible.audio !== false) audioLabels.push(...addAudioTrack({ args, filters, inputs, segments: project.audioSegments || [], mediaEntry: media.audio, extractedFiles, duration, prefix: "voice" })); + if (visible.music !== false) audioLabels.push(...addAudioTrack({ args, filters, inputs, segments: project.musicSegments || [], mediaEntry: media.music, extractedFiles, duration, prefix: "music", defaultVolume: Number(project.musicVolume) || 0.35 })); + if (audioLabels.length) filters.push(`${audioLabels.join("")}amix=inputs=${audioLabels.length}:duration=longest:normalize=0,atrim=duration=${duration}[aout]`); + + args.push("-filter_complex", filters.join(";"), "-map", "[vout]"); + if (audioLabels.length) args.push("-map", "[aout]", "-c:a", "aac", "-b:a", "192k"); + else args.push("-an"); + args.push("-c:v", "libx264", "-preset", settings.preset || "medium", "-crf", String(Number(settings.crf) || 18), "-pix_fmt", "yuv420p", "-r", String(frameRate), "-t", String(duration), "-movflags", "+faststart"); + return { args, duration, width, height, frameRate, hasAudio: audioLabels.length > 0 }; +} diff --git a/src/lib/projectRenderPlan.test.js b/src/lib/projectRenderPlan.test.js new file mode 100644 index 0000000..b738896 --- /dev/null +++ b/src/lib/projectRenderPlan.test.js @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; +import { buildFfmpegRenderPlan } from "./projectRenderPlan.js"; + +const files = new Map([ + ["media/visuals/image.png", "/tmp/image.png"], + ["media/audio/voice.wav", "/tmp/voice.wav"], +]); + +describe("buildFfmpegRenderPlan", () => { + it("builds a deterministic MP4 plan for portable visual and voice media", () => { + const plan = buildFfmpegRenderPlan({ + project: { + ratioId: "9:16", + visualSegments: [{ id: "visual", type: "image", duration: 2 }], + audioSegments: [{ id: "voice", start: 0.5, duration: 1, volume: 0.8 }], + }, + media: { + visuals: [{ id: "visual", path: "media/visuals/image.png" }], + audio: { path: "media/audio/voice.wav" }, + }, + extractedFiles: files, + settings: { width: 360, height: 640, frameRate: 24 }, + }); + expect(plan).toMatchObject({ duration: 2, width: 360, height: 640, frameRate: 24, hasAudio: true }); + expect(plan.args.join(" ")).toContain("concat=n=1:v=1:a=0[vout]"); + expect(plan.args.join(" ")).toContain("amix=inputs=1"); + }); + + it("refuses features that would otherwise be silently omitted", () => { + expect(() => buildFfmpegRenderPlan({ + project: { + visualSegments: [{ id: "visual", type: "image", duration: 2 }], + captionSegments: [{ id: "caption", text: "Visible", start: 0, end: 1 }], + }, + media: { visuals: [{ id: "visual", path: "media/visuals/image.png" }] }, + extractedFiles: files, + })).toThrowError(expect.objectContaining({ code: "UNSUPPORTED_RENDER_FEATURE" })); + }); +}); + diff --git a/src/lib/timelineCommandCli.test.js b/src/lib/timelineCommandCli.test.js index 0f7a19d..2db572e 100644 --- a/src/lib/timelineCommandCli.test.js +++ b/src/lib/timelineCommandCli.test.js @@ -8,29 +8,126 @@ import { describe, expect, it } from "vitest"; const execute = promisify(execFile); +function silentWav(seconds = 0.1, sampleRate = 8000) { + const samples = Math.max(1, Math.round(seconds * sampleRate)); + const buffer = Buffer.alloc(44 + samples * 2); + buffer.write("RIFF", 0); buffer.writeUInt32LE(buffer.length - 8, 4); buffer.write("WAVE", 8); + buffer.write("fmt ", 12); buffer.writeUInt32LE(16, 16); buffer.writeUInt16LE(1, 20); buffer.writeUInt16LE(1, 22); + buffer.writeUInt32LE(sampleRate, 24); buffer.writeUInt32LE(sampleRate * 2, 28); buffer.writeUInt16LE(2, 32); buffer.writeUInt16LE(16, 34); + buffer.write("data", 36); buffer.writeUInt32LE(samples * 2, 40); + return buffer; +} + +function solidPpm(width = 16, height = 16) { + const header = Buffer.from(`P6\n${width} ${height}\n255\n`); + return Buffer.concat([header, Buffer.alloc(width * height * 3, 128)]); +} + describe("timeline command CLI", () => { + it("renders and verifies a portable image and voice project transactionally", async () => { + const directory = await mkdtemp(join(tmpdir(), "timeline-render-")); + const input = join(directory, "input.timeline"); + const output = join(directory, "output.mp4"); + const requestPath = join(directory, "render.json"); + const payload = { + format: "timeline-studio-archive", + version: 2, + project: { + ratioId: "16:9", + visualSegments: [{ id: "visual", type: "image", duration: 0.25 }], + audioSegments: [{ id: "voice", start: 0, duration: 0.1 }], + }, + media: { + visuals: [{ id: "visual", path: "media/visuals/image.ppm", type: "image/x-portable-pixmap" }], + audio: { path: "media/audio/voice.wav", type: "audio/wav" }, + }, + }; + await writeFile(input, zipSync({ + "project.json": strToU8(JSON.stringify(payload)), + "media/visuals/image.ppm": solidPpm(), + "media/audio/voice.wav": silentWav(), + })); + await writeFile(requestPath, JSON.stringify({ + schemaVersion: 1, + project: input, + output: { video: output }, + render: { width: 160, height: 90, frameRate: 24, preset: "ultrafast", crf: 28 }, + })); + const rendered = JSON.parse((await execute(process.execPath, ["scripts/timeline-command.mjs", "project.render", requestPath], { cwd: process.cwd() })).stdout); + expect(rendered).toMatchObject({ + ok: true, + artifacts: { project: input, video: output }, + render: { width: 160, height: 90, frameRate: 24, hasAudio: true, codec: "h264", container: "mp4" }, + verification: { width: 160, height: 90, hasAudio: true }, + }); + expect(rendered.verification.duration).toBeGreaterThanOrEqual(0.2); + expect((await readFile(output)).length).toBeGreaterThan(1000); + }, 20_000); + it("dry-runs and writes a new archive while preserving media entries", async () => { const directory = await mkdtemp(join(tmpdir(), "timeline-command-")); const input = join(directory, "input.timeline"); const output = join(directory, "output.timeline"); const planPath = join(directory, "plan.json"); + const importedImage = join(directory, "imported.png"); + const importedMusic = join(directory, "music.wav"); + await writeFile(importedImage, new Uint8Array([137, 80, 78, 71, 1, 2, 3, 4])); + await writeFile(importedMusic, silentWav()); const payload = { format: "timeline-studio-archive", version: 2, project: { audioSegments: [{ id: "voice", start: 0, duration: 2 }], - captionSegments: [{ id: "caption", text: "Old", start: 0, end: 2, audioSegmentId: "voice" }], + captionSegments: [{ id: "caption", text: "Old", start: 0, end: 2, audioSegmentId: "voice", speaker: "Host", words: [{ text: "Old", start: 0.1, end: 0.5 }] }], + visualSegments: [{ id: "visual", type: "video", duration: 4, sourceStart: 0, sourceDuration: 4, playbackRate: 1 }], }, - media: {}, + media: { visuals: [{ id: "visual", path: "media/visuals/source.mp4", name: "source.mp4", type: "video/mp4", size: 3 }] }, }; - await writeFile(input, zipSync({ "project.json": strToU8(JSON.stringify(payload)), "media/audio/voice.wav": new Uint8Array([1, 2, 3]) })); + await writeFile(input, zipSync({ + "project.json": strToU8(JSON.stringify(payload)), + "media/audio/voice.wav": new Uint8Array([1, 2, 3]), + "media/visuals/source.mp4": new Uint8Array([4, 5, 6]), + })); const basePlan = { schemaVersion: 1, project: input, baseRevision: 0, - operations: [{ id: "edit-caption", type: "caption.update", clipId: "caption", text: "New" }], + operations: [ + { id: "edit-caption", type: "caption.update", clipId: "caption", text: "New" }, + { id: "add-caption", type: "caption.add", clipId: "caption-2", text: "Later", start: 2, end: 3 }, + { id: "split-visual", type: "visual.split", clipId: "visual", at: 1.5, rightClipId: "visual-b" }, + { id: "set-volume", type: "clip.set_property", clipId: "voice", property: "volume", value: 0.75 }, + { id: "speed-visual", type: "clip.set_speed", clipId: "visual-b", speed: 2 }, + { id: "mute-visual", type: "clip.set_muted", clipId: "visual", muted: true }, + { id: "hide-audio", type: "track.set_visibility", track: "audio", visible: false }, + { id: "set-ratio", type: "project.set_ratio", ratio: "9:16" }, + { id: "append-visual", type: "visual.append", sourceClipId: "visual", clipId: "visual-c", duration: 1 }, + { id: "add-overlay", type: "overlay.add", sourceClipId: "visual", clipId: "overlay-a", start: 0.5, duration: 1, layer: 1 }, + { id: "set-transition", type: "transition.set", clipId: "visual", transitionId: "fade", duration: 0.5 }, + { id: "import-image", type: "asset.import", file: importedImage, track: "visuals", clipId: "visual-imported", name: "imported.png", duration: 2, width: 640, height: 480 }, + { id: "import-music", type: "asset.import", file: importedMusic, track: "music", clipId: "music-imported", name: "music.wav", start: 1 }, + ], output: { project: output }, }; + const projectInspect = JSON.parse((await execute(process.execPath, ["scripts/timeline-command.mjs", "project.inspect", input], { cwd: process.cwd() })).stdout); + expect(projectInspect).toMatchObject({ ok: true, archiveVersion: 2, revision: 0, duration: 4, mediaInventory: { count: 2, paths: ["media/audio/voice.wav", "media/visuals/source.mp4"] } }); + const trackInspect = JSON.parse((await execute(process.execPath, ["scripts/timeline-command.mjs", "track.inspect", input, "visuals"], { cwd: process.cwd() })).stdout); + expect(trackInspect).toMatchObject({ ok: true, track: "visuals", clipCount: 1, clips: [{ id: "visual", start: 0, end: 4 }] }); + const clipInspect = JSON.parse((await execute(process.execPath, ["scripts/timeline-command.mjs", "clip.inspect", input, "visual"], { cwd: process.cwd() })).stdout); + expect(clipInspect).toMatchObject({ ok: true, track: "visuals", source: { sourceStart: 0, sourceDuration: 4, playbackRate: 1 } }); + const transcriptInspect = JSON.parse((await execute(process.execPath, ["scripts/timeline-command.mjs", "transcript.inspect", input, "voice"], { cwd: process.cwd() })).stdout); + expect(transcriptInspect).toMatchObject({ ok: true, audioClipId: "voice", segmentCount: 1, wordCount: 1, segments: [{ id: "caption", speaker: "Host", words: [{ text: "Old" }] }] }); + + await writeFile(planPath, JSON.stringify({ ...basePlan, dryRun: false })); + const archiveBeforeDiff = await readFile(input); + const diff = JSON.parse((await execute(process.execPath, ["scripts/timeline-command.mjs", "project.diff", planPath], { cwd: process.cwd() })).stdout); + expect(diff).toMatchObject({ ok: true, revision: 1, artifacts: {}, diff: { before: { revision: 0 }, after: { revision: 1 } } }); + expect(diff.diff.changes.projectFields).toEqual(expect.arrayContaining([{ field: "ratioId", before: null, after: "9:16" }])); + expect(diff.diff.changes.tracks.visuals).toMatchObject({ added: ["visual-b", "visual-c", "visual-imported"], modified: [{ id: "visual" }] }); + expect(diff.diff.changes.tracks.overlays).toMatchObject({ added: ["overlay-a"] }); + expect(await readFile(input)).toEqual(archiveBeforeDiff); + await expect(readFile(output)).rejects.toThrow(); + await writeFile(planPath, JSON.stringify({ ...basePlan, dryRun: true })); const dryRun = JSON.parse((await execute(process.execPath, ["scripts/timeline-command.mjs", "run", planPath], { cwd: process.cwd() })).stdout); expect(dryRun).toMatchObject({ ok: true, revision: 1, artifacts: {} }); @@ -41,9 +138,27 @@ describe("timeline command CLI", () => { expect(run.artifacts.project).toBe(output); const files = unzipSync(new Uint8Array(await readFile(output))); expect([...files["media/audio/voice.wav"]]).toEqual([1, 2, 3]); - expect(JSON.parse(strFromU8(files["project.json"])).project).toMatchObject({ - captionSegments: [{ id: "caption", text: "New" }], - commandState: { revision: 1, appliedOperationIds: ["edit-caption"] }, + expect([...files["media/visuals/source.mp4"]]).toEqual([4, 5, 6]); + const savedPayload = JSON.parse(strFromU8(files["project.json"])); + const importedManifest = savedPayload.media.visuals.find((item) => item.id === "visual-imported"); + expect(importedManifest).toMatchObject({ name: "imported.png", type: "image/png", size: 8 }); + expect([...files[importedManifest.path]]).toEqual([137, 80, 78, 71, 1, 2, 3, 4]); + expect(savedPayload.media.music).toMatchObject({ id: "music-imported", name: "music.wav", type: "audio/wav" }); + expect(files[savedPayload.media.music.path].length).toBeGreaterThan(44); + expect(savedPayload.project).toMatchObject({ + captionSegments: [{ id: "caption", text: "New" }, { id: "caption-2", text: "Later" }], + audioSegments: [{ id: "voice", volume: 0.75 }], + visualSegments: [ + { id: "visual", duration: 1.5, muted: true, transition: { id: "fade", duration: 0.5 } }, + { id: "visual-b", duration: 1.25, sourceStart: 1.5, sourceDuration: 2.5, playbackRate: 2 }, + { id: "visual-c", duration: 1, archiveMediaId: "visual" }, + { id: "visual-imported", duration: 2, archiveMediaId: "visual-imported", width: 640, height: 480 }, + ], + visualOverlaySegments: [{ id: "overlay-a", start: 0.5, duration: 1, archiveMediaId: "visual" }], + ratioId: "9:16", + trackVisibility: { audio: false }, + musicSegments: [{ id: "music-imported", start: 1, name: "music.wav" }], + commandState: { revision: 1, appliedOperationIds: ["edit-caption", "add-caption", "split-visual", "set-volume", "speed-visual", "mute-visual", "hide-audio", "set-ratio", "append-visual", "add-overlay", "set-transition", "import-image", "import-music"] }, }); - }); + }, 20_000); });