feat: add browser LivePortrait avatar pipeline

This commit is contained in:
haixin.yang
2026-07-11 14:36:04 +08:00
parent 32e8a0fbb4
commit e302b25288
14 changed files with 825 additions and 230 deletions
+12
View File
@@ -42,6 +42,8 @@ A browser-first AI video editor for image/video timelines, AI voiceover generati
- Detect the primary subject in an image or scan a complete video into a timestamped YOLOS tiny subject track in a browser worker.
- Generate transparent portrait cutouts with MODNet; videos are pre-analyzed into a full-duration temporal mask track used by preview and export.
- Use normalized subject geometry for smart caption avoidance and subject-aware cropping across aspect ratios.
- Generate a talking portrait from a front-facing image and the current voiceover with the browser JoyVASA audio driver and LivePortrait neural renderer.
- Choose a mixed-FP16 256px fast-preview path or a mixed-FP16 512px quality path, with adaptive neural keyframes and WebM visual-track replacement.
- Export MP4 when browser support is available, with WebM fallback.
- Show export progress during browser-side rendering.
- Persist language selection in `localStorage`.
@@ -52,6 +54,8 @@ A browser-first AI video editor for image/video timelines, AI voiceover generati
- Added installable PWA metadata, app icons, service-worker app shell/model caching, and cached sticker assets for faster repeat sessions.
- Added browser-side automatic caption groundwork using Whisper ONNX workers, plus timeline snapping/alignment improvements for captions, source audio, and generated voiceover.
- Added lazy-loaded YOLOS tiny and MODNet vision workers with subject overlays, portrait matting, caption avoidance, and preview/export-consistent smart crop geometry.
- Added an end-to-end browser talking-avatar pipeline: JoyVASA audio-to-motion ONNX, LivePortrait WebGPU rendering, adaptive 12fps neural keyframes, safe frame holding at the output frame rate, and automatic visual-track replacement.
- Moved the 906MB avatar model bundle to the revision-pinned [Timeline Studio ONNX model repository](https://huggingface.co/haixin/timeline-studio-onnx-models/tree/a201b681c8f96672b5c3f624e32d4dc932f150af), keeping large model binaries out of this Git repository.
## AI Features
@@ -66,6 +70,9 @@ Current AI capabilities are designed to run in the browser as much as possible:
- Browser voice recording for manually captured narration.
- Local waveform decoding for voiceover, source audio, and background music.
- YOLOS tiny q8 ONNX subject detection and MODNet q8 ONNX portrait matting with revision-pinned, service-worker-cached model assets.
- JoyVASA and LivePortrait ONNX talking-avatar generation with WebGPU execution, mixed-FP16 generators, GPU feature reuse, parallel model-part downloads, and explicit GPU-session isolation between the two large pipelines.
Talking-avatar model files are loaded from Hugging Face at the immutable revision `a201b681c8f96672b5c3f624e32d4dc932f150af`. Generated keyframes are checked for non-finite or temporally anomalous output; a suspect frame is retried once and, if necessary, replaced with the previous valid frame instead of being encoded as a corrupted texture.
MODNet is portrait-oriented, while YOLOS tiny covers common COCO categories. Images support the full matting path. Videos are pre-analyzed across their full duration before export, then resolve timestamped YOLOS geometry and MODNet masks for preview, caption avoidance, smart crop, and every rendered export time instead of freezing the first-frame result. Long videos automatically use a wider temporal sampling interval to keep WASM inference and memory bounded.
@@ -130,6 +137,8 @@ src/
config/
editor.js
models.js
joyVasa.js
livePortrait.js
lib/
asr.js
media.js
@@ -140,6 +149,8 @@ src/
vision.js
visualGeometry.js
workers/
joyvasa.worker.js
liveportrait.worker.js
vision.worker.js
i18n.js
main.jsx
@@ -198,6 +209,7 @@ The following are intentionally excluded from git:
- `.netlify/`
- local npm cache and machine-specific config
- local Codex/agent notes
- large avatar ONNX binaries, which are hosted in the revision-pinned Hugging Face model repository
## License
+16
View File
@@ -39,10 +39,18 @@
- 使用 YOLOS tiny 在浏览器 Worker 中识别图片主体,或扫描整段视频生成带时间戳的主体轨迹。
- 使用 MODNet 为图片生成透明人像抠图;视频会先生成覆盖全片的时序遮罩,再用于播放与导出。
- 基于归一化主体框自动做字幕避让和跨画幅智能裁切。
- 使用当前正脸图片与配音,通过浏览器 JoyVASA 音频驱动和 LivePortrait 神经渲染生成数字人口型视频。
- 数字人支持混合 FP16 256px 快速预览与 512px 高质量档、自适应神经关键帧以及自动替换画面轨。
- 浏览器支持时导出 MP4,不支持时回退到 WebM。
- 本地渲染导出时显示导出进度。
- 首次语言选择会保存到 `localStorage`
## 最近更新
- 完成 JoyVASA 音频到运动 ONNX 与 LivePortrait WebGPU 的浏览器端到端数字人链路。
- 使用自适应 1–2fps 神经关键帧、输出帧保持和画面轨自动替换,避免整张人脸交叉淡化造成重影。
- 将约 906MB 数字人模型迁移到锁定 revision 的 [Timeline Studio ONNX 模型仓库](https://huggingface.co/haixin/timeline-studio-onnx-models/tree/a201b681c8f96672b5c3f624e32d4dc932f150af),Git 仓库不再保存大模型二进制文件。
## AI 功能
当前 AI 能力尽量在浏览器端运行:
@@ -55,6 +63,9 @@
- 浏览器录音,可手动录制旁白并写入配音轨。
- 本地解析配音、视频原声、背景音乐波形。
- YOLOS tiny q8 ONNX 主体检测与 MODNet q8 ONNX 人像抠图,模型按需加载并由 Service Worker 缓存。
- JoyVASA + LivePortrait ONNX 数字人生成,包含 WebGPU 推理、混合 FP16 生成器、人物 GPU 特征复用、模型分片并行下载,以及两套大型 GPU 模型之间的显存隔离。
数字人模型固定从 Hugging Face revision `a201b681c8f96672b5c3f624e32d4dc932f150af` 加载。每个神经关键帧都会检查非有限数值和时序离群;异常帧会原位重推一次,仍异常则沿用上一正常帧,不会把损坏纹理编码进视频。
MODNet 主要面向人像抠图,YOLOS tiny 覆盖 COCO 常见类别。图片支持完整抠图;视频采用导出前全片预分析,按时间解析 YOLOS 主体轨迹与 MODNet 遮罩,因此预览、字幕避让、智能裁切和导出不再静态复用首帧。为控制浏览器内存与 WASM 推理耗时,长视频会自动降低时序采样密度并在相邻主体帧之间插值。
@@ -119,12 +130,16 @@ src/
ui.jsx
config/
editor.js
joyVasa.js
livePortrait.js
lib/
media.js
timeline.js
vision.js
visualGeometry.js
workers/
joyvasa.worker.js
liveportrait.worker.js
vision.worker.js
i18n.js
main.jsx
@@ -179,6 +194,7 @@ npx netlify-cli deploy --prod --dir=dist
- `.netlify/`
- 本地 npm 缓存和机器相关配置
- 本地 Codex/agent 工作说明
- 已迁移到固定 Hugging Face revision 的大型数字人 ONNX 文件
## License
@@ -0,0 +1,110 @@
"""Build browser-oriented LivePortrait generator variants.
The preview graph keeps the complete final SPADE residual block but runs it at
128px before the original sub-pixel RGB head. That preserves the semantic path
and produces a genuine neural 256px output while reducing the most expensive
high-resolution convolutions. Both preview and quality graphs are then converted
to mixed FP16 with FP32 public inputs/outputs for browser compatibility.
"""
from __future__ import annotations
import argparse
from copy import deepcopy
from pathlib import Path
import numpy as np
import onnx
from onnx import numpy_helper
from onnxconverter_common import float16
PREVIEW_SOURCE = "/spade_generator/up_0/Add_output_0"
SHORTCUT_NODE = "/spade_generator/up_1/conv_s/Conv"
RGB_NODE = "/spade_generator/conv_img/conv_img.0/Conv"
def ancestors(model: onnx.ModelProto, outputs: set[str]) -> list[onnx.NodeProto]:
producers = {output: node for node in model.graph.node for output in node.output}
needed: set[str] = set()
def visit(value: str) -> None:
node = producers.get(value)
if node is None or node.name in needed:
return
needed.add(node.name)
for input_name in node.input:
visit(input_name)
for output in outputs:
visit(output)
return [deepcopy(node) for node in model.graph.node if node.name in needed]
def build_preview(model: onnx.ModelProto) -> onnx.ModelProto:
preview = deepcopy(model)
# Keep the complete final SPADE block, but run it at 128px instead of
# upsampling to 256px first. The unchanged sub-pixel RGB head then produces
# a genuine 256px output. This avoids the edge-like artifacts caused by
# bypassing the semantic refinement block while cutting its spatial work 4x.
preview_scale_name = "preview_up_1_scales"
preview.graph.initializer.append(
numpy_helper.from_array(np.asarray([1, 1, 1, 1], dtype=np.float32), preview_scale_name),
)
for node in preview.graph.node:
if node.name == "/spade_generator/up_1/Resize":
node.input[2] = preview_scale_name
for index, value in enumerate(preview.graph.initializer):
if value.name == "/spade_generator/up_1/norm_s/Concat_1_output_0":
preview.graph.initializer[index].CopyFrom(
numpy_helper.from_array(
np.asarray([1, 256, 128, 128], dtype=np.int64),
value.name,
),
)
output = preview.graph.output[0]
for dimension, size in zip(output.type.tensor_type.shape.dim, [1, 3, 256, 256]):
dimension.dim_value = size
del preview.graph.value_info[:]
preview.producer_name = "Timeline Studio LivePortrait preview optimizer"
preview.producer_version = "2"
onnx.checker.check_model(preview)
return preview
def convert_fp16(model: onnx.ModelProto) -> onnx.ModelProto:
converted = float16.convert_float_to_float16(
model,
keep_io_types=True,
disable_shape_infer=False,
# Resize requires float32 roi/scales inputs in the ONNX schema. These
# numerically sensitive/reduction ops also stay in float32 while the
# convolution-heavy path and its weights use float16.
op_block_list=["GridSample", "InstanceNormalization", "ReduceSum", "Resize"],
)
converted.producer_name = "Timeline Studio LivePortrait mixed FP16 optimizer"
converted.producer_version = "1"
onnx.checker.check_model(converted)
return converted
def save(model: onnx.ModelProto, path: Path) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
onnx.save(model, path)
print(f"{path.name}: nodes={len(model.graph.node)} bytes={path.stat().st_size}")
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("input", type=Path)
parser.add_argument("output_dir", type=Path)
args = parser.parse_args()
source = onnx.load(args.input, load_external_data=True)
preview = build_preview(source)
save(preview, args.output_dir / "liveportrait-generator-preview-fp32.onnx")
save(convert_fp16(preview), args.output_dir / "liveportrait-generator-preview-fp16.onnx")
save(convert_fp16(source), args.output_dir / "liveportrait-generator-quality-fp16.onnx")
if __name__ == "__main__":
main()
+157 -99
View File
@@ -125,7 +125,33 @@ async function decodeAvatarAudio16k(blob) {
}
}
async function encodeAvatarFrames(blobs, width, height, fps) {
function runAvatarWorkerTask(worker, message, transfer, terminalType, onProgress) {
return new Promise((resolve, reject) => {
worker.onmessage = (event) => {
if (event.data?.type === "progress") {
onProgress?.(event.data);
return;
}
if (event.data?.type === "error") {
reject(new Error(event.data.message));
return;
}
if (event.data?.type === terminalType) resolve(event.data);
};
worker.onerror = (event) => reject(new Error(event.message || "Worker error"));
worker.postMessage(message, transfer);
});
}
function formatAvatarProgress(t, progress) {
const template = progress.phaseKey ? t(progress.phaseKey) : progress.phase || t("avatarGenerating");
return Object.entries(progress.phaseParams || {}).reduce(
(text, [key, value]) => text.replaceAll(`{${key}}`, String(value)),
template,
);
}
async function encodeAvatarFrames(blobs, width, height, fps, keyframeTimes = [], duration = blobs.length / fps) {
const canvas = document.createElement("canvas");
canvas.width = width;
canvas.height = height;
@@ -141,15 +167,32 @@ async function encodeAvatarFrames(blobs, width, height, fps) {
recorder.onerror = () => reject(recorder.error || new Error("数字人视频编码失败"));
});
recorder.start();
for (const blob of blobs) {
const bitmap = await createImageBitmap(blob);
context.drawImage(bitmap, 0, 0, width, height);
bitmap.close();
const bitmaps = await Promise.all(blobs.map((blob) => createImageBitmap(blob)));
const totalFrames = Math.max(1, Math.ceil(duration * fps));
for (let frame = 0; frame < totalFrames; frame += 1) {
const frameTime = frame / fps;
let nearestIndex = 0;
let nearestDistance = Number.POSITIVE_INFINITY;
for (let index = 0; index < bitmaps.length; index += 1) {
const time = keyframeTimes[index] ?? (index * duration) / Math.max(1, bitmaps.length - 1);
const distance = Math.abs(time - frameTime);
if (distance < nearestDistance) {
nearestDistance = distance;
nearestIndex = index;
}
}
// Never alpha-blend two complete portraits: even tiny head motion produces
// double eyes/mouth and the "ghost face" artifact. Adaptive keyframes make
// the held frame intervals short while preserving a single coherent face.
context.drawImage(bitmaps[nearestIndex], 0, 0, width, height);
await new Promise((resolve) => window.setTimeout(resolve, 1000 / fps));
}
bitmaps.forEach((bitmap) => bitmap.close());
recorder.stop();
const output = await stopped;
stream.getTracks().forEach((track) => track.stop());
return stopped;
if (!output.size) throw new Error("数字人视频编码结果为空");
return output;
}
function getNearestRatioIdForSize(width, height) {
@@ -363,11 +406,17 @@ export function App() {
const visionObjectUrlsRef = useRef(new Map());
const visionAbortControllerRef = useRef(null);
const visionJobGenerationRef = useRef(0);
const avatarWorkerRef = useRef(null);
const avatarMotionWorkerRef = useRef(null);
const avatarRenderWorkerRef = useRef(null);
const avatarMotionCacheRef = useRef({ audioBlob: null, motion: null });
const avatarTestImportedRef = useRef(false);
const avatarTestAudioImportedRef = useRef(false);
const activeLanguage = uiLanguage || "zh";
const t = useMemo(() => createTranslator(activeLanguage), [activeLanguage]);
useEffect(() => () => {
avatarMotionWorkerRef.current?.terminate();
avatarRenderWorkerRef.current?.terminate();
}, []);
const trOption = (name, option) => {
if (option?.kind === "stickerCategory") {
return activeLanguage !== "zh" && option.nameEn ? option.nameEn : name;
@@ -384,9 +433,15 @@ export function App() {
function openAvatarPanel() {
setAvatarPanelOpen(true);
if (!avatarMotionWorkerRef.current) {
avatarMotionWorkerRef.current = new Worker(new URL("./workers/joyvasa.worker.js", import.meta.url), { type: "module" });
}
if (!avatarRenderWorkerRef.current) {
avatarRenderWorkerRef.current = new Worker(new URL("./workers/liveportrait.worker.js", import.meta.url), { type: "module" });
}
}
async function generateAvatarAcceptanceFrame() {
async function generateAvatarAcceptanceFrame(quality = "preview") {
if (avatarJob.running) return;
if (!previewVisualSrc || previewVisualType !== "image") {
notify(t("avatarNeedsPortrait"));
@@ -405,88 +460,89 @@ export function App() {
if (!response.ok) throw new Error(`读取肖像失败(HTTP ${response.status}`);
return response.blob();
});
const sourceDuration = Math.max(MIN_VISUAL_SEGMENT_SECONDS, Math.min(4, audioDuration || imageDuration || 4));
const audioSamples = await decodeAvatarAudio16k(audioBlob);
const motionWorker = new Worker(new URL("./workers/joyvasa.worker.js", import.meta.url), { type: "module" });
avatarWorkerRef.current?.terminate();
avatarWorkerRef.current = motionWorker;
motionWorker.onmessage = (event) => {
if (event.data?.type === "progress") {
setAvatarJob({ running: true, progress: event.data.progress, phase: event.data.phase });
return;
const testDuration = import.meta.env.DEV ? Number(import.meta.env.VITE_AVATAR_TEST_DURATION || 0) : 0;
const sourceDuration = testDuration > 0
? Math.max(0.5, Math.min(4, testDuration))
: Math.max(MIN_VISUAL_SEGMENT_SECONDS, Math.min(4, audioDuration || imageDuration || 4));
let motionBuffer;
if (avatarMotionCacheRef.current.audioBlob === audioBlob && avatarMotionCacheRef.current.motion) {
motionBuffer = avatarMotionCacheRef.current.motion.slice(0);
setAvatarJob({ running: true, progress: 65, phase: t("avatarProgressReuseMotion") });
} else {
// Never allocate JoyVASA while a previous LivePortrait graph remains
// resident. Some WebGPU drivers return corrupted texture-like frames
// under that memory pressure without reporting an inference error.
if (avatarRenderWorkerRef.current) {
await runAvatarWorkerTask(
avatarRenderWorkerRef.current,
{ type: "releaseGpuSessions" },
[],
"gpuReleased",
);
}
if (event.data?.type === "error") {
console.error("LivePortrait worker error", event.data.message);
setAvatarJob({ running: false, progress: 0, phase: `${t("avatarGenerationFailed")}${event.data.message}` });
notify(`${t("avatarGenerationFailed")}${event.data.message}`);
motionWorker.terminate();
if (avatarWorkerRef.current === motionWorker) avatarWorkerRef.current = null;
return;
const audioSamples = await decodeAvatarAudio16k(audioBlob);
if (!avatarMotionWorkerRef.current) {
avatarMotionWorkerRef.current = new Worker(new URL("./workers/joyvasa.worker.js", import.meta.url), { type: "module" });
}
if (event.data?.type !== "motion") return;
motionWorker.terminate();
const renderWorker = new Worker(new URL("./workers/liveportrait.worker.js", import.meta.url), { type: "module" });
avatarWorkerRef.current = renderWorker;
renderWorker.onmessage = async (renderEvent) => {
if (renderEvent.data?.type === "progress") {
setAvatarJob({ running: true, progress: renderEvent.data.progress, phase: renderEvent.data.phase });
return;
}
if (renderEvent.data?.type === "error") {
setAvatarJob({ running: false, progress: 0, phase: `${t("avatarGenerationFailed")}${renderEvent.data.message}` });
notify(`${t("avatarGenerationFailed")}${renderEvent.data.message}`);
renderWorker.terminate();
if (avatarWorkerRef.current === renderWorker) avatarWorkerRef.current = null;
return;
}
if (renderEvent.data?.type !== "videoFrames") return;
try {
setAvatarJob({ running: true, progress: 99, phase: "编码数字人视频" });
const blob = await encodeAvatarFrames(renderEvent.data.blobs, renderEvent.data.width, renderEvent.data.height, renderEvent.data.fps);
const url = URL.createObjectURL(blob);
imageUrlRefs.current.add(url);
const asset = {
id: crypto.randomUUID(), type: "video", src: url, name: "liveportrait-joyvasa.webm",
meta: `${renderEvent.data.width} x ${renderEvent.data.height} · JoyVASA + LivePortrait Web`, blob,
duration: sourceDuration, width: renderEvent.data.width, height: renderEvent.data.height, trackFrames: [],
};
setUserAssets((assets) => [asset, ...assets]);
replaceVisualTimeline(asset, sourceDuration);
setCurrentTime(0);
setAvatarJob({ running: false, progress: 100, phase: t("avatarAcceptanceDone") });
notify(t("avatarTrackReplaced"));
} catch (error) {
setAvatarJob({ running: false, progress: 0, phase: `${t("avatarGenerationFailed")}${error.message}` });
notify(`${t("avatarGenerationFailed")}${error.message}`);
} finally {
renderWorker.terminate();
if (avatarWorkerRef.current === renderWorker) avatarWorkerRef.current = null;
}
};
renderWorker.postMessage({
type: "generateVideo", portraitBlob: sourceBlob, motionBuffer: event.data.motion,
const motionResult = await runAvatarWorkerTask(
avatarMotionWorkerRef.current,
{ type: "generate", audioSamples: audioSamples.buffer, modelBaseUrl: JOYVASA_PROJECT_MODEL_BASE_URL },
[audioSamples.buffer],
"motion",
(progress) => setAvatarJob({ running: true, progress: progress.progress, phase: formatAvatarProgress(t, progress) }),
);
avatarMotionCacheRef.current = { audioBlob, motion: motionResult.motion.slice(0) };
motionBuffer = motionResult.motion;
// Keep downloaded JoyVASA bytes warm, but release its GPU weights before
// the portrait renderer is created.
await runAvatarWorkerTask(
avatarMotionWorkerRef.current,
{ type: "release", modelBaseUrl: JOYVASA_PROJECT_MODEL_BASE_URL },
[],
"released",
);
}
if (!avatarRenderWorkerRef.current) {
avatarRenderWorkerRef.current = new Worker(new URL("./workers/liveportrait.worker.js", import.meta.url), { type: "module" });
}
const renderResult = await runAvatarWorkerTask(
avatarRenderWorkerRef.current,
{
type: "generateVideo", portraitBlob: sourceBlob, motionBuffer,
modelBaseUrl: import.meta.env.VITE_LIVE_PORTRAIT_MODEL_BASE_URL || "",
joyVasaModelBaseUrl: JOYVASA_PROJECT_MODEL_BASE_URL,
webGpuModelBaseUrl: LIVE_PORTRAIT_WEBGPU_PROJECT_MODEL_BASE_URL,
quality,
renderFps: Math.max(1, Number(import.meta.env.VITE_AVATAR_RENDER_FPS || 8)),
}, [event.data.motion]);
neuralFps: Math.max(1, Number(import.meta.env.VITE_AVATAR_NEURAL_FPS || 2)),
duration: sourceDuration,
portraitKey: previewVisualSegment?.id || previewVisualSrc,
},
[motionBuffer],
"videoFrames",
(progress) => setAvatarJob({ running: true, progress: progress.progress, phase: formatAvatarProgress(t, progress) }),
);
setAvatarJob({ running: true, progress: 99, phase: t("avatarProgressEncodeVideo") });
const blob = await encodeAvatarFrames(
renderResult.blobs,
renderResult.width,
renderResult.height,
renderResult.fps,
renderResult.keyframeTimes,
renderResult.duration,
);
const url = URL.createObjectURL(blob);
imageUrlRefs.current.add(url);
const asset = {
id: crypto.randomUUID(), type: "video", src: url, name: "liveportrait-joyvasa.webm",
meta: `${renderResult.width} x ${renderResult.height} · JoyVASA + LivePortrait FP16 WebGPU`, blob,
duration: sourceDuration, width: renderResult.width, height: renderResult.height, trackFrames: [],
};
motionWorker.onerror = (event) => {
console.error("LivePortrait worker module error", event.message, event);
setAvatarJob({ running: false, progress: 0, phase: `${t("avatarGenerationFailed")}${event.message || "Worker error"}` });
notify(`${t("avatarGenerationFailed")}${event.message || "Worker error"}`);
motionWorker.terminate();
if (avatarWorkerRef.current === motionWorker) avatarWorkerRef.current = null;
};
motionWorker.postMessage({
type: "generate",
audioSamples: audioSamples.buffer,
modelBaseUrl: JOYVASA_PROJECT_MODEL_BASE_URL,
}, [audioSamples.buffer]);
setUserAssets((assets) => [asset, ...assets]);
replaceVisualTimeline(asset, sourceDuration);
setCurrentTime(0);
setAvatarJob({ running: false, progress: 100, phase: t("avatarAcceptanceDone") });
notify(t("avatarTrackReplaced"));
} catch (error) {
setAvatarJob({ running: false, progress: 0, phase: "" });
notify(`${t("avatarGenerationFailed")}${error instanceof Error ? error.message : String(error)}`);
@@ -2572,7 +2628,9 @@ export function App() {
try {
preparedText = prepareTextForVoice(rawText, selectedVoice);
} catch (error) {
const message = error instanceof Error ? error.message : "当前文案不适合所选语音";
const message = error instanceof TtsInputError
? t(error.code)
: error instanceof Error ? error.message : t("ttsErrorVoiceMismatch");
setStatus("error");
setStatusText(message);
setProgress(0);
@@ -2582,10 +2640,10 @@ export function App() {
setVoiceTab("synthesis");
setStatus("generating");
setStatusText("准备本地模型");
setStatusText(t("ttsStatusPreparingModel"));
setProgress(6);
if (preparedText.warning) {
notify(preparedText.warning);
if (preparedText.warningKey) {
notify(t(preparedText.warningKey));
}
try {
@@ -2595,9 +2653,9 @@ export function App() {
const tts = await import("@diffusionstudio/vits-web");
const cacheWasCleared = await clearPiperCacheIfStorageTight(tts);
if (cacheWasCleared) {
notify("浏览器模型缓存空间紧张,已清理 Piper 缓存后继续生成。");
notify(t("ttsNoticePiperCacheCleared"));
}
setStatusText("下载或读取中文 ONNX 模型");
setStatusText(t("ttsStatusLoadingChineseModel"));
const progressCallback = (event) => {
if (event?.total) {
const nextProgress = Math.round((event.loaded / event.total) * 76);
@@ -2616,13 +2674,13 @@ export function App() {
throw error;
}
setStatusText("模型缓存空间不足,正在清理后重试");
setStatusText(t("ttsStatusClearingCache"));
await tts.flush?.();
blob = await tts.predict(piperInput, progressCallback);
}
} else {
const { KokoroTTS } = await import("kokoro-js");
setStatusText("加载 Kokoro 82M q8");
setStatusText(t("ttsStatusLoadingKokoro"));
const tts = await KokoroTTS.from_pretrained(MODEL_ID, {
dtype: "q8",
device: "wasm",
@@ -2634,7 +2692,7 @@ export function App() {
}
},
});
setStatusText("生成英文配音");
setStatusText(t("ttsStatusGeneratingEnglish"));
const audio = await tts.generate(preparedText.text, {
voice: selectedVoice.id,
speed,
@@ -2642,21 +2700,21 @@ export function App() {
blob = audio.toBlob();
}
setStatusText("解析音频波形");
await commitAudio(blob, `${selectedVoice.name} 已生成`);
notify("配音已生成并写入时间线");
setStatusText(t("ttsStatusDecodingWaveform"));
await commitAudio(blob, `${selectedVoice.name} · ${t("ttsGenerated")}`);
notify(t("ttsNoticeGenerated"));
} catch (error) {
console.error(error);
const message =
error instanceof TtsInputError
? error.message
? t(error.code)
: selectedVoice.engine === "piper" && isPiperSymbolError(error)
? "当前中文语音模型不支持这段文案里的部分字符,请清理英文/特殊符号,或切换 English 声音。"
? t("ttsErrorUnsupportedPiperSymbols")
: isStorageQuotaError(error)
? "浏览器模型缓存空间不足,请在设置里清理模型缓存后重试。"
? t("ttsErrorStorageQuota")
: error instanceof Error
? error.message
: "生成失败,请重试";
: t("ttsErrorGenerationFailed");
setStatus("error");
setStatusText(message);
setProgress(0);
+5 -5
View File
@@ -60,28 +60,28 @@ export function Topbar({
<div className="project-title">{t("projectTitle")}</div>
<div className="menu-anchor">
<button className="project-file-button" type="button" onClick={() => setShowFileMenu((open) => !open)}>
文件 <CaretDown size={13} />
{t("fileMenu")} <CaretDown size={13} />
</button>
{showFileMenu ? (
<Popover className="project-file-popover" onClose={() => setShowFileMenu(false)}>
<div className="file-menu-card">
<div className="file-menu-heading">
<span>工程</span>
<span>{t("projectMenuHeading")}</span>
<small>Timeline Studio</small>
</div>
<button className="file-menu-action file-menu-new" type="button" onClick={handleNewProject}>
<span className="file-menu-icon"><FilePlus size={17} /></span>
<span className="file-menu-copy"><strong>新建工程</strong><small>从空白时间线开始</small></span>
<span className="file-menu-copy"><strong>{t("newProject")}</strong><small>{t("newProjectHint")}</small></span>
</button>
<div className="file-menu-divider" />
<button className="file-menu-action" type="button" onClick={() => handleImportProject()}>
<span className="file-menu-icon"><FileArrowUp size={17} /></span>
<span className="file-menu-copy"><strong>导入工程包</strong><small>恢复时间线与全部媒体</small></span>
<span className="file-menu-copy"><strong>{t("importProject")}</strong><small>{t("importProjectHint")}</small></span>
<span className="file-menu-format">.timeline</span>
</button>
<button className="file-menu-action is-primary" type="button" onClick={handleExportProject}>
<span className="file-menu-icon"><FileArrowDown size={17} /></span>
<span className="file-menu-copy"><strong>导出工程包</strong><small>打包图片视频和音频</small></span>
<span className="file-menu-copy"><strong>{t("exportProject")}</strong><small>{t("exportProjectHint")}</small></span>
<span className="file-menu-format">.timeline</span>
</button>
</div>
+10 -1
View File
@@ -143,6 +143,7 @@ function AvatarContextPanel({ t, hasVisual, visualType, audioBlob, audioDuration
const hasPortrait = hasVisual && visualType === "image";
const [probeState, setProbeState] = useState("idle");
const [probeResult, setProbeResult] = useState(null);
const [avatarQuality, setAvatarQuality] = useState("preview");
const runProbe = async () => {
setProbeState("running");
@@ -167,6 +168,14 @@ function AvatarContextPanel({ t, hasVisual, visualType, audioBlob, audioDuration
<div className={captionSegments.length ? "is-ready" : ""}><ClosedCaptioning size={17} weight="duotone" /><span><strong>{t("avatarLipSyncSource")}</strong><em>{captionSegments.length ? `${captionSegments.length} ${t("captionSegmentsUnit")} · ${t("avatarCaptionSync")}` : t("avatarNeedsCaptions")}</em></span></div>
</div>
<div className="avatar-sync-mode"><span>{t("avatarModelSource")}</span><strong>{LIVE_PORTRAIT_WEB_MODEL.id}</strong></div>
<div className="avatar-quality-picker" aria-label={t("avatarQuality")}>
<button type="button" className={avatarQuality === "preview" ? "is-active" : ""} onClick={() => setAvatarQuality("preview")}>
<strong>{t("avatarQualityPreview")}</strong><em>{t("avatarQualityPreviewHint")}</em>
</button>
<button type="button" className={avatarQuality === "quality" ? "is-active" : ""} onClick={() => setAvatarQuality("quality")}>
<strong>{t("avatarQualityFull")}</strong><em>{t("avatarQualityFullHint")}</em>
</button>
</div>
<p className="avatar-context-note">{t("avatarGenerationNote")}</p>
<div className="avatar-porting-stages" aria-label={t("avatarPortingStatus")}>
<div className="is-done"><span>1</span><strong>{t("avatarStagePinned")}</strong></div>
@@ -191,7 +200,7 @@ function AvatarContextPanel({ t, hasVisual, visualType, audioBlob, audioDuration
className="panel-primary avatar-generate-button"
type="button"
disabled={!hasPortrait || avatarJob?.running}
onClick={generateAvatarAcceptanceFrame}
onClick={() => generateAvatarAcceptanceFrame(avatarQuality)}
>
<PersonSimpleRun size={17} weight="duotone" />
{avatarJob?.running ? t("avatarGenerating") : t("avatarGenerate")}
+3 -1
View File
@@ -1,4 +1,5 @@
const JOYVASA_REVISION = "b8f13fe9c23679c56f21b1baafb92ed00dc087c3";
const TIMELINE_STUDIO_MODEL_REVISION = "a201b681c8f96672b5c3f624e32d4dc932f150af";
export const JOYVASA_WEB_MODEL = Object.freeze({
id: "jdh-algo/JoyVASA",
@@ -31,4 +32,5 @@ export const JOYVASA_WEB_MODEL = Object.freeze({
runtime: Object.freeze({ sampleRate: 16_000, windowSamples: 64_000, paddedSamples: 64_080, fps: 25, frames: 100, diffusionSteps: 50 }),
});
export const JOYVASA_PROJECT_MODEL_BASE_URL = "/models/joyvasa/";
export const JOYVASA_PROJECT_MODEL_BASE_URL =
`https://huggingface.co/haixin/timeline-studio-onnx-models/resolve/${TIMELINE_STUDIO_MODEL_REVISION}/joyvasa/`;
+23 -14
View File
@@ -1,4 +1,5 @@
const LIVE_PORTRAIT_REVISION = "e6c5d2407593a39f29c92ffd5ea3eaf5e59d52a1";
const TIMELINE_STUDIO_MODEL_REVISION = "a201b681c8f96672b5c3f624e32d4dc932f150af";
export const LIVE_PORTRAIT_WEB_MODEL = Object.freeze({
id: "dyicnc/Live-Portrait-ONNX",
@@ -16,16 +17,19 @@ export const LIVE_PORTRAIT_WEB_MODEL = Object.freeze({
stitching: "stitching.onnx",
stitchingLip: "stitching_lip.onnx",
stitchingRetargeting: "stitching_retargeting.onnx",
generatorWebGpu: Object.freeze([
"liveportrait-generator-webgpu.onnx.part-aa?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-ab?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-ac?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-ad?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-ae?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-af?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-ag?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-ah?v=03defe3d",
"liveportrait-generator-webgpu.onnx.part-ai?v=03defe3d",
generatorPreviewFp16: Object.freeze([
"liveportrait-generator-preview-fp16.onnx.part-aa?v=5fdb50d2",
"liveportrait-generator-preview-fp16.onnx.part-ab?v=5fdb50d2",
"liveportrait-generator-preview-fp16.onnx.part-ac?v=5fdb50d2",
"liveportrait-generator-preview-fp16.onnx.part-ad?v=5fdb50d2",
"liveportrait-generator-preview-fp16.onnx.part-ae?v=5fdb50d2",
]),
generatorQualityFp16: Object.freeze([
"liveportrait-generator-quality-fp16.onnx.part-aa?v=1b4630bf",
"liveportrait-generator-quality-fp16.onnx.part-ab?v=1b4630bf",
"liveportrait-generator-quality-fp16.onnx.part-ac?v=1b4630bf",
"liveportrait-generator-quality-fp16.onnx.part-ad?v=1b4630bf",
"liveportrait-generator-quality-fp16.onnx.part-ae?v=1b4630bf",
]),
appearanceFeatureExtractorWebGpu: "liveportrait-appearance_feature_extractor.onnx",
motionExtractorWebGpu: Object.freeze([
@@ -68,9 +72,13 @@ export const LIVE_PORTRAIT_WEB_MODEL = Object.freeze({
bytes: 150_609,
sha256: "33489d795915b78a8e96787c42c367cac23a0d5d3d2bd3efbb4af5ee758d42bb",
}),
generatorWebGpu: Object.freeze({
bytes: 421_246_756,
sha256: "03defe3d3a391a897ae4ed4059d19ff12c9a3f46f9c605471c365775388bc551",
generatorPreviewFp16: Object.freeze({
bytes: 210_713_705,
sha256: "5fdb50d2fdaf1d52a65f39dddf7b79c968725eaae735418f5494831ba4d45706",
}),
generatorQualityFp16: Object.freeze({
bytes: 210_713_678,
sha256: "1b4630bfbe499dd1d28697fa1e479ab4b305c80421aa485bfd04d36698c6fe7f",
}),
appearanceFeatureExtractorWebGpu: Object.freeze({
bytes: 3_355_896,
@@ -87,7 +95,8 @@ export const LIVE_PORTRAIT_WEB_MODEL = Object.freeze({
}),
});
export const LIVE_PORTRAIT_WEBGPU_PROJECT_MODEL_BASE_URL = "/models/liveportrait-webgpu/";
export const LIVE_PORTRAIT_WEBGPU_PROJECT_MODEL_BASE_URL =
`https://huggingface.co/haixin/timeline-studio-onnx-models/resolve/${TIMELINE_STUDIO_MODEL_REVISION}/liveportrait-webgpu/`;
export function getLivePortraitModelUrl(file) {
return `https://huggingface.co/${LIVE_PORTRAIT_WEB_MODEL.id}/resolve/${LIVE_PORTRAIT_WEB_MODEL.revision}/${file}`;
+110
View File
@@ -178,8 +178,63 @@ export const UI_COPY = {
avatarSyncModeCaption: "字幕 + 配音精确对齐",
avatarGenerationNote: "模型已随项目按 50MB 分片托管并由服务工作线程缓存;首次运行需要下载,后续直接复用本地缓存。",
avatarGenerate: "生成口型视频",
avatarQuality: "生成质量",
avatarQualityPreview: "快速预览",
avatarQualityPreviewHint: "256px · FP16 · 自适应 12→8fps",
avatarQualityFull: "高质量",
avatarQualityFullHint: "512px · FP16 · 自适应 12→8fps",
avatarGenerating: "正在生成数字人视频",
avatarGenerationHint: "首次运行会下载模型;请保持此页面打开。",
avatarProgressDownloadFile: "下载 {file}",
avatarProgressDownloadModel: "下载 {model}",
avatarProgressInitHubert: "初始化 JoyVASA HuBERT WebGPU",
avatarProgressInitMotion: "初始化 JoyVASA 运动 WebGPU",
avatarProgressInitModel: "初始化 {model}",
avatarProgressMotionStep: "生成口型运动 {current}/{total}",
avatarProgressPrepareAudio: "准备真实音频驱动",
avatarProgressAudioReady: "音频特征完成",
avatarProgressMotionReady: "真实音频运动生成完成",
avatarProgressReuseMotion: "复用当前配音口型运动",
avatarProgressPreparePortraitMotion: "准备肖像与真实口型运动",
avatarProgressExtract3d: "提取人物 3D 特征",
avatarProgressReuseGpuFeature: "复用人物 GPU 特征",
avatarProgressKeyframe: "FP16 WebGPU 关键帧 {current}/{total} · {seconds}s",
avatarProgressFrameEncoded: "完成帧 {current}/{total} · 编码 {seconds}s",
avatarProgressRetryCorruptFrame: "检测到异常关键帧 {current}/{total},正在原位重推(第 {attempt} 次)",
avatarProgressDroppedCorruptFrame: "异常关键帧 {current}/{total} 已丢弃并沿用上一正常帧",
avatarProgressInterpolate: "关键帧生成完成,准备运动插帧",
avatarProgressExtractFeature: "提取人物特征",
avatarProgressWarp3d: "计算 3D 形变(可能需要约 1 分钟)",
avatarProgressRenderPortrait: "渲染 512×512 人像",
avatarProgressEncodeVideo: "编码数字人视频",
fileMenu: "文件",
projectMenuHeading: "工程",
newProject: "新建工程",
newProjectHint: "从空白时间线开始",
importProject: "导入工程包",
importProjectHint: "恢复时间线与全部媒体",
exportProject: "导出工程包",
exportProjectHint: "打包图片、视频和音频",
ttsWarningChineseSymbolsCleaned: "已自动清理中文语音不支持的符号,避免本地 ONNX 运行失败。",
ttsWarningEnglishCharactersCleaned: "已自动清理英文语音不支持的字符。",
ttsErrorChineseVoiceEnglishText: "当前中文语音不适合大量英文文案,请切换 English 声音后再生成。",
ttsErrorNoChineseContent: "当前中文语音没有可朗读的中文内容,请输入中文或切换英文声音。",
ttsErrorEnglishVoiceChineseText: "当前英文语音不适合中文文案,请切换中文声音后再生成。",
ttsErrorNoEnglishContent: "当前英文语音没有可朗读的英文内容,请输入英文或切换中文声音。",
ttsErrorEmptyScript: "请输入要生成的文案。",
ttsErrorVoiceMismatch: "当前文案不适合所选语音。",
ttsStatusPreparingModel: "准备本地模型",
ttsStatusLoadingChineseModel: "下载或读取中文 ONNX 模型",
ttsStatusClearingCache: "模型缓存空间不足,正在清理后重试",
ttsStatusLoadingKokoro: "加载 Kokoro 82M q8",
ttsStatusGeneratingEnglish: "生成英文配音",
ttsStatusDecodingWaveform: "解析音频波形",
ttsNoticePiperCacheCleared: "浏览器模型缓存空间紧张,已清理 Piper 缓存后继续生成。",
ttsGenerated: "已生成",
ttsNoticeGenerated: "配音已生成并写入时间线",
ttsErrorUnsupportedPiperSymbols: "当前中文语音模型不支持这段文案里的部分字符,请清理英文/特殊符号,或切换 English 声音。",
ttsErrorStorageQuota: "浏览器模型缓存空间不足,请在设置里清理模型缓存后重试。",
ttsErrorGenerationFailed: "生成失败,请重试",
top: "顶部",
middle: "居中",
bottom: "底部",
@@ -422,8 +477,63 @@ export const UI_COPY = {
avatarSyncModeCaption: "Captions + voiceover alignment",
avatarGenerationNote: "Models ship with the project in 50 MB chunks and are cached by the service worker. First use downloads them; later runs reuse the local cache.",
avatarGenerate: "Generate lip-sync video",
avatarQuality: "Render quality",
avatarQualityPreview: "Fast preview",
avatarQualityPreviewHint: "256px · FP16 · adaptive 12→8fps",
avatarQualityFull: "High quality",
avatarQualityFullHint: "512px · FP16 · adaptive 12→8fps",
avatarGenerating: "Generating avatar video",
avatarGenerationHint: "First use downloads models. Keep this page open.",
avatarProgressDownloadFile: "Downloading {file}",
avatarProgressDownloadModel: "Downloading {model}",
avatarProgressInitHubert: "Initializing JoyVASA HuBERT WebGPU",
avatarProgressInitMotion: "Initializing JoyVASA motion WebGPU",
avatarProgressInitModel: "Initializing {model}",
avatarProgressMotionStep: "Generating lip motion {current}/{total}",
avatarProgressPrepareAudio: "Preparing audio-driven motion",
avatarProgressAudioReady: "Audio features ready",
avatarProgressMotionReady: "Audio-driven motion ready",
avatarProgressReuseMotion: "Reusing motion from the current voiceover",
avatarProgressPreparePortraitMotion: "Preparing portrait and lip motion",
avatarProgressExtract3d: "Extracting 3D portrait features",
avatarProgressReuseGpuFeature: "Reusing portrait GPU features",
avatarProgressKeyframe: "FP16 WebGPU keyframe {current}/{total} · {seconds}s",
avatarProgressFrameEncoded: "Frame {current}/{total} complete · encoded in {seconds}s",
avatarProgressRetryCorruptFrame: "Corrupt keyframe {current}/{total} detected; retrying in place (attempt {attempt})",
avatarProgressDroppedCorruptFrame: "Corrupt keyframe {current}/{total} dropped; holding the previous valid frame",
avatarProgressInterpolate: "Keyframes ready · preparing motion interpolation",
avatarProgressExtractFeature: "Extracting portrait features",
avatarProgressWarp3d: "Computing 3D warping (may take about 1 minute)",
avatarProgressRenderPortrait: "Rendering 512×512 portrait",
avatarProgressEncodeVideo: "Encoding talking-avatar video",
fileMenu: "File",
projectMenuHeading: "Project",
newProject: "New project",
newProjectHint: "Start with a blank timeline",
importProject: "Import project package",
importProjectHint: "Restore the timeline and all media",
exportProject: "Export project package",
exportProjectHint: "Bundle images, video, and audio",
ttsWarningChineseSymbolsCleaned: "Unsupported symbols were cleaned automatically to prevent the local ONNX voice model from failing.",
ttsWarningEnglishCharactersCleaned: "Unsupported characters were cleaned automatically for the English voice.",
ttsErrorChineseVoiceEnglishText: "The selected Chinese voice is not suitable for mostly English text. Switch to an English voice and try again.",
ttsErrorNoChineseContent: "No readable Chinese content remains. Enter Chinese text or switch to an English voice.",
ttsErrorEnglishVoiceChineseText: "The selected English voice is not suitable for Chinese text. Switch to a Chinese voice and try again.",
ttsErrorNoEnglishContent: "No readable English content remains. Enter English text or switch to a Chinese voice.",
ttsErrorEmptyScript: "Enter a script to generate voiceover.",
ttsErrorVoiceMismatch: "The script is not suitable for the selected voice.",
ttsStatusPreparingModel: "Preparing local model",
ttsStatusLoadingChineseModel: "Downloading or loading the Chinese ONNX model",
ttsStatusClearingCache: "Model cache is full · clearing it before retrying",
ttsStatusLoadingKokoro: "Loading Kokoro 82M q8",
ttsStatusGeneratingEnglish: "Generating English voiceover",
ttsStatusDecodingWaveform: "Analyzing audio waveform",
ttsNoticePiperCacheCleared: "Browser model storage was low. The Piper cache was cleared and generation will continue.",
ttsGenerated: "Generated",
ttsNoticeGenerated: "Voiceover generated and added to the timeline.",
ttsErrorUnsupportedPiperSymbols: "The Chinese voice model does not support some characters in this script. Remove English or special symbols, or switch to an English voice.",
ttsErrorStorageQuota: "Browser model storage is full. Clear the model cache in Settings and try again.",
ttsErrorGenerationFailed: "Generation failed. Try again.",
top: "Top",
middle: "Middle",
bottom: "Bottom",
+4 -1
View File
@@ -1,6 +1,9 @@
import { LIVE_PORTRAIT_WEB_MODEL } from "../config/livePortrait.js";
const REQUIRED_GENERATOR_BYTES = LIVE_PORTRAIT_WEB_MODEL.knownArtifacts.generator.bytes;
const REQUIRED_GENERATOR_BYTES = Math.max(
LIVE_PORTRAIT_WEB_MODEL.knownArtifacts.generatorPreviewFp16.bytes,
LIVE_PORTRAIT_WEB_MODEL.knownArtifacts.generatorQualityFp16.bytes,
);
function makeCheck(id, state, detail) {
return { id, state, detail };
+11 -10
View File
@@ -50,9 +50,10 @@ const ASCII_PUNCTUATION_MAP = new Map([
]);
export class TtsInputError extends Error {
constructor(message) {
super(message);
constructor(code) {
super(code);
this.name = "TtsInputError";
this.code = code;
}
}
@@ -81,7 +82,7 @@ function prepareChinesePiperText(rawText) {
const hanCount = countMatches(normalized, /\p{Script=Han}/gu);
if (latinCount > Math.max(14, hanCount * 0.8)) {
throw new TtsInputError("当前中文语音不适合大量英文文案,请切换 English 声音后再生成。");
throw new TtsInputError("ttsErrorChineseVoiceEnglishText");
}
let changed = false;
@@ -113,12 +114,12 @@ function prepareChinesePiperText(rawText) {
.trim();
if (!text) {
throw new TtsInputError("当前中文语音没有可朗读的中文内容,请输入中文或切换英文声音。");
throw new TtsInputError("ttsErrorNoChineseContent");
}
return {
text,
warning: changed ? "已自动清理中文语音不支持的符号,避免本地 ONNX 运行失败。" : "",
warningKey: changed ? "ttsWarningChineseSymbolsCleaned" : "",
};
}
@@ -128,17 +129,17 @@ function prepareKokoroText(rawText) {
const latinCount = countMatches(normalized, /[A-Za-z]/g);
if (hanCount > Math.max(4, latinCount * 0.5)) {
throw new TtsInputError("当前英文语音不适合中文文案,请切换中文声音后再生成。");
throw new TtsInputError("ttsErrorEnglishVoiceChineseText");
}
const text = normalized.replace(/[^\p{Script=Latin}0-9\s.,!?;:'"()\-]/gu, "").trim();
if (!text) {
throw new TtsInputError("当前英文语音没有可朗读的英文内容,请输入英文或切换中文声音。");
throw new TtsInputError("ttsErrorNoEnglishContent");
}
return {
text,
warning: text !== normalized ? "已自动清理英文语音不支持的字符。" : "",
warningKey: text !== normalized ? "ttsWarningEnglishCharactersCleaned" : "",
};
}
@@ -153,9 +154,9 @@ export function prepareTextForVoice(rawText, voice) {
const text = normalizeBaseText(rawText);
if (!text) {
throw new TtsInputError("请输入要生成的文案。");
throw new TtsInputError("ttsErrorEmptyScript");
}
return { text, warning: "" };
return { text, warningKey: "" };
}
export function isPiperSymbolError(error) {
+13 -5
View File
@@ -422,7 +422,7 @@ button:disabled {
position: relative;
z-index: 1;
display: grid;
grid-template-columns: 66px minmax(250px, 0.66fr) minmax(520px, 1.62fr) minmax(300px, 0.78fr);
grid-template-columns: 90px minmax(250px, 0.66fr) minmax(520px, 1.62fr) minmax(300px, 0.78fr);
grid-template-rows: minmax(0, 1fr);
gap: 6px;
min-height: 0;
@@ -655,7 +655,7 @@ button:disabled {
gap: 9px;
min-height: 0;
max-height: 100%;
padding: 12px 7px;
padding: 12px 6px;
overflow-x: hidden;
overflow-y: auto;
overscroll-behavior: contain;
@@ -696,18 +696,25 @@ button:disabled {
place-items: center;
flex: 0 0 auto;
gap: 5px;
width: 52px;
width: 100%;
min-height: 58px;
border: 0;
border-radius: 7px;
padding: 0;
color: #9ca7b2;
background: transparent;
cursor: pointer;
}
.rail-tool span {
font-size: 12px;
width: 100%;
overflow: hidden;
font-size: 11.5px;
font-weight: 600;
line-height: 1.2;
text-align: center;
text-overflow: ellipsis;
white-space: nowrap;
}
.rail-tool:hover,
@@ -1493,6 +1500,7 @@ button:disabled {
.avatar-input-list > div.is-ready { border-color: rgba(53,234,217,.2); color: #61dace; background: rgba(53,234,217,.045); }
.avatar-input-list span { display: grid; gap: 3px; }.avatar-input-list strong { color: #dfe9ef; font-size: 11px; }.avatar-input-list em { color: #778592; font-size: 9px; font-style: normal; }
.avatar-sync-mode { display: flex; justify-content: space-between; align-items: center; border-radius: 8px; padding: 10px; color: #93a2b0; background: rgba(255,255,255,.035); font-size: 10px; }.avatar-sync-mode strong { color: #77e5da; font-size: 10px; }.avatar-context-note { margin: 0; color: #778592; font-size: 10px; line-height: 1.55; }.avatar-generate-button { display: flex; align-items: center; justify-content: center; gap: 7px; min-height: 42px; width: 100%; }
.avatar-quality-picker { display: grid; grid-template-columns: 1fr 1fr; gap: 7px; }.avatar-quality-picker button { display: grid; gap: 3px; min-width: 0; border: 1px solid rgba(255,255,255,.08); border-radius: 8px; padding: 9px; text-align: left; color: #9aa8b8; background: rgba(255,255,255,.025); cursor: pointer; }.avatar-quality-picker button.is-active { border-color: rgba(53,234,217,.32); background: rgba(53,234,217,.075); box-shadow: inset 0 0 0 1px rgba(53,234,217,.06); }.avatar-quality-picker strong { color: #dfe9ef; font-size: 10px; }.avatar-quality-picker button.is-active strong { color: #82e9df; }.avatar-quality-picker em { overflow: hidden; color: #71808e; font-size: 8px; font-style: normal; line-height: 1.35; text-overflow: ellipsis; }
.avatar-official-notice { display: grid; grid-template-columns: 26px minmax(0, 1fr); gap: 8px; align-items: start; border: 1px solid rgba(108, 137, 255, .24); border-radius: 8px; padding: 10px; color: #9bb6ff; background: rgba(93, 122, 240, .08); }.avatar-official-notice > span { display: grid; gap: 3px; }.avatar-official-notice strong { color: #dce7ff; font-size: 11px; }.avatar-official-notice em { color: #94a2b4; font-size: 9px; font-style: normal; line-height: 1.45; }
.avatar-porting-stages { display: grid; gap: 7px; }
.avatar-porting-stages > div { display: grid; grid-template-columns: 22px minmax(0, 1fr); align-items: center; gap: 8px; color: #6f7c8a; font-size: 10px; }
@@ -4073,7 +4081,7 @@ button:disabled {
}
.editor-grid {
grid-template-columns: 58px minmax(0, 1fr);
grid-template-columns: 90px minmax(0, 1fr);
grid-template-areas:
"rail preview"
"rail media"
+79 -31
View File
@@ -9,9 +9,11 @@ ort.env.wasm.simd = true;
ort.env.wasm.wasmPaths = { mjs: ortWasmMjsUrl, wasm: ortWasmUrl };
const tensor = (data, dims) => new ort.Tensor("float32", data, dims);
const runtimePromises = new Map();
const artifactPromises = new Map();
function progress(value, phase) {
self.postMessage({ type: "progress", progress: Math.round(value), phase });
function progress(value, phaseKey, phaseParams = {}) {
self.postMessage({ type: "progress", progress: Math.round(value), phaseKey, phaseParams });
}
function modelUrl(file, baseUrl) {
@@ -24,16 +26,15 @@ async function fetchArtifact(key, baseUrl, start, span) {
const files = Array.isArray(configuredFiles) ? configuredFiles : [configuredFiles];
const total = JOYVASA_WEB_MODEL.knownArtifacts[key].bytes;
let loaded = 0;
const parts = [];
for (const file of files) {
const parts = await Promise.all(files.map(async (file) => {
const response = await fetch(modelUrl(file, baseUrl));
if (!response.ok) throw new Error(`${file} 下载失败(HTTP ${response.status}`);
const reader = response.body?.getReader();
if (!reader) {
const part = new Uint8Array(await response.arrayBuffer());
parts.push(part);
loaded += part.byteLength;
continue;
progress(start + Math.min(1, loaded / total) * span, "avatarProgressDownloadFile", { file });
return part;
}
const chunks = [];
let partBytes = 0;
@@ -43,7 +44,7 @@ async function fetchArtifact(key, baseUrl, start, span) {
chunks.push(value);
partBytes += value.byteLength;
loaded += value.byteLength;
progress(start + Math.min(1, loaded / total) * span, `下载 ${file}`);
progress(start + Math.min(1, loaded / total) * span, "avatarProgressDownloadFile", { file });
}
const part = new Uint8Array(partBytes);
let offset = 0;
@@ -51,8 +52,8 @@ async function fetchArtifact(key, baseUrl, start, span) {
part.set(chunk, offset);
offset += chunk.byteLength;
}
parts.push(part);
}
return part;
}));
const combined = new Uint8Array(loaded);
let offset = 0;
for (const part of parts) {
@@ -70,6 +71,41 @@ async function createSession(key, bytes) {
});
}
function getRuntime(modelBaseUrl) {
if (runtimePromises.has(modelBaseUrl)) return runtimePromises.get(modelBaseUrl);
const promise = (async () => {
let artifacts = artifactPromises.get(modelBaseUrl);
if (!artifacts) {
artifacts = Promise.all([
fetchArtifact("audio", modelBaseUrl, 2, 47),
fetchArtifact("denoiser", modelBaseUrl, 49, 12),
fetchArtifact("conditioning", modelBaseUrl, 61, 1),
fetchArtifact("schedule", modelBaseUrl, 62, 1),
]).catch((error) => {
artifactPromises.delete(modelBaseUrl);
throw error;
});
artifactPromises.set(modelBaseUrl, artifacts);
}
const [audioBytes, denoiserBytes, conditioningBytes, scheduleBytes] = await artifacts;
progress(63, "avatarProgressInitHubert");
const audioSession = await createSession("audio", audioBytes);
progress(65, "avatarProgressInitMotion");
const denoiserSession = await createSession("denoiser", denoiserBytes);
return {
audioSession,
denoiserSession,
conditioning: new Float32Array(conditioningBytes),
schedule: new Float32Array(scheduleBytes),
};
})().catch((error) => {
runtimePromises.delete(modelBaseUrl);
throw error;
});
runtimePromises.set(modelBaseUrl, promise);
return promise;
}
function reflectPadOnce(input, amount) {
const output = new Float32Array(input.length + amount * 2);
output.set(input, amount);
@@ -168,41 +204,53 @@ async function sampleMotion(denoiser, audioFeatures, conditioning, schedule) {
const next = new Float32Array(motion.length);
for (let i = 0; i < next.length; i += 1) next[i] = c0 * motion[i] + c1 * target[i] + (noise ? sigma * noise[i] : 0);
motion = next;
progress(66 + ((51 - step) / 50) * 32, `生成口型运动 ${51 - step}/50`);
progress(66 + ((51 - step) / 50) * 32, "avatarProgressMotionStep", { current: 51 - step, total: 50 });
}
return motion;
}
async function generate({ audioSamples, modelBaseUrl }) {
progress(1, "准备真实音频驱动");
const [audioBytes, denoiserBytes, conditioningBytes, scheduleBytes] = await Promise.all([
fetchArtifact("audio", modelBaseUrl, 2, 47),
fetchArtifact("denoiser", modelBaseUrl, 49, 12),
fetchArtifact("conditioning", modelBaseUrl, 61, 1),
fetchArtifact("schedule", modelBaseUrl, 62, 1),
]);
progress(63, "初始化 JoyVASA HuBERT WebGPU");
const audioSession = await createSession("audio", audioBytes);
progress(1, "avatarProgressPrepareAudio");
const runtime = await getRuntime(modelBaseUrl);
const window = prepareWindow(new Float32Array(audioSamples));
const audioResult = await audioSession.run({ audio_padded: tensor(window, [1, 64_080]) });
await audioSession.release();
progress(65, "初始化 JoyVASA 运动 WebGPU");
const denoiserSession = await createSession("denoiser", denoiserBytes);
progress(66, "音频特征完成");
const audioResult = await runtime.audioSession.run({ audio_padded: tensor(window, [1, 64_080]) });
progress(66, "avatarProgressAudioReady");
const motion = await sampleMotion(
denoiserSession,
runtime.denoiserSession,
audioResult.audio_features.data,
new Float32Array(conditioningBytes),
new Float32Array(scheduleBytes),
runtime.conditioning,
runtime.schedule,
);
await denoiserSession.release();
progress(100, "真实音频运动生成完成");
progress(100, "avatarProgressMotionReady");
self.postMessage({ type: "motion", motion: motion.buffer, frames: 100, fps: 25 }, [motion.buffer]);
}
async function prepare({ modelBaseUrl }) {
await getRuntime(modelBaseUrl);
self.postMessage({ type: "prepared" });
}
async function release({ modelBaseUrl }) {
const promise = runtimePromises.get(modelBaseUrl);
if (promise) {
const runtime = await promise;
runtime.audioSession.release();
runtime.denoiserSession.release();
runtimePromises.delete(modelBaseUrl);
}
self.postMessage({ type: "released" });
}
self.onmessage = (event) => {
if (event.data?.type !== "generate") return;
generate(event.data).catch((error) => {
const task = event.data?.type === "generate"
? generate
: event.data?.type === "prepare"
? prepare
: event.data?.type === "release"
? release
: null;
if (!task) return;
task(event.data).catch((error) => {
self.postMessage({ type: "error", message: error instanceof Error ? error.message : String(error) });
});
};
+272 -63
View File
@@ -9,9 +9,13 @@ ort.env.wasm.simd = true;
ort.env.wasm.wasmPaths = { mjs: ortWasmMjsUrl, wasm: ortWasmUrl };
const tensor = (data, dims) => new ort.Tensor("float32", data, dims);
const sessionPromises = new Map();
let portraitCache = null;
let motionTemplatePromise = null;
let activeGeneratorKey = null;
function postProgress(progress, phase) {
self.postMessage({ type: "progress", progress: Math.max(0, Math.min(100, Math.round(progress))), phase });
function postProgress(progress, phaseKey, phaseParams = {}) {
self.postMessage({ type: "progress", progress: Math.max(0, Math.min(100, Math.round(progress))), phaseKey, phaseParams });
}
function resolveModelUrl(file, modelBaseUrl) {
@@ -22,16 +26,15 @@ function resolveModelUrl(file, modelBaseUrl) {
async function fetchModel(key, file, modelBaseUrl, completedBytes, totalBytes) {
const files = Array.isArray(file) ? file : [file];
const parts = [];
let loaded = 0;
for (const partFile of files) {
const parts = await Promise.all(files.map(async (partFile) => {
const response = await fetch(resolveModelUrl(partFile, modelBaseUrl));
if (!response.ok) throw new Error(`${partFile} 下载失败(HTTP ${response.status}`);
const bytes = new Uint8Array(await response.arrayBuffer());
parts.push(bytes);
loaded += bytes.byteLength;
postProgress(5 + ((completedBytes + loaded) / totalBytes) * 55, `下载 ${key}`);
}
postProgress(5 + ((completedBytes + loaded) / totalBytes) * 55, "avatarProgressDownloadModel", { model: key });
return bytes;
}));
const combined = new Uint8Array(loaded);
let offset = 0;
for (const part of parts) {
@@ -42,18 +45,48 @@ async function fetchModel(key, file, modelBaseUrl, completedBytes, totalBytes) {
}
async function loadSession(key, modelBaseUrl, downloadState, executionProvider = "wasm") {
const cacheKey = `${executionProvider}:${modelBaseUrl || "project"}:${key}`;
if (sessionPromises.has(cacheKey)) return sessionPromises.get(cacheKey);
const promise = createSession(key, modelBaseUrl, downloadState, executionProvider).catch((error) => {
sessionPromises.delete(cacheKey);
throw error;
});
sessionPromises.set(cacheKey, promise);
return promise;
}
async function createSession(key, modelBaseUrl, downloadState, executionProvider) {
const file = LIVE_PORTRAIT_WEB_MODEL.files[key];
const bytes = LIVE_PORTRAIT_WEB_MODEL.knownArtifacts[key]?.bytes ?? 0;
const model = await fetchModel(key, file, modelBaseUrl, downloadState.completed, downloadState.total);
downloadState.completed += bytes;
postProgress(5 + (downloadState.completed / downloadState.total) * 55, `初始化 ${key}`);
postProgress(5 + (downloadState.completed / downloadState.total) * 55, "avatarProgressInitModel", { model: key });
if (executionProvider === "webgpu" && !self.navigator?.gpu) {
throw new Error("当前浏览器没有可用的 WebGPU,无法运行全 GPU LivePortrait");
}
const provider = executionProvider === "webgpu" && key === "generatorWebGpu"
const provider = executionProvider === "webgpu" && key.startsWith("generator")
? { name: "webgpu", preferredLayout: "NHWC" }
: executionProvider;
return ort.InferenceSession.create(model, { executionProviders: [provider], graphOptimizationLevel: "all" });
const options = { executionProviders: [provider], graphOptimizationLevel: "all" };
if (executionProvider === "webgpu" && key === "appearanceFeatureExtractorWebGpu") {
options.preferredOutputLocation = { output: "gpu-buffer" };
}
return ort.InferenceSession.create(model, options);
}
function getMotionTemplate(baseUrl) {
if (!motionTemplatePromise) {
motionTemplatePromise = fetch(new URL("joyvasa-motion-template.json", new URL(baseUrl, self.location.origin)))
.then((response) => {
if (!response.ok) throw new Error(`JoyVASA 运动模板下载失败(HTTP ${response.status}`);
return response.json();
})
.catch((error) => {
motionTemplatePromise = null;
throw error;
});
}
return motionTemplatePromise;
}
function preprocessPortrait(blob) {
@@ -154,14 +187,14 @@ async function retargetAndStitch(lipSession, stitchingSession, source, targetRat
return driving;
}
async function outputToBlob(output) {
const [, , height, width] = output.dims;
async function frameDataToBlob(data, dims) {
const [, , height, width] = dims;
const plane = width * height;
const rgba = new Uint8ClampedArray(plane * 4);
for (let i = 0; i < plane; i += 1) {
rgba[i * 4] = Math.round(Math.max(0, Math.min(1, output.data[i])) * 255);
rgba[i * 4 + 1] = Math.round(Math.max(0, Math.min(1, output.data[plane + i])) * 255);
rgba[i * 4 + 2] = Math.round(Math.max(0, Math.min(1, output.data[plane * 2 + i])) * 255);
rgba[i * 4] = Math.round(Math.max(0, Math.min(1, data[i])) * 255);
rgba[i * 4 + 1] = Math.round(Math.max(0, Math.min(1, data[plane + i])) * 255);
rgba[i * 4 + 2] = Math.round(Math.max(0, Math.min(1, data[plane * 2 + i])) * 255);
rgba[i * 4 + 3] = 255;
}
const canvas = new OffscreenCanvas(width, height);
@@ -169,6 +202,40 @@ async function outputToBlob(output) {
return canvas.convertToBlob({ type: "image/png" });
}
function outputToBlob(output) {
return frameDataToBlob(output.data, output.dims);
}
function sampledFrameDistance(data, dims, reference, referenceDims) {
const [, channels, height, width] = dims;
const [, referenceChannels, referenceHeight, referenceWidth] = referenceDims;
const sampleSize = 32;
let difference = 0;
let samples = 0;
for (let channel = 0; channel < Math.min(3, channels, referenceChannels); channel += 1) {
const planeOffset = channel * width * height;
const referenceOffset = channel * referenceWidth * referenceHeight;
for (let row = 0; row < sampleSize; row += 1) {
const y = Math.min(height - 1, Math.round((row / (sampleSize - 1)) * (height - 1)));
const referenceY = Math.min(referenceHeight - 1, Math.round((row / (sampleSize - 1)) * (referenceHeight - 1)));
for (let column = 0; column < sampleSize; column += 1) {
const x = Math.min(width - 1, Math.round((column / (sampleSize - 1)) * (width - 1)));
const referenceX = Math.min(referenceWidth - 1, Math.round((column / (sampleSize - 1)) * (referenceWidth - 1)));
const value = data[planeOffset + y * width + x];
const referenceValue = reference[referenceOffset + referenceY * referenceWidth + referenceX];
if (!Number.isFinite(value) || !Number.isFinite(referenceValue)) return Number.POSITIVE_INFINITY;
difference += Math.abs(value - referenceValue);
samples += 1;
}
}
}
return samples ? difference / samples : Number.POSITIVE_INFINITY;
}
function disposeOutputs(result) {
Object.values(result || {}).forEach((value) => value?.dispose?.());
}
function templateValue(template, key, index = 0) {
return Array.isArray(template[key]) ? template[key][index] : template[key];
}
@@ -187,6 +254,40 @@ function decodeJoyVasaFrame(coefficients, frame, template) {
return { exp, scale, translation, rotation: rotationMatrix(pitch, yaw, roll) };
}
function selectAdaptiveMotionFrames(coefficients, duration, maximumFps) {
const lastFrame = Math.min(99, Math.max(1, Math.round(duration * 25)));
const mandatoryGap = 25;
const selected = new Set([0, lastFrame]);
for (let frame = mandatoryGap; frame < lastFrame; frame += mandatoryGap) selected.add(frame);
const candidates = [];
for (let frame = 1; frame < lastFrame; frame += 1) {
const offset = frame * 73;
const previous = (frame - 1) * 73;
let squared = 0;
// Expression coefficients dominate visible lip motion; head pose receives
// a smaller contribution so a quick turn can still request a keyframe.
for (let index = 0; index < 63; index += 1) {
const delta = coefficients[offset + index] - coefficients[previous + index];
squared += delta * delta;
}
for (let index = 67; index < 70; index += 1) {
const delta = coefficients[offset + index] - coefficients[previous + index];
squared += delta * delta * 0.2;
}
candidates.push({ frame, energy: Math.sqrt(squared / 63) });
}
const energies = candidates.map(({ energy }) => energy).sort((a, b) => a - b);
const threshold = energies[Math.floor(energies.length * 0.62)] || 0;
const maximumCount = Math.max(selected.size, Math.ceil(duration * maximumFps) + 1);
for (const candidate of candidates.sort((a, b) => b.energy - a.energy)) {
if (selected.size >= maximumCount || candidate.energy < threshold) break;
const separated = [...selected].every((frame) => Math.abs(frame - candidate.frame) >= 8);
if (separated) selected.add(candidate.frame);
}
return [...selected].sort((a, b) => a - b);
}
function buildDrivingKeypoints(motion, source, driving, initialDriving) {
const sourceRotation = rotationMatrix(headposeDegree(motion.pitch.data), headposeDegree(motion.yaw.data), headposeDegree(motion.roll.data));
const relativeRotation = multiply3x3(multiply3x3(driving.rotation, transpose3x3(initialDriving.rotation)), sourceRotation);
@@ -208,68 +309,172 @@ function buildDrivingKeypoints(motion, source, driving, initialDriving) {
return output;
}
async function generateVideo({ portraitBlob, motionBuffer, modelBaseUrl, joyVasaModelBaseUrl, webGpuModelBaseUrl, renderFps = 8 }) {
postProgress(2, "准备肖像与真实口型运动");
async function generateVideo({
portraitBlob,
motionBuffer,
modelBaseUrl,
joyVasaModelBaseUrl,
webGpuModelBaseUrl,
quality = "preview",
renderFps = 8,
neuralFps = 2,
duration = 4,
portraitKey = "",
}) {
postProgress(2, "avatarProgressPreparePortraitMotion");
const image = await preprocessPortrait(portraitBlob);
const keys = ["appearanceFeatureExtractorWebGpu", "motionExtractorWebGpu", "stitchingWebGpu", "generatorWebGpu"];
const preferredGeneratorKey = quality === "quality" ? "generatorQualityFp16" : "generatorPreviewFp16";
if (activeGeneratorKey && activeGeneratorKey !== preferredGeneratorKey) {
for (const [cacheKey, promise] of sessionPromises.entries()) {
if (!cacheKey.endsWith(`:${activeGeneratorKey}`)) continue;
sessionPromises.delete(cacheKey);
const session = await promise.catch(() => null);
session?.release();
}
}
activeGeneratorKey = preferredGeneratorKey;
const keys = ["appearanceFeatureExtractorWebGpu", "motionExtractorWebGpu", "stitchingWebGpu", preferredGeneratorKey];
const downloadState = { completed: 0, total: keys.reduce((sum, key) => sum + (LIVE_PORTRAIT_WEB_MODEL.knownArtifacts[key]?.bytes ?? 0), 0) };
const sessions = {};
for (const key of keys) {
sessions[key] = await loadSession(
key,
webGpuModelBaseUrl,
downloadState,
"webgpu",
);
}
const templateResponse = await fetch(new URL("joyvasa-motion-template.json", new URL(joyVasaModelBaseUrl, self.location.origin)));
if (!templateResponse.ok) throw new Error(`JoyVASA 运动模板下载失败(HTTP ${templateResponse.status}`);
const template = await templateResponse.json();
for (const key of keys) sessions[key] = await loadSession(key, webGpuModelBaseUrl, downloadState, "webgpu");
const template = await getMotionTemplate(joyVasaModelBaseUrl);
const coefficients = new Float32Array(motionBuffer);
postProgress(63, "提取人物 3D 特征");
const feature = (await sessions.appearanceFeatureExtractorWebGpu.run({ img: image })).output;
const sourceMotion = await sessions.motionExtractorWebGpu.run({ img: image });
postProgress(63, "avatarProgressExtract3d");
let feature;
let sourceMotion;
let portraitPixels;
if (portraitKey && portraitCache?.key === portraitKey) {
({ feature, sourceMotion, portraitPixels } = portraitCache);
image.dispose();
postProgress(66, "avatarProgressReuseGpuFeature");
} else {
portraitPixels = new Float32Array(image.data);
feature = (await sessions.appearanceFeatureExtractorWebGpu.run({ img: image })).output;
sourceMotion = await sessions.motionExtractorWebGpu.run({ img: image });
image.dispose();
if (portraitCache) {
portraitCache.feature.dispose();
Object.values(portraitCache.sourceMotion).forEach((value) => value?.dispose?.());
}
portraitCache = { key: portraitKey, feature, sourceMotion, portraitPixels };
}
const source = transformKeypoints(sourceMotion);
const sourceTensor = tensor(source, [1, 21, 3]);
const stitchInput = new Float32Array(126);
const stitchTensor = tensor(stitchInput, [1, 126]);
const drivingInput = new Float32Array(63);
const drivingTensor = tensor(drivingInput, [1, 21, 3]);
const initialDriving = decodeJoyVasaFrame(coefficients, 0, template);
const frameCount = Math.ceil(4 * renderFps);
const safeDuration = Math.max(0.25, Math.min(4, Number(duration) || 4));
const safeOutputFps = Math.max(1, Math.min(30, Number(renderFps) || 8));
const safeNeuralFps = Math.max(1, Math.min(safeOutputFps, Number(neuralFps) || 2));
const motionFrames = selectAdaptiveMotionFrames(coefficients, safeDuration, safeNeuralFps);
const frameCount = motionFrames.length;
const blobs = [];
let previousFramePixels = null;
let previousFrameDims = null;
for (let outputFrame = 0; outputFrame < frameCount; outputFrame += 1) {
const motionFrame = Math.min(99, Math.round((outputFrame / renderFps) * 25));
const motionFrame = motionFrames[outputFrame];
const frameTime = Math.min(safeDuration, motionFrame / 25);
const driving = decodeJoyVasaFrame(coefficients, motionFrame, template);
const rawDriving = buildDrivingKeypoints(sourceMotion, source, driving, initialDriving);
const stitched = await (async () => {
const input = new Float32Array(126);
input.set(source);
input.set(rawDriving, 63);
const result = await sessions.stitchingWebGpu.run({ input: tensor(input, [1, 126]) });
const value = new Float32Array(rawDriving);
for (let i = 0; i < 63; i += 1) value[i] += result.output.data[i];
for (let point = 0; point < 21; point += 1) {
value[point * 3] += result.output.data[63];
value[point * 3 + 1] += result.output.data[64];
stitchInput.set(source);
stitchInput.set(rawDriving, 63);
const stitchResult = await sessions.stitchingWebGpu.run({ input: stitchTensor });
drivingInput.set(rawDriving);
for (let i = 0; i < 63; i += 1) drivingInput[i] += stitchResult.output.data[i];
for (let point = 0; point < 21; point += 1) {
drivingInput[point * 3] += stitchResult.output.data[63];
drivingInput[point * 3 + 1] += stitchResult.output.data[64];
}
disposeOutputs(stitchResult);
let acceptedPixels = null;
let acceptedDims = null;
let inferenceMs = 0;
for (let attempt = 0; attempt < 2; attempt += 1) {
const inferenceStarted = performance.now();
const generated = await sessions[preferredGeneratorKey].run({
feature_3d: feature,
kp_source: sourceTensor,
kp_driving: drivingTensor,
});
inferenceMs += performance.now() - inferenceStarted;
const output = generated.out;
const pixels = new Float32Array(output.data);
const dims = [...output.dims];
const reference = previousFramePixels || portraitPixels;
const referenceDims = previousFrameDims || [1, 3, 256, 256];
const distance = sampledFrameDistance(pixels, dims, reference, referenceDims);
const threshold = previousFramePixels ? 0.22 : 0.34;
disposeOutputs(generated);
if (distance <= threshold) {
acceptedPixels = pixels;
acceptedDims = dims;
break;
}
return value;
})();
const inferenceStarted = performance.now();
const generated = await sessions.generatorWebGpu.run({
feature_3d: feature,
kp_source: tensor(source, [1, 21, 3]),
kp_driving: tensor(stitched, [1, 21, 3]),
});
const inferenceMs = performance.now() - inferenceStarted;
postProgress(68 + (outputFrame / frameCount) * 31, `WebGPU 帧 ${outputFrame + 1}/${frameCount} · ${(inferenceMs / 1000).toFixed(1)}s`);
postProgress(68 + (outputFrame / frameCount) * 31, "avatarProgressRetryCorruptFrame", { current: outputFrame + 1, total: frameCount, attempt: attempt + 1 });
}
postProgress(68 + (outputFrame / frameCount) * 31, "avatarProgressKeyframe", { current: outputFrame + 1, total: frameCount, seconds: (inferenceMs / 1000).toFixed(1) });
const encodeStarted = performance.now();
blobs.push(await outputToBlob(generated.out));
if (acceptedPixels) {
blobs.push(await frameDataToBlob(acceptedPixels, acceptedDims));
previousFramePixels = acceptedPixels;
previousFrameDims = acceptedDims;
} else if (blobs.length) {
blobs.push(blobs[blobs.length - 1]);
postProgress(68 + (outputFrame / frameCount) * 31, "avatarProgressDroppedCorruptFrame", { current: outputFrame + 1, total: frameCount });
} else {
throw new Error("WebGPU 首帧连续异常,已阻止损坏画面写入视频;请释放其他 GPU 页面后重试");
}
const encodeMs = performance.now() - encodeStarted;
postProgress(68 + ((outputFrame + 1) / frameCount) * 31, `完成帧 ${outputFrame + 1}/${frameCount} · 编码 ${(encodeMs / 1000).toFixed(1)}s`);
postProgress(68 + ((outputFrame + 1) / frameCount) * 31, "avatarProgressFrameEncoded", { current: outputFrame + 1, total: frameCount, seconds: (encodeMs / 1000).toFixed(1) });
}
postProgress(100, "真实口型帧生成完成");
self.postMessage({ type: "videoFrames", blobs, width: 512, height: 512, fps: renderFps, duration: 4 });
sourceTensor.dispose();
stitchTensor.dispose();
drivingTensor.dispose();
const size = quality === "quality" ? 512 : 256;
postProgress(100, "avatarProgressInterpolate");
self.postMessage({
type: "videoFrames",
blobs,
width: size,
height: size,
fps: safeOutputFps,
keyframeFps: safeNeuralFps,
keyframeTimes: motionFrames.map((frame) => Math.min(safeDuration, frame / 25)),
duration: safeDuration,
quality,
precision: "mixed-fp16",
});
}
async function prepare({ webGpuModelBaseUrl, quality = "preview" }) {
const generator = quality === "quality" ? "generatorQualityFp16" : "generatorPreviewFp16";
const keys = ["appearanceFeatureExtractorWebGpu", "motionExtractorWebGpu", "stitchingWebGpu", generator];
const downloadState = { completed: 0, total: keys.reduce((sum, key) => sum + (LIVE_PORTRAIT_WEB_MODEL.knownArtifacts[key]?.bytes ?? 0), 0) };
for (const key of keys) await loadSession(key, webGpuModelBaseUrl, downloadState, "webgpu");
self.postMessage({ type: "prepared", quality });
}
async function releaseGpuSessions() {
if (portraitCache) {
portraitCache.feature.dispose();
Object.values(portraitCache.sourceMotion).forEach((value) => value?.dispose?.());
portraitCache = null;
}
const pending = [...sessionPromises.entries()];
sessionPromises.clear();
activeGeneratorKey = null;
for (const [, promise] of pending) {
const session = await promise.catch(() => null);
session?.release();
}
self.postMessage({ type: "gpuReleased" });
}
async function generate({ portraitBlob, modelBaseUrl }) {
postProgress(2, "准备肖像");
postProgress(2, "avatarPreparing");
const image = await preprocessPortrait(portraitBlob);
const keys = ["appearanceFeatureExtractor", "motionExtractor", "stitchingLip", "stitching", "warping", "spadeGenerator"];
const downloadState = {
@@ -279,22 +484,22 @@ async function generate({ portraitBlob, modelBaseUrl }) {
const sessions = {};
for (const key of keys) sessions[key] = await loadSession(key, modelBaseUrl, downloadState);
postProgress(63, "提取人物特征");
postProgress(63, "avatarProgressExtractFeature");
const feature = (await sessions.appearanceFeatureExtractor.run({ img: image })).output;
const motion = await sessions.motionExtractor.run({ img: image });
const source = transformKeypoints(motion);
const driving = await retargetAndStitch(sessions.stitchingLip, sessions.stitching, source, 0.35);
postProgress(70, "计算 3D 形变(可能需要约 1 分钟)");
postProgress(70, "avatarProgressWarp3d");
const warped = await sessions.warping.run({
feature_3d: feature,
kp_source: tensor(source, [1, 21, 3]),
kp_driving: tensor(driving, [1, 21, 3]),
});
postProgress(86, "渲染 512×512 人像");
postProgress(86, "avatarProgressRenderPortrait");
const generated = await sessions.spadeGenerator.run({ input: warped["879"] });
const blob = await outputToBlob(generated.output);
postProgress(100, "单帧验收完成");
postProgress(100, "avatarAcceptanceDone");
self.postMessage({ type: "result", blob, width: 512, height: 512 });
}
@@ -308,6 +513,10 @@ async function probe({ modelBaseUrl }) {
self.onmessage = (event) => {
const task = event.data?.type === "generateVideo"
? generateVideo
: event.data?.type === "prepare"
? prepare
: event.data?.type === "releaseGpuSessions"
? releaseGpuSessions
: event.data?.type === "generate"
? generate
: event.data?.type === "probe"