feat: VRSub 单体应用（WOV 单机版）初始提交
为视频生成 VR 双眼字幕的单体实现：FastAPI 后端、调度器与全部节点 （提音/转写/翻译/ASS/抽帧/OCR/LLM 过滤）在单进程内运行。 - 节点协议（wov_sdk 数据模型）与分布式版保持一致，预留回退桥梁 - 工作流即数据：DAG 存于 workflows/*.json，模型/链路改动只改数据 - 调度器：拓扑顺序执行、断点续跑（产物重建）、任务暂停/继续 - 抽帧按帧间隔（select 按帧号精确取帧），VLM OCR 与 LLM 过滤使用 自适应线程池弹性并发，并打印数据处理速度进度日志 - 100% 行覆盖率（pytest --cov-fail-under=100）
This commit is contained in:
+22
@@ -0,0 +1,22 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.coverage
|
||||
htmlcov/
|
||||
.venv/
|
||||
*.egg-info/
|
||||
build/
|
||||
dist/
|
||||
|
||||
# 本地环境变量（含 API 密钥，禁止入库）
|
||||
.env
|
||||
|
||||
# 本地数据与产物（上传文件、SQLite、节点中间产物）
|
||||
data/
|
||||
|
||||
# 模型权重（大文件，不入库；如需共享请走对象存储或模型仓库）
|
||||
model/
|
||||
# 编辑器与系统
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
.idea/
|
||||
@@ -0,0 +1,339 @@
|
||||
# VRSub（单体版）
|
||||
|
||||
本仓库是"为视频生成 VR 双眼字幕"的单体应用（WOV AI Workflow Platform 的
|
||||
单机实现）：FastAPI 后端、工作流调度器与全部节点（提音 / 转写 / 翻译 /
|
||||
ASS）在**同一个进程**内运行，不再启动子进程、不再走节点 HTTP 协议。
|
||||
由原分布式多仓库（wov-api / wov-web / wov-sdk / wov-node-*）合并而来，
|
||||
本 AGENTS.md 汇总了各仓库的约定与规范。
|
||||
|
||||
## 架构概览
|
||||
|
||||
```
|
||||
vrsub/
|
||||
├── src/wov_sdk/ # 协议数据模型（NodeManifest/InvokeRequest/InvokeResponse/
|
||||
│ # WorkflowDefinition 等），与分布式版保持一致
|
||||
├── src/wov_app/ # 应用层：main/config/db/registry/scheduler/seed/routers
|
||||
│ └── routers/ # apps.py（用户端）、workflows.py（管理端）
|
||||
├── nodes/ # 进程内节点实现：echo/ffmpeg/whisper/llm/ass
|
||||
├── manifests/ # 各节点清单 JSON（echo.json/ffmpeg.json/...）
|
||||
├── workflows/ # 默认工作流定义 JSON（模型/链路均为数据，改模型不改代码）
|
||||
├── web/ # 静态前端（index/tasks/admin/workflow + assets）
|
||||
├── model/ # 本地 whisper 权重（gitignored）
|
||||
├── data/ # SQLite + 上传/产物存储（gitignored）
|
||||
└── tests/ # 全部单元/API/冒烟测试（100% 覆盖率）
|
||||
```
|
||||
|
||||
### 核心机制
|
||||
|
||||
- **节点注册表**（`src/wov_app/registry.py`）：启动时把 `manifests/*.json` 与
|
||||
`nodes/*.py` 的 `invoke` 处理器静态注册到进程内字典，调度器按
|
||||
`node_type` 直接调用。协议数据模型不变，为将来回退分布式保留兼容桥梁。
|
||||
- **调度器**（`src/wov_app/scheduler.py`）：后台线程轮询 SQLite 中的 QUEUED
|
||||
任务，按工作流 DAG 拓扑顺序调用节点，产物按
|
||||
`data/storage/runs/<run_id>/steps/<node_id>/` 落盘并登记到 artifacts 表。
|
||||
- **前端**：由 FastAPI 静态挂载 `web/`，节点注册/实例管理页面已移除，
|
||||
仅保留应用中心、任务管理、管理后台（工作流）与工作流编排。
|
||||
|
||||
## 节点输入/输出协议
|
||||
|
||||
节点统一签名 `invoke(request: InvokeRequest) -> InvokeResponse`，通过产物 URI
|
||||
交换数据（节点之间不直接调用，不共享内存状态）。
|
||||
|
||||
| 节点 ID（node_type） | 输入 | 输出 | 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| `echo` | `text` / `file_uri` | `text`、`file_uri` | 示例节点，验证协议链路 |
|
||||
| `ffmpeg-extract` | `video_uri` | `audio_uri`（WAV） | 参数：`sample_rate`、`channels` |
|
||||
| `faster-whisper` | `audio_uri`（16kHz 单声道） | `srt_uri` | 参数：`language`、`task`、`model_path`、`device`、`compute_type`、`beam_size`、`vad_filter`（默认开）、`condition_on_previous_text`、`chunk_seconds` |
|
||||
| `llm-translate` | `srt_uri` | `cn_srt_uri` | 参数：`target_language`、`model` |
|
||||
| `vlm-ocr` | `image_uri` | `text`、`text_uri` | 直接调本地 Ollama 多模态模型（glm-ocr）的 `/api/chat` 做视频帧 OCR（流式 + 5s 上限），参数：`model`、`ollama_host`、`prompt`、`timeout_seconds`、`keep_alive`、`num_predict`、`temperature`、`repeat_penalty` |
|
||||
| `frame-extract` | `video_uri` | `frames_manifest`、`frame_count` | 按**帧间隔**抽帧（解析 fps → step=round(间隔秒×fps)，ffmpeg select 按帧号精确取帧，帧时间=帧号/fps 无累计偏差）并 crop 裁切字幕区域，参数：`interval_seconds`（默认 0.5）、`crop`（[x,y,w,h] 0~1） |
|
||||
| `subtitle-ocr` | `frames_manifest` | `srt_uri`、`count` | 自适应线程池并发逐帧调 vlm-ocr → 垃圾过滤（无文字帧）→ 相同字幕合并（记录最后可见帧）→ 组装 SRT，消失时间=最后可见帧+采样间隔（间隔从帧清单推导），参数：`min_chars`、`min_alnum_ratio`、`garbage_tokens`、`pool_min_workers`/`pool_max_workers`/`pool_window_seconds`/`pool_fast_threshold`/`pool_slow_threshold` |
|
||||
| `llm-filter` | `srt_uri` | `srt_uri`、`kept`、`removed` | LLM 过滤无意义字幕（自适应线程池并发判断）：每条连同前后各 `context_size`（默认 10）条纯文本（不含时间戳）分批给 LLM，仅判断目标字幕是否多余/无意义，判定删除则该条连同时间戳移除并重新编号，参数：`context_size`、`model`、`pool_min_workers`/`pool_max_workers`/`pool_window_seconds`/`pool_fast_threshold`/`pool_slow_threshold` |
|
||||
| `srt-to-dual-eye-ass` | `cn_srt_uri` | `ass_uri` | 参数：`resolution`，如 `3840x1920` |
|
||||
|
||||
### 模型权重解析（本地优先）
|
||||
|
||||
whisper 节点按以下顺序解析模型路径，默认避免从远端下载：
|
||||
|
||||
1. 请求参数 `model_path`；裸模型名（不含路径分隔符）会在 `model/<名称>` 下解析。
|
||||
2. 环境变量 `WHISPER_MODEL_PATH`。
|
||||
3. 本地候选目录（存在且含 `model.bin` 即使用）：
|
||||
- 单体根目录 `model/faster-whisper-large-v3`。
|
||||
- `nodes/model/faster-whisper-large-v3`。
|
||||
4. 兜底：`large-v3`（需要联网从 Hugging Face 下载）。
|
||||
|
||||
把权重放在 `model/` 目录即可完全离线运行。当前已下载模型：
|
||||
|
||||
- `model/faster-whisper-large-v3`：通用转写模型（demo 工作流）。
|
||||
- `model/whisper-large-v2-translate-zh-v0.2-st-ct2`：中文直出模型
|
||||
（`chickenrice0721/whisper-large-v2-translate-zh-v0.2-st-ct2`），配合
|
||||
`task=translate` 直接生成中文，无需 LLM 翻译（zh-direct 工作流）。
|
||||
- `model/whisper-large-v3-translate-zh-v0.1-lt-ct2`：早期中文直出模型，
|
||||
已无工作流引用，保留在盘上待处理。
|
||||
|
||||
### 内置工作流
|
||||
|
||||
| ID | 名称 | 链路 | 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| `demo` | 视频字幕生成 | 提音 → 转写 → LLM 翻译 → ASS | 通用链路，翻译走 SiliconFlow |
|
||||
| `zh-direct` | 中文直出字幕 | 提音 → 中文转写 → ASS | 中文直出模型，无 LLM 步骤 |
|
||||
| `ocr-subtitle` | 字幕OCR提取 | 抽帧 → 逐帧 OCR → 汇总 SRT → LLM 过滤 | 提取烧录字幕做基准数据；前端框选 crop；LLM 过滤多余/无意义字幕 |
|
||||
|
||||
最终产物按 `上传文件名.标识.时间戳` 重命名（如 `test01.zh-CN.20260815123000.srt`），
|
||||
标识优先取节点的 `target_language` 参数，否则用产物别名。
|
||||
|
||||
### 任务参数覆盖（前端框选）
|
||||
|
||||
创建任务时可携带可选 `params` 表单字段（JSON）：`{"节点ID": {"参数": 值}}`，
|
||||
随任务持久化（param_overrides），调度执行时合并进对应节点参数。字幕 OCR
|
||||
前端把框选的 `crop` 按此传给 `frame-extract` 节点。
|
||||
|
||||
### 切换模型不改代码
|
||||
|
||||
- 模型是工作流 DAG 中 asr 节点的 `model_path` 参数（**数据**），两个内置工作流
|
||||
均已显式声明：demo 用 `faster-whisper-large-v3`，zh-direct 用中文直出模型。
|
||||
- 切换模型 = 改 `workflows/*.json` 或管理页面 DAG JSON → 保存新版本 → 发布，
|
||||
全程不涉及代码；新库启动时从 JSON 重新 seed。
|
||||
- 默认工作流定义存放在 `workflows/*.json`（数据文件），代码只负责加载。
|
||||
|
||||
### 长音频处理
|
||||
|
||||
**当前策略：分块转写，默认每 1 分钟一块**（`chunk_seconds=60`，2026-08 调整）。
|
||||
whisper 节点内部用 ffmpeg 把音频切成块 → 逐块转写 → 按偏移合并为完整 SRT：
|
||||
|
||||
- 内存/显存有界（模型 + 单块音频），任意时长可处理，失败粒度小。
|
||||
- 分块是**应用层工程策略**，与模型训练格式无关：whisper 训练/推理都按 30s
|
||||
窗口解码，任意块大小均适用。
|
||||
- 每块 `offset = 块序号 × chunk_seconds`，SRT 序号连续；切块失败自动回退
|
||||
整段单次转写。
|
||||
- `chunk_seconds=0` 可关闭分块；大小按工作流 DAG 参数（数据）调整。
|
||||
- 同时默认 `condition_on_previous_text=false`（每块/每窗口独立解码，防重复）。
|
||||
- **`vad_filter` 默认开启**（2026-08 用户决定）：过滤静音段提速并减少无语音处
|
||||
幻觉。注意 VAD 靠压缩时间轴回映射（SpeechTimestampsMap），长静音场景曾实测
|
||||
错位（30s 静音致第二段语音从 ~40s 落到 10s）；如需极致对齐可显式传
|
||||
`vad_filter=false`。
|
||||
- **分块偏移按每块实际时长累积**（WAV 头精确）：ffmpeg 切出的块实际时长不等于
|
||||
块长（如 60.05s），用 `块序号×块长` 的假设值会随块数累积漂移；改为按真实
|
||||
时长累加后，字幕时间轴与原始音频严格一致。
|
||||
- 参考：openai/whisper 重复问题（issue #1026/#1046，PR #1052/#1253）、
|
||||
SYSTRAN/faster-whisper issue #465。
|
||||
|
||||
### glm-ocr 重复循环问题与源头修复（2026-08）
|
||||
|
||||
- **根因**：glm-ocr 生成阶段存在已知 bug（M-RoPE delta 未传递，大图触发
|
||||
重复循环；GitHub #454 / #16892）。`keep_alive` 与其无关（实测无效）。
|
||||
- **源头修复**：
|
||||
1. `frame-extract` 裁切后把帧**压缩到 720p 内**（仅缩小，保持宽高比）——
|
||||
过大输入图是触发条件之一。
|
||||
2. `vlm` 请求体 `options.repeat_penalty`（默认 1.2）+ `num_predict`
|
||||
（默认 256）压制重复。
|
||||
3. `subtitle-ocr` 增加 `max_result_chars`（默认 200）：模型输出超长视为
|
||||
异常（重复循环等），**直接报错并跳过该帧**。
|
||||
- **glm-ocr 调用结构**：走 Ollama `/api/chat`，识别指令放**系统提示词**，
|
||||
用户消息只携带图片（content 为空、images 传 base64），`stream=True`
|
||||
逐行接收，`stop: ["\n", "\n答", "答"]` 命中即停止（输出首个换行即停 + 阻止“答：”式重复循环），`temperature` 默认
|
||||
0.3、`repeat_penalty` 默认 1、`num_predict`（默认 256）随请求透传。
|
||||
- **gettext 标签防御性提取**（2026-08）：若模型输出含 `<gettext></gettext>`
|
||||
标签（旧提示词要求）则取第一个标签内文本，多个标签取第一个防重复循环；
|
||||
未按格式输出时回退原文。当前默认提示词为"提取图像中的文字，不要描述
|
||||
图片中的内容"（字幕流水线在 ocr-subtitle 工作流的 subtitle-ocr 节点参数
|
||||
中显式指定，经 subtitle-ocr 透传给 vlm-ocr）。
|
||||
- **每次调用 5 秒上限**（2026-08 调整）：vlm 请求 `stream=True` 逐行读取，
|
||||
每次调用整体受 5 秒截止时间约束（`timeout_seconds` 参数 /
|
||||
`VLM_TIMEOUT_SECONDS`，默认 5），超过即终止返回 failed，不再等待后续
|
||||
流式块。
|
||||
- **不做文本加工**：除协议要求的 gettext 标签提取外，不再对模型输出做
|
||||
过滤/去重等文本加工，结果原样使用，仅受长度上限约束。
|
||||
|
||||
### VLM OCR 自适应并发（2026-08）
|
||||
|
||||
subtitle-ocr 逐帧调 vlm-ocr 时使用 `nodes/adaptive_pool.py` 的自适应线程池
|
||||
弹性并发：
|
||||
|
||||
- 从 `pool_min_workers`（默认 1）起步，按**滚动窗口**（`pool_window_seconds`，
|
||||
默认 10s）统计已完成任务的平均响应时间；
|
||||
- 平均响应 < `pool_fast_threshold`（默认 0.3s）→ 线程数 +1（上限
|
||||
`pool_max_workers`，默认 16）——服务端空闲就加大并发加速处理；
|
||||
- 平均响应 > `pool_slow_threshold`（默认 1.0s）→ 线程数 -1（下限 1）——
|
||||
服务端变慢就退避，避免盲目并发压垮本地 Ollama；
|
||||
- 结果按帧顺序返回，SRT 时间轴不受并发影响；worker 需无共享可变状态
|
||||
（vlm-ocr 处理器为纯函数，线程安全）。
|
||||
### 前端 OCR 框选
|
||||
|
||||
首页选择工作流后，若 DAG 中存在声明 `crop` 参数的节点（frame-extract），
|
||||
自动切换到框选面板：视频预览 + 拖动框选字幕区域 → 生成 crop 比例 →
|
||||
框选完成后才可提交（未框选时提交按钮禁用）。矩形↔crop 换算为纯函数
|
||||
（`web/assets/crop.js`，含 letterbox 处理），由 node 单测覆盖。
|
||||
|
||||
## 环境变量
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `WOV_DATA_DIR` | `<根>/data` | 数据目录 |
|
||||
| `WOV_DB_PATH` | `<根>/data/wov.db` | SQLite 路径 |
|
||||
| `WOV_STORAGE_DIR` | `<根>/data/storage` | 上传与产物根目录 |
|
||||
| `WOV_AUTO_SEED` | `1` | 启动时创建 demo 工作流 |
|
||||
| `WOV_SCHEDULER_ENABLED` | `1` | 启动后台调度器 |
|
||||
| `WOV_SCHEDULER_INTERVAL_SECONDS` | `1.0` | 调度轮询间隔 |
|
||||
| `WOV_CLEANUP_ENABLED` | `1` | 开启孤儿数据定时清理 |
|
||||
| `WOV_CLEANUP_INTERVAL_SECONDS` | `3600` | 孤儿清理扫描周期（秒） |
|
||||
| `WOV_CLEANUP_GRACE_SECONDS` | `3600` | 孤儿清理宽限期（秒） |
|
||||
| `WHISPER_MODEL_PATH` | 见上 | 显式指定 whisper 模型路径 |
|
||||
| `WHISPER_DEVICE` | `auto` | 转写设备 |
|
||||
| `LLM_API_BASE` | `https://api.siliconflow.cn/v1/chat/completions` | LLM 兼容接口 |
|
||||
| `LLM_API_KEY` | 空（读 `.env`） | SiliconFlow Bearer Key，存于 gitignored 的 `.env` |
|
||||
| `LLM_MODEL` | `Qwen/Qwen3.6-35B-A3B` | LLM 模型名 |
|
||||
| `LLM_TIMEOUT_SECONDS` | `600` | LLM 单请求超时 |
|
||||
| `OLLAMA_HOST` | `http://192.168.123.70:11434` | Ollama 服务地址 |
|
||||
| `VLM_MODEL` | `glm-ocr:latest` | VLM OCR 模型 |
|
||||
| `VLM_PROMPT` | 提取图像中的文字，不要描述图片中的内容 | OCR 提示词（字幕流水线在 ocr-subtitle 工作流的 subtitle-ocr 节点参数中显式指定同一提示词） |
|
||||
| `VLM_TIMEOUT_SECONDS` | `5` | VLM 单请求整体超时上限（每次调用 5 秒，超时即终止；流式读取同样受此截止约束） |
|
||||
| `FFMPEG_BIN` | 空 | 显式 ffmpeg 路径（否则 PATH → imageio-ffmpeg） |
|
||||
|
||||
## 启动
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
uv run uvicorn wov_app.main:app --reload
|
||||
```
|
||||
|
||||
访问：
|
||||
|
||||
```
|
||||
http://127.0.0.1:8000/ 应用中心（上传视频 → 字幕生成）
|
||||
http://127.0.0.1:8000/tasks.html 任务管理
|
||||
http://127.0.0.1:8000/admin.html 管理后台（工作流）
|
||||
http://127.0.0.1:8000/workflow.html 工作流编排（DAG JSON）
|
||||
http://127.0.0.1:8000/docs API 文档
|
||||
```
|
||||
|
||||
## Python 环境与 uv 管理
|
||||
|
||||
- 统一使用 uv 管理虚拟环境和依赖，禁止直接使用 pip 修改依赖。
|
||||
- 基础命令：`uv sync`（安装含 dev 组依赖）、`uv run <command>`、
|
||||
`uv add <package>`、`uv lock`。
|
||||
- 虚拟环境位于 `.venv`，测试依赖在 `[dependency-groups] dev`。
|
||||
- 新增依赖时使用 `uv add`，不修改系统 Python 或全局环境。
|
||||
|
||||
## 孤儿数据清理
|
||||
|
||||
应用内置后台清理器（`src/wov_app/maintenance.py`），按周期自动清理死数据：
|
||||
|
||||
- **自动删除**：无任务记录的上传/步骤残留目录；COMPLETED 且产物文件全部丢失、
|
||||
超过宽限期（默认 1 小时）的任务记录（下载已全部 404）。
|
||||
- **绝不自动删除**：FAILED 任务（可重试）、QUEUED/RUNNING 任务、宽限期内的任务、
|
||||
仍有产物文件的任务。
|
||||
- 手动删除任务仅通过删除接口（`DELETE /api/runs/{run_id}`）或管理界面进行。
|
||||
|
||||
## 任务暂停/继续（2026-08）
|
||||
|
||||
- **状态机**：`QUEUED / RUNNING / PAUSED / COMPLETED / FAILED`。排队中或运行中的
|
||||
任务可暂停（`POST /api/runs/{run_id}/pause`），PAUSED 可继续
|
||||
（`POST /api/runs/{run_id}/resume` → 恢复 QUEUED）。
|
||||
- **调度器语义**：`next_queued_run` 同时取 QUEUED 与 PAUSED；`execute_run` 在每个
|
||||
节点边界检查状态，被暂停则停下保持 PAUSED（当前节点执行完后才停）；
|
||||
继续时从产物表（`restore_run_outputs`，剥去"节点ID."前缀还原输出名）重建已完成
|
||||
节点的输出，**跳过已完成节点断点续跑**，最后补做 final_outputs 收尾。
|
||||
- **前端**：任务管理页为 QUEUED/RUNNING 提供"暂停"、PAUSED 提供"继续"按钮。
|
||||
- **进度日志（数据处理速度）**：
|
||||
- 调度器：每节点完成打印"任务 X 进度 i/N 节点: Y 耗时 Zs, 运行累计 Ws"；
|
||||
- subtitle-ocr：`OCR 进度 X/Y 帧 (Z 帧/s)`；llm-filter：`字幕判定进度 X/Y 条 (Z 条/s)`
|
||||
（线程池 `on_progress` 回调，每任务完成触发）；
|
||||
- whisper：分块转写打印"分块 X/Y 完成 offset=... 耗时 Zs (Nx 实时, 累计 ...s)"。
|
||||
|
||||
## 测试与覆盖率
|
||||
|
||||
- 必须达到 100% 行覆盖率（pytest 已配置 `--cov-fail-under=100`，范围
|
||||
`src/` 与 `nodes/`）。
|
||||
- 测试必须调用真实代码路径，不得在测试类中重写业务逻辑来模拟被测功能。
|
||||
- **测试必须使用真实数据**：真实音频（合法 WAV/PCM）、真实 JSON/数据库/文件；
|
||||
禁止用占位字节（如 `b"x"`）或伪造结构冒充被测数据——假数据测试只能凑覆盖率，
|
||||
无法验证真实行为，视为无意义测试。
|
||||
- 只允许在 I/O 边界使用 mock/stub：文件系统、网络、子进程、环境变量、时间、
|
||||
**模型推理**（重模型不进入单元测试；注入的假模型必须返回结构真实的分段，
|
||||
且必须配套真实模型集成测试，见下）。
|
||||
- **真实模型集成测试**：使用真实 faster-whisper 模型 + 真实音频素材验证
|
||||
端到端转写（`tests/test_integration_whisper.py`）；本地无模型或素材时跳过，
|
||||
有则必须执行，作为对假模型单测的校准。
|
||||
- **测试资产存放 `testdata/`**：图片（`ocr_text.png`）、语音（`speech_60s.wav`）
|
||||
等测试媒体一次性生成后入库，测试直接复用，**禁止在测试执行时再生成**；
|
||||
缺失时测试跳过而非现场生成。大体积视频素材放 `data/testdata/`（gitignored）。OCR 相关资产：
|
||||
`ocr_text.png`（有文字）、`ocr_notext.png`（无文字帧）、`subtitle_10s.mp4`
|
||||
（烧录 SUB 001@1-4s / SUB 002@6-9s 的 10s 测试视频）、`test_real_hav_sub.png`
|
||||
（真实视频字幕截图，VLM 集成测试期望识别出"还有没有什么困扰 或者奇怪的地方吗"）。
|
||||
- **开发流程强制 TDD（红-绿-重构）**：任何新功能/修复必须先写失败测试（红），
|
||||
再实现最小代码让其通过（绿），最后重构保持整洁；不允许先写实现后补测试。
|
||||
- 测试运行：`uv run pytest`；全部测试位于 `tests/`。
|
||||
- 100% 行覆盖率只保证代码路径被覆盖，不覆盖端口占用、防火墙、权限等
|
||||
外部环境状态；端口问题用启动检查、端口检查与 uvicorn 冒烟测试补充。
|
||||
- 本地出现 `WinError 10013` / `WinError 10048` 时，先用
|
||||
`netstat -ano | findstr :<port>` 确认是否有残留监听进程。
|
||||
|
||||
## 代码注释规范
|
||||
|
||||
- 本仓库所有源码（Python、JavaScript、HTML、CSS、TOML 等支持注释的文件）
|
||||
必须配有详细中文注释，说明模块/文件职责、核心类与函数的作用以及关键逻辑，
|
||||
确保后续维护人员无需通读全部实现即可快速理解工作原理。
|
||||
- 新增或修改代码时，必须同步补充或更新对应注释；不得删除已有注释。
|
||||
- 测试代码同样必须配有中文注释，说明每条测试验证的行为与覆盖的路径。
|
||||
- JSON 数据文件（`manifests/*.json`）按 JSON 规范不支持注释，字段语义以
|
||||
`src/wov_sdk/models.py` 的 `NodeManifest` 模型注释和本文档为准；
|
||||
修改 JSON 字段时须同步更新文档。
|
||||
|
||||
## 目标运行环境
|
||||
|
||||
- 本服务的最终部署目标是 Linux，通常以 Docker/Kubernetes 容器运行。
|
||||
- 当前 Windows 只作为本地开发环境，不允许在业务代码中写死 Windows 路径、
|
||||
盘符或 Windows 专用命令。
|
||||
- 路径处理统一使用 `pathlib`。
|
||||
- ffmpeg 在 Linux 上可使用系统包，也允许通过 `imageio-ffmpeg` 使用内置
|
||||
二进制，节点代码不能假设 ffmpeg 一定在 PATH。
|
||||
- 测试必须可以在 Windows 和 Linux 上运行；涉及平台分支的代码应同时覆盖
|
||||
两种路径解析。
|
||||
|
||||
## Windows / PowerShell 执行规则
|
||||
|
||||
- 默认 shell 视为 Windows PowerShell 5.1；不要假设 Bash、zsh 或 PowerShell 7。
|
||||
必要时先查 `$PSVersionTable.PSVersion`。
|
||||
- 禁止把 Bash 语法交给 PowerShell：`python - <<'PY'`、`cat <<EOF`、`export`、
|
||||
`source`、`rm -rf`、Bash 后台 `&` 等。
|
||||
- PowerShell 中 `&` 是调用运算符；URL 或参数含 `&` 时整体单引号引用。
|
||||
- 避免 PowerShell 5.1 下使用 Bash 风格 `&&` / `||`；顺序步骤用多行 PowerShell。
|
||||
- 参数含空格、括号、中文、`&|;><$` 或引号时默认用单引号。
|
||||
- 外部程序路径可能有空格时，用 `& 'C:\path with spaces\tool.exe' arg1`。
|
||||
- 文件操作优先 PowerShell 原生命令和 `-LiteralPath`。
|
||||
- 复杂 Python 不用 `python -c`；涉及 SQL、JSON、中文、反斜杠路径、换行或
|
||||
多层引号时，用仓库脚本或临时 `.py` 文件。
|
||||
- 禁止在 PowerShell 用 Bash here-doc。临时传 Python 源码只允许 PowerShell
|
||||
here-string，且尽量保持 ASCII。
|
||||
- Python 源码含中文常量时，不通过 PowerShell 管道传给 `python -`；用 UTF-8
|
||||
脚本文件、仓库脚本或 `\uXXXX`。
|
||||
- 搜索文本/文件优先 `rg` / `rg --files`。
|
||||
- 数据库或生产内容写操作前先查询当前数据；写入必须有明确筛选条件，禁止
|
||||
无条件 `DELETE` / `UPDATE`。
|
||||
- 同一 PowerShell 命令连续失败两次后，停止微调长命令；改短命令、脚本文件、
|
||||
数组 splatting 或分步验证。
|
||||
|
||||
## 关键设计约束（北极星不变式）
|
||||
|
||||
- 节点之间不直接调用，只通过产物 URI 交换数据；中间产物落在共享存储
|
||||
（`data/storage`），不放在节点模块内部。
|
||||
- 工作流必须是数据文件或数据库记录（workflow_versions 表存 DAG JSON），
|
||||
不允许把步骤顺序写死在应用代码里。
|
||||
- 节点注册表是节点调用的唯一入口；API 与调度器不绕过 registry 直接执行
|
||||
节点逻辑。
|
||||
- 协议数据模型（wov_sdk）要长期稳定，宁可先少做功能，也不轻易改协议。
|
||||
- 存储、队列、调度器都要通过抽象边界隔离，方便从单机实现替换为分布式实现。
|
||||
- 用户端永远只看到"输入 -> 进度 -> 结果"，不暴露工作流细节。
|
||||
- 单体对分布式版的三处降级：无子进程隔离、无空闲 TTL 回收（模型常驻，
|
||||
仅懒加载）、慢任务无法强制中断（由节点自身超时兜底）。
|
||||
|
||||
## 单体化说明
|
||||
|
||||
- 由原 7 个独立仓库合并：wov-api、wov-web、wov-sdk、wov-node-echo、
|
||||
wov-node-ffmpeg、wov-node-whisper、wov-node-llm、wov-node-ass。
|
||||
- 删除内容：`NodeManager`（子进程生命周期）、节点 HTTP 服务端
|
||||
（`wov_sdk.server`）、节点注册/实例管理 API 与页面、node_instances 表、
|
||||
各节点的 `__main__` 进程入口。
|
||||
- 保留内容：协议数据模型、工作流 DAG 数据化、调度拓扑执行、上传/进度/下载/
|
||||
重试 API、静态前端、SQLite Repository 层、本地优先模型加载。
|
||||
@@ -0,0 +1,45 @@
|
||||
# VRSub
|
||||
|
||||
为视频生成 **VR 双眼字幕**的单体应用：API、调度器与全部节点在同一进程内运行，
|
||||
单仓库、单环境、单命令启动。由原分布式 WOV 多仓库合并而来，详细约定见
|
||||
[AGENTS.md](./AGENTS.md)。
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
uv run uvicorn wov_app.main:app --reload
|
||||
```
|
||||
|
||||
访问 `http://127.0.0.1:8000/` 上传视频，自动执行"提音 → 转写 → 翻译 → ASS"。
|
||||
|
||||
- 模型权重本地优先：把 whisper 模型放在 `model/faster-whisper-large-v3`
|
||||
（含 `model.bin`）即可完全离线运行。
|
||||
- LLM 翻译默认使用 SiliconFlow（`Qwen/Qwen3.6-35B-A3B`），API Key 放在
|
||||
gitignored 的 `.env`（`LLM_API_KEY=sk-...`），启动时自动加载；也可用
|
||||
`LLM_API_BASE` / `LLM_MODEL` 环境变量覆盖。
|
||||
|
||||
## 目录
|
||||
|
||||
| 路径 | 说明 |
|
||||
| --- | --- |
|
||||
| `src/wov_sdk/` | 协议数据模型（与分布式版兼容） |
|
||||
| `src/wov_app/` | 应用层：API、注册表、调度器、数据库 |
|
||||
| `nodes/` | 进程内节点实现（echo/ffmpeg/whisper/llm/ass） |
|
||||
| `manifests/` | 节点清单 JSON |
|
||||
| `web/` | 静态前端 |
|
||||
| `model/` | 本地模型权重（gitignored） |
|
||||
| `data/` | SQLite 与存储（gitignored） |
|
||||
| `tests/` | 测试（100% 行覆盖率） |
|
||||
|
||||
## 测试
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
## 与分布式版的关系
|
||||
|
||||
- 协议数据模型、工作流 DAG 数据化、调度拓扑执行保持不变。
|
||||
- 已移除：子进程节点、节点 HTTP 协议、节点注册/实例管理、TTL 回收。
|
||||
- 未来回退分布式时，只需为 `registry.invoke` 重新加上进程边界。
|
||||
Executable
+21
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"id": "srt-to-dual-eye-ass",
|
||||
"name": "SRT to Dual-Eye ASS",
|
||||
"version": "0.1.0",
|
||||
"capability": "subtitle",
|
||||
"repo_dir": "wov-node-ass",
|
||||
"command": ["python", "-m", "wov_node_ass"],
|
||||
"env": {
|
||||
"WOV_NODE_PORT": "0"
|
||||
},
|
||||
"input_schema": {
|
||||
"cn_srt_uri": "file"
|
||||
},
|
||||
"output_schema": {
|
||||
"ass_uri": "file"
|
||||
},
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 60,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
Executable
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"id": "echo",
|
||||
"name": "Echo Node",
|
||||
"version": "0.1.0",
|
||||
"capability": "echo",
|
||||
"repo_dir": "wov-node-echo",
|
||||
"command": ["python", "-m", "wov_node_echo"],
|
||||
"env": {
|
||||
"WOV_NODE_PORT": "0"
|
||||
},
|
||||
"input_schema": {
|
||||
"text": "string"
|
||||
},
|
||||
"output_schema": {
|
||||
"text": "string",
|
||||
"file_uri": "file"
|
||||
},
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 15,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
Executable
+21
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"id": "ffmpeg-extract",
|
||||
"name": "FFmpeg Audio Extract",
|
||||
"version": "0.1.0",
|
||||
"capability": "media",
|
||||
"repo_dir": "wov-node-ffmpeg",
|
||||
"command": ["python", "-m", "wov_node_ffmpeg"],
|
||||
"env": {
|
||||
"WOV_NODE_PORT": "0"
|
||||
},
|
||||
"input_schema": {
|
||||
"video_uri": "file"
|
||||
},
|
||||
"output_schema": {
|
||||
"audio_uri": "file"
|
||||
},
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 60,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"id": "frame-extract",
|
||||
"name": "Frame Extract",
|
||||
"version": "0.1.0",
|
||||
"capability": "frame-extract",
|
||||
"repo_dir": "nodes",
|
||||
"command": ["python", "-m", "frame_extract"],
|
||||
"env": {},
|
||||
"input_schema": { "video_uri": "file" },
|
||||
"output_schema": { "frames_manifest": "file", "frame_count": "integer" },
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 300,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"id": "llm-filter",
|
||||
"name": "LLM Subtitle Filter",
|
||||
"version": "0.1.0",
|
||||
"capability": "llm-filter",
|
||||
"repo_dir": "nodes",
|
||||
"command": ["python", "-m", "llm_filter"],
|
||||
"env": {},
|
||||
"input_schema": { "srt_uri": "file" },
|
||||
"output_schema": { "srt_uri": "file", "kept": "integer", "removed": "integer" },
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 300,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
Executable
+21
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"id": "llm-translate",
|
||||
"name": "LLM Subtitle Translate",
|
||||
"version": "0.1.0",
|
||||
"capability": "llm",
|
||||
"repo_dir": "wov-node-llm",
|
||||
"command": ["python", "-m", "wov_node_llm"],
|
||||
"env": {
|
||||
"WOV_NODE_PORT": "0"
|
||||
},
|
||||
"input_schema": {
|
||||
"srt_uri": "file"
|
||||
},
|
||||
"output_schema": {
|
||||
"cn_srt_uri": "file"
|
||||
},
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 60,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"id": "subtitle-ocr",
|
||||
"name": "Subtitle OCR",
|
||||
"version": "0.1.0",
|
||||
"capability": "subtitle-ocr",
|
||||
"repo_dir": "nodes",
|
||||
"command": ["python", "-m", "subtitle_ocr"],
|
||||
"env": {},
|
||||
"input_schema": { "frames_manifest": "file" },
|
||||
"output_schema": { "srt_uri": "file", "count": "integer" },
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 300,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"id": "vlm-ocr",
|
||||
"name": "VLM OCR",
|
||||
"version": "0.1.0",
|
||||
"capability": "ocr",
|
||||
"repo_dir": "nodes",
|
||||
"command": ["python", "-m", "vlm_ocr"],
|
||||
"env": {},
|
||||
"input_schema": { "image_uri": "file" },
|
||||
"output_schema": { "text": "string", "text_uri": "file" },
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 300,
|
||||
"health_timeout_seconds": 10,
|
||||
"keep_warm": false
|
||||
}
|
||||
Executable
+21
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"id": "faster-whisper",
|
||||
"name": "Faster Whisper ASR",
|
||||
"version": "0.1.0",
|
||||
"capability": "asr",
|
||||
"repo_dir": "wov-node-whisper",
|
||||
"command": ["python", "-m", "wov_node_whisper"],
|
||||
"env": {
|
||||
"WOV_NODE_PORT": "0"
|
||||
},
|
||||
"input_schema": {
|
||||
"audio_uri": "file"
|
||||
},
|
||||
"output_schema": {
|
||||
"srt_uri": "file"
|
||||
},
|
||||
"max_concurrency": 1,
|
||||
"idle_ttl_seconds": 300,
|
||||
"health_timeout_seconds": 30,
|
||||
"keep_warm": false
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
"""进程内节点实现包。
|
||||
|
||||
每个模块对应一个节点，提供统一的 invoke(request) -> InvokeResponse 处理器，
|
||||
由 wov_app.registry 在启动时静态注册，调度器按 node_type 直接调用。
|
||||
"""
|
||||
|
||||
from nodes import ass, echo, ffmpeg, llm, whisper
|
||||
|
||||
__all__ = ["ass", "echo", "ffmpeg", "llm", "whisper"]
|
||||
@@ -0,0 +1,177 @@
|
||||
"""自适应线程池。
|
||||
|
||||
用于对耗时的独立子任务（如逐帧 VLM OCR）做弹性并发加速：
|
||||
- 以滚动时间窗口统计已完成任务的平均响应时间；
|
||||
- 窗口内平均响应 < fast_threshold（默认 0.3s）→ 增加 1 个工作线程（上限 max_workers）；
|
||||
- 窗口内平均响应 > slow_threshold（默认 1.0s）→ 减少 1 个工作线程（下限 min_workers）。
|
||||
|
||||
线程数从 min_workers（默认 1）起步，按实测负载自适应：服务端空闲（响应快）
|
||||
就加大并发，服务端变慢就退避，避免盲目并发压垮上游（如本地 Ollama）。
|
||||
|
||||
线程安全说明：worker 会在多个线程中并发调用，调用方需保证 worker 无共享
|
||||
可变状态（registry 处理器是纯函数，符合要求）；结果按输入顺序返回。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import Callable
|
||||
|
||||
# 停止哨兵：压入队列让空闲工作线程退出（用于缩容）。
|
||||
_POISON = object()
|
||||
|
||||
|
||||
def decide(
|
||||
current: int,
|
||||
avg: float,
|
||||
min_workers: int,
|
||||
max_workers: int,
|
||||
fast_threshold: float,
|
||||
slow_threshold: float,
|
||||
) -> int:
|
||||
"""根据窗口平均响应时间返回调整后的目标线程数（纯决策函数）。
|
||||
|
||||
响应快（avg < fast_threshold）且未达上限 → 加 1；响应慢
|
||||
（avg > slow_threshold）且未达下限 → 减 1；其余情况保持不变。
|
||||
"""
|
||||
if avg < fast_threshold and current < max_workers:
|
||||
return current + 1
|
||||
if avg > slow_threshold and current > min_workers:
|
||||
return current - 1
|
||||
return current
|
||||
|
||||
|
||||
class AdaptiveThreadPool:
|
||||
"""自适应线程池：单次 map 按输入顺序返回全部结果。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
worker: Callable,
|
||||
min_workers: int = 1,
|
||||
max_workers: int = 16,
|
||||
window_seconds: float = 10.0,
|
||||
fast_threshold: float = 0.3,
|
||||
slow_threshold: float = 1.0,
|
||||
clock=time.monotonic,
|
||||
on_progress: Callable[[int, int, float], None] | None = None,
|
||||
) -> None:
|
||||
"""初始化；clock 可注入便于测试；on_progress(done,total,rate) 每次完成回调。"""
|
||||
self._worker = worker
|
||||
self.min_workers = max(1, min_workers)
|
||||
self.max_workers = max(self.min_workers, max_workers)
|
||||
self.window_seconds = window_seconds
|
||||
self.fast_threshold = fast_threshold
|
||||
self.slow_threshold = slow_threshold
|
||||
self._clock = clock
|
||||
self._queue: queue.Queue = queue.Queue()
|
||||
# 并发目标线程数：决策/缩容的权威依据（线程退出是异步的，不能用
|
||||
# len(_threads) 判断，否则并发缩容会重复放哨兵把全部线程毒死）。
|
||||
self._target_workers = 0
|
||||
self._threads: list[threading.Thread] = []
|
||||
self._results: list = []
|
||||
self._lock = threading.Lock()
|
||||
self._stop = threading.Event()
|
||||
# 滚动窗口起点与已记录的单次耗时。
|
||||
self._window_start = clock()
|
||||
# 观测到的最大并发线程数（供测试与监控）。
|
||||
self.max_concurrency = 0
|
||||
# 进度回调与计数：on_progress(已完成数, 总数, 平均速度/秒)。
|
||||
self._on_progress = on_progress
|
||||
self._completed = 0
|
||||
self._total = 0
|
||||
self._started_at = 0.0
|
||||
self._window_times: list[float] = []
|
||||
|
||||
def _run(self) -> None:
|
||||
"""工作线程主循环：取任务 → 执行 → 记录耗时并自适应评估。"""
|
||||
try:
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
seq, item = self._queue.get(timeout=0.2)
|
||||
except queue.Empty:
|
||||
continue
|
||||
if item is _POISON:
|
||||
# 缩容哨兵：处理完即可退出（队列计数照常）。
|
||||
self._queue.task_done()
|
||||
break
|
||||
start = self._clock()
|
||||
try:
|
||||
result = self._worker(item)
|
||||
except Exception as exc:
|
||||
# 单任务异常不拖垮整体：以异常对象作为结果，由调用方判定。
|
||||
result = exc
|
||||
finally:
|
||||
elapsed = self._clock() - start
|
||||
self._results.append((seq, result))
|
||||
# 进度回调：已完成数、总数与平均处理速度（条/秒）。
|
||||
self._completed += 1
|
||||
if self._on_progress is not None:
|
||||
elapsed_total = max(self._clock() - self._started_at, 1e-9)
|
||||
self._on_progress(
|
||||
self._completed, self._total, self._completed / elapsed_total
|
||||
)
|
||||
self._tick(elapsed)
|
||||
self._queue.task_done()
|
||||
finally:
|
||||
# 无论何种退出路径都从线程列表移除，保证线程数统计准确。
|
||||
with self._lock:
|
||||
if threading.current_thread() in self._threads:
|
||||
self._threads.remove(threading.current_thread())
|
||||
|
||||
def _tick(self, elapsed: float) -> None:
|
||||
"""记录一次完成耗时；窗口满时按平均响应时间调整线程数。"""
|
||||
self._window_times.append(elapsed)
|
||||
if self._clock() - self._window_start < self.window_seconds:
|
||||
return
|
||||
avg = sum(self._window_times) / len(self._window_times)
|
||||
self._window_start = self._clock()
|
||||
self._window_times.clear()
|
||||
with self._lock:
|
||||
current = self._target_workers
|
||||
self._resize(
|
||||
decide(
|
||||
current, avg, self.min_workers, self.max_workers,
|
||||
self.fast_threshold, self.slow_threshold,
|
||||
)
|
||||
)
|
||||
|
||||
def _resize(self, target: int) -> None:
|
||||
"""调整并发目标：扩容启动新线程；缩容压入等量停止哨兵（幂等）。
|
||||
|
||||
以 _target_workers 为当前值：重复调用同一 target 不会重复放哨兵，
|
||||
避免并发缩容把所有线程毒死导致队列任务无人处理而挂起。
|
||||
"""
|
||||
with self._lock:
|
||||
current = self._target_workers
|
||||
if target > current:
|
||||
self.max_concurrency = max(self.max_concurrency, target)
|
||||
for _ in range(target - current):
|
||||
thread = threading.Thread(target=self._run, daemon=True)
|
||||
thread.start()
|
||||
self._threads.append(thread)
|
||||
self._target_workers = target
|
||||
elif target < current:
|
||||
for _ in range(current - target):
|
||||
self._queue.put((None, _POISON))
|
||||
self._target_workers = target
|
||||
|
||||
def map(self, items) -> list:
|
||||
"""按输入顺序返回每个 item 经 worker 处理后的结果列表。"""
|
||||
self._results = []
|
||||
self._completed = 0
|
||||
self._total = len(items)
|
||||
self._started_at = self._clock()
|
||||
self._stop.clear()
|
||||
self._resize(self.min_workers)
|
||||
for seq, item in enumerate(items):
|
||||
self._queue.put((seq, item))
|
||||
self._queue.join()
|
||||
self._stop.set()
|
||||
with self._lock:
|
||||
threads = list(self._threads)
|
||||
for thread in threads:
|
||||
thread.join(1.0)
|
||||
self._results.sort(key=lambda pair: pair[0])
|
||||
return [result for _, result in self._results]
|
||||
Executable
+98
@@ -0,0 +1,98 @@
|
||||
"""SRT 转 ASS 节点。
|
||||
|
||||
单体版中作为进程内节点模块，由调度器直接调用。解析标准 SRT 后生成 ASS
|
||||
文件，其中同一句字幕同时输出 LeftEye 与 RightEye 两个样式，分别落在屏幕
|
||||
左右两半，形成 VR 双眼叠加效果。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
def _ass_header(resolution: str) -> str:
|
||||
"""生成 ASS 文件头：脚本信息、左右眼样式和事件格式。"""
|
||||
width, height = resolution.lower().split("x", 1)
|
||||
# 左眼样式占左半边，右眼样式占右半边，各留 50px 内边距。
|
||||
left_margin = 50
|
||||
right_margin = int(width) - 50
|
||||
return f"""[Script Info]
|
||||
Title: VR Dual-Eye Subtitle
|
||||
ScriptType: v4.00+
|
||||
Collisions: Normal
|
||||
PlayResX: {width}
|
||||
PlayResY: {height}
|
||||
WrapStyle: 1
|
||||
ScaledBorderAndShadow: yes
|
||||
|
||||
[V4+ Styles]
|
||||
Format: Name,Fontname,Fontsize,PrimaryColour,SecondaryColour,OutlineColour,BackColour,Bold,Italic,Underline,StrikeOut,ScaleX,ScaleY,Spacing,Angle,BorderStyle,Outline,Shadow,Alignment,MarginL,MarginR,MarginV,Encoding
|
||||
Style: LeftEye,Arial,50,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,50,100,0,0,1,4,0,2,{left_margin},{int(width) // 2},{int(height) // 2 + 60},1
|
||||
Style: RightEye,Arial,50,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,50,100,0,0,1,4,0,2,{int(width) // 2},{right_margin},{int(height) // 2 + 60},1
|
||||
|
||||
[Events]
|
||||
Format: Layer,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text
|
||||
"""
|
||||
|
||||
|
||||
def parse_srt(text: str) -> list[tuple[str, str, str]]:
|
||||
"""把 SRT 文本解析为 (开始时间, 结束时间, 文本) 条目列表。"""
|
||||
entries: list[tuple[str, str, str]] = []
|
||||
lines = text.splitlines()
|
||||
index = 0
|
||||
while index < len(lines):
|
||||
# 跳过序号前的空行，兼容文件开头有换行的情况。
|
||||
if not lines[index].strip():
|
||||
index += 1
|
||||
continue
|
||||
# 跳过序号行，直接读取下一行时间轴。
|
||||
index += 1
|
||||
if index >= len(lines):
|
||||
break
|
||||
time_line = lines[index].strip()
|
||||
index += 1
|
||||
# 时间轴必须包含分隔符，否则按畸形输入跳过。
|
||||
if " --> " not in time_line:
|
||||
continue
|
||||
# SRT 使用逗号毫秒，ASS 使用点号，需要转换。
|
||||
start, end = [part.replace(",", ".") for part in time_line.split(" --> ")]
|
||||
# 连续读取非空行作为字幕文本，多行用 ASS 换行符 \N 连接。
|
||||
text_lines: list[str] = []
|
||||
while index < len(lines) and lines[index].strip():
|
||||
text_lines.append(lines[index])
|
||||
index += 1
|
||||
entries.append((start, end, r"\N".join(text_lines)))
|
||||
index += 1
|
||||
return entries
|
||||
|
||||
|
||||
def write_ass(entries: list[tuple[str, str, str]], output_path: Path, resolution: str) -> None:
|
||||
"""把解析后的条目写入 ASS 文件，每个条目输出左右眼两行 Dialogue。"""
|
||||
lines = [_ass_header(resolution)]
|
||||
for start, end, text in entries:
|
||||
# an2 对齐到屏幕中央偏下，保证双眼字幕视线自然。
|
||||
lines.append(f"Dialogue: 0,{start},{end},LeftEye,,0,0,0,,{{\\an2}}{text}")
|
||||
lines.append(f"Dialogue: 0,{start},{end},RightEye,,0,0,0,,{{\\an2}}{text}")
|
||||
output_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""把 cn_srt_uri 指向的 SRT 转为 dual_eye.ass 产物。"""
|
||||
srt_uri = request.inputs.get("cn_srt_uri")
|
||||
if not srt_uri:
|
||||
return InvokeResponse(status="failed", error="cn_srt_uri is required")
|
||||
|
||||
srt_path = Path(srt_uri)
|
||||
if not srt_path.is_file():
|
||||
return InvokeResponse(status="failed", error="srt file not found")
|
||||
|
||||
entries = parse_srt(srt_path.read_text(encoding="utf-8"))
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / "dual_eye.ass"
|
||||
# 分辨率默认 3840x1920，覆盖常见 VR 视频尺寸。
|
||||
resolution = str(request.params.get("resolution", "3840x1920"))
|
||||
write_ass(entries, output_path, resolution)
|
||||
return InvokeResponse(status="completed", outputs={"ass_uri": str(output_path)})
|
||||
|
||||
Executable
+50
@@ -0,0 +1,50 @@
|
||||
"""Echo 节点。
|
||||
|
||||
单体版中作为进程内节点模块存在，由调度器直接调用 invoke 处理器，不再启动
|
||||
独立 HTTP 服务。保留该节点用于验证节点协议与注册表链路。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
def _resolve_input_text(request: InvokeRequest, node_root: Path) -> str:
|
||||
"""按优先级解析输入文本：直接文本 > 文件 URI > 默认字符串。"""
|
||||
# 优先使用请求中直接携带的 text 字段。
|
||||
text = request.inputs.get("text")
|
||||
if text is not None:
|
||||
return str(text)
|
||||
|
||||
# 其次读取 file_uri 指向的文件；相对路径以单体根目录为基准。
|
||||
file_uri = request.inputs.get("file_uri")
|
||||
if file_uri:
|
||||
path = Path(file_uri)
|
||||
if not path.is_absolute():
|
||||
path = node_root / path
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
# 都没有时返回固定文本，保证节点总有可演示的输出。
|
||||
return "echo"
|
||||
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""处理节点调用：把解析出的文本写入产物并返回 URI。"""
|
||||
# 单体根目录用于解析相对文件路径（nodes/ 的上一级）。
|
||||
node_root = Path(__file__).resolve().parent.parent
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
text = _resolve_input_text(request, node_root)
|
||||
# 产物必须落在请求给定的 output_dir，调度器按 run 与节点组织目录。
|
||||
output_path = output_dir / "echo.txt"
|
||||
output_path.write_text(text, encoding="utf-8")
|
||||
|
||||
return InvokeResponse(
|
||||
status="completed",
|
||||
outputs={
|
||||
"text": text,
|
||||
"file_uri": str(output_path),
|
||||
},
|
||||
)
|
||||
Executable
+79
@@ -0,0 +1,79 @@
|
||||
"""FFmpeg 提音节点。
|
||||
|
||||
单体版中作为进程内节点模块，由调度器直接调用。ffmpeg 解析顺序为：
|
||||
FFMPEG_BIN 环境变量 > PATH 中的 ffmpeg > imageio-ffmpeg 内置二进制。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
def _bundled_ffmpeg() -> str | None:
|
||||
"""尝试获取 imageio-ffmpeg 内置的 ffmpeg 可执行文件路径。"""
|
||||
try:
|
||||
import imageio_ffmpeg
|
||||
|
||||
return imageio_ffmpeg.get_ffmpeg_exe()
|
||||
except Exception: # noqa: BLE001
|
||||
# 未安装 imageio-ffmpeg 或获取失败时返回 None，交由上层回退。
|
||||
return None
|
||||
|
||||
|
||||
def _ffmpeg_bin() -> str:
|
||||
"""按优先级解析 ffmpeg 可执行文件，返回最终命令路径。"""
|
||||
# 显式配置优先，便于部署环境指定自定义二进制。
|
||||
configured = os.getenv("FFMPEG_BIN")
|
||||
if configured:
|
||||
return configured
|
||||
# 其次查找 PATH 中的系统 ffmpeg。
|
||||
found = shutil.which("ffmpeg")
|
||||
if found:
|
||||
return found
|
||||
# 最后回退到 imageio-ffmpeg 内置二进制；都没有时保留 "ffmpeg" 交给调用失败处理。
|
||||
return _bundled_ffmpeg() or "ffmpeg"
|
||||
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""提取输入视频/音频的标准化音频，产物为 audio.wav。"""
|
||||
video_uri = request.inputs.get("video_uri")
|
||||
if not video_uri:
|
||||
return InvokeResponse(status="failed", error="video_uri is required")
|
||||
|
||||
# 找不到可用 ffmpeg 时直接返回失败，避免子进程报晦涩错误。
|
||||
ffmpeg = _ffmpeg_bin()
|
||||
if shutil.which(ffmpeg) is None and not Path(ffmpeg).is_file():
|
||||
return InvokeResponse(status="failed", error="ffmpeg not found")
|
||||
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / "audio.wav"
|
||||
# ASR 节点默认期望 16kHz 单声道；参数可覆盖。
|
||||
channels = str(request.params.get("channels", 1))
|
||||
sample_rate = str(request.params.get("sample_rate", 16000))
|
||||
command = [
|
||||
ffmpeg,
|
||||
"-y", # 覆盖可能存在的同名输出文件。
|
||||
"-i",
|
||||
str(video_uri),
|
||||
"-vn", # 丢弃视频流，只保留音频。
|
||||
"-ac",
|
||||
channels,
|
||||
"-ar",
|
||||
sample_rate,
|
||||
str(output_path),
|
||||
]
|
||||
result = subprocess.run(command, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
# 返回 stderr 尾部，保留最有诊断价值的错误信息。
|
||||
return InvokeResponse(
|
||||
status="failed",
|
||||
error=result.stderr[-2000:] or "ffmpeg failed",
|
||||
)
|
||||
return InvokeResponse(status="completed", outputs={"audio_uri": str(output_path)})
|
||||
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
"""视频抽帧节点。
|
||||
|
||||
按**帧间隔**从视频抽取帧：解析视频帧率后，帧间隔 step = round(间隔秒 × fps)，
|
||||
用 ffmpeg 的 select 过滤器按帧号（n mod step == 0）精确取帧——每帧都是真实
|
||||
视频帧，帧时间 = 帧号 / fps，避免按时间 seek（-ss 秒）造成的取整漂移。
|
||||
再按 crop 相对比例 [x,y,w,h]（0~1）裁切出字幕区域，供 vlm-ocr 等下游节点
|
||||
使用。产物为 frames.json 清单：[{"time": 帧时间秒, "image_uri": 帧图片路径}, ...]。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from nodes.ffmpeg import _ffmpeg_bin
|
||||
from wov_app.logging import get_logger
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
logger = get_logger("frame-extract")
|
||||
|
||||
# 默认裁切：画面底部 18% 区域（常见字幕位置）。
|
||||
DEFAULT_CROP = [0.0, 0.82, 1.0, 0.18]
|
||||
|
||||
|
||||
def _parse_crop(raw) -> list[float] | None:
|
||||
"""解析并校验 crop 相对比例 [x,y,w,h]（0~1 且不越出画面）。"""
|
||||
try:
|
||||
crop = [float(value) for value in raw]
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if len(crop) != 4:
|
||||
return None
|
||||
x, y, w, h = crop
|
||||
if not (0 <= x <= 1 and 0 <= y <= 1 and 0 <= w <= 1 and 0 <= h <= 1):
|
||||
return None
|
||||
if x + w > 1.001 or y + h > 1.001:
|
||||
return None
|
||||
return crop
|
||||
|
||||
|
||||
def _video_size(video: Path, ffmpeg_bin: str) -> tuple[int, int] | None:
|
||||
"""从 ffmpeg -i 输出解析视频分辨率，避免依赖 ffprobe。"""
|
||||
result = subprocess.run(
|
||||
[ffmpeg_bin, "-i", str(video)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
for line in (result.stderr or "").splitlines():
|
||||
if "Video:" not in line:
|
||||
continue
|
||||
match = re.search(r"(\d{2,5})x(\d{2,5})", line)
|
||||
if match:
|
||||
return int(match.group(1)), int(match.group(2))
|
||||
return None
|
||||
|
||||
|
||||
def _video_duration(video: Path, ffmpeg_bin: str) -> float | None:
|
||||
"""从 ffmpeg -i 输出解析总时长（秒）。"""
|
||||
result = subprocess.run(
|
||||
[ffmpeg_bin, "-i", str(video)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
match = re.search(r"Duration:\s*(\d+):(\d+):(\d+(?:\.\d+)?)", result.stderr or "")
|
||||
if match:
|
||||
hours = int(match.group(1))
|
||||
minutes = int(match.group(2))
|
||||
seconds = float(match.group(3))
|
||||
return hours * 3600 + minutes * 60 + seconds
|
||||
return None
|
||||
|
||||
|
||||
def _video_fps(video: Path, ffmpeg_bin: str) -> float | None:
|
||||
"""从 ffmpeg -i 输出解析帧率，支持小数（29.97 fps）与有理数（30000/1001 fps）。"""
|
||||
result = subprocess.run(
|
||||
[ffmpeg_bin, "-i", str(video)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
for line in (result.stderr or "").splitlines():
|
||||
if "Video:" not in line:
|
||||
continue
|
||||
match = re.search(r"(\d+)/(\d+)\s*fps", line)
|
||||
if match:
|
||||
return int(match.group(1)) / int(match.group(2))
|
||||
match = re.search(r"(\d+(?:\.\d+)?)\s*fps", line)
|
||||
if match:
|
||||
return float(match.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def _frame_step(fps: float, interval: float) -> int:
|
||||
"""帧间隔换算：每 step 帧取一帧（step=round(间隔秒×fps)，至少为 1）。"""
|
||||
return max(1, int(round(interval * fps)))
|
||||
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""按帧间隔抽取并裁切视频帧，输出 frames.json 清单。"""
|
||||
video_uri = request.inputs.get("video_uri")
|
||||
if not video_uri:
|
||||
return InvokeResponse(status="failed", error="video_uri is required")
|
||||
video = Path(video_uri)
|
||||
if not video.is_file():
|
||||
return InvokeResponse(status="failed", error="video file not found")
|
||||
|
||||
interval = float(request.params.get("interval_seconds", 0.5))
|
||||
crop = _parse_crop(request.params.get("crop", DEFAULT_CROP))
|
||||
if crop is None:
|
||||
return InvokeResponse(status="failed", error="invalid crop")
|
||||
if interval <= 0:
|
||||
return InvokeResponse(status="failed", error="invalid interval_seconds")
|
||||
|
||||
ffmpeg_bin = _ffmpeg_bin()
|
||||
size = _video_size(video, ffmpeg_bin)
|
||||
if size is None:
|
||||
return InvokeResponse(status="failed", error="cannot read video size")
|
||||
duration = _video_duration(video, ffmpeg_bin)
|
||||
if duration is None or duration <= 0:
|
||||
return InvokeResponse(status="failed", error="cannot read video duration")
|
||||
fps = _video_fps(video, ffmpeg_bin)
|
||||
if fps is None or fps <= 0:
|
||||
return InvokeResponse(status="failed", error="cannot read video fps")
|
||||
|
||||
# 把"每多少秒一帧"换算为"每多少帧取一帧"，按帧号取帧是帧精确的。
|
||||
step = _frame_step(fps, interval)
|
||||
width, height = size
|
||||
x_px = int(round(crop[0] * width))
|
||||
y_px = int(round(crop[1] * height))
|
||||
w_px = max(1, int(round(crop[2] * width)))
|
||||
h_px = max(1, int(round(crop[3] * height)))
|
||||
|
||||
output_dir = Path(request.output_dir)
|
||||
frames_dir = output_dir / "frames"
|
||||
frames_dir.mkdir(parents=True, exist_ok=True)
|
||||
# 单次解码全片：select 按帧号（n mod step == 0）精确取帧，随后对选中帧
|
||||
# 裁切字幕区域并压缩到 720p 内（仅缩小）——过大的输入图会触发 glm-ocr
|
||||
# 的重复循环 bug（M-RoPE delta），源头规避。
|
||||
result = subprocess.run(
|
||||
[
|
||||
ffmpeg_bin,
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-i",
|
||||
str(video),
|
||||
"-vf",
|
||||
(
|
||||
f"select='not(mod(n\\,{step}))',"
|
||||
f"crop={w_px}:{h_px}:{x_px}:{y_px},"
|
||||
"scale=1280:720:force_original_aspect_ratio=decrease:force_divisible_by=2"
|
||||
),
|
||||
# 只写出被选中的帧，避免 CFR 补帧产生重复文件。
|
||||
"-vsync",
|
||||
"vfr",
|
||||
str(frames_dir / "frame_%04d.png"),
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return InvokeResponse(status="failed", error=result.stderr[-500:] or "ffmpeg failed")
|
||||
|
||||
# 第 k 个输出文件对应原始帧号 k×step，时间 = 帧号 / fps（帧精确，无累计偏差）。
|
||||
files = sorted(frames_dir.glob("frame_*.png"))
|
||||
manifest = [
|
||||
{"time": round((index * step) / fps, 3), "image_uri": str(path)}
|
||||
for index, path in enumerate(files)
|
||||
]
|
||||
manifest_path = output_dir / "frames.json"
|
||||
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False), encoding="utf-8")
|
||||
logger.info("抽帧完成: %d 帧, 帧间隔 %d, 裁切 %dx%d+%d+%d", len(files), step, w_px, h_px, x_px, y_px)
|
||||
return InvokeResponse(
|
||||
status="completed",
|
||||
outputs={"frames_manifest": str(manifest_path), "frame_count": len(files)},
|
||||
)
|
||||
Executable
+103
@@ -0,0 +1,103 @@
|
||||
"""LLM 翻译节点。
|
||||
|
||||
单体版中作为进程内节点模块，由调度器直接调用。接收 SRT，提取纯文本行
|
||||
分批调用 LLM，再把译文回填到原 SRT 结构并输出 cn.srt。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
# 单次 LLM 请求携带的字幕行数；过大会超出模型上下文，过小则请求次数过多。
|
||||
CHUNK_SIZE = 20
|
||||
|
||||
|
||||
def translate_lines(lines: list[str], params: dict) -> list[str]:
|
||||
"""分批调用 LLM 翻译纯文本行，返回顺序一致的译文列表。"""
|
||||
# 接口地址、Key 和模型均可通过环境变量配置（.env 自动加载），
|
||||
# 默认指向 SiliconFlow 兼容接口，模型为 DeepSeek-V4-Flash。
|
||||
api_base = os.getenv(
|
||||
"LLM_API_BASE",
|
||||
"https://api.siliconflow.cn/v1/chat/completions",
|
||||
)
|
||||
api_key = os.getenv("LLM_API_KEY", "")
|
||||
# 单次请求超时可配置，长文本翻译场景下需要放宽。
|
||||
request_timeout = float(os.getenv("LLM_TIMEOUT_SECONDS", "600"))
|
||||
model = str(params.get("model") or os.getenv("LLM_MODEL", "Qwen/Qwen3.6-35B-A3B"))
|
||||
target_language = str(params.get("target_language", "zh-CN"))
|
||||
# 系统提示词约束模型只输出译文，保证行数和顺序可回填。
|
||||
system_prompt = (
|
||||
"你是专业字幕翻译。将用户提供的日文字幕翻译为"
|
||||
f"{target_language}。只返回译文，保持行数和顺序，不要添加解释。"
|
||||
)
|
||||
translated: list[str] = []
|
||||
# 按 CHUNK_SIZE 分批发送，避免单次请求超过模型上下文限制。
|
||||
for start in range(0, len(lines), CHUNK_SIZE):
|
||||
chunk = lines[start : start + CHUNK_SIZE]
|
||||
body = {
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": "\n".join(chunk)},
|
||||
],
|
||||
# 关闭推理模型的思考模式：Qwen3 等模型默认会把推理过程写入
|
||||
# reasoning_content，导致 content 为空或截断译文；关闭后直接输出译文。
|
||||
"enable_thinking": False,
|
||||
# 放宽输出上限，避免长批次翻译被模型默认 max_tokens 截断。
|
||||
"max_tokens": 8192,
|
||||
}
|
||||
headers = {"Content-Type": "application/json"}
|
||||
# 配置了 Key 时附带 Bearer 鉴权头。
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
request = urllib.request.Request(
|
||||
api_base,
|
||||
data=json.dumps(body).encode("utf-8"),
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=request_timeout) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
# 兼容 OpenAI Chat Completions 响应格式，取第一条消息内容。
|
||||
content = payload["choices"][0]["message"]["content"]
|
||||
# 忽略空行，保证译文列表与输入行一一对应。
|
||||
translated.extend(
|
||||
[line.strip() for line in content.splitlines() if line.strip()]
|
||||
)
|
||||
return translated
|
||||
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""翻译 SRT 文件中的字幕文本，输出 cn.srt。"""
|
||||
srt_uri = request.inputs.get("srt_uri")
|
||||
if not srt_uri:
|
||||
return InvokeResponse(status="failed", error="srt_uri is required")
|
||||
|
||||
srt_path = Path(srt_uri)
|
||||
if not srt_path.is_file():
|
||||
return InvokeResponse(status="failed", error="srt file not found")
|
||||
|
||||
# 标准 SRT 每 4 行一组：序号、时间轴、文本、空行；文本位于第 3 行。
|
||||
lines = srt_path.read_text(encoding="utf-8").splitlines()
|
||||
text_indices = list(range(2, len(lines), 4))
|
||||
source_lines = [lines[index] for index in text_indices]
|
||||
translated_lines = translate_lines(source_lines, request.params)
|
||||
# 防止模型返回行数偏差：多出的截断，缺少的用空串补齐。
|
||||
translated_lines = translated_lines[: len(source_lines)]
|
||||
translated_lines += [""] * max(0, len(source_lines) - len(translated_lines))
|
||||
# 只替换文本行，序号、时间轴和空行保持不变。
|
||||
for index, text_index in enumerate(text_indices):
|
||||
lines[text_index] = translated_lines[index]
|
||||
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / "cn.srt"
|
||||
# 末尾补一个换行，让文件满足常见文本工具习惯。
|
||||
output_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
return InvokeResponse(status="completed", outputs={"cn_srt_uri": str(output_path)})
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
"""LLM 字幕过滤节点。
|
||||
|
||||
对 OCR 识别出的 SRT 字幕做二次过滤：为避免字幕上下文过长，把每条字幕连同其
|
||||
前后各 context_size 条字幕（纯文本，**不含时间戳**）分批提供给 LLM，模型仅判断
|
||||
目标字幕是否属于多余、无意义的字符（如重复、残缺、无实际语义的杂项）；
|
||||
判定无意义则删除该条（连同其时间戳），其余字幕保持原样并重新编号输出。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from nodes.adaptive_pool import AdaptiveThreadPool
|
||||
from wov_app.logging import get_logger
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
logger = get_logger("llm-filter")
|
||||
|
||||
# 匹配 SRT 条目：时间轴行 + 文本（文本可多行），到下一个序号行或文末结束。
|
||||
_SRT_BLOCK_RE = re.compile(
|
||||
r"(\d{2}:\d{2}:\d{2},\d{3})\s*-->\s*(\d{2}:\d{2}:\d{2},\d{3})\s*\n(.*?)(?=\n\s*\d+\s*\n|\Z)",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
# 目标字幕标记：提示词用该标记指明需要判断的那一条字幕。
|
||||
TARGET_MARK = "【目标】"
|
||||
|
||||
# 默认上下文窗口：目标字幕前后各取 10 条。
|
||||
DEFAULT_CONTEXT_SIZE = 10
|
||||
|
||||
|
||||
def parse_srt(text: str) -> list[dict]:
|
||||
"""解析 SRT 文本为条目列表：[{"start", "end", "text"}]。"""
|
||||
entries: list[dict] = []
|
||||
for match in _SRT_BLOCK_RE.finditer(text):
|
||||
entries.append(
|
||||
{
|
||||
"start": match.group(1),
|
||||
"end": match.group(2),
|
||||
"text": match.group(3).strip(),
|
||||
}
|
||||
)
|
||||
return entries
|
||||
|
||||
|
||||
def serialize_srt(entries: list[dict]) -> str:
|
||||
"""把条目列表序列化为标准 SRT 文本（序号重新从 1 编号）。"""
|
||||
blocks = [
|
||||
f"{index}\n{entry['start']} --> {entry['end']}\n{entry['text']}"
|
||||
for index, entry in enumerate(entries, start=1)
|
||||
]
|
||||
return "\n\n".join(blocks) + "\n"
|
||||
|
||||
|
||||
def _judge_target(
|
||||
entries: list[dict], index: int, context_size: int, params: dict
|
||||
) -> bool:
|
||||
"""调用 LLM 判断目标字幕是否多余/无意义；返回 True 表示应删除。
|
||||
|
||||
请求体只含目标字幕及其前后各 context_size 条字幕的纯文本（无时间戳），
|
||||
目标字幕用 TARGET_MARK 标记；模型只需回答"保留"或"删除"。
|
||||
"""
|
||||
start = max(0, index - context_size)
|
||||
end = min(len(entries), index + context_size + 1)
|
||||
target_pos = index - start
|
||||
lines = [
|
||||
f"{TARGET_MARK}{text}" if pos == target_pos else text
|
||||
for pos, text in enumerate(entry["text"] for entry in entries[start:end])
|
||||
]
|
||||
|
||||
# LLM 兼容接口配置：地址/Key/模型/超时均可通过环境变量覆盖（默认 SiliconFlow）。
|
||||
api_base = os.getenv(
|
||||
"LLM_API_BASE",
|
||||
"https://api.siliconflow.cn/v1/chat/completions",
|
||||
)
|
||||
api_key = os.getenv("LLM_API_KEY", "")
|
||||
request_timeout = float(os.getenv("LLM_TIMEOUT_SECONDS", "60"))
|
||||
model = str(params.get("model") or os.getenv("LLM_MODEL", "Qwen/Qwen3.6-35B-A3B"))
|
||||
system_prompt = (
|
||||
"你是字幕质量过滤器。用户会提供一段字幕序列（纯文本，不含时间戳），"
|
||||
f"其中用{TARGET_MARK}标记的字幕是需要判断的目标。请判断该字幕是否属于"
|
||||
"多余、无意义的字符（如重复、残缺、无实际语义的杂项）。"
|
||||
"只回答两个字：保留 或 删除，不要输出其他内容。"
|
||||
)
|
||||
body = {
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": "\n".join(lines)},
|
||||
],
|
||||
# 关闭推理模式：Qwen3 等模型默认会把思考过程写入 reasoning_content，
|
||||
# 导致 content 为空或包含多余内容。
|
||||
"enable_thinking": False,
|
||||
# 只输出"保留/删除"，输出上限给得很小即可。
|
||||
"max_tokens": 16,
|
||||
}
|
||||
headers = {"Content-Type": "application/json"}
|
||||
# 配置了 Key 时附带 Bearer 鉴权头。
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
request = urllib.request.Request(
|
||||
api_base,
|
||||
data=json.dumps(body).encode("utf-8"),
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=request_timeout) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
content = str(payload["choices"][0]["message"]["content"])
|
||||
# 模型回答含"删除"即视为该条无意义；其余情况（保留/异常）一律保留，宁多勿删。
|
||||
return "删除" in content
|
||||
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""过滤 SRT 中多余/无意义的字幕，产物为 filtered.srt。"""
|
||||
srt_uri = request.inputs.get("srt_uri")
|
||||
if not srt_uri:
|
||||
return InvokeResponse(status="failed", error="srt_uri is required")
|
||||
srt_path = Path(srt_uri)
|
||||
if not srt_path.is_file():
|
||||
return InvokeResponse(status="failed", error="srt file not found")
|
||||
|
||||
entries = parse_srt(srt_path.read_text(encoding="utf-8"))
|
||||
context_size = int(request.params.get("context_size", DEFAULT_CONTEXT_SIZE))
|
||||
# 单条判断的工作函数：返回 True 表示该条应删除。
|
||||
def judge_one(index) -> bool:
|
||||
return _judge_target(entries, index, context_size, request.params)
|
||||
|
||||
# 进度日志：打印已判定条数、总数与平均处理速度（条/s）。
|
||||
def log_progress(done: int, total: int, rate: float) -> None:
|
||||
logger.info("字幕判定进度 %d/%d 条 (%.1f 条/s)", done, total, rate)
|
||||
|
||||
# 自适应并发调用 LLM：10s 窗口内平均响应 < 0.3s 则加 1 线程（上限
|
||||
# pool_max_workers），> pool_slow_threshold 则减 1 线程（下限 1），
|
||||
# 按实测负载弹性伸缩，避免盲目并发压垮 LLM 接口。
|
||||
pool = AdaptiveThreadPool(
|
||||
worker=judge_one,
|
||||
on_progress=log_progress,
|
||||
min_workers=int(request.params.get("pool_min_workers", 1)),
|
||||
max_workers=int(request.params.get("pool_max_workers", 16)),
|
||||
window_seconds=float(request.params.get("pool_window_seconds", 10.0)),
|
||||
fast_threshold=float(request.params.get("pool_fast_threshold", 0.3)),
|
||||
slow_threshold=float(request.params.get("pool_slow_threshold", 1.0)),
|
||||
)
|
||||
verdicts = pool.map(range(len(entries)))
|
||||
|
||||
kept: list[dict] = []
|
||||
removed = 0
|
||||
for index, (entry, verdict) in enumerate(zip(entries, verdicts)):
|
||||
# 并行下 LLM 异常被线程池隔离为异常结果：任一条失败即整体失败，
|
||||
# 避免静默输出未过滤结果。
|
||||
if isinstance(verdict, Exception):
|
||||
return InvokeResponse(status="failed", error=str(verdict))
|
||||
if verdict:
|
||||
removed += 1
|
||||
logger.info("删除无意义字幕 %d: %r", index + 1, entry["text"][:40])
|
||||
else:
|
||||
kept.append(entry)
|
||||
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / "filtered.srt"
|
||||
output_path.write_text(serialize_srt(kept), encoding="utf-8")
|
||||
logger.info("字幕过滤完成: 保留 %d 条, 删除 %d 条", len(kept), removed)
|
||||
return InvokeResponse(
|
||||
status="completed",
|
||||
outputs={"srt_uri": str(output_path), "kept": len(kept), "removed": removed},
|
||||
)
|
||||
@@ -0,0 +1,162 @@
|
||||
"""字幕 OCR 汇总节点。
|
||||
|
||||
读取 frame-extract 产出的 frames.json，逐帧调用 vlm-ocr 节点识别字幕文字；
|
||||
过滤无文字帧的垃圾输出（glm-ocr 在空帧上会输出无用文字），折叠模型重复
|
||||
循环输出，合并连续相同的字幕（记录最后可见帧时间），最终组装为带时间轴的
|
||||
SRT 基准数据：每条字幕消失时间 = 最后可见帧时间 + 采样间隔（间隔从帧清单
|
||||
时间轴推导），与视频烧录时间对齐。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from nodes.adaptive_pool import AdaptiveThreadPool
|
||||
from nodes.whisper import format_timestamp
|
||||
from wov_app import registry
|
||||
from wov_app.logging import get_logger
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
logger = get_logger("subtitle-ocr")
|
||||
|
||||
# 默认垃圾词：无文字帧的模型输出可能反复出现这些词。
|
||||
def _sampling_interval(manifest: list[dict], default: float) -> float:
|
||||
"""从帧清单时间轴推导采样间隔（相邻帧时间差的中位数）。
|
||||
|
||||
帧时间由 frame-extract 按固定间隔生成，取相邻差的中位数即可还原真实
|
||||
采样间隔，避免结束时间与抽取参数不一致。清单不足两帧时回退默认值。
|
||||
"""
|
||||
diffs = [
|
||||
float(manifest[i + 1]["time"]) - float(manifest[i]["time"])
|
||||
for i in range(len(manifest) - 1)
|
||||
if float(manifest[i + 1]["time"]) > float(manifest[i]["time"])
|
||||
]
|
||||
if not diffs:
|
||||
return default
|
||||
diffs.sort()
|
||||
# 取 3 位小数：与 frame-extract 的 round(秒,3) 时间戳精度一致，避免浮点漂移。
|
||||
return round(diffs[len(diffs) // 2], 3)
|
||||
|
||||
|
||||
def _assemble_srt(
|
||||
kept: list[tuple[float, float, str]],
|
||||
interval_seconds: float,
|
||||
) -> list[str]:
|
||||
"""把 (起始帧时间, 最后可见帧时间, 文本) 序列组装为 SRT 行列表。
|
||||
|
||||
每条字幕的结束时间 = 最后可见帧时间 + 采样间隔：字幕在最后一个被识别
|
||||
到的帧之后的一个采样间隔内消失，与视频烧录时间对齐；两段字幕之间的
|
||||
空白段（无字幕帧）不再被并入前一条字幕。
|
||||
"""
|
||||
lines: list[str] = []
|
||||
for index, (start, last_seen, text) in enumerate(kept):
|
||||
end = last_seen + interval_seconds
|
||||
lines.extend(
|
||||
[
|
||||
str(index + 1),
|
||||
f"{format_timestamp(start)} --> {format_timestamp(end)}",
|
||||
text,
|
||||
"",
|
||||
]
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""逐帧 OCR 并汇总字幕，产物为 subtitle.srt。"""
|
||||
manifest_uri = request.inputs.get("frames_manifest")
|
||||
if not manifest_uri:
|
||||
return InvokeResponse(status="failed", error="frames_manifest is required")
|
||||
manifest_path = Path(manifest_uri)
|
||||
if not manifest_path.is_file():
|
||||
return InvokeResponse(status="failed", error="frames manifest not found")
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
|
||||
# 结果长度上限：超过即视为模型异常（重复循环等），该帧直接报错跳过。
|
||||
max_result_chars = int(request.params.get("max_result_chars", 200))
|
||||
# 采样间隔优先从帧清单时间轴推导（与 frame-extract 实际抽取间隔一致），
|
||||
# 参数仅作清单退化时的兜底。
|
||||
interval = _sampling_interval(
|
||||
manifest, float(request.params.get("interval_seconds", 2.0))
|
||||
)
|
||||
# 透传给 vlm-ocr 的参数（仅传已提供的，避免覆盖其默认值）。
|
||||
vlm_params = {
|
||||
key: request.params.get(key)
|
||||
for key in (
|
||||
"model", "ollama_host", "prompt", "timeout_seconds", "keep_alive",
|
||||
"temperature", "repeat_penalty", "num_predict",
|
||||
)
|
||||
if request.params.get(key) is not None
|
||||
}
|
||||
|
||||
# 单帧 OCR：并行池的工作函数，返回该帧识别文本（失败/空/超长均返回空串）。
|
||||
def ocr_frame(payload) -> str:
|
||||
index, item = payload
|
||||
response = registry.invoke(
|
||||
"vlm-ocr",
|
||||
InvokeRequest(
|
||||
run_id=request.run_id,
|
||||
node_instance_id="",
|
||||
inputs={"image_uri": str(item["image_uri"])},
|
||||
params=vlm_params,
|
||||
output_dir=str(Path(request.output_dir) / "ocr_frames" / f"{index:04d}"),
|
||||
),
|
||||
)
|
||||
if response.status != "completed":
|
||||
# 单帧失败不中断整体，跳过该帧继续汇总。
|
||||
logger.warning("帧 %d OCR 失败，跳过: %s", index, response.error)
|
||||
return ""
|
||||
logger.info("帧 %d/%d OCR 完成: %r", index + 1, len(manifest), response.outputs.get("text"))
|
||||
text = str(response.outputs.get("text", "")).strip()
|
||||
if not text:
|
||||
return ""
|
||||
# 超长输出视为模型异常（重复循环等），直接报错并跳过该帧。
|
||||
if len(text) > max_result_chars:
|
||||
logger.warning(
|
||||
"帧 %d OCR 输出超长(%d > %d)，跳过: %r",
|
||||
index, len(text), max_result_chars, text[:60],
|
||||
)
|
||||
return ""
|
||||
return text
|
||||
|
||||
# 进度日志：打印已识别帧数、总数与平均处理速度（帧/s）。
|
||||
def log_progress(done: int, total: int, rate: float) -> None:
|
||||
logger.info("OCR 进度 %d/%d 帧 (%.1f 帧/s)", done, total, rate)
|
||||
|
||||
# 自适应并发调用 vlm-ocr：10s 窗口内平均响应 < 0.3s 则加 1 线程（上限
|
||||
# pool_max_workers），> pool_slow_threshold 则减 1 线程（下限 1），
|
||||
# 按实测负载弹性伸缩，避免盲目并发压垮本地 Ollama。
|
||||
pool = AdaptiveThreadPool(
|
||||
worker=ocr_frame,
|
||||
on_progress=log_progress,
|
||||
min_workers=int(request.params.get("pool_min_workers", 1)),
|
||||
max_workers=int(request.params.get("pool_max_workers", 16)),
|
||||
window_seconds=float(request.params.get("pool_window_seconds", 10.0)),
|
||||
fast_threshold=float(request.params.get("pool_fast_threshold", 0.3)),
|
||||
slow_threshold=float(request.params.get("pool_slow_threshold", 1.0)),
|
||||
)
|
||||
texts = pool.map(list(enumerate(manifest)))
|
||||
|
||||
# kept 元素为 (起始帧时间, 最后可见帧时间, 文本)；按帧顺序合并连续相同字幕。
|
||||
kept: list[tuple[float, float, str]] = []
|
||||
for index, text in enumerate(texts):
|
||||
if not text:
|
||||
continue
|
||||
time = float(manifest[index]["time"])
|
||||
# 连续帧相同字幕合并为一条（字幕停留多帧属正常现象）：
|
||||
# 仅更新最后可见帧时间，起始时间保持首次出现。
|
||||
if kept and kept[-1][2] == text:
|
||||
kept[-1] = (kept[-1][0], time, text)
|
||||
continue
|
||||
kept.append((time, time, text))
|
||||
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / "subtitle.srt"
|
||||
output_path.write_text("\n".join(_assemble_srt(kept, interval)), encoding="utf-8")
|
||||
logger.info("字幕汇总完成: %d 条", len(kept))
|
||||
return InvokeResponse(
|
||||
status="completed",
|
||||
outputs={"srt_uri": str(output_path), "count": len(kept)},
|
||||
)
|
||||
+193
@@ -0,0 +1,193 @@
|
||||
"""VLM OCR 节点。
|
||||
|
||||
调用本地 Ollama 服务的多模态模型（默认 glm-ocr:latest）对图片做文字识别
|
||||
（OCR），用于从视频帧中提取字幕文字，构建"真实音频 + 正确字幕"的测试数据。
|
||||
|
||||
与 llm-translate 节点相互独立：本节点只做视觉 OCR，不做翻译，避免把两类
|
||||
职责混在一起。调用协议见 Ollama 官方文档：POST /api/chat。
|
||||
|
||||
请求约定（2026-08 调整，直接请求 API 版本）：
|
||||
- 使用流式传输（stream=True），逐行接收生成内容，命中终止序列立即停止接收，
|
||||
避免模型重复循环时无限拉取输出；响应体整体受 5 秒截止时间约束。
|
||||
- 采样 temperature 默认 0.3（可参数覆盖），并携带终止序列列表
|
||||
`["\n", "\n答", "答"]`（输出首个换行即停 + 阻止“答：”式重复循环）；
|
||||
模型生成遇到任一标记即停止。
|
||||
- 每次调用整体超时 5 秒（timeout_seconds 参数 / VLM_TIMEOUT_SECONDS 环境变量），
|
||||
超过即终止，不再继续等待后续流式块。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
# 终止序列列表：模型输出一旦出现任一标记（如换行 "\n"、"答"）立即停止生成
|
||||
# （Ollama 的 stop 参数，同时作为流式读取时的本地截断标记）。
|
||||
# 语义：输出第一个换行即停（单行字幕），"答" 系列阻止"答："式重复循环。
|
||||
STOP_SEQUENCE = ["\n","\n答","答"]
|
||||
|
||||
|
||||
def _default_host() -> str:
|
||||
"""默认 Ollama 服务地址，可通过环境变量覆盖。"""
|
||||
return os.getenv("OLLAMA_HOST", "http://192.168.123.70:11434")
|
||||
|
||||
|
||||
def _clean_ocr_text(content: str) -> str:
|
||||
"""清洗模型输出：去掉 markdown 围栏与空行，只保留识别到的文字。
|
||||
|
||||
glm-ocr 的自定义解析器会在识别文本后追加大量 ``` 围栏，必须剔除，
|
||||
否则会污染字幕文本。
|
||||
"""
|
||||
lines: list[str] = []
|
||||
for line in content.splitlines():
|
||||
stripped = line.strip()
|
||||
# 整行只有反引号（可带语言标记，如 ```markdown）的围栏行丢弃。
|
||||
if re.fullmatch(r"`+[A-Za-z0-9]*", stripped):
|
||||
continue
|
||||
if stripped:
|
||||
lines.append(stripped)
|
||||
return "\n".join(lines)
|
||||
# 标签提取正则：匹配 <gettext>...</gettext>（DOTALL 让 . 也能匹配换行）。
|
||||
_GETTEXT_RE = re.compile(r"<gettext>(.*?)</gettext>", re.DOTALL)
|
||||
|
||||
|
||||
def _extract_gettext(raw: str) -> str:
|
||||
"""从模型输出中提取 <gettext></gettext> 标签内的内容。
|
||||
|
||||
提示词要求模型用该标签包裹识别结果；模型未按格式输出（找不到标签）
|
||||
时回退返回原始文本，保持旧行为。多个标签只取第一个（防重复循环）。
|
||||
"""
|
||||
match = _GETTEXT_RE.search(raw)
|
||||
if match is None:
|
||||
return raw
|
||||
return match.group(1)
|
||||
|
||||
|
||||
def _truncate_at_stop(raw: str) -> str:
|
||||
"""在最先命中的终止序列处截断文本。
|
||||
|
||||
STOP_SEQUENCE 为终止序列列表（如 ["\n", "\n答", "答"]）：取最早出现的位置
|
||||
截断，返回截断后的文本；未命中任何序列时原样返回。
|
||||
"""
|
||||
positions = [pos for seq in STOP_SEQUENCE if (pos := raw.find(seq)) != -1]
|
||||
if not positions:
|
||||
return raw
|
||||
return raw[: min(positions)]
|
||||
|
||||
|
||||
def _consume_stream(response, deadline: float) -> str:
|
||||
"""逐行读取流式响应，直到命中终止序列 / 流结束 / 超过截止时间。
|
||||
|
||||
response 为 urllib 打开的响应对象，readline() 返回 bytes（每行一个
|
||||
Ollama 流式 JSON 块）。返回拼接后的原始文本（未清洗、未截断终止序列）。
|
||||
"""
|
||||
parts: list[str] = []
|
||||
while True:
|
||||
# 每次调用整体 5 秒上限：超过立即终止，不再等待下一个流式块。
|
||||
if time.monotonic() >= deadline:
|
||||
raise TimeoutError("timed out")
|
||||
line = response.readline()
|
||||
if not line:
|
||||
# 流式输出正常结束（模型未发出终止序列，直接收完）。
|
||||
break
|
||||
chunk = json.loads(line.decode("utf-8"))
|
||||
message = chunk.get("message")
|
||||
if message is None:
|
||||
# done 行是流结束标记，可能不带 message；其余缺失视为格式错误。
|
||||
if chunk.get("done"):
|
||||
break
|
||||
raise KeyError("missing message in stream chunk")
|
||||
parts.append(str(message.get("content", "")))
|
||||
# 命中任一终止序列即停止接收后续内容（模型应已结束生成）。
|
||||
if any(seq in "".join(parts) for seq in STOP_SEQUENCE):
|
||||
break
|
||||
if chunk.get("done"):
|
||||
# 模型侧完成（未命中终止序列也自然结束）。
|
||||
break
|
||||
return "".join(parts)
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""识别 image_uri 指向的图片中的文字，产物为 ocr.txt。
|
||||
|
||||
参数：model（默认 glm-ocr:latest）、ollama_host、prompt、
|
||||
timeout_seconds（默认 5，每次调用整体上限）；均可通过环境变量
|
||||
VLM_MODEL / OLLAMA_HOST / VLM_TIMEOUT_SECONDS 覆盖。
|
||||
"""
|
||||
image_uri = request.inputs.get("image_uri")
|
||||
if not image_uri:
|
||||
return InvokeResponse(status="failed", error="image_uri is required")
|
||||
|
||||
image_path = Path(image_uri)
|
||||
# 文件不存在时提前失败，避免无谓的网络请求。
|
||||
if not image_path.is_file():
|
||||
return InvokeResponse(status="failed", error="image file not found")
|
||||
|
||||
host = str(request.params.get("ollama_host") or _default_host())
|
||||
model = str(request.params.get("model") or os.getenv("VLM_MODEL", "glm-ocr:latest"))
|
||||
prompt = str(
|
||||
request.params.get("prompt")
|
||||
or os.getenv("VLM_PROMPT", "提取图像中的文字，不要描述图片中的内容")
|
||||
)
|
||||
timeout = float(
|
||||
request.params.get("timeout_seconds")
|
||||
or os.getenv("VLM_TIMEOUT_SECONDS", "5")
|
||||
)
|
||||
|
||||
# 图片按 base64 随请求体发送（Ollama 多模态标准格式）。
|
||||
image_b64 = base64.b64encode(image_path.read_bytes()).decode("ascii")
|
||||
body = {
|
||||
"model": model,
|
||||
# 流式传输：逐行接收生成内容，命中终止序列或超时即停止。
|
||||
"stream": True,
|
||||
# keep_alive 让模型在服务端常驻，避免逐帧调用反复加载模型。
|
||||
"keep_alive": str(request.params.get("keep_alive", "5m")),
|
||||
# 采样选项：temperature 默认 0.3（可参数覆盖）；glm-ocr 在大图上有已知
|
||||
# 重复循环 bug，repeat_penalty 惩罚重复 token、num_predict 限制输出上限。
|
||||
"options": {
|
||||
"temperature": float(request.params.get("temperature", 0.3)),
|
||||
"repeat_penalty": float(request.params.get("repeat_penalty", 1)),
|
||||
"num_predict": int(request.params.get("num_predict", 256)),
|
||||
},
|
||||
# /api/chat 的输入结构：识别指令放系统提示词，用户消息只携带图片
|
||||
# （content 为空、images 传 base64，与 glm-ocr 期望结构一致）。
|
||||
"messages": [
|
||||
{"role": "system", "content": prompt},
|
||||
{"role": "user", "content": "", "images": [image_b64]},
|
||||
],
|
||||
# 模型遇到任一终止序列即停止生成，遏制重复循环；同时作为流式读取截断点。
|
||||
"stop": STOP_SEQUENCE,
|
||||
}
|
||||
request_url = f"{host.rstrip('/')}/api/chat"
|
||||
http_request = urllib.request.Request(
|
||||
request_url,
|
||||
data=json.dumps(body).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
# timeout 同时作用于连接与每次 socket 读取；deadline 保证整体 5 秒上限。
|
||||
deadline = time.monotonic() + timeout
|
||||
with urllib.request.urlopen(http_request, timeout=timeout) as response:
|
||||
raw = _consume_stream(response, deadline)
|
||||
# 终止序列可能随最后一个流式块一起返回，在最先命中的序列处截断再清洗。
|
||||
raw = _truncate_at_stop(raw)
|
||||
# 提取提示词约定的 <gettext> 标签内容；模型未按格式输出时回退原始文本。
|
||||
text = _clean_ocr_text(_extract_gettext(raw))
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / "ocr.txt"
|
||||
output_path.write_text(text + "\n", encoding="utf-8")
|
||||
return InvokeResponse(
|
||||
status="completed",
|
||||
outputs={"text": text, "text_uri": str(output_path)},
|
||||
)
|
||||
except (urllib.error.URLError, KeyError, ValueError, OSError) as exc:
|
||||
# 网络失败、响应格式异常、超时等统一转换为 failed 响应。
|
||||
return InvokeResponse(status="failed", error=str(exc))
|
||||
Executable
+274
@@ -0,0 +1,274 @@
|
||||
"""faster-whisper ASR 节点。
|
||||
|
||||
单体版中作为进程内节点模块，由调度器直接调用。模型权重默认优先从本地
|
||||
目录加载，避免从远端下载，仅在本地找不到模型时才回退到远端 large-v3。
|
||||
CUDA 动态库通过 ctypes 在进程内预加载，替代分布式版的 LD_LIBRARY_PATH 注入。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ctypes
|
||||
import os
|
||||
import subprocess
|
||||
import sysconfig
|
||||
import time
|
||||
import wave
|
||||
from pathlib import Path
|
||||
|
||||
from nodes.ffmpeg import _ffmpeg_bin
|
||||
from wov_app.logging import get_logger
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
# 转写进度日志：输出到主进程控制台，长音频分块时可见每块进度。
|
||||
logger = get_logger("whisper")
|
||||
|
||||
|
||||
def _is_windows() -> bool:
|
||||
"""判断当前是否为 Windows，供测试单独注入覆盖。"""
|
||||
return os.name == "nt"
|
||||
|
||||
|
||||
def _load_cuda_libraries() -> None:
|
||||
"""在进程内预加载 pip 安装的 NVIDIA 动态库。
|
||||
|
||||
分布式版通过给节点子进程注入 LD_LIBRARY_PATH（Windows 为 PATH）解决；
|
||||
单体版没有进程边界，必须在导入 faster-whisper 前加载 nvidia 轮子自带
|
||||
的 .so/.dll，否则 ctranslate2 初始化 CUDA 时找不到 libcublas.so.12。
|
||||
"""
|
||||
# site-packages 目录，nvidia 各包的动态库位于其下。
|
||||
site_packages = Path(sysconfig.get_paths()["purelib"])
|
||||
for vendor in ("cublas", "cudnn", "cuda_nvrtc"):
|
||||
# Windows 轮子把 dll 放在 bin/，Linux 放在 lib/。
|
||||
for subdir in ("bin", "lib"):
|
||||
lib_dir = site_packages / "nvidia" / vendor / subdir
|
||||
if not lib_dir.is_dir():
|
||||
continue
|
||||
if _is_windows():
|
||||
# Windows 通过 DLL 搜索目录注册，等价于进程内 PATH 注入。
|
||||
os.add_dll_directory(str(lib_dir))
|
||||
else:
|
||||
for so_file in sorted(lib_dir.glob("*.so*")):
|
||||
try:
|
||||
ctypes.CDLL(str(so_file))
|
||||
except OSError:
|
||||
# 个别依赖缺失（如 libcudart）时跳过，交由 ctranslate2 报错。
|
||||
continue
|
||||
|
||||
def _local_model_candidates() -> list[Path]:
|
||||
"""返回本地模型候选目录：单体根目录 model/ 优先，其次 nodes/ 同级 model/。
|
||||
|
||||
单体根目录 model/ 对应仓库根下的 model/faster-whisper-large-v3，
|
||||
nodes/ 同级 model/ 允许部署时把权重随代码目录一起携带。
|
||||
"""
|
||||
monolith_root = Path(__file__).resolve().parent.parent
|
||||
return [
|
||||
monolith_root / "model" / "faster-whisper-large-v3",
|
||||
monolith_root / "nodes" / "model" / "faster-whisper-large-v3",
|
||||
]
|
||||
|
||||
|
||||
def resolve_model_path(
|
||||
params: dict,
|
||||
env: dict | None = None,
|
||||
candidates: list[Path] | None = None,
|
||||
) -> str:
|
||||
"""按 参数 > 环境变量 > 本地候选目录 > 远端 large-v3 的顺序解析模型路径。
|
||||
|
||||
本地优先是默认行为：只要候选目录存在且包含 model.bin 就使用本地权重，
|
||||
避免从 Hugging Face 下载；远端下载仅在全部本地候选缺失时作为兜底。
|
||||
参数/环境变量传入的是裸模型名（不含路径分隔符）时，会先在本地模型
|
||||
目录（model/）下按名解析，方便工作流直接引用下载好的模型。
|
||||
candidates 参数供测试注入临时目录，默认使用 _local_model_candidates()。
|
||||
"""
|
||||
env = env if env is not None else os.environ
|
||||
candidates = candidates if candidates is not None else _local_model_candidates()
|
||||
explicit = params.get("model_path") or env.get("WHISPER_MODEL_PATH")
|
||||
if explicit:
|
||||
explicit_str = str(explicit)
|
||||
# 裸模型名按 <模型目录>/<名称> 在本地解析，例如
|
||||
# "whisper-large-v3-translate-zh-v0.1-lt-ct2"。
|
||||
if "/" not in explicit_str and "\\" not in explicit_str:
|
||||
named = candidates[0].parent / explicit_str
|
||||
if (named / "model.bin").is_file():
|
||||
return str(named)
|
||||
return explicit_str
|
||||
for candidate in candidates:
|
||||
# model.bin 是 CTranslate2 权重的必需文件，存在才认为模型完整。
|
||||
if candidate.is_dir() and (candidate / "model.bin").is_file():
|
||||
return str(candidate)
|
||||
return "large-v3"
|
||||
|
||||
def format_timestamp(seconds: float) -> str:
|
||||
"""把秒数格式化为 SRT 时间戳，例如 01:00:00,500。"""
|
||||
# 先换算成毫秒再逐级拆分为时/分/秒/毫秒，避免浮点误差。
|
||||
total_ms = int(seconds * 1000)
|
||||
hours, remainder = divmod(total_ms, 3600000)
|
||||
minutes, remainder = divmod(remainder, 60000)
|
||||
secs, millis = divmod(remainder, 1000)
|
||||
return f"{hours:02d}:{minutes:02d}:{secs:02d},{millis:03d}"
|
||||
|
||||
|
||||
def _split_audio(
|
||||
audio_path: Path,
|
||||
output_dir: Path,
|
||||
chunk_seconds: int,
|
||||
ffmpeg_bin: str | None,
|
||||
) -> list[Path]:
|
||||
"""用 ffmpeg 把音频切成 chunk_seconds 秒一块的 wav，返回块路径列表。
|
||||
|
||||
分块是应用层工程策略（内存有界、失败粒度小），与模型 30s 窗口无关；
|
||||
whisper 训练与推理都按 30s 窗口解码，任意块大小都适用。以下情况回退
|
||||
为整段单次转写：chunk_seconds <= 0、找不到 ffmpeg、切块失败、音频本身
|
||||
不足一块（ffmpeg 产出单块）。
|
||||
"""
|
||||
if chunk_seconds <= 0 or not ffmpeg_bin:
|
||||
return [audio_path]
|
||||
chunk_dir = output_dir / "chunks"
|
||||
chunk_dir.mkdir(parents=True, exist_ok=True)
|
||||
pattern = str(chunk_dir / "chunk_%03d.wav")
|
||||
# 音频已是 16kHz 单声道 WAV，流拷贝切块即可，无需重编码。
|
||||
result = subprocess.run(
|
||||
[
|
||||
ffmpeg_bin,
|
||||
"-y",
|
||||
"-i",
|
||||
str(audio_path),
|
||||
"-f",
|
||||
"segment",
|
||||
"-segment_time",
|
||||
str(chunk_seconds),
|
||||
"-c",
|
||||
"copy",
|
||||
pattern,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
# 切块失败（输入损坏等）回退整段，不让转写流程中断。
|
||||
return [audio_path]
|
||||
chunks = sorted(chunk_dir.glob("chunk_*.wav"))
|
||||
return chunks or [audio_path]
|
||||
|
||||
|
||||
def _wav_duration_seconds(path: Path, fallback: float) -> float:
|
||||
"""从 WAV 头精确读取时长；文件非法/非 WAV 时用 fallback 兜底。
|
||||
|
||||
分块偏移必须用每块的实际时长累积，而不是 块序号×块长 的假设值——
|
||||
ffmpeg 切出的块实际时长并不精确等于块长（如 60.05s），假设值会随
|
||||
块数累积漂移，造成字幕时间轴逐渐错位。
|
||||
"""
|
||||
try:
|
||||
with wave.open(str(path), "rb") as wav:
|
||||
rate = wav.getframerate()
|
||||
return wav.getnframes() / rate if rate else fallback
|
||||
except (wave.Error, EOFError, OSError):
|
||||
# 文件损坏/非 WAV（如 mp4 直传）时用 fallback 兜底。
|
||||
return fallback
|
||||
|
||||
|
||||
def _append_srt_lines(lines: list[str], segments, offset: float, start_index: int) -> int:
|
||||
"""把一段转写结果按 SRT 格式追加到 lines，时间加上 offset 偏移。
|
||||
|
||||
分块合并时每块 offset 为前面所有块的实际时长累积；单次调用 offset=0。
|
||||
每写出一条字幕就打印其编号与完整视频角度的时间范围，便于对照对齐。
|
||||
返回本段新增的条数，用于全局序号递增。
|
||||
"""
|
||||
count = 0
|
||||
for segment in segments:
|
||||
# 完整视频角度的时间 = 模型预测时间 + 累积偏移。
|
||||
start_time = segment.start + offset
|
||||
end_time = segment.end + offset
|
||||
lines.extend(
|
||||
[
|
||||
str(start_index + count),
|
||||
f"{format_timestamp(start_time)} --> {format_timestamp(end_time)}",
|
||||
segment.text.strip(),
|
||||
"",
|
||||
]
|
||||
)
|
||||
logger.info(
|
||||
"分段 #%d: %s --> %s",
|
||||
start_index + count,
|
||||
format_timestamp(start_time),
|
||||
format_timestamp(end_time),
|
||||
)
|
||||
count += 1
|
||||
return count
|
||||
|
||||
def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
"""转写音频并生成 SRT 字幕，产物为 transcript.srt。"""
|
||||
audio_uri = request.inputs.get("audio_uri")
|
||||
if not audio_uri:
|
||||
return InvokeResponse(status="failed", error="audio_uri is required")
|
||||
|
||||
# 文件不存在时提前失败，避免进入耗时的模型加载流程。
|
||||
audio_path = Path(audio_uri)
|
||||
if not audio_path.is_file():
|
||||
return InvokeResponse(status="failed", error="audio file not found")
|
||||
|
||||
try:
|
||||
# 延迟导入 faster-whisper，保证节点注册与调度等轻量路径不依赖重型依赖；
|
||||
# 导入前先预加载 NVIDIA 动态库，否则 ctranslate2 找不到 libcublas。
|
||||
_load_cuda_libraries()
|
||||
from faster_whisper import WhisperModel
|
||||
# 模型路径默认本地优先：参数 > 环境变量 > 工作区本地目录 > 远端兜底。
|
||||
model_path = resolve_model_path(request.params)
|
||||
device = str(request.params.get("device") or os.getenv("WHISPER_DEVICE", "auto"))
|
||||
# auto 让 faster-whisper 根据硬件自动选择 float16/int8 等计算类型。
|
||||
compute_type = str(request.params.get("compute_type") or "auto")
|
||||
model = WhisperModel(
|
||||
model_path,
|
||||
device=device,
|
||||
compute_type=compute_type,
|
||||
)
|
||||
# language 默认日语；vad_filter 默认开启（用户 2026-08 决定）：过滤静音
|
||||
# 段以提速并减少无语音处幻觉；长静音时 VAD 压缩时间轴可能轻微错位，
|
||||
# 如需极致对齐可在工作流参数中显式关闭。
|
||||
# task 默认 transcribe，中文直出模型可传 translate 直接翻译为目标语言。
|
||||
# condition_on_previous_text 默认 False：长音频下开启会导致重复/漂移，
|
||||
# 关闭后每个 30s 窗口独立解码，是 faster-whisper 官方建议的长音频方案。
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
# 分块转写：默认每 1 分钟一块（chunk_seconds=60），切块失败自动回退整段。
|
||||
chunk_seconds = int(request.params.get("chunk_seconds", 60))
|
||||
chunks = _split_audio(audio_path, output_dir, chunk_seconds, _ffmpeg_bin())
|
||||
# 逐块转写并合并：offset 用每块实际时长累积（WAV 头精确），SRT 序号连续。
|
||||
logger.info("转写开始: %d 个分块", len(chunks))
|
||||
lines: list[str] = []
|
||||
offset = 0.0
|
||||
# SRT 序号从 1 开始，跨块连续递增。
|
||||
srt_number = 1
|
||||
transcribe_started = time.monotonic()
|
||||
for chunk_index, chunk in enumerate(chunks, start=1):
|
||||
chunk_started = time.monotonic()
|
||||
segments, _info = model.transcribe(
|
||||
str(chunk),
|
||||
language=str(request.params.get("language", "ja")),
|
||||
task=str(request.params.get("task", "transcribe")),
|
||||
beam_size=int(request.params.get("beam_size", 1)),
|
||||
vad_filter=bool(request.params.get("vad_filter", True)),
|
||||
condition_on_previous_text=bool(
|
||||
request.params.get("condition_on_previous_text", False)
|
||||
),
|
||||
)
|
||||
# 进度日志：块序号/总数、单块耗时、实时倍率（块音频时长/墙钟耗时）
|
||||
# 与转写累计耗时，直观反映数据处理速度。
|
||||
chunk_elapsed = time.monotonic() - chunk_started
|
||||
srt_number += _append_srt_lines(lines, segments, offset, srt_number)
|
||||
# 偏移按本块实际时长推进，避免假设块长导致的累积漂移。
|
||||
offset += _wav_duration_seconds(chunk, chunk_seconds)
|
||||
logger.info(
|
||||
"分块 %d/%d 完成 offset=%.2fs 耗时 %.1fs (%.2fx 实时, 累计 %.1fs)",
|
||||
chunk_index, len(chunks), offset, chunk_elapsed,
|
||||
chunk_seconds / chunk_elapsed if chunk_elapsed > 0 else 0.0,
|
||||
time.monotonic() - transcribe_started,
|
||||
)
|
||||
output_path = output_dir / "transcript.srt"
|
||||
output_path.write_text("\n".join(lines), encoding="utf-8")
|
||||
return InvokeResponse(status="completed", outputs={"srt_uri": str(output_path)})
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 模型加载或转写异常统一转换为 failed 响应，不让调度线程崩溃。
|
||||
return InvokeResponse(status="failed", error=str(exc))
|
||||
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
[project]
|
||||
name = "vrsub"
|
||||
version = "0.1.0"
|
||||
description = "VRSub：为视频生成 VR 双眼字幕的单体应用（API、调度器与全部节点在单进程内运行）"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
# Web 框架：FastAPI 负责 API 与静态前端挂载。
|
||||
"fastapi",
|
||||
"uvicorn",
|
||||
"python-multipart",
|
||||
# 转写节点：faster-whisper 及其 CUDA 动态库（同进程直接加载，无需路径注入）。
|
||||
"faster-whisper>=1.2.1",
|
||||
"nvidia-cublas-cu12>=12.9.2.10",
|
||||
"nvidia-cudnn-cu12>=9.24.0.43",
|
||||
# 提音节点：imageio-ffmpeg 提供内置 ffmpeg 兜底。
|
||||
"imageio-ffmpeg>=0.6",
|
||||
"python-dotenv>=1.2.2",
|
||||
]
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"pytest",
|
||||
"pytest-cov",
|
||||
"httpx",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
# wov_sdk / wov_app 位于 src/，nodes 位于仓库根，一并作为可安装包。
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src", "."]
|
||||
include = ["wov_sdk*", "wov_app*", "nodes*"]
|
||||
|
||||
# pytest 配置：扫描 tests 目录并强制 100% 行覆盖率；同时把仓库根加入
|
||||
# sys.path，保证 nodes 包在未重新安装时也能被导入。
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["."]
|
||||
addopts = "--cov=src --cov=nodes --cov-fail-under=100 -p no:cacheprovider"
|
||||
@@ -0,0 +1,5 @@
|
||||
"""WOV 单体应用包。
|
||||
|
||||
包含 FastAPI 应用入口、SQLite 数据访问、进程内节点注册表、工作流调度器
|
||||
以及管理端/用户端路由。所有节点在同一进程内直接调用，无子进程边界。
|
||||
"""
|
||||
@@ -0,0 +1,28 @@
|
||||
"""应用配置中心。
|
||||
|
||||
集中读取环境变量并推导路径常量，避免业务代码散落魔法值。路径统一使用
|
||||
pathlib，Windows 与 Linux 开发环境均可用。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# WOV 单体根目录：本文件位于 src/wov_app/config.py，向上三级即仓库根。
|
||||
WORKSPACE_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
|
||||
# 数据目录、SQLite 文件与产物存储目录均可通过环境变量覆盖，便于测试隔离。
|
||||
DATA_DIR = Path(os.getenv("WOV_DATA_DIR", str(WORKSPACE_ROOT / "data")))
|
||||
DB_PATH = Path(os.getenv("WOV_DB_PATH", str(DATA_DIR / "wov.db")))
|
||||
STORAGE_DIR = Path(os.getenv("WOV_STORAGE_DIR", str(DATA_DIR / "storage")))
|
||||
|
||||
# 调度器轮询排队任务的间隔（秒）。
|
||||
SCHEDULER_INTERVAL_SECONDS = float(os.getenv("WOV_SCHEDULER_INTERVAL_SECONDS", "1.0"))
|
||||
|
||||
# 孤儿数据清理器配置：定时扫描并清理无对应文件/记录的死数据。
|
||||
CLEANUP_ENABLED = os.getenv("WOV_CLEANUP_ENABLED", "1") == "1"
|
||||
# 清理扫描周期（秒），默认每小时一次。
|
||||
CLEANUP_INTERVAL_SECONDS = float(os.getenv("WOV_CLEANUP_INTERVAL_SECONDS", "3600"))
|
||||
# 宽限期（秒）：任务最后更新距今超过该时长且满足孤儿条件才清理。
|
||||
CLEANUP_GRACE_SECONDS = float(os.getenv("WOV_CLEANUP_GRACE_SECONDS", "3600"))
|
||||
@@ -0,0 +1,398 @@
|
||||
"""SQLite 数据访问层。
|
||||
|
||||
所有持久化逻辑集中在本模块，业务代码只依赖 Database 提供的方法。后续切换
|
||||
PostgreSQL 时只需替换本层实现，不修改调度器与路由的业务逻辑。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterator
|
||||
|
||||
|
||||
class Database:
|
||||
"""SQLite 数据库封装：负责建表以及工作流/任务/产物的 CRUD。"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
"""打开数据库并确保父目录存在、表结构已初始化。"""
|
||||
self.path = path
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._init_schema()
|
||||
|
||||
@contextmanager
|
||||
def _connect(self) -> Iterator[sqlite3.Connection]:
|
||||
"""提供带事务提交的数据库连接上下文。"""
|
||||
conn = sqlite3.connect(self.path)
|
||||
# 按列名读取结果，返回 dict 更直观。
|
||||
conn.row_factory = sqlite3.Row
|
||||
# 开启外键约束，保证子表记录引用有效。
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
try:
|
||||
yield conn
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def _init_schema(self) -> None:
|
||||
"""创建全部业务表；已存在的表保持不变。"""
|
||||
with self._connect() as conn:
|
||||
conn.executescript(
|
||||
"""
|
||||
-- 工作流表：只保存概要信息，完整定义存版本表。
|
||||
CREATE TABLE IF NOT EXISTS workflows (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
description TEXT NOT NULL DEFAULT '',
|
||||
published INTEGER NOT NULL DEFAULT 0,
|
||||
latest_version INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- 工作流版本表：每个版本保存一份 DAG 定义 JSON。
|
||||
CREATE TABLE IF NOT EXISTS workflow_versions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
workflow_id TEXT NOT NULL,
|
||||
version INTEGER NOT NULL,
|
||||
definition_json TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(workflow_id, version),
|
||||
FOREIGN KEY(workflow_id) REFERENCES workflows(id)
|
||||
);
|
||||
|
||||
-- 工作流运行表：记录任务从排队到完成/失败的状态机。
|
||||
CREATE TABLE IF NOT EXISTS workflow_runs (
|
||||
id TEXT PRIMARY KEY,
|
||||
workflow_id TEXT NOT NULL,
|
||||
workflow_version INTEGER NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
current_node_id TEXT,
|
||||
progress REAL NOT NULL DEFAULT 0,
|
||||
error TEXT,
|
||||
input_uri TEXT,
|
||||
param_overrides TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
FOREIGN KEY(workflow_id) REFERENCES workflows(id)
|
||||
);
|
||||
|
||||
-- 产物表：记录每个任务各节点的输出 URI，按名称唯一。
|
||||
CREATE TABLE IF NOT EXISTS artifacts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
run_id TEXT NOT NULL,
|
||||
node_id TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
uri TEXT NOT NULL,
|
||||
mime_type TEXT NOT NULL DEFAULT '',
|
||||
size INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(run_id, name),
|
||||
FOREIGN KEY(run_id) REFERENCES workflow_runs(id)
|
||||
);
|
||||
"""
|
||||
)
|
||||
|
||||
# 旧库迁移：workflow_runs 补充 param_overrides 列（前端框选覆盖）。
|
||||
columns = [
|
||||
row["name"]
|
||||
for row in conn.execute("PRAGMA table_info(workflow_runs)").fetchall()
|
||||
]
|
||||
if "param_overrides" not in columns:
|
||||
conn.execute("ALTER TABLE workflow_runs ADD COLUMN param_overrides TEXT")
|
||||
|
||||
def upsert_workflow(self, workflow: dict[str, Any]) -> None:
|
||||
"""插入或更新工作流概要信息。"""
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO workflows (id, name, description, published, latest_version)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
name = excluded.name,
|
||||
description = excluded.description,
|
||||
published = excluded.published,
|
||||
latest_version = excluded.latest_version
|
||||
""",
|
||||
(
|
||||
workflow["id"],
|
||||
workflow["name"],
|
||||
workflow.get("description", ""),
|
||||
int(workflow.get("published", 0)),
|
||||
int(workflow.get("latest_version", 0)),
|
||||
),
|
||||
)
|
||||
|
||||
def get_workflow(self, workflow_id: str) -> dict[str, Any] | None:
|
||||
"""按 ID 读取工作流概要。"""
|
||||
with self._connect() as conn:
|
||||
row = conn.execute("SELECT * FROM workflows WHERE id = ?", (workflow_id,)).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
def list_workflows(self) -> list[dict[str, Any]]:
|
||||
"""按创建时间倒序返回全部工作流。"""
|
||||
with self._connect() as conn:
|
||||
rows = conn.execute("SELECT * FROM workflows ORDER BY created_at DESC").fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
def delete_workflow(self, workflow_id: str) -> None:
|
||||
"""级联删除工作流相关的产物、任务、版本和概要记录。"""
|
||||
with self._connect() as conn:
|
||||
# 外键没有级联删除配置，手动按依赖顺序清理。
|
||||
conn.execute("DELETE FROM artifacts WHERE run_id IN (SELECT id FROM workflow_runs WHERE workflow_id = ?)", (workflow_id,))
|
||||
conn.execute("DELETE FROM workflow_runs WHERE workflow_id = ?", (workflow_id,))
|
||||
conn.execute("DELETE FROM workflow_versions WHERE workflow_id = ?", (workflow_id,))
|
||||
conn.execute("DELETE FROM workflows WHERE id = ?", (workflow_id,))
|
||||
|
||||
def create_workflow_version(self, workflow_id: str, version: int, definition: dict[str, Any]) -> None:
|
||||
"""为工作流新增一个版本，definition 以 JSON 保存。"""
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO workflow_versions (workflow_id, version, definition_json)
|
||||
VALUES (?, ?, ?)
|
||||
""",
|
||||
(workflow_id, version, json.dumps(definition, ensure_ascii=False)),
|
||||
)
|
||||
|
||||
def get_latest_workflow_version(self, workflow_id: str) -> dict[str, Any] | None:
|
||||
"""返回工作流最新版本，并把 definition_json 反序列化为 definition。"""
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT * FROM workflow_versions
|
||||
WHERE workflow_id = ?
|
||||
ORDER BY version DESC
|
||||
LIMIT 1
|
||||
""",
|
||||
(workflow_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
result = dict(row)
|
||||
# 对外统一暴露 definition 字典，隐藏 JSON 存储细节。
|
||||
result["definition"] = json.loads(result.pop("definition_json"))
|
||||
return result
|
||||
|
||||
def get_workflow_version(self, workflow_id: str, version: int) -> dict[str, Any] | None:
|
||||
"""按版本号读取指定工作流版本。"""
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT * FROM workflow_versions
|
||||
WHERE workflow_id = ? AND version = ?
|
||||
""",
|
||||
(workflow_id, version),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
result = dict(row)
|
||||
result["definition"] = json.loads(result.pop("definition_json"))
|
||||
return result
|
||||
|
||||
def list_workflow_versions(self, workflow_id: str) -> list[dict[str, Any]]:
|
||||
"""按版本倒序返回工作流全部版本。"""
|
||||
with self._connect() as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT * FROM workflow_versions
|
||||
WHERE workflow_id = ?
|
||||
ORDER BY version DESC
|
||||
""",
|
||||
(workflow_id,),
|
||||
).fetchall()
|
||||
versions = []
|
||||
for row in rows:
|
||||
item = dict(row)
|
||||
item["definition"] = json.loads(item.pop("definition_json"))
|
||||
versions.append(item)
|
||||
return versions
|
||||
|
||||
def create_run(self, run: dict[str, Any]) -> None:
|
||||
"""创建一条排队中的工作流运行记录。"""
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO workflow_runs (
|
||||
id, workflow_id, workflow_version, status, current_node_id,
|
||||
progress, error, input_uri, param_overrides, created_at, updated_at
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
run["id"],
|
||||
run["workflow_id"],
|
||||
run["workflow_version"],
|
||||
run["status"],
|
||||
run.get("current_node_id"),
|
||||
float(run.get("progress", 0)),
|
||||
run.get("error"),
|
||||
run.get("input_uri"),
|
||||
json.dumps(run["param_overrides"], ensure_ascii=False)
|
||||
if run.get("param_overrides")
|
||||
else None,
|
||||
run["created_at"],
|
||||
run["updated_at"],
|
||||
),
|
||||
)
|
||||
|
||||
def get_run(self, run_id: str) -> dict[str, Any] | None:
|
||||
"""按 ID 读取任务运行记录。"""
|
||||
with self._connect() as conn:
|
||||
row = conn.execute("SELECT * FROM workflow_runs WHERE id = ?", (run_id,)).fetchone()
|
||||
return self._parse_overrides(row) if row else None
|
||||
|
||||
|
||||
@staticmethod
|
||||
def _parse_overrides(row) -> dict:
|
||||
"""把查询行中的 param_overrides JSON 字符串解析为字典。"""
|
||||
result = dict(row)
|
||||
raw = result.get("param_overrides")
|
||||
result["param_overrides"] = json.loads(raw) if raw else None
|
||||
return result
|
||||
|
||||
def list_runs(self, limit: int = 20) -> list[dict[str, Any]]:
|
||||
"""按创建时间倒序返回最近的运行记录。"""
|
||||
with self._connect() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM workflow_runs ORDER BY created_at DESC LIMIT ?",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [self._parse_overrides(row) for row in rows]
|
||||
|
||||
def list_run_ids(self) -> list[str]:
|
||||
"""返回全部任务 ID，供孤儿数据清理对照磁盘目录使用。"""
|
||||
with self._connect() as conn:
|
||||
rows = conn.execute("SELECT id FROM workflow_runs").fetchall()
|
||||
return [row["id"] for row in rows]
|
||||
|
||||
def update_run(self, run_id: str, **fields: Any) -> None:
|
||||
"""更新运行状态字段，同时刷新 updated_at；未知字段会被忽略。"""
|
||||
# 只允许更新状态机相关字段，防止任意列被改写。
|
||||
allowed = {
|
||||
"status",
|
||||
"current_node_id",
|
||||
"progress",
|
||||
"error",
|
||||
}
|
||||
updates = {key: value for key, value in fields.items() if key in allowed}
|
||||
if not updates:
|
||||
return
|
||||
updates["updated_at"] = fields.get("updated_at")
|
||||
# 动态拼接 SET 子句，键来自白名单，不存在 SQL 注入风险。
|
||||
assignments = ", ".join(f"{key} = ?" for key in updates)
|
||||
values = list(updates.values()) + [run_id]
|
||||
with self._connect() as conn:
|
||||
conn.execute(f"UPDATE workflow_runs SET {assignments} WHERE id = ?", values)
|
||||
|
||||
def reset_run(self, run_id: str, updated_at: str) -> None:
|
||||
"""把失败任务重置为 QUEUED，并清空进度与旧产物，供重试使用。"""
|
||||
with self._connect() as conn:
|
||||
# 清空错误和进度，恢复到首次排队时的状态。
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE workflow_runs
|
||||
SET status = 'QUEUED', current_node_id = NULL, progress = 0,
|
||||
error = NULL, updated_at = ?
|
||||
WHERE id = ?
|
||||
""",
|
||||
(updated_at, run_id),
|
||||
)
|
||||
# 删除旧产物，避免重试后残留过期下载链接。
|
||||
conn.execute("DELETE FROM artifacts WHERE run_id = ?", (run_id,))
|
||||
|
||||
def next_queued_run(self) -> dict[str, Any] | None:
|
||||
"""按创建时间返回最早一条可执行任务（排队或已暂停待续跑）。"""
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT * FROM workflow_runs
|
||||
WHERE status IN ('QUEUED', 'PAUSED')
|
||||
ORDER BY created_at ASC
|
||||
LIMIT 1
|
||||
"""
|
||||
).fetchone()
|
||||
return self._parse_overrides(row) if row else None
|
||||
|
||||
def pause_run(self, run_id: str, updated_at: str) -> None:
|
||||
"""暂停任务：置为 PAUSED；调度器会在节点边界检查并停止推进。"""
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"UPDATE workflow_runs SET status = 'PAUSED', updated_at = ? WHERE id = ?",
|
||||
(updated_at, run_id),
|
||||
)
|
||||
|
||||
def resume_run(self, run_id: str, updated_at: str) -> None:
|
||||
"""继续任务：PAUSED 恢复为 QUEUED，等待调度器从断点续跑。"""
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"UPDATE workflow_runs SET status = 'QUEUED', updated_at = ? WHERE id = ?",
|
||||
(updated_at, run_id),
|
||||
)
|
||||
|
||||
def restore_run_outputs(self, run_id: str) -> dict[str, dict[str, str]]:
|
||||
"""从已登记的产物重建各节点输出，供暂停后断点续跑使用。
|
||||
|
||||
返回 {节点ID: {输出名: URI}}；已完成节点的产物可直接作为后续节点的输入。
|
||||
"""
|
||||
outputs: dict[str, dict[str, str]] = {}
|
||||
for artifact in self.list_artifacts(run_id):
|
||||
name = artifact["name"]
|
||||
# 产物名形如 "节点ID.输出名"（如 a.data_uri），还原为 {输出名: URI}。
|
||||
prefix = artifact["node_id"] + "."
|
||||
if name.startswith(prefix):
|
||||
name = name[len(prefix):]
|
||||
outputs.setdefault(artifact["node_id"], {})[name] = artifact["uri"]
|
||||
return outputs
|
||||
def create_artifact(self, artifact: dict[str, Any]) -> None:
|
||||
"""记录任务产物；同 run 与 name 冲突时覆盖。"""
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT OR REPLACE INTO artifacts (
|
||||
run_id, node_id, name, uri, mime_type, size
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
artifact["run_id"],
|
||||
artifact["node_id"],
|
||||
artifact["name"],
|
||||
artifact["uri"],
|
||||
artifact.get("mime_type", ""),
|
||||
int(artifact.get("size", 0)),
|
||||
),
|
||||
)
|
||||
|
||||
def list_artifacts(self, run_id: str) -> list[dict[str, Any]]:
|
||||
"""按创建时间返回任务的全部产物。"""
|
||||
with self._connect() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM artifacts WHERE run_id = ? ORDER BY created_at",
|
||||
(run_id,),
|
||||
).fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
def get_artifact(self, run_id: str, name: str) -> dict[str, Any] | None:
|
||||
"""按任务与产物名读取单个产物记录。"""
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM artifacts WHERE run_id = ? AND name = ?",
|
||||
(run_id, name),
|
||||
).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
def delete_run_artifacts(self, run_id: str) -> None:
|
||||
"""删除任务的全部产物记录。"""
|
||||
with self._connect() as conn:
|
||||
conn.execute("DELETE FROM artifacts WHERE run_id = ?", (run_id,))
|
||||
|
||||
def delete_run(self, run_id: str) -> None:
|
||||
"""删除任务记录本身及其产物记录。
|
||||
|
||||
产物表外键引用任务表，必须先删产物再删任务，否则违反外键约束。
|
||||
"""
|
||||
with self._connect() as conn:
|
||||
conn.execute("DELETE FROM artifacts WHERE run_id = ?", (run_id,))
|
||||
conn.execute("DELETE FROM workflow_runs WHERE id = ?", (run_id,))
|
||||
@@ -0,0 +1,31 @@
|
||||
"""轻量日志配置。
|
||||
|
||||
单体版所有节点在 API 主进程内运行，这里把节点运行日志直接输出到主进程
|
||||
控制台（uvicorn 的 stderr），便于观察各节点的执行过程与耗时。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
# 应用日志统一前缀，便于与其他库日志区分。
|
||||
_APP_LOGGER_NAME = "vrsub"
|
||||
|
||||
|
||||
def _ensure_console_handler(logger: logging.Logger) -> None:
|
||||
"""为日志器附加控制台输出；已配置过则跳过，避免重复打印。"""
|
||||
if any(isinstance(handler, logging.StreamHandler) for handler in logger.handlers):
|
||||
return
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(message)s"))
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(logging.INFO)
|
||||
# 不向 uvicorn 根日志传播，防止消息重复输出。
|
||||
logger.propagate = False
|
||||
|
||||
|
||||
def get_logger(name: str) -> logging.Logger:
|
||||
"""获取并确保输出到主进程控制台的日志器。"""
|
||||
logger = logging.getLogger(f"{_APP_LOGGER_NAME}.{name}")
|
||||
_ensure_console_handler(logger)
|
||||
return logger
|
||||
@@ -0,0 +1,82 @@
|
||||
"""FastAPI 应用入口。
|
||||
|
||||
负责组装数据库、进程内节点注册表、调度器与静态前端，并在应用生命周期内
|
||||
管理后台线程的启动与清理。单体版所有节点在同一进程内直接调用。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 先加载 .env（含 LLM API Key 等本地配置），再导入读取环境变量的 config。
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
import os # noqa: E402
|
||||
from contextlib import asynccontextmanager # noqa: E402
|
||||
from pathlib import Path # noqa: E402
|
||||
|
||||
from fastapi import FastAPI # noqa: E402
|
||||
from fastapi.middleware.cors import CORSMiddleware # noqa: E402
|
||||
from fastapi.staticfiles import StaticFiles # noqa: E402
|
||||
|
||||
from wov_app import registry
|
||||
from wov_app.config import DB_PATH, STORAGE_DIR, WORKSPACE_ROOT
|
||||
from wov_app.db import Database
|
||||
from wov_app.maintenance import OrphanCleaner
|
||||
from wov_app.routers import apps, workflows
|
||||
from wov_app.scheduler import WorkflowScheduler
|
||||
from wov_app.seed import seed_default_workflows
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""应用生命周期：启动时初始化数据、注册节点和后台服务，退出时回收资源。"""
|
||||
# 确保数据与存储目录存在，避免首次启动写文件失败。
|
||||
db = Database(DB_PATH)
|
||||
STORAGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
# 静态注册全部内置节点到进程内注册表（内存态，不落库）。
|
||||
registry.register_all()
|
||||
# 默认创建演示工作流，可关闭便于测试。
|
||||
if os.getenv("WOV_AUTO_SEED", "1") == "1":
|
||||
seed_default_workflows(db)
|
||||
|
||||
scheduler = WorkflowScheduler(db, STORAGE_DIR)
|
||||
# 调度器默认开启，处理排队中的任务；测试可关闭后手动执行。
|
||||
if os.getenv("WOV_SCHEDULER_ENABLED", "1") == "1":
|
||||
scheduler.start()
|
||||
cleaner = OrphanCleaner(db, STORAGE_DIR)
|
||||
# 孤儿数据清理默认开启，定时清除死数据；测试可关闭。
|
||||
if os.getenv("WOV_CLEANUP_ENABLED", "1") == "1":
|
||||
cleaner.start()
|
||||
# 共享对象挂到 app.state，路由通过 Depends 延迟获取。
|
||||
app.state.db = db
|
||||
app.state.scheduler = scheduler
|
||||
app.state.cleaner = cleaner
|
||||
yield
|
||||
# 退出时先停调度器与清理器，避免残留后台线程。
|
||||
cleaner.stop()
|
||||
scheduler.stop()
|
||||
|
||||
|
||||
app = FastAPI(title="VRSub API（单体版）", version="0.1.0", lifespan=lifespan)
|
||||
|
||||
# MVP 阶段不做鉴权，允许跨域便于本地调试与静态页面访问。
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
app.include_router(workflows.router)
|
||||
app.include_router(apps.router)
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict:
|
||||
"""进程存活探针，供部署环境与前端检测后端可用性。"""
|
||||
return {"status": "ok", "service": "wov-api", "mode": "monolith"}
|
||||
|
||||
|
||||
# 静态前端目录位于仓库根下的 web，由 FastAPI 直接挂载。
|
||||
FRONTEND_DIR = WORKSPACE_ROOT / "web"
|
||||
app.mount("/", StaticFiles(directory=FRONTEND_DIR, html=True), name="frontend")
|
||||
@@ -0,0 +1,129 @@
|
||||
"""孤儿数据清理器。
|
||||
|
||||
定时扫描存储目录与数据库，清理不再有意义的死数据：
|
||||
|
||||
1. 磁盘上存在但没有对应任务记录的上传/步骤目录（删除任务中断等残留）。
|
||||
2. 状态为 COMPLETED 但产物文件已全部丢失、且超过宽限期的任务记录
|
||||
（这类任务在任务页会显示"完成"但下载全部 404，属于孤儿数据）。
|
||||
|
||||
出于安全考虑，以下数据**不会**被自动清理：
|
||||
|
||||
- FAILED 任务（用户可能重试，且失败任务本就可能没有文件）。
|
||||
- 状态非终态（QUEUED/RUNNING）的任务。
|
||||
- 最近宽限期内的任务，避免误删刚完成的运行。
|
||||
|
||||
手动删除任务仍走删除接口，本模块只做保守的孤儿兜底。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from wov_app.config import CLEANUP_GRACE_SECONDS, CLEANUP_INTERVAL_SECONDS
|
||||
from wov_app.db import Database
|
||||
|
||||
|
||||
class OrphanCleaner:
|
||||
"""后台孤儿清理器：周期扫描并清理孤儿数据，只保留明确的死数据。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
db: Database,
|
||||
storage_dir: Path,
|
||||
interval_seconds: float | None = None,
|
||||
grace_seconds: float | None = None,
|
||||
) -> None:
|
||||
"""保存依赖并初始化轮询线程控制字段。"""
|
||||
self.db = db
|
||||
self.storage_dir = storage_dir
|
||||
self.interval_seconds = interval_seconds or CLEANUP_INTERVAL_SECONDS
|
||||
self.grace_seconds = grace_seconds or CLEANUP_GRACE_SECONDS
|
||||
self._thread: threading.Thread | None = None
|
||||
self._stopping = False
|
||||
|
||||
def start(self) -> None:
|
||||
"""启动清理线程；重复调用无副作用。"""
|
||||
if self._thread is not None:
|
||||
return
|
||||
self._stopping = False
|
||||
self._thread = threading.Thread(
|
||||
target=self._loop,
|
||||
name="wov-orphan-cleaner",
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
"""请求停止并等待清理线程退出。"""
|
||||
self._stopping = True
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=5)
|
||||
self._thread = None
|
||||
|
||||
def _loop(self) -> None:
|
||||
"""周期循环：每隔一个间隔执行一次清理。"""
|
||||
while not self._stopping:
|
||||
time.sleep(self.interval_seconds)
|
||||
self.clean_once()
|
||||
|
||||
def clean_once(self) -> int:
|
||||
"""执行一次孤儿清理，返回清理的数据条目数。"""
|
||||
run_ids = set(self.db.list_run_ids())
|
||||
removed = 0
|
||||
# 1) 无对应任务记录的上传/步骤目录视为残留，直接删除。
|
||||
removed += self._clean_dangling(self.storage_dir / "uploads", run_ids)
|
||||
removed += self._clean_dangling(self.storage_dir / "runs", run_ids)
|
||||
# 2) COMPLETED 且产物文件全失、超过宽限期的任务记录删除。
|
||||
for run_id in run_ids:
|
||||
run = self.db.get_run(run_id)
|
||||
if run is None:
|
||||
continue
|
||||
if run["status"] != "COMPLETED":
|
||||
continue
|
||||
if not self._expired(run.get("updated_at")):
|
||||
continue
|
||||
if self._has_files(self.storage_dir / "runs" / run_id):
|
||||
continue
|
||||
removed += self._remove_run(run_id, run)
|
||||
return removed
|
||||
|
||||
def _clean_dangling(self, root: Path, run_ids: set[str]) -> int:
|
||||
"""删除 root 下没有对应任务记录的残留子目录，返回删除数。"""
|
||||
if not root.is_dir():
|
||||
return 0
|
||||
removed = 0
|
||||
for child in root.iterdir():
|
||||
if child.is_dir() and child.name not in run_ids:
|
||||
shutil.rmtree(child, ignore_errors=True)
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
def _expired(self, updated_at: str | None) -> bool:
|
||||
"""判断任务最后更新时间是否已超过宽限期；无法解析时保守视为未过期。"""
|
||||
if not updated_at:
|
||||
return False
|
||||
try:
|
||||
updated = datetime.fromisoformat(updated_at)
|
||||
return (datetime.now(timezone.utc) - updated).total_seconds() > self.grace_seconds
|
||||
except ValueError:
|
||||
# 时间格式损坏时保守保留，避免误删。
|
||||
return False
|
||||
|
||||
def _has_files(self, run_dir: Path) -> bool:
|
||||
"""判断任务目录下是否仍存在产物文件。"""
|
||||
if not run_dir.is_dir():
|
||||
return False
|
||||
return any(path.is_file() for path in run_dir.rglob("*"))
|
||||
|
||||
def _remove_run(self, run_id: str, run: dict) -> int:
|
||||
"""删除孤儿任务：数据库记录（含产物）、上传目录与步骤目录。"""
|
||||
self.db.delete_run(run_id)
|
||||
input_uri = run.get("input_uri")
|
||||
if input_uri:
|
||||
shutil.rmtree(Path(input_uri).parent, ignore_errors=True)
|
||||
shutil.rmtree(self.storage_dir / "runs" / run_id, ignore_errors=True)
|
||||
return 1
|
||||
@@ -0,0 +1,110 @@
|
||||
"""进程内节点注册表。
|
||||
|
||||
单体版不再启动子进程：节点清单与 invoke 处理器在启动时静态注册到本模块，
|
||||
调度器通过 invoke(node_id, request) 在同一个进程内直接调用处理器。
|
||||
协议数据模型（NodeManifest / InvokeRequest / InvokeResponse）保持不变，
|
||||
为将来回退分布式保留兼容桥梁。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
from wov_app.logging import get_logger
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse, NodeManifest
|
||||
|
||||
# 节点运行日志：输出到主进程控制台。
|
||||
logger = get_logger("node")
|
||||
|
||||
# 节点调用处理器签名：接收调用请求，返回调用结果。
|
||||
NodeHandler = Callable[[InvokeRequest], InvokeResponse]
|
||||
|
||||
|
||||
@dataclass
|
||||
class NodeEntry:
|
||||
"""注册表条目：节点清单与其进程内处理器。"""
|
||||
|
||||
manifest: NodeManifest
|
||||
handler: NodeHandler
|
||||
|
||||
|
||||
# 进程内注册表：key 为节点 ID（即工作流中的 node_type），value 为注册条目。
|
||||
_registry: dict[str, NodeEntry] = {}
|
||||
|
||||
|
||||
def _load_manifest(name: str) -> NodeManifest:
|
||||
"""从 manifests/ 目录加载节点清单文件并校验。"""
|
||||
path = Path(__file__).resolve().parent.parent.parent / "manifests" / f"{name}.json"
|
||||
return NodeManifest.load(str(path))
|
||||
|
||||
|
||||
def register(manifest: NodeManifest, handler: NodeHandler) -> None:
|
||||
"""注册单个节点；manifest 校验失败时抛出 ValueError。"""
|
||||
manifest.validate()
|
||||
_registry[manifest.id] = NodeEntry(manifest=manifest, handler=handler)
|
||||
|
||||
|
||||
def register_all() -> None:
|
||||
"""注册全部内置节点，启动时调用一次；重复调用按 ID 覆盖，幂等。"""
|
||||
from nodes import ass, echo, ffmpeg, frame_extract, llm, llm_filter, subtitle_ocr, vlm, whisper
|
||||
register(_load_manifest("echo"), echo.invoke)
|
||||
register(_load_manifest("ffmpeg"), ffmpeg.invoke)
|
||||
register(_load_manifest("whisper"), whisper.invoke)
|
||||
register(_load_manifest("llm"), llm.invoke)
|
||||
register(_load_manifest("vlm"), vlm.invoke)
|
||||
register(_load_manifest("frame-extract"), frame_extract.invoke)
|
||||
register(_load_manifest("subtitle-ocr"), subtitle_ocr.invoke)
|
||||
register(_load_manifest("llm-filter"), llm_filter.invoke)
|
||||
register(_load_manifest("ass"), ass.invoke)
|
||||
|
||||
|
||||
def list_nodes() -> list[NodeManifest]:
|
||||
"""按节点 ID 顺序返回全部已注册节点清单。"""
|
||||
return [entry.manifest for _, entry in sorted(_registry.items())]
|
||||
|
||||
|
||||
def get_node(node_id: str) -> NodeManifest | None:
|
||||
"""按节点 ID 返回清单；未注册时返回 None。"""
|
||||
entry = _registry.get(node_id)
|
||||
return entry.manifest if entry else None
|
||||
|
||||
|
||||
def invoke(node_id: str, request: InvokeRequest) -> InvokeResponse:
|
||||
"""调用指定节点的进程内处理器；节点未注册时抛出 ValueError。
|
||||
|
||||
统一在这里记录节点的开始/结束/耗时/产物日志，所有节点自动获得
|
||||
主进程可见的运行日志，无需在各节点实现内重复埋点。
|
||||
"""
|
||||
entry = _registry.get(node_id)
|
||||
if entry is None:
|
||||
raise ValueError(f"node not registered: {node_id}")
|
||||
logger.info(
|
||||
"节点 %s 开始 run=%s inputs=%s params=%s",
|
||||
node_id,
|
||||
request.run_id,
|
||||
request.inputs,
|
||||
request.params,
|
||||
)
|
||||
start = time.perf_counter()
|
||||
response = entry.handler(request)
|
||||
elapsed = time.perf_counter() - start
|
||||
if response.status == "completed":
|
||||
logger.info(
|
||||
"节点 %s 完成 run=%s 耗时=%.2fs outputs=%s",
|
||||
node_id,
|
||||
request.run_id,
|
||||
elapsed,
|
||||
response.outputs,
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"节点 %s 失败 run=%s 耗时=%.2fs error=%s",
|
||||
node_id,
|
||||
request.run_id,
|
||||
elapsed,
|
||||
response.error,
|
||||
)
|
||||
return response
|
||||
@@ -0,0 +1,5 @@
|
||||
"""WOV 单体 API 路由包。
|
||||
|
||||
按职责拆分为用户应用与工作流管理两组路由，统一由 wov_app.main 挂载。
|
||||
节点注册/实例管理路由已随单体化移除。
|
||||
"""
|
||||
Executable
+213
@@ -0,0 +1,213 @@
|
||||
"""用户端应用路由。
|
||||
|
||||
面向普通用户暴露“应用中心”能力：列出已发布工作流、上传输入创建任务、
|
||||
查询进度、重试失败任务以及下载产物。用户只看到输入 -> 进度 -> 结果。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from wov_app.db import Database
|
||||
|
||||
router = APIRouter(tags=["apps"])
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
"""返回当前 UTC 时间的 ISO 格式字符串。"""
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _get_db() -> Database:
|
||||
"""从 FastAPI 应用状态中延迟获取数据库实例。"""
|
||||
from wov_app.main import app
|
||||
|
||||
return app.state.db
|
||||
|
||||
|
||||
@router.get("/api/apps")
|
||||
def list_apps(db: Database = Depends(_get_db)) -> list[dict]:
|
||||
"""返回全部已发布工作流及其最新版本定义。"""
|
||||
apps = []
|
||||
for workflow in db.list_workflows():
|
||||
# 草稿工作流不对用户端可见。
|
||||
if not workflow["published"]:
|
||||
continue
|
||||
latest = db.get_latest_workflow_version(workflow["id"])
|
||||
apps.append(
|
||||
{
|
||||
"id": workflow["id"],
|
||||
"name": workflow["name"],
|
||||
"description": workflow["description"],
|
||||
"version": workflow["latest_version"],
|
||||
"definition": latest["definition"] if latest else None,
|
||||
}
|
||||
)
|
||||
return apps
|
||||
|
||||
|
||||
@router.post("/api/apps/{workflow_id}/runs")
|
||||
async def create_run(
|
||||
workflow_id: str,
|
||||
file: UploadFile = File(...),
|
||||
params: str = Form(default=""),
|
||||
db: Database = Depends(_get_db),
|
||||
) -> dict:
|
||||
"""接收用户上传文件，创建排队中的工作流任务。"""
|
||||
workflow = db.get_workflow(workflow_id)
|
||||
# 只允许对已发布且存在版本的工作流发起任务。
|
||||
if workflow is None or not workflow["published"]:
|
||||
raise HTTPException(status_code=404, detail="published workflow not found")
|
||||
|
||||
latest = db.get_latest_workflow_version(workflow_id)
|
||||
if latest is None:
|
||||
raise HTTPException(status_code=422, detail="workflow has no version")
|
||||
|
||||
run_id = f"run_{uuid.uuid4().hex[:12]}"
|
||||
# 使用安全文件名，避免路径穿越。
|
||||
filename = Path(file.filename or "upload.bin").name
|
||||
from wov_app.config import STORAGE_DIR
|
||||
|
||||
# 上传文件按 run 隔离存放，调度器通过 input_uri 引用。
|
||||
input_dir = STORAGE_DIR / "uploads" / run_id
|
||||
input_dir.mkdir(parents=True, exist_ok=True)
|
||||
input_uri = input_dir / filename
|
||||
content = await file.read()
|
||||
input_uri.write_bytes(content)
|
||||
|
||||
# 可选参数覆盖（如前端框选的 crop）：{节点ID: {参数: 值}}，随任务持久化。
|
||||
param_overrides = None
|
||||
if params.strip():
|
||||
try:
|
||||
parsed = json.loads(params)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise HTTPException(status_code=422, detail="params must be valid JSON") from exc
|
||||
if not isinstance(parsed, dict):
|
||||
raise HTTPException(status_code=422, detail="params must be a JSON object")
|
||||
param_overrides = parsed
|
||||
|
||||
now = _now_iso()
|
||||
db.create_run(
|
||||
{
|
||||
"id": run_id,
|
||||
"workflow_id": workflow_id,
|
||||
"workflow_version": latest["version"],
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(input_uri),
|
||||
"param_overrides": param_overrides,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
return {
|
||||
"id": run_id,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"artifacts": [],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/runs")
|
||||
def list_runs(db: Database = Depends(_get_db)) -> list[dict]:
|
||||
"""返回最近的运行记录，供任务管理页展示。"""
|
||||
return db.list_runs()
|
||||
|
||||
|
||||
@router.get("/api/runs/{run_id}")
|
||||
def get_run(run_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""返回任务详情，并附带当前产物列表。"""
|
||||
run = db.get_run(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
run["artifacts"] = db.list_artifacts(run_id)
|
||||
return run
|
||||
|
||||
|
||||
@router.post("/api/runs/{run_id}/retry")
|
||||
def retry_run(run_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""重置失败任务为排队状态，清空旧产物后重新执行。"""
|
||||
run = db.get_run(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
if run["status"] != "FAILED":
|
||||
raise HTTPException(status_code=422, detail="only failed runs can be retried")
|
||||
# reset_run 会清空进度、错误和旧产物，确保从头开始。
|
||||
db.reset_run(run_id, _now_iso())
|
||||
return {"id": run_id, "status": "QUEUED"}
|
||||
|
||||
|
||||
@router.post("/api/runs/{run_id}/pause")
|
||||
def pause_run(run_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""暂停任务：排队中或运行中的任务可暂停，运行中的任务在节点边界停下。"""
|
||||
run = db.get_run(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
if run["status"] not in ("QUEUED", "RUNNING"):
|
||||
raise HTTPException(status_code=422, detail="only queued or running runs can be paused")
|
||||
db.pause_run(run_id, _now_iso())
|
||||
return {"id": run_id, "status": "PAUSED"}
|
||||
|
||||
|
||||
@router.post("/api/runs/{run_id}/resume")
|
||||
def resume_run(run_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""继续任务：暂停的任务恢复排队，由调度器从断点继续执行。"""
|
||||
run = db.get_run(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
if run["status"] != "PAUSED":
|
||||
raise HTTPException(status_code=422, detail="only paused runs can be resumed")
|
||||
db.resume_run(run_id, _now_iso())
|
||||
return {"id": run_id, "status": "QUEUED"}
|
||||
|
||||
@router.delete("/api/runs/{run_id}")
|
||||
def delete_run(run_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""删除任务：清理产物记录、上传文件与步骤产物目录。"""
|
||||
run = db.get_run(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
from wov_app.config import STORAGE_DIR
|
||||
|
||||
# 先删数据库记录（含产物表），再清理磁盘上的上传与中间产物。
|
||||
db.delete_run(run_id)
|
||||
input_uri = run.get("input_uri")
|
||||
if input_uri:
|
||||
# 上传文件位于 <storage>/uploads/<run_id>/，整目录一并删除。
|
||||
shutil.rmtree(Path(input_uri).parent, ignore_errors=True)
|
||||
# 步骤产物位于 <storage>/runs/<run_id>/，整目录一并删除。
|
||||
shutil.rmtree(STORAGE_DIR / "runs" / run_id, ignore_errors=True)
|
||||
return {"deleted": run_id}
|
||||
|
||||
@router.get("/api/runs/{run_id}/artifacts")
|
||||
def list_run_artifacts(run_id: str, db: Database = Depends(_get_db)) -> list[dict]:
|
||||
"""返回任务全部产物记录。"""
|
||||
if db.get_run(run_id) is None:
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
return db.list_artifacts(run_id)
|
||||
|
||||
|
||||
@router.get("/api/runs/{run_id}/artifacts/{artifact_name}")
|
||||
def download_artifact(
|
||||
run_id: str,
|
||||
artifact_name: str,
|
||||
db: Database = Depends(_get_db),
|
||||
) -> FileResponse:
|
||||
"""按任务与产物名下载文件，文件缺失时返回 404。"""
|
||||
artifact = db.get_artifact(run_id, artifact_name)
|
||||
if artifact is None:
|
||||
raise HTTPException(status_code=404, detail="artifact not found")
|
||||
path = Path(artifact["uri"])
|
||||
if not path.is_file():
|
||||
raise HTTPException(status_code=404, detail="artifact file missing")
|
||||
return FileResponse(
|
||||
path,
|
||||
media_type=artifact["mime_type"],
|
||||
filename=path.name,
|
||||
)
|
||||
Executable
+140
@@ -0,0 +1,140 @@
|
||||
"""工作流管理路由。
|
||||
|
||||
提供工作流的创建、查询、校验、发布和删除能力。工作流以版本化 DAG 数据保存，
|
||||
不写死在业务代码中。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
|
||||
from wov_app.db import Database
|
||||
from wov_app.schemas import WorkflowCreate
|
||||
from wov_sdk.models import WorkflowDefinition
|
||||
|
||||
router = APIRouter(prefix="/api/admin/workflows", tags=["workflows"])
|
||||
|
||||
|
||||
def _get_db() -> Database:
|
||||
"""从应用状态延迟获取数据库实例。"""
|
||||
from wov_app.main import app
|
||||
|
||||
return app.state.db
|
||||
|
||||
|
||||
def _slugify(value: str) -> str:
|
||||
"""把工作流名称转换为小写连字符 ID；无有效字符时生成随机 ID。"""
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
|
||||
return slug or uuid.uuid4().hex[:8]
|
||||
|
||||
|
||||
def _validate_definition(raw: dict) -> WorkflowDefinition:
|
||||
"""解析并校验 DAG 定义，非法时转换为 422 HTTP 异常。"""
|
||||
try:
|
||||
definition = WorkflowDefinition.from_dict(raw)
|
||||
definition.validate()
|
||||
return definition
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_workflows(db: Database = Depends(_get_db)) -> list[dict]:
|
||||
"""返回全部工作流概要。"""
|
||||
return db.list_workflows()
|
||||
|
||||
|
||||
@router.post("")
|
||||
def create_workflow(
|
||||
payload: WorkflowCreate,
|
||||
db: Database = Depends(_get_db),
|
||||
) -> dict:
|
||||
"""创建新工作流或为已有工作流追加一个版本。"""
|
||||
definition = _validate_definition(payload.definition)
|
||||
# 未显式指定 ID 时由名称生成；已有工作流则版本号递增。
|
||||
workflow_id = payload.id or _slugify(payload.name)
|
||||
existing = db.get_workflow(workflow_id)
|
||||
version = (existing or {}).get("latest_version", 0) + 1
|
||||
# 每次创建都保存新版本，发布操作只切换 published 标记。
|
||||
db.upsert_workflow(
|
||||
{
|
||||
"id": workflow_id,
|
||||
"name": payload.name,
|
||||
"description": payload.description,
|
||||
"published": 0,
|
||||
"latest_version": version,
|
||||
}
|
||||
)
|
||||
db.create_workflow_version(workflow_id, version, definition.to_dict())
|
||||
return {
|
||||
"id": workflow_id,
|
||||
"name": payload.name,
|
||||
"description": payload.description,
|
||||
"published": False,
|
||||
"latest_version": version,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{workflow_id}")
|
||||
def get_workflow(workflow_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""返回工作流概要及最新版本定义。"""
|
||||
workflow = db.get_workflow(workflow_id)
|
||||
if workflow is None:
|
||||
raise HTTPException(status_code=404, detail="workflow not found")
|
||||
latest = db.get_latest_workflow_version(workflow_id)
|
||||
workflow["latest_version_data"] = latest
|
||||
return workflow
|
||||
|
||||
|
||||
@router.delete("/{workflow_id}")
|
||||
def delete_workflow(workflow_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""删除工作流及其版本、任务和产物记录。"""
|
||||
if db.get_workflow(workflow_id) is None:
|
||||
raise HTTPException(status_code=404, detail="workflow not found")
|
||||
db.delete_workflow(workflow_id)
|
||||
return {"deleted": workflow_id}
|
||||
|
||||
|
||||
@router.post("/{workflow_id}/validate")
|
||||
def validate_workflow(
|
||||
workflow_id: str,
|
||||
definition: dict,
|
||||
db: Database = Depends(_get_db),
|
||||
) -> dict:
|
||||
"""在不保存的情况下校验一份 DAG 定义。"""
|
||||
if db.get_workflow(workflow_id) is None:
|
||||
raise HTTPException(status_code=404, detail="workflow not found")
|
||||
parsed = _validate_definition(definition)
|
||||
return {"valid": True, "node_ids": [node.id for node in parsed.nodes]}
|
||||
|
||||
|
||||
@router.post("/{workflow_id}/publish")
|
||||
def publish_workflow(workflow_id: str, db: Database = Depends(_get_db)) -> dict:
|
||||
"""把工作流标记为已发布，使其出现在用户应用中心。"""
|
||||
workflow = db.get_workflow(workflow_id)
|
||||
if workflow is None:
|
||||
raise HTTPException(status_code=404, detail="workflow not found")
|
||||
if workflow["latest_version"] == 0:
|
||||
raise HTTPException(status_code=422, detail="workflow has no version")
|
||||
# 发布只是状态切换，不修改已保存的版本数据。
|
||||
db.upsert_workflow(
|
||||
{
|
||||
"id": workflow_id,
|
||||
"name": workflow["name"],
|
||||
"description": workflow["description"],
|
||||
"published": 1,
|
||||
"latest_version": workflow["latest_version"],
|
||||
}
|
||||
)
|
||||
return {"published": workflow_id}
|
||||
|
||||
|
||||
@router.get("/{workflow_id}/versions")
|
||||
def list_versions(workflow_id: str, db: Database = Depends(_get_db)) -> list[dict]:
|
||||
"""返回工作流全部版本定义。"""
|
||||
if db.get_workflow(workflow_id) is None:
|
||||
raise HTTPException(status_code=404, detail="workflow not found")
|
||||
return db.list_workflow_versions(workflow_id)
|
||||
Executable
+308
@@ -0,0 +1,308 @@
|
||||
"""工作流调度器。
|
||||
|
||||
轮询 SQLite 中的排队任务，按工作流 DAG 的拓扑顺序依次调用进程内节点
|
||||
处理器，并把节点产物登记为任务产物。单体版使用单线程顺序执行，节点在
|
||||
同一进程内直接调用，不再经过子进程与 HTTP 协议。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, WorkflowDefinition
|
||||
|
||||
from wov_app import registry
|
||||
from wov_app.config import SCHEDULER_INTERVAL_SECONDS
|
||||
from wov_app.db import Database
|
||||
from wov_app.logging import get_logger
|
||||
|
||||
# 调度器运行日志：节点进度、暂停/续跑等状态变化。
|
||||
logger = get_logger("scheduler")
|
||||
|
||||
def _now_iso() -> str:
|
||||
"""返回当前 UTC 时间的 ISO 格式字符串。"""
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def topological_sort(definition: WorkflowDefinition) -> list[str]:
|
||||
"""对工作流 DAG 做拓扑排序，返回可执行的节点 ID 顺序。"""
|
||||
nodes = {node.id: node for node in definition.nodes}
|
||||
# 统计每个节点的入度，并记录依赖关系。
|
||||
indegree = {node_id: 0 for node_id in nodes}
|
||||
dependents: dict[str, list[str]] = {node_id: [] for node_id in nodes}
|
||||
|
||||
for edge in definition.edges:
|
||||
# 边引用了不存在的节点时直接报错。
|
||||
if edge.from_node not in nodes or edge.to_node not in nodes:
|
||||
raise ValueError(f"unknown edge: {edge.from_node} -> {edge.to_node}")
|
||||
indegree[edge.to_node] += 1
|
||||
dependents[edge.from_node].append(edge.to_node)
|
||||
|
||||
# Kahn 算法：从入度为 0 的节点开始逐层取出。
|
||||
queue = [node_id for node_id, degree in indegree.items() if degree == 0]
|
||||
ordered: list[str] = []
|
||||
while queue:
|
||||
current = queue.pop(0)
|
||||
ordered.append(current)
|
||||
for dependent in dependents[current]:
|
||||
indegree[dependent] -= 1
|
||||
if indegree[dependent] == 0:
|
||||
queue.append(dependent)
|
||||
|
||||
# 排序结果数量不足说明存在环，无法确定执行顺序。
|
||||
if len(ordered) != len(nodes):
|
||||
raise ValueError("workflow contains a cycle")
|
||||
return ordered
|
||||
|
||||
|
||||
class WorkflowScheduler:
|
||||
"""后台任务调度器：单线程轮询并执行排队中的工作流运行。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
db: Database,
|
||||
storage_dir: Path,
|
||||
interval_seconds: float | None = None,
|
||||
) -> None:
|
||||
"""保存依赖并初始化轮询线程控制字段。"""
|
||||
self.db = db
|
||||
self.storage_dir = storage_dir
|
||||
self.interval_seconds = interval_seconds or SCHEDULER_INTERVAL_SECONDS
|
||||
self._thread: threading.Thread | None = None
|
||||
self._stopping = False
|
||||
|
||||
def start(self) -> None:
|
||||
"""启动调度线程；重复调用无副作用。"""
|
||||
if self._thread is not None:
|
||||
return
|
||||
self._stopping = False
|
||||
self._thread = threading.Thread(
|
||||
target=self._loop,
|
||||
name="wov-workflow-scheduler",
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
"""请求停止并等待轮询线程退出。"""
|
||||
self._stopping = True
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=5)
|
||||
self._thread = None
|
||||
|
||||
def _loop(self) -> None:
|
||||
"""轮询循环：有排队任务就立即执行，否则休眠一个间隔。"""
|
||||
while not self._stopping:
|
||||
run = self.db.next_queued_run()
|
||||
if run is not None:
|
||||
self.execute_run(run["id"])
|
||||
else:
|
||||
time.sleep(self.interval_seconds)
|
||||
|
||||
def _resolve_ref(
|
||||
self,
|
||||
ref: str,
|
||||
run_input_uri: str | None,
|
||||
outputs_by_node: dict[str, dict[str, str]],
|
||||
) -> str | None:
|
||||
"""解析输入引用：input.xxx 取任务入口，node.key 取前序节点产物。"""
|
||||
# 入口引用以 input. 为前缀。
|
||||
if ref.startswith("input."):
|
||||
return run_input_uri
|
||||
# 其余引用必须形如 "节点ID.输出名"。
|
||||
node_id, separator, key = ref.partition(".")
|
||||
if not separator:
|
||||
return None
|
||||
return outputs_by_node.get(node_id, {}).get(key)
|
||||
|
||||
def execute_run(self, run_id: str) -> None:
|
||||
"""执行单个任务：加载 DAG、按拓扑顺序调用节点并登记产物。"""
|
||||
run = self.db.get_run(run_id)
|
||||
# 任务不存在或不在可执行状态（排队/暂停）时直接返回，避免重复执行。
|
||||
if run is None or run["status"] not in ("QUEUED", "PAUSED"):
|
||||
return
|
||||
|
||||
# 工作流或版本记录丢失时把任务标记为失败。
|
||||
workflow = self.db.get_workflow(run["workflow_id"])
|
||||
if workflow is None:
|
||||
self.db.update_run(run_id, status="FAILED", error="workflow not found", updated_at=_now_iso())
|
||||
return
|
||||
|
||||
version = self.db.get_workflow_version(run["workflow_id"], run["workflow_version"])
|
||||
if version is None:
|
||||
self.db.update_run(run_id, status="FAILED", error="workflow version not found", updated_at=_now_iso())
|
||||
return
|
||||
|
||||
# 解析并校验 DAG，随后计算拓扑执行顺序。
|
||||
definition = WorkflowDefinition.from_dict(version["definition"])
|
||||
definition.validate()
|
||||
ordered = topological_sort(definition)
|
||||
# 从已登记产物重建已完成节点的输出，支持暂停后断点续跑。
|
||||
outputs_by_node = self.db.restore_run_outputs(run_id)
|
||||
run_started = time.monotonic()
|
||||
self.db.update_run(run_id, status="RUNNING", progress=0, updated_at=_now_iso())
|
||||
try:
|
||||
for index, node_id in enumerate(ordered):
|
||||
# 暂停检查：用户暂停后调度器在节点边界停下，保持 PAUSED 等待续跑。
|
||||
current = self.db.get_run(run_id)
|
||||
if current is None or current["status"] == "PAUSED":
|
||||
logger.info("任务 %s 已暂停，停止在节点 %s 之前", run_id, node_id)
|
||||
return
|
||||
# 断点续跑：跳过已产出结果的节点（其产物已作为输入可用）。
|
||||
if node_id in outputs_by_node:
|
||||
continue
|
||||
# 当前节点进度 = 已完成节点数 / 总节点数。
|
||||
node_spec = next(item for item in definition.nodes if item.id == node_id)
|
||||
self.db.update_run(
|
||||
run_id,
|
||||
current_node_id=node_id,
|
||||
progress=index / len(ordered),
|
||||
updated_at=_now_iso(),
|
||||
)
|
||||
# 解析节点声明的每个输入引用，缺任一输入即失败。
|
||||
invoke_inputs: dict[str, str] = {}
|
||||
for input_name, ref in node_spec.inputs.items():
|
||||
value = self._resolve_ref(ref, run.get("input_uri"), outputs_by_node)
|
||||
if value is None:
|
||||
raise ValueError(f"missing input {input_name} for node {node_id}")
|
||||
invoke_inputs[input_name] = value
|
||||
|
||||
# 前端框选的 crop 等参数覆盖：按节点 ID 合并进节点参数。
|
||||
node_params = dict(node_spec.params)
|
||||
overrides = run.get("param_overrides") or {}
|
||||
node_params.update(overrides.get(node_id, {}))
|
||||
|
||||
# 每个任务的每个节点使用独立产物目录，避免并发冲突。
|
||||
node_started = time.monotonic()
|
||||
output_dir = (
|
||||
self.storage_dir
|
||||
/ "runs"
|
||||
/ run_id
|
||||
/ "steps"
|
||||
/ node_id
|
||||
)
|
||||
response = registry.invoke(
|
||||
node_spec.node_type,
|
||||
InvokeRequest(
|
||||
run_id=run_id,
|
||||
node_instance_id="",
|
||||
inputs=invoke_inputs,
|
||||
params=node_params,
|
||||
output_dir=str(output_dir),
|
||||
),
|
||||
)
|
||||
# 节点返回非 completed 即视为步骤失败。
|
||||
if response.status != "completed":
|
||||
raise RuntimeError(response.error or f"node {node_id} failed")
|
||||
|
||||
# 记录节点输出，供后续节点引用和最终产物映射使用。
|
||||
outputs_by_node[node_id] = {
|
||||
str(key): str(value) for key, value in response.outputs.items()
|
||||
}
|
||||
for key, uri in outputs_by_node[node_id].items():
|
||||
# 产物名带节点前缀，例如 asr.srt_uri，避免跨节点重名。
|
||||
artifact = {
|
||||
"run_id": run_id,
|
||||
"node_id": node_id,
|
||||
"name": f"{node_id}.{key}",
|
||||
"uri": uri,
|
||||
"mime_type": self._mime_type(uri),
|
||||
"size": self._file_size(uri),
|
||||
}
|
||||
self.db.create_artifact(artifact)
|
||||
# 进度日志：节点序号/总数、耗时与任务累计运行时间（数据速度可观测）。
|
||||
logger.info(
|
||||
"任务 %s 进度 %d/%d 节点: %s 耗时 %.1fs, 运行累计 %.1fs",
|
||||
run_id, index + 1, len(ordered), node_id,
|
||||
time.monotonic() - node_started,
|
||||
time.monotonic() - run_started,
|
||||
)
|
||||
# 处理 final_outputs，为用户端提供简洁的下载别名。
|
||||
for alias, ref in definition.final_outputs.items():
|
||||
resolved = self._resolve_ref(ref, run.get("input_uri"), outputs_by_node)
|
||||
if resolved is not None:
|
||||
# 最终产物按 上传文件名.标识.时间戳 重命名，区分语言与版本。
|
||||
resolved = self._final_artifact_uri(resolved, run, definition, alias, ref)
|
||||
self.db.create_artifact(
|
||||
{
|
||||
"run_id": run_id,
|
||||
"node_id": ref.partition(".")[0],
|
||||
"name": alias,
|
||||
"uri": resolved,
|
||||
"mime_type": self._mime_type(resolved),
|
||||
"size": self._file_size(resolved),
|
||||
}
|
||||
)
|
||||
|
||||
# 全部节点成功后标记完成；期间被暂停则保持 PAUSED，等待续跑补做收尾。
|
||||
if self.db.get_run(run_id)["status"] == "PAUSED":
|
||||
logger.info("任务 %s 节点全部完成但已暂停，保持 PAUSED", run_id)
|
||||
return
|
||||
self.db.update_run(
|
||||
run_id,
|
||||
status="COMPLETED",
|
||||
current_node_id=None,
|
||||
progress=1.0,
|
||||
updated_at=_now_iso(),
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 任一步骤异常都结束任务并记录错误，等待用户重试。
|
||||
self.db.update_run(
|
||||
run_id,
|
||||
status="FAILED",
|
||||
error=str(exc),
|
||||
updated_at=_now_iso(),
|
||||
)
|
||||
|
||||
def _final_artifact_uri(
|
||||
self,
|
||||
resolved: str,
|
||||
run: dict,
|
||||
definition: WorkflowDefinition,
|
||||
alias: str,
|
||||
ref: str,
|
||||
) -> str:
|
||||
"""把最终产物重命名为 上传文件名.标识.时间戳 并返回新 URI。
|
||||
|
||||
标识优先取产出节点的 target_language 参数（如 zh-CN），否则回退为
|
||||
产物别名；时间戳取当前时刻，用于区分同一上传文件的多次运行版本。
|
||||
重命名在原地进行（同目录），不复制文件。
|
||||
"""
|
||||
source = Path(resolved)
|
||||
# 续跑等场景下源文件可能已被上次收尾重命名过：不再重命名，原样返回。
|
||||
if not source.is_file():
|
||||
return resolved
|
||||
# 基础名来自上传文件名；无上传文件时退回通用名称 subtitle。
|
||||
base = Path(run["input_uri"]).stem if run.get("input_uri") else "subtitle"
|
||||
# 通过最终输出引用定位产出节点，取其语言参数作为标识。
|
||||
node_id = ref.partition(".")[0]
|
||||
node = next((item for item in definition.nodes if item.id == node_id), None)
|
||||
tag = (node.params.get("target_language") if node else None) or alias
|
||||
timestamp = datetime.now(timezone.utc).strftime("%Y%m%d%H%M%S")
|
||||
new_path = source.with_name(f"{base}.{tag}.{timestamp}{source.suffix}")
|
||||
source.rename(new_path)
|
||||
return str(new_path)
|
||||
|
||||
@staticmethod
|
||||
def _mime_type(uri: str) -> str:
|
||||
"""按扩展名推断产物 MIME 类型，未知类型使用通用二进制类型。"""
|
||||
path = Path(uri)
|
||||
suffix = path.suffix.lower()
|
||||
return {
|
||||
".srt": "application/x-subrip",
|
||||
".ass": "text/plain",
|
||||
".wav": "audio/wav",
|
||||
".mp4": "video/mp4",
|
||||
".txt": "text/plain",
|
||||
}.get(suffix, "application/octet-stream")
|
||||
|
||||
@staticmethod
|
||||
def _file_size(uri: str) -> int:
|
||||
"""读取产物文件大小；文件缺失时按 0 处理。"""
|
||||
try:
|
||||
return Path(uri).stat().st_size
|
||||
except OSError:
|
||||
return 0
|
||||
@@ -0,0 +1,22 @@
|
||||
"""FastAPI 请求/响应 schema。
|
||||
|
||||
使用 Pydantic 模型校验管理 API 的 JSON 请求体。节点管理功能已移除，仅保留
|
||||
工作流相关请求模型。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class WorkflowCreate(BaseModel):
|
||||
"""创建工作流或新增版本的请求体。"""
|
||||
|
||||
# 缺省时由后端根据名称生成 slug ID。
|
||||
id: str | None = None
|
||||
name: str = Field(min_length=1)
|
||||
description: str = ""
|
||||
# DAG 原始字典，后端会解析并校验为 WorkflowDefinition。
|
||||
definition: dict[str, Any]
|
||||
@@ -0,0 +1,48 @@
|
||||
"""种子数据模块。
|
||||
|
||||
从 workflows/*.json 数据文件加载默认工作流并写入数据库（幂等）。
|
||||
工作流定义是**数据**（JSON）：切换模型、调整链路只改数据文件，不涉及代码，
|
||||
满足"工作流即数据"与"切换模型不改代码"的设计约束。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import WorkflowDefinition
|
||||
|
||||
from wov_app.config import WORKSPACE_ROOT
|
||||
from wov_app.db import Database
|
||||
|
||||
|
||||
def seed_default_workflows(db: Database, workflows_dir: Path | None = None) -> int:
|
||||
"""从数据目录加载默认工作流，已存在的工作流跳过，返回创建数量。
|
||||
|
||||
每个 JSON 文件结构：
|
||||
{"id", "name", "description", "version", "definition"}，
|
||||
definition 为 WorkflowDefinition 的标准 DAG 字典。
|
||||
"""
|
||||
workflows_dir = workflows_dir or (WORKSPACE_ROOT / "workflows")
|
||||
created = 0
|
||||
for path in sorted(workflows_dir.glob("*.json")):
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
workflow_id = str(payload["id"])
|
||||
# 已存在的工作流不覆盖，避免启动时反复改写用户数据。
|
||||
if db.get_workflow(workflow_id) is not None:
|
||||
continue
|
||||
definition = WorkflowDefinition.from_dict(payload["definition"])
|
||||
definition.validate()
|
||||
version = int(payload.get("version", 1))
|
||||
db.upsert_workflow(
|
||||
{
|
||||
"id": workflow_id,
|
||||
"name": str(payload["name"]),
|
||||
"description": str(payload.get("description", "")),
|
||||
"published": 1,
|
||||
"latest_version": version,
|
||||
}
|
||||
)
|
||||
db.create_workflow_version(workflow_id, version, definition.to_dict())
|
||||
created += 1
|
||||
return created
|
||||
@@ -0,0 +1,28 @@
|
||||
"""WOV SDK 公共导出入口。
|
||||
|
||||
单体版中调度器、节点与 API 统一从 wov_sdk 导入协议模型，而无需关心具体
|
||||
模块路径。协议数据模型保持与分布式版一致，为将来回退保留兼容桥梁。
|
||||
"""
|
||||
|
||||
from wov_sdk.models import (
|
||||
HealthResponse,
|
||||
InvokeRequest,
|
||||
InvokeResponse,
|
||||
NodeManifest,
|
||||
ProgressEvent,
|
||||
WorkflowDefinition,
|
||||
WorkflowEdge,
|
||||
WorkflowNode,
|
||||
)
|
||||
|
||||
# 对外稳定的公共 API 清单；新增模型时必须同步追加到这里。
|
||||
__all__ = [
|
||||
"HealthResponse",
|
||||
"InvokeRequest",
|
||||
"InvokeResponse",
|
||||
"NodeManifest",
|
||||
"ProgressEvent",
|
||||
"WorkflowDefinition",
|
||||
"WorkflowEdge",
|
||||
"WorkflowNode",
|
||||
]
|
||||
Executable
+321
@@ -0,0 +1,321 @@
|
||||
"""WOV 节点协议核心数据模型。
|
||||
|
||||
本模块定义节点 Manifest、调用请求/响应、健康检查、进度事件以及工作流 DAG 的
|
||||
通用数据结构。单体版中调度器、节点与 API 共用这些类，字段语义必须长期保持
|
||||
稳定，新增能力时只能向后兼容地扩展字段。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _require_non_empty(value: str, name: str) -> None:
|
||||
"""校验必填字符串字段，空字符串或纯空白字符串都会被拒绝。"""
|
||||
if not value or not value.strip():
|
||||
raise ValueError(f"{name} must not be empty")
|
||||
|
||||
|
||||
@dataclass
|
||||
class NodeManifest:
|
||||
"""节点注册清单：描述节点能力、输入输出与资源参数。
|
||||
|
||||
该清单由 manifests/ 目录下的 JSON 文件提供，单体启动时注册到进程内
|
||||
节点注册表，调度器据此把 node_type 解析到对应的 invoke 处理器。
|
||||
"""
|
||||
|
||||
# 节点稳定唯一 ID，例如 faster-whisper；注册后不可随意更改。
|
||||
id: str
|
||||
# 展示名称，仅用于管理后台等界面。
|
||||
name: str
|
||||
# 节点版本号，与节点仓库 Git tag 保持一致。
|
||||
version: str
|
||||
# 能力标识，工作流通过 node_type 引用能力，而不是直接绑定具体仓库。
|
||||
capability: str
|
||||
# 启动命令；单体版不再启动子进程，字段保留仅为协议兼容。
|
||||
command: list[str]
|
||||
# 节点代码所在目录；单体版保留仅为协议兼容。
|
||||
repo_dir: str = "."
|
||||
# 节点环境变量；单体版保留仅为协议兼容。
|
||||
env: dict[str, str] = field(default_factory=dict)
|
||||
# 输入字段 schema，当前主要用于文档展示，后续可用于运行时校验。
|
||||
input_schema: dict[str, Any] = field(default_factory=dict)
|
||||
# 输出字段 schema，用于描述节点产物的名称与类型。
|
||||
output_schema: dict[str, Any] = field(default_factory=dict)
|
||||
# 单进程内无并发槽位概念，字段保留仅为协议兼容。
|
||||
max_concurrency: int = 1
|
||||
# 单体内模型常驻不回收，字段保留仅为协议兼容。
|
||||
idle_ttl_seconds: int = 300
|
||||
# 无进程启动等待，字段保留仅为协议兼容。
|
||||
health_timeout_seconds: int = 10
|
||||
# 单体内模型常驻不回收，字段保留仅为协议兼容。
|
||||
keep_warm: bool = False
|
||||
|
||||
def validate(self) -> None:
|
||||
"""校验 manifest 必填字段与数值边界，非法配置抛出 ValueError。"""
|
||||
_require_non_empty(self.id, "id")
|
||||
_require_non_empty(self.name, "name")
|
||||
_require_non_empty(self.version, "version")
|
||||
_require_non_empty(self.capability, "capability")
|
||||
_require_non_empty(self.repo_dir, "repo_dir")
|
||||
# 命令不能为空，否则节点进程无法启动。
|
||||
if not self.command:
|
||||
raise ValueError("command must not be empty")
|
||||
if self.max_concurrency < 1:
|
||||
raise ValueError("max_concurrency must be >= 1")
|
||||
if self.idle_ttl_seconds < 0:
|
||||
raise ValueError("idle_ttl_seconds must be >= 0")
|
||||
if self.health_timeout_seconds < 1:
|
||||
raise ValueError("health_timeout_seconds must be >= 1")
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为可 JSON 序列化的普通字典。"""
|
||||
return {
|
||||
"id": self.id,
|
||||
"name": self.name,
|
||||
"version": self.version,
|
||||
"capability": self.capability,
|
||||
"command": self.command,
|
||||
"repo_dir": self.repo_dir,
|
||||
"env": self.env,
|
||||
"input_schema": self.input_schema,
|
||||
"output_schema": self.output_schema,
|
||||
"max_concurrency": self.max_concurrency,
|
||||
"idle_ttl_seconds": self.idle_ttl_seconds,
|
||||
"health_timeout_seconds": self.health_timeout_seconds,
|
||||
"keep_warm": self.keep_warm,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "NodeManifest":
|
||||
"""从注册 API 或 JSON 文件解析出的字典恢复 manifest。"""
|
||||
return cls(
|
||||
id=str(data["id"]),
|
||||
name=str(data["name"]),
|
||||
version=str(data["version"]),
|
||||
capability=str(data["capability"]),
|
||||
# command 可能缺省，解析时提供空列表兜底。
|
||||
command=[str(item) for item in data.get("command", [])],
|
||||
repo_dir=str(data.get("repo_dir", ".")),
|
||||
env={str(k): str(v) for k, v in data.get("env", {}).items()},
|
||||
input_schema=dict(data.get("input_schema", {})),
|
||||
output_schema=dict(data.get("output_schema", {})),
|
||||
# 数值字段缺省时使用与 dataclass 一致的默认值。
|
||||
max_concurrency=int(data.get("max_concurrency", 1)),
|
||||
idle_ttl_seconds=int(data.get("idle_ttl_seconds", 300)),
|
||||
health_timeout_seconds=int(data.get("health_timeout_seconds", 10)),
|
||||
keep_warm=bool(data.get("keep_warm", False)),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: str) -> "NodeManifest":
|
||||
"""从磁盘上的 node.manifest.json 加载并校验 manifest。"""
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
manifest = cls.from_dict(json.load(f))
|
||||
manifest.validate()
|
||||
return manifest
|
||||
|
||||
|
||||
@dataclass
|
||||
class InvokeRequest:
|
||||
"""节点调用请求：由调度器或管理后台发送给节点 HTTP 服务。"""
|
||||
|
||||
# 工作流运行 ID，用于追踪一次完整执行。
|
||||
run_id: str
|
||||
# 实际承载本次调用的节点实例 ID，由 NodeManager 回填。
|
||||
node_instance_id: str
|
||||
# 输入产物映射，key 为输入名，value 为产物 URI 或直接文本。
|
||||
inputs: dict[str, Any] = field(default_factory=dict)
|
||||
# 节点运行参数，例如采样率、语言、模型路径等。
|
||||
params: dict[str, Any] = field(default_factory=dict)
|
||||
# 节点产物输出目录。
|
||||
output_dir: str = "."
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为可 JSON 序列化的普通字典。"""
|
||||
return {
|
||||
"run_id": self.run_id,
|
||||
"node_instance_id": self.node_instance_id,
|
||||
"inputs": self.inputs,
|
||||
"params": self.params,
|
||||
"output_dir": self.output_dir,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "InvokeRequest":
|
||||
"""从 HTTP 请求 JSON 解析调用请求。"""
|
||||
return cls(
|
||||
run_id=str(data["run_id"]),
|
||||
node_instance_id=str(data["node_instance_id"]),
|
||||
inputs=dict(data.get("inputs", {})),
|
||||
params=dict(data.get("params", {})),
|
||||
output_dir=str(data.get("output_dir", ".")),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class InvokeResponse:
|
||||
"""节点调用响应：completed 表示成功，failed 表示执行失败。"""
|
||||
|
||||
# 执行状态，固定为 completed / failed。
|
||||
status: str
|
||||
# 输出产物映射，key 为输出名，value 为产物 URI。
|
||||
outputs: dict[str, Any] = field(default_factory=dict)
|
||||
# 失败原因，仅在 failed 时有意义。
|
||||
error: str | None = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为可 JSON 序列化的普通字典。"""
|
||||
return {
|
||||
"status": self.status,
|
||||
"outputs": self.outputs,
|
||||
"error": self.error,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "InvokeResponse":
|
||||
"""从 HTTP 响应 JSON 解析调用结果。"""
|
||||
return cls(
|
||||
# 缺省按失败处理，避免未知状态被误判为成功。
|
||||
status=str(data.get("status", "failed")),
|
||||
outputs=dict(data.get("outputs", {})),
|
||||
error=data.get("error"),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class HealthResponse:
|
||||
"""节点健康检查响应：节点进程就绪后返回 ok 与自身标识。"""
|
||||
|
||||
status: str
|
||||
node_id: str
|
||||
version: str
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为可 JSON 序列化的普通字典。"""
|
||||
return {
|
||||
"status": self.status,
|
||||
"node_id": self.node_id,
|
||||
"version": self.version,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProgressEvent:
|
||||
"""进度事件：预留用于节点向调度器上报执行进度。"""
|
||||
|
||||
run_id: str
|
||||
node_id: str
|
||||
progress: float
|
||||
message: str | None = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为可 JSON 序列化的普通字典。"""
|
||||
return {
|
||||
"run_id": self.run_id,
|
||||
"node_id": self.node_id,
|
||||
"progress": self.progress,
|
||||
"message": self.message,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkflowNode:
|
||||
"""工作流中的一个节点：声明节点类型、参数和输入引用。"""
|
||||
|
||||
# 节点在 DAG 内的唯一 ID，例如 extract、asr。
|
||||
id: str
|
||||
# 引用的节点能力，例如 ffmpeg-extract、faster-whisper。
|
||||
node_type: str
|
||||
# 传递给节点 invoke 的 params。
|
||||
params: dict[str, Any] = field(default_factory=dict)
|
||||
# 输入引用，value 形如 "前序节点ID.输出名" 或 "input.入口字段"。
|
||||
inputs: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为可 JSON 序列化的普通字典。"""
|
||||
return {
|
||||
"id": self.id,
|
||||
"node_type": self.node_type,
|
||||
"params": self.params,
|
||||
"inputs": self.inputs,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "WorkflowNode":
|
||||
"""从工作流定义 JSON 解析节点。"""
|
||||
return cls(
|
||||
id=str(data["id"]),
|
||||
node_type=str(data["node_type"]),
|
||||
params=dict(data.get("params", {})),
|
||||
inputs={str(k): str(v) for k, v in data.get("inputs", {}).items()},
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkflowEdge:
|
||||
"""工作流有向边：from_node 的输出流向 to_node 的输入。"""
|
||||
|
||||
from_node: str
|
||||
to_node: str
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为 JSON 时使用 from/to 短字段名。"""
|
||||
return {"from": self.from_node, "to": self.to_node}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "WorkflowEdge":
|
||||
"""从工作流定义 JSON 解析边。"""
|
||||
return cls(from_node=str(data["from"]), to_node=str(data["to"]))
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkflowDefinition:
|
||||
"""工作流 DAG 定义：包含节点列表、依赖边和输入输出映射。"""
|
||||
|
||||
name: str
|
||||
version: int
|
||||
nodes: list[WorkflowNode] = field(default_factory=list)
|
||||
edges: list[WorkflowEdge] = field(default_factory=list)
|
||||
# 用户上传入口与入口字段名的映射，例如 {"video_uri": "file"}。
|
||||
entry_inputs: dict[str, Any] = field(default_factory=dict)
|
||||
# 最终对外暴露的产物别名映射，例如 {"ass": "ass.ass_uri"}。
|
||||
final_outputs: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def validate(self) -> None:
|
||||
"""校验 DAG 基本约束：名称、版本、节点 ID 唯一、边引用有效。"""
|
||||
_require_non_empty(self.name, "name")
|
||||
if self.version < 1:
|
||||
raise ValueError("version must be >= 1")
|
||||
# 节点 ID 集合用于检查重复和边引用。
|
||||
node_ids = {node.id for node in self.nodes}
|
||||
if len(node_ids) != len(self.nodes):
|
||||
raise ValueError("workflow node ids must be unique")
|
||||
for edge in self.edges:
|
||||
if edge.from_node not in node_ids or edge.to_node not in node_ids:
|
||||
raise ValueError(f"edge references unknown node: {edge}")
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""转换为可 JSON 序列化的普通字典。"""
|
||||
return {
|
||||
"name": self.name,
|
||||
"version": self.version,
|
||||
"nodes": [node.to_dict() for node in self.nodes],
|
||||
"edges": [edge.to_dict() for edge in self.edges],
|
||||
"entry_inputs": self.entry_inputs,
|
||||
"final_outputs": self.final_outputs,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "WorkflowDefinition":
|
||||
"""从工作流定义 JSON 解析 DAG。"""
|
||||
return cls(
|
||||
name=str(data["name"]),
|
||||
version=int(data.get("version", 1)),
|
||||
nodes=[WorkflowNode.from_dict(item) for item in data.get("nodes", [])],
|
||||
edges=[WorkflowEdge.from_dict(item) for item in data.get("edges", [])],
|
||||
entry_inputs=dict(data.get("entry_inputs", {})),
|
||||
final_outputs=dict(data.get("final_outputs", {})),
|
||||
)
|
||||
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 1.5 KiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 127 KiB |
@@ -0,0 +1,43 @@
|
||||
"""pytest 全局配置。
|
||||
|
||||
在测试进程启动时创建独立临时目录，并通过环境变量把应用的数据目录、数据库、
|
||||
存储和后台服务全部指向测试环境，避免污染本地开发数据；同时隔离进程内节点
|
||||
注册表，防止测试之间互相泄漏注册条目。
|
||||
"""
|
||||
|
||||
import atexit
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
# 每个测试进程使用独立临时根目录，保证测试之间互不干扰。
|
||||
TEST_ROOT = Path(tempfile.mkdtemp(prefix="vrsub-test-"))
|
||||
os.environ["WOV_DATA_DIR"] = str(TEST_ROOT / "data")
|
||||
os.environ["WOV_DB_PATH"] = str(TEST_ROOT / "data" / "wov.db")
|
||||
os.environ["WOV_STORAGE_DIR"] = str(TEST_ROOT / "storage")
|
||||
# 默认关闭自动种子和后台调度，测试显式控制执行时机。
|
||||
os.environ["WOV_AUTO_SEED"] = "0"
|
||||
os.environ["WOV_SCHEDULER_ENABLED"] = "0"
|
||||
os.environ["WOV_CLEANUP_ENABLED"] = "0"
|
||||
|
||||
|
||||
def _cleanup() -> None:
|
||||
"""进程退出时清理临时测试目录。"""
|
||||
shutil.rmtree(TEST_ROOT, ignore_errors=True)
|
||||
|
||||
|
||||
atexit.register(_cleanup)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_registry():
|
||||
"""快照并恢复进程内节点注册表，避免测试之间互相污染。"""
|
||||
from wov_app import registry
|
||||
|
||||
snapshot = dict(registry._registry)
|
||||
yield
|
||||
registry._registry.clear()
|
||||
registry._registry.update(snapshot)
|
||||
@@ -0,0 +1,148 @@
|
||||
"""自适应线程池测试。
|
||||
|
||||
覆盖决策函数（增/减/保持/边界）、map 顺序返回、worker 异常隔离，
|
||||
以及"10s 窗口内平均响应 < 0.3s 加线程 / > 1.0s 减线程"的弹性行为
|
||||
（通过注入假时钟做确定性验证）。
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
from nodes.adaptive_pool import AdaptiveThreadPool, decide
|
||||
|
||||
|
||||
class FakeClock:
|
||||
"""可手动拨动的假时钟，用于确定性验证弹性窗口逻辑。"""
|
||||
|
||||
def __init__(self, now: float = 0.0) -> None:
|
||||
self.now = now
|
||||
|
||||
def __call__(self) -> float:
|
||||
return self.now
|
||||
|
||||
def advance(self, seconds: float) -> None:
|
||||
self.now += seconds
|
||||
|
||||
|
||||
def test_decide_increase_when_fast() -> None:
|
||||
"""平均响应低于 fast_threshold 且未达上限：线程数 +1。"""
|
||||
assert decide(1, 0.1, 1, 16, 0.3, 1.0) == 2
|
||||
|
||||
|
||||
def test_decide_decrease_when_slow() -> None:
|
||||
"""平均响应高于 slow_threshold 且高于下限：线程数 -1。"""
|
||||
assert decide(3, 2.0, 1, 16, 0.3, 1.0) == 2
|
||||
|
||||
|
||||
def test_decide_keep_when_mid() -> None:
|
||||
"""平均响应介于两阈值之间：保持不变。"""
|
||||
assert decide(2, 0.5, 1, 16, 0.3, 1.0) == 2
|
||||
|
||||
|
||||
def test_decide_bounds() -> None:
|
||||
"""已达上限不再增、已达下限不再减。"""
|
||||
assert decide(16, 0.1, 1, 16, 0.3, 1.0) == 16
|
||||
assert decide(1, 2.0, 1, 16, 0.3, 1.0) == 1
|
||||
|
||||
|
||||
def test_pool_map_ordered_results() -> None:
|
||||
"""map 按输入顺序返回结果，worker 简单映射。"""
|
||||
pool = AdaptiveThreadPool(worker=lambda item: item * 2)
|
||||
assert pool.map([1, 2, 3, 4]) == [2, 4, 6, 8]
|
||||
|
||||
|
||||
def test_pool_on_progress_callback() -> None:
|
||||
"""进度回调：每次完成触发一次，携带已完成数/总数/速度。"""
|
||||
progress: list[tuple[int, int, float]] = []
|
||||
pool = AdaptiveThreadPool(
|
||||
worker=lambda item: item,
|
||||
on_progress=lambda done, total, rate: progress.append((done, total, rate)),
|
||||
)
|
||||
pool.map([10, 20, 30])
|
||||
assert [item[0] for item in progress] == [1, 2, 3] # 已完成数递增。
|
||||
assert all(item[1] == 3 for item in progress) # 总数固定。
|
||||
assert all(item[2] > 0 for item in progress) # 速度为正值。
|
||||
|
||||
def test_pool_map_empty() -> None:
|
||||
"""空输入：不启动任务，直接返回空列表。"""
|
||||
pool = AdaptiveThreadPool(worker=lambda item: item)
|
||||
assert pool.map([]) == []
|
||||
|
||||
|
||||
def test_pool_worker_exception_isolated() -> None:
|
||||
"""worker 抛异常时以异常对象作为结果，不拖垮整体。"""
|
||||
def boom(item):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
pool = AdaptiveThreadPool(worker=boom)
|
||||
results = pool.map([1, 2])
|
||||
assert len(results) == 2
|
||||
assert all(isinstance(result, RuntimeError) for result in results)
|
||||
|
||||
|
||||
def test_pool_grows_when_fast() -> None:
|
||||
"""10s 窗口内平均响应 < 0.3s：线程数从 1 增至 2（弹性扩容）。"""
|
||||
clock = FakeClock()
|
||||
pool = AdaptiveThreadPool(
|
||||
worker=lambda item: item,
|
||||
min_workers=1, max_workers=16,
|
||||
window_seconds=10.0, fast_threshold=0.3,
|
||||
clock=clock,
|
||||
)
|
||||
# 拨快时钟越过窗口：首个任务完成即触发评估 → 平均响应≈0 < 0.3 → +1 线程。
|
||||
clock.advance(11)
|
||||
pool.map(list(range(4)))
|
||||
assert pool.max_concurrency == 2
|
||||
|
||||
|
||||
def test_pool_shrink_when_slow() -> None:
|
||||
"""窗口平均响应 > 1.0s：线程数从 2 减至 1（弹性退避）。"""
|
||||
clock = FakeClock()
|
||||
pool = AdaptiveThreadPool(
|
||||
worker=lambda item: item,
|
||||
min_workers=1, max_workers=16,
|
||||
window_seconds=10.0, fast_threshold=0.3, slow_threshold=1.0,
|
||||
clock=clock,
|
||||
)
|
||||
pool._resize(2) # 先扩到 2 个线程。
|
||||
clock.advance(11)
|
||||
pool._tick(2.0) # 窗口内平均 2.0 > 1.0 → 缩回 1。
|
||||
deadline = time.monotonic() + 2
|
||||
while len(pool._threads) > 1 and time.monotonic() < deadline:
|
||||
time.sleep(0.01)
|
||||
assert len(pool._threads) == 1
|
||||
pool._stop.set()
|
||||
|
||||
|
||||
def test_resize_shrink_idempotent() -> None:
|
||||
"""回归：重复缩容到同一目标不会重复放哨兵（曾因并发缩容毒死全部线程而死锁）。"""
|
||||
pool = AdaptiveThreadPool(worker=lambda item: item, min_workers=1, max_workers=8)
|
||||
pool._resize(3)
|
||||
assert pool._target_workers == 3
|
||||
pool._resize(2)
|
||||
pool._resize(2) # 目标已是 2：幂等，不再放哨兵。
|
||||
assert pool._target_workers == 2
|
||||
# 只有 1 个线程被哨兵退出，最终存活 2 个。
|
||||
deadline = time.monotonic() + 2
|
||||
while len(pool._threads) > 2 and time.monotonic() < deadline:
|
||||
time.sleep(0.01)
|
||||
assert len(pool._threads) == 2
|
||||
pool._stop.set()
|
||||
|
||||
|
||||
def test_pool_survives_mixed_grow_shrink() -> None:
|
||||
"""回归：扩容+缩容混合场景 map 必须完成且保序（修复前会死锁挂起）。"""
|
||||
clock = FakeClock()
|
||||
state = {"count": 0}
|
||||
|
||||
def worker(item):
|
||||
state["count"] += 1
|
||||
clock.advance(0.06 if state["count"] <= 20 else 0.6)
|
||||
return item
|
||||
|
||||
pool = AdaptiveThreadPool(
|
||||
worker=worker, min_workers=1, max_workers=4,
|
||||
window_seconds=0.5, fast_threshold=0.2, slow_threshold=0.4,
|
||||
clock=clock,
|
||||
)
|
||||
out = pool.map(list(range(60)))
|
||||
assert out == list(range(60))
|
||||
@@ -0,0 +1,34 @@
|
||||
"""应用级 API 冒烟测试。
|
||||
|
||||
使用 FastAPI TestClient 验证健康检查、静态页面与 OpenAPI 文档可访问。
|
||||
节点注册/实例管理 API 已随单体化移除，不再有对应路由。
|
||||
"""
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from wov_app.main import app
|
||||
|
||||
|
||||
def test_health_and_static() -> None:
|
||||
"""验证静态首页、OpenAPI 文档与健康探针均可访问。"""
|
||||
with TestClient(app) as client:
|
||||
root = client.get("/", follow_redirects=False)
|
||||
assert root.status_code == 200
|
||||
assert "VRSub 字幕生成" in root.text
|
||||
|
||||
docs = client.get("/docs")
|
||||
assert docs.status_code == 200
|
||||
|
||||
response = client.get("/health")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["service"] == "wov-api"
|
||||
assert response.json()["mode"] == "monolith"
|
||||
|
||||
|
||||
def test_node_admin_routes_removed() -> None:
|
||||
"""验证节点注册与实例管理路由在单体版中已移除（404/405）。"""
|
||||
with TestClient(app) as client:
|
||||
# GET 落到静态文件挂载后返回 404；POST 对静态挂载返回 405。
|
||||
assert client.post("/api/admin/nodes", json={}).status_code == 405
|
||||
assert client.get("/api/admin/nodes").status_code == 404
|
||||
assert client.get("/api/admin/node-instances").status_code == 404
|
||||
@@ -0,0 +1,313 @@
|
||||
"""用户应用 API 测试。
|
||||
|
||||
覆盖已发布应用的上传建任务、进度查询、产物下载、失败重试以及
|
||||
未发布/无版本工作流的拒绝逻辑。节点为内置注册，无需再手动注册。
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from wov_app.main import app
|
||||
|
||||
|
||||
def _create_published_echo_workflow(client) -> str:
|
||||
"""创建一个已发布的单节点 Echo 工作流（echo 为内置节点）。"""
|
||||
definition = {
|
||||
"name": "echo-flow",
|
||||
"version": 1,
|
||||
"nodes": [
|
||||
{
|
||||
"id": "step",
|
||||
"node_type": "echo",
|
||||
"inputs": {"file_uri": "input.video_uri"},
|
||||
}
|
||||
],
|
||||
"edges": [],
|
||||
"entry_inputs": {"video_uri": "file"},
|
||||
"final_outputs": {"result": "step.file_uri"},
|
||||
}
|
||||
client.post(
|
||||
"/api/admin/workflows",
|
||||
json={
|
||||
"id": "echo-app",
|
||||
"name": "Echo App",
|
||||
"description": "upload a file",
|
||||
"definition": definition,
|
||||
},
|
||||
)
|
||||
client.post("/api/admin/workflows/echo-app/publish")
|
||||
return "echo-app"
|
||||
|
||||
|
||||
def test_upload_run_progress_and_download() -> None:
|
||||
"""验证上传文件建任务、手动执行、查询产物与下载的完整流程。"""
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
apps = client.get("/api/apps")
|
||||
assert apps.status_code == 200
|
||||
assert any(item["id"] == workflow_id for item in apps.json())
|
||||
|
||||
uploaded = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"hello from upload", "text/plain")},
|
||||
)
|
||||
assert uploaded.status_code == 200
|
||||
run_id = uploaded.json()["id"]
|
||||
assert uploaded.json()["status"] == "QUEUED"
|
||||
|
||||
run = client.get(f"/api/runs/{run_id}")
|
||||
assert run.status_code == 200
|
||||
assert run.json()["input_uri"].endswith("sample.txt")
|
||||
assert run.json()["artifacts"] == []
|
||||
|
||||
scheduler = app.state.scheduler
|
||||
scheduler.execute_run(run_id)
|
||||
|
||||
completed = client.get(f"/api/runs/{run_id}")
|
||||
assert completed.status_code == 200
|
||||
assert completed.json()["status"] == "COMPLETED"
|
||||
artifact_names = [item["name"] for item in completed.json()["artifacts"]]
|
||||
assert "result" in artifact_names
|
||||
|
||||
artifacts = client.get(f"/api/runs/{run_id}/artifacts")
|
||||
assert artifacts.status_code == 200
|
||||
assert len(artifacts.json()) >= 1
|
||||
|
||||
downloaded = client.get(f"/api/runs/{run_id}/artifacts/result")
|
||||
assert downloaded.status_code == 200
|
||||
assert b"hello from upload" in downloaded.content
|
||||
|
||||
assert client.get(f"/api/runs/{run_id}/artifacts/missing").status_code == 404
|
||||
assert client.get("/api/runs/missing").status_code == 404
|
||||
assert client.get("/api/runs/missing/artifacts").status_code == 404
|
||||
|
||||
db = app.state.db
|
||||
db.create_artifact(
|
||||
{
|
||||
"run_id": run_id,
|
||||
"node_id": "step",
|
||||
"name": "missing-file",
|
||||
"uri": str(Path(__file__).resolve().parent / "not-exists.bin"),
|
||||
"mime_type": "text/plain",
|
||||
"size": 0,
|
||||
}
|
||||
)
|
||||
assert client.get(f"/api/runs/{run_id}/artifacts/missing-file").status_code == 404
|
||||
|
||||
runs = client.get("/api/runs")
|
||||
assert runs.status_code == 200
|
||||
assert any(item["id"] == run_id for item in runs.json())
|
||||
|
||||
|
||||
def test_upload_rejects_unpublished_workflow() -> None:
|
||||
"""验证草稿或不存在的工作流不能被用户发起任务。"""
|
||||
with TestClient(app) as client:
|
||||
client.post(
|
||||
"/api/admin/workflows",
|
||||
json={
|
||||
"id": "draft",
|
||||
"name": "Draft",
|
||||
"definition": {
|
||||
"name": "Draft",
|
||||
"version": 1,
|
||||
"nodes": [],
|
||||
"edges": [],
|
||||
},
|
||||
},
|
||||
)
|
||||
response = client.post(
|
||||
"/api/apps/draft/runs",
|
||||
files={"file": ("x.txt", b"x", "text/plain")},
|
||||
)
|
||||
assert response.status_code == 404
|
||||
|
||||
response = client.post(
|
||||
"/api/apps/missing/runs",
|
||||
files={"file": ("x.txt", b"x", "text/plain")},
|
||||
)
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
def test_upload_rejects_workflow_without_version() -> None:
|
||||
"""验证已发布但没有任何版本的工作流返回 422。"""
|
||||
with TestClient(app) as client:
|
||||
db = app.state.db
|
||||
db.upsert_workflow(
|
||||
{"id": "empty", "name": "Empty", "published": 1, "latest_version": 0}
|
||||
)
|
||||
response = client.post(
|
||||
"/api/apps/empty/runs",
|
||||
files={"file": ("x.txt", b"x", "text/plain")},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
def test_retry_failed_run_requeues_and_reruns() -> None:
|
||||
"""验证失败任务重试会清空旧产物并重新执行成功。"""
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
uploaded = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"hello retry", "text/plain")},
|
||||
)
|
||||
run_id = uploaded.json()["id"]
|
||||
db = app.state.db
|
||||
db.update_run(
|
||||
run_id,
|
||||
status="FAILED",
|
||||
error="boom",
|
||||
updated_at="2026-01-01T00:00:00+00:00",
|
||||
)
|
||||
db.create_artifact(
|
||||
{
|
||||
"run_id": run_id,
|
||||
"node_id": "step",
|
||||
"name": "stale",
|
||||
"uri": "stale.txt",
|
||||
"mime_type": "text/plain",
|
||||
"size": 1,
|
||||
}
|
||||
)
|
||||
|
||||
response = client.post(f"/api/runs/{run_id}/retry")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"id": run_id, "status": "QUEUED"}
|
||||
run = client.get(f"/api/runs/{run_id}").json()
|
||||
assert run["status"] == "QUEUED"
|
||||
assert run["error"] is None
|
||||
assert run["artifacts"] == []
|
||||
|
||||
app.state.scheduler.execute_run(run_id)
|
||||
completed = client.get(f"/api/runs/{run_id}").json()
|
||||
assert completed["status"] == "COMPLETED"
|
||||
assert any(item["name"] == "result" for item in completed["artifacts"])
|
||||
|
||||
|
||||
def test_pause_resume_run_api() -> None:
|
||||
"""验证暂停/继续接口：QUEUED→PAUSED→QUEUED，状态非法时报 422。"""
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
uploaded = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"hello pause", "text/plain")},
|
||||
)
|
||||
run_id = uploaded.json()["id"]
|
||||
assert uploaded.json()["status"] == "QUEUED"
|
||||
|
||||
paused = client.post(f"/api/runs/{run_id}/pause")
|
||||
assert paused.status_code == 200
|
||||
assert paused.json() == {"id": run_id, "status": "PAUSED"}
|
||||
assert client.get(f"/api/runs/{run_id}").json()["status"] == "PAUSED"
|
||||
|
||||
resumed = client.post(f"/api/runs/{run_id}/resume")
|
||||
assert resumed.status_code == 200
|
||||
assert resumed.json() == {"id": run_id, "status": "QUEUED"}
|
||||
assert client.get(f"/api/runs/{run_id}").json()["status"] == "QUEUED"
|
||||
|
||||
# 非 PAUSED 任务不可继续。
|
||||
assert client.post(f"/api/runs/{run_id}/resume").status_code == 422
|
||||
# 不存在的任务 404。
|
||||
assert client.post("/api/runs/missing/pause").status_code == 404
|
||||
assert client.post("/api/runs/missing/resume").status_code == 404
|
||||
|
||||
|
||||
def test_pause_rejects_terminal_states() -> None:
|
||||
"""验证已完成任务不可暂停。"""
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
uploaded = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"hello done", "text/plain")},
|
||||
)
|
||||
run_id = uploaded.json()["id"]
|
||||
db = app.state.db
|
||||
db.update_run(run_id, status="COMPLETED", progress=1.0, updated_at="2026-01-01T00:00:00+00:00")
|
||||
assert client.post(f"/api/runs/{run_id}/pause").status_code == 422
|
||||
|
||||
|
||||
def test_retry_rejects_non_failed_and_missing_runs() -> None:
|
||||
"""验证只有 FAILED 状态且存在的任务才能重试。"""
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
uploaded = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"x", "text/plain")},
|
||||
)
|
||||
run_id = uploaded.json()["id"]
|
||||
|
||||
assert client.post(f"/api/runs/{run_id}/retry").status_code == 422
|
||||
assert client.post("/api/runs/missing/retry").status_code == 404
|
||||
|
||||
|
||||
def test_delete_run_removes_record_and_files() -> None:
|
||||
"""验证删除任务会清理数据库记录与磁盘上的上传/步骤文件。"""
|
||||
import shutil
|
||||
|
||||
from wov_app.config import STORAGE_DIR
|
||||
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
uploaded = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"hello delete", "text/plain")},
|
||||
)
|
||||
run_id = uploaded.json()["id"]
|
||||
|
||||
# 执行任务以生成步骤产物目录。
|
||||
app.state.scheduler.execute_run(run_id)
|
||||
run = client.get(f"/api/runs/{run_id}").json()
|
||||
steps_dir = STORAGE_DIR / "runs" / run_id
|
||||
assert steps_dir.is_dir()
|
||||
# 上传文件目录也应存在。
|
||||
upload_dir = Path(run["input_uri"]).parent
|
||||
assert upload_dir.is_dir()
|
||||
|
||||
deleted = client.delete(f"/api/runs/{run_id}")
|
||||
assert deleted.status_code == 200
|
||||
assert deleted.json() == {"deleted": run_id}
|
||||
|
||||
assert client.get(f"/api/runs/{run_id}").status_code == 404
|
||||
assert not steps_dir.exists()
|
||||
assert not upload_dir.exists()
|
||||
|
||||
# 删除不存在的任务返回 404。
|
||||
assert client.delete(f"/api/runs/missing").status_code == 404
|
||||
|
||||
# 清理测试遗留的 runs 目录，避免跨用例残留。
|
||||
shutil.rmtree(STORAGE_DIR / "runs", ignore_errors=True)
|
||||
|
||||
|
||||
def test_create_run_with_param_overrides() -> None:
|
||||
"""验证创建任务时可携带 params 覆盖（如前端框选的 crop），并持久化。"""
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
uploaded = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"x", "text/plain")},
|
||||
data={"params": '{"step": {"crop": [0, 0.82, 1, 0.18]}}'},
|
||||
)
|
||||
assert uploaded.status_code == 200
|
||||
run_id = uploaded.json()["id"]
|
||||
run = client.get(f"/api/runs/{run_id}").json()
|
||||
assert run["param_overrides"] == {"step": {"crop": [0, 0.82, 1, 0.18]}}
|
||||
# 非法 JSON 返回 422。
|
||||
bad = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"x", "text/plain")},
|
||||
data={"params": "not-json"},
|
||||
)
|
||||
assert bad.status_code == 422
|
||||
|
||||
|
||||
def test_create_run_params_non_object_rejected() -> None:
|
||||
"""params 为 JSON 数组时返回 422。"""
|
||||
with TestClient(app) as client:
|
||||
workflow_id = _create_published_echo_workflow(client)
|
||||
response = client.post(
|
||||
f"/api/apps/{workflow_id}/runs",
|
||||
files={"file": ("sample.txt", b"x", "text/plain")},
|
||||
data={"params": "[1,2,3]"},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
@@ -0,0 +1,32 @@
|
||||
// crop 归一化纯函数单测：由 pytest 通过 node 执行（TDD 红阶段先失败）。
|
||||
"use strict";
|
||||
const assert = require("assert");
|
||||
const { videoDisplayRect, rectToCrop, cropToRect } = require("../web/assets/crop.js");
|
||||
|
||||
// 1) 无留边（容器比例与视频一致）：底部 20% 矩形 → crop [0, 0.8, 1, 0.2]
|
||||
let d = videoDisplayRect(1280, 720, 1280, 720);
|
||||
assert.deepStrictEqual(d, { x: 0, y: 0, w: 1280, h: 720 });
|
||||
assert.deepStrictEqual(
|
||||
rectToCrop({ x: 0, y: 576, w: 1280, h: 144 }, 1280, 720, 1280, 720),
|
||||
[0, 0.8, 1, 0.2]
|
||||
);
|
||||
|
||||
// 2) letterbox（容器比视频宽）：视频显示在中间，矩形映射要考虑左右留边
|
||||
d = videoDisplayRect(1280, 720, 1600, 720);
|
||||
assert.deepStrictEqual(d, { x: 160, y: 0, w: 1280, h: 720 });
|
||||
// 在渲染视频内框选右下 25% 区域
|
||||
let crop = rectToCrop({ x: 160 + 640, y: 360, w: 640, h: 360 }, 1280, 720, 1600, 720);
|
||||
assert.deepStrictEqual(crop, [0.5, 0.5, 0.5, 0.5]);
|
||||
|
||||
// 3) 回显一致性：crop → rect → crop 应还原（含 letterbox）
|
||||
let back = cropToRect(crop, 1280, 720, 1600, 720);
|
||||
assert.deepStrictEqual(
|
||||
rectToCrop(back, 1280, 720, 1600, 720),
|
||||
crop
|
||||
);
|
||||
|
||||
// 4) 越界钳制：矩形超出画面时 crop 值被限制在 0~1
|
||||
crop = rectToCrop({ x: -100, y: -50, w: 2000, h: 900 }, 1280, 720, 1280, 720);
|
||||
assert.ok(crop.every((v) => v >= 0 && v <= 1));
|
||||
|
||||
console.log("crop.js 全部断言通过");
|
||||
@@ -0,0 +1,294 @@
|
||||
"""数据库层单元测试。
|
||||
|
||||
直接对 Database 方法调用真实 SQLite 路径，覆盖工作流、版本、任务与产物的
|
||||
增删改查。节点注册表已改为进程内内存态，不再落库。
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from wov_app.db import Database
|
||||
|
||||
|
||||
def test_workflow_crud(tmp_path) -> None:
|
||||
"""验证工作流概要的插入、发布标记更新与删除。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
workflow = {
|
||||
"id": "demo",
|
||||
"name": "Demo",
|
||||
"description": "desc",
|
||||
"published": 0,
|
||||
"latest_version": 0,
|
||||
}
|
||||
db.upsert_workflow(workflow)
|
||||
assert db.get_workflow("demo")["name"] == "Demo"
|
||||
assert [item["id"] for item in db.list_workflows()] == ["demo"]
|
||||
|
||||
db.upsert_workflow({**workflow, "published": 1, "latest_version": 1})
|
||||
assert db.get_workflow("demo")["published"] == 1
|
||||
|
||||
db.delete_workflow("demo")
|
||||
assert db.get_workflow("demo") is None
|
||||
|
||||
|
||||
def test_workflow_versions(tmp_path) -> None:
|
||||
"""验证工作流版本的写入、最新版本查询与列表。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow(
|
||||
{"id": "demo", "name": "Demo", "published": 1, "latest_version": 2}
|
||||
)
|
||||
definition = {"name": "Demo", "version": 1, "nodes": [], "edges": []}
|
||||
db.create_workflow_version("demo", 1, definition)
|
||||
db.create_workflow_version("demo", 2, {**definition, "version": 2})
|
||||
|
||||
latest = db.get_latest_workflow_version("demo")
|
||||
assert latest["version"] == 2
|
||||
assert latest["definition"]["version"] == 2
|
||||
|
||||
version = db.get_workflow_version("demo", 1)
|
||||
assert version["version"] == 1
|
||||
assert db.get_workflow_version("demo", 99) is None
|
||||
assert len(db.list_workflow_versions("demo")) == 2
|
||||
|
||||
empty_db = Database(tmp_path / "empty.db")
|
||||
assert empty_db.get_latest_workflow_version("missing") is None
|
||||
|
||||
|
||||
def test_run_and_artifact_crud(tmp_path) -> None:
|
||||
"""验证任务与产物的创建、查询、更新与删除。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow({"id": "demo", "name": "Demo", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_1",
|
||||
"workflow_id": "demo",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": "in.txt",
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
assert db.get_run("run_1")["status"] == "QUEUED"
|
||||
assert db.next_queued_run()["id"] == "run_1"
|
||||
|
||||
db.update_run("run_1", status="RUNNING", progress=0.5, updated_at=now)
|
||||
db.update_run("run_1")
|
||||
assert db.get_run("run_1")["status"] == "RUNNING"
|
||||
assert db.get_run("run_1")["progress"] == 0.5
|
||||
assert db.next_queued_run() is None
|
||||
assert len(db.list_runs()) == 1
|
||||
|
||||
db.create_artifact(
|
||||
{
|
||||
"run_id": "run_1",
|
||||
"node_id": "echo",
|
||||
"name": "result",
|
||||
"uri": "out.txt",
|
||||
"mime_type": "text/plain",
|
||||
"size": 3,
|
||||
}
|
||||
)
|
||||
assert db.get_artifact("run_1", "result")["uri"] == "out.txt"
|
||||
assert db.get_artifact("run_1", "missing") is None
|
||||
assert len(db.list_artifacts("run_1")) == 1
|
||||
|
||||
db.delete_run_artifacts("run_1")
|
||||
assert db.list_artifacts("run_1") == []
|
||||
|
||||
|
||||
def test_reset_run_clears_error_and_artifacts(tmp_path) -> None:
|
||||
"""验证 reset_run 会把失败任务恢复到排队状态并清空旧产物。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow({"id": "demo", "name": "Demo", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_1",
|
||||
"workflow_id": "demo",
|
||||
"workflow_version": 1,
|
||||
"status": "FAILED",
|
||||
"progress": 0.75,
|
||||
"current_node_id": "translate",
|
||||
"error": "timed out",
|
||||
"input_uri": "in.txt",
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
db.create_artifact(
|
||||
{
|
||||
"run_id": "run_1",
|
||||
"node_id": "asr",
|
||||
"name": "asr.srt_uri",
|
||||
"uri": "out.srt",
|
||||
"mime_type": "application/x-subrip",
|
||||
"size": 3,
|
||||
}
|
||||
)
|
||||
|
||||
db.reset_run("run_1", "2026-01-02T00:00:00+00:00")
|
||||
|
||||
run = db.get_run("run_1")
|
||||
assert run["status"] == "QUEUED"
|
||||
assert run["progress"] == 0
|
||||
assert run["current_node_id"] is None
|
||||
assert run["error"] is None
|
||||
assert run["updated_at"] == "2026-01-02T00:00:00+00:00"
|
||||
assert run["created_at"] == now
|
||||
assert db.list_artifacts("run_1") == []
|
||||
|
||||
|
||||
def test_delete_run(tmp_path) -> None:
|
||||
"""验证 delete_run 会删除任务记录及其产物记录。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow({"id": "demo", "name": "Demo", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_1",
|
||||
"workflow_id": "demo",
|
||||
"workflow_version": 1,
|
||||
"status": "COMPLETED",
|
||||
"progress": 1,
|
||||
"input_uri": "in.txt",
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
db.create_artifact(
|
||||
{
|
||||
"run_id": "run_1",
|
||||
"node_id": "asr",
|
||||
"name": "asr.srt_uri",
|
||||
"uri": "out.srt",
|
||||
"mime_type": "application/x-subrip",
|
||||
"size": 3,
|
||||
}
|
||||
)
|
||||
db.delete_run("run_1")
|
||||
assert db.get_run("run_1") is None
|
||||
assert db.list_artifacts("run_1") == []
|
||||
|
||||
|
||||
def test_list_run_ids(tmp_path) -> None:
|
||||
"""验证 list_run_ids 返回全部任务 ID，供孤儿清理对照使用。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow({"id": "demo", "name": "Demo", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
assert db.list_run_ids() == []
|
||||
for run_id in ("run_a", "run_b"):
|
||||
db.create_run(
|
||||
{
|
||||
"id": run_id,
|
||||
"workflow_id": "demo",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
assert set(db.list_run_ids()) == {"run_a", "run_b"}
|
||||
|
||||
|
||||
def test_run_param_overrides_persist(tmp_path) -> None:
|
||||
"""验证 param_overrides 随任务持久化并可读回。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow({"id": "demo", "name": "D", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_ov",
|
||||
"workflow_id": "demo",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"param_overrides": {"extract": {"crop": [0, 0.5, 1, 0.5]}},
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
run = db.get_run("run_ov")
|
||||
assert run["param_overrides"] == {"extract": {"crop": [0, 0.5, 1, 0.5]}}
|
||||
assert db.next_queued_run()["param_overrides"] == {"extract": {"crop": [0, 0.5, 1, 0.5]}}
|
||||
|
||||
|
||||
def test_db_migration_adds_param_overrides(tmp_path) -> None:
|
||||
"""旧库迁移：缺少 param_overrides 列的库打开后自动补列。"""
|
||||
import sqlite3
|
||||
|
||||
db_path = tmp_path / "old.db"
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.execute(
|
||||
"CREATE TABLE workflow_runs (id TEXT PRIMARY KEY, workflow_id TEXT NOT NULL,"
|
||||
" workflow_version INTEGER NOT NULL, status TEXT NOT NULL, current_node_id TEXT,"
|
||||
" progress REAL NOT NULL DEFAULT 0, error TEXT, input_uri TEXT,"
|
||||
" created_at TEXT NOT NULL, updated_at TEXT NOT NULL)"
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
Database(db_path)
|
||||
conn = sqlite3.connect(db_path)
|
||||
columns = [row[1] for row in conn.execute("PRAGMA table_info(workflow_runs)")]
|
||||
conn.close()
|
||||
assert "param_overrides" in columns
|
||||
|
||||
|
||||
def test_pause_resume_run(tmp_path) -> None:
|
||||
"""验证 pause_run/resume_run 的状态流转与 PAUSED 任务可被调度器取到。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow({"id": "demo", "name": "Demo", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_p",
|
||||
"workflow_id": "demo",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
db.pause_run("run_p", now)
|
||||
assert db.get_run("run_p")["status"] == "PAUSED"
|
||||
# PAUSED 任务会被 next_queued_run 取到（等待续跑）。
|
||||
assert db.next_queued_run()["id"] == "run_p"
|
||||
db.resume_run("run_p", now)
|
||||
assert db.get_run("run_p")["status"] == "QUEUED"
|
||||
assert db.next_queued_run()["id"] == "run_p"
|
||||
|
||||
|
||||
def test_restore_run_outputs(tmp_path) -> None:
|
||||
"""验证从产物重建节点输出（断点续跑的依据）。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
db.upsert_workflow({"id": "demo", "name": "Demo", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_r",
|
||||
"workflow_id": "demo",
|
||||
"workflow_version": 1,
|
||||
"status": "PAUSED",
|
||||
"progress": 0,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
db.create_artifact(
|
||||
{
|
||||
"run_id": "run_r",
|
||||
"node_id": "extract",
|
||||
"name": "frames_manifest",
|
||||
"uri": "frames.json",
|
||||
"mime_type": "application/json",
|
||||
"size": 1,
|
||||
}
|
||||
)
|
||||
assert db.restore_run_outputs("run_r") == {
|
||||
"extract": {"frames_manifest": "frames.json"}
|
||||
}
|
||||
assert db.restore_run_outputs("run_none") == {}
|
||||
@@ -0,0 +1,28 @@
|
||||
"""前端 crop 归一化纯函数测试。
|
||||
|
||||
通过 node 执行 tests/test_crop_js.js（真实 JS 断言），验证框选矩形与
|
||||
crop 比例的互转（含 letterbox 与越界钳制）。
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
# 单体根目录：tests/ 的上一级。
|
||||
WORKSPACE = Path(__file__).resolve().parent.parent
|
||||
JS_TEST = WORKSPACE / "tests" / "test_crop_js.js"
|
||||
|
||||
|
||||
def test_crop_js_normalization() -> None:
|
||||
"""node 执行 crop 纯函数断言（TDD 红阶段先失败）。"""
|
||||
if shutil.which("node") is None:
|
||||
pytest.skip("环境无 node，跳过前端 crop 单测")
|
||||
result = subprocess.run(
|
||||
["node", str(JS_TEST)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(WORKSPACE),
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
@@ -0,0 +1,90 @@
|
||||
"""字幕 OCR 整链真实集成测试。
|
||||
|
||||
使用 testdata/subtitle_10s.mp4（烧录 SUB 001@1-4s、SUB 002@6-9s）与真实
|
||||
glm-ocr 模型：抽帧（frame-extract）→ 逐帧 OCR（subtitle-ocr）→ 汇总 SRT，
|
||||
断言烧录文字与时间轴对齐。Ollama 服务或资产缺失时自动跳过。
|
||||
"""
|
||||
|
||||
import json
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from nodes.frame_extract import invoke as frame_invoke
|
||||
from nodes.subtitle_ocr import invoke as ocr_invoke
|
||||
from wov_sdk.models import InvokeRequest
|
||||
|
||||
OLLAMA_HOST = "http://192.168.123.70:11434"
|
||||
MODEL = "glm-ocr:latest"
|
||||
|
||||
# 单体根目录：tests/ 的上一级。
|
||||
WORKSPACE = Path(__file__).resolve().parent.parent
|
||||
VIDEO = WORKSPACE / "testdata" / "subtitle_10s.mp4"
|
||||
|
||||
|
||||
def _register_nodes() -> None:
|
||||
"""注册全部内置节点，供 subtitle-ocr 内部调 vlm-ocr 使用。"""
|
||||
from wov_app import registry
|
||||
|
||||
registry.register_all()
|
||||
|
||||
|
||||
def _ollama_reachable() -> bool:
|
||||
"""探测 Ollama 服务与目标模型是否可用。"""
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
f"{OLLAMA_HOST}/api/show",
|
||||
data=b'{"model": "%s"}' % MODEL.encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=5) as resp:
|
||||
return resp.status == 200
|
||||
except (urllib.error.URLError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_subtitle_ocr_full_chain(tmp_path) -> None:
|
||||
"""抽帧→OCR→汇总：SRT 应含 SUB 001/SUB 002 且时间轴落在各自区间。"""
|
||||
if not _ollama_reachable():
|
||||
pytest.skip("Ollama 服务或 glm-ocr 模型不可用，跳过真实模型集成测试")
|
||||
if not VIDEO.is_file():
|
||||
pytest.skip("缺少 testdata/subtitle_10s.mp4 测试资产，跳过集成测试")
|
||||
_register_nodes()
|
||||
|
||||
# 抽帧：1s 间隔，字幕在底部，裁切下半 30% 区域（y=0.7,h=0.3）。
|
||||
frames_resp = frame_invoke(
|
||||
InvokeRequest(
|
||||
run_id="chain_fx",
|
||||
node_instance_id="",
|
||||
inputs={"video_uri": str(VIDEO)},
|
||||
params={"interval_seconds": 1, "crop": [0, 0.7, 1, 0.3]},
|
||||
output_dir=str(tmp_path / "frames"),
|
||||
)
|
||||
)
|
||||
assert frames_resp.status == "completed", frames_resp.error
|
||||
manifest = json.loads(Path(frames_resp.outputs["frames_manifest"]).read_text(encoding="utf-8"))
|
||||
assert len(manifest) >= 8
|
||||
|
||||
# OCR 汇总：真实 glm-ocr 逐帧识别。
|
||||
ocr_resp = ocr_invoke(
|
||||
InvokeRequest(
|
||||
run_id="chain_ocr",
|
||||
node_instance_id="",
|
||||
inputs={"frames_manifest": str(frames_resp.outputs["frames_manifest"])},
|
||||
params={"model": MODEL, "ollama_host": OLLAMA_HOST, "min_chars": 2},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert ocr_resp.status == "completed", ocr_resp.error
|
||||
srt = Path(ocr_resp.outputs["srt_uri"]).read_text(encoding="utf-8")
|
||||
# 两条烧录字幕都应被识别（文字可能带噪声，但至少含关键片段）。
|
||||
assert "SUB" in srt
|
||||
# 时间轴：SUB 001 应在 1-4s，SUB 002 应在 6-9s（允许模型/抽帧容差）。
|
||||
first_line = next(line for line in srt.splitlines() if "-->" in line)
|
||||
start = first_line.split(" --> ")[0].replace(",", ".")
|
||||
hours, minutes, seconds = start.split(":")
|
||||
total = int(hours) * 3600 + int(minutes) * 60 + float(seconds)
|
||||
assert total < 5
|
||||
@@ -0,0 +1,65 @@
|
||||
"""VLM OCR 节点真实集成测试。
|
||||
|
||||
复用 testdata/test_real_hav_sub.png（真实视频字幕截图，一次性入库，避免
|
||||
每次测试生成）。调用本地 Ollama 服务（192.168.123.70:11434）的真实
|
||||
glm-ocr 模型做 OCR。Ollama 服务或测试资产缺失时自动跳过；可用时必须执行。
|
||||
"""
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from nodes.vlm import invoke
|
||||
from wov_sdk.models import InvokeRequest
|
||||
|
||||
OLLAMA_HOST = "http://192.168.123.70:11434"
|
||||
MODEL = "glm-ocr:latest"
|
||||
# 测试图片（真实视频字幕帧）上应识别出的字幕文本。
|
||||
EXPECTED_TEXT = "还有没有什么困扰 或者奇怪的地方吗"
|
||||
|
||||
# 单体根目录：tests/ 的上一级。
|
||||
WORKSPACE = Path(__file__).resolve().parent.parent
|
||||
TEST_IMAGE = WORKSPACE / "testdata" / "test_real_hav_sub.png"
|
||||
|
||||
|
||||
def _ollama_reachable() -> bool:
|
||||
"""探测 Ollama 服务与目标模型是否可用。"""
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
f"{OLLAMA_HOST}/api/show",
|
||||
data=b'{"model": "%s"}' % MODEL.encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=5) as resp:
|
||||
return resp.status == 200
|
||||
except (urllib.error.URLError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_vlm_ocr_real_model(tmp_path) -> None:
|
||||
"""复用真实字幕截图 + 真实 glm-ocr：应识别出关键字幕文本并清洗围栏垃圾。"""
|
||||
if not _ollama_reachable():
|
||||
pytest.skip("Ollama 服务或 glm-ocr 模型不可用，跳过真实模型集成测试")
|
||||
if not TEST_IMAGE.is_file():
|
||||
pytest.skip("缺少 testdata/test_real_hav_sub.png 测试资产，跳过集成测试")
|
||||
|
||||
response = invoke(
|
||||
InvokeRequest(
|
||||
run_id="vlm_integration",
|
||||
node_instance_id="",
|
||||
inputs={"image_uri": str(TEST_IMAGE)},
|
||||
params={"model": MODEL, "ollama_host": OLLAMA_HOST},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
text = response.outputs["text"]
|
||||
print(text)
|
||||
# 关键字幕文本应被识别出来。注意 glm-ocr 在此图上存在已知重复循环 bug：
|
||||
# 识别出正确文本后可能继续循环输出，因此用"包含"断言而非全等，
|
||||
# 下游 subtitle-ocr 的 max_result_chars 守卫会拦截超长输出。
|
||||
assert EXPECTED_TEXT in text
|
||||
# 围栏垃圾（```）不应出现在输出里。
|
||||
assert "```" not in text
|
||||
@@ -0,0 +1,52 @@
|
||||
"""真实模型集成测试。
|
||||
|
||||
复用 testdata/speech_60s.wav（真实语音 WAV，一次性生成、入库，避免每次
|
||||
测试从视频提取）。使用真实 faster-whisper 模型端到端验证 whisper 节点的
|
||||
分块转写与 SRT 生成。本地缺少模型或测试资产时自动跳过；具备条件时必须
|
||||
执行，作为对假模型单元测试的校准。
|
||||
|
||||
约定（见 AGENTS.md「测试与覆盖率」）：单元测试允许在模型推理这一 I/O
|
||||
边界使用返回真实结构的薄桩，但必须配套本集成测试验证真实行为。
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from nodes.whisper import invoke
|
||||
from wov_sdk.models import InvokeRequest
|
||||
|
||||
# 单体根目录：tests/ 的上一级。
|
||||
WORKSPACE = Path(__file__).resolve().parent.parent
|
||||
MODEL_DIR = WORKSPACE / "model" / "faster-whisper-large-v3"
|
||||
TEST_AUDIO = WORKSPACE / "testdata" / "speech_60s.wav"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_whisper_real_model_chunked_transcription(tmp_path) -> None:
|
||||
"""复用 testdata 语音 + 真实模型：分块转写产出真实 SRT，时间不越出素材范围。"""
|
||||
if not (MODEL_DIR / "model.bin").is_file():
|
||||
pytest.skip("本地无 faster-whisper-large-v3 模型，跳过真实模型集成测试")
|
||||
if not TEST_AUDIO.is_file():
|
||||
pytest.skip("缺少 testdata/speech_60s.wav 测试资产，跳过真实模型集成测试")
|
||||
|
||||
response = invoke(
|
||||
InvokeRequest(
|
||||
run_id="integration_1",
|
||||
node_instance_id="",
|
||||
inputs={"audio_uri": str(TEST_AUDIO)},
|
||||
params={"language": "ja", "chunk_seconds": 60, "vad_filter": False},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
srt_path = Path(response.outputs["srt_uri"])
|
||||
assert srt_path.is_file()
|
||||
srt = srt_path.read_text(encoding="utf-8")
|
||||
time_lines = [line for line in srt.splitlines() if "-->" in line]
|
||||
# 60s 语音若含可识别内容，则应有字幕，且时间轴不越出素材时长（允许少量超窗）。
|
||||
if time_lines:
|
||||
last_end = time_lines[-1].split(" --> ")[1].replace(",", ".")
|
||||
hours, minutes, seconds = last_end.split(":")
|
||||
total = int(hours) * 3600 + int(minutes) * 60 + float(seconds)
|
||||
assert total < 90
|
||||
@@ -0,0 +1,257 @@
|
||||
"""LLM 字幕过滤节点测试。
|
||||
|
||||
覆盖 SRT 解析/序列化、±N 上下文窗口组装（纯文本无时间戳、目标标记）、
|
||||
LLM 调用（按 I/O 边界 mock urlopen）与删除判定、invoke 全链路与异常路径。
|
||||
"""
|
||||
|
||||
import json
|
||||
import urllib.error
|
||||
from pathlib import Path
|
||||
|
||||
from nodes.llm_filter import invoke, parse_srt, serialize_srt
|
||||
from wov_sdk.models import InvokeRequest
|
||||
|
||||
# 4 条字幕的 SRT：第 3 条为"答："开头的无意义杂项，模拟 OCR 噪声。
|
||||
_SRT = (
|
||||
"1\n00:00:01,000 --> 00:00:04,000\n还有没有什么困扰\n\n"
|
||||
"2\n00:00:05,000 --> 00:00:08,000\n或者奇怪的地方吗\n\n"
|
||||
"3\n00:00:09,000 --> 00:00:12,000\n答：无意义杂项\n\n"
|
||||
"4\n00:00:13,000 --> 00:00:16,000\n第二句正常字幕\n"
|
||||
)
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
"""模拟 urllib 响应：read() 返回 LLM 兼容接口的 JSON 载荷。"""
|
||||
|
||||
def __init__(self, payload: bytes) -> None:
|
||||
self._payload = payload
|
||||
|
||||
def read(self) -> bytes:
|
||||
return self._payload
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
"""模拟 LLM 兼容接口：记录请求体，按策略返回"保留/删除"。
|
||||
|
||||
支持两种策略：contents（按队列顺序，用于单次直调 _judge_target 的
|
||||
确定性测试）或 decision_fn（按请求体内容决策，用于并发 invoke 测试，
|
||||
保证任何线程执行顺序下判定结果都确定）。
|
||||
"""
|
||||
|
||||
def __init__(self, contents: list[str] | None = None, decision_fn=None) -> None:
|
||||
self._contents = list(contents) if contents is not None else None
|
||||
self._decision_fn = decision_fn
|
||||
self.bodies: list[dict] = []
|
||||
self.headers: list[dict] = []
|
||||
|
||||
def __call__(self, request, timeout=None):
|
||||
body = json.loads(request.data.decode("utf-8"))
|
||||
self.bodies.append(body)
|
||||
self.headers.append(dict(request.headers))
|
||||
if self._decision_fn is not None:
|
||||
content = self._decision_fn(body)
|
||||
else:
|
||||
content = self._contents.pop(0)
|
||||
payload = json.dumps({"choices": [{"message": {"content": content}}]}).encode()
|
||||
return FakeResponse(payload)
|
||||
|
||||
|
||||
def _patch_llm(monkeypatch, contents: list[str] | None = None, decision_fn=None) -> FakeLLM:
|
||||
"""替换 nodes.llm_filter 的 urlopen 为 FakeLLM 并返回实例。"""
|
||||
fake = FakeLLM(contents=contents, decision_fn=decision_fn)
|
||||
monkeypatch.setattr("nodes.llm_filter.urllib.request.urlopen", fake)
|
||||
return fake
|
||||
|
||||
|
||||
def _decision_by_target(body) -> str:
|
||||
"""按目标字幕内容决策：含"答："判为删除，其余保留（与 _SRT 的噪声对应）。"""
|
||||
target = next(
|
||||
line for line in body["messages"][1]["content"].splitlines()
|
||||
if line.startswith("【目标】")
|
||||
)
|
||||
return "删除" if "答：" in target else "保留"
|
||||
|
||||
|
||||
def test_parse_srt_multiline_and_last_block() -> None:
|
||||
"""解析 SRT：多行文本与末条无空行结尾均能正确解析。"""
|
||||
text = (
|
||||
"1\n00:00:01,000 --> 00:00:04,000\n第一行\n第二行\n\n"
|
||||
"2\n00:00:05,000 --> 00:00:08,000\n末条无空行结尾\n"
|
||||
)
|
||||
entries = parse_srt(text)
|
||||
assert len(entries) == 2
|
||||
assert entries[0]["start"] == "00:00:01,000"
|
||||
assert entries[0]["end"] == "00:00:04,000"
|
||||
assert entries[0]["text"] == "第一行\n第二行"
|
||||
assert entries[1]["text"] == "末条无空行结尾"
|
||||
|
||||
|
||||
def test_serialize_srt_renumbers() -> None:
|
||||
"""序列化：序号从 1 重新编号，保留原始时间轴。"""
|
||||
entries = [
|
||||
{"start": "00:00:09,000", "end": "00:00:12,000", "text": "答：无意义杂项"},
|
||||
{"start": "00:00:13,000", "end": "00:00:16,000", "text": "第二句正常字幕"},
|
||||
]
|
||||
out = serialize_srt(entries)
|
||||
assert out == (
|
||||
"1\n00:00:09,000 --> 00:00:12,000\n答：无意义杂项\n\n"
|
||||
"2\n00:00:13,000 --> 00:00:16,000\n第二句正常字幕\n"
|
||||
)
|
||||
|
||||
|
||||
def test_judge_target_window_and_keep(monkeypatch) -> None:
|
||||
"""窗口只含纯文本（无时间戳）、目标带标记；模型答"保留"则返回 False。"""
|
||||
from nodes.llm_filter import _judge_target
|
||||
|
||||
entries = parse_srt(_SRT)
|
||||
fake = _patch_llm(monkeypatch, ["保留"])
|
||||
# context_size=1，目标为第 2 条（index=1）：窗口 0..2 共 3 行，目标在中间。
|
||||
assert _judge_target(entries, 1, context_size=1, params={}) is False
|
||||
body = fake.bodies[0]
|
||||
lines = body["messages"][1]["content"].splitlines()
|
||||
assert len(lines) == 3
|
||||
assert lines[0] == "还有没有什么困扰"
|
||||
assert lines[1] == "【目标】或者奇怪的地方吗"
|
||||
assert lines[2] == "答：无意义杂项"
|
||||
# 不含时间戳。
|
||||
assert "00:00" not in body["messages"][1]["content"]
|
||||
assert body["enable_thinking"] is False
|
||||
assert body["max_tokens"] == 16
|
||||
|
||||
|
||||
def test_judge_target_delete(monkeypatch) -> None:
|
||||
"""模型答"删除"时返回 True（判定该条无意义）。"""
|
||||
from nodes.llm_filter import _judge_target
|
||||
|
||||
entries = parse_srt(_SRT)
|
||||
_patch_llm(monkeypatch, ["删除"])
|
||||
assert _judge_target(entries, 2, context_size=10, params={}) is True
|
||||
|
||||
|
||||
def test_judge_target_model_and_auth(monkeypatch) -> None:
|
||||
"""模型名从参数取；配置 API Key 时附带 Bearer 鉴权头。"""
|
||||
from nodes.llm_filter import _judge_target
|
||||
|
||||
entries = parse_srt(_SRT)
|
||||
monkeypatch.setenv("LLM_API_KEY", "sk-test")
|
||||
fake = _patch_llm(monkeypatch, ["保留"])
|
||||
assert _judge_target(entries, 0, context_size=10, params={"model": "m/1"}) is False
|
||||
assert fake.bodies[0]["model"] == "m/1"
|
||||
assert fake.headers[0]["Authorization"] == "Bearer sk-test"
|
||||
|
||||
|
||||
def test_invoke_filters_and_renumbers(monkeypatch, tmp_path) -> None:
|
||||
"""全链路（并发）：按 LLM 判定删除无意义条，保留条重新编号输出。"""
|
||||
srt = tmp_path / "in.srt"
|
||||
srt.write_text(_SRT, encoding="utf-8")
|
||||
# 内容决策：目标字幕含"答："判删除，其余保留（任何线程顺序下结果确定）。
|
||||
_patch_llm(monkeypatch, decision_fn=_decision_by_target)
|
||||
response = invoke(
|
||||
InvokeRequest(
|
||||
run_id="r", node_instance_id="",
|
||||
inputs={"srt_uri": str(srt)},
|
||||
params={},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
assert response.outputs["kept"] == 3
|
||||
assert response.outputs["removed"] == 1
|
||||
out = Path(response.outputs["srt_uri"]).read_text(encoding="utf-8")
|
||||
assert out.count("-->") == 3
|
||||
# 被删除的"答：无意义杂项"（第 3 条）时间轴不再出现。
|
||||
assert "00:00:09,000" not in out
|
||||
# 保留条重新编号且时间轴不变。
|
||||
assert out.startswith("1\n00:00:01,000 --> 00:00:04,000\n还有没有什么困扰\n\n2\n")
|
||||
assert "00:00:13,000 --> 00:00:16,000\n第二句正常字幕\n" in out
|
||||
|
||||
|
||||
def test_invoke_context_size_param(monkeypatch, tmp_path) -> None:
|
||||
"""context_size 参数生效：窗口大小=2×context_size+1（两端截断除外）。"""
|
||||
srt = tmp_path / "in.srt"
|
||||
srt.write_text(_SRT, encoding="utf-8")
|
||||
fake = _patch_llm(monkeypatch, decision_fn=_decision_by_target)
|
||||
response = invoke(
|
||||
InvokeRequest(
|
||||
run_id="r", node_instance_id="",
|
||||
inputs={"srt_uri": str(srt)},
|
||||
params={"context_size": 1},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
# 找到目标为第 2 条（"或者奇怪的地方吗"）的请求体：窗口应含 3 行。
|
||||
body = next(
|
||||
b for b in fake.bodies
|
||||
if "【目标】或者奇怪的地方吗" in b["messages"][1]["content"]
|
||||
)
|
||||
window = body["messages"][1]["content"].splitlines()
|
||||
assert len(window) == 3
|
||||
|
||||
|
||||
def test_invoke_missing_input(tmp_path) -> None:
|
||||
"""缺少 srt_uri 时返回失败。"""
|
||||
response = invoke(
|
||||
InvokeRequest(run_id="r", node_instance_id="", inputs={}, output_dir=str(tmp_path))
|
||||
)
|
||||
assert response.status == "failed"
|
||||
assert "srt_uri" in response.error
|
||||
|
||||
|
||||
def test_invoke_file_missing(tmp_path) -> None:
|
||||
"""srt 文件不存在时返回失败。"""
|
||||
response = invoke(
|
||||
InvokeRequest(
|
||||
run_id="r", node_instance_id="",
|
||||
inputs={"srt_uri": str(tmp_path / "none.srt")},
|
||||
output_dir=str(tmp_path),
|
||||
)
|
||||
)
|
||||
assert response.status == "failed"
|
||||
assert "not found" in response.error
|
||||
|
||||
|
||||
def test_invoke_llm_error(monkeypatch, tmp_path) -> None:
|
||||
"""LLM 调用失败（网络错误）时返回 failed，不静默输出未过滤结果。"""
|
||||
srt = tmp_path / "in.srt"
|
||||
srt.write_text(_SRT, encoding="utf-8")
|
||||
|
||||
def boom(request, timeout=None):
|
||||
raise urllib.error.URLError("llm down")
|
||||
|
||||
monkeypatch.setattr("nodes.llm_filter.urllib.request.urlopen", boom)
|
||||
response = invoke(
|
||||
InvokeRequest(
|
||||
run_id="r", node_instance_id="",
|
||||
inputs={"srt_uri": str(srt)},
|
||||
params={},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "failed"
|
||||
assert "llm down" in response.error
|
||||
|
||||
|
||||
def test_invoke_empty_srt(monkeypatch, tmp_path) -> None:
|
||||
"""空 SRT（无条目）正常完成，输出空文件且不调用 LLM。"""
|
||||
srt = tmp_path / "empty.srt"
|
||||
srt.write_text("", encoding="utf-8")
|
||||
fake = _patch_llm(monkeypatch, [])
|
||||
response = invoke(
|
||||
InvokeRequest(
|
||||
run_id="r", node_instance_id="",
|
||||
inputs={"srt_uri": str(srt)},
|
||||
params={},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
assert response.outputs["kept"] == 0
|
||||
assert response.outputs["removed"] == 0
|
||||
assert fake.bodies == []
|
||||
@@ -0,0 +1,165 @@
|
||||
"""孤儿数据清理器测试。
|
||||
|
||||
覆盖 COMPLETED 无文件任务的删除、各类保留分支（有文件/失败/宽限期内）、
|
||||
无任务记录的残留目录清理、清理线程启停以及防御性分支。
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from wov_app.db import Database
|
||||
from wov_app.maintenance import OrphanCleaner
|
||||
|
||||
|
||||
def _db(tmp_path) -> Database:
|
||||
"""在临时目录创建独立数据库。"""
|
||||
return Database(tmp_path / "wov.db")
|
||||
|
||||
|
||||
def _make_run(db, run_id, status="COMPLETED", updated="2020-01-01T00:00:00+00:00", input_uri=None):
|
||||
"""创建指定状态与更新时间的工作流任务记录。"""
|
||||
db.upsert_workflow({"id": "flow", "name": "F", "published": 1, "latest_version": 1})
|
||||
db.create_run(
|
||||
{
|
||||
"id": run_id,
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": status,
|
||||
"progress": 1,
|
||||
"input_uri": input_uri,
|
||||
"created_at": updated,
|
||||
"updated_at": updated,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
"""返回当前 UTC 时间的 ISO 字符串。"""
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def test_cleaner_removes_completed_orphan_run(tmp_path) -> None:
|
||||
"""验证 COMPLETED 且无任何产物文件、超过宽限期的任务被整体清理。"""
|
||||
db = _db(tmp_path)
|
||||
upload_dir = tmp_path / "storage" / "uploads" / "run_orphan"
|
||||
upload_dir.mkdir(parents=True)
|
||||
upload_file = upload_dir / "in.mp4"
|
||||
upload_file.write_bytes(b"x")
|
||||
_make_run(db, "run_orphan", input_uri=str(upload_file))
|
||||
db.create_artifact(
|
||||
{
|
||||
"run_id": "run_orphan",
|
||||
"node_id": "asr",
|
||||
"name": "asr.srt_uri",
|
||||
"uri": str(tmp_path / "storage" / "runs" / "run_orphan" / "out.srt"),
|
||||
"mime_type": "application/x-subrip",
|
||||
"size": 1,
|
||||
}
|
||||
)
|
||||
cleaner = OrphanCleaner(db, tmp_path / "storage", grace_seconds=3600)
|
||||
assert cleaner.clean_once() == 1
|
||||
assert db.get_run("run_orphan") is None
|
||||
assert db.list_artifacts("run_orphan") == []
|
||||
assert not upload_dir.exists()
|
||||
|
||||
|
||||
def test_cleaner_keeps_completed_run_with_files(tmp_path) -> None:
|
||||
"""验证仍有产物文件的 COMPLETED 任务不会被清理。"""
|
||||
db = _db(tmp_path)
|
||||
steps = tmp_path / "storage" / "runs" / "run_keep"
|
||||
(steps / "asr").mkdir(parents=True)
|
||||
(steps / "asr" / "out.srt").write_text("1\n00:00:00,000 --> 00:00:01,000\nok\n", encoding="utf-8")
|
||||
_make_run(db, "run_keep")
|
||||
cleaner = OrphanCleaner(db, tmp_path / "storage", grace_seconds=3600)
|
||||
assert cleaner.clean_once() == 0
|
||||
assert db.get_run("run_keep") is not None
|
||||
|
||||
|
||||
def test_cleaner_keeps_failed_and_recent_runs(tmp_path) -> None:
|
||||
"""验证 FAILED 任务与宽限期内的任务都不会被自动清理。"""
|
||||
db = _db(tmp_path)
|
||||
_make_run(db, "run_failed", status="FAILED")
|
||||
_make_run(db, "run_recent", updated=_now_iso())
|
||||
cleaner = OrphanCleaner(db, tmp_path / "storage", grace_seconds=3600)
|
||||
assert cleaner.clean_once() == 0
|
||||
assert db.get_run("run_failed") is not None
|
||||
assert db.get_run("run_recent") is not None
|
||||
|
||||
|
||||
def test_cleaner_removes_dangling_dirs_only(tmp_path) -> None:
|
||||
"""验证无任务记录的残留目录被删除，已有任务的上传目录被保留。"""
|
||||
db = _db(tmp_path)
|
||||
ghost_upload = tmp_path / "storage" / "uploads" / "ghost"
|
||||
ghost_upload.mkdir(parents=True)
|
||||
ghost_steps = tmp_path / "storage" / "runs" / "ghost"
|
||||
ghost_steps.mkdir(parents=True)
|
||||
keep_upload = tmp_path / "storage" / "uploads" / "run_keep"
|
||||
keep_upload.mkdir(parents=True)
|
||||
(keep_upload / "in.mp4").write_bytes(b"x")
|
||||
# run_keep 存在产物文件，不属于孤儿任务。
|
||||
keep_steps = tmp_path / "storage" / "runs" / "run_keep" / "asr"
|
||||
keep_steps.mkdir(parents=True)
|
||||
(keep_steps / "out.srt").write_text("ok", encoding="utf-8")
|
||||
_make_run(db, "run_keep")
|
||||
cleaner = OrphanCleaner(db, tmp_path / "storage", grace_seconds=3600)
|
||||
assert cleaner.clean_once() == 2
|
||||
assert not ghost_upload.exists()
|
||||
assert not ghost_steps.exists()
|
||||
assert keep_upload.exists()
|
||||
|
||||
|
||||
def test_cleaner_removes_run_with_empty_steps_dir(tmp_path) -> None:
|
||||
"""验证步骤目录存在但为空（无文件）时仍视为孤儿清理。"""
|
||||
db = _db(tmp_path)
|
||||
steps = tmp_path / "storage" / "runs" / "run_empty"
|
||||
(steps / "asr").mkdir(parents=True)
|
||||
_make_run(db, "run_empty", input_uri="")
|
||||
cleaner = OrphanCleaner(db, tmp_path / "storage", grace_seconds=3600)
|
||||
assert cleaner.clean_once() == 1
|
||||
assert db.get_run("run_empty") is None
|
||||
assert not steps.exists()
|
||||
|
||||
|
||||
def test_cleaner_default_config_and_defensive_branches(tmp_path, monkeypatch) -> None:
|
||||
"""验证默认配置构造、缺失/非法时间与缺失任务记录的防御分支。"""
|
||||
db = _db(tmp_path)
|
||||
# 默认配置（interval/grace 走 config 默认值）。
|
||||
cleaner = OrphanCleaner(db, tmp_path / "storage")
|
||||
assert cleaner.interval_seconds > 0
|
||||
assert cleaner.grace_seconds > 0
|
||||
# 无更新时间 / 非法时间均保守视为未过期。
|
||||
assert cleaner._expired(None) is False
|
||||
assert cleaner._expired("not-a-date") is False
|
||||
# 不存在的根目录直接返回 0。
|
||||
assert cleaner._clean_dangling(tmp_path / "missing", set()) == 0
|
||||
# list_run_ids 返回的 ID 在读取详情前已不存在时跳过。
|
||||
monkeypatch.setattr(db, "list_run_ids", lambda: ["ghost"])
|
||||
monkeypatch.setattr(db, "get_run", lambda run_id: None)
|
||||
assert cleaner.clean_once() == 0
|
||||
|
||||
|
||||
def test_cleaner_start_stop_loop(tmp_path) -> None:
|
||||
"""验证清理线程可启动、周期执行并正常停止。"""
|
||||
import time
|
||||
|
||||
db = _db(tmp_path)
|
||||
cleaner = OrphanCleaner(db, tmp_path / "storage", interval_seconds=0.05, grace_seconds=3600)
|
||||
cleaner.start()
|
||||
try:
|
||||
cleaner.start()
|
||||
time.sleep(0.2)
|
||||
finally:
|
||||
cleaner.stop()
|
||||
assert cleaner._thread is None
|
||||
|
||||
|
||||
def test_lifespan_starts_cleaner(monkeypatch) -> None:
|
||||
"""验证启用清理器时应用生命周期会启动清理线程并随退出停止。"""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from wov_app.main import app
|
||||
|
||||
monkeypatch.setenv("WOV_CLEANUP_ENABLED", "1")
|
||||
with TestClient(app) as client:
|
||||
assert client.get("/health").status_code == 200
|
||||
assert app.state.cleaner._thread is not None
|
||||
Executable
+174
@@ -0,0 +1,174 @@
|
||||
"""wov_sdk.models 的单元测试。
|
||||
|
||||
测试覆盖所有数据模型的 JSON 往返序列化、字段校验和 manifest 文件加载，
|
||||
确保协议模型的稳定性。
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from wov_sdk.models import (
|
||||
HealthResponse,
|
||||
InvokeRequest,
|
||||
InvokeResponse,
|
||||
NodeManifest,
|
||||
ProgressEvent,
|
||||
WorkflowDefinition,
|
||||
WorkflowEdge,
|
||||
WorkflowNode,
|
||||
)
|
||||
|
||||
|
||||
def valid_manifest() -> NodeManifest:
|
||||
"""构造一个覆盖全部字段的合法 NodeManifest，供测试复用。"""
|
||||
return NodeManifest(
|
||||
id="echo",
|
||||
name="Echo",
|
||||
version="1.0.0",
|
||||
capability="echo",
|
||||
command=["python", "-m", "echo"],
|
||||
repo_dir="wov-node-echo",
|
||||
env={"PORT": "0"},
|
||||
input_schema={"text": "string"},
|
||||
output_schema={"text": "string"},
|
||||
max_concurrency=2,
|
||||
idle_ttl_seconds=15,
|
||||
health_timeout_seconds=5,
|
||||
keep_warm=True,
|
||||
)
|
||||
|
||||
|
||||
def test_manifest_round_trip() -> None:
|
||||
"""验证 manifest 经过 to_dict/from_dict 后保持原值。"""
|
||||
manifest = valid_manifest()
|
||||
restored = NodeManifest.from_dict(manifest.to_dict())
|
||||
assert restored == manifest
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("field", "value"),
|
||||
[
|
||||
("id", ""),
|
||||
("name", ""),
|
||||
("version", ""),
|
||||
("capability", ""),
|
||||
("repo_dir", ""),
|
||||
("command", []),
|
||||
("max_concurrency", 0),
|
||||
("idle_ttl_seconds", -1),
|
||||
("health_timeout_seconds", 0),
|
||||
],
|
||||
)
|
||||
def test_manifest_validation(field: str, value: object) -> None:
|
||||
"""验证必填字段为空或数值越界时抛出 ValueError。"""
|
||||
manifest = valid_manifest()
|
||||
setattr(manifest, field, value)
|
||||
with pytest.raises(ValueError):
|
||||
manifest.validate()
|
||||
|
||||
|
||||
def test_manifest_load(tmp_path) -> None:
|
||||
"""验证 NodeManifest.load 能从 JSON 文件读取并校验。"""
|
||||
path = tmp_path / "node.manifest.json"
|
||||
path.write_text(json.dumps(valid_manifest().to_dict()), encoding="utf-8")
|
||||
loaded = NodeManifest.load(str(path))
|
||||
assert loaded.id == "echo"
|
||||
|
||||
|
||||
def test_invoke_request_round_trip() -> None:
|
||||
"""验证 InvokeRequest 的 JSON 往返序列化。"""
|
||||
request = InvokeRequest(
|
||||
run_id="run_1",
|
||||
node_instance_id="ni_1",
|
||||
inputs={"text": "hello"},
|
||||
params={"temperature": 0.2},
|
||||
output_dir="out",
|
||||
)
|
||||
restored = InvokeRequest.from_dict(request.to_dict())
|
||||
assert restored == request
|
||||
|
||||
|
||||
def test_invoke_response_round_trip() -> None:
|
||||
"""验证 InvokeResponse 的 JSON 往返序列化。"""
|
||||
response = InvokeResponse(status="completed", outputs={"text": "hello"})
|
||||
restored = InvokeResponse.from_dict(response.to_dict())
|
||||
assert restored == response
|
||||
|
||||
|
||||
def test_health_and_progress_serialization() -> None:
|
||||
"""验证健康检查和进度事件模型的字典输出。"""
|
||||
health = HealthResponse(status="ok", node_id="echo", version="1.0.0")
|
||||
assert health.to_dict() == {
|
||||
"status": "ok",
|
||||
"node_id": "echo",
|
||||
"version": "1.0.0",
|
||||
}
|
||||
|
||||
progress = ProgressEvent(run_id="run_1", node_id="echo", progress=0.5, message="half")
|
||||
assert progress.to_dict() == {
|
||||
"run_id": "run_1",
|
||||
"node_id": "echo",
|
||||
"progress": 0.5,
|
||||
"message": "half",
|
||||
}
|
||||
|
||||
|
||||
def test_workflow_node_and_edge_round_trip() -> None:
|
||||
"""验证工作流节点与边的 JSON 往返序列化。"""
|
||||
node = WorkflowNode(
|
||||
id="asr",
|
||||
node_type="faster-whisper",
|
||||
params={"language": "ja"},
|
||||
inputs={"audio_uri": "extract.audio_uri"},
|
||||
)
|
||||
edge = WorkflowEdge(from_node="extract", to_node="asr")
|
||||
assert WorkflowNode.from_dict(node.to_dict()) == node
|
||||
assert WorkflowEdge.from_dict(edge.to_dict()) == edge
|
||||
assert edge.to_dict() == {"from": "extract", "to": "asr"}
|
||||
|
||||
assert node.to_dict()["inputs"] == {"audio_uri": "extract.audio_uri"}
|
||||
|
||||
|
||||
def test_workflow_definition_round_trip_and_validation() -> None:
|
||||
"""验证完整 DAG 定义可往返序列化并通过校验。"""
|
||||
definition = WorkflowDefinition(
|
||||
name="demo",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(id="extract", node_type="ffmpeg"),
|
||||
WorkflowNode(id="asr", node_type="whisper"),
|
||||
],
|
||||
edges=[WorkflowEdge(from_node="extract", to_node="asr")],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
final_outputs={"srt": "asr.srt_uri"},
|
||||
)
|
||||
restored = WorkflowDefinition.from_dict(definition.to_dict())
|
||||
assert restored == definition
|
||||
restored.validate()
|
||||
|
||||
|
||||
def test_workflow_definition_invalid() -> None:
|
||||
"""验证非法 DAG（空名、版本为 0、重复节点、未知边）被拒绝。"""
|
||||
with pytest.raises(ValueError):
|
||||
WorkflowDefinition(name="", version=1).validate()
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
WorkflowDefinition(name="demo", version=0).validate()
|
||||
|
||||
duplicate = WorkflowDefinition(
|
||||
name="demo",
|
||||
version=1,
|
||||
nodes=[WorkflowNode(id="a", node_type="x"), WorkflowNode(id="a", node_type="y")],
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
duplicate.validate()
|
||||
|
||||
unknown_edge = WorkflowDefinition(
|
||||
name="demo",
|
||||
version=1,
|
||||
nodes=[WorkflowNode(id="a", node_type="x")],
|
||||
edges=[WorkflowEdge(from_node="a", to_node="missing")],
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
unknown_edge.validate()
|
||||
+1295
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,401 @@
|
||||
"""抽帧与字幕 OCR 节点单元测试。
|
||||
|
||||
frame-extract 用 testdata 真实视频抽帧+裁切+720p 压缩；subtitle-ocr 的 OCR
|
||||
网络调用（vlm-ocr）按 I/O 边界 mock，但喂给它的帧图片是真实的 testdata 资产。
|
||||
应用层不再加工模型输出文本，只做长度上限校验（超长报错跳过）。
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
from nodes.frame_extract import invoke as frame_invoke
|
||||
from nodes.subtitle_ocr import _assemble_srt
|
||||
from nodes.subtitle_ocr import invoke as ocr_invoke
|
||||
|
||||
# 单体根目录：tests/ 的上一级。
|
||||
WORKSPACE = Path(__file__).resolve().parent.parent
|
||||
TESTDATA = WORKSPACE / "testdata"
|
||||
# 10s 测试视频：SUB 001 在 1-4s、SUB 002 在 6-9s。
|
||||
VIDEO = TESTDATA / "subtitle_10s.mp4"
|
||||
TEXT_IMG = TESTDATA / "ocr_text.png"
|
||||
|
||||
|
||||
def _png_size(path: Path) -> tuple[int, int]:
|
||||
"""从 PNG 头读取宽高（真实图片尺寸断言）。"""
|
||||
data = path.read_bytes()
|
||||
assert data[:8] == b"\x89PNG\r\n\x1a\n", "not a real png"
|
||||
width = int.from_bytes(data[16:20], "big")
|
||||
height = int.from_bytes(data[20:24], "big")
|
||||
return width, height
|
||||
|
||||
|
||||
def _frame_request(tmp_path, video=VIDEO, **params) -> InvokeRequest:
|
||||
"""构造 frame-extract 调用请求。"""
|
||||
return InvokeRequest(
|
||||
run_id="run_fx",
|
||||
node_instance_id="",
|
||||
inputs={"video_uri": str(video)},
|
||||
params=params,
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# frame-extract：抽帧 + 裁切 + 720p 压缩
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_frame_extract_crop_and_manifest(tmp_path) -> None:
|
||||
"""真实视频抽帧：裁切下半 50% 后帧尺寸为 1280x360，清单时间轴正确。"""
|
||||
response = frame_invoke(
|
||||
_frame_request(tmp_path, interval_seconds=1, crop=[0, 0.5, 1, 0.5])
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
manifest = json.loads(Path(response.outputs["frames_manifest"]).read_text(encoding="utf-8"))
|
||||
assert len(manifest) >= 9
|
||||
assert [round(item["time"], 3) for item in manifest] == [
|
||||
round(i * 1.0, 3) for i in range(len(manifest))
|
||||
]
|
||||
first = Path(manifest[0]["image_uri"])
|
||||
assert first.is_file()
|
||||
# 1280x360 已在 720p 内，压缩不改变尺寸。
|
||||
assert _png_size(first) == (1280, 360)
|
||||
|
||||
|
||||
def test_frame_extract_default_params(tmp_path) -> None:
|
||||
"""未指定参数时使用默认值：抽帧间隔 0.5 秒 + 默认底部裁切区域。"""
|
||||
response = frame_invoke(_frame_request(tmp_path))
|
||||
assert response.status == "completed", response.error
|
||||
manifest = json.loads(Path(response.outputs["frames_manifest"]).read_text(encoding="utf-8"))
|
||||
assert manifest
|
||||
# 默认间隔 0.5s：25fps 下 step=round(12.5)=12（银行家舍入），
|
||||
# 帧时间按 step/fps=12/25=0.48s 步进（帧号精确，采样周期由帧量化决定）。
|
||||
assert [round(item["time"], 3) for item in manifest] == [
|
||||
round(i * 12 / 25, 3) for i in range(len(manifest))
|
||||
]
|
||||
|
||||
def test_frame_extract_missing_video(tmp_path) -> None:
|
||||
"""缺少 video_uri 时返回失败。"""
|
||||
response = frame_invoke(
|
||||
InvokeRequest(
|
||||
run_id="r", node_instance_id="", inputs={}, output_dir=str(tmp_path)
|
||||
)
|
||||
)
|
||||
assert response.status == "failed"
|
||||
|
||||
|
||||
def test_frame_extract_bad_crop(tmp_path) -> None:
|
||||
"""crop 比例越界（超出画面）时返回失败。"""
|
||||
assert frame_invoke(_frame_request(tmp_path, crop=[0, 0.5, 1, 1.5])).status == "failed"
|
||||
assert frame_invoke(_frame_request(tmp_path, crop=[-0.1, 0, 1, 0.5])).status == "failed"
|
||||
assert frame_invoke(_frame_request(tmp_path, crop="abc")).status == "failed"
|
||||
assert frame_invoke(_frame_request(tmp_path, crop=[0, 0.5, 1])).status == "failed"
|
||||
# 各值域合法但 x+w 越出画面。
|
||||
assert frame_invoke(_frame_request(tmp_path, crop=[0.6, 0, 0.5, 0.3])).status == "failed"
|
||||
|
||||
|
||||
def test_frame_extract_video_missing_file(tmp_path) -> None:
|
||||
"""video_uri 指向不存在的文件时返回失败。"""
|
||||
response = frame_invoke(_frame_request(tmp_path, video=tmp_path / "none.mp4"))
|
||||
assert response.status == "failed"
|
||||
assert "not found" in response.error
|
||||
|
||||
|
||||
def test_frame_extract_bad_interval(tmp_path) -> None:
|
||||
"""间隔 <= 0 时返回失败。"""
|
||||
response = frame_invoke(_frame_request(tmp_path, interval_seconds=0))
|
||||
assert response.status == "failed"
|
||||
|
||||
|
||||
def test_frame_extract_ffmpeg_fails(monkeypatch, tmp_path) -> None:
|
||||
"""ffmpeg 抽帧失败时透传错误。"""
|
||||
import subprocess as sp
|
||||
|
||||
monkeypatch.setattr("nodes.frame_extract._video_size", lambda *a, **k: (1280, 720))
|
||||
monkeypatch.setattr("nodes.frame_extract._video_fps", lambda *a, **k: 25.0)
|
||||
monkeypatch.setattr("nodes.frame_extract._video_duration", lambda *a, **k: 10.0)
|
||||
monkeypatch.setattr(
|
||||
"nodes.frame_extract.subprocess.run",
|
||||
lambda *a, **k: sp.CompletedProcess([], 1, stderr="boom"),
|
||||
)
|
||||
response = frame_invoke(_frame_request(tmp_path))
|
||||
assert response.status == "failed"
|
||||
assert "boom" in response.error
|
||||
|
||||
|
||||
def test_video_size_unreadable(monkeypatch) -> None:
|
||||
"""ffmpeg -i 输出不含视频流信息时返回 None。"""
|
||||
import subprocess as sp
|
||||
|
||||
from nodes.frame_extract import _video_size
|
||||
|
||||
monkeypatch.setattr(
|
||||
"nodes.frame_extract.subprocess.run",
|
||||
lambda *a, **k: sp.CompletedProcess([], 0, stderr="no video stream"),
|
||||
)
|
||||
assert _video_size(Path("/tmp/x.mp4"), "ffmpeg") is None
|
||||
|
||||
|
||||
def test_frame_extract_video_size_unknown(monkeypatch, tmp_path) -> None:
|
||||
"""无法读取视频分辨率时返回失败。"""
|
||||
monkeypatch.setattr("nodes.frame_extract._video_size", lambda *a, **k: None)
|
||||
response = frame_invoke(_frame_request(tmp_path))
|
||||
assert response.status == "failed"
|
||||
assert "video size" in response.error
|
||||
|
||||
|
||||
def test_video_duration_unreadable(monkeypatch) -> None:
|
||||
"""ffmpeg -i 输出缺少 Duration 时返回 None。"""
|
||||
import subprocess as sp
|
||||
|
||||
from nodes.frame_extract import _video_duration
|
||||
|
||||
monkeypatch.setattr(
|
||||
"nodes.frame_extract.subprocess.run",
|
||||
lambda *a, **k: sp.CompletedProcess([], 0, stderr="no duration info"),
|
||||
)
|
||||
assert _video_duration(Path("/tmp/x.mp4"), "ffmpeg") is None
|
||||
|
||||
|
||||
def test_frame_extract_duration_unknown(monkeypatch, tmp_path) -> None:
|
||||
"""无法读取视频时长时返回失败。"""
|
||||
monkeypatch.setattr("nodes.frame_extract._video_size", lambda *a, **k: (1280, 720))
|
||||
monkeypatch.setattr("nodes.frame_extract._video_duration", lambda *a, **k: None)
|
||||
response = frame_invoke(_frame_request(tmp_path))
|
||||
assert response.status == "failed"
|
||||
assert "duration" in response.error
|
||||
|
||||
|
||||
def test_frame_extract_fps_unknown(monkeypatch, tmp_path) -> None:
|
||||
"""无法读取视频帧率时返回失败。"""
|
||||
monkeypatch.setattr("nodes.frame_extract._video_size", lambda *a, **k: (1280, 720))
|
||||
monkeypatch.setattr("nodes.frame_extract._video_duration", lambda *a, **k: 10.0)
|
||||
monkeypatch.setattr("nodes.frame_extract._video_fps", lambda *a, **k: None)
|
||||
response = frame_invoke(_frame_request(tmp_path))
|
||||
assert response.status == "failed"
|
||||
assert "fps" in response.error
|
||||
|
||||
def test_frame_step_conversion() -> None:
|
||||
"""帧间隔换算：step=round(间隔秒×fps)，至少为 1。"""
|
||||
from nodes.frame_extract import _frame_step
|
||||
|
||||
assert _frame_step(fps=25.0, interval=0.2) == 5
|
||||
assert _frame_step(fps=25.0, interval=1.0) == 25
|
||||
assert _frame_step(fps=29.97, interval=1.0) == 30
|
||||
# fps 很低时 step 也不会小于 1（每帧都取）。
|
||||
assert _frame_step(fps=1.0, interval=0.2) == 1
|
||||
|
||||
|
||||
def test_video_fps_parse(monkeypatch) -> None:
|
||||
"""帧率解析：支持小数（29.97）与有理数（30000/1001）。"""
|
||||
import subprocess as sp
|
||||
|
||||
from nodes.frame_extract import _video_fps
|
||||
|
||||
monkeypatch.setattr(
|
||||
"nodes.frame_extract.subprocess.run",
|
||||
lambda *a, **k: sp.CompletedProcess(
|
||||
[], 0, stderr="Stream #0:0: Video: h264, 1280x720, 30000/1001 fps, 30000/1001 tbr"
|
||||
),
|
||||
)
|
||||
assert _video_fps(Path("/tmp/x.mp4"), "ffmpeg") == 30000 / 1001
|
||||
monkeypatch.setattr(
|
||||
"nodes.frame_extract.subprocess.run",
|
||||
lambda *a, **k: sp.CompletedProcess(
|
||||
[], 0, stderr="Stream #0:0: Video: h264, 1280x720, 25 fps, 25 tbr"
|
||||
),
|
||||
)
|
||||
assert _video_fps(Path("/tmp/x.mp4"), "ffmpeg") == 25.0
|
||||
monkeypatch.setattr(
|
||||
"nodes.frame_extract.subprocess.run",
|
||||
lambda *a, **k: sp.CompletedProcess([], 0, stderr="no video stream"),
|
||||
)
|
||||
assert _video_fps(Path("/tmp/x.mp4"), "ffmpeg") is None
|
||||
|
||||
def test_frame_extract_one_second_exact_frames(tmp_path) -> None:
|
||||
"""真实视频 1s 间隔按秒 seek 精确抽帧：10s 视频应得 10 帧，时间 0..9。"""
|
||||
response = frame_invoke(
|
||||
_frame_request(tmp_path, interval_seconds=1, crop=[0, 0.7, 1, 0.3])
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
manifest = json.loads(Path(response.outputs["frames_manifest"]).read_text(encoding="utf-8"))
|
||||
assert [round(item["time"], 3) for item in manifest] == [
|
||||
round(i * 1.0, 3) for i in range(len(manifest))
|
||||
]
|
||||
assert len(manifest) == 10
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# subtitle-ocr：OCR 循环 + 长度上限 + 合并 + SRT 组装
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _frames_manifest(tmp_path, frame_specs) -> Path:
|
||||
"""构造真实 frames.json；frame_specs=[(time, image_path), ...]。"""
|
||||
items = [{"time": time, "image_uri": str(image)} for time, image in frame_specs]
|
||||
path = tmp_path / "frames.json"
|
||||
path.write_text(json.dumps(items), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def test_assemble_srt_real_timeline() -> None:
|
||||
"""SRT 组装：起始=帧时间，结束=最后可见帧时间+采样间隔。"""
|
||||
lines = _assemble_srt([(0.0, 6.0, "A"), (6.0, 8.0, "B")], interval_seconds=2.0)
|
||||
text = "\n".join(lines)
|
||||
assert text.startswith("1\n")
|
||||
# A 最后可见帧 6.0 + 间隔 2.0 = 8.0（而非下一条字幕的出现时间）。
|
||||
assert "00:00:00,000 --> 00:00:08,000" in text
|
||||
assert "00:00:06,000 --> 00:00:10,000" in text
|
||||
|
||||
|
||||
def test_sampling_interval_from_manifest() -> None:
|
||||
"""采样间隔从帧清单时间轴推导：均匀间隔取相邻差，退化清单回退默认值。"""
|
||||
from nodes.subtitle_ocr import _sampling_interval
|
||||
|
||||
manifest = [{"time": i * 0.2, "image_uri": f"f{i}.png"} for i in range(10)]
|
||||
assert _sampling_interval(manifest, 2.0) == 0.2
|
||||
# 单帧（无法算差）与异常时间序：回退默认值。
|
||||
assert _sampling_interval([{"time": 0.0, "image_uri": "f0.png"}], 2.0) == 2.0
|
||||
assert _sampling_interval(
|
||||
[{"time": 0.0}, {"time": 0.0}, {"time": 0.2}], 2.0
|
||||
) == 0.2
|
||||
|
||||
def test_ocr_merges_consecutive_same_text(monkeypatch, tmp_path) -> None:
|
||||
"""连续帧相同字幕合并为一条；消失时间=最后可见帧+间隔，空白段保留。"""
|
||||
# SUB 001 在 0/2s，4s 为空帧，SUB 002 在 6/8s。
|
||||
frame_texts = [
|
||||
(0.0, "SUB 001"), (2.0, "SUB 001"), (4.0, ""),
|
||||
(6.0, "SUB 002"), (8.0, "SUB 002"),
|
||||
]
|
||||
frames = []
|
||||
for index, (time, _text) in enumerate(frame_texts):
|
||||
image = tmp_path / f"f{index}.png"
|
||||
image.write_bytes(TEXT_IMG.read_bytes())
|
||||
frames.append((time, image))
|
||||
mapping = {str(image): text for (time, image), (_, text) in zip(frames, frame_texts)}
|
||||
|
||||
def fake_vlm(node_id, request):
|
||||
return InvokeResponse(status="completed", outputs={"text": mapping[request.inputs["image_uri"]]})
|
||||
|
||||
monkeypatch.setattr("wov_app.registry.invoke", fake_vlm)
|
||||
manifest = _frames_manifest(tmp_path, frames)
|
||||
response = ocr_invoke(
|
||||
InvokeRequest(
|
||||
run_id="run_ocr",
|
||||
node_instance_id="",
|
||||
inputs={"frames_manifest": str(manifest)},
|
||||
params={},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
srt = Path(response.outputs["srt_uri"]).read_text(encoding="utf-8")
|
||||
assert srt.count("SUB 001") == 1
|
||||
assert srt.count("SUB 002") == 1
|
||||
# SUB 001 最后可见帧 2.0 + 间隔 2.0 = 4.0 消失（而非拖到 SUB 002 出现）。
|
||||
assert "00:00:00,000 --> 00:00:04,000" in srt
|
||||
assert "00:00:06,000 --> 00:00:10,000" in srt
|
||||
|
||||
|
||||
def test_ocr_skips_failed_frames(monkeypatch, tmp_path) -> None:
|
||||
"""个别帧 OCR 失败时跳过，不影响其余帧汇总。"""
|
||||
frames = []
|
||||
for index in range(3):
|
||||
image = tmp_path / f"f{index}.png"
|
||||
image.write_bytes(TEXT_IMG.read_bytes())
|
||||
frames.append((index * 2.0, image))
|
||||
|
||||
def fake_vlm(node_id, request):
|
||||
if "f1" in request.inputs["image_uri"]:
|
||||
return InvokeResponse(status="failed", error="boom")
|
||||
return InvokeResponse(status="completed", outputs={"text": "SUB 001"})
|
||||
|
||||
monkeypatch.setattr("wov_app.registry.invoke", fake_vlm)
|
||||
manifest = _frames_manifest(tmp_path, frames)
|
||||
response = ocr_invoke(
|
||||
InvokeRequest(
|
||||
run_id="run_ocr",
|
||||
node_instance_id="",
|
||||
inputs={"frames_manifest": str(manifest)},
|
||||
params={},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
srt = Path(response.outputs["srt_uri"]).read_text(encoding="utf-8")
|
||||
assert "SUB 001" in srt
|
||||
|
||||
|
||||
def test_ocr_missing_manifest(tmp_path) -> None:
|
||||
"""缺少 frames_manifest 或清单文件不存在时返回失败。"""
|
||||
response = ocr_invoke(
|
||||
InvokeRequest(run_id="r", node_instance_id="", inputs={}, output_dir=str(tmp_path))
|
||||
)
|
||||
assert response.status == "failed"
|
||||
response = ocr_invoke(
|
||||
InvokeRequest(
|
||||
run_id="r", node_instance_id="",
|
||||
inputs={"frames_manifest": str(tmp_path / "none.json")},
|
||||
output_dir=str(tmp_path),
|
||||
)
|
||||
)
|
||||
assert response.status == "failed"
|
||||
|
||||
|
||||
def test_ocr_skips_oversized_output(monkeypatch, tmp_path) -> None:
|
||||
"""超长输出（模型重复循环等）直接报错跳过该帧，不进入 SRT。"""
|
||||
frames = []
|
||||
for index in range(3):
|
||||
image = tmp_path / f"o{index}.png"
|
||||
image.write_bytes(TEXT_IMG.read_bytes())
|
||||
frames.append((index * 2.0, image))
|
||||
|
||||
def fake_vlm(node_id, request):
|
||||
if "o0" in request.inputs["image_uri"]:
|
||||
return InvokeResponse(status="completed", outputs={"text": "重复字幕\n" * 50})
|
||||
if "o1" in request.inputs["image_uri"]:
|
||||
# 空输出帧跳过。
|
||||
return InvokeResponse(status="completed", outputs={"text": ""})
|
||||
return InvokeResponse(status="completed", outputs={"text": "SUB 001"})
|
||||
|
||||
monkeypatch.setattr("wov_app.registry.invoke", fake_vlm)
|
||||
manifest = _frames_manifest(tmp_path, frames)
|
||||
response = ocr_invoke(
|
||||
InvokeRequest(
|
||||
run_id="run_ocr",
|
||||
node_instance_id="",
|
||||
inputs={"frames_manifest": str(manifest)},
|
||||
params={"max_result_chars": 200},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
srt = Path(response.outputs["srt_uri"]).read_text(encoding="utf-8")
|
||||
assert "SUB 001" in srt
|
||||
assert "重复字幕" not in srt
|
||||
|
||||
|
||||
def test_ocr_passes_short_text_through(monkeypatch, tmp_path) -> None:
|
||||
"""不超过上限的模型输出原样进入 SRT（不再做应用层过滤）。"""
|
||||
image = tmp_path / "s0.png"
|
||||
image.write_bytes(TEXT_IMG.read_bytes())
|
||||
frames = [(0.0, image)]
|
||||
|
||||
def fake_vlm(node_id, request):
|
||||
return InvokeResponse(status="completed", outputs={"text": " SUB 001 "})
|
||||
|
||||
monkeypatch.setattr("wov_app.registry.invoke", fake_vlm)
|
||||
manifest = _frames_manifest(tmp_path, frames)
|
||||
response = ocr_invoke(
|
||||
InvokeRequest(
|
||||
run_id="run_ocr", node_instance_id="",
|
||||
inputs={"frames_manifest": str(manifest)},
|
||||
params={},
|
||||
output_dir=str(tmp_path / "out"),
|
||||
)
|
||||
)
|
||||
assert response.status == "completed", response.error
|
||||
srt = Path(response.outputs["srt_uri"]).read_text(encoding="utf-8")
|
||||
assert "SUB 001" in srt
|
||||
@@ -0,0 +1,116 @@
|
||||
"""进程内节点注册表测试。
|
||||
|
||||
覆盖节点注册、全量注册、查询、进程内调用以及未注册节点的报错路径，
|
||||
验证注册表作为调度器唯一调用入口的正确性。
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from wov_app import registry
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse, NodeManifest
|
||||
|
||||
|
||||
def _echo_manifest() -> NodeManifest:
|
||||
"""构造最小合法 echo 节点清单。"""
|
||||
return NodeManifest(
|
||||
id="echo",
|
||||
name="Echo",
|
||||
version="1.0.0",
|
||||
capability="echo",
|
||||
command=["python", "-m", "echo"],
|
||||
repo_dir="nodes",
|
||||
)
|
||||
|
||||
|
||||
def test_register_and_list() -> None:
|
||||
"""验证注册后可查询与列出节点，且按 ID 排序。"""
|
||||
registry.register(_echo_manifest(), lambda request: InvokeResponse(status="completed"))
|
||||
registry.register(
|
||||
NodeManifest(
|
||||
id="z-node",
|
||||
name="Z",
|
||||
version="1",
|
||||
capability="x",
|
||||
command=["python", "-m", "z"],
|
||||
repo_dir="nodes",
|
||||
),
|
||||
lambda request: InvokeResponse(status="completed"),
|
||||
)
|
||||
assert [node.id for node in registry.list_nodes()] == ["echo", "z-node"]
|
||||
assert registry.get_node("echo").capability == "echo"
|
||||
assert registry.get_node("missing") is None
|
||||
|
||||
|
||||
def test_register_validation() -> None:
|
||||
"""验证非法 manifest 注册会被协议校验拒绝。"""
|
||||
invalid = _echo_manifest()
|
||||
invalid.id = ""
|
||||
with pytest.raises(ValueError):
|
||||
registry.register(invalid, lambda request: InvokeResponse(status="completed"))
|
||||
|
||||
|
||||
def test_register_all_loads_builtin_nodes() -> None:
|
||||
"""验证 register_all 会加载 manifests/ 下全部内置节点。"""
|
||||
registry.register_all()
|
||||
ids = {node.id for node in registry.list_nodes()}
|
||||
assert {
|
||||
"echo",
|
||||
"ffmpeg-extract",
|
||||
"faster-whisper",
|
||||
"llm-translate",
|
||||
"vlm-ocr",
|
||||
"srt-to-dual-eye-ass",
|
||||
} <= ids
|
||||
|
||||
|
||||
def test_invoke_calls_handler() -> None:
|
||||
"""验证 invoke 会把请求转发给注册的进程内处理器。"""
|
||||
captured = {}
|
||||
|
||||
def handler(request: InvokeRequest) -> InvokeResponse:
|
||||
captured["run_id"] = request.run_id
|
||||
return InvokeResponse(status="completed", outputs={"text": "ok"})
|
||||
|
||||
registry.register(_echo_manifest(), handler)
|
||||
response = registry.invoke("echo", InvokeRequest(run_id="run_1", node_instance_id=""))
|
||||
assert response.status == "completed"
|
||||
assert response.outputs == {"text": "ok"}
|
||||
assert captured["run_id"] == "run_1"
|
||||
|
||||
|
||||
def test_invoke_unknown_node() -> None:
|
||||
"""验证调用未注册节点时抛出 ValueError。"""
|
||||
with pytest.raises(ValueError, match="not registered"):
|
||||
registry.invoke("missing", InvokeRequest(run_id="run_1", node_instance_id=""))
|
||||
|
||||
|
||||
def test_invoke_logs_node_lifecycle(caplog) -> None:
|
||||
"""验证 invoke 会记录节点的开始/完成/耗时日志（主进程可见）。"""
|
||||
registry.register(
|
||||
_echo_manifest(), lambda request: InvokeResponse(status="completed", outputs={"text": "ok"})
|
||||
)
|
||||
with caplog.at_level("INFO", logger="vrsub.node"):
|
||||
registry.invoke("echo", InvokeRequest(run_id="run_1", node_instance_id=""))
|
||||
assert any("节点 echo 开始" in record.message for record in caplog.records)
|
||||
assert any("节点 echo 完成" in record.message for record in caplog.records)
|
||||
|
||||
|
||||
def test_invoke_logs_node_failure(caplog) -> None:
|
||||
"""验证节点返回 failed 时记录失败日志。"""
|
||||
registry.register(
|
||||
_echo_manifest(), lambda request: InvokeResponse(status="failed", error="boom")
|
||||
)
|
||||
with caplog.at_level("INFO", logger="vrsub.node"):
|
||||
registry.invoke("echo", InvokeRequest(run_id="run_1", node_instance_id=""))
|
||||
assert any("节点 echo 失败" in record.message for record in caplog.records)
|
||||
|
||||
|
||||
def test_get_logger_idempotent() -> None:
|
||||
"""验证日志器重复获取不会重复附加控制台处理器。"""
|
||||
from wov_app.logging import get_logger
|
||||
|
||||
logger = get_logger("idempotent")
|
||||
handler_count = len(logger.handlers)
|
||||
again = get_logger("idempotent")
|
||||
assert again is logger
|
||||
assert len(again.handlers) == handler_count
|
||||
@@ -0,0 +1,655 @@
|
||||
"""调度器单元测试。
|
||||
|
||||
覆盖拓扑排序、任务执行成功/失败分支、输入引用解析、MIME 推断以及
|
||||
后台轮询线程的启动与停止。节点调用改为进程内注册表直接调用。
|
||||
"""
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from wov_app import registry
|
||||
from wov_app.db import Database
|
||||
from wov_app.scheduler import WorkflowScheduler, topological_sort
|
||||
from wov_sdk.models import (
|
||||
InvokeResponse,
|
||||
NodeManifest,
|
||||
WorkflowDefinition,
|
||||
WorkflowEdge,
|
||||
WorkflowNode,
|
||||
)
|
||||
|
||||
# 单体根目录：tests/ 的上一级。
|
||||
WORKSPACE = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def _register_echo() -> None:
|
||||
"""把内置 echo 节点注册到进程内注册表。"""
|
||||
from nodes.echo import invoke
|
||||
|
||||
registry.register(NodeManifest.load(str(WORKSPACE / "manifests" / "echo.json")), invoke)
|
||||
|
||||
|
||||
def _db(tmp_path) -> Database:
|
||||
"""在临时目录创建独立数据库。"""
|
||||
return Database(tmp_path / "wov.db")
|
||||
|
||||
|
||||
def _echo_definition() -> WorkflowDefinition:
|
||||
"""构造引用 Echo 节点的单步骤工作流定义。"""
|
||||
return WorkflowDefinition(
|
||||
name="echo-flow",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(
|
||||
id="step",
|
||||
node_type="echo",
|
||||
inputs={"file_uri": "input.video_uri"},
|
||||
)
|
||||
],
|
||||
edges=[],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
final_outputs={"result": "step.file_uri"},
|
||||
)
|
||||
|
||||
|
||||
def test_topological_sort() -> None:
|
||||
"""验证 DAG 排序保持依赖顺序，并拒绝环与未知边。"""
|
||||
definition = WorkflowDefinition(
|
||||
name="dag",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(id="a", node_type="x"),
|
||||
WorkflowNode(id="b", node_type="x"),
|
||||
WorkflowNode(id="c", node_type="x"),
|
||||
],
|
||||
edges=[
|
||||
WorkflowEdge(from_node="a", to_node="b"),
|
||||
WorkflowEdge(from_node="a", to_node="c"),
|
||||
],
|
||||
)
|
||||
order = topological_sort(definition)
|
||||
assert order.index("a") < order.index("b")
|
||||
assert order.index("a") < order.index("c")
|
||||
|
||||
cycle = WorkflowDefinition(
|
||||
name="cycle",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(id="a", node_type="x"),
|
||||
WorkflowNode(id="b", node_type="x"),
|
||||
],
|
||||
edges=[
|
||||
WorkflowEdge(from_node="a", to_node="b"),
|
||||
WorkflowEdge(from_node="b", to_node="a"),
|
||||
],
|
||||
)
|
||||
with pytest.raises(ValueError, match="cycle"):
|
||||
topological_sort(cycle)
|
||||
|
||||
with pytest.raises(ValueError, match="unknown edge"):
|
||||
topological_sort(
|
||||
WorkflowDefinition(
|
||||
name="bad",
|
||||
version=1,
|
||||
nodes=[WorkflowNode(id="a", node_type="x")],
|
||||
edges=[WorkflowEdge(from_node="a", to_node="missing")],
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_execute_echo_workflow(tmp_path) -> None:
|
||||
"""验证排队任务可被完整执行并登记全部产物。"""
|
||||
db = _db(tmp_path)
|
||||
input_file = tmp_path / "input.txt"
|
||||
input_file.write_text("hello scheduler", encoding="utf-8")
|
||||
_register_echo()
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, _echo_definition().to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_1",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(input_file),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_1")
|
||||
|
||||
run = db.get_run("run_1")
|
||||
assert run["status"] == "COMPLETED"
|
||||
artifacts = db.list_artifacts("run_1")
|
||||
assert {item["name"] for item in artifacts} == {"step.text", "step.file_uri", "result"}
|
||||
|
||||
|
||||
def test_execute_run_missing_workflow(tmp_path, monkeypatch) -> None:
|
||||
"""验证工作流记录缺失时任务被标记为失败。"""
|
||||
db = _db(tmp_path)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_missing",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr(db, "get_workflow", lambda workflow_id: None)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_missing")
|
||||
assert db.get_run("run_missing")["status"] == "FAILED"
|
||||
|
||||
|
||||
def test_execute_run_missing_version(tmp_path) -> None:
|
||||
"""验证版本记录缺失时任务被标记为失败。"""
|
||||
db = _db(tmp_path)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_version",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_version")
|
||||
assert db.get_run("run_version")["status"] == "FAILED"
|
||||
|
||||
|
||||
def test_execute_run_missing_node(tmp_path) -> None:
|
||||
"""验证未注册节点被调用时任务失败。"""
|
||||
db = _db(tmp_path)
|
||||
definition = WorkflowDefinition(
|
||||
name="bad",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(
|
||||
id="step",
|
||||
node_type="missing-node",
|
||||
inputs={"text": "input.video_uri"},
|
||||
)
|
||||
],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, definition.to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_node",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(tmp_path / "in.txt"),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_node")
|
||||
assert db.get_run("run_node")["status"] == "FAILED"
|
||||
|
||||
|
||||
def test_resolve_ref_and_mime(tmp_path) -> None:
|
||||
"""验证输入引用解析、MIME 推断与文件大小读取。"""
|
||||
db = _db(tmp_path)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
assert scheduler._resolve_ref("input.video", "in.mp4", {}) == "in.mp4"
|
||||
assert (
|
||||
scheduler._resolve_ref(
|
||||
"a.out", None, {"a": {"out": "result.txt"}}
|
||||
)
|
||||
== "result.txt"
|
||||
)
|
||||
assert scheduler._resolve_ref("a.out", None, {}) is None
|
||||
assert scheduler._resolve_ref("nodot", "in.mp4", {}) is None
|
||||
assert scheduler._mime_type("x.srt") == "application/x-subrip"
|
||||
assert scheduler._mime_type("x.ass") == "text/plain"
|
||||
assert scheduler._mime_type("x.wav") == "audio/wav"
|
||||
assert scheduler._mime_type("x.mp4") == "video/mp4"
|
||||
assert scheduler._mime_type("x.txt") == "text/plain"
|
||||
assert scheduler._mime_type("x.bin") == "application/octet-stream"
|
||||
existing = tmp_path / "existing.txt"
|
||||
existing.write_text("x", encoding="utf-8")
|
||||
assert scheduler._file_size(str(existing)) == 1
|
||||
missing = tmp_path / "missing.bin"
|
||||
assert scheduler._file_size(str(missing)) == 0
|
||||
|
||||
|
||||
def test_execute_unknown_or_non_queued_run(tmp_path) -> None:
|
||||
"""验证未知任务或非排队任务会被忽略。"""
|
||||
db = _db(tmp_path)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_done",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "COMPLETED",
|
||||
"progress": 1,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("missing")
|
||||
scheduler.execute_run("run_done")
|
||||
assert db.get_run("run_done")["status"] == "COMPLETED"
|
||||
|
||||
|
||||
def test_execute_missing_input(tmp_path) -> None:
|
||||
"""验证输入引用无法解析时任务失败。"""
|
||||
db = _db(tmp_path)
|
||||
definition = WorkflowDefinition(
|
||||
name="missing-input",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(
|
||||
id="step",
|
||||
node_type="echo",
|
||||
inputs={"text": "missing.output"},
|
||||
)
|
||||
],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, definition.to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_input",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(tmp_path / "in.txt"),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_input")
|
||||
assert db.get_run("run_input")["status"] == "FAILED"
|
||||
|
||||
|
||||
def test_execute_node_failed_response(tmp_path) -> None:
|
||||
"""验证节点返回 failed 时任务被标记为失败。"""
|
||||
db = _db(tmp_path)
|
||||
registry.register(
|
||||
NodeManifest(
|
||||
id="fail-node",
|
||||
name="Fail",
|
||||
version="1",
|
||||
capability="echo",
|
||||
repo_dir="nodes",
|
||||
command=["python", "-m", "fail"],
|
||||
),
|
||||
lambda request: InvokeResponse(status="failed", error="boom"),
|
||||
)
|
||||
definition = WorkflowDefinition(
|
||||
name="fail-flow",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(
|
||||
id="step",
|
||||
node_type="fail-node",
|
||||
inputs={"text": "input.video_uri"},
|
||||
)
|
||||
],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, definition.to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_fail_node",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(tmp_path / "in.txt"),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_fail_node")
|
||||
assert db.get_run("run_fail_node")["status"] == "FAILED"
|
||||
|
||||
|
||||
def test_scheduler_start_stop_loop(tmp_path) -> None:
|
||||
"""验证调度线程可重复启动并正常停止。"""
|
||||
db = _db(tmp_path)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage", interval_seconds=0.05)
|
||||
scheduler.start()
|
||||
try:
|
||||
scheduler.start()
|
||||
time.sleep(0.15)
|
||||
finally:
|
||||
scheduler.stop()
|
||||
assert scheduler._thread is None
|
||||
|
||||
|
||||
def test_scheduler_background_executes_queued_run(tmp_path) -> None:
|
||||
"""验证后台线程会自动执行排队中的任务。"""
|
||||
db = _db(tmp_path)
|
||||
input_file = tmp_path / "input.txt"
|
||||
input_file.write_text("background", encoding="utf-8")
|
||||
_register_echo()
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, _echo_definition().to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_bg",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(input_file),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage", interval_seconds=0.05)
|
||||
scheduler.start()
|
||||
try:
|
||||
deadline = time.monotonic() + 10
|
||||
while time.monotonic() < deadline:
|
||||
if db.get_run("run_bg")["status"] in {"COMPLETED", "FAILED"}:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
finally:
|
||||
scheduler.stop()
|
||||
assert db.get_run("run_bg")["status"] == "COMPLETED"
|
||||
|
||||
|
||||
def test_final_artifact_renamed_with_language_tag(tmp_path) -> None:
|
||||
"""验证最终产物按 上传文件名.语言.时间戳 重命名并登记新 URI。"""
|
||||
db = _db(tmp_path)
|
||||
input_file = tmp_path / "movie01.mp4"
|
||||
input_file.write_text("video", encoding="utf-8")
|
||||
_register_echo()
|
||||
definition = WorkflowDefinition(
|
||||
name="lang-flow",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(
|
||||
id="step",
|
||||
node_type="echo",
|
||||
params={"target_language": "zh-CN"},
|
||||
inputs={"file_uri": "input.video_uri"},
|
||||
)
|
||||
],
|
||||
edges=[],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
final_outputs={"cn_srt": "step.file_uri"},
|
||||
)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, definition.to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_1",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(input_file),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_1")
|
||||
artifacts = db.list_artifacts("run_1")
|
||||
final = next(item for item in artifacts if item["name"] == "cn_srt")
|
||||
filename = Path(final["uri"]).name
|
||||
# 命名规则：movie01.zh-CN.<14位时间戳>.txt
|
||||
assert filename.startswith("movie01.zh-CN.")
|
||||
assert filename.endswith(".txt")
|
||||
assert Path(final["uri"]).is_file()
|
||||
# 原始未重命名文件不应残留。
|
||||
step_artifacts = [item for item in artifacts if item["name"] == "step.file_uri"]
|
||||
assert not Path(step_artifacts[0]["uri"]).exists()
|
||||
|
||||
|
||||
def test_final_artifact_renamed_fallback_base_and_tag(tmp_path) -> None:
|
||||
"""验证无上传文件时基础名回退 subtitle，无语言参数时标识回退别名。"""
|
||||
db = _db(tmp_path)
|
||||
_register_echo()
|
||||
definition = WorkflowDefinition(
|
||||
name="fallback-flow",
|
||||
version=1,
|
||||
nodes=[
|
||||
# 空输入让 echo 走默认文本路径，避免 input_uri 缺失导致解析失败。
|
||||
WorkflowNode(id="step", node_type="echo", inputs={})
|
||||
],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
final_outputs={"result": "step.file_uri"},
|
||||
)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, definition.to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
# 故意不提供 input_uri，验证基础名回退。
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_1",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": None,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_1")
|
||||
final = next(item for item in db.list_artifacts("run_1") if item["name"] == "result")
|
||||
filename = Path(final["uri"]).name
|
||||
# 基础名回退 subtitle、标识回退别名 result。
|
||||
assert filename.startswith("subtitle.result.")
|
||||
assert filename.endswith(".txt")
|
||||
|
||||
|
||||
def test_execute_run_merges_param_overrides(tmp_path) -> None:
|
||||
"""验证调度执行时把 param_overrides 合并进节点参数。"""
|
||||
db = _db(tmp_path)
|
||||
input_file = tmp_path / "input.txt"
|
||||
input_file.write_text("x", encoding="utf-8")
|
||||
captured = {}
|
||||
|
||||
def recording_handler(request):
|
||||
captured["params"] = dict(request.params)
|
||||
return InvokeResponse(status="completed", outputs={"text": "ok"})
|
||||
|
||||
registry.register(
|
||||
NodeManifest(
|
||||
id="record-node",
|
||||
name="Record",
|
||||
version="1",
|
||||
capability="echo",
|
||||
repo_dir="nodes",
|
||||
command=["python", "-m", "record"],
|
||||
),
|
||||
recording_handler,
|
||||
)
|
||||
definition = WorkflowDefinition(
|
||||
name="ov-flow",
|
||||
version=1,
|
||||
nodes=[WorkflowNode(id="step", node_type="record-node", params={"base": 1})],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
)
|
||||
db.upsert_workflow({"id": "flow", "name": "F", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, definition.to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_ov",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"param_overrides": {"step": {"crop": [0, 0.5, 1, 0.5]}},
|
||||
"input_uri": str(input_file),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_ov")
|
||||
assert captured["params"] == {"base": 1, "crop": [0, 0.5, 1, 0.5]}
|
||||
|
||||
|
||||
def _two_node_definition() -> WorkflowDefinition:
|
||||
"""构造 a→b 两节点工作流：b 引用 a 的输出。"""
|
||||
return WorkflowDefinition(
|
||||
name="two-flow",
|
||||
version=1,
|
||||
nodes=[
|
||||
WorkflowNode(id="a", node_type="x", inputs={"video_uri": "input.video_uri"}),
|
||||
WorkflowNode(id="b", node_type="x", inputs={"data_uri": "a.data_uri"}),
|
||||
],
|
||||
edges=[WorkflowEdge(from_node="a", to_node="b")],
|
||||
entry_inputs={"video_uri": "file"},
|
||||
final_outputs={"result": "b.data_uri"},
|
||||
)
|
||||
|
||||
|
||||
def test_execute_pause_between_nodes_and_resume(tmp_path, monkeypatch) -> None:
|
||||
"""验证运行中暂停：节点边界停下保持 PAUSED；续跑时跳过已完成节点。"""
|
||||
db = _db(tmp_path)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, _two_node_definition().to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_pause",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(tmp_path / "in.txt"),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
data_file = tmp_path / "data.bin"
|
||||
data_file.write_bytes(b"x")
|
||||
|
||||
calls: list[str] = []
|
||||
|
||||
def fake_invoke(node_type, request):
|
||||
# registry.invoke 首参是 node_type；用产物目录名（steps/<node_id>）识别节点。
|
||||
node_id = Path(request.output_dir).name
|
||||
calls.append(node_id)
|
||||
# 第一个节点完成后立刻暂停任务，模拟用户在运行中点暂停。
|
||||
if node_id == "a":
|
||||
db.pause_run("run_pause", now)
|
||||
return InvokeResponse(status="completed", outputs={"data_uri": str(data_file)})
|
||||
|
||||
monkeypatch.setattr(registry, "invoke", fake_invoke)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_pause")
|
||||
assert db.get_run("run_pause")["status"] == "PAUSED"
|
||||
assert calls == ["a"] # 节点 b 未执行。
|
||||
|
||||
# 继续：恢复排队并再次执行，节点 a 已产出结果应被跳过，只执行 b。
|
||||
db.resume_run("run_pause", now)
|
||||
scheduler.execute_run("run_pause")
|
||||
assert db.get_run("run_pause")["status"] == "COMPLETED"
|
||||
assert calls == ["a", "b"]
|
||||
artifacts = db.list_artifacts("run_pause")
|
||||
assert {item["name"] for item in artifacts} == {"a.data_uri", "b.data_uri", "result"}
|
||||
|
||||
|
||||
def test_execute_paused_run_not_run(tmp_path, monkeypatch) -> None:
|
||||
"""验证非可执行状态（如 RUNNING 之外的值）的任务不会被执行。"""
|
||||
db = _db(tmp_path)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, _two_node_definition().to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_done",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "COMPLETED",
|
||||
"progress": 1.0,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
called = []
|
||||
|
||||
def fake_invoke(node_id, request):
|
||||
called.append(node_id)
|
||||
return InvokeResponse(status="completed", outputs={})
|
||||
|
||||
monkeypatch.setattr(registry, "invoke", fake_invoke)
|
||||
WorkflowScheduler(db, tmp_path / "storage").execute_run("run_done")
|
||||
assert called == []
|
||||
|
||||
|
||||
def test_execute_pause_after_last_node_keeps_paused(tmp_path, monkeypatch) -> None:
|
||||
"""验证全部节点完成但运行中被暂停时保持 PAUSED；续跑补做收尾后完成。"""
|
||||
db = _db(tmp_path)
|
||||
db.upsert_workflow({"id": "flow", "name": "Flow", "published": 1, "latest_version": 1})
|
||||
db.create_workflow_version("flow", 1, _two_node_definition().to_dict())
|
||||
now = "2026-01-01T00:00:00+00:00"
|
||||
db.create_run(
|
||||
{
|
||||
"id": "run_tail",
|
||||
"workflow_id": "flow",
|
||||
"workflow_version": 1,
|
||||
"status": "QUEUED",
|
||||
"progress": 0,
|
||||
"input_uri": str(tmp_path / "in.txt"),
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}
|
||||
)
|
||||
data_file = tmp_path / "data.bin"
|
||||
data_file.write_bytes(b"x")
|
||||
calls: list[str] = []
|
||||
|
||||
def fake_invoke(node_type, request):
|
||||
node_id = Path(request.output_dir).name
|
||||
calls.append(node_id)
|
||||
if node_id == "b": # 最后一个节点执行时暂停。
|
||||
db.pause_run("run_tail", now)
|
||||
return InvokeResponse(status="completed", outputs={"data_uri": str(data_file)})
|
||||
|
||||
monkeypatch.setattr(registry, "invoke", fake_invoke)
|
||||
scheduler = WorkflowScheduler(db, tmp_path / "storage")
|
||||
scheduler.execute_run("run_tail")
|
||||
# 全部节点已执行，但收尾前被暂停 → 保持 PAUSED 而不是 COMPLETED。
|
||||
assert db.get_run("run_tail")["status"] == "PAUSED"
|
||||
assert calls == ["a", "b"]
|
||||
|
||||
# 续跑：节点产物齐备全部跳过，补做收尾后完成。
|
||||
db.resume_run("run_tail", now)
|
||||
scheduler.execute_run("run_tail")
|
||||
assert db.get_run("run_tail")["status"] == "COMPLETED"
|
||||
assert calls == ["a", "b"]
|
||||
@@ -0,0 +1,108 @@
|
||||
"""种子数据测试。
|
||||
|
||||
验证从 workflows/*.json 数据文件加载默认工作流、幂等性，以及
|
||||
开启种子与调度器后的应用生命周期。工作流定义来自数据文件而非代码。
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from wov_app.db import Database
|
||||
from wov_app.main import app
|
||||
from wov_app.seed import seed_default_workflows
|
||||
|
||||
# 单体根目录：tests/ 的上一级。
|
||||
WORKSPACE = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def test_seed_default_workflows_idempotent(tmp_path) -> None:
|
||||
"""验证从数据文件加载 demo/zh-direct 两个工作流且重复调用幂等。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
created = seed_default_workflows(db)
|
||||
assert created == 3
|
||||
assert db.get_workflow("demo") is not None
|
||||
assert db.get_workflow("zh-direct") is not None
|
||||
|
||||
# demo：asr 显式声明 model_path 与长音频参数，模型选择完全数据化。
|
||||
demo = db.get_latest_workflow_version("demo")["definition"]
|
||||
assert demo["name"] == "视频字幕生成"
|
||||
demo_asr = next(node for node in demo["nodes"] if node["id"] == "asr")
|
||||
assert demo_asr["params"]["model_path"] == "faster-whisper-large-v3"
|
||||
assert demo_asr["params"]["condition_on_previous_text"] is False
|
||||
|
||||
# zh-direct：使用中文直出模型并开启翻译任务。
|
||||
zh = db.get_latest_workflow_version("zh-direct")["definition"]
|
||||
assert zh["name"] == "中文直出字幕"
|
||||
zh_asr = next(node for node in zh["nodes"] if node["id"] == "asr")
|
||||
assert zh_asr["params"]["model_path"] == "whisper-large-v2-translate-zh-v0.2-st-ct2"
|
||||
assert zh_asr["params"]["task"] == "translate"
|
||||
|
||||
# 再次调用不重复创建。
|
||||
assert seed_default_workflows(db) == 0
|
||||
assert len(db.list_workflow_versions("demo")) == 1
|
||||
|
||||
|
||||
def test_seed_custom_dir_and_empty(tmp_path) -> None:
|
||||
"""验证自定义数据目录的加载与空目录返回 0。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
custom = tmp_path / "workflows"
|
||||
custom.mkdir()
|
||||
(custom / "a.json").write_text(
|
||||
"""
|
||||
{
|
||||
"id": "flow-a",
|
||||
"name": "Flow A",
|
||||
"description": "custom",
|
||||
"version": 1,
|
||||
"definition": {
|
||||
"name": "Flow A",
|
||||
"version": 1,
|
||||
"nodes": [{"id": "step", "node_type": "echo"}],
|
||||
"edges": [],
|
||||
"entry_inputs": {},
|
||||
"final_outputs": {}
|
||||
}
|
||||
}
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert seed_default_workflows(db, custom) == 1
|
||||
assert db.get_workflow("flow-a") is not None
|
||||
# 空目录返回 0。
|
||||
empty = tmp_path / "empty"
|
||||
empty.mkdir()
|
||||
assert seed_default_workflows(db, empty) == 0
|
||||
# 已存在的工作流被跳过。
|
||||
assert seed_default_workflows(db, custom) == 0
|
||||
|
||||
|
||||
def test_lifespan_with_seed_and_scheduler(monkeypatch) -> None:
|
||||
"""验证启用自动种子与调度器后应用正常启动，demo 与中文直出应用均可见。"""
|
||||
monkeypatch.setenv("WOV_AUTO_SEED", "1")
|
||||
monkeypatch.setenv("WOV_SCHEDULER_ENABLED", "1")
|
||||
with TestClient(app) as client:
|
||||
apps = client.get("/api/apps")
|
||||
assert apps.status_code == 200
|
||||
assert any(item["id"] == "demo" for item in apps.json())
|
||||
assert any(item["id"] == "zh-direct" for item in apps.json())
|
||||
|
||||
|
||||
def test_seed_workflows_have_chunk_seconds(tmp_path) -> None:
|
||||
"""验证内置工作流的 asr 节点均显式声明分块参数。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
seed_default_workflows(db)
|
||||
for workflow_id in ("demo", "zh-direct"):
|
||||
definition = db.get_latest_workflow_version(workflow_id)["definition"]
|
||||
asr = next(node for node in definition["nodes"] if node["id"] == "asr")
|
||||
assert asr["params"]["chunk_seconds"] == 60
|
||||
|
||||
|
||||
def test_seed_workflows_vad_filter_off(tmp_path) -> None:
|
||||
"""验证内置工作流 asr 显式开启 VAD。"""
|
||||
db = Database(tmp_path / "wov.db")
|
||||
seed_default_workflows(db)
|
||||
for workflow_id in ("demo", "zh-direct"):
|
||||
definition = db.get_latest_workflow_version(workflow_id)["definition"]
|
||||
asr = next(node for node in definition["nodes"] if node["id"] == "asr")
|
||||
assert asr["params"]["vad_filter"] is True
|
||||
@@ -0,0 +1,34 @@
|
||||
"""uvicorn 冒烟测试。
|
||||
|
||||
用真实套接字启动 uvicorn 服务并请求 /health，验证应用能脱离 TestClient
|
||||
在实际 Web 服务环境中正常工作。
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
from uvicorn import Config, Server
|
||||
|
||||
from wov_app.main import app
|
||||
|
||||
|
||||
def test_uvicorn_serves_app_over_real_socket() -> None:
|
||||
"""验证 uvicorn 监听真实端口后健康检查可用。"""
|
||||
config = Config(app=app, host="127.0.0.1", port=0, log_level="error")
|
||||
server = Server(config)
|
||||
thread = threading.Thread(target=server.run, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
deadline = time.monotonic() + 10
|
||||
while not server.started and time.monotonic() < deadline:
|
||||
time.sleep(0.05)
|
||||
assert server.started
|
||||
|
||||
port = server.servers[0].sockets[0].getsockname()[1]
|
||||
with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=5) as response:
|
||||
assert response.status == 200
|
||||
assert b'"wov-api"' in response.read()
|
||||
finally:
|
||||
server.should_exit = True
|
||||
thread.join(timeout=10)
|
||||
@@ -0,0 +1,117 @@
|
||||
"""工作流管理 API 测试。
|
||||
|
||||
覆盖工作流的创建、查询、校验、发布、版本列表与删除等管理接口。
|
||||
"""
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from wov_app.main import app
|
||||
|
||||
|
||||
def definition() -> dict:
|
||||
"""构造一个引用 Echo 节点的合法工作流定义。"""
|
||||
return {
|
||||
"name": "echo-flow",
|
||||
"version": 1,
|
||||
"nodes": [
|
||||
{
|
||||
"id": "step",
|
||||
"node_type": "echo",
|
||||
"inputs": {"file_uri": "input.video_uri"},
|
||||
}
|
||||
],
|
||||
"edges": [],
|
||||
"entry_inputs": {"video_uri": "file"},
|
||||
"final_outputs": {"result": "step.file_uri"},
|
||||
}
|
||||
|
||||
|
||||
def test_workflow_crud_and_publish() -> None:
|
||||
"""验证工作流 CRUD、校验、发布与版本列表的完整流程。"""
|
||||
with TestClient(app) as client:
|
||||
created = client.post(
|
||||
"/api/admin/workflows",
|
||||
json={
|
||||
"id": "echo-flow",
|
||||
"name": "Echo Flow",
|
||||
"description": "demo",
|
||||
"definition": definition(),
|
||||
},
|
||||
)
|
||||
assert created.status_code == 200
|
||||
assert created.json()["id"] == "echo-flow"
|
||||
|
||||
assert client.get("/api/admin/workflows").status_code == 200
|
||||
assert client.get("/api/admin/workflows/echo-flow").status_code == 200
|
||||
assert client.get("/api/admin/workflows/missing").status_code == 404
|
||||
|
||||
validated = client.post(
|
||||
"/api/admin/workflows/echo-flow/validate",
|
||||
json=definition(),
|
||||
)
|
||||
assert validated.status_code == 200
|
||||
assert validated.json()["valid"] is True
|
||||
assert client.post(
|
||||
"/api/admin/workflows/missing/validate",
|
||||
json=definition(),
|
||||
).status_code == 404
|
||||
|
||||
published = client.post("/api/admin/workflows/echo-flow/publish")
|
||||
assert published.status_code == 200
|
||||
assert published.json()["published"] == "echo-flow"
|
||||
assert client.post("/api/admin/workflows/missing/publish").status_code == 404
|
||||
|
||||
versions = client.get("/api/admin/workflows/echo-flow/versions")
|
||||
assert versions.status_code == 200
|
||||
assert len(versions.json()) == 1
|
||||
assert client.get("/api/admin/workflows/missing/versions").status_code == 404
|
||||
|
||||
assert client.delete("/api/admin/workflows/echo-flow").status_code == 200
|
||||
assert client.delete("/api/admin/workflows/echo-flow").status_code == 404
|
||||
|
||||
|
||||
def test_workflow_slug_without_id() -> None:
|
||||
"""验证未提供 ID 时后端会从名称生成 slug。"""
|
||||
with TestClient(app) as client:
|
||||
created = client.post(
|
||||
"/api/admin/workflows",
|
||||
json={
|
||||
"name": "Echo Flow",
|
||||
"definition": definition(),
|
||||
},
|
||||
)
|
||||
assert created.status_code == 200
|
||||
assert created.json()["id"] == "echo-flow"
|
||||
|
||||
|
||||
def test_workflow_validation_error() -> None:
|
||||
"""验证重复节点 ID 的 DAG 会被拒绝。"""
|
||||
with TestClient(app) as client:
|
||||
response = client.post(
|
||||
"/api/admin/workflows",
|
||||
json={
|
||||
"id": "bad",
|
||||
"name": "Bad",
|
||||
"definition": {
|
||||
"name": "Bad",
|
||||
"version": 1,
|
||||
"nodes": [
|
||||
{"id": "a", "node_type": "x"},
|
||||
{"id": "a", "node_type": "y"},
|
||||
],
|
||||
"edges": [],
|
||||
},
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
def test_publish_workflow_without_version() -> None:
|
||||
"""验证没有版本记录的工作流不能发布。"""
|
||||
with TestClient(app) as client:
|
||||
db = app.state.db
|
||||
db.upsert_workflow(
|
||||
{"id": "empty", "name": "Empty", "published": 0, "latest_version": 0}
|
||||
)
|
||||
response = client.post("/api/admin/workflows/empty/publish")
|
||||
assert response.status_code == 422
|
||||
Executable
+46
@@ -0,0 +1,46 @@
|
||||
<!doctype html>
|
||||
<!-- VRSub 管理后台：工作流管理（创建/发布/删除）。节点注册与实例管理已随单体化移除。 -->
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>VRSub 管理后台</title>
|
||||
<link rel="stylesheet" href="/assets/styles.css?v=2" />
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<a class="brand" href="/">VRSub</a>
|
||||
<nav>
|
||||
<a href="/">应用中心</a>
|
||||
<a href="/tasks.html">任务管理</a>
|
||||
<a href="/admin.html">管理后台</a>
|
||||
<a href="/workflow.html">工作流</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<main class="container">
|
||||
<h1>管理后台</h1>
|
||||
|
||||
<!-- 已发布工作流：展示发布状态，支持发布与删除操作。 -->
|
||||
<section class="panel">
|
||||
<h2>已发布工作流</h2>
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>ID</th>
|
||||
<th>名称</th>
|
||||
<th>版本</th>
|
||||
<th>状态</th>
|
||||
<th>操作</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="workflowList"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<script src="/assets/app.js?v=2"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,647 @@
|
||||
// VRSub 静态前端公共脚本：所有页面共用的 API 封装、渲染函数与事件绑定。
|
||||
// 首页只负责发起任务；任务管理页展示全部任务的进度、产物下载与失败重试。
|
||||
|
||||
// 演示"视频字幕生成"工作流的 DAG 定义，预填在工作流编排页。
|
||||
const DEMO_WORKFLOW = {
|
||||
name: "视频字幕生成",
|
||||
version: 1,
|
||||
nodes: [
|
||||
{
|
||||
id: "extract",
|
||||
node_type: "ffmpeg-extract",
|
||||
params: { sample_rate: 16000, channels: 1 },
|
||||
inputs: { video_uri: "input.video_uri" },
|
||||
},
|
||||
{
|
||||
id: "asr",
|
||||
node_type: "faster-whisper",
|
||||
params: { language: "ja" },
|
||||
inputs: { audio_uri: "extract.audio_uri" },
|
||||
},
|
||||
{
|
||||
id: "translate",
|
||||
node_type: "llm-translate",
|
||||
params: { target_language: "zh-CN" },
|
||||
inputs: { srt_uri: "asr.srt_uri" },
|
||||
},
|
||||
{
|
||||
id: "ass",
|
||||
node_type: "srt-to-dual-eye-ass",
|
||||
params: { resolution: "3840x1920" },
|
||||
inputs: { cn_srt_uri: "translate.cn_srt_uri" },
|
||||
},
|
||||
],
|
||||
edges: [
|
||||
{ from: "extract", to: "asr" },
|
||||
{ from: "asr", to: "translate" },
|
||||
{ from: "translate", to: "ass" },
|
||||
],
|
||||
entry_inputs: { video_uri: "file" },
|
||||
final_outputs: {
|
||||
cn_srt: "translate.cn_srt_uri",
|
||||
ass: "ass.ass_uri",
|
||||
},
|
||||
};
|
||||
|
||||
// 统一封装 fetch：自动携带 JSON 头、解析响应并在失败时抛出可读错误。
|
||||
async function api(path, options = {}) {
|
||||
const response = await fetch(path, {
|
||||
headers: { "Content-Type": "application/json", ...(options.headers || {}) },
|
||||
...options,
|
||||
});
|
||||
const data = await response.json().catch(() => null);
|
||||
if (!response.ok) {
|
||||
// FastAPI 的校验错误 detail 可能是数组，统一序列化为字符串展示。
|
||||
const detail = data && data.detail ? JSON.stringify(data.detail) : response.statusText;
|
||||
throw new Error(`${response.status} ${detail}`);
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
// 转义用户可控文本，防止 XSS 注入到表格或状态 HTML 中。
|
||||
function escapeHtml(value) {
|
||||
return String(value)
|
||||
.replaceAll("&", "&")
|
||||
.replaceAll("<", "<")
|
||||
.replaceAll(">", ">")
|
||||
.replaceAll('"', """)
|
||||
.replaceAll("'", "'");
|
||||
}
|
||||
|
||||
// 根据状态生成带语义颜色的徽章 HTML。
|
||||
function badge(status) {
|
||||
// 统一转小写比较，兼容后端返回的不同大小写。
|
||||
const value = String(status).toLowerCase();
|
||||
const className =
|
||||
value === "completed" || value === "ok"
|
||||
? "ok"
|
||||
: value === "error" || value === "failed"
|
||||
? "error"
|
||||
: "warn";
|
||||
return `<span class="badge ${className}">${escapeHtml(status)}</span>`;
|
||||
}
|
||||
|
||||
// 把 ISO 时间格式化为本地时间；非法值原样返回。
|
||||
function formatTime(value) {
|
||||
if (!value) return "";
|
||||
const date = new Date(value);
|
||||
return Number.isNaN(date.getTime()) ? value : date.toLocaleString();
|
||||
}
|
||||
|
||||
// 计算距给定时间的流逝时长，用于任务页展示已运行时间。
|
||||
function formatElapsed(value) {
|
||||
if (!value) return "";
|
||||
const seconds = Math.floor((Date.now() - new Date(value).getTime()) / 1000);
|
||||
if (Number.isNaN(seconds) || seconds < 0) return "";
|
||||
if (seconds < 60) return `${seconds} 秒`;
|
||||
return `${Math.floor(seconds / 60)} 分钟`;
|
||||
}
|
||||
|
||||
// 健康检查失败时在页面顶部展示后端不可用提示。
|
||||
async function loadHealth() {
|
||||
try {
|
||||
await api("/health");
|
||||
} catch (error) {
|
||||
const banner = document.createElement("div");
|
||||
banner.className = "result";
|
||||
banner.textContent = `后端不可用：${error.message}`;
|
||||
document.body.prepend(banner);
|
||||
}
|
||||
}
|
||||
|
||||
// 加载工作流列表并渲染发布状态与操作按钮。
|
||||
async function loadWorkflows() {
|
||||
const workflows = await api("/api/admin/workflows");
|
||||
const tbody = document.getElementById("workflowList");
|
||||
if (!tbody) return;
|
||||
tbody.innerHTML = workflows.length
|
||||
? workflows
|
||||
.map(
|
||||
(workflow) => `
|
||||
<tr>
|
||||
<td>${escapeHtml(workflow.id)}</td>
|
||||
<td>${escapeHtml(workflow.name)}</td>
|
||||
<td>${escapeHtml(workflow.latest_version)}</td>
|
||||
<td>${workflow.published ? badge("ok") : badge("draft")}</td>
|
||||
<td>
|
||||
<button class="danger" data-delete-workflow="${escapeHtml(workflow.id)}">删除</button>
|
||||
${workflow.published ? "" : `<button data-publish-workflow="${escapeHtml(workflow.id)}">发布</button>`}
|
||||
</td>
|
||||
</tr>`,
|
||||
)
|
||||
.join("")
|
||||
: '<tr><td colspan="5">暂无工作流</td></tr>';
|
||||
}
|
||||
|
||||
// 加载已发布的工作流（应用），填充首页的下拉选择框。
|
||||
// 已发布工作流缓存（id → {name, definition}），供按需切换 OCR 面板。
|
||||
let workflowApps = [];
|
||||
// 需要 crop 的节点 ID（如 frame-extract）与用户框选的 crop 值。
|
||||
let cropNodeId = null;
|
||||
let selectedCrop = null;
|
||||
// 框选模式：开启后才让画布接收鼠标事件，平时不拦截视频控件。
|
||||
let drawMode = false;
|
||||
// 当前选择的视频文件：change 时保存，提交时使用（避免被重置清空 input 丢失）。
|
||||
let selectedVideoFile = null;
|
||||
|
||||
async function loadWorkflowOptions() {
|
||||
const select = document.getElementById("workflowSelect");
|
||||
if (!select) return;
|
||||
workflowApps = await api("/api/apps");
|
||||
select.innerHTML = workflowApps.length
|
||||
? workflowApps
|
||||
.map(
|
||||
(app) =>
|
||||
`<option value="${escapeHtml(app.id)}">${escapeHtml(app.name)}</option>`,
|
||||
)
|
||||
.join("")
|
||||
: '<option value="">暂无可用工作流</option>';
|
||||
// 切换工作流时按需展示框选面板。
|
||||
select.addEventListener("change", onWorkflowChange);
|
||||
await onWorkflowChange();
|
||||
}
|
||||
|
||||
// 切换工作流：若 DAG 中存在带 crop 参数的节点（字幕 OCR 流程），
|
||||
// 切换到框选面板；否则使用标准上传。
|
||||
async function onWorkflowChange() {
|
||||
const select = document.getElementById("workflowSelect");
|
||||
const ocrCard = document.getElementById("ocrCard");
|
||||
const standardCard = document.getElementById("standardCard");
|
||||
if (!select || !ocrCard || !standardCard) return;
|
||||
cropNodeId = null;
|
||||
selectedCrop = null;
|
||||
const app = workflowApps.find((item) => item.id === select.value);
|
||||
// 数据驱动判断：节点 params 中声明了 crop 即需要框选。
|
||||
const cropNode = (app?.definition?.nodes || []).find(
|
||||
(node) => node.params && node.params.crop !== undefined,
|
||||
);
|
||||
const needsCrop = Boolean(cropNode);
|
||||
cropNodeId = needsCrop ? cropNode.id : null;
|
||||
ocrCard.hidden = !needsCrop;
|
||||
standardCard.hidden = needsCrop;
|
||||
resetOcrPanel();
|
||||
if (needsCrop) {
|
||||
document.getElementById("ocrProgress").textContent = "请选择视频并框选字幕区域";
|
||||
}
|
||||
}
|
||||
|
||||
// 重置 OCR 面板：清除视频、画布与框选状态。
|
||||
function resetOcrPanel() {
|
||||
const video = document.getElementById("ocrVideo");
|
||||
const canvas = document.getElementById("ocrCanvas");
|
||||
const fileInput = document.getElementById("ocrVideoFile");
|
||||
const cropValue = document.getElementById("cropValue");
|
||||
const submit = document.getElementById("ocrSubmit");
|
||||
selectedCrop = null;
|
||||
if (video) video.removeAttribute("src");
|
||||
if (canvas) {
|
||||
canvas.width = 0;
|
||||
canvas.height = 0;
|
||||
}
|
||||
if (fileInput) fileInput.value = "";
|
||||
if (cropValue) cropValue.value = "";
|
||||
if (submit) submit.disabled = true;
|
||||
selectedVideoFile = null;
|
||||
exitDrawMode();
|
||||
}
|
||||
|
||||
// 用 crop.js 把画布上的框选矩形归一化为 crop 比例并展示。
|
||||
function applyCropRect(rect) {
|
||||
const video = document.getElementById("ocrVideo");
|
||||
const box = document.getElementById("videoBox");
|
||||
if (!video.videoWidth || !box) return;
|
||||
const crop = rectToCrop(
|
||||
rect,
|
||||
video.videoWidth,
|
||||
video.videoHeight,
|
||||
box.clientWidth,
|
||||
box.clientHeight,
|
||||
);
|
||||
selectedCrop = crop;
|
||||
document.getElementById("cropValue").value = crop.join(", ");
|
||||
document.getElementById("ocrSubmit").disabled = false;
|
||||
document.getElementById("ocrProgress").textContent =
|
||||
`已框选 crop=[${crop.join(", ")}]，可提交任务`;
|
||||
}
|
||||
|
||||
// 进入框选模式：暂停视频、隐藏原生控件、启用画布绘制。
|
||||
function enterDrawMode() {
|
||||
const video = document.getElementById("ocrVideo");
|
||||
const canvas = document.getElementById("ocrCanvas");
|
||||
const toggle = document.getElementById("ocrDrawMode");
|
||||
if (!video.videoWidth) {
|
||||
document.getElementById("ocrProgress").textContent = "请先选择视频并定位到有字幕的画面";
|
||||
return;
|
||||
}
|
||||
drawMode = true;
|
||||
video.pause();
|
||||
video.removeAttribute("controls"); // 隐藏进度条，避免遮挡框选操作。
|
||||
canvas.classList.add("drawable");
|
||||
if (toggle) toggle.textContent = "退出框选模式";
|
||||
document.getElementById("ocrProgress").textContent = "请拖动框选字幕区域";
|
||||
}
|
||||
|
||||
// 退出框选模式：恢复原生控件，画布不再拦截指针。
|
||||
function exitDrawMode() {
|
||||
const video = document.getElementById("ocrVideo");
|
||||
const canvas = document.getElementById("ocrCanvas");
|
||||
const toggle = document.getElementById("ocrDrawMode");
|
||||
drawMode = false;
|
||||
if (video) video.setAttribute("controls", "");
|
||||
if (canvas) canvas.classList.remove("drawable");
|
||||
if (toggle) toggle.textContent = "进入框选模式";
|
||||
}
|
||||
|
||||
// 初始化 OCR 面板：视频预览 + 画布拖动框选。
|
||||
function setupOcrPanel() {
|
||||
const fileInput = document.getElementById("ocrVideoFile");
|
||||
const video = document.getElementById("ocrVideo");
|
||||
const canvas = document.getElementById("ocrCanvas");
|
||||
const resetButton = document.getElementById("ocrReset");
|
||||
if (!fileInput || !video || !canvas) return;
|
||||
|
||||
// 选择视频后显示预览并设置画布尺寸与坐标换算。
|
||||
// 注意顺序：必须先 reset（会清空旧 src），再设置新 src，否则被清掉。
|
||||
fileInput.addEventListener("change", () => {
|
||||
const file = fileInput.files[0];
|
||||
if (!file) return;
|
||||
resetOcrPanel();
|
||||
// 重置会清空 input，这里把文件引用保存下来供提交使用。
|
||||
selectedVideoFile = file;
|
||||
video.src = URL.createObjectURL(file);
|
||||
video.load();
|
||||
video.onloadedmetadata = () => {
|
||||
const box = document.getElementById("videoBox");
|
||||
canvas.width = box.clientWidth;
|
||||
canvas.height = box.clientHeight;
|
||||
document.getElementById("ocrProgress").textContent = "请在预览中拖动框选字幕区域";
|
||||
};
|
||||
});
|
||||
|
||||
// 框选模式开关：进入/退出。
|
||||
const drawToggle = document.getElementById("ocrDrawMode");
|
||||
if (drawToggle) {
|
||||
drawToggle.addEventListener("click", () => {
|
||||
if (drawMode) {
|
||||
exitDrawMode();
|
||||
} else {
|
||||
enterDrawMode();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// 拖动绘制框选矩形：mousedown 起点 → mousemove 更新 → mouseup 生成 crop。
|
||||
let startX = 0;
|
||||
let startY = 0;
|
||||
let drawing = false;
|
||||
canvas.addEventListener("mousedown", (event) => {
|
||||
if (!drawMode) return;
|
||||
const rect = canvas.getBoundingClientRect();
|
||||
startX = event.clientX - rect.left;
|
||||
startY = event.clientY - rect.top;
|
||||
drawing = true;
|
||||
});
|
||||
canvas.addEventListener("mousemove", (event) => {
|
||||
if (!drawing || !drawMode) return;
|
||||
const rect = canvas.getBoundingClientRect();
|
||||
const x = event.clientX - rect.left;
|
||||
const y = event.clientY - rect.top;
|
||||
const ctx = canvas.getContext("2d");
|
||||
ctx.clearRect(0, 0, canvas.width, canvas.height);
|
||||
ctx.strokeStyle = "#ff5252";
|
||||
ctx.lineWidth = 2;
|
||||
ctx.strokeRect(Math.min(startX, x), Math.min(startY, y), Math.abs(x - startX), Math.abs(y - startY));
|
||||
});
|
||||
canvas.addEventListener("mouseup", (event) => {
|
||||
if (!drawing || !drawMode) return;
|
||||
drawing = false;
|
||||
const rect = canvas.getBoundingClientRect();
|
||||
const x = event.clientX - rect.left;
|
||||
const y = event.clientY - rect.top;
|
||||
const w = Math.abs(x - startX);
|
||||
const h = Math.abs(y - startY);
|
||||
if (w < 5 || h < 5) return;
|
||||
applyCropRect({
|
||||
x: Math.min(startX, x),
|
||||
y: Math.min(startY, y),
|
||||
w,
|
||||
h,
|
||||
});
|
||||
});
|
||||
|
||||
// 清除框选：清空画布与 crop，禁用提交。
|
||||
resetButton.addEventListener("click", () => {
|
||||
const ctx = canvas.getContext("2d");
|
||||
ctx.clearRect(0, 0, canvas.width, canvas.height);
|
||||
selectedCrop = null;
|
||||
document.getElementById("cropValue").value = "";
|
||||
document.getElementById("ocrSubmit").disabled = true;
|
||||
document.getElementById("ocrProgress").textContent = "请在预览中拖动框选字幕区域";
|
||||
});
|
||||
}
|
||||
|
||||
// 提交字幕 OCR 任务：携带 crop 覆盖参数，框选完成才可用。
|
||||
async function submitOcr() {
|
||||
const fileInput = document.getElementById("ocrVideoFile");
|
||||
const progress = document.getElementById("ocrProgress");
|
||||
if (!selectedCrop) {
|
||||
progress.textContent = "请先框选字幕区域";
|
||||
return;
|
||||
}
|
||||
const file = selectedVideoFile;
|
||||
if (!file) {
|
||||
progress.textContent = "请先选择视频文件";
|
||||
return;
|
||||
}
|
||||
const workflowId = document.getElementById("workflowSelect").value;
|
||||
const form = new FormData();
|
||||
form.append("file", file);
|
||||
// 把框选的 crop 传给需要它的节点（如 frame-extract）。
|
||||
form.append("params", JSON.stringify({ [cropNodeId]: { crop: selectedCrop } }));
|
||||
progress.textContent = "上传中...";
|
||||
try {
|
||||
const response = await fetch(`/api/apps/${encodeURIComponent(workflowId)}/runs`, {
|
||||
method: "POST",
|
||||
body: form,
|
||||
});
|
||||
const data = await response.json().catch(() => null);
|
||||
if (!response.ok) {
|
||||
throw new Error(data && data.detail ? JSON.stringify(data.detail) : response.statusText);
|
||||
}
|
||||
progress.textContent = `任务 ${data.id} 已创建，正在跳转到任务管理...`;
|
||||
window.location.href = `/tasks.html?run=${encodeURIComponent(data.id)}`;
|
||||
} catch (error) {
|
||||
progress.textContent = `创建失败：${error.message}`;
|
||||
}
|
||||
}
|
||||
|
||||
// 渲染单个任务的进度条 HTML；失败任务用红色填充。
|
||||
function progressBar(percent, failed) {
|
||||
return `<div class="progress"><div class="progress-bar ${failed ? "error" : ""}" style="width:${Math.min(100, percent)}%"></div></div>`;
|
||||
}
|
||||
|
||||
// 为已完成任务生成最终产物下载链接（cn_srt / ass）。
|
||||
function artifactLinks(runId, status) {
|
||||
if (status !== "COMPLETED") return "-";
|
||||
return `
|
||||
<div class="downloads-inline">
|
||||
<a class="download-link" href="/api/runs/${encodeURIComponent(runId)}/artifacts/cn_srt">中文 SRT</a>
|
||||
<a class="download-link" href="/api/runs/${encodeURIComponent(runId)}/artifacts/ass">VR ASS</a>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// 加载最近任务并渲染任务表格：状态、当前节点、进度条、产物与重试。
|
||||
async function loadRuns() {
|
||||
const tbody = document.getElementById("runList");
|
||||
if (!tbody) return;
|
||||
const runs = await api("/api/runs");
|
||||
tbody.innerHTML = runs.length
|
||||
? runs
|
||||
.map((run) => {
|
||||
const percent = Math.round((run.progress || 0) * 100);
|
||||
const failed = run.status === "FAILED";
|
||||
// RUNNING/QUEUED 附加耗时或排队提示，其余状态只显示徽章。
|
||||
const statusHtml =
|
||||
run.status === "RUNNING"
|
||||
? `${badge(run.status)} <span class="muted">已运行 ${formatElapsed(run.updated_at)}</span>`
|
||||
: run.status === "QUEUED"
|
||||
? `${badge(run.status)} <span class="muted">排队中</span>`
|
||||
: run.status === "PAUSED"
|
||||
? `${badge(run.status)} <span class="muted">已暂停 ${formatElapsed(run.updated_at)}</span>`
|
||||
: badge(run.status);
|
||||
// 失败任务提供重试，所有任务均可删除。
|
||||
// 排队/运行中可暂停，暂停后可继续，失败可重试，所有任务可删除。
|
||||
const canPause = run.status === "RUNNING" || run.status === "QUEUED";
|
||||
const canResume = run.status === "PAUSED";
|
||||
const actions = `
|
||||
${canPause ? `<button class="warn" data-pause-run="${escapeHtml(run.id)}">暂停</button>` : ""}
|
||||
${canResume ? `<button class="warn" data-resume-run="${escapeHtml(run.id)}">继续</button>` : ""}
|
||||
${failed ? `<button class="danger" data-retry-run="${escapeHtml(run.id)}">重试</button>` : ""}
|
||||
<button class="danger" data-delete-run="${escapeHtml(run.id)}">删除</button>
|
||||
`;
|
||||
return `
|
||||
<tr>
|
||||
<td title="${escapeHtml(run.error || "")}">${escapeHtml(run.id)}</td>
|
||||
<td>${escapeHtml(run.workflow_id)}</td>
|
||||
<td>${statusHtml}</td>
|
||||
<td>${escapeHtml(run.current_node_id || "-")}</td>
|
||||
<td>${progressBar(percent, failed)} ${percent}%</td>
|
||||
<td>${escapeHtml(formatTime(run.created_at))}</td>
|
||||
<td>${artifactLinks(run.id, run.status)}</td>
|
||||
<td>${actions}</td>
|
||||
</tr>`;
|
||||
})
|
||||
.join("")
|
||||
: '<tr><td colspan="8">暂无任务，请到首页发起。</td></tr>';
|
||||
}
|
||||
|
||||
// 请求后端重试失败任务，成功后刷新列表。
|
||||
async function retryRun(runId) {
|
||||
try {
|
||||
const result = await api(`/api/runs/${runId}/retry`, { method: "POST" });
|
||||
alert(`任务 ${result.id} 已重新排队`);
|
||||
await loadRuns();
|
||||
} catch (error) {
|
||||
alert(`重试失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 暂停任务：排队或运行中的任务置为 PAUSED，运行中的任务在节点边界停下。
|
||||
async function pauseRun(runId) {
|
||||
try {
|
||||
const result = await api(`/api/runs/${runId}/pause`, { method: "POST" });
|
||||
alert(`任务 ${result.id} 已暂停`);
|
||||
await loadRuns();
|
||||
} catch (error) {
|
||||
alert(`暂停失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 继续任务：PAUSED 恢复排队，由调度器从断点继续执行。
|
||||
async function resumeRun(runId) {
|
||||
try {
|
||||
const result = await api(`/api/runs/${runId}/resume`, { method: "POST" });
|
||||
alert(`任务 ${result.id} 已恢复执行`);
|
||||
await loadRuns();
|
||||
} catch (error) {
|
||||
alert(`继续失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
// 删除任务：二次确认后调用后端删除接口并刷新列表。
|
||||
async function deleteRun(runId) {
|
||||
if (!window.confirm(`确认删除任务 ${runId}？相关产物文件将一并删除。`)) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await api(`/api/runs/${runId}`, { method: "DELETE" });
|
||||
await loadRuns();
|
||||
} catch (error) {
|
||||
alert(`删除失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 创建或更新工作流：解析 DAG JSON 后提交，随后刷新列表。
|
||||
async function createWorkflow() {
|
||||
const workflowId = document.getElementById("workflowId").value.trim();
|
||||
const name = document.getElementById("workflowName").value.trim();
|
||||
const description = document.getElementById("workflowDescription").value.trim();
|
||||
let definition;
|
||||
try {
|
||||
definition = JSON.parse(document.getElementById("workflowDefinition").value);
|
||||
} catch (error) {
|
||||
alert(`DAG JSON 无效：${error.message}`);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await api("/api/admin/workflows", {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
id: workflowId || undefined,
|
||||
name,
|
||||
description,
|
||||
definition,
|
||||
}),
|
||||
});
|
||||
alert("工作流已保存");
|
||||
await loadWorkflows();
|
||||
} catch (error) {
|
||||
alert(`保存失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 发布指定工作流，使其出现在用户应用中心。
|
||||
async function publishWorkflow(workflowId) {
|
||||
try {
|
||||
await api(`/api/admin/workflows/${workflowId}/publish`, { method: "POST" });
|
||||
await loadWorkflows();
|
||||
} catch (error) {
|
||||
alert(`发布失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 删除指定工作流。
|
||||
async function deleteWorkflow(workflowId) {
|
||||
try {
|
||||
await api(`/api/admin/workflows/${workflowId}`, { method: "DELETE" });
|
||||
await loadWorkflows();
|
||||
} catch (error) {
|
||||
alert(`删除失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 首页发起任务：选择工作流并上传视频，创建任务后跳转到任务管理页查看进度。
|
||||
async function uploadVideo() {
|
||||
const workflowId = document.getElementById("workflowSelect").value;
|
||||
const fileInput = document.getElementById("videoFile");
|
||||
const progress = document.getElementById("runProgress");
|
||||
if (!workflowId) {
|
||||
progress.textContent = "暂无可用工作流";
|
||||
return;
|
||||
}
|
||||
if (!fileInput.files.length) {
|
||||
progress.textContent = "请先选择视频文件";
|
||||
return;
|
||||
}
|
||||
const form = new FormData();
|
||||
form.append("file", fileInput.files[0]);
|
||||
progress.textContent = "上传中...";
|
||||
try {
|
||||
const response = await fetch(`/api/apps/${encodeURIComponent(workflowId)}/runs`, {
|
||||
method: "POST",
|
||||
body: form,
|
||||
});
|
||||
const data = await response.json().catch(() => null);
|
||||
if (!response.ok) {
|
||||
throw new Error(data && data.detail ? JSON.stringify(data.detail) : response.statusText);
|
||||
}
|
||||
progress.textContent = `任务 ${data.id} 已创建，正在跳转到任务管理...`;
|
||||
// 首页只负责发起任务，进度展示交给任务管理页。
|
||||
window.location.href = `/tasks.html?run=${encodeURIComponent(data.id)}`;
|
||||
} catch (error) {
|
||||
progress.textContent = `创建失败：${error.message}`;
|
||||
}
|
||||
}
|
||||
|
||||
// 全局点击委托：按按钮上的 data-* 属性分发到对应操作。
|
||||
document.addEventListener("click", (event) => {
|
||||
const deleteWorkflowButton = event.target.closest("[data-delete-workflow]");
|
||||
if (deleteWorkflowButton) {
|
||||
deleteWorkflow(deleteWorkflowButton.dataset.deleteWorkflow);
|
||||
return;
|
||||
}
|
||||
const publishWorkflowButton = event.target.closest("[data-publish-workflow]");
|
||||
if (publishWorkflowButton) {
|
||||
publishWorkflow(publishWorkflowButton.dataset.publishWorkflow);
|
||||
return;
|
||||
}
|
||||
const retryButton = event.target.closest("[data-retry-run]");
|
||||
if (retryButton) {
|
||||
retryRun(retryButton.dataset.retryRun);
|
||||
return;
|
||||
}
|
||||
const pauseButton = event.target.closest("[data-pause-run]");
|
||||
if (pauseButton) {
|
||||
pauseRun(pauseButton.dataset.pauseRun);
|
||||
return;
|
||||
}
|
||||
const resumeButton = event.target.closest("[data-resume-run]");
|
||||
if (resumeButton) {
|
||||
resumeRun(resumeButton.dataset.resumeRun);
|
||||
return;
|
||||
}
|
||||
const deleteRunButton = event.target.closest("[data-delete-run]");
|
||||
if (deleteRunButton) {
|
||||
deleteRun(deleteRunButton.dataset.deleteRun);
|
||||
}
|
||||
});
|
||||
|
||||
// 页面初始化：预填 DAG、绑定按钮事件并加载对应页面数据。
|
||||
document.addEventListener("DOMContentLoaded", async () => {
|
||||
const workflowDefinition = document.getElementById("workflowDefinition");
|
||||
if (workflowDefinition) {
|
||||
workflowDefinition.value = JSON.stringify(DEMO_WORKFLOW, null, 2);
|
||||
}
|
||||
const createWorkflowButton = document.getElementById("createWorkflow");
|
||||
if (createWorkflowButton) {
|
||||
createWorkflowButton.addEventListener("click", createWorkflow);
|
||||
}
|
||||
const publishWorkflowButton = document.getElementById("publishWorkflow");
|
||||
if (publishWorkflowButton) {
|
||||
publishWorkflowButton.addEventListener("click", () => {
|
||||
publishWorkflow(document.getElementById("workflowId").value.trim());
|
||||
});
|
||||
}
|
||||
const uploadButton = document.getElementById("uploadVideo");
|
||||
if (uploadButton) {
|
||||
uploadButton.addEventListener("click", uploadVideo);
|
||||
// 首页加载已发布工作流供用户选择。
|
||||
await loadWorkflowOptions();
|
||||
// 字幕 OCR 面板：视频预览 + 拖动框选（仅需 crop 的工作流展示）。
|
||||
setupOcrPanel();
|
||||
}
|
||||
const ocrSubmitButton = document.getElementById("ocrSubmit");
|
||||
if (ocrSubmitButton) {
|
||||
ocrSubmitButton.addEventListener("click", submitOcr);
|
||||
}
|
||||
|
||||
await loadHealth();
|
||||
if (document.getElementById("workflowList")) {
|
||||
await loadWorkflows();
|
||||
}
|
||||
// 任务管理页：每 3 秒刷新一次全部任务的进度，支持高亮跳转参数 run=ID。
|
||||
if (document.getElementById("runList")) {
|
||||
await loadRuns();
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
const target = params.get("run");
|
||||
if (target) {
|
||||
const row = [...document.querySelectorAll("#runList tr")].find((item) =>
|
||||
item.textContent.includes(target),
|
||||
);
|
||||
if (row) {
|
||||
row.scrollIntoView({ block: "center" });
|
||||
row.style.background = "#fff7db";
|
||||
}
|
||||
}
|
||||
setInterval(loadRuns, 3000);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,38 @@
|
||||
// WOV OCR 前端 crop 归一化工具（纯函数，供 node 单测与浏览器共用）。
|
||||
// 负责把用户在视频预览上框选的矩形（显示坐标）与 crop 比例 [x,y,w,h]（0~1）
|
||||
// 互转，映射基于视频固有分辨率并处理 object-fit: contain 的留边（letterbox）。
|
||||
|
||||
// 计算 <video> 在指定容器内 contain 显示后的实际渲染矩形（容器坐标）。
|
||||
function videoDisplayRect(videoWidth, videoHeight, boxWidth, boxHeight) {
|
||||
const scale = Math.min(boxWidth / videoWidth, boxHeight / videoHeight);
|
||||
const w = videoWidth * scale;
|
||||
const h = videoHeight * scale;
|
||||
return { x: (boxWidth - w) / 2, y: (boxHeight - h) / 2, w, h };
|
||||
}
|
||||
|
||||
// 框选矩形（容器坐标）→ crop 比例 [x, y, w, h]，钳制到 0~1，保留 3 位小数。
|
||||
function rectToCrop(rect, videoWidth, videoHeight, boxWidth, boxHeight) {
|
||||
const display = videoDisplayRect(videoWidth, videoHeight, boxWidth, boxHeight);
|
||||
const clamp = (v) => Math.min(1, Math.max(0, Math.round(v * 1000) / 1000));
|
||||
return [
|
||||
clamp((rect.x - display.x) / display.w),
|
||||
clamp((rect.y - display.y) / display.h),
|
||||
clamp(rect.w / display.w),
|
||||
clamp(rect.h / display.h),
|
||||
];
|
||||
}
|
||||
|
||||
// crop 比例 → 框选矩形（容器坐标），用于回显。
|
||||
function cropToRect(crop, videoWidth, videoHeight, boxWidth, boxHeight) {
|
||||
const display = videoDisplayRect(videoWidth, videoHeight, boxWidth, boxHeight);
|
||||
return {
|
||||
x: display.x + crop[0] * display.w,
|
||||
y: display.y + crop[1] * display.h,
|
||||
w: crop[2] * display.w,
|
||||
h: crop[3] * display.h,
|
||||
};
|
||||
}
|
||||
|
||||
if (typeof module !== "undefined" && module.exports) {
|
||||
module.exports = { videoDisplayRect, rectToCrop, cropToRect };
|
||||
}
|
||||
Executable
+348
@@ -0,0 +1,348 @@
|
||||
/* VRSub 静态前端全局样式：定义色彩变量、布局与通用组件样式。 */
|
||||
|
||||
/* 设计令牌：集中管理配色，后续换肤只需修改变量。 */
|
||||
:root {
|
||||
--bg: #f5f7fa;
|
||||
--surface: #ffffff;
|
||||
--border: #d7dde6;
|
||||
--text: #1c2733;
|
||||
--muted: #66748a;
|
||||
--primary: #1769aa;
|
||||
--danger: #b42318;
|
||||
--ok: #177245;
|
||||
--radius: 8px;
|
||||
}
|
||||
|
||||
/* 全局盒模型与基础排版重置。 */
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
/* 页面主体：浅灰背景与默认文字颜色。 */
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: "Segoe UI", "Microsoft YaHei", sans-serif;
|
||||
color: var(--text);
|
||||
background: var(--bg);
|
||||
}
|
||||
|
||||
/* 顶部导航栏：品牌标题与页面导航。 */
|
||||
.topbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 12px 28px;
|
||||
background: var(--surface);
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
/* 品牌标题样式。 */
|
||||
.brand {
|
||||
font-size: 20px;
|
||||
font-weight: 700;
|
||||
color: var(--primary);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
/* 导航链接横向排列。 */
|
||||
nav {
|
||||
display: flex;
|
||||
gap: 18px;
|
||||
}
|
||||
|
||||
/* 导航链接默认使用弱化色，悬停时高亮。 */
|
||||
nav a {
|
||||
color: var(--muted);
|
||||
text-decoration: none;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
nav a:hover {
|
||||
color: var(--primary);
|
||||
}
|
||||
|
||||
/* 内容容器：限制最大宽度并居中。 */
|
||||
.container {
|
||||
max-width: 1080px;
|
||||
margin: 0 auto;
|
||||
padding: 28px 20px 60px;
|
||||
}
|
||||
|
||||
/* 一级标题与二级标题的字号控制。 */
|
||||
h1 {
|
||||
margin: 0 0 8px;
|
||||
font-size: 28px;
|
||||
}
|
||||
|
||||
h2 {
|
||||
margin: 0 0 12px;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
/* 弱化文字：用于说明、时间等次要信息。 */
|
||||
.muted {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
/* 应用中心卡片网格：自适应列数的最小宽度布局。 */
|
||||
.app-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
|
||||
gap: 18px;
|
||||
margin-top: 24px;
|
||||
}
|
||||
|
||||
/* 卡片与面板共用白底、边框和圆角外观。 */
|
||||
.card,
|
||||
.panel {
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px;
|
||||
}
|
||||
|
||||
/* 面板之间保持纵向间距。 */
|
||||
.panel {
|
||||
margin-top: 22px;
|
||||
}
|
||||
|
||||
/* 禁用态样式：降低透明度表达不可用。 */
|
||||
.disabled {
|
||||
opacity: 0.6;
|
||||
}
|
||||
|
||||
/* 表单标签：块级显示并加粗。 */
|
||||
label {
|
||||
display: block;
|
||||
margin: 10px 0 6px;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
/* 输入控件统一样式，宽度撑满容器。 */
|
||||
input,
|
||||
select,
|
||||
textarea {
|
||||
width: 100%;
|
||||
padding: 9px 11px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
font: inherit;
|
||||
background: #fff;
|
||||
}
|
||||
|
||||
/* 文本域使用等宽字体并允许纵向拉伸。 */
|
||||
textarea {
|
||||
font-family: Consolas, monospace;
|
||||
resize: vertical;
|
||||
}
|
||||
|
||||
/* 按钮基础样式。 */
|
||||
button {
|
||||
margin-top: 12px;
|
||||
padding: 9px 16px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--surface);
|
||||
color: var(--text);
|
||||
font: inherit;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
/* 按钮悬停强调边框与文字颜色。 */
|
||||
button:hover {
|
||||
border-color: var(--primary);
|
||||
color: var(--primary);
|
||||
}
|
||||
|
||||
/* 主按钮：填充品牌色。 */
|
||||
button.primary {
|
||||
background: var(--primary);
|
||||
border-color: var(--primary);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
/* 幽灵按钮：透明背景，用于次要操作。 */
|
||||
button.ghost {
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
/* 操作区与表单网格：弹性换行排列。 */
|
||||
.actions,
|
||||
.form-grid {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
/* 表单网格：默认两列，宽字段占整行。 */
|
||||
.form-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, 1fr);
|
||||
}
|
||||
|
||||
/* 宽字段跨满整行。 */
|
||||
.form-grid .wide {
|
||||
grid-column: 1 / -1;
|
||||
}
|
||||
|
||||
/* 表格容器：窄屏时允许横向滚动。 */
|
||||
.table-wrap {
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
/* 表格基础样式。 */
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
/* 表头与单元格的间距、对齐与分隔线。 */
|
||||
th,
|
||||
td {
|
||||
padding: 10px;
|
||||
text-align: left;
|
||||
border-bottom: 1px solid var(--border);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* 表头使用弱化色与小字号。 */
|
||||
th {
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
/* 结果块：深色等宽字体，适合展示 JSON 或日志。 */
|
||||
.result {
|
||||
margin: 16px 0 0;
|
||||
padding: 14px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: #0f1720;
|
||||
color: #d7e5f3;
|
||||
font-family: Consolas, monospace;
|
||||
font-size: 13px;
|
||||
white-space: pre-wrap;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
/* 下载区：横向排列的下载链接。 */
|
||||
.downloads {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
flex-wrap: wrap;
|
||||
margin-top: 14px;
|
||||
}
|
||||
|
||||
/* 下载链接：描边按钮式链接。 */
|
||||
.download-link {
|
||||
display: inline-block;
|
||||
padding: 8px 14px;
|
||||
border: 1px solid var(--primary);
|
||||
border-radius: 6px;
|
||||
color: var(--primary);
|
||||
font-weight: 600;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
/* 下载链接悬停时轻微填充背景。 */
|
||||
.download-link:hover {
|
||||
background: #e8f1f9;
|
||||
}
|
||||
|
||||
/* 状态徽章基础样式。 */
|
||||
.badge {
|
||||
display: inline-block;
|
||||
padding: 3px 8px;
|
||||
border-radius: 999px;
|
||||
font-size: 12px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
/* 成功徽章：绿色背景。 */
|
||||
.badge.ok {
|
||||
background: #e2f4ea;
|
||||
color: var(--ok);
|
||||
}
|
||||
|
||||
/* 警告徽章：黄色背景。 */
|
||||
.badge.warn {
|
||||
background: #fff2d9;
|
||||
color: #8a5a00;
|
||||
}
|
||||
|
||||
/* 错误徽章：红色背景。 */
|
||||
.badge.error {
|
||||
background: #fde8e6;
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
/* 危险操作按钮文字颜色。 */
|
||||
.danger {
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
/* 任务进度条容器：圆角底槽。 */
|
||||
.progress {
|
||||
width: 120px;
|
||||
height: 8px;
|
||||
border-radius: 999px;
|
||||
background: #e5e7eb;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* 任务进度条填充：按百分比宽度显示进度。 */
|
||||
.progress-bar {
|
||||
height: 100%;
|
||||
border-radius: 999px;
|
||||
background: var(--primary, #1a73e8);
|
||||
transition: width 0.4s ease;
|
||||
}
|
||||
|
||||
/* 失败任务的进度条使用红色填充。 */
|
||||
.progress-bar.error {
|
||||
background: var(--danger, #d93025);
|
||||
}
|
||||
|
||||
/* 任务页产物下载链接组。 */
|
||||
.downloads-inline {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* 字幕 OCR 面板：视频预览容器，画布绝对覆盖用于框选。 */
|
||||
.video-box {
|
||||
position: relative;
|
||||
width: 100%;
|
||||
aspect-ratio: 16 / 9;
|
||||
background: #000;
|
||||
border-radius: 8px;
|
||||
overflow: hidden;
|
||||
margin: 8px 0;
|
||||
}
|
||||
|
||||
.video-box video {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: contain;
|
||||
}
|
||||
|
||||
.video-box canvas {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
/* 默认不拦截指针事件，保证原生视频控件（进度条等）可操作。 */
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
/* 框选模式下画布才接收鼠标事件。 */
|
||||
.video-box canvas.drawable {
|
||||
pointer-events: auto;
|
||||
cursor: crosshair;
|
||||
}
|
||||
Executable
+70
@@ -0,0 +1,70 @@
|
||||
<!doctype html>
|
||||
<!-- VRSub 首页：发起任务。工作流选择始终可见；选择后若该工作流需要 crop
|
||||
（如字幕 OCR 的抽帧节点），则展示视频预览与框选字幕区域面板，
|
||||
框选后才可提交。 -->
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>VRSub - 发起任务</title>
|
||||
<link rel="stylesheet" href="/assets/styles.css?v=3" />
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<a class="brand" href="/">VRSub</a>
|
||||
<nav>
|
||||
<a href="/">发起任务</a>
|
||||
<a href="/tasks.html">任务管理</a>
|
||||
<a href="/admin.html">管理后台</a>
|
||||
<a href="/workflow.html">工作流</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<main class="container">
|
||||
<h1>VRSub 字幕生成</h1>
|
||||
<p class="muted">选择工作流并上传视频。字幕 OCR 类工作流需要先在预览中框选字幕区域。</p>
|
||||
|
||||
<!-- 工作流选择放在两种模式之外，任何情况下都可见、可切换。 -->
|
||||
<label for="workflowSelect">选择工作流</label>
|
||||
<select id="workflowSelect"></select>
|
||||
|
||||
<div class="app-grid">
|
||||
<!-- 标准任务：选择文件后提交。 -->
|
||||
<article class="card" id="standardCard">
|
||||
<h2>视频字幕生成</h2>
|
||||
<p>上传视频，后台自动执行字幕生成。</p>
|
||||
<label for="videoFile">选择视频</label>
|
||||
<input id="videoFile" type="file" accept="video/*,.mp4,.mkv,.avi,.mov,.m4a,.mp3" />
|
||||
<button id="uploadVideo" class="primary">上传并生成</button>
|
||||
<pre id="runProgress" class="result">等待上传...</pre>
|
||||
</article>
|
||||
|
||||
<!-- 字幕 OCR 面板：选择视频后在预览中拖动框选字幕区域。 -->
|
||||
<article class="card" id="ocrCard" hidden>
|
||||
<h2>字幕 OCR 提取</h2>
|
||||
<p>选择视频并在预览中<b>拖动框选字幕区域</b>，框选完成后才能提交任务。</p>
|
||||
<label for="ocrVideoFile">选择视频</label>
|
||||
<input id="ocrVideoFile" type="file" accept="video/*,.mp4,.mkv,.avi,.mov" />
|
||||
<div class="ocr-stage">
|
||||
<div id="videoBox" class="video-box">
|
||||
<video id="ocrVideo" controls preload="metadata"></video>
|
||||
<canvas id="ocrCanvas"></canvas>
|
||||
</div>
|
||||
</div>
|
||||
<label for="cropValue">字幕区域 crop [x, y, w, h]</label>
|
||||
<input id="cropValue" type="text" readonly placeholder="选择视频并框选后自动生成" />
|
||||
<div class="actions">
|
||||
<button id="ocrDrawMode" class="ghost">进入框选模式</button>
|
||||
<button id="ocrReset" class="ghost">清除框选</button>
|
||||
<button id="ocrSubmit" class="primary" disabled>提交任务</button>
|
||||
</div>
|
||||
<p class="muted">提示：先用进度条定位到有字幕的画面，再点击「进入框选模式」拖动框选字幕区域。</p>
|
||||
<pre id="ocrProgress" class="result">等待框选...</pre>
|
||||
</article>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<script src="/assets/crop.js?v=2"></script>
|
||||
<script src="/assets/app.js?v=6"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,49 @@
|
||||
<!doctype html>
|
||||
<!-- VRSub 任务管理页：展示全部任务的实时进度、产物下载与失败重试。 -->
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>VRSub - 任务管理</title>
|
||||
<link rel="stylesheet" href="/assets/styles.css?v=2" />
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<a class="brand" href="/">VRSub</a>
|
||||
<nav>
|
||||
<a href="/">发起任务</a>
|
||||
<a href="/tasks.html">任务管理</a>
|
||||
<a href="/admin.html">管理后台</a>
|
||||
<a href="/workflow.html">工作流</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<main class="container">
|
||||
<h1>任务管理</h1>
|
||||
<p class="muted">查看全部任务的执行进度；运行中的任务可暂停/继续，完成的任务可下载产物，失败的任务可以重新排队执行。</p>
|
||||
|
||||
<!-- 任务列表由 assets/app.js 的 loadRuns 定期刷新填充，展示进度条与下载链接。 -->
|
||||
<section class="panel">
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>任务 ID</th>
|
||||
<th>工作流</th>
|
||||
<th>状态</th>
|
||||
<th>当前节点</th>
|
||||
<th>进度</th>
|
||||
<th>创建时间</th>
|
||||
<th>产物</th>
|
||||
<th>操作</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="runList"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<script src="/assets/app.js?v=2"></script>
|
||||
</body>
|
||||
</html>
|
||||
Executable
+63
@@ -0,0 +1,63 @@
|
||||
<!doctype html>
|
||||
<!-- VRSub 工作流编排页：以 DAG JSON 创建/更新工作流并发布。 -->
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>VRSub 工作流</title>
|
||||
<link rel="stylesheet" href="/assets/styles.css?v=2" />
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<a class="brand" href="/">VRSub</a>
|
||||
<nav>
|
||||
<a href="/">应用中心</a>
|
||||
<a href="/tasks.html">任务管理</a>
|
||||
<a href="/admin.html">管理后台</a>
|
||||
<a href="/workflow.html">工作流</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<main class="container">
|
||||
<h1>工作流编排</h1>
|
||||
|
||||
<!-- 工作流定义：输入概要信息与 DAG JSON，创建或追加新版本。 -->
|
||||
<section class="panel">
|
||||
<h2>工作流定义</h2>
|
||||
<label for="workflowId">工作流 ID</label>
|
||||
<input id="workflowId" type="text" value="demo" />
|
||||
<label for="workflowName">名称</label>
|
||||
<input id="workflowName" type="text" value="视频字幕生成" />
|
||||
<label for="workflowDescription">描述</label>
|
||||
<input id="workflowDescription" type="text" value="上传视频，自动生成中文字幕和 VR 双眼 ASS。" />
|
||||
<label for="workflowDefinition">DAG JSON</label>
|
||||
<textarea id="workflowDefinition" rows="22"></textarea>
|
||||
<div class="actions">
|
||||
<button id="createWorkflow" class="primary">创建/更新工作流</button>
|
||||
<button id="publishWorkflow" class="ghost">发布</button>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- 已发布工作流：展示发布状态，支持发布与删除操作。 -->
|
||||
<section class="panel">
|
||||
<h2>已发布工作流</h2>
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>ID</th>
|
||||
<th>名称</th>
|
||||
<th>版本</th>
|
||||
<th>状态</th>
|
||||
<th>操作</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="workflowList"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<script src="/assets/app.js?v=2"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"id": "demo",
|
||||
"name": "视频字幕生成",
|
||||
"description": "上传视频，自动生成中文字幕和 VR 双眼 ASS。",
|
||||
"version": 1,
|
||||
"definition": {
|
||||
"name": "视频字幕生成",
|
||||
"version": 1,
|
||||
"nodes": [
|
||||
{
|
||||
"id": "extract",
|
||||
"node_type": "ffmpeg-extract",
|
||||
"params": {
|
||||
"sample_rate": 16000,
|
||||
"channels": 1
|
||||
},
|
||||
"inputs": {
|
||||
"video_uri": "input.video_uri"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "asr",
|
||||
"node_type": "faster-whisper",
|
||||
"params": {
|
||||
"language": "ja",
|
||||
"model_path": "faster-whisper-large-v3",
|
||||
"condition_on_previous_text": false,
|
||||
"chunk_seconds": 60,
|
||||
"vad_filter": true
|
||||
},
|
||||
"inputs": {
|
||||
"audio_uri": "extract.audio_uri"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "translate",
|
||||
"node_type": "llm-translate",
|
||||
"params": {
|
||||
"target_language": "zh-CN"
|
||||
},
|
||||
"inputs": {
|
||||
"srt_uri": "asr.srt_uri"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "ass",
|
||||
"node_type": "srt-to-dual-eye-ass",
|
||||
"params": {
|
||||
"resolution": "3840x1920"
|
||||
},
|
||||
"inputs": {
|
||||
"cn_srt_uri": "translate.cn_srt_uri"
|
||||
}
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"from": "extract",
|
||||
"to": "asr"
|
||||
},
|
||||
{
|
||||
"from": "asr",
|
||||
"to": "translate"
|
||||
},
|
||||
{
|
||||
"from": "translate",
|
||||
"to": "ass"
|
||||
}
|
||||
],
|
||||
"entry_inputs": {
|
||||
"video_uri": "file"
|
||||
},
|
||||
"final_outputs": {
|
||||
"cn_srt": "translate.cn_srt_uri",
|
||||
"ass": "ass.ass_uri"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"id": "ocr-subtitle",
|
||||
"name": "字幕OCR提取",
|
||||
"description": "抽帧并 OCR 提取视频烧录字幕，经 LLM 过滤无意义内容后生成带时间轴的 SRT 基准数据。",
|
||||
"version": 3,
|
||||
"definition": {
|
||||
"name": "字幕OCR提取",
|
||||
"version": 3,
|
||||
"nodes": [
|
||||
{
|
||||
"id": "extract",
|
||||
"node_type": "frame-extract",
|
||||
"params": {
|
||||
"interval_seconds": 0.5,
|
||||
"crop": [
|
||||
0,
|
||||
0.82,
|
||||
1,
|
||||
0.18
|
||||
]
|
||||
},
|
||||
"inputs": {
|
||||
"video_uri": "input.video_uri"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "ocr",
|
||||
"node_type": "subtitle-ocr",
|
||||
"params": {
|
||||
"prompt": "提取图像中的文字，不要描述图片中的内容"
|
||||
},
|
||||
"inputs": {
|
||||
"frames_manifest": "extract.frames_manifest"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "filter",
|
||||
"node_type": "llm-filter",
|
||||
"params": {},
|
||||
"inputs": {
|
||||
"srt_uri": "ocr.srt_uri"
|
||||
}
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"from": "extract",
|
||||
"to": "ocr"
|
||||
},
|
||||
{
|
||||
"from": "ocr",
|
||||
"to": "filter"
|
||||
}
|
||||
],
|
||||
"entry_inputs": {
|
||||
"video_uri": "file"
|
||||
},
|
||||
"final_outputs": {
|
||||
"srt": "filter.srt_uri"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"id": "zh-direct",
|
||||
"name": "中文直出字幕",
|
||||
"description": "使用中文直出模型直接生成中文字幕和 VR 双眼 ASS，无需 LLM 翻译。",
|
||||
"version": 1,
|
||||
"definition": {
|
||||
"name": "中文直出字幕",
|
||||
"version": 1,
|
||||
"nodes": [
|
||||
{
|
||||
"id": "extract",
|
||||
"node_type": "ffmpeg-extract",
|
||||
"params": {
|
||||
"sample_rate": 16000,
|
||||
"channels": 1
|
||||
},
|
||||
"inputs": {
|
||||
"video_uri": "input.video_uri"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "asr",
|
||||
"node_type": "faster-whisper",
|
||||
"params": {
|
||||
"language": "ja",
|
||||
"task": "translate",
|
||||
"model_path": "whisper-large-v2-translate-zh-v0.2-st-ct2",
|
||||
"target_language": "zh-CN",
|
||||
"condition_on_previous_text": false,
|
||||
"chunk_seconds": 60,
|
||||
"vad_filter": true
|
||||
},
|
||||
"inputs": {
|
||||
"audio_uri": "extract.audio_uri"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "ass",
|
||||
"node_type": "srt-to-dual-eye-ass",
|
||||
"params": {
|
||||
"resolution": "3840x1920"
|
||||
},
|
||||
"inputs": {
|
||||
"cn_srt_uri": "asr.srt_uri"
|
||||
}
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"from": "extract",
|
||||
"to": "asr"
|
||||
},
|
||||
{
|
||||
"from": "asr",
|
||||
"to": "ass"
|
||||
}
|
||||
],
|
||||
"entry_inputs": {
|
||||
"video_uri": "file"
|
||||
},
|
||||
"final_outputs": {
|
||||
"cn_srt": "asr.srt_uri",
|
||||
"ass": "ass.ass_uri"
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user