feat: 字幕质量提升——长时寒暄幻觉清洗 + 专名/成人隐语不直译
两部分同属字幕质量优化,共用 llm.py 翻译链路:
1) 长时寒暄幻觉词清洗(nodes/subtitle_cleanup.py)
对展示时长超过阈值(默认 15s,实测 30s 幻觉占位 vs 2s 真实词的分界)
且文本含收尾/开场寒暄(晚安、感谢观看等)的字幕,文本替换为 '-',
由后续过滤流程移除;短时寒暄(如剧情中真实互道晚安)保留不误删。
写文件前调用 。
2) 专名/隐语不直译(nodes/proper_nouns.py)
片假名专名(人名/品牌/角色)与成人语境隐语是 LLM 误译重灾区:
- ジンゴ 被误译成'芒果'、カンタくん 被保留日文而非音译;
- マンゴー/バナナ/リンゴ/金玉/おまんこ/ちんちん 等在色情语境中
是生殖器官代称,字面直译严重错译。
整理三张规则表(专名/拟声词/成人隐语,用户提供隐语表),
检测原文命中后动态注入翻译提示词,
让 LLM 按正确语义处理。文档见 docs/proper_nouns.md。
测试(红→绿):
- tests/test_hallucination_mask.py:长时掩码/短时保留/非寒暄保留/阈值边界
- tests/test_proper_nouns.py:专名命中/拟声词/成人隐语/真实数据集成
- 真实 LLM 集成测试(完整 1440 行翻译 + 清洗 + 专名注入)通过
This commit is contained in:
+23
-5
@@ -28,7 +28,8 @@ import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from wov_sdk.models import InvokeRequest, InvokeResponse
|
||||
|
||||
from nodes.subtitle_cleanup import clean_srt_text
|
||||
from nodes.proper_nouns import build_proper_noun_rule
|
||||
# 单次 LLM 请求携带的字幕行数;过大会超出模型上下文,过小则请求次数过多。
|
||||
CHUNK_SIZE = 20
|
||||
|
||||
@@ -144,18 +145,28 @@ def _translate_batch(
|
||||
system_prompt: str,
|
||||
request_timeout: float,
|
||||
) -> list[str]:
|
||||
"""翻译单个批次:行数不一致时多行合并、少行重试,返回与 chunk 等长译文。"""
|
||||
"""翻译单个批次:行数不一致时多行合并、少行重试,返回与 chunk 等长译文。
|
||||
|
||||
每批调用前根据本批原文命中情况动态拼接专名/隐语规则(build_proper_noun_rule),
|
||||
注入到系统提示词,让 LLM 正确处理片假名专名与成人语境隐语。"""
|
||||
# 本批命中的专名/隐语规则(无命中返回 None)。
|
||||
rule = build_proper_noun_rule(chunk)
|
||||
batch_system = system_prompt
|
||||
if rule:
|
||||
batch_system = system_prompt + "\n\n" + rule
|
||||
attempt = 0
|
||||
while True:
|
||||
content = _call_llm(
|
||||
api_base,
|
||||
api_key,
|
||||
model,
|
||||
system_prompt,
|
||||
batch_system,
|
||||
"\n".join(chunk),
|
||||
request_timeout,
|
||||
)
|
||||
batch = [line.strip() for line in content.splitlines() if line.strip()]
|
||||
# 保留所有行:先 rstrip 尾随换行避免多出末尾空行,再 splitlines 保留
|
||||
# 内容中的空串行(空行可能是合法的空字幕,过滤掉会误判行数)。
|
||||
batch = content.rstrip("\n").splitlines()
|
||||
if len(batch) == len(chunk):
|
||||
return batch
|
||||
if len(batch) > len(chunk):
|
||||
@@ -193,8 +204,15 @@ def invoke(request: InvokeRequest) -> InvokeResponse:
|
||||
for index, text_index in enumerate(text_indices):
|
||||
lines[text_index] = translated_lines[index]
|
||||
|
||||
# 长时寒暄幻觉词清洗:对展示时长超过阈值且含收尾/开场寒暄(晚安、感谢观看
|
||||
# 等)的条目,文本替换为 '-'(由后续过滤流程移除),避免幻觉占位污染正片;
|
||||
# 短时(≤阈值)如剧情中真实互道'晚安'则保留,不误删。见
|
||||
# nodes/subtitle_cleanup.py。
|
||||
srt_body = "\n".join(lines) + "\n"
|
||||
srt_body = clean_srt_text(srt_body)
|
||||
|
||||
output_dir = Path(request.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / "cn.srt"
|
||||
output_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
output_path.write_text(srt_body, encoding="utf-8")
|
||||
return InvokeResponse(status="completed", outputs={"cn_srt_uri": str(output_path)})
|
||||
@@ -0,0 +1,121 @@
|
||||
"""专有名词与隐语(不应直译)处理规则。
|
||||
|
||||
日语字幕"日 → 中文"翻译中,两类词是 LLM 误译重灾区:
|
||||
1. **片假名专有名词**(人名/品牌/角色/道具名):模型常按读音硬译
|
||||
(如 ジンゴ → "芒果"),产生与原文无对应的荒谬结果。
|
||||
2. **成人语境隐语/俗称**(AV/色情内容的委婉说法):模型常按字面直译
|
||||
(如 マンゴー → 芒果、バナナ → 香蕉、金玉 → 金玉),实际这些词在
|
||||
色情语境中是生殖器官或性行为的代称。
|
||||
|
||||
本模块维护三张规则表(专名、拟声/口语、成人语境隐语),并提供规则构建
|
||||
函数 build_proper_noun_rule,供翻译节点在系统提示词中动态注入,让 LLM
|
||||
按正确语义处理。
|
||||
|
||||
处理策略(详见 docs/proper_nouns.md):
|
||||
- 人名:音译(保留姓氏/称谓),不直译字面义;
|
||||
- 品牌/产品名:保留原文或使用约定译名;
|
||||
- 角色/道具专名:保留原文或按上下文意译,禁止按读音硬译;
|
||||
- 拟声/拟态词:用中文对应拟声词,不直译;
|
||||
- **成人语境隐语:按"实际含义"的常用中文翻译处理,禁止字面直译**。
|
||||
|
||||
匹配基于"原文文本是否包含专名"判断,未命中时不注入规则(避免干扰普通
|
||||
翻译)。纯函数,可独立测试(tests/test_proper_nouns.py)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 一、专名表:{日文原文: (中文处理建议, 说明)}。
|
||||
# 命中即注入对应指令。
|
||||
PROPER_NOUNS: dict[str, tuple[str, str]] = {
|
||||
# 角色/道具专名:勿按读音硬译
|
||||
"ジンゴ": ("保留原英文'Jingo'或按上下文意译;禁止译作'芒果'",
|
||||
"实测被误译'芒果'(4500s),片假名专名"),
|
||||
"マンゴー": ("保留'マンゴー/Mango';仅当确指水果时译'芒果'",
|
||||
"与'芒果'同音易误译,见成人隐语表"),
|
||||
# 人名:音译,勿留日文
|
||||
"カンタくん": ("音译'康太君/坎塔君',勿保留'カンタ君'", "人名(3803s)"),
|
||||
"松井": ("保留'松井'(姓氏汉字)", "人名,参考'松井小姐'"),
|
||||
"ひな子": ("音译'日奈子/雏子'", "人名,参考'松井日奈子'"),
|
||||
"カリン": ("音译'果林/花梨'", "人名,参考'北冈果林'"),
|
||||
"北岡": ("保留'北冈'(姓氏汉字)", "人名"),
|
||||
"権藤": ("音译'权藤/昆藤'", "人名,参考'医务室长权藤'"),
|
||||
"金山": ("保留'金山'(姓氏汉字)", "人名,参考'金山先生'"),
|
||||
}
|
||||
|
||||
# 二、高频"伪专名"(拟声/口语):勿按字面直译
|
||||
ONOMATOPOEIA: dict[str, tuple[str, str]] = {
|
||||
"パンパン": ("按语境译'鼓胀、饱满、涨满'", "勿译'砰砰'"),
|
||||
"グリグリ": ("用力碾/钻、搅动", "勿译'咕噜咕噜'"),
|
||||
"チンポ": ("按上下文译为俗语(肉棒/鸡巴)", "勿音译'金宝'"),
|
||||
"ビクビク": ("一颤一颤、哆嗦", "勿译'比库比库'"),
|
||||
"ヌルヌル": ("滑溜溜、黏糊糊", "勿译'奴鲁奴鲁'"),
|
||||
"ビンビン": ("硬邦邦、精神十足", "勿译'宾宾'"),
|
||||
"マット": ("垫子(日语借词)", "勿译'马特'"),
|
||||
"リラックス": ("放松(外来语还原含义)", "勿音译'丽拉库斯'"),
|
||||
}
|
||||
|
||||
# 三、成人语境隐语/俗称:{日文原文: (实际含义, 对应中文常用翻译, 说明)}。
|
||||
# 这些词在色情语境中是生殖器官/性行为的代称,LLM 若按字面直译会严重错译。
|
||||
# 结构:词条 -> (实际含义, 常用中文翻译, 补充说明/约束)
|
||||
ADULT_EUPHEMISMS: dict[str, tuple[str, str, str]] = {
|
||||
# 水果谐音/形状类(画面常见比喻)
|
||||
"マンゴー": ("女性生殖器", "小穴、鲍鱼、妹妹;若为谐音梗可保留'芒果'",
|
||||
"与'マンコ'同音,直译'芒果'为常见误译"),
|
||||
"バナナ": ("男性生殖器", "肉棒、鸡鸡、老二;画面打码可译'香肠'",
|
||||
"水果形状比喻,直译'香蕉'错误"),
|
||||
"リンゴ": ("睾丸(阴囊)", "蛋蛋;若出自'淫梦'梗可保留'苹果'并加注释",
|
||||
"食物比喻,直译'苹果'错误"),
|
||||
# 物品比喻类
|
||||
"ちんぽう": ("男性生殖器", "阳具、鸡巴(粗俗)", "与'珍宝(ちんぽう)'谐音"),
|
||||
"ちんちん": ("男性生殖器", "小鸡鸡(委婉)", "小朋友用语,女优常用来装可爱"),
|
||||
"金玉": ("睾丸", "蛋蛋、睾丸(正式)", "直译'金玉'会让人摸不着头脑"),
|
||||
"にくつぼ": ("女性生殖器", "肉洞、蜜穴(偏文学性);小穴(口语)",
|
||||
"字面'肉壶',实为色情比喻"),
|
||||
"おまんこ": ("女性生殖器", "小穴、阴部(正式/粗俗)", "最常用称,勿照搬"),
|
||||
"おそそ": ("女性生殖器", "那里、下面(委婉)", "儿童语/婉语,直译会破坏语气"),
|
||||
"オチンチン": ("男性生殖器", "小弟弟(常用);网络流行语可直译'欧金金'",
|
||||
"与'欧金金'同源网络梗"),
|
||||
# 状态/动作类
|
||||
"ほんばん": ("真实插入的性行为", "真枪实弹、来真的、本番(音译)",
|
||||
"AV 术语,勿直译'本番'为'正本'"),
|
||||
"すまた": ("股间摩擦(不插入)", "素股(音译最常用)、腿交、磨大腿",
|
||||
"AV 术语"),
|
||||
"せいかん": ("户外/野外性行为", "野战、户外做爱", "字面'青姦',勿照搬"),
|
||||
"エッチ": ("性行为/色情的", "做爱、嘿咻、H(音译)", "通用隐语,勿直译'H'"),
|
||||
"アヘ顔": ("高潮时翻白眼吐舌的表情", "阿黑颜(音译)、高潮脸、失神脸",
|
||||
"网络亚文化词,直译'阿嘿脸'外行"),
|
||||
}
|
||||
|
||||
|
||||
def build_proper_noun_rule(terms: str | list[str]) -> str | None:
|
||||
"""根据待翻译文本中的专名/隐语,构建翻译提示词片段。
|
||||
|
||||
参数 terms: 待翻译的原文行(字符串列表或整体文本)。
|
||||
返回注入提示词的规则字符串;若原文未命中任何专名/隐语则返回 None
|
||||
(不注入,避免干扰普通翻译)。
|
||||
"""
|
||||
if isinstance(terms, str):
|
||||
joined = terms
|
||||
else:
|
||||
joined = "\n".join(terms)
|
||||
|
||||
rules: list[str] = []
|
||||
for token, (advice, reason) in PROPER_NOUNS.items():
|
||||
if token in joined:
|
||||
rules.append(f" 日文'{token}':{advice}({reason})。")
|
||||
for token, (advice, reason) in ONOMATOPOEIA.items():
|
||||
if token in joined:
|
||||
rules.append(f" 日文'{token}':{advice}({reason})。")
|
||||
for token, (meaning, cn, note) in ADULT_EUPHEMISMS.items():
|
||||
if token in joined:
|
||||
rules.append(
|
||||
f" 日文'{token}'是成人语境隐语(实际含义:{meaning}),"
|
||||
f"请译为'{cn}',切勿按字面直译。{note}。"
|
||||
)
|
||||
|
||||
if not rules:
|
||||
return None
|
||||
return (
|
||||
"注意以下专有名词/隐语按规则处理(不要按读音或字面硬译):\n"
|
||||
+ "\n".join(rules)
|
||||
)
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Subtitle cleanup: mask long-duration closing/greeting hallucinations.
|
||||
|
||||
Background (real run 20260905115050): after fixing the timing alignment,
|
||||
subtitles still contain "closing/greeting hallucination words" - fixed
|
||||
phrases like 'wan an / gan xie guan kan / gan xie nin de guan kan'
|
||||
(good night / thanks for watching) that the ASR/LLM repeatedly emits on
|
||||
empty segments, filling a full 30s block, unrelated to video content.
|
||||
Some 2s 'good night' might be real dialogue, so it must be kept.
|
||||
|
||||
Plan (confirmed by user): after translation, mask subtitle entries whose
|
||||
*display duration* exceeds a threshold AND whose text contains a greeting
|
||||
hallucination token - replace the text with '-' so the downstream SRT/filter
|
||||
pipeline drops it. The duration threshold protects short real greetings.
|
||||
|
||||
Threshold is derived from real run data: 30s hallucinations vs 2s real words,
|
||||
a clear gap; default 15s (>=15s masks, <15s keeps).
|
||||
|
||||
Pure functions, unit-testable (tests/test_hallucination_mask.py).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Greeting/closing hallucination tokens that LLM repeats on empty/end segments.
|
||||
HALLUCINATION_TOKENS = (
|
||||
"谢谢观看", "感谢观看", "感谢收看", "谢谢收看", "感谢您的观看", "感谢您的收看",
|
||||
"晚安", "下次再见", "再会", "敬请期待", "感谢您的光临", "欢迎光临",
|
||||
"再见", "多谢观看", "观看愉快",
|
||||
)
|
||||
|
||||
# Display-duration threshold (seconds): only mask entries longer than this.
|
||||
DEFAULT_THRESHOLD_SECONDS = 15.0
|
||||
|
||||
_SRT_BLOCK = re.compile(
|
||||
r"(\d{2}:\d{2}:\d{2},\d{3})\s*-->\s*(\d{2}:\d{2}:\d{2},\d{3})\s*\n(.*?)(?=\n\s*\d+\s*\n|\Z)",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
def mask_hallucination_text(
|
||||
entries: list[dict],
|
||||
threshold_seconds: float = DEFAULT_THRESHOLD_SECONDS,
|
||||
) -> list[dict]:
|
||||
"""Return a new list where long-duration greeting entries have text='-'.
|
||||
|
||||
duration = end - start. Only entries whose duration >= threshold AND text
|
||||
contains any HALLUCINATION_TOKENS are masked. Input list is not mutated.
|
||||
"""
|
||||
cleaned = []
|
||||
for entry in entries:
|
||||
duration = entry.get("end", 0.0) - entry.get("start", 0.0)
|
||||
text = entry.get("text", "")
|
||||
if duration >= threshold_seconds and any(t in text for t in HALLUCINATION_TOKENS):
|
||||
entry = dict(entry, text="-")
|
||||
cleaned.append(entry)
|
||||
return cleaned
|
||||
|
||||
|
||||
def _ts_to_seconds(ts: str) -> float:
|
||||
"""Convert an SRT timestamp HH:MM:SS,mmm to seconds (float)."""
|
||||
hours, minutes, rest = ts.split(":")
|
||||
seconds, millis = rest.split(",")
|
||||
return int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(millis) / 1000
|
||||
|
||||
|
||||
def clean_srt_text(
|
||||
srt_text: str,
|
||||
threshold_seconds: float = DEFAULT_THRESHOLD_SECONDS,
|
||||
) -> str:
|
||||
"""Mask long-duration greeting hallucinations in an SRT string.
|
||||
|
||||
Parses each cue's start/end/text, applies mask_hallucination_text, and
|
||||
rewrites the block keeping the original time line when not masked.
|
||||
"""
|
||||
def _replace(match) -> str:
|
||||
start = _ts_to_seconds(match.group(1))
|
||||
end = _ts_to_seconds(match.group(2))
|
||||
text = match.group(3).strip()
|
||||
entry = {"start": start, "end": end, "text": text}
|
||||
cleaned = mask_hallucination_text([entry], threshold_seconds)
|
||||
new_text = cleaned[0]["text"]
|
||||
return f"{match.group(1)} --> {match.group(2)}\n{new_text}"
|
||||
|
||||
return _SRT_BLOCK.sub(_replace, srt_text)
|
||||
Reference in New Issue
Block a user