feat: 字幕质量提升——长时寒暄幻觉清洗 + 专名/成人隐语不直译

两部分同属字幕质量优化,共用 llm.py 翻译链路:

1) 长时寒暄幻觉词清洗(nodes/subtitle_cleanup.py)
   对展示时长超过阈值(默认 15s,实测 30s 幻觉占位 vs 2s 真实词的分界)
   且文本含收尾/开场寒暄(晚安、感谢观看等)的字幕,文本替换为 '-',
   由后续过滤流程移除;短时寒暄(如剧情中真实互道晚安)保留不误删。
    写文件前调用 。

2) 专名/隐语不直译(nodes/proper_nouns.py)
   片假名专名(人名/品牌/角色)与成人语境隐语是 LLM 误译重灾区:
   - ジンゴ 被误译成'芒果'、カンタくん 被保留日文而非音译;
   - マンゴー/バナナ/リンゴ/金玉/おまんこ/ちんちん 等在色情语境中
     是生殖器官代称,字面直译严重错译。
   整理三张规则表(专名/拟声词/成人隐语,用户提供隐语表),
    检测原文命中后动态注入翻译提示词,
   让 LLM 按正确语义处理。文档见 docs/proper_nouns.md。

测试(红→绿):
- tests/test_hallucination_mask.py:长时掩码/短时保留/非寒暄保留/阈值边界
- tests/test_proper_nouns.py:专名命中/拟声词/成人隐语/真实数据集成
- 真实 LLM 集成测试(完整 1440 行翻译 + 清洗 + 专名注入)通过
This commit is contained in:
2026-09-05 22:32:40 +08:00
parent fcdcfe020b
commit a032855210
6 changed files with 508 additions and 5 deletions
+85
View File
@@ -0,0 +1,85 @@
"""Subtitle cleanup: mask long-duration closing/greeting hallucinations.
Background (real run 20260905115050): after fixing the timing alignment,
subtitles still contain "closing/greeting hallucination words" - fixed
phrases like 'wan an / gan xie guan kan / gan xie nin de guan kan'
(good night / thanks for watching) that the ASR/LLM repeatedly emits on
empty segments, filling a full 30s block, unrelated to video content.
Some 2s 'good night' might be real dialogue, so it must be kept.
Plan (confirmed by user): after translation, mask subtitle entries whose
*display duration* exceeds a threshold AND whose text contains a greeting
hallucination token - replace the text with '-' so the downstream SRT/filter
pipeline drops it. The duration threshold protects short real greetings.
Threshold is derived from real run data: 30s hallucinations vs 2s real words,
a clear gap; default 15s (>=15s masks, <15s keeps).
Pure functions, unit-testable (tests/test_hallucination_mask.py).
"""
from __future__ import annotations
import re
# Greeting/closing hallucination tokens that LLM repeats on empty/end segments.
HALLUCINATION_TOKENS = (
"谢谢观看", "感谢观看", "感谢收看", "谢谢收看", "感谢您的观看", "感谢您的收看",
"晚安", "下次再见", "再会", "敬请期待", "感谢您的光临", "欢迎光临",
"再见", "多谢观看", "观看愉快",
)
# Display-duration threshold (seconds): only mask entries longer than this.
DEFAULT_THRESHOLD_SECONDS = 15.0
_SRT_BLOCK = re.compile(
r"(\d{2}:\d{2}:\d{2},\d{3})\s*-->\s*(\d{2}:\d{2}:\d{2},\d{3})\s*\n(.*?)(?=\n\s*\d+\s*\n|\Z)",
re.DOTALL,
)
def mask_hallucination_text(
entries: list[dict],
threshold_seconds: float = DEFAULT_THRESHOLD_SECONDS,
) -> list[dict]:
"""Return a new list where long-duration greeting entries have text='-'.
duration = end - start. Only entries whose duration >= threshold AND text
contains any HALLUCINATION_TOKENS are masked. Input list is not mutated.
"""
cleaned = []
for entry in entries:
duration = entry.get("end", 0.0) - entry.get("start", 0.0)
text = entry.get("text", "")
if duration >= threshold_seconds and any(t in text for t in HALLUCINATION_TOKENS):
entry = dict(entry, text="-")
cleaned.append(entry)
return cleaned
def _ts_to_seconds(ts: str) -> float:
"""Convert an SRT timestamp HH:MM:SS,mmm to seconds (float)."""
hours, minutes, rest = ts.split(":")
seconds, millis = rest.split(",")
return int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(millis) / 1000
def clean_srt_text(
srt_text: str,
threshold_seconds: float = DEFAULT_THRESHOLD_SECONDS,
) -> str:
"""Mask long-duration greeting hallucinations in an SRT string.
Parses each cue's start/end/text, applies mask_hallucination_text, and
rewrites the block keeping the original time line when not masked.
"""
def _replace(match) -> str:
start = _ts_to_seconds(match.group(1))
end = _ts_to_seconds(match.group(2))
text = match.group(3).strip()
entry = {"start": start, "end": end, "text": text}
cleaned = mask_hallucination_text([entry], threshold_seconds)
new_text = cleaned[0]["text"]
return f"{match.group(1)} --> {match.group(2)}\n{new_text}"
return _SRT_BLOCK.sub(_replace, srt_text)