feat: 字幕质量提升——长时寒暄幻觉清洗 + 专名/成人隐语不直译
两部分同属字幕质量优化,共用 llm.py 翻译链路:
1) 长时寒暄幻觉词清洗(nodes/subtitle_cleanup.py)
对展示时长超过阈值(默认 15s,实测 30s 幻觉占位 vs 2s 真实词的分界)
且文本含收尾/开场寒暄(晚安、感谢观看等)的字幕,文本替换为 '-',
由后续过滤流程移除;短时寒暄(如剧情中真实互道晚安)保留不误删。
写文件前调用 。
2) 专名/隐语不直译(nodes/proper_nouns.py)
片假名专名(人名/品牌/角色)与成人语境隐语是 LLM 误译重灾区:
- ジンゴ 被误译成'芒果'、カンタくん 被保留日文而非音译;
- マンゴー/バナナ/リンゴ/金玉/おまんこ/ちんちん 等在色情语境中
是生殖器官代称,字面直译严重错译。
整理三张规则表(专名/拟声词/成人隐语,用户提供隐语表),
检测原文命中后动态注入翻译提示词,
让 LLM 按正确语义处理。文档见 docs/proper_nouns.md。
测试(红→绿):
- tests/test_hallucination_mask.py:长时掩码/短时保留/非寒暄保留/阈值边界
- tests/test_proper_nouns.py:专名命中/拟声词/成人隐语/真实数据集成
- 真实 LLM 集成测试(完整 1440 行翻译 + 清洗 + 专名注入)通过
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
"""Subtitle cleanup: mask long-duration closing/greeting hallucinations.
|
||||
|
||||
Background (real run 20260905115050): after fixing the timing alignment,
|
||||
subtitles still contain "closing/greeting hallucination words" - fixed
|
||||
phrases like 'wan an / gan xie guan kan / gan xie nin de guan kan'
|
||||
(good night / thanks for watching) that the ASR/LLM repeatedly emits on
|
||||
empty segments, filling a full 30s block, unrelated to video content.
|
||||
Some 2s 'good night' might be real dialogue, so it must be kept.
|
||||
|
||||
Plan (confirmed by user): after translation, mask subtitle entries whose
|
||||
*display duration* exceeds a threshold AND whose text contains a greeting
|
||||
hallucination token - replace the text with '-' so the downstream SRT/filter
|
||||
pipeline drops it. The duration threshold protects short real greetings.
|
||||
|
||||
Threshold is derived from real run data: 30s hallucinations vs 2s real words,
|
||||
a clear gap; default 15s (>=15s masks, <15s keeps).
|
||||
|
||||
Pure functions, unit-testable (tests/test_hallucination_mask.py).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Greeting/closing hallucination tokens that LLM repeats on empty/end segments.
|
||||
HALLUCINATION_TOKENS = (
|
||||
"谢谢观看", "感谢观看", "感谢收看", "谢谢收看", "感谢您的观看", "感谢您的收看",
|
||||
"晚安", "下次再见", "再会", "敬请期待", "感谢您的光临", "欢迎光临",
|
||||
"再见", "多谢观看", "观看愉快",
|
||||
)
|
||||
|
||||
# Display-duration threshold (seconds): only mask entries longer than this.
|
||||
DEFAULT_THRESHOLD_SECONDS = 15.0
|
||||
|
||||
_SRT_BLOCK = re.compile(
|
||||
r"(\d{2}:\d{2}:\d{2},\d{3})\s*-->\s*(\d{2}:\d{2}:\d{2},\d{3})\s*\n(.*?)(?=\n\s*\d+\s*\n|\Z)",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
def mask_hallucination_text(
|
||||
entries: list[dict],
|
||||
threshold_seconds: float = DEFAULT_THRESHOLD_SECONDS,
|
||||
) -> list[dict]:
|
||||
"""Return a new list where long-duration greeting entries have text='-'.
|
||||
|
||||
duration = end - start. Only entries whose duration >= threshold AND text
|
||||
contains any HALLUCINATION_TOKENS are masked. Input list is not mutated.
|
||||
"""
|
||||
cleaned = []
|
||||
for entry in entries:
|
||||
duration = entry.get("end", 0.0) - entry.get("start", 0.0)
|
||||
text = entry.get("text", "")
|
||||
if duration >= threshold_seconds and any(t in text for t in HALLUCINATION_TOKENS):
|
||||
entry = dict(entry, text="-")
|
||||
cleaned.append(entry)
|
||||
return cleaned
|
||||
|
||||
|
||||
def _ts_to_seconds(ts: str) -> float:
|
||||
"""Convert an SRT timestamp HH:MM:SS,mmm to seconds (float)."""
|
||||
hours, minutes, rest = ts.split(":")
|
||||
seconds, millis = rest.split(",")
|
||||
return int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(millis) / 1000
|
||||
|
||||
|
||||
def clean_srt_text(
|
||||
srt_text: str,
|
||||
threshold_seconds: float = DEFAULT_THRESHOLD_SECONDS,
|
||||
) -> str:
|
||||
"""Mask long-duration greeting hallucinations in an SRT string.
|
||||
|
||||
Parses each cue's start/end/text, applies mask_hallucination_text, and
|
||||
rewrites the block keeping the original time line when not masked.
|
||||
"""
|
||||
def _replace(match) -> str:
|
||||
start = _ts_to_seconds(match.group(1))
|
||||
end = _ts_to_seconds(match.group(2))
|
||||
text = match.group(3).strip()
|
||||
entry = {"start": start, "end": end, "text": text}
|
||||
cleaned = mask_hallucination_text([entry], threshold_seconds)
|
||||
new_text = cleaned[0]["text"]
|
||||
return f"{match.group(1)} --> {match.group(2)}\n{new_text}"
|
||||
|
||||
return _SRT_BLOCK.sub(_replace, srt_text)
|
||||
Reference in New Issue
Block a user