feat: 新增种子目录清理脚本与配套 skill
主程序只能把 magnet 交给 Transmission,无法挑选文件,种子内的广告 (*.url、宣传页、样片/预告)会一并下载,同一部作品还会被多个发布站各下一份。 新增离线清理脚本处理下载完成后的目录。 clean_seed_extras.py: - 保留:>=100MB 的视频(40+ 扩展名;无扩展名时读文件头嗅探)、字幕 (含 .srt.old-<日期> 备份)、与同目录视频同名的海报、folder.jpg 等通用封面 - 删除:其余文件;同目录内内容相同的大视频只保留一份,校验方式为 sha256(体积 + 头 4MB + 尾 4MB),体积相同但内容不同的不会误删 - 删除后自底向上级联清理空目录(可删到影片级目录,但不删 --root 本身) - 默认预演,需 --apply 才真正删除;--manifest 导出待删清单供逐段复核 - 其他开关:--no-subtitles/--no-posters/--no-covers/--no-dedupe/--min-size-mb/ --prefer-source/--keep-empty-dirs/--skip-recent-min .agents/skills/torrent-download-cleanup: - 记录完整流程(预演 -> 出清单 -> 用户确认 -> 执行 -> 复核)、保留/删除规则、 去重策略、代码维护注意(decide() 必须是 视频 -> 去重 -> 海报匹配 的顺序) - 附只读盘点脚本 find_missing_torrents.py:找出 Transmission 中数据已不在磁盘 的任务(自包含,只用 requests) tests/test_clean_seed_extras.py: 30 个离线用例(tmp_path 临时目录) tests/README.md: 补充脚本用法与测试说明
This commit is contained in:
@@ -0,0 +1,146 @@
|
|||||||
|
---
|
||||||
|
name: torrent-download-cleanup
|
||||||
|
description: 用 clean_seed_extras.py 清理 LookingWeb 的种子下载目录(默认 /mnt/fnOS/123):保留大体积视频、字幕、同名海报与通用封面,删除广告/样片/重复视频并级联清理空目录。当用户要求"清理/整理下载目录、删除重复视频、看看有哪些垃圾可以删、下载完成了清一下",或需要继续上一次的清理流程与 Transmission 收尾时使用。
|
||||||
|
---
|
||||||
|
|
||||||
|
# 种子下载目录清理(LookingWeb)
|
||||||
|
|
||||||
|
主程序 `LookingWebMain.py` 只能把 magnet 交给 Transmission,**无法挑选文件**,所以种子里的广告
|
||||||
|
(`*.url`、宣传页、样片、预告)会和正片一起下载;同一部作品还会被多个发布站各下一份。
|
||||||
|
本 skill 负责下载完成后的离线清理,与主程序互不依赖。
|
||||||
|
|
||||||
|
## 硬性流程(必须遵守)
|
||||||
|
|
||||||
|
1. **先预演**:`--quiet` 看汇总,`--manifest` 导出完整清单。绝不直接 `--apply`。
|
||||||
|
2. **把清单给用户看**:报数量、体积、分段行号,并把"可能误伤"的类别单独列出来
|
||||||
|
(小体积视频、通用封面、会被整体清空的影片级目录)。
|
||||||
|
3. **拿到明确确认后才执行** `--apply`。删除不可逆,这是硬边界。
|
||||||
|
4. **执行后复核**:文件数是否等于预演的"保留"数、垃圾是否清零、再跑一次预演是否 0 删除。
|
||||||
|
5. **不要用 `df` 宣称"释放了空间"**(原因见下文"磁盘空间")。
|
||||||
|
|
||||||
|
## 脚本
|
||||||
|
|
||||||
|
`clean_seed_extras.py`(仓库根目录,仅依赖 loguru)。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd <repo> # 例如 ~/Documents/myProject/lookingWeb
|
||||||
|
.venv/bin/python clean_seed_extras.py --quiet --manifest clean_manifest.txt # 预演 + 导出清单
|
||||||
|
.venv/bin/python clean_seed_extras.py --apply --quiet # 确认后执行
|
||||||
|
```
|
||||||
|
|
||||||
|
| 参数 | 作用 |
|
||||||
|
|---|---|
|
||||||
|
| `--root PATH` | 目标目录,默认 `/mnt/fnOS/123/` |
|
||||||
|
| `--min-size-mb N` | "大视频"阈值,默认 100 |
|
||||||
|
| `--apply` | 真正删除;不加则只预演 |
|
||||||
|
| `--manifest PATH` | 导出待删清单(文件 + 空目录) |
|
||||||
|
| `--list-all` / `--list-limit N` | 预演时是否列出全部待删项 |
|
||||||
|
| `--no-subtitles` | 连字幕一起删 |
|
||||||
|
| `--no-posters` | 不保留任何图片 |
|
||||||
|
| `--no-covers` | 不保留 `folder.jpg` 等通用封面(只留同名海报) |
|
||||||
|
| `--no-dedupe` | 关闭重复视频去重 |
|
||||||
|
| `--dup-check-bytes N` | 重复校验头/尾各读多少字节,默认 4MB |
|
||||||
|
| `--prefer-source a,b` | 优先保留的站点前缀 |
|
||||||
|
| `--keep-empty-dirs` | 不删空目录 |
|
||||||
|
| `--skip-recent-min N` | 跳过最近 N 分钟内改动过的文件(防误删在下的) |
|
||||||
|
| `--quiet` | 只输出统计,不逐条列文件 |
|
||||||
|
|
||||||
|
## 保留 / 删除规则
|
||||||
|
|
||||||
|
保留(命中任一):
|
||||||
|
|
||||||
|
1. **视频**:扩展名在 `VIDEO_EXTS` 内(mp4/mkv/avi/wmv/ts/m2ts/rmvb/iso…共 40+ 种)
|
||||||
|
且体积 `>= --min-size-mb`;扩展名未知或无扩展名时读**文件头**嗅探
|
||||||
|
(ftyp / EBML / RIFF-AVI / FLV / ASF / MPEG-PS / TS 双同步字节 / RMF)。
|
||||||
|
2. **字幕**:文件名含 `.srt/.ass/.ssa/.vtt/.sub/.idx/.sup/.smi` 等,
|
||||||
|
含 `xxx.CN.srt.old-<日期>` 这类备份。
|
||||||
|
3. **海报**:图片扩展名,且文件名包含**同目录**下某个已保留视频的文件名主干。
|
||||||
|
4. **通用封面**:`folder.jpg` / `poster.jpg` / `cover.jpg` / `backdrop.jpg` / `fanart.jpg`
|
||||||
|
等媒体库约定名(Jellyfin/Emby/Kodi 当封面用),且同目录下确实还有保留的视频
|
||||||
|
(目录里没视频时封面算无主垃圾,一并删)。
|
||||||
|
|
||||||
|
删除:其余全部;外加**同目录内内容相同的大视频只保留一份**。
|
||||||
|
删除完成后**自底向上级联清空目录**:子目录先删,父目录变空后继续向上删,
|
||||||
|
可一路删到影片级目录,但**不删 `--root` 本身**。
|
||||||
|
不跟随符号链接目录;删符号链接只删链接本身。
|
||||||
|
|
||||||
|
## 重复视频去重
|
||||||
|
|
||||||
|
同一部作品被多个发布站各下一份时,文件名只差站点前缀,例如:
|
||||||
|
|
||||||
|
```
|
||||||
|
masex.tv@vrkm01874_1_8k.mp4 ← 保留
|
||||||
|
hkbisi.com@vrkm01874_1_8k.mp4 ← 删除(体积与头尾指纹完全一致)
|
||||||
|
```
|
||||||
|
|
||||||
|
- 先按「同目录 + 同体积」分组(几乎不耗 IO),再对每组算
|
||||||
|
`sha256(体积 + 头 4MB + 尾 4MB)` 确认内容一致。**体积相同但内容不同的不会误删。**
|
||||||
|
- 不做全量哈希:740 个视频约 3.5TB,读全量不现实。
|
||||||
|
- 保留哪一份依次比较:**无站点前缀** > `--prefer-source` 指定站点 >
|
||||||
|
同目录内该前缀出现次数更多(保持目录内命名一致)> 文件名短 > 字典序。
|
||||||
|
- 实测站点前缀:`masex.tv@`、`4k2.com@`、`4k2.me@`、`hkbisi.com@`、`twojav.com@`、`489155.com@`。
|
||||||
|
|
||||||
|
## 代码维护注意(改 `decide()` 时)
|
||||||
|
|
||||||
|
处理顺序**必须是**:① 标记达阈值视频 → ② 去重 → ③ 字幕/海报/封面匹配。
|
||||||
|
|
||||||
|
去重必须排在海报匹配之前:否则"被删副本的同名海报"要等到下一轮才被清掉,脚本就不是一次收敛
|
||||||
|
(历史上踩过这个坑,测试 `test_去重后副本的同名海报一并删除` 会断言"再跑一次无事可做")。
|
||||||
|
另外第 ③ 轮的入口是 `if entry.keep or entry.reason: continue`——`reason` 非空表示已定案
|
||||||
|
(保留或已判为重复视频),不能被覆盖成"待删除"。
|
||||||
|
|
||||||
|
改完必须同步:`tests/test_clean_seed_extras.py`(离线用例)与 `tests/README.md`。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m pytest tests/test_clean_seed_extras.py -v
|
||||||
|
```
|
||||||
|
|
||||||
|
## 执行后的复核清单
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /mnt/fnOS/123
|
||||||
|
find . -type f | wc -l # 应等于预演的"保留"数
|
||||||
|
find . -type f -size +100M -name '*.mp4' -printf '%h\t%s\n' | sort | uniq -d | wc -l # 应为 0 组
|
||||||
|
find . -name '*.url' | wc -l # 应为 0
|
||||||
|
find . -type d -name '.pad' | wc -l # 应为 0
|
||||||
|
find . -type d -name '*.trickplay' | wc -l # 应为 0
|
||||||
|
.venv/bin/python <repo>/clean_seed_extras.py --quiet # 再跑一次应为"删除: 0"(幂等)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Transmission 收尾(每次清理后都要看一眼)
|
||||||
|
|
||||||
|
清理会让 Transmission 里的任务缺文件:被清掉的样片/预告 → 部分缺失;
|
||||||
|
被去重删掉的那一侧副本 → 该任务**数据全部不在磁盘上**。
|
||||||
|
|
||||||
|
先用只读脚本盘点(`scripts/find_missing_torrents.py`,自包含,只用 requests,不修改任何东西):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python .agents/skills/torrent-download-cleanup/scripts/find_missing_torrents.py
|
||||||
|
.venv/bin/python .../find_missing_torrents.py --json # 机器可读输出
|
||||||
|
.venv/bin/python .../find_missing_torrents.py --host 192.168.123.199 --port 9095
|
||||||
|
```
|
||||||
|
|
||||||
|
输出里标 `!` 的任务仍在活动状态(会空转/报错),最值得处理。
|
||||||
|
|
||||||
|
注意:Transmission 的 `downloadDir` 是 **NAS 侧路径**(实测为 `/downloads`),
|
||||||
|
在客户端命名空间里不存在,所以**不能**用 `os.path.join(downloadDir, f["name"])` 判断存在性——
|
||||||
|
那样会把所有任务误报为"全部缺失"。必须按**文件名 basename** 与本机目录比对。
|
||||||
|
|
||||||
|
要把这些任务从 Transmission 移除时(`torrent-remove`,只删任务记录不删数据),
|
||||||
|
**必须先列出清单(id / 名称 / 状态 / 缺失文件数)给用户确认**,不要自行决定。
|
||||||
|
|
||||||
|
## 磁盘空间:不要用 df 下结论
|
||||||
|
|
||||||
|
`/mnt/fnOS/123` 是 CIFS 挂载(`//192.168.123.199/123`),它报告的容量是**缓存的**:
|
||||||
|
实测写入 256MB 并 `sync` 后 `df` 读数**一位都没变**,间隔 20 秒的两次读数也完全相同。
|
||||||
|
所以 `df` 既不能证明"空间已释放",也不能证明"没释放"。
|
||||||
|
释放空间请让用户到 **fnOS 存储空间页面**确认;若那边也没降,考虑快照或回收站占用。
|
||||||
|
|
||||||
|
## 背景数据(2026-10-18 首次完整清理)
|
||||||
|
|
||||||
|
- 清理前 4540 个文件 → 清理后 **3021 个**(视频 701 / 字幕 1658 / 海报 624 / 通用封面 38),目录 276。
|
||||||
|
- 累计删除 1519 个文件 ≈ 226.7 GB:1441 个垃圾(2.9GB)+ 39 个重复大视频(223.8GB)
|
||||||
|
+ 39 个孤儿海报(26.8MB);另删除 751 个空目录。
|
||||||
|
- 站点常见垃圾:`*.url`(广告)、`twojav.com@*` 等前缀的样片/预告、
|
||||||
|
`.pad/`(BT BEP47 填充文件,全 0 字节)、`*.trickplay/`(Jellyfin 缩略图)。
|
||||||
|
- 预告片特征:同一支 19.3MB 的 `三 上 悠 亚 想 要 跟 你 决 胜 负.mp4` 在 80+ 个目录重复。
|
||||||
@@ -0,0 +1,180 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
只读盘点:Transmission 里哪些任务的数据文件已经不在本机下载目录上。
|
||||||
|
|
||||||
|
用途:`clean_seed_extras.py` 清理后,被删掉的样片/预告会让对应任务"部分缺失",
|
||||||
|
被去重删掉的那一侧副本会让对应任务"数据全部不在磁盘上"。这个脚本只查询、不修改,
|
||||||
|
输出清单供人工确认后,再决定是否用 torrent-remove 移除这些任务。
|
||||||
|
|
||||||
|
自包含:只用 requests 直连 Transmission RPC,不依赖仓库里其他模块。
|
||||||
|
|
||||||
|
用法:
|
||||||
|
.venv/bin/python .agents/skills/torrent-download-cleanup/scripts/find_missing_torrents.py
|
||||||
|
.venv/bin/python .../find_missing_torrents.py --root /mnt/fnOS/123 --json
|
||||||
|
.venv/bin/python .../find_missing_torrents.py --host 192.168.123.199 --port 9095
|
||||||
|
|
||||||
|
要点:Transmission 返回的 downloadDir 是 **NAS 侧路径**(实测为 /downloads),
|
||||||
|
在客户端命名空间里并不存在,所以不能拿它拼路径判断文件是否存在;
|
||||||
|
必须按文件名的 basename 与本机目录里的文件比对,否则会把所有任务误报为"全部缺失"。
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
DEFAULT_HOST = "192.168.123.199"
|
||||||
|
DEFAULT_PORT = 9095
|
||||||
|
DEFAULT_RPC_PATH = "/transmission/rpc"
|
||||||
|
DEFAULT_USER = "admin"
|
||||||
|
DEFAULT_PASSWORD = "GUOxy.5157"
|
||||||
|
DEFAULT_ROOT = "/mnt/fnOS/123"
|
||||||
|
REQUEST_TIMEOUT = (5, 30)
|
||||||
|
|
||||||
|
STATUS_TEXT = {
|
||||||
|
0: "stopped",
|
||||||
|
1: "check-wait",
|
||||||
|
2: "checking",
|
||||||
|
3: "download-wait",
|
||||||
|
4: "downloading",
|
||||||
|
5: "seed-wait",
|
||||||
|
6: "seeding",
|
||||||
|
}
|
||||||
|
# 仍在活动的状态:这些任务缺文件会一直空转/报错,最值得处理
|
||||||
|
ACTIVE_STATUS = {1, 2, 3, 4, 5, 6}
|
||||||
|
|
||||||
|
TORRENT_FIELDS = ["id", "name", "status", "error", "errorString", "downloadDir", "files"]
|
||||||
|
|
||||||
|
|
||||||
|
class TransmissionRpc:
|
||||||
|
"""最小可用的 Transmission RPC 客户端(处理 409 会话握手 + Basic 认证)"""
|
||||||
|
|
||||||
|
def __init__(self, host, port, rpc_path, username, password):
|
||||||
|
self.url = f"http://{host}:{port}{rpc_path}"
|
||||||
|
self.session = requests.Session()
|
||||||
|
if username and password:
|
||||||
|
self.session.auth = (username, password)
|
||||||
|
self.session_id = None
|
||||||
|
|
||||||
|
def call(self, method, arguments=None):
|
||||||
|
"""发一次 RPC;409 时取 session id 重发一次。返回解析后的 JSON 或 None"""
|
||||||
|
payload = {"method": method}
|
||||||
|
if arguments:
|
||||||
|
payload["arguments"] = arguments
|
||||||
|
|
||||||
|
for attempt in range(2):
|
||||||
|
headers = {}
|
||||||
|
if self.session_id:
|
||||||
|
headers["X-Transmission-Session-Id"] = self.session_id
|
||||||
|
try:
|
||||||
|
response = self.session.post(self.url, json=payload, headers=headers,
|
||||||
|
timeout=REQUEST_TIMEOUT)
|
||||||
|
except Exception as err:
|
||||||
|
print(f"RPC 请求异常({method}): {err}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
|
||||||
|
if response.status_code == 409 and attempt == 0:
|
||||||
|
new_id = response.headers.get("X-Transmission-Session-Id")
|
||||||
|
if new_id:
|
||||||
|
self.session_id = new_id
|
||||||
|
continue
|
||||||
|
print("Transmission 会话握手失败(409 且无 session id)", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
|
||||||
|
if response.status_code == 401:
|
||||||
|
print("Transmission 认证失败(401),检查用户名/密码", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
if response.status_code != 200:
|
||||||
|
print(f"Transmission 返回非 200: {response.status_code}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
|
||||||
|
try:
|
||||||
|
body = response.json()
|
||||||
|
except ValueError:
|
||||||
|
print("Transmission 响应不是合法 JSON", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
if body.get("result") != "success":
|
||||||
|
print(f"Transmission 业务失败: result={body.get('result')!r}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
return body.get("arguments") or {}
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser():
|
||||||
|
parser = argparse.ArgumentParser(description="只读盘点 Transmission 中缺数据的任务")
|
||||||
|
parser.add_argument("--root", default=DEFAULT_ROOT, help=f"本机下载目录(默认 {DEFAULT_ROOT})")
|
||||||
|
parser.add_argument("--host", default=DEFAULT_HOST)
|
||||||
|
parser.add_argument("--port", type=int, default=DEFAULT_PORT)
|
||||||
|
parser.add_argument("--rpc-path", default=DEFAULT_RPC_PATH)
|
||||||
|
parser.add_argument("--user", default=DEFAULT_USER)
|
||||||
|
parser.add_argument("--password", default=DEFAULT_PASSWORD)
|
||||||
|
parser.add_argument("--json", action="store_true", help="以 JSON 输出")
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
args = build_parser().parse_args(argv)
|
||||||
|
|
||||||
|
if not os.path.isdir(args.root):
|
||||||
|
print(f"下载目录不存在: {args.root}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
on_disk = {name.casefold() for _, _, files in os.walk(args.root) for name in files}
|
||||||
|
|
||||||
|
rpc = TransmissionRpc(args.host, args.port, args.rpc_path, args.user, args.password)
|
||||||
|
arguments = rpc.call("torrent-get", {"fields": TORRENT_FIELDS})
|
||||||
|
if arguments is None:
|
||||||
|
return 1
|
||||||
|
|
||||||
|
torrents = arguments.get("torrents", [])
|
||||||
|
gone, partial = [], []
|
||||||
|
for torrent in torrents:
|
||||||
|
files = torrent.get("files") or []
|
||||||
|
if not files:
|
||||||
|
continue
|
||||||
|
present = sum(1 for f in files if os.path.basename(f["name"]).casefold() in on_disk)
|
||||||
|
if present == 0:
|
||||||
|
gone.append(torrent)
|
||||||
|
elif present < len(files):
|
||||||
|
partial.append((present, len(files), torrent))
|
||||||
|
|
||||||
|
if args.json:
|
||||||
|
print(json.dumps({
|
||||||
|
"root": args.root,
|
||||||
|
"total_torrents": len(torrents),
|
||||||
|
"all_missing": [{"id": t["id"], "name": t["name"],
|
||||||
|
"status": STATUS_TEXT.get(t["status"], t["status"]),
|
||||||
|
"error": t["error"], "files": len(t.get("files") or []),
|
||||||
|
"active": t["status"] in ACTIVE_STATUS} for t in gone],
|
||||||
|
"partially_missing": [{"id": t["id"], "name": t["name"],
|
||||||
|
"present": present, "files": total,
|
||||||
|
"status": STATUS_TEXT.get(t["status"], t["status"]),
|
||||||
|
"active": t["status"] in ACTIVE_STATUS}
|
||||||
|
for present, total, t in partial],
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
active_gone = sum(1 for t in gone if t["status"] in ACTIVE_STATUS)
|
||||||
|
print(f"下载目录: {args.root}")
|
||||||
|
print(f"Transmission 任务总数: {len(torrents)}")
|
||||||
|
print(f"数据文件已全部不在磁盘上的任务: {len(gone)} 个(其中仍在活动状态: {active_gone} 个)")
|
||||||
|
for t in gone:
|
||||||
|
mark = "!" if t["status"] in ACTIVE_STATUS else " "
|
||||||
|
print(f" {mark} id={t['id']:<5} {STATUS_TEXT.get(t['status'], t['status']):<13} "
|
||||||
|
f"error={t['error']:<2} 文件数={len(t.get('files') or []):<3} {t['name'][:56]}")
|
||||||
|
|
||||||
|
print(f"\n部分文件缺失的任务: {len(partial)} 个")
|
||||||
|
for present, total, t in sorted(partial, key=lambda r: r[0] / r[1]):
|
||||||
|
mark = "!" if t["status"] in ACTIVE_STATUS else " "
|
||||||
|
print(f" {mark} id={t['id']:<5} 缺 {total - present}/{total} "
|
||||||
|
f"{STATUS_TEXT.get(t['status'], t['status']):<13} error={t['error']:<2} {t['name'][:56]}")
|
||||||
|
|
||||||
|
print("\n标记 ! 的任务仍在活动状态(会空转/报错),最值得处理。")
|
||||||
|
print("要移除请先让用户确认,再执行 torrent-remove(只删任务记录,不删数据)。")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -6,3 +6,6 @@ __pycache__/
|
|||||||
venv/
|
venv/
|
||||||
cf_cookies.json
|
cf_cookies.json
|
||||||
.idea/
|
.idea/
|
||||||
|
|
||||||
|
# 清理脚本生成的待删清单(仅本地复核用)
|
||||||
|
clean_manifest*.txt
|
||||||
|
|||||||
@@ -0,0 +1,589 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
种子下载目录清理脚本:只保留「大体积视频 + 字幕 + 与视频同名的海报」,其余全部删除。
|
||||||
|
|
||||||
|
背景:
|
||||||
|
主程序 (LookingWebMain.py) 只能把 magnet 交给 Transmission,无法挑选文件,
|
||||||
|
种子里的广告(twojav.com@*.url、宣传页、样片等)会被一起下载下来。本脚本在
|
||||||
|
下载完成后单独运行,按「文件类型 + 体积」做一次清理,与主程序互不依赖。
|
||||||
|
|
||||||
|
保留规则(命中任意一条即保留):
|
||||||
|
1. 视频: 扩展名属于 VIDEO_EXTS 且体积 >= --min-size-mb(默认 100MB)。
|
||||||
|
扩展名未知/没有扩展名时,读取文件头 (magic bytes) 嗅探,识别为视频容器也保留。
|
||||||
|
2. 字幕: 文件名中出现字幕扩展名(.srt/.ass/.ssa/.vtt/.sub/.idx/.sup/.smi 等),
|
||||||
|
含 `xxx.CN.srt.old-20260918` 这类字幕备份。
|
||||||
|
3. 海报: 图片扩展名,且文件名包含「同目录下某个已保留视频」的文件名主干
|
||||||
|
(如 `ATID-643 ... 松井日奈子-poster.jpg` 命中同名视频,`OTHER-999-poster.jpg` 不命中)。
|
||||||
|
4. 通用封面图: `folder.jpg` / `poster.jpg` / `cover.jpg` / `backdrop.jpg` / `fanart.jpg`
|
||||||
|
等媒体库约定的封面名,且**同目录下确实还有保留的视频**(避免留下只有封面的空壳目录)。
|
||||||
|
不想要这一类加 `--no-covers`。
|
||||||
|
|
||||||
|
删除规则:
|
||||||
|
以上四条都不命中的文件一律删除;另外**同一目录内内容相同的大视频只保留一份**
|
||||||
|
(同一部作品被多个发布站各下一份时,文件名只差站点前缀,见 `--no-dedupe`)。
|
||||||
|
删除完成后再自底向上清理空目录:子目录先删,父目录随即变空则继续向上删
|
||||||
|
(例如 `影片目录/xxx.trickplay/320 - 10x10/` 会整条链删掉),
|
||||||
|
包括整个影片级目录;但**不会删除 --root 本身**。
|
||||||
|
|
||||||
|
安全性:
|
||||||
|
- 默认是**预演模式**,只统计和打印,不删除任何文件;确认无误后加 --apply 才真正删除。
|
||||||
|
- 不跟随符号链接目录;删除符号链接时只删链接本身,不会删到链接指向的目标。
|
||||||
|
- 只处理 --root 目录内部;空目录清理不会删除 root 本身。
|
||||||
|
|
||||||
|
用法:
|
||||||
|
python clean_seed_extras.py # 预演(默认,安全)
|
||||||
|
python clean_seed_extras.py --manifest clean_manifest.txt # 导出完整待删清单
|
||||||
|
python clean_seed_extras.py --apply # 真正删除
|
||||||
|
python clean_seed_extras.py --apply --no-dedupe # 不做重复视频去重
|
||||||
|
python clean_seed_extras.py --apply --list-all # 删除并列出每个被删文件
|
||||||
|
python clean_seed_extras.py --apply --no-subtitles # 连字幕一起删,只留视频+海报
|
||||||
|
python clean_seed_extras.py --apply --no-posters # 不保留海报
|
||||||
|
python clean_seed_extras.py --min-size-mb 50 # 视频体积阈值改成 50MB
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from collections import Counter
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
# 默认清理目标:本机下载目录
|
||||||
|
DEFAULT_ROOT = "/mnt/fnOS/123/"
|
||||||
|
# “大视频”阈值(MB)。>= 该体积的视频才保留
|
||||||
|
DEFAULT_MIN_SIZE_MB = 100
|
||||||
|
# 重复视频指纹读取字节数:只哈希「头部 N + 尾部 N + 体积」
|
||||||
|
# (大文件全量哈希要读 TB 级数据,不值得;同源发布的同名文件头尾一致即判定重复)
|
||||||
|
DEFAULT_DUP_CHECK_BYTES = 4 * 1024 * 1024
|
||||||
|
|
||||||
|
# 站点发布前缀,如 masex.tv@xxx.mp4、hkbisi.com@xxx.mp4
|
||||||
|
SITE_PREFIX_RE = re.compile(r"^([^@/\\]{1,40})@")
|
||||||
|
|
||||||
|
# 视频扩展名(尽量覆盖常见容器/编码格式;扩展名未知时另有文件头嗅探兜底)
|
||||||
|
VIDEO_EXTS = {
|
||||||
|
# 常见容器
|
||||||
|
".mp4", ".m4v", ".mkv", ".webm", ".avi", ".wmv", ".asf", ".mov", ".qt",
|
||||||
|
".flv", ".f4v", ".rm", ".rmvb", ".divx", ".xvid", ".ogv", ".ogm", ".dv",
|
||||||
|
".3gp", ".3g2", ".mxf", ".vob", ".dat",
|
||||||
|
# 传输流 / 高清录制
|
||||||
|
".ts", ".m2ts", ".mts", ".tp", ".trp",
|
||||||
|
# MPEG 系
|
||||||
|
".mpg", ".mpeg", ".mpe", ".m1v", ".m2v", ".mpv",
|
||||||
|
# 裸流 / 编码名
|
||||||
|
".h264", ".h265", ".hevc", ".av1", ".vc1", ".avs",
|
||||||
|
# 光盘镜像(多为原盘/BDMV 打包,常含正片)
|
||||||
|
".iso", ".img", ".ifo",
|
||||||
|
}
|
||||||
|
|
||||||
|
# 字幕:用「文件名中出现字幕扩展名」判断,这样 xxx.srt.old-20260918 这类备份也会保留
|
||||||
|
SUBTITLE_IN_NAME_RE = re.compile(
|
||||||
|
r"\.(srt|ass|ssa|vtt|sub|idx|sup|smi|sami|ttml|dfxp|sbv|usf)(\.|$)",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 图片(海报候选)
|
||||||
|
IMAGE_EXTS = {".jpg", ".jpeg", ".jfif", ".png", ".webp", ".bmp", ".gif", ".tbn"}
|
||||||
|
|
||||||
|
# 媒体库通用封面图名(Jellyfin/Emby/Kodi 约定)。这些图不包含视频文件名,
|
||||||
|
# 但会被当作影片封面使用,因此默认保留(仅当同目录还有保留的视频时)。
|
||||||
|
COVER_IMAGE_NAMES = {
|
||||||
|
"folder.jpg", "folder.png", "folder.jpeg",
|
||||||
|
"poster.jpg", "poster.png", "poster.jpeg",
|
||||||
|
"cover.jpg", "cover.png", "cover.jpeg",
|
||||||
|
"default.jpg", "default.png",
|
||||||
|
"movie.jpg", "movie.png", "show.jpg", "show.png",
|
||||||
|
"backdrop.jpg", "backdrop.png", "fanart.jpg", "fanart.png",
|
||||||
|
"banner.jpg", "banner.png", "thumb.jpg", "thumb.png",
|
||||||
|
"landscape.jpg", "landscape.png", "disc.jpg", "disc.png",
|
||||||
|
"logo.png", "clearlogo.png", "clearart.png",
|
||||||
|
}
|
||||||
|
|
||||||
|
# 视频文件头嗅探只读取的字节数
|
||||||
|
SNIFF_BYTES = 512
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Entry:
|
||||||
|
"""扫描到的单个文件"""
|
||||||
|
path: str
|
||||||
|
dir: str
|
||||||
|
name: str
|
||||||
|
size: int
|
||||||
|
ext: str
|
||||||
|
is_link: bool
|
||||||
|
keep: bool = False
|
||||||
|
reason: str = ""
|
||||||
|
dup_of: str = "" # 被判定为重复时,记录保留的那一份的路径
|
||||||
|
|
||||||
|
|
||||||
|
def site_prefix(name):
|
||||||
|
"""取文件名开头的站点前缀(无前缀返回空字符串)"""
|
||||||
|
match = SITE_PREFIX_RE.match(name)
|
||||||
|
return match.group(1).lower() if match else ""
|
||||||
|
|
||||||
|
|
||||||
|
def file_fingerprint(path, check_bytes):
|
||||||
|
"""文件指纹 = sha256(体积 + 头部 check_bytes + 尾部 check_bytes)
|
||||||
|
|
||||||
|
不读中间部分,避免为了校验重复去读 TB 级数据;对“同一源发布两次”的文件足够可靠。
|
||||||
|
"""
|
||||||
|
size = os.path.getsize(path)
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
digest.update(str(size).encode())
|
||||||
|
with open(path, "rb") as fp:
|
||||||
|
digest.update(fp.read(check_bytes))
|
||||||
|
if size > check_bytes * 2:
|
||||||
|
fp.seek(-check_bytes, os.SEEK_END)
|
||||||
|
digest.update(fp.read(check_bytes))
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def plan_duplicate_videos(entries, min_size, check_bytes, prefer_source=()):
|
||||||
|
"""找出同目录内内容相同的大视频,返回 [(保留项, [重复项, ...]), ...]
|
||||||
|
|
||||||
|
先按「同目录 + 同体积」分组(几乎不耗 IO),再对每组算指纹确认内容一致,
|
||||||
|
指纹不同(体积相同但内容不同)的一律不动。
|
||||||
|
|
||||||
|
保留哪一份(依次比较):
|
||||||
|
1. 无站点前缀(文件名最干净);
|
||||||
|
2. --prefer-source 里列出的站点;
|
||||||
|
3. 同目录内该前缀出现次数多的(与目录内命名保持一致);
|
||||||
|
4. 文件名短、字典序。
|
||||||
|
"""
|
||||||
|
prefer_source = tuple(p.lower() for p in prefer_source)
|
||||||
|
|
||||||
|
# 同目录内各站点前缀的出现次数(仅统计保留的视频)
|
||||||
|
prefix_count = {}
|
||||||
|
for entry in entries:
|
||||||
|
if entry.keep and entry.reason.startswith("视频"):
|
||||||
|
key = (entry.dir, site_prefix(entry.name))
|
||||||
|
prefix_count[key] = prefix_count.get(key, 0) + 1
|
||||||
|
|
||||||
|
def rank(entry):
|
||||||
|
prefix = site_prefix(entry.name)
|
||||||
|
if not prefix:
|
||||||
|
order = 0
|
||||||
|
elif prefix in prefer_source:
|
||||||
|
order = 1 + prefer_source.index(prefix)
|
||||||
|
else:
|
||||||
|
order = 1 + len(prefer_source)
|
||||||
|
popularity = -prefix_count.get((entry.dir, prefix), 0)
|
||||||
|
return (order, popularity, len(entry.name), entry.name.casefold())
|
||||||
|
|
||||||
|
# 候选:会被保留的视频(跳过符号链接,不拿链接做删除依据)
|
||||||
|
groups = {}
|
||||||
|
for entry in entries:
|
||||||
|
if entry.keep and entry.reason.startswith("视频") \
|
||||||
|
and entry.size >= min_size and not entry.is_link:
|
||||||
|
groups.setdefault((entry.dir, entry.size), []).append(entry)
|
||||||
|
|
||||||
|
plans = []
|
||||||
|
for items in groups.values():
|
||||||
|
if len(items) < 2:
|
||||||
|
continue
|
||||||
|
by_fingerprint = {}
|
||||||
|
for entry in items:
|
||||||
|
try:
|
||||||
|
fingerprint = file_fingerprint(entry.path, check_bytes)
|
||||||
|
except OSError as err:
|
||||||
|
logger.warning("无法读取文件内容,跳过去重: {} ({})", entry.path, err)
|
||||||
|
continue
|
||||||
|
by_fingerprint.setdefault(fingerprint, []).append(entry)
|
||||||
|
|
||||||
|
for same in by_fingerprint.values():
|
||||||
|
if len(same) < 2:
|
||||||
|
continue
|
||||||
|
same.sort(key=rank)
|
||||||
|
plans.append((same[0], same[1:]))
|
||||||
|
return plans
|
||||||
|
|
||||||
|
|
||||||
|
def looks_like_video(path):
|
||||||
|
"""按文件头判断是否为视频容器(用于扩展名未知/无扩展名的大文件)
|
||||||
|
|
||||||
|
只做「像视频」的保守判断:无法确定时返回 False(宁可留下垃圾,也不误删视频)。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
with open(path, "rb") as fp:
|
||||||
|
head = fp.read(SNIFF_BYTES)
|
||||||
|
except OSError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
if len(head) >= 12 and head[4:8] == b"ftyp": # mp4 / mov / m4v / 3gp
|
||||||
|
return True
|
||||||
|
if head.startswith(b"\x1a\x45\xdf\xa3"): # mkv / webm (EBML)
|
||||||
|
return True
|
||||||
|
if head.startswith(b"RIFF") and head[8:12] == b"AVI ": # avi
|
||||||
|
return True
|
||||||
|
if head.startswith(b"FLV\x01"): # flv
|
||||||
|
return True
|
||||||
|
if head.startswith(b"\x30\x26\xb2\x75\x8e\x66\xcf\x11"): # asf / wmv
|
||||||
|
return True
|
||||||
|
if head.startswith(b"\x00\x00\x01\xba"): # mpeg program stream
|
||||||
|
return True
|
||||||
|
if head.startswith(b"\x00\x00\x01\xb3"): # mpeg video es
|
||||||
|
return True
|
||||||
|
if head.startswith(b".RMF"): # rm / rmvb
|
||||||
|
return True
|
||||||
|
# mpeg-ts:同步字节 0x47,间隔 188 再校验一次,避免误判
|
||||||
|
if len(head) > 188 and head[0:1] == b"\x47" and head[188:189] == b"\x47":
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def collect(root, skip_recent_min=0):
|
||||||
|
"""遍历 root,收集文件与目录清单(不跟随符号链接目录)"""
|
||||||
|
entries = []
|
||||||
|
dirs = []
|
||||||
|
now = time.time()
|
||||||
|
|
||||||
|
for dirpath, dirnames, filenames in os.walk(root, followlinks=False):
|
||||||
|
dirs.append(dirpath)
|
||||||
|
# 符号链接目录不进入,避免跨出目标目录删除外部数据
|
||||||
|
dirnames[:] = [d for d in dirnames if not os.path.islink(os.path.join(dirpath, d))]
|
||||||
|
|
||||||
|
for name in filenames:
|
||||||
|
path = os.path.join(dirpath, name)
|
||||||
|
is_link = os.path.islink(path)
|
||||||
|
try:
|
||||||
|
st = os.stat(path) # 跟随链接取体积
|
||||||
|
size, mtime = st.st_size, st.st_mtime
|
||||||
|
except OSError as err:
|
||||||
|
logger.warning("无法读取文件信息,跳过: {} ({})", path, err)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if skip_recent_min and now - mtime < skip_recent_min * 60:
|
||||||
|
logger.warning("文件最近被修改过,跳过(可能仍在下载): {}", path)
|
||||||
|
continue
|
||||||
|
|
||||||
|
entries.append(Entry(path, dirpath, name, size,
|
||||||
|
os.path.splitext(name)[1].lower(), is_link))
|
||||||
|
|
||||||
|
return entries, dirs
|
||||||
|
|
||||||
|
|
||||||
|
def decide(entries, min_size, keep_subtitles, keep_posters, keep_covers=True,
|
||||||
|
dedupe=True, check_bytes=DEFAULT_DUP_CHECK_BYTES, prefer_source=()):
|
||||||
|
"""标记每个文件保留/删除(就地修改 entries)
|
||||||
|
|
||||||
|
处理顺序:先定下要保留的视频 → 再去重 → 最后才匹配字幕/海报/封面。
|
||||||
|
去重必须在海报匹配之前,否则「被删副本的同名海报」会留到下一轮才被清掉,
|
||||||
|
脚本就不是一次收敛。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
keep_covers: 是否保留 folder.jpg / poster.jpg 等通用封面图
|
||||||
|
dedupe: 是否对同目录内内容相同的大视频去重(只留一份)
|
||||||
|
"""
|
||||||
|
# 第一轮:找出所有达到阈值的视频
|
||||||
|
for entry in entries:
|
||||||
|
if entry.size < min_size:
|
||||||
|
continue
|
||||||
|
if entry.ext in VIDEO_EXTS:
|
||||||
|
entry.keep, entry.reason = True, "视频"
|
||||||
|
elif looks_like_video(entry.path):
|
||||||
|
entry.keep, entry.reason = True, "视频(按文件头识别)"
|
||||||
|
|
||||||
|
# 第二轮:同目录内内容相同的大视频只保留一份
|
||||||
|
if dedupe:
|
||||||
|
for kept, duplicates in plan_duplicate_videos(entries, min_size, check_bytes, prefer_source):
|
||||||
|
for entry in duplicates:
|
||||||
|
entry.keep = False
|
||||||
|
entry.reason = "重复视频"
|
||||||
|
entry.dup_of = kept.path
|
||||||
|
logger.info("重复视频: 保留 {},删除 {} 个内容相同的副本({})",
|
||||||
|
kept.name, len(duplicates), human_size(sum(e.size for e in duplicates)))
|
||||||
|
|
||||||
|
# 去重完成后,按「真正留下来的视频」建索引,供海报/封面匹配
|
||||||
|
video_stems = {}
|
||||||
|
for entry in entries:
|
||||||
|
if entry.keep and entry.reason.startswith("视频"):
|
||||||
|
video_stems.setdefault(entry.dir, []).append(os.path.splitext(entry.name)[0].casefold())
|
||||||
|
# 长主干优先,避免短名字误命中别的海报
|
||||||
|
for stems in video_stems.values():
|
||||||
|
stems.sort(key=len, reverse=True)
|
||||||
|
|
||||||
|
# 第三轮:字幕、同名海报、通用封面
|
||||||
|
for entry in entries:
|
||||||
|
if entry.keep or entry.reason: # 已定案(保留或已判为重复视频)的不再覆盖
|
||||||
|
continue
|
||||||
|
if keep_subtitles and SUBTITLE_IN_NAME_RE.search(entry.name):
|
||||||
|
entry.keep, entry.reason = True, "字幕"
|
||||||
|
continue
|
||||||
|
if keep_posters and entry.ext in IMAGE_EXTS:
|
||||||
|
# 同目录必须还有保留的视频,否则封面图本身也成了无主垃圾
|
||||||
|
if video_stems.get(entry.dir):
|
||||||
|
if keep_covers and entry.name.casefold() in COVER_IMAGE_NAMES:
|
||||||
|
entry.keep, entry.reason = True, "通用封面"
|
||||||
|
continue
|
||||||
|
stem = os.path.splitext(entry.name)[0].casefold()
|
||||||
|
if any(v in stem for v in video_stems[entry.dir]):
|
||||||
|
entry.keep, entry.reason = True, "海报"
|
||||||
|
continue
|
||||||
|
entry.reason = "待删除"
|
||||||
|
|
||||||
|
|
||||||
|
def delete_files(entries):
|
||||||
|
"""删除标记为待删的文件,返回 (删除数, 释放字节, 失败数)"""
|
||||||
|
deleted = freed = failed = 0
|
||||||
|
for entry in entries:
|
||||||
|
if entry.keep:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
os.remove(entry.path) # 符号链接只删链接本身
|
||||||
|
except OSError as err:
|
||||||
|
failed += 1
|
||||||
|
logger.warning("删除失败: {} ({})", entry.path, err)
|
||||||
|
continue
|
||||||
|
deleted += 1
|
||||||
|
freed += entry.size
|
||||||
|
logger.debug("已删除: {}", entry.path)
|
||||||
|
return deleted, freed, failed
|
||||||
|
|
||||||
|
|
||||||
|
def plan_empty_dirs(entries, dirs, root):
|
||||||
|
"""计算删除后会被清空的目录(自底向上级联,不含 root 本身)
|
||||||
|
|
||||||
|
逻辑上与真实删除一致:子目录先判定,父目录在子目录全被清空后也随之清空,
|
||||||
|
因此能一路级联到影片级目录。预演(--manifest)和执行使用同一份结果。
|
||||||
|
"""
|
||||||
|
real_root = os.path.realpath(root)
|
||||||
|
has_kept = {e.dir for e in entries if e.keep}
|
||||||
|
children = {}
|
||||||
|
for d in dirs:
|
||||||
|
if os.path.realpath(d) != real_root and not os.path.islink(d):
|
||||||
|
children.setdefault(os.path.dirname(d), []).append(d)
|
||||||
|
|
||||||
|
alive = set()
|
||||||
|
for d in sorted(dirs, key=lambda p: p.count(os.sep), reverse=True):
|
||||||
|
if d in has_kept or any(ch in alive for ch in children.get(d, ())):
|
||||||
|
alive.add(d)
|
||||||
|
|
||||||
|
return [d for d in dirs
|
||||||
|
if d not in alive and os.path.realpath(d) != real_root and not os.path.islink(d)]
|
||||||
|
|
||||||
|
|
||||||
|
def remove_empty_dirs(paths):
|
||||||
|
"""删除 plan_empty_dirs 规划出的空目录(保留 root 本身)"""
|
||||||
|
removed = 0
|
||||||
|
for path in sorted(paths, key=lambda p: p.count(os.sep), reverse=True):
|
||||||
|
try:
|
||||||
|
if not os.listdir(path): # 文件删除失败时目录不为空,这里会自然跳过
|
||||||
|
os.rmdir(path)
|
||||||
|
removed += 1
|
||||||
|
logger.debug("已删除空目录: {}", path)
|
||||||
|
except OSError as err:
|
||||||
|
logger.debug("空目录清理跳过: {} ({})", path, err)
|
||||||
|
return removed
|
||||||
|
|
||||||
|
|
||||||
|
def write_manifest(path, entries, empty_dirs, root, min_size_mb):
|
||||||
|
"""把待删清单写入文件,便于逐段复核(不执行任何删除)"""
|
||||||
|
to_delete = [e for e in entries if not e.keep]
|
||||||
|
duplicates = [e for e in to_delete if e.dup_of]
|
||||||
|
others = [e for e in to_delete if not e.dup_of]
|
||||||
|
|
||||||
|
buckets = [
|
||||||
|
("A. 小体积视频(未达阈值)",
|
||||||
|
[e for e in others if e.ext in VIDEO_EXTS]),
|
||||||
|
("B. 无扩展名非视频(BT 填充 .pad 等)",
|
||||||
|
[e for e in others if not e.ext]),
|
||||||
|
("C. 图片(非同名海报/封面)",
|
||||||
|
[e for e in others if e.ext in IMAGE_EXTS]),
|
||||||
|
("D. 广告/文本/其他",
|
||||||
|
[e for e in others if e.ext and e.ext not in VIDEO_EXTS and e.ext not in IMAGE_EXTS]),
|
||||||
|
]
|
||||||
|
|
||||||
|
total_files = len(to_delete)
|
||||||
|
total_bytes = sum(e.size for e in to_delete)
|
||||||
|
lines = [
|
||||||
|
"# clean_seed_extras 待删除清单(本文件只是清单,不代表已删除)",
|
||||||
|
f"# 生成时间: {time.strftime('%Y-%m-%d %H:%M:%S')}",
|
||||||
|
f"# 根目录 : {root}",
|
||||||
|
f"# 视频阈值: >= {min_size_mb:g} MB",
|
||||||
|
f"# 合计 : {total_files} 个文件({human_size(total_bytes)}) + {len(empty_dirs)} 个空目录",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
for title, items in buckets:
|
||||||
|
items.sort(key=lambda e: -e.size)
|
||||||
|
lines.append(f"===== {title} — {len(items)} 个,"
|
||||||
|
f"{human_size(sum(e.size for e in items))} =====")
|
||||||
|
lines.extend(f"{e.size:>13} {e.path}" for e in items)
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
# 重复视频:按“保留哪一份”分组展示,方便核对
|
||||||
|
groups = {}
|
||||||
|
for entry in duplicates:
|
||||||
|
groups.setdefault(entry.dup_of, []).append(entry)
|
||||||
|
lines.append(f"===== E. 重复大视频(同目录内容相同,每组只保留一份) — "
|
||||||
|
f"{len(duplicates)} 个,{human_size(sum(e.size for e in duplicates))} =====")
|
||||||
|
for kept_path, dups in sorted(groups.items()):
|
||||||
|
dups.sort(key=lambda e: -e.size)
|
||||||
|
lines.append(f"--- 保留: {kept_path} (删除 {len(dups)} 份副本,"
|
||||||
|
f"释放 {human_size(sum(e.size for e in dups))})")
|
||||||
|
lines.extend(f"{e.size:>13} {e.path}" for e in dups)
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
lines.append(f"===== F. 待删除的空目录 — {len(empty_dirs)} 个 =====")
|
||||||
|
lines.extend(f"DIR {d}" for d in sorted(empty_dirs))
|
||||||
|
|
||||||
|
with open(path, "w", encoding="utf-8") as fp:
|
||||||
|
fp.write("\n".join(lines) + "\n")
|
||||||
|
logger.info("待删清单已写入: {}({} 个文件 + {} 个空目录,{})",
|
||||||
|
os.path.abspath(path), total_files, len(empty_dirs), human_size(total_bytes))
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def human_size(num_bytes):
|
||||||
|
"""字节数转可读体积"""
|
||||||
|
value = float(num_bytes)
|
||||||
|
for unit in ("B", "KB", "MB", "GB", "TB"):
|
||||||
|
if value < 1024 or unit == "TB":
|
||||||
|
return f"{value:.1f} {unit}"
|
||||||
|
value /= 1024
|
||||||
|
return f"{value:.1f} TB"
|
||||||
|
|
||||||
|
|
||||||
|
def print_summary(entries, root, min_size_mb, dry_run, deleted, freed, failed, removed_dirs,
|
||||||
|
list_all, list_limit):
|
||||||
|
"""打印统计结果"""
|
||||||
|
keep_counter = Counter(e.reason for e in entries if e.keep)
|
||||||
|
del_counter = Counter(e.ext or "(无扩展名)" for e in entries if not e.keep)
|
||||||
|
duplicates = [e for e in entries if not e.keep and e.dup_of]
|
||||||
|
|
||||||
|
print("\n" + "=" * 68)
|
||||||
|
print(f"目录: {root}")
|
||||||
|
print(f"视频阈值: >= {min_size_mb:g} MB")
|
||||||
|
print("-" * 68)
|
||||||
|
print(f"扫描文件总数: {len(entries)} 保留: {len(entries) - deleted} 删除: {deleted}")
|
||||||
|
if keep_counter:
|
||||||
|
print("保留明细: " + ",".join(f"{k} {v} 个" for k, v in keep_counter.most_common()))
|
||||||
|
if del_counter:
|
||||||
|
print("删除明细(按扩展名): " + ",".join(f"{k} {v} 个" for k, v in del_counter.most_common(10)))
|
||||||
|
if duplicates:
|
||||||
|
kept_groups = {e.dup_of for e in duplicates}
|
||||||
|
print(f"其中重复大视频: {len(duplicates)} 个({human_size(sum(e.size for e in duplicates))}),"
|
||||||
|
f"共 {len(kept_groups)} 组,每组保留一份")
|
||||||
|
if not dry_run:
|
||||||
|
print(f"释放空间: {human_size(freed)}" + (f" 删除失败: {failed}" if failed else ""))
|
||||||
|
print(f"清理空目录: {removed_dirs} 个")
|
||||||
|
print("=" * 68)
|
||||||
|
|
||||||
|
to_delete = [e for e in entries if not e.keep]
|
||||||
|
if dry_run and to_delete:
|
||||||
|
limit = len(to_delete) if list_all else list_limit
|
||||||
|
print(f"待删除文件(最多显示 {limit} 个,实际 {len(to_delete)} 个):")
|
||||||
|
for entry in to_delete[:limit]:
|
||||||
|
print(f" [删] {human_size(entry.size):>10} {entry.path}")
|
||||||
|
if len(to_delete) > limit:
|
||||||
|
print(f" ... 其余 {len(to_delete) - limit} 个已省略(--list-all 可全部显示)")
|
||||||
|
elif list_all:
|
||||||
|
for entry in to_delete:
|
||||||
|
print(f" [已删] {human_size(entry.size):>10} {entry.path}")
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
print("\n>>> 预演模式,未删除任何文件。确认无误后加 --apply 执行:")
|
||||||
|
print(f">>> python {os.path.basename(__file__)} --apply")
|
||||||
|
else:
|
||||||
|
print(f"\n>>> 清理完成,共删除 {deleted} 个文件,释放 {human_size(freed)}。")
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="清理种子下载目录:只保留大体积视频 + 字幕 + 与视频同名的海报",
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||||
|
)
|
||||||
|
parser.add_argument("--root", default=DEFAULT_ROOT,
|
||||||
|
help=f"清理目标目录(默认 {DEFAULT_ROOT})")
|
||||||
|
parser.add_argument("--min-size-mb", type=float, default=DEFAULT_MIN_SIZE_MB,
|
||||||
|
help=f"视频体积阈值,>= 该值才保留(默认 {DEFAULT_MIN_SIZE_MB})")
|
||||||
|
parser.add_argument("--apply", action="store_true",
|
||||||
|
help="真正执行删除;不加则只预演(默认)")
|
||||||
|
parser.add_argument("--no-subtitles", action="store_true", help="不保留字幕")
|
||||||
|
parser.add_argument("--no-posters", action="store_true", help="不保留任何图片(含同名海报与通用封面)")
|
||||||
|
parser.add_argument("--no-covers", action="store_true",
|
||||||
|
help="只保留与视频同名的海报,删除 folder.jpg/poster.jpg 等通用封面图")
|
||||||
|
parser.add_argument("--no-dedupe", action="store_true",
|
||||||
|
help="不对「同目录内容相同的大视频」去重(默认去重,每组只留一份)")
|
||||||
|
parser.add_argument("--dup-check-bytes", type=int, default=DEFAULT_DUP_CHECK_BYTES,
|
||||||
|
help=f"重复校验读取的字节数(头/尾各读这么多,默认 {DEFAULT_DUP_CHECK_BYTES})")
|
||||||
|
parser.add_argument("--prefer-source", default="",
|
||||||
|
help="优先保留的站点前缀,逗号分隔(如 masex.tv,hkbisi.com)")
|
||||||
|
parser.add_argument("--manifest", metavar="PATH", default=None,
|
||||||
|
help="把待删清单(文件 + 空目录)写入指定文件,便于逐段复核")
|
||||||
|
parser.add_argument("--keep-empty-dirs", action="store_true",
|
||||||
|
help="保留删除后变空的目录(默认会级联删除空目录,但不删 root 本身)")
|
||||||
|
parser.add_argument("--skip-recent-min", type=int, default=0,
|
||||||
|
help="跳过最近 N 分钟内修改过的文件(默认 0,不跳过)")
|
||||||
|
parser.add_argument("--list-all", action="store_true", help="列出全部待删/已删文件")
|
||||||
|
parser.add_argument("--list-limit", type=int, default=200,
|
||||||
|
help="预演时最多显示多少条待删文件(默认 200)")
|
||||||
|
parser.add_argument("--quiet", action="store_true", help="只输出统计,不输出逐条明细")
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
args = build_parser().parse_args(argv)
|
||||||
|
|
||||||
|
logger.remove()
|
||||||
|
logger.add(sys.stderr, format="<level>{level: <7}</level> {message}", level="INFO")
|
||||||
|
|
||||||
|
root = os.path.abspath(os.path.expanduser(args.root))
|
||||||
|
if not os.path.isdir(root):
|
||||||
|
logger.error("目标目录不存在或不是目录: {}", root)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
min_size = int(args.min_size_mb * 1024 * 1024)
|
||||||
|
dry_run = not args.apply
|
||||||
|
|
||||||
|
print("=" * 68)
|
||||||
|
print("种子目录清理" + ("(预演模式,不会删除任何文件)" if dry_run else "(删除模式)"))
|
||||||
|
print("=" * 68)
|
||||||
|
print(f"目标目录 : {root}")
|
||||||
|
print(f"视频阈值 : >= {args.min_size_mb:g} MB")
|
||||||
|
print(f"保留字幕 : {'否' if args.no_subtitles else '是'}")
|
||||||
|
print(f"保留同名海报: {'否' if args.no_posters else '是'}")
|
||||||
|
print(f"保留通用封面: {'否' if (args.no_posters or args.no_covers) else '是'}")
|
||||||
|
print(f"重复视频去重: {'否' if args.no_dedupe else '是'}")
|
||||||
|
print(f"清理空目录 : {'否' if args.keep_empty_dirs else '是'}")
|
||||||
|
|
||||||
|
entries, dirs = collect(root, skip_recent_min=args.skip_recent_min)
|
||||||
|
logger.info("扫描完成:{} 个文件,{} 个目录", len(entries), len(dirs))
|
||||||
|
|
||||||
|
prefer_source = tuple(p.strip() for p in args.prefer_source.split(",") if p.strip())
|
||||||
|
decide(entries, min_size,
|
||||||
|
keep_subtitles=not args.no_subtitles,
|
||||||
|
keep_posters=not args.no_posters,
|
||||||
|
keep_covers=not args.no_covers,
|
||||||
|
dedupe=not args.no_dedupe,
|
||||||
|
check_bytes=args.dup_check_bytes,
|
||||||
|
prefer_source=prefer_source)
|
||||||
|
|
||||||
|
# 空目录先规划后执行:预演/清单/真实删除使用同一份结果
|
||||||
|
planned_dirs = [] if args.keep_empty_dirs else plan_empty_dirs(entries, dirs, root)
|
||||||
|
if args.manifest:
|
||||||
|
write_manifest(args.manifest, entries, planned_dirs, root, args.min_size_mb)
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
deleted = sum(1 for e in entries if not e.keep)
|
||||||
|
freed = sum(e.size for e in entries if not e.keep)
|
||||||
|
failed = removed_dirs = 0
|
||||||
|
if args.quiet:
|
||||||
|
args.list_all = False
|
||||||
|
else:
|
||||||
|
deleted, freed, failed = delete_files(entries)
|
||||||
|
removed_dirs = remove_empty_dirs(planned_dirs)
|
||||||
|
|
||||||
|
if args.quiet:
|
||||||
|
args.list_all = False
|
||||||
|
args.list_limit = 0
|
||||||
|
|
||||||
|
print_summary(entries, root, args.min_size_mb, dry_run, deleted, freed, failed,
|
||||||
|
removed_dirs, args.list_all, args.list_limit)
|
||||||
|
return 1 if failed else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
+134
-13
@@ -9,8 +9,8 @@
|
|||||||
- 测试连接对象属性和方法
|
- 测试连接对象属性和方法
|
||||||
- 测试关闭连接
|
- 测试关闭连接
|
||||||
|
|
||||||
### seed_is_exist 测试
|
### seed_is_exist / all_seed_codes 测试
|
||||||
- `test_seed_is_exist.py` - seed_is_exist 功能真实数据库测试
|
- `test_seed_is_exist.py` - 真实数据库测试(含 all_seed_codes 作品级去重)
|
||||||
- 首次运行自动创建 `LookingWeb` 库和 `seeds_info` 表(幂等)
|
- 首次运行自动创建 `LookingWeb` 库和 `seeds_info` 表(幂等)
|
||||||
- 测试种子不存在 → False
|
- 测试种子不存在 → False
|
||||||
- 真实插入种子后 → True
|
- 真实插入种子后 → True
|
||||||
@@ -19,15 +19,58 @@
|
|||||||
- 数据库不可达 → False(不抛异常)
|
- 数据库不可达 → False(不抛异常)
|
||||||
- 测试数据测后自动清理
|
- 测试数据测后自动清理
|
||||||
|
|
||||||
### qBittorrent 下载器测试
|
### 作品级去重与 magnet 校验测试
|
||||||
|
- `test_seed_dedupe.py` - 离线测试(mock 网络/数据库/下载器,22 个用例)
|
||||||
|
- magnet 链接校验:40 位十六进制 / 32 位 base32 合法;拒绝站点“复制模板”行
|
||||||
|
(多个 magnet 用 `__JOIN_ITEM_SEP__magnet_link__` 拼接)与非法 btih
|
||||||
|
(如 `17i6okorc0zvocd3tzfompbktvveifcwt`)
|
||||||
|
- 影片番号解析(标题中的“影片番号为XXX”)与 `SeedInfo.code`
|
||||||
|
- `get_seed_from_element_detail` 只保留合法 magnet
|
||||||
|
- `collect_new_seeds` 两级去重:同一作品(番号)只下载一次、同一链接不重复收集、
|
||||||
|
解析不到番号时退回链接去重
|
||||||
|
- `main()` 端到端:详情页 2 个合法 magnet(同一番号)+ 1 个模板行 → 只下载 1 个;
|
||||||
|
数据库已有该番号 → 一条都不下载
|
||||||
|
|
||||||
|
### qBittorrent 下载器测试(备用下载器)
|
||||||
- `test_qbittorrent_downloader.py` - qBittorrent 下载器单元测试
|
- `test_qbittorrent_downloader.py` - qBittorrent 下载器单元测试
|
||||||
- 测试初始化(带/不带认证)
|
- `test_qbittorrent_downloader.py` - qBittorrent 下载器离线单元测试(19 个用例,全部 mock 网络)
|
||||||
- 测试添加下载任务(成功/失败/异常)
|
- 测试初始化(带/不带认证、登录成功/被拒/异常)
|
||||||
- 测试种子信息下载
|
- 测试添加下载任务(成功 / HTTP 200 + "Fails." 被拒 / 非 200 / 异常 / 空链接)
|
||||||
|
- 测试种子信息下载(含 seed_info 为 None)
|
||||||
- 测试带选项的下载(保存路径、分类、标签)
|
- 测试带选项的下载(保存路径、分类、标签)
|
||||||
- 测试连接测试
|
- 测试连接测试与获取下载列表
|
||||||
- 测试获取下载列表
|
- 说明:mock 目标为 `requests.Session.post/get`;`QbittorrentDownloader.session` 是实例属性,
|
||||||
- 注意:当前 11/12 用例因测试 mock 目标与实现不一致(`QbittorrentDownloader.session` 是实例属性而非模块属性)而失败,属既有问题,待修复。
|
patch 模块属性会 AttributeError。qBittorrent Web API 的成功/失败不体现在 HTTP 状态码,
|
||||||
|
添加 torrent 成功返回 200 + `Ok.`、被拒绝返回 200 + `Fails.`,判定必须读响应体。
|
||||||
|
|
||||||
|
### Transmission 下载器测试(主程序当前使用)
|
||||||
|
- `test_transmission_downloader.py` - Transmission 下载器离线单元测试(26 个用例,全部 mock 网络)
|
||||||
|
- 测试初始化(带/不带认证、409 握手后认证成功、401、非 200、异常)
|
||||||
|
- 测试 409 会话握手:session id 过期自动更新重发、连续 409 不死循环
|
||||||
|
- 测试添加下载任务(成功 / HTTP 200 + `result != success` 被拒 / 重复种子 / 401 / 非 JSON / 异常 / 空链接)
|
||||||
|
- 测试保存路径映射为 `download-dir`、分类与标签映射为 `labels`
|
||||||
|
- 测试种子信息下载(含 seedInfo 为 None、seedUrl 为空)
|
||||||
|
- 测试连接测试与获取下载列表
|
||||||
|
- 说明:下载器连接 `http://192.168.123.199:9095/transmission/rpc`(Transmission Web UI,
|
||||||
|
Basic 认证 admin / GUOxy.5157)。mock 目标为 `requests.Session.post`;
|
||||||
|
Transmission 首次请求返回 409 + `X-Transmission-Session-Id` 响应头,需要带上该头重发;
|
||||||
|
业务失败不体现在 HTTP 状态码上,必须读响应体的 `result` 字段。
|
||||||
|
|
||||||
|
### 下载目录清理脚本测试
|
||||||
|
- `test_clean_seed_extras.py` - 离线单元测试(30 个用例,全部使用 tmp_path 临时目录)
|
||||||
|
- 文件头嗅探 `looks_like_video`:mp4/ftyp、mkv、avi、flv、wmv、mpg、rmvb 识别;
|
||||||
|
mpeg-ts 需两次 0x47 同步字节校验;随机数据判为非视频
|
||||||
|
- 保留规则:>=100MB 视频保留、小体积样片删除、无扩展名大视频按文件头保留
|
||||||
|
- 字幕保留(含 `xxx.CN.srt.old-20260918` 备份);`--no-subtitles` / `--no-posters` 可关闭
|
||||||
|
- 海报只保留与「同目录视频文件名主干」匹配的图片;
|
||||||
|
通用封面(folder.jpg/poster.jpg/cover.jpg 等)默认保留,`--no-covers` 关闭;
|
||||||
|
目录内没有视频时封面图视为无主垃圾一并删除
|
||||||
|
- 重复视频去重:同目录同体积 → 头尾指纹校验 → 只留一份;
|
||||||
|
体积相同但内容不同不去重、不同目录不去重、`--no-dedupe` 关闭、
|
||||||
|
`--prefer-source` 指定优先站点、`dup_of` 记录保留的那一份
|
||||||
|
- 安全行为:默认预演不删除;`--apply` 才删除;空目录级联删除但保留 root;
|
||||||
|
`--keep-empty-dirs` 保留空目录;不跟随符号链接目录;目录不存在返回码 2
|
||||||
|
|
||||||
## 运行测试
|
## 运行测试
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -40,10 +83,36 @@ uv run python -m pytest tests/ -v
|
|||||||
# 运行 MySQLDatabase 测试
|
# 运行 MySQLDatabase 测试
|
||||||
uv run python -m pytest tests/test_database.py -v
|
uv run python -m pytest tests/test_database.py -v
|
||||||
|
|
||||||
# 运行 qBittorrent 测试
|
# 运行 qBittorrent 测试(已不再被主程序使用,保留备用)
|
||||||
uv run python -m pytest tests/test_qbittorrent_downloader.py -v
|
uv run python -m pytest tests/test_qbittorrent_downloader.py -v
|
||||||
|
|
||||||
|
# 运行 Transmission 测试(主程序当前使用)
|
||||||
|
uv run python -m pytest tests/test_transmission_downloader.py -v
|
||||||
|
|
||||||
|
# 运行作品级去重与 magnet 校验测试(离线)
|
||||||
|
uv run python -m pytest tests/test_seed_dedupe.py -v
|
||||||
|
|
||||||
|
# 运行下载目录清理脚本测试(离线)
|
||||||
|
uv run python -m pytest tests/test_clean_seed_extras.py -v
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## 去重规则(已修复的重复下载问题)
|
||||||
|
|
||||||
|
freejavbt 的同一个详情页会为同一部影片发布多个 magnet(如 VRKM-1872 的
|
||||||
|
“5个文件”与“6个文件”,info-hash 不同、内容重叠)。主程序原来只用
|
||||||
|
`seed_url`(完整 magnet 链接)去重,因此同一部影片的其他发布会被当成新种子
|
||||||
|
再次下载。现在改为两级去重(`LookingWebMain.collect_new_seeds`):
|
||||||
|
|
||||||
|
1. **作品级(影片番号)**:标题中的“影片番号为XXX”为去重键,数据库里已有该番号
|
||||||
|
(`MySqlService.all_seed_codes()`)或本次运行已收集过,就跳过;
|
||||||
|
2. **链接级**:完整 magnet 链接(原有 `seed_is_exist` 逻辑),并额外记录本次运行
|
||||||
|
已收集但尚未写库的链接(写库发生在下载之后)。
|
||||||
|
|
||||||
|
解析阶段还会丢弃两类脏数据(`is_valid_seed_url`):站点“复制模板”行拼接的伪 magnet,
|
||||||
|
以及站点发布的非法 btih;这两类链接发给 Transmission 只会被拒收(`have_download=-1`)。
|
||||||
|
|
||||||
|
关闭作品级去重可设置 `LookingWebMain.DEDUPE_BY_SEED_CODE = False`(退回仅按链接去重)。
|
||||||
|
|
||||||
## 环境要求
|
## 环境要求
|
||||||
|
|
||||||
- Python 3.7+
|
- Python 3.7+
|
||||||
@@ -51,11 +120,63 @@ uv run python -m pytest tests/test_qbittorrent_downloader.py -v
|
|||||||
- loguru
|
- loguru
|
||||||
- beautifulsoup4
|
- beautifulsoup4
|
||||||
- pymysql
|
- pymysql
|
||||||
|
- cryptography(MySQL 8 caching_sha2_password 认证必需)
|
||||||
- pytest
|
- pytest
|
||||||
|
|
||||||
## 测试覆盖
|
## 测试覆盖
|
||||||
|
|
||||||
- MySQLDatabase: 4 个测试用例
|
- MySQLDatabase: 4 个测试用例
|
||||||
- seed_is_exist: 5 个测试用例(真实数据库)
|
- seed_is_exist / all_seed_codes: 8 个测试用例(真实数据库)
|
||||||
- qBittorrentDownloader: 12 个测试用例(11 个待修复)
|
- 作品级去重与 magnet 校验: 22 个测试用例(离线 mock)
|
||||||
- 总计: 25 个测试用例
|
- qBittorrentDownloader: 19 个测试用例(离线 mock,备用下载器)
|
||||||
|
- TransmissionDownloader: 26 个测试用例(离线 mock,主程序当前使用)
|
||||||
|
- clean_seed_extras: 30 个测试用例(离线,tmp_path 临时目录)
|
||||||
|
- 其他: cf_session / 主程序函数(真实网络)
|
||||||
|
|
||||||
|
## 下载完成后的清理脚本(clean_seed_extras.py)
|
||||||
|
|
||||||
|
主程序只能把 magnet 交给 Transmission,无法挑选文件,种子内的广告文件会被一并下载。
|
||||||
|
下载完成后单独运行 `clean_seed_extras.py` 清理,保留「大体积视频 + 字幕 + 与视频同名的海报」:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 预演(默认,只统计不删除)
|
||||||
|
uv run python clean_seed_extras.py
|
||||||
|
|
||||||
|
# 导出完整待删清单供逐段复核(不删除)
|
||||||
|
uv run python clean_seed_extras.py --manifest clean_manifest.txt
|
||||||
|
|
||||||
|
# 确认无误后真正删除
|
||||||
|
uv run python clean_seed_extras.py --apply
|
||||||
|
|
||||||
|
# 常用选项
|
||||||
|
uv run python clean_seed_extras.py --apply --list-all # 列出每个被删文件
|
||||||
|
uv run python clean_seed_extras.py --min-size-mb 50 # 大视频阈值改为 50MB
|
||||||
|
uv run python clean_seed_extras.py --apply --no-subtitles # 连字幕一起删
|
||||||
|
uv run python clean_seed_extras.py --apply --no-covers # 不保留 folder.jpg 等通用封面
|
||||||
|
uv run python clean_seed_extras.py --apply --no-dedupe # 不做重复视频去重
|
||||||
|
uv run python clean_seed_extras.py --prefer-source masex.tv,hkbisi.com
|
||||||
|
uv run python clean_seed_extras.py --root /mnt/fnOS/123 # 指定目录(默认值)
|
||||||
|
```
|
||||||
|
|
||||||
|
保留规则:
|
||||||
|
|
||||||
|
1. **视频**:扩展名在 `VIDEO_EXTS` 内且体积 >= `--min-size-mb`(默认 100MB);
|
||||||
|
扩展名未知时按文件头(magic bytes)嗅探识别;
|
||||||
|
2. **字幕**:文件名含字幕扩展名(`.srt/.ass/.ssa/.vtt/.sub/.idx/.sup/.smi` 等),
|
||||||
|
含 `.srt.old-<日期>` 备份;
|
||||||
|
3. **海报**:图片扩展名且文件名包含**同目录**下某个已保留视频的文件名主干;
|
||||||
|
4. **通用封面**:`folder.jpg`/`poster.jpg`/`cover.jpg`/`backdrop.jpg`/`fanart.jpg` 等
|
||||||
|
媒体库约定名(Jellyfin/Emby/Kodi 会当封面用),且同目录下确实还有保留的视频。
|
||||||
|
|
||||||
|
删除规则:其余文件删除;另外**同一目录内内容相同的大视频只保留一份**
|
||||||
|
(同一部作品被多个发布站各下一份时,文件名只差站点前缀,如
|
||||||
|
`masex.tv@vrkm01874_1_8k.mp4` 与 `hkbisi.com@vrkm01874_1_8k.mp4`)。
|
||||||
|
|
||||||
|
去重实现:先按「同目录 + 同体积」分组(几乎不耗 IO),再对每组算
|
||||||
|
`sha256(体积 + 头部 4MB + 尾部 4MB)` 确认内容一致——**体积相同但内容不同的不会误删**。
|
||||||
|
保留哪一份依次比较:无站点前缀 > `--prefer-source` 指定站点 >
|
||||||
|
同目录内该前缀出现次数多 > 文件名短 > 字典序。
|
||||||
|
|
||||||
|
删除文件后自底向上级联清理空目录(子目录先删,父目录变空后继续向上删,
|
||||||
|
可一路删到影片级目录,但不删 `--root` 本身;如 Jellyfin 的 `*.trickplay/` 缩略图目录、
|
||||||
|
BT 的 BEP47 填充目录 `.pad/`)。脚本不跟随符号链接目录,删除符号链接只删链接本身。
|
||||||
@@ -0,0 +1,320 @@
|
|||||||
|
"""clean_seed_extras 清理脚本离线单元测试
|
||||||
|
|
||||||
|
全部使用 tmp_path 临时目录,不触碰真实下载目录(/mnt/fnOS/123)。
|
||||||
|
运行: uv run python -m pytest tests/test_clean_seed_extras.py -v
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import clean_seed_extras as c
|
||||||
|
|
||||||
|
MB = 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
def make_file(path, size, header=b""):
|
||||||
|
"""创建指定体积的文件(稀疏文件,速度快),可选写入文件头"""
|
||||||
|
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||||
|
with open(path, "wb") as fp:
|
||||||
|
if header:
|
||||||
|
fp.write(header)
|
||||||
|
os.truncate(path, size)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def run(root, *extra):
|
||||||
|
"""以预演/删除模式运行脚本主函数"""
|
||||||
|
return c.main(["--root", str(root), "--quiet", *extra])
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 文件头嗅探
|
||||||
|
|
||||||
|
def test_looks_like_video_常见容器(tmp_path):
|
||||||
|
cases = {
|
||||||
|
"a.mp4": b"\x00\x00\x00\x20ftypisom",
|
||||||
|
"b.mkv": b"\x1a\x45\xdf\xa3\x01\x00\x00\x00",
|
||||||
|
"c.avi": b"RIFF\x00\x00\x00\x00AVI LIST",
|
||||||
|
"d.flv": b"FLV\x01\x05\x00\x00\x00",
|
||||||
|
"e.wmv": b"\x30\x26\xb2\x75\x8e\x66\xcf\x11",
|
||||||
|
"f.mpg": b"\x00\x00\x01\xba\x44\x00",
|
||||||
|
"g.rmvb": b".RMF\x00\x00\x00\x12",
|
||||||
|
}
|
||||||
|
for name, header in cases.items():
|
||||||
|
path = make_file(str(tmp_path / name), 200 * MB, header)
|
||||||
|
assert c.looks_like_video(path) is True, name
|
||||||
|
|
||||||
|
|
||||||
|
def test_looks_like_video_ts_需要两次同步字节(tmp_path):
|
||||||
|
ts = make_file(str(tmp_path / "ok.ts"), 200 * MB, b"\x47" + b"\x00" * 187 + b"\x47")
|
||||||
|
assert c.looks_like_video(ts) is True
|
||||||
|
fake = make_file(str(tmp_path / "fake.ts"), 200 * MB, b"\x47" + b"\x00" * 511)
|
||||||
|
assert c.looks_like_video(fake) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_looks_like_video_非视频(tmp_path):
|
||||||
|
junk = make_file(str(tmp_path / "junk.bin"), 200 * MB, os.urandom(64))
|
||||||
|
assert c.looks_like_video(junk) is False
|
||||||
|
assert c.looks_like_video(str(tmp_path / "not-exist")) is False
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ 保留规则
|
||||||
|
|
||||||
|
def test_大视频保留_小视频删除(tmp_path):
|
||||||
|
big = make_file(str(tmp_path / "x/big.mp4"), 120 * MB)
|
||||||
|
small = make_file(str(tmp_path / "x/sample.mp4"), 20 * MB)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(big) is True
|
||||||
|
assert os.path.exists(small) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_无扩展名大视频按文件头保留(tmp_path):
|
||||||
|
video = make_file(str(tmp_path / "noext_video"), 150 * MB, b"\x00\x00\x00\x20ftypisom")
|
||||||
|
junk = make_file(str(tmp_path / "noext_junk"), 150 * MB, os.urandom(64))
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(video) is True
|
||||||
|
assert os.path.exists(junk) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_字幕保留_含old备份(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/movie.mp4"), 120 * MB)
|
||||||
|
subs = [
|
||||||
|
make_file(str(tmp_path / "x/movie.srt"), 1024),
|
||||||
|
make_file(str(tmp_path / "x/movie.CN.srt"), 1024),
|
||||||
|
make_file(str(tmp_path / "x/movie.CN_dual_eye.ass"), 1024),
|
||||||
|
make_file(str(tmp_path / "x/movie.CN.srt.old-20260918"), 1024),
|
||||||
|
]
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert all(os.path.exists(p) for p in subs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_海报仅保留与视频同名的(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/ATID-643 淫魔病棟.mp4"), 120 * MB)
|
||||||
|
keep = make_file(str(tmp_path / "x/ATID-643 淫魔病棟-poster.jpg"), 4096)
|
||||||
|
drop_folder = make_file(str(tmp_path / "x/folder.jpg"), 4096)
|
||||||
|
drop_other = make_file(str(tmp_path / "x/OTHER-999-poster.jpg"), 4096)
|
||||||
|
run(tmp_path, "--apply", "--no-covers") # --no-covers = 严格模式,只留同名海报
|
||||||
|
assert os.path.exists(keep) is True
|
||||||
|
assert os.path.exists(drop_folder) is False
|
||||||
|
assert os.path.exists(drop_other) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_通用封面图默认保留(tmp_path):
|
||||||
|
"""folder.jpg / poster.jpg 等媒体库封面默认保留(同目录有大视频时)"""
|
||||||
|
make_file(str(tmp_path / "x/movie.mp4"), 120 * MB)
|
||||||
|
covers = [
|
||||||
|
make_file(str(tmp_path / "x/folder.jpg"), 4096),
|
||||||
|
make_file(str(tmp_path / "x/poster.jpg"), 4096),
|
||||||
|
make_file(str(tmp_path / "x/backdrop.jpg"), 4096),
|
||||||
|
make_file(str(tmp_path / "x/fanart.png"), 4096),
|
||||||
|
]
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert all(os.path.exists(p) for p in covers)
|
||||||
|
|
||||||
|
|
||||||
|
def test_无视频目录的通用封面不保留(tmp_path):
|
||||||
|
"""只有 folder.jpg 没有视频的目录属于无主垃圾,封面和目录都应删掉"""
|
||||||
|
make_file(str(tmp_path / "shell/folder.jpg"), 4096)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(str(tmp_path / "shell/folder.jpg")) is False
|
||||||
|
assert os.path.exists(str(tmp_path / "shell")) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_跨目录不误匹配海报(tmp_path):
|
||||||
|
"""海报只与同目录的视频匹配,避免 A 目录海报被 B 目录视频保住"""
|
||||||
|
make_file(str(tmp_path / "a/movie.mp4"), 120 * MB)
|
||||||
|
other = make_file(str(tmp_path / "b/movie-poster.jpg"), 4096)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(other) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_关闭字幕与海报保留(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/movie.mp4"), 120 * MB)
|
||||||
|
sub = make_file(str(tmp_path / "x/movie.srt"), 1024)
|
||||||
|
poster = make_file(str(tmp_path / "x/movie-poster.jpg"), 4096)
|
||||||
|
cover = make_file(str(tmp_path / "x/folder.jpg"), 4096)
|
||||||
|
run(tmp_path, "--apply", "--no-subtitles", "--no-posters")
|
||||||
|
assert os.path.exists(sub) is False
|
||||||
|
assert os.path.exists(poster) is False
|
||||||
|
assert os.path.exists(cover) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_阈值可调(tmp_path):
|
||||||
|
small = make_file(str(tmp_path / "x/sample.mp4"), 20 * MB)
|
||||||
|
run(tmp_path, "--apply", "--min-size-mb", "10")
|
||||||
|
assert os.path.exists(small) is True
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ 重复视频去重
|
||||||
|
|
||||||
|
HEADER = b"\x00\x00\x00\x20ftypisom"
|
||||||
|
|
||||||
|
|
||||||
|
def test_重复大视频只保留一份(tmp_path):
|
||||||
|
a = make_file(str(tmp_path / "x/hkbisi.com@abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
b = make_file(str(tmp_path / "x/masex.tv@abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert sum(os.path.exists(p) for p in (a, b)) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_体积相同内容不同不去重(tmp_path):
|
||||||
|
a = make_file(str(tmp_path / "x/a.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
b = make_file(str(tmp_path / "x/b.com@abc001_1.mp4"), 120 * MB, b"\x00\x00\x00\x20ftypmp42")
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(a) is True
|
||||||
|
assert os.path.exists(b) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_去重时保留无站点前缀的一份(tmp_path):
|
||||||
|
plain = make_file(str(tmp_path / "x/abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
prefixed = make_file(str(tmp_path / "x/masex.tv@abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(plain) is True
|
||||||
|
assert os.path.exists(prefixed) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_prefer_source_优先保留指定站点(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/masex.tv@abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
other = make_file(str(tmp_path / "x/hkbisi.com@abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
run(tmp_path, "--apply", "--prefer-source", "hkbisi.com")
|
||||||
|
assert os.path.exists(other) is True
|
||||||
|
assert os.path.exists(str(tmp_path / "x/masex.tv@abc001_1_8k.mp4")) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_不同目录内容相同不去重(tmp_path):
|
||||||
|
a = make_file(str(tmp_path / "x/abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
b = make_file(str(tmp_path / "y/abc001_1_8k.mp4"), 120 * MB, HEADER)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(a) is True
|
||||||
|
assert os.path.exists(b) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_dedupe_关闭去重(tmp_path):
|
||||||
|
a = make_file(str(tmp_path / "x/a.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
b = make_file(str(tmp_path / "x/b.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
run(tmp_path, "--apply", "--no-dedupe")
|
||||||
|
assert os.path.exists(a) is True
|
||||||
|
assert os.path.exists(b) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_去重后副本的同名海报一并删除(tmp_path):
|
||||||
|
"""去重必须在海报匹配之前:副本的海报不能留到下一轮才清(要求一次收敛)"""
|
||||||
|
make_file(str(tmp_path / "x/a.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
make_file(str(tmp_path / "x/b.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
keep_poster = make_file(str(tmp_path / "x/a.com@abc001_1-poster.jpg"), 4096)
|
||||||
|
dup_poster = make_file(str(tmp_path / "x/b.com@abc001_1-poster.jpg"), 4096)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.exists(keep_poster) is True
|
||||||
|
assert os.path.exists(dup_poster) is False
|
||||||
|
# 再跑一次应当无事可做(一次收敛)
|
||||||
|
entries, _ = c.collect(str(tmp_path))
|
||||||
|
c.decide(entries, 100 * MB, True, True)
|
||||||
|
assert [e.path for e in entries if not e.keep] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_去重会记录保留的那一份(tmp_path):
|
||||||
|
a = make_file(str(tmp_path / "x/a.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
make_file(str(tmp_path / "x/b.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
entries, _ = c.collect(str(tmp_path))
|
||||||
|
c.decide(entries, 100 * MB, True, True)
|
||||||
|
dups = [e for e in entries if e.dup_of]
|
||||||
|
assert len(dups) == 1
|
||||||
|
assert dups[0].reason == "重复视频"
|
||||||
|
assert os.path.exists(dups[0].dup_of) is True
|
||||||
|
assert dups[0].dup_of in (a, str(tmp_path / "x/b.com@abc001_1.mp4"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_重复视频单独成段写入清单(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/a.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
make_file(str(tmp_path / "x/b.com@abc001_1.mp4"), 120 * MB, HEADER)
|
||||||
|
manifest = tmp_path / "m.txt"
|
||||||
|
c.main(["--root", str(tmp_path), "--quiet", "--manifest", str(manifest)])
|
||||||
|
text = manifest.read_text(encoding="utf-8")
|
||||||
|
assert "E. 重复大视频" in text
|
||||||
|
assert "--- 保留:" in text
|
||||||
|
assert "F. 待删除的空目录" in text
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ 删除行为
|
||||||
|
|
||||||
|
def test_预演模式不删除任何文件(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/movie.mp4"), 120 * MB)
|
||||||
|
junk = make_file(str(tmp_path / "x/a.url"), 16)
|
||||||
|
assert run(tmp_path) == 0
|
||||||
|
assert os.path.exists(junk) is True
|
||||||
|
assert os.path.exists(str(tmp_path / "x/movie.mp4")) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_清理空目录_但保留root(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/movie.mp4"), 120 * MB)
|
||||||
|
make_file(str(tmp_path / "x/movie.trickplay/320 - 10x10/0.jpg"), 1024)
|
||||||
|
empty = tmp_path / "empty-dir"
|
||||||
|
empty.mkdir()
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.isdir(str(tmp_path)) is True
|
||||||
|
assert os.path.exists(str(tmp_path / "x/movie.trickplay")) is False
|
||||||
|
assert os.path.exists(str(empty)) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_父目录级联删除到影片级目录(tmp_path):
|
||||||
|
"""多层子目录被清空后,父目录应逐级向上删除,直到影片级目录"""
|
||||||
|
make_file(str(tmp_path / "vrkm-1763/a-poster.jpg"), 4096)
|
||||||
|
make_file(str(tmp_path / "vrkm-1763/sub/b.url"), 16)
|
||||||
|
make_file(str(tmp_path / "vrkm-1763/sub/deep/deeper/c.jpg"), 1024)
|
||||||
|
make_file(str(tmp_path / "keepme/x/movie.mp4"), 120 * MB)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
# 整条空目录链连同影片级目录一起消失
|
||||||
|
assert os.path.exists(str(tmp_path / "vrkm-1763")) is False
|
||||||
|
assert os.path.exists(str(tmp_path / "vrkm-1763/sub/deep/deeper")) is False
|
||||||
|
# 有视频的目录不受影响,root 本身保留
|
||||||
|
assert os.path.exists(str(tmp_path / "keepme/x/movie.mp4")) is True
|
||||||
|
assert os.path.isdir(str(tmp_path)) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_全部删除后root仍保留(tmp_path):
|
||||||
|
make_file(str(tmp_path / "only/a.url"), 16)
|
||||||
|
make_file(str(tmp_path / "only/b.jpg"), 1024)
|
||||||
|
run(tmp_path, "--apply")
|
||||||
|
assert os.path.isdir(str(tmp_path)) is True
|
||||||
|
assert os.listdir(str(tmp_path)) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_keep_empty_dirs(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/movie.mp4"), 120 * MB)
|
||||||
|
make_file(str(tmp_path / "x/movie.trickplay/0.jpg"), 1024)
|
||||||
|
run(tmp_path, "--apply", "--keep-empty-dirs")
|
||||||
|
assert os.path.isdir(str(tmp_path / "x/movie.trickplay")) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_不跟随符号链接目录(tmp_path):
|
||||||
|
"""符号链接目录内的文件不应被扫描/删除"""
|
||||||
|
outside = tmp_path / "outside"
|
||||||
|
victim = make_file(str(outside / "keep-me.jpg"), 1024)
|
||||||
|
target = tmp_path / "root" / "link"
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
os.symlink(str(outside), str(target))
|
||||||
|
run(target.parent, "--apply")
|
||||||
|
assert os.path.exists(victim) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_目录不存在返回2(tmp_path):
|
||||||
|
assert run(tmp_path / "not-exist") == 2
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ 汇总统计
|
||||||
|
|
||||||
|
def test_汇总统计(tmp_path):
|
||||||
|
make_file(str(tmp_path / "x/movie.mp4"), 120 * MB)
|
||||||
|
make_file(str(tmp_path / "x/movie.srt"), 1024)
|
||||||
|
make_file(str(tmp_path / "x/junk.url"), 16)
|
||||||
|
entries, dirs = c.collect(str(tmp_path))
|
||||||
|
c.decide(entries, 100 * MB, True, True, keep_covers=False)
|
||||||
|
assert len(entries) == 3
|
||||||
|
assert sum(1 for e in entries if e.keep) == 2
|
||||||
|
assert sum(1 for e in entries if not e.keep) == 1
|
||||||
|
assert [e.reason for e in entries if e.name == "movie.srt"] == ["字幕"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_human_size():
|
||||||
|
assert c.human_size(0) == "0.0 B"
|
||||||
|
assert c.human_size(1024) == "1.0 KB"
|
||||||
|
assert c.human_size(100 * MB) == "100.0 MB"
|
||||||
Reference in New Issue
Block a user