Jellyfin Web VR plugin for Quest 3 (VR180 stereo playback)

Injects a 「VR 播放」 button into Jellyfin Web's video player page and plays
VR180 / VR360 stereo content on Meta Quest 3 directly in the browser, with
DirectPlay 8K HEVC, in-VR subtitles and an in-VR control panel.

Modules (src/, concatenated in filename order into dist/jfvr.js):
  00-core.js       namespace, shared state, GL helpers, layer/frame-hook registry
  10-subtitles.js  libass canvas -> head-locked quad layer, auto-cropped to ink,
                   with depth/size decoupling (angular-size lock)
  20-controls.js   in-VR control panel: progress/seek, transport, subtitle rows,
                   aiming beam, padlock, pure 2D canvas -> XR quad layer
  30-hands.js      system hand model (Three.js + generic-hand GLB) via a manually
                   driven projection layer
  90-main.js       boot, XR session lifecycle, rAF broker (rAF stops during XR),
                   input dispatch, settings panel, localStorage persistence

Key on-device findings are documented in HANDOFF.md §6 and README.md addenda:
requestAnimationFrame stops during an XR session, 3:2 pulldown at 90 Hz needs
updateTargetFrameRate(120), libass must be resized before entering VR, a quad's
local -Z is its plane normal, and hand aiming must use the gaze ray because the
emulated targetRaySpace points sideways. Quest does not expose eye tracking.

Docs: INTERFACES.md (module contract + addenda), README.md (measurements),
HANDOFF.md (handover notes). Credentials are NOT in the repo: deploy.py reads
the SSH password from the JF_PW environment variable.
This commit is contained in:
2026-10-11 19:19:06 +08:00
commit 838675baa6
14 changed files with 13981 additions and 0 deletions
+20
View File
@@ -0,0 +1,20 @@
# --- secrets (never commit) -------------------------------------------
# Local copy of the NAS / Jellyfin credentials. The repo only carries
# placeholders; see HANDOFF.md §2.1.1.
secrets.local.md
secrets*.md
# --- duplicate / scratch ----------------------------------------------
# Root copy is byte-identical to legacy/jfvr-mvp-v6-working.js (which
# deploy.py --legacy actually uses), so only the legacy one is tracked.
/jfvr-mvp.js
# Temporary offline test page (fake <video> + libass canvas + XR stubs).
_harness.html
# --- build / tooling noise --------------------------------------------
__pycache__/
*.py[cod]
.DS_Store
Thumbs.db
*.swp
*~
+454
View File
@@ -0,0 +1,454 @@
# Jellyfin Web VR180 — 项目交接文档
> 给新对话用。读完这份就能接手,不需要重新调查环境。
> 最后更新:已部署版本 `7616c9`(`NS.version = hud-5`;ADDENDUM 6 字幕深度/大小解耦
> + ADDENDUM 7 唤出门限/进 VR 前面板精简 + ADDENDUM 7b 锁按钮与「锁定」面板模式)
> 已在 Quest 浏览器里实机验证模块加载 + 锁定数学(`subLockAngle/subAngleDeg`,`errors: []`);
> **VR 会话内的观感还没看过**。
---
## 1. 项目目标
给 Jellyfin Web 播放页注入插件,让 **Meta Quest 3 浏览器**能直接播放内网 Jellyfin 里的
**VR180 立体视频**,不用手动切全屏、选投影。在网页里点「VR 播放」→ 选格式 → 进 VR。
---
## 2. 环境事实(已核实,不用重查)
### 2.1 服务器
| 项 | 值 |
|---|---|
| 主机 | `192.168.123.199`(fnOS NAS,CW8505fnOS,Intel Pentium Gold 8505 + UHD 核显) |
| **SSH** | `admin` / `<NAS 密码>`(密码不写进仓库,见 §2.1.1) |
| Jellyfin | **12.1.0**,Docker 容器名 `Jellyfin`,`jellyfin/jellyfin:latest` |
| 配置目录 | `/vol2/jellyfin/config` |
| web 目录(注入点) | 容器内 `/jellyfin/jellyfin-web/` |
| 备份 | `/vol2/jellyfin/config/_vr_backup/index.html.orig` |
| 反向代理 | Traefik v3.0(`http://192.168.123.199:8080/api/rawdata` 可读配置) |
#### 2.1.1 密码怎么取
**仓库里不存密码。** 真实密码在本机一个被 `.gitignore` 忽略的文件里:
```
secrets.local.md # 不入库,只存在于开发机
```
`deploy.py` 也只从环境变量 `JF_PW` 读,代码里没有硬编码。换机器时要么把那份
`secrets.local.md` 拷过去,要么手动 `export JF_PW=...`。
### 2.2 两个访问入口(关键区别)
| 入口 | 安全上下文 | WebXR | 速度 |
|---|---|---|---|
| **`https://jellyfin.cat-shark.xyz`** | ✅ | **直接可用,零配置** | **IPv6 局域网直连,实测 700 Mbps,TTFB 7–12ms** |
| `http://192.168.123.199:8096` | ❌ | ❌ 需每台设备在 Quest 浏览器 `chrome://flags` 里启用 `unsafely-treat-insecure-origin-as-secure` 并填该地址 → Relaunch | 内网 |
> **推荐一律用 HTTPS 域名。** 该域名的 IPv6 AAAA 记录就是 NAS 自己的局域网地址
> (`2409:8a14:44a:e360:520:8aa2:7736:4f12`),Quest 与 NAS 在同一个 /64,纯内网直连。
> IPv4 的 A 记录 `39.105.149.197` 是阿里云中转(NAS 上跑着 `npc` 内网穿透客户端),走 IPv4 才慢。
### 2.3 Quest 3
| 项 | 值 |
|---|---|
| 系统 | Android 14 (SDK 34),build `UP1A.231005.007.A1` |
| 浏览器 | **`152.1.0.34.52.1079628208`**(Chromium 152.0.7977.83) |
| 无线 ADB | `192.168.123.229:34485`,由 SideQuest 维持 |
**ADB 与 CDP 通路:**
```bash
ADB=/opt/SideQuest/resources/platform-tools/adb # SideQuest 自带的 adb
$ADB devices -l # 确认设备在线
$ADB forward tcp:9222 localabstract:chrome_devtools_remote
curl -s http://127.0.0.1:9222/json/version # 确认 CDP 通
```
项目里有 `quest.sh` 一键做上面这些。拿到 9222 后用 `agent_browser --cdp http://127.0.0.1:9222` 接管 Quest 浏览器。
> ⚠️ **CDP 转发会被无线 ADB 重置**,断了就重跑 `quest.sh`。
**截图与「看」画面:**
```bash
$ADB exec-out screencap -p > shot.png # 4128x2208,双眼并排合成输出
tesseract shot.png - -l chi_sim --psm 6 # OCR 可读中文
```
> ⚠️ 当前模型**看不到图片**,只能靠 OCR + 像素统计 + 差分分析。
---
## 3. 片源
```
61+ 个 VR180 文件,全部:
8192 x 4096 | HEVC Main / level 6.1 | ~60 fps | 8–25 Mbps | .mp4 (faststart)
```
即 **VR180 SBS**,每眼 4096×4096 的 180° 等距柱状。
**字幕文件是天生的双目设计**(853 部是这样的):
```
Title: VR Dual-Eye Subtitle
PlayResX: 3840 PlayResY: 1920 ← 2:1,与视频同比例
Style: LeftEye , Alignment=8, MarginL=50 , MarginR=1920, MarginV=700
Style: RightEye, Alignment=8, MarginL=1920, MarginR=50 , MarginV=700
```
左半屏=左眼、右半屏=右眼,正好匹配 `stereo-left-right`。
> ⚠️ **必须保持 DirectPlay。** Pentium 8505 + UHD 核显编不了 8K(Alder Lake-N 编码上限约 4K),
> 一旦 Jellyfin 转码,分辨率腰斩且 CPU 打满。
---
## 4. Quest 3 实测能力(真机数据)
| 能力 | 结果 |
|---|---|
| `isSecureContext` / `navigator.xr`(HTTPS 下) | true / true |
| `immersive-vr` | **true** |
| `XRMediaBinding.createEquirectLayer` | **true** |
| `XRWebGLBinding.createQuadLayer` | **true** |
| `XRWebGLBinding.getSubImage` | **需要 2 个参数** `(layer, frame)` |
| `MAX_TEXTURE_SIZE` / `MAX_RENDERBUFFER_SIZE` | **8192 / 16384** |
| 8K HEVC `decodingInfo` | `supported:true, powerEfficient:true` |
| `supportedFrameRates` | **[72, 80, 90, 120]** |
| `enabledFeatures` | `[local, viewer, bounded-floor, layers, hand-tracking, webxr, local-floor]` |
| 手部追踪 | ✅ 25 关节齐全,profiles `["oculus-hand","generic-hand","generic-hand-select","generic-trigger"]` |
| **眼动追踪** | ❌ **Meta 不向 WebXR 暴露** |
---
## 5. 架构
```
jelllyfin_vr_web_plugin/
├── INTERFACES.md ← 绑定契约(所有模块必须遵守)+ 5 个 ADDENDUM
├── README.md ← 技术记录(含全部实测数据)
├── HANDOFF.md ← 本文档
├── src/
│ ├── 00-core.js (343 行) 命名空间、共享状态、GL 助手、层/帧钩子注册表、能力探测
│ ├── 10-subtitles.js (1022 行) 字幕:libass 画布 → 头锁定 quad 层(HUD)+ 视角尺寸锁定
│ ├── 20-controls.js (1930 行) 控制面板:进度条/按钮/光束/光标/锁按钮,canvas 2D → quad 层
│ ├── 30-hands.js (569 行) **系统手部模型**:Three.js + 官方 generic-hand GLB → projection 层
│ └── 90-main.js (1420 行) 引导、会话生命周期、rAF 代理、输入分发、设置面板、持久化
├── build.py ← src/*.js 按文件名顺序拼接 → dist/jfvr.js;`--check` 只做语法检查
├── deploy.py ← 构建 + 上传进容器 + 注入 script 标签;`rollback` 还原
├── quest.sh ← 一键重连 Quest 的 ADB + CDP 转发
├── dist/jfvr.js ← 构建产物
└── legacy/jfvr-mvp-v6-working.js ← 早期验证可用的单文件版(回退用)
```
### 5.1 层栈(**顺序即合成顺序,后的在上**)
```
video equirect XRMediaBinding.createEquirectLayer, 180°, stereo-left-right
hands projection Three.js 渲染官方手模型(手动渲染循环,不占 baseLayer)
subtitles quad viewer 空间,头锁定,自动裁剪墨迹包围盒
controls quad local 空间,固定位置 (0,-0.34,-1.45),身份旋转
```
> ⚠️ **Three.js 不能接管 XR 会话**:`renderer.xr.setSession()` 会设 `baseLayer`,
> 与 `layers` 数组冲突。所以手模型走**手动渲染循环**(自建 projection layer、
> 自绑帧缓冲、自遍历 view 设投影矩阵)。
### 5.2 关键 API
```js
__JFVR.addLayer(layer) / removeLayer(layer) // 层注册表,自动推 updateRenderState
__JFVR.addFrameHook(fn) / removeFrameHook(fn) // fn(frame, time, session) 每 XR 帧
__JFVR.state // 共享状态;window.__JFVR_STATE__ 也是它(供 CDP 读取)
__JFVR.stats // 诊断信息,全部落在 state.stats
```
---
## 6. 关键技术发现(**血泪教训,不要重犯**)
### 6.1 `window.requestAnimationFrame` 在 XR 会话中完全停摆
```
实测:6 秒内 0 次回调
timeupdate : 22 次 ✅
requestVideoFrameCallback : 288 次 ✅
```
Jellyfin 用 rAF 渲染 libass 字幕 → **进 VR 后字幕画布冻结在最后一帧**。
**解法**:90-main.js 劫持 `requestAnimationFrame`,XR 中把回调收进私有队列,
用 `video.requestVideoFrameCallback` 驱动;不在 XR 时透传原生实现。
### 6.2 帧率不匹配导致"卡顿"
59.94fps 视频 + 90Hz 会话 = **1.5 倍** → 每帧交替显示 1/2 次 = 3:2 pulldown 抖动。
**解法**:`session.updateTargetFrameRate(120)`(120/60 = 2 整除)。
实测 87.9 → 114.9 fps,20 秒稳定、丢帧 0.000%。
### 6.3 libass 画布必须在**进入 VR 之前**放大
libass 按屏幕视频框尺寸渲染(约 1250×708 = **3.5 px/度**,糊)。
**XR 会话运行期间 libass 不重算画布尺寸**,所以放大动作挂在「打开 VR 面板」这个点击上。
实测放大到 **4320×2160**(libass 封顶)。
**踩过的坑**:退出 VR 时如果还原尺寸,而面板还开着,再进 VR 就不会重新放大 → 字幕糊。
现在:面板开着就不还原。
### 6.4 quad 的局部 `-Z` 是**面法线**,不是轴向
想在光束朝向里把 `-Z` 对齐射线是**错的** —— 会让平面垂直于射线、`width` 横在射线上,
看到的是一条正对眼睛的细横条。正确:**`+X`(长轴)对齐射线**,`-Z` 与射线垂直。
### 6.5 单条平面条带沿视线方向时**必然侧对** → 十字双条带
光束指向"远离你"的方向时,任何包含光束轴的平面都包含视线方向 → 侧对 → 投影宽度趋零。
实测有效宽度只剩 `cos(68.7°) = 0.36` 倍 ≈ 2mm ≈ **3 像素,等于隐身**。
**解法**:**两条正交条带**(第二条绕光束轴转 90°),`max(cosθ, sinθ) ≥ 0.707` 恒成立。
### 6.6 手部 `targetRaySpace` 是**模拟射线**
Meta 官方文档原文:
> the `targetRaySpace` for hands is populated with an **emulated ray** that matches the
> behaviors of the pointing ray of hands present in **Meta Horizon Home**, which assumes
> that the UI is in front of the user, so the target rays will **always point in that
> general direction**
- **优点**:系统做了**稳定化**,落点平稳
- **缺点**:不跟手指
> ❌ 我一度改成用食指关节自己算射线 → **指尖逐帧抖动直接传导到落点,明显比系统射线不稳**。
> **默认必须用系统射线。** 面板里留了「手部指向:系统射线 / 食指」开关供切换。
### 6.7 `requestSession()` 必须**同步**写在点击处理里
WebXR 规范:需要 *transient activation*。前面不能有 `await`,否则被拒。
实测用 `element.click()` 从 CDP eval 调用会失败(`The requested session requires user activation`),
但用 `agent_browser click` 可以(CDP 输入事件算用户手势)。
### 6.8 `frame.getPose()` 的结果**只对该帧有效**
反复读同一个旧 `XRFrame` 会一直返回缓存值 —— 我因此误判过"射线冻结"。
**采样必须在帧循环内做。**
### 6.9 其余坐标陷阱
- 面板带 yaw 旋转后,"平面垂直于局部 z 轴"的假设失效 → `panelHit` 必须
**把射线变换到层自身空间**再求交(含四元数旋转/共轭/姿态合成)
- 字幕层在 **viewer 空间**而指针射线在 **local 空间** → 两者不同坐标系,必须转换
### 6.10 系统保留手势
**掌心肌肤捏合**(palm pinch)是系统保留:左手 = 菜单键,**会直接退出会话**。
只能用普通点捏。
### 6.11 面板“一捏就弹”有两个独立原因(ADDENDUM 7)
只修 `pointerDown` 里的 `show()` 不够——`selectstart` 监听里也有一句无条件 `show()`。
两处删掉后才真的变成“指哪儿弹哪儿”。另外**唤出那一下不算点击**,否则手从远处
扫到面板的路径上会顺手碰到一个按钮。
### 6.12 锁按钮为什么一度做成了独立图层(错误,已改回)
第一版把锁做成了控制条右侧的**独立 quad 图层**(x=+0.45),实测选不中。教训:
- **能用手的射线扫到的范围 = 光标自己的容差窗口**(`panelHit` 对 0.72m 面板用
0.25/0.6 → 只能到 x≈0.45,即面板右边缘)。放在更外面就是**任何射线都到不了**。
- 锁的命中检测当时用的是**面板的图层几何**,所以“面板显示时点不中、隐藏时能点中”——
行为随面板尺寸变化。
正确做法:锁就是**面板上的一个普通按钮**(`lock` 动作),走光标→hitTest→handleSelect
→onAction 这条已验证能用的路。锁定后切到第三种面板模式 `locked`(0.26×0.10),
**只画锁、只命中锁**。
### 6.13 「头部锁定」的东西永远瞄不到
“VR 空间右侧边缘”听起来像视野边缘的头锁定 HUD,但这个项目**只能用手的射线**指向
(手柄连不上)。头锁定元素的相对角度永远不变,射线**永远追不上它**,所以任何可点的
元素都必须和控制条在同一个 `refSpace` 里。
---
## 7. 当前状态(部署 `8ae73b`)
### 7.1 可用
| 功能 | 状态 |
|---|---|
| VR180 立体播放 | ✅ DirectPlay 8K HEVC,120Hz,~112 fps |
| 字幕 HUD(头锁定 quad) | ✅ 自动裁剪墨迹包围盒、双目半屏提取 |
| 控制面板(fixed 位置) | ✅ 进度条/播放暂停/±10s/字幕开关/字号±/透明±/复位 |
| 中文按钮标签 | ✅ |
| 引导线(十字双光束) | ✅ 任何时候都显示 |
| 面板唤出门限 | ✅ 只有指示线指向/靠近面板时才唤出;唤出那一下不算点击;500ms 内忽略 |
| 锁按钮(面板不启动) | ✅ 面板媒体行的第五个按钮:上锁后切到「只显示锁」的最小面板,并同样自动隐藏;每次进 VR 重置为未锁 |
| 按压反馈 | ✅ 圆环放大 1.9× + 蓝填充 + 变白,300ms |
| 字幕抓拖 | ✅ 注视选中 + 手移动拖(与视线解耦,避免转头就飘) |
| 字幕深度/大小解耦 | ✅ 默认「锁定视角尺寸」:拉近/拉远只改深度,视角大小与画面位置不变 |
| VR 内「近/远」字幕按钮 | ✅ 控制条字幕行现为 9 键:上 下 近 远 小 大 暗 亮 复位 |
| 字幕行题注显示距离 | ✅ `字幕 距1.45m · 69°`(距离永远是第一个数字,便于和物体对深度) |
| 字幕 CC 按钮状态 | ✅ 开=蓝底+`字幕开`,关=斜杠 CC+`字幕关` |
| 设置持久化 | ✅ localStorage `jfvr.settings.v1` |
| 系统手模型 | ✅ 官方 `generic-hand` GLB,Three.js 渲染,帧率无损失 |
### 7.2 已知未完成 / 待验证
- **锁定角尺寸的默认值还没定** —— 代码默认 69.2°,但头显上已存的是 **85°**
(之前用面板滑杆设的,ADDENDUM 7 已移除那个滑杆)。要回默认就在 VR 里按几次「小」
(每次 −5°),或者直接改 localStorage。
- **ADDENDUM 7/7b 的 VR 手感未验证** —— 唤出灵不灵、锁按钮好不好矒、500ms 合不合适,
只能戴上去试。模块级冒烟测试已过(见 README ADDENDUM 7b 的 Verification)。
- **手柄无法连接** —— **系统层问题,不是插件问题**。
`adb shell getevent -i` 只列出 5 个设备(无任何控制器),WebXR 的 `inputSources`
也只有双手。需要在头显里检查手柄电量 / 重新配对 / 确认「自动切换」开关。
- 手模型实际渲染效果**还没在头显里确认**(subagent 交付后只验证了加载成功 `handsReady: true`)
- 可疑现象:不捏合时 `inputSources` 仍显示左手 `@0`(按钮 0 卡在按下)
---
## 8. 字幕距离 / 角尺寸 —— 已解决(ADDENDUM 6)
### 用户要解决的问题
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
Quest 不给 WebXR 眼动(§4),**没法自动跟视线距离**;而且近处物体足够强行的方案是
让「距离」变成一个**纯深度**旋钮:不管拉到多近,字幕的角尺寸和画面位置都不变。
### 实现(默认开启)
`state.subLockAngle = true`,`state.subAngleDeg = 69.2`。锁定时拉近/拉远:
```
width = 2 · |z| · tan(subAngleDeg / 2) 保持角尺寸 θ
y = (y0 / z0) · z 保持角高度 y/z
```
69.2° 就是旧的 `2.0 m @ 1.45 m`,所以**默认观感零变化**。近端限位从 0.5 m 收到
**0.35 m**。ADDENDUM 7 之后锁**固定在开**(面板里的「距离与大小」开关已按用户要求移除),
所以不再有「拉近=变大」的自由模式入口。
- VR 内控制条字幕那一行现在是 **9 个按钮**:`上 下 近 远 小 大 暗 亮 复位`。
- `上/下`:垂直位置 ±0.04 m(以前 VR 里只能抓拖,没有按钮)
- `近/远`:深度 ±0.15 m(锁定下只改深度,角尺寸不变)
- `小/大`:锁定下视角宽度 ±5°(解锁后是宽度 ±0.15 m)
- 题注永远先报**深度**:`字幕 距1.45m · 69°`,右边是不透明度。要用它和眼前物体对深度。
- 顺手修了两个 bug:① 控制条 CC 按钮以前读的是 controls 模块自己的、main 从不写入的
`S.subtitlesOn`,所以**永远显示为关**——现在改为回退读共享的 `state.subtitles`,
并把开/关做成蓝底 `字幕开` / 斜杠 CC `字幕关`;② VR 里没有垂直位置按钮(现已补上)。
2D 面板的「垂直位置」滑杆实测是好的,所以缺的只是 VR 侧。
- 抓拖在锁定下会把深度位移按抓取起点的 y/z 比例同步到 y,所以拉到很近时字幕
不会掉到视野下方。
- 新增诊断 `stats.subTextDeg`:**实际画出来的字幕墨迹角高度(度)**。用它对照
文献就能定默认 `subAngleDeg`。
### 文献结论(保留供参考)
核心指标是 **角尺寸(angular size)**= 距离 × 字体大小的组合,不是单独某一个:
- *User Experience of Reading in Virtual Reality — Finding Values for Text Distance, Size and Contrast*
(arXiv 2004.01545,全文表 I 已复现):以 dmm(1 dmm = 1 m 处 1 mm 字高)为单位,
**中/长文本偏好 16–18 dmm ≈ 0.9–1.0° 字高**,2 词短文本 27–32 dmm ≈ 1.6–1.8°;
Google 指南:正文/说明 ~15–20 dmm,标题 32 dmm ≈ 1.8°,大标题 40 dmm ≈ 2.3°。
参与者明确认为**角尺寸过大**是「最差」读数体验之一 → 存在上限。
- *VR Reading UIs: Assessing Text Parameters for Reading in VR*(Dingler 等,CHI 2018 EA):
偏好字高 **41 ± 14 dmm ≈ 2.35° ± 0.8°**。
- *The influence of text rotation, font and distance on legibility in VR*(IEEE VR 2020)
与 *An effective method for measuring text legibility in XR*(Frontiers in VR 2023,含 Quest 2)
都用 dmm/角度测最小可读字号,结论同向:可读性由角尺寸决定。
上述都是**字高**角尺寸,而本插件的滑杆/参数是**字幕条整幅角宽**,两者靠 `subTextDeg`
换算。默认值暂不动(保持已调好的观感),等头显实测后再定。
### 待办
1. 头显里读 `stats.subTextDeg`,若明显偏离 1–2° 再考虑改默认 `subAngleDeg`
2. (可选)如果实际用下来「近/远」每次 0.15 m 太粗,再加长按连续调整
---
## 9. 工作流速查
```bash
cd /home/cat/Desktop/myProject/jelllyfin_vr_web_plugin
# 语法检查(全部模块)
python3 build.py --check
# 构建 + 部署(自动构建 → 上传 → 注入 script 标签)
# 密码从环境变量读;真实值放在本地 secrets.local.md(不入库)
export JF_PW='<NAS 密码>'
python3 deploy.py
# 回滚到原始 index.html
python3 deploy.py rollback
# 部署早期单文件版
python3 deploy.py --legacy
# 重连 Quest 的 ADB + CDP
./quest.sh
```
**部署后必须让用户刷新页面**(脚本按页面加载缓存,URL 里的 `?r=<md5前6位>` 是破缓存用的)。
### 诊断信息
页面里「VR 播放」面板 → 展开「诊断信息」,或经 CDP 读 `window.__JFVR_STATE__.stats`。关键字段:
```
layersBuilt 实际建立的层
fps / targetFrameRate
pointer 当前瞄准源与命中坐标
inputSources 系统报告的输入源
pressAt 上次按压命中/未命中
handsReady / handsErr / handsSources / handsFrames
subMode / subCanvasSize / subCrop / subTex / subOpacity / subPlacement
subLockAngle / subAngleDeg / subTextDeg / subYOverZ ← ADDENDUM 6
ctlSubsOn / ctlSubDist / ctlSubAngle / ctlSubLock ← 控制条实际画出的值
ctlLock / ctlMode / pressAt / lastActivate ← ADDENDUM 7/7b 锁与唤出判定
ctlHidden / ctlVisible / ctlBeamOn / ctlDraws / ctlErr
panelPose / grab / grabValue / saved
errors[] 最近错误
```
---
## 10. 长期隐患
1. **注入写在容器内文件**(`/jellyfin/jellyfin-web/index.html`)→
`jellyfin/jellyfin:latest` 一升级就没了,需重跑 `deploy.py`。
正式版应改用 **JavaScript Injector**(`n00bcodr/jellyfin-plugins` 的 `12/manifest.json`,
targetAbi 12.0.0.0)或 **File Transformation**(`IAmParadox27/jellyfin-plugin-file-transformation`,554⭐)。
2. **Three.js 从 CDN 加载**(1.27MB)→ 依赖外网。可改为本地托管。
3. **Quest 浏览器版本会随系统更新变化** → 能力探测(诊断信息里的能力段)是唯一的自检手段。
---
## 11. 参考项目
| 项目 | 形态 | 备注 |
|---|---|---|
| `JiaruiGe/jellyfinvr`(镜像 `allexxex`) | 单 JS,Three.js | VR180/360/SBS/OU;每帧上传纹理,8K 下是瓶颈 |
| `Mix1C/Jellyfin-360VR-Player` | 单 JS,A-Frame | 仅 360°,无 180 |
| `Lqlsoftware/Jellyfin-vr-plugin` | C# DLL | 独立播放页 `/VR/Video/{id}/Play` |
| `Toastyice/DeoVRDeeplink` | C# DLL + `deovr://` | 可完全自动唤起 Quest 上的 DeoVR(deeplink 走系统,不受 user gesture 限制) |
| `IAmParadox27/jellyfin-plugin-file-transformation` | C# 插件 | 正式版注入通道 |
| `n00bcodr/jellyfin-plugins` → JavaScript Injector | 插件 | `12/manifest.json` |
---
## 12. 自检清单(改完代码后)
```
[ ] python3 build.py --check 全部模块语法通过
[ ] JF_PW=... python3 deploy.py 部署成功,md5 一致
[ ] 让用户刷新页面
[ ] 读 window.__JFVR_STATE__.stats 确认 layersBuilt / errors 为空
[ ] 确认 fps 仍在 ~110+,targetFrameRate 120
[ ] 确认面板/字幕/手模型均无异常
```
+734
View File
@@ -0,0 +1,734 @@
# JFVR module interfaces (contract)
Everything lives under a single global namespace `window.__JFVR`, created by
`src/00-core.js` before any other module runs. Modules are plain
`(function(){ ... })()` IIFEs that attach themselves to `__JFVR`.
Load order (build script concatenates in this order):
```
00-core.js -> __JFVR core: state, helpers, layer/frame-hook registry
10-subtitles.js -> __JFVR.Subtitles
20-controls.js -> __JFVR.Controls
90-main.js -> boot, DOM scan, XR session lifecycle, settings panel, wiring
```
Nothing may read `document` / `navigator.xr` at load time except `00-core.js`
and `90-main.js`. The two module files must only *define* functions and are
called later with a session/frame.
---
## 00-core.js — always available as `window.__JFVR`
```js
__JFVR.state // shared mutable state object (see below)
__JFVR.log(...args) // console.log with [JFVR] prefix + colour
__JFVR.version // string
// --- helpers ---
__JFVR.makeEl(tag, className?, text?) // document.createElement wrapper
__JFVR.compileShader(gl, type, source) // throws with the info log on failure
__JFVR.invert4(mat4ArrayLike) // -> Float32Array(16)
__JFVR.QUAD_VS // vertex shader source: fullscreen triangle,
// `in vec2 aPos; out vec2 vUv;`
// vUv = vec2(aPos.x*.5+.5, .5 - aPos.y*.5)
__JFVR.bindFullscreenTriangle(gl, program) // creates the VBO, enables aPos, returns attribLoc
// --- XR layer registry (owned by 90-main.js) ---
__JFVR.addLayer(layer) // register + push updateRenderState if a session is live
__JFVR.removeLayer(layer) // unregister + push
__JFVR.addFrameHook(fn) // fn(frame, time, session) every XR frame
__JFVR.removeFrameHook(fn)
// --- capabilities (filled by 00-core.js at boot) ---
__JFVR.caps = {
secureContext, hasXR, immersiveVR, hasXRMediaBinding, hasEquirectLayer,
hasXRWebGLBinding, hasXRHand, hasXRJointSpace, maxTextureSize,
hevc: { ... }, ua
}
// --- live handles ---
__JFVR.session // current XRSession or null
__JFVR.video // current <video> element or null
__JFVR.refSpace // 'local' XRReferenceSpace for the running session
__JFVR.viewerSpace // 'viewer' XRReferenceSpace (head-locked layers)
```
### `__JFVR.state` fields that modules may use
```js
state.video // <video> or null
state.session // XRSession or null
state.refSpace // local space
state.viewerSpace // viewer space (head-locked)
state.projection // '180' | '360'
state.stereo // 'mono' | 'sbs' | 'ou'
state.subtitles // bool — render subtitles in VR
state.subMode // 'hud' (head-locked quad, default)
// 'sphere'(on the equirect layer, legacy)
state.subAlpha // 'premultiplied' | 'straight'
state.subPlacement // { y, z, width } metres, hud mode only
state.stats // diagnostics bag, free to add keys
state.gl // { refSpace, session } set by main during a session
```
### Layer / frame-hook lifecycle a module must follow
1. `create(session, glContextOrNull)` — build GL resources + the XR layer,
call `__JFVR.addLayer(layer)`, `__JFVR.addFrameHook(hook)`.
Return the layer, or `null` when it cannot run (missing canvas, missing API…).
**Never throw** — return null and `__JFVR.log()` the reason.
2. `hook(frame, time, session)` — per-frame work. Must be cheap; skip work when
nothing changed.
3. `destroy()` — remove the frame hook, remove the layer, delete GL resources,
drop canvas references. Must be safe to call twice and safe to call when
`create()` returned null.
---
## 10-subtitles.js — `__JFVR.Subtitles`
Renders Jellyfin's libass subtitle canvas into VR.
```js
__JFVR.Subtitles = {
// Called once at boot, before any session.
init(): void,
// BEFORE entering VR. Jellyfin sizes the libass canvas to the on-screen video
// box (~1250x625), which is far too small for VR. Blowing up the player box
// makes libass re-render bigger. Must be called while NOT in an immersive
// session (libass does not recompute its backing store during XR).
// Idempotent: calling twice must not double-apply.
prepare(): void,
// Undo prepare(). Safe when prepare() was never called.
restore(): void,
// True when a libass canvas exists and has non-zero size.
isReady(): boolean,
// Current libass canvas size, for diagnostics: { w, h } or null
canvasSize(): { w, h } | null,
// --- session lifecycle ---
create(session): XRLayer | null,
destroy(): void,
// --- settings, safe to call before or during a session ---
setMode(mode), // 'hud' | 'sphere' | 'off'
setAlphaMode(mode), // 'premultiplied' | 'straight'
setPlacement({ y, z, width }), // partial update, hud only
getMode(): string
};
```
### HUD mode requirements (the default, and the point of this module)
- A **head-locked quad layer** in `__JFVR.viewerSpace`
(`XRWebGLBinding.createQuadLayer({ space: viewerSpace, layout: 'mono', ... })`).
- Source pixels come from `canvas.libassjs-canvas`. That canvas holds a
**stereo left-right** image: the left half is the `LeftEye` subtitle, the
right half the `RightEye` one. For a mono HUD use **one half only**
(the left half is correct for this content).
- **Auto-crop to the ink bounding box** so the strip is tight and the text is
as large as possible. Recompute the bbox at a low rate (≈4 Hz) on a
downscaled copy — never `getImageData` the full-resolution canvas every frame
(a 4320x2160 full read is ~37 MB and has already crashed a renderer once).
Add hysteresis / smoothing so the strip does not jitter between frames.
- Upload only when the source changed (or at most ~30 Hz) — the earlier
implementation burned a 4K texture upload every frame.
- Target ≈30 px/degree. A 2.0 m wide strip at 1.45 m is ≈66° → texture width
2048 is comfortable on Quest 3.
- Respect `state.subAlpha`:
- `'premultiplied'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true)`
- `'straight'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false)`
- Draw with `XRWebGLBinding.getSubImage(layer, frame)` → bind
`sub.colorTexture` to a framebuffer → viewport to
`sub.colorTextureWidth/Height` → clear to `(0,0,0,0)` → draw.
`getSubImage` takes **two** arguments.
### `prepare()` detail
The measured behaviour: setting the player box to `7680x3840` makes libass
produce a **4320x2160** canvas (it does not go higher). The previous bug was
that this was only applied on "open panel" and then restored on session end
while the panel stayed open, so a re-entry got a 1250x625 canvas.
`prepare()` must therefore be idempotent and re-callable.
Target DOM: `document.querySelector('.videoPlayerContainer')` and the `<video>`.
Save/restore their inline `style.cssText` exactly.
---
## 20-controls.js — `__JFVR.Controls`
A head-locked control strip: progress bar + seek, play/pause, skip ±10 s,
subtitle toggle. **Pure 2D canvas rendering + a hit-test function, plus a thin
XR quad layer wrapper.**
```js
__JFVR.Controls = {
init(): void,
// --- pure rendering, no WebXR, must be unit-testable in isolation ---
// Draws the bar into `ctx` sized `w` x `h` (CSS pixels).
// `view` = {
// playing, currentTime, duration, buffered, subtitlesOn,
// mode: 'collapsed' | 'expanded',
// hover: null | 'playpause' | 'back10' | 'fwd10' | 'subs' | 'track'
// }
draw(ctx, w, h, view): void,
// Maps a point in normalized panel coords (0..1, origin top-left) to an action.
// Returns one of:
// { type: 'seek', value: 0..1 }
// { type: 'playpause' | 'back10' | 'fwd10' | 'subs' }
// null
hitTest(nx, ny, w, h): action | null,
// --- XR lifecycle ---
create(session): XRLayer | null,
destroy(): void,
// --- state / input ---
setVideo(videoEl, getExtra | null): void, // main provides time/duration/buffered readers
onAction(cb): void, // cb(action) — main performs the media op
show(autoHideMs?), hide(): void, // collapsed <-> expanded + auto-hide
isVisible(): boolean,
handleSelect(inputSource, frame): void // called on XR `select`
};
```
### Panel design requirements
- **Two visual states**: `collapsed` = a slim progress bar only (≈0.9 m x 0.07 m,
centre-bottom of view). `expanded` = progress row + buttons
(≈1.1 m x 0.22 m).
- Hit targets must be **generous**: the progress bar should be visually ~1 cm
tall but hit-testable over ~3–4 cm. Meta's docs warn that the emulated hand
target ray for hand tracking is only a coarse forward pointer, so **do not
rely on pixel-precise aiming**.
- Colours must read on top of arbitrary video: dark translucent pill background,
white text, accent `#4ea1ff` (matches the existing in-page panel).
- Everything must be drawn from scratch on a canvas — no DOM, no images, no
external fonts (use a generic stack: `'system-ui, -apple-system, sans-serif'`).
- `draw()` must be deterministic and side-effect free (no timers inside).
- `hitTest` must be a pure geometry function of `(nx, ny, w, h)`.
### Input model (main.js supplies)
- **Gaze**: the panel is head-locked, so the reticle is the view centre.
`main.js` converts that into `(nx, ny)` and calls `hitTest`.
- **Controller ray**: `main.js` intersects the controller ray with the panel
plane and converts to `(nx, ny)`.
- Both end in `handleSelect()` → the action from `hitTest` → `onAction` callback.
---
## 90-main.js — owned by the integrator
Boot, DOM scan, the in-page `VR 播放` button + settings panel, the XR session
lifecycle, the rAF broker (see below), input dispatch, and the settings UI for
subtitle mode / alpha mode / placement.
### rAF broker (already proven, keep it)
While an immersive XR session is running `window.requestAnimationFrame` is
**completely stopped** (measured: 0 callbacks in 6 s), and Jellyfin renders
libass subtitles from an rAF loop. `90-main.js` intercepts
`requestAnimationFrame`/`cancelAnimationFrame`, holds callbacks while in XR and
drives them from `video.requestVideoFrameCallback`. Outside XR it delegates to
the native implementation.
---
# ADDENDUM 2 — in-VR UX pass (after first on-device test)
Owner feedback from the first headset test:
1. **The control bar is permanently visible — wrong.** It must be hidden by
default and only appear on interaction, then auto-hide.
2. **Hand/controller input has no guide line.** There is no visible ray and no
cursor, so the user cannot tell where they are aiming.
3. **Subtitles must sit below the visual centre** by default, and both the
**position and the opacity must be user-adjustable**.
## New shared state (owned by 90-main.js, read by modules)
```js
state.subOpacity // 0..1, default 0.85
state.subPlacement // { y, z, width } — y is the quad CENTRE, negative = below eye level
state.ctlAutoHideMs // default 4000
state.pointer // { visible, source, origin:{x,y,z}, dir:{x,y,z}, hit:{nx,ny}|null }
// written every XR frame by 90-main.js; modules must treat it as read-only
```
## 10-subtitles.js — additions
```js
setOpacity(a) // 0..1; applied to the XR layer (layer.opacity or the
// shader's alpha), NOT by rebuilding the layer
getOpacity(): number
```
- Default `state.subPlacement` becomes `{ y: -0.42, z: -1.45, width: 2.0 }`.
- `setPlacement()` must take effect live (update `layer.transform` if the runtime
allows mutating it, otherwise rebuild just this layer and keep the frame hook).
- Keep the existing tight-crop behaviour and the 4320x2160 canvas preparation.
## 20-controls.js — additions
```js
setPointer(p) // p as in state.pointer; called every frame by main.
// Draw a beam from origin along dir and, when hit != null, a
// cursor dot at (hit.nx, hit.ny) on the panel.
setOpacity(a)
getOpacity(): number
```
- **Default state is HIDDEN.** `create()` must not make the bar visible.
- `show(autoHideMs)` → visible + expanded, auto-hides after `autoHideMs`
(default `state.ctlAutoHideMs`).
- `hide()` → fully hidden (no bar, no beam, no cursor) until `show()` again.
- `isVisible()` → true only while shown.
- The beam must be a **separate, always-updated visual** that is only visible
while the bar is shown. Implement it as a thin quad layer whose transform is
placed at `origin + dir * (L/2)` and oriented so its local -Z follows `dir`
(build the quaternion from the direction; L = distance to the hit, or a fixed
1.2 m when there is no hit). Height ≈ 0.006 m, width = L, with a soft gradient
texture (bright at the origin end, fading out) so it reads as a laser.
- The cursor: when `hit` is non-null, draw a ring/dot on the control panel canvas
at `(hit.nx, hit.ny)` so the user sees exactly what will be activated.
- Do not re-upload either texture more than ~30 Hz; skip everything when hidden.
- Add stats keys `ctlPointer`, `ctlHidden`, `ctlBeamOn`.
## Input model change (90-main.js) — unified pointing
Hand tracking and controllers both expose `targetRaySpace`. Use one code path:
- For **every** input source with `targetRayMode === 'tracked-pointer'`
(controllers *and* hands), intersect its ray with the control quad plane and
produce `hit`.
- If no source produces a hit, fall back to gaze: `hit = {nx:0.5, ny:0.5}` centred.
- `state.pointer.source` records which one won.
- Show the beam for the winning source; hide it when the controls are hidden.
Meta's docs note the hand target ray is an *emulated* forward-pointing ray, so
hit targets stay generous — that is already the case.
### Beam orientation — CORRECTED convention (my first spec was wrong)
Do **not** map the quad's local `-Z` onto the ray direction. A quad's local `-Z`
is its plane **normal**, so that would put the quad's plane perpendicular to the
ray and lay its `width` extent across the beam — a thin face-on bar, not a beam.
Correct:
```
position = origin + dir * (L / 2) // L = hit distance, else 1.2 m
width = L (the long axis)
height = 0.006
local +X -> dir // the beam axis lies IN the plane
local -Z -> perpendicular to dir, biased up so the ribbon is never edge-on
gradient -> bright at the origin end, fading to transparent at the hit end
```
Assertion for tests: `R(q)·(1,0,0) ≈ dir` and `R(q)·(0,0,-1) ⟂ dir`.
---
# ADDENDUM 4 — in-VR subtitle placement (grab & drag)
Owner requirement: adjust the subtitles **without leaving VR**. Chosen model:
- **Position**: grab the subtitle strip with the laser and drag it in space.
- **Reset** button restores the defaults.
- **Size** and **opacity** also adjustable in VR.
- Values **persist in localStorage** (per browser, no server changes).
## Grab & drag model (owned by 90-main.js)
The strip is a head-locked quad in viewer space, so both the ray and the strip
live in the same space — no world-space math needed.
```
on grab start:
grabActive = true
grabOrigin0 = ray.origin // viewer space
grabHit0 = ray ∩ plane(z = placement.z)
grabPlace0 = { y, z, width }
per frame while grabbed:
P = currentRay ∩ plane(z = grabPlace0.z) // fixed reference plane
y' = grabPlace0.y + (P.y - grabHit0.y) // drag up/down
z' = grabPlace0.z + (ray.origin.z - grabOrigin0.z) // pull toward / push away
clamp y' to [-0.95, -0.05], z' to [-2.6, -0.7]
Subtitles.setPlacement({ y: y', z: z', width: current })
on grab end: clear grab state, persist
```
Rationale for `z`: in viewer space forward is `-Z`, so pulling the hand back
(increasing `origin.z`) makes `z'` less negative, i.e. the strip comes **closer**
— which matches how pulling a real object behaves.
Grab **must not** steal the normal control-bar interaction: only start a grab when
the ray actually hits the subtitle quad; otherwise fall through to the bar.
## Input: press **and release** are now needed
The existing activation path in `90-main.js` only detects a rising edge. Grabbing
needs the held state too, so track per input source:
- `selectstart` / `selectend` session events, **and**
- `inputSource.gamepad` button press/release (hand pinch; see ADDENDUM 3)
Both feed one `pointerDown(src)` / `pointerUp(src)` pair, deduped the same way.
## 10-subtitles.js — additions
```js
getLayer(): XRLayer | null // so main can ray-cast against the strip
resetPlacement(): void // back to { y: -0.42, z: -1.45, width: 2.0 }
```
- `width` **is** the size control: `setPlacement({ width })` already rescales the
quad while keeping the angular text height consistent with the texture aspect.
- All of `setPlacement` / `setOpacity` / `resetPlacement` stay live during a session.
## 20-controls.js — additions
The expanded control bar gains a subtitle row with large −/+ buttons:
| Action returned by `hitTest` | Meaning |
| --- | --- |
| `sub_smaller` / `sub_bigger` | width by ±0.15 m, clamped to [0.8, 4.0] |
| `sub_opaque_down` / `sub_opaque_up` | opacity by ∓0.05, clamped to [0.1, 1.0] |
| `sub_reset` | restore all subtitle defaults |
- Add `view.sub = { width, opacity }` so `draw()` can render the current values.
- Keep hit targets generous; the rows may overlap visually but must not overlap
in hit-test order (buttons before track).
- Add a new expanded height that fits the extra row.
## 90-main.js — persistence
```
localStorage key: 'jfvr.settings.v1'
{ subPlacement: {y,z,width}, subOpacity, subMode, subAlpha,
projection, stereo, ctlAutoHideMs }
```
Load at boot (merge over defaults, validate ranges), save debounced ~400 ms after
any change. A corrupt/absent value must fall back to defaults silently.
---
# ADDENDUM 5 — gaze pointing + lazy-follow panel + panel size
## Why (measured on device)
The hand's `targetRaySpace` is an **emulated** ray (Meta documents this). Measured
on a real Quest 3 while the panel was straight ahead:
```
hand:right origin=(-0.317,-0.054,-0.248) dir=(-0.874, 0.461, -0.154)
^ 87 % sideways, 15 % forward
```
Aiming with it lands somewhere the user is not looking. Gaze + pinch is Meta's
documented recommendation for hands and needs no ray accuracy at all.
## Changes
| # | Change |
| --- | --- |
| 1 | Pointer rays are computed in **LOCAL space** (`__JFVR.refSpace`), not viewer space. |
| 2 | Hands aim with the **gaze ray** (head forward in local space). No beam - a guide line from the hand would be misleading because the hand is not aiming. The on-panel reticle is the indicator. |
| 3 | Controllers keep their own `targetRaySpace` ray **and** the laser beam; a controller ray is accurate and a laser is expected. Selected by `pointer.beam`. |
| 4 | The control panel layer's **space is `refSpace`** and holds a stable world pose. A head-locked panel would always put the view centre on its own centre, so no button could ever be targeted by gaze. |
| 5 | 90-main.js drives a **lazy follow**: it only re-centres the panel once the head has turned > 26 deg away from it, easing with a 0.45 s time constant. |
| 6 | `subtitles` is **no longer persisted**. Persisting it meant a session where the user toggled subtitles off came back with the layer never created - which looked like a load failure. Always ON at boot; the toggle is per-session. |
| 7 | Subtitle **distance** range extended: the near clamp is now **0.5 m** (was 0.7 m). |
---
# ADDENDUM 6 — subtitle depth vs size decoupling (angular-size lock)
## Why (owner requirement)
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
Eye tracking is **not** exposed to WebXR on Quest (HANDOFF §4), so the strip's depth
cannot follow the user's gaze automatically. What matters instead is that the DEPTH
control must not change the strip's apparent size: otherwise pulling the subtitles to
a nearby object makes the text huge (and, with the head-locked quad, drops it out of
the lower view). See HANDOFF §8 for the literature this is based on.
## Model
`state.subLockAngle` (default `true`) + `state.subAngleDeg` (default `69.2`).
While **locked**, a depth change holds TWO things:
```
width = 2 * |z| * tan(subAngleDeg / 2) angular width (theta)
y = (y0 / z0) * z angular height (the y/z ratio)
```
so the strip keeps both its angular size and its screen position — only depth
changes. The default pair is bit-for-bit the pre-ADDENDUM-6 look (2.0 m at 1.45 m →
69.2°), so nothing moves for anyone who does not touch the control.
While **unlocked** everything behaves exactly as at ADDENDUM 4: `width` and `z` are
independent ("pull closer = bigger").
Ranges: `theta` ∈ [20, 120]°, `z` ∈ [-2.6, -0.35] m (near clamp moved from 0.5 m to
0.35 m — the whole point is matching very close objects). Under the lock the width
floor drops to 0.05 m (the angle clamp is the real limit); free mode keeps 0.2 m.
## 10-subtitles.js — API additions
```js
setLockAngle(on) / getLockAngle(): boolean
setAngleDeg(deg) / getAngleDeg(): number // locked angular width, degrees
getPlacement(): { y, z, width } // the resolved, clamped triple
```
- `setPlacement({ y, z, width })` under the lock:
- `z` without `y` → depth-only: `y` is re-derived from the held y/z ratio and
`width` from `subAngleDeg`;
- `width` without `z` → size change: `subAngleDeg` is re-derived from the requested
width at the current depth (the derived width then wins);
- `y` + `z` (grab & drag) → the caller's `y` wins, only `width` is derived.
- `resetPlacement()` restores `{ y:-0.42, z:-1.45, width:2.0 }` **and**
`subAngleDeg = 69.2`; the lock flag is left alone.
- New diagnostics: `stats.subAngleDeg`, `stats.subLockAngle`, `stats.subYOverZ`, and
`stats.subTextDeg` — the ACTUAL angular height of the drawn subtitle ink, in
degrees. arXiv 2004.01545 measures preferred angular size (character height) at
~16–18 dmm ≈ 0.9–1.0° for medium/long text and ~32–40 dmm ≈ 1.8–2.3° for
titles/headlines; Google's dmm is 1 mm of character height at 1 m. Use
`subTextDeg` to pick a `subAngleDeg` default from real measurements later; the
shipped default is deliberately the on-device-proven look.
## 20-controls.js — subtitle row is now NINE buttons
```
sub_up sub_down sub_near sub_far sub_smaller sub_bigger sub_opaque_down sub_opaque_up sub_reset
上 下 近 远 小 大 暗 亮 复位
```
- The vertical-position pair (up/down) and the depth pair (near/far) are the
ADDENDUM-6 additions.
- `view.sub` gains `angleDeg`, `lock` and `dist` (= `|z|`). The caption ALWAYS
leads with the depth in metres — `字幕 距1.45m · 69°` while locked,
`字幕 距1.45m · 2.0m` while free — because the depth is the number the user has
to match against the object being watched. Opacity stays right-aligned.
- **`readSubtitles()` bug fixed**: it only read the module-local `S.subtitlesOn`,
which `90-main.js` never set (it calls `setVideo(video)` with no `getExtra`), so
the CC button always rendered as OFF. It now falls back to the shared
`state.subtitles`. The button also got an unambiguous on/off treatment: an
accent pill + `字幕开` when on, a slashed CC glyph + `字幕关` when off.
- Hit pills: pitch 0.10, half width 0.052 (a hair of overlap → no dead zone);
visual width 0.092; caption font 0.048·h, button label font 0.040·h. The bar
itself is unchanged (`PANEL_M.expanded` = 0.72 × 0.21 m, canvas 1440 × 420 px).
## 90-main.js
- Persists `subLockAngle` / `subAngleDeg` in `jfvr.settings.v1`; while locked the
stored `subPlacement.width` is re-derived on load, so `subAngleDeg` + `z` is what
restores the user's look.
- In-page panel: a `距离与大小` toggle (锁定视角尺寸 / 自由), and a `视角宽度`
slider in degrees shown while locked / the old `宽度` slider in metres while free.
All sliders re-read their value whenever a setter runs, so VR-bar and grab changes
show up in an already-open panel.
- `sub_up` / `sub_down` step `y` by ±0.04 m. `sub_near` / `sub_far` step `z` by
±0.15 m. `sub_smaller` / `sub_bigger` step `subAngleDeg` by ∓5° while locked,
else `width` by ∓0.15 m.
- Grab & drag: while locked, a depth drag also carries `y` by the grab-start y/z
ratio (vertical hand motion still adds on top), so the strip cannot slide out of
the lower view when it is pulled close.
---
# ADDENDUM 7 — summon gating, lock badge, pre-VR panel trim
Owner feedback after the ADDENDUM 6 pass:
> 控制面版的唤出太灵敏了,只有当指示线指向面板附近时,面板再显示出来即可;在 vr 空间
> 右侧边缘增加一个锁的图标,当选中上锁后,面板不会启动;在刚唤出面板时,0.5s 内的操作
> 应该忽略,否则可能误触。在进入"vr 模式"时,字幕相关有很多滑块配置,移除哪些配置,
> 在没有在真实 vr 空间中的调整,都是不准确的,没必要在进入前指定。视角宽度保持默认即可。
> 字幕相关只保留开启和关闭即可。"vr 播放"这个按钮,不要挡住原有播放器的选项,现在似乎
> 挡住了原播放器的全屏按钮;手部指向选项直接移除吧,没人会只使用食指作为指向。
## 1. Summon gating (90-main.js)
There were **two** independent reasons the bar appeared on *every* pinch, and both
are gone:
| Was | Now |
| --- | --- |
| `onSelectStart` called `Controls.show()` unconditionally | listener removed; summoning is decided in `pointerDown()` from the actual aim |
| `pointerDown()` called `Controls.show()` before checking the hit | `show()` is called only when the ray lands on / near the bar |
```
bar hidden + aim ON the bar -> show(), record barShownAt, return (the summoning press never clicks)
bar hidden + aim NEAR the bar-> same (padded window, see below)
bar hidden + aim elsewhere -> nothing
bar hidden + locked -> nothing
bar shown + < PANEL_ARM_MS(500) since it appeared -> ignore
bar shown + aim ON the bar -> show() (extends auto-hide) + handleSelect
```
Barely-visible: the summoning press is *consumed* — it does not also press a button,
and for 500 ms after the bar appears every activation is ignored. Otherwise the hand
travelling onto the bar right after the summon fires a random button.
The summon window is deliberately wider than the button window: `pointer.hit` (used
for the cursor and for button hits) uses `panelHit(..., 0.25, 0.6)`, while summoning
falls back to `panelHit(..., 0.5, 0.9)`. A ray that merely grazes the bar can still
bring it up; the 500 ms arm is what protects against mis-clicks.
Stats: `stats.pressAt` now reads `SUMMON (armed 500ms)`, `IGNORED (arming)`,
`MISS (not aiming at the panel)` or `MISS (panel locked)`.
## 2. Lock badge (20-controls.js + 90-main.js)
An **always-visible** small quad (`LOCK_X/Y/Z = 0.86 / -0.34 / -1.45`, `LOCK_M = 0.075`,
texture 128²) placed in the **same `refSpace`** as the control bar — to the right of
it. While `state.ctlLock` is true, nothing summons the bar; the badge stays visible
and hit-testable so it can be unlocked again.
Why `refSpace` and not a head-locked HUD corner: on this setup the only pointing
device is the hand's ray, and a head-locked badge could never be aimed at. "Right-hand
edge of the VR space" is therefore implemented as "right of the control bar, in the
same world-locked space".
- Unlocked: dim (60 % white) padlock with the shackle open.
- Locked: accent (`#4ea1ff`) filled padlock, shackle closed, keyhole punched out.
- Hit window: `panelHit(..., 0.6, 0.6)` ≈ 6.5° across at 1.45 m — the badge itself is
only ~3°, so it is easy to hit with a coarse hand ray.
- The lock is **per VR session**: `enterVR()` sets `ctlLock = false`, `onSessionEnd()`
resets it too. A stuck lock would look exactly like a broken plugin, so it is never
restored from `localStorage`.
New API:
```js
Controls.getLockLayer(): XRLayer | null
Controls.setLock(on) / getLock(): boolean
```
Stats: `stats.ctlLock`.
## 3. Pre-VR settings panel trimmed (90-main.js)
Sliders set on a flat screen do not transfer to a headset, so the only subtitle
setting left before entering VR is **开 / 关**:
| Removed | Why |
| --- | --- |
| 模式 (hud/sphere/off) | legacy A/B debug toggle |
| 透明合成, 不透明度 | judge-able only in VR |
| 垂直位置, 距离, 宽度 | now fully replaced by the in-VR row |
| 距离与大小 (lock toggle) | the lock is always ON; it is the whole point |
| 视角宽度 | keeps its default; in-VR `小/大` is the control |
| 手部指向 | the index-ray mode was removed outright |
`视角宽度` / `subAngleDeg` still **persists** — the in-VR `小/大` buttons write it,
and the panel just no longer offers a second, less accurate way to set it. Same for
placement / opacity: set in VR, remembered next time.
Removed with it: `setSubModeLive()`, `setSubLockLive()`, `handRay()` and every
`handRayMode` read/write.
## 4. "VR 播放" button position (90-main.js CSS)
`.jfvr-btn` moved from `bottom: 24px` to `bottom: 120px` — at the viewport's
bottom-right it sat on top of Jellyfin's own OSD, covering its fullscreen button.
`.jfvr-panel` moved to `bottom: 180px` and `max-height: calc(100vh - 200px)` so it
still stacks above the button.
---
# ADDENDUM 7b — the padlock is a PANEL BUTTON, not its own layer
## What was wrong with ADDENDUM 7's badge
On device the badge could not be selected while the panel was shown, but could be
while it was hidden. ADDENDUM 7 had given the padlock **its own quad layer** parked at
`(0.45, -0.34, -1.45)` — 0.45 m to the right of the panel centre, i.e. 0.09 m to the
right of the panel's own right edge.
Aiming is the binding constraint and it was underestimated: the only pointing device
here is the hand ray, and the cursor's own acceptance window (`panelHit` padding
0.25/0.6 on a 0.72 m panel) reaches `x ≈ 0.45` — the panel's right edge. The badge was
**outside every reachable ray**. Worse, its hit test ran on the *panel's* layer
geometry, so whether it fired depended on where the panel happened to be, which is
exactly the "works when hidden, not when shown" report.
## What it is now
An ordinary button on the panel canvas — the `lock` action, fifth in the media row:
```
expanded: [<<<] [>||] [>>>] [CC] [padlock]
0.18 0.36 0.54 0.72 0.90 (hit pill half width 0.092)
```
It travels the same code path as every button that already works: cursor → `hitTest`
→ `handleSelect` → `onAction` → `90-main.js handleControlAction('lock')`.
## The "locked" panel mode
`show()` picks the mode from the lock state, so locking answers "what does a summon
look like now?" with a third mode:
| mode | size (m) | canvas | drawn / hit-testable |
| --- | --- | --- | --- |
| `expanded` | 0.72 × 0.21 | 1440 × 420 | everything |
| `locked` | 0.26 × 0.10 | 520 × 200 | padlock + 解锁 label only |
| `collapsed` | 0.60 × 0.05 | 1200 × 100 | (legacy; no longer entered) |
`hitTest(nx, ny, w, h, modeHint)` now takes the mode explicitly, because "collapsed" (a
thin strip) and "locked" (a small square-ish pill) share no aspect-ratio invariant.
`handleSelect` passes `S.mode`; the aspect-ratio `modeFor()` remains only as a fallback
for external callers.
`hide()` **no longer resets the mode**. The layer therefore keeps the size it will be
shown at, so the summon window that `90-main.js` ray-casts is the full panel rect
(0.72 × 0.21 unlocked, 0.26 × 0.10 locked) instead of a 5 cm strip — and re-showing
never needs a resize pop.
## Lifecycle (all of it in 90-main.js `toggleControlsLock`)
```
lock: S.ctlLock = true; Controls.setLock(true) -> mode 'locked'
Controls.show() -> padlock-only panel, auto-hide restarts
barShownAt = now
unlock: S.ctlLock = false; Controls.setLock(false) -> mode 'expanded'
Controls.show() -> full panel
barShownAt = now -> 0.5 s ignored, so the unlocking
press cannot land on a button
```
Summoning while locked is allowed (it is the only way back out): `pointerDown` no longer
early-returns on `S.ctlLock`, and the summon shows the padlock panel alone.
## Also fixed
`handleSelect()` used to fall back to `(nx, ny) = (0.5, 0.5)` when there was no pointer
hit and no reticle — which silently pressed whatever button lives at the centre. It now
returns `null` and performs nothing.
## API
```js
Controls.setLock(on) / getLock(): boolean // setLock also swaps the panel in place
Controls.hitTest(nx, ny, w, h, mode?) // mode: 'expanded' | 'locked' | 'collapsed'
Controls.PANEL_M / PANEL_PX // now include .locked
```
Stats: `stats.ctlMode` (`expanded` | `locked`), `stats.ctlLock`,
`stats.pressAt` (`SUMMON (locked panel)`, `LOCKED`/`UNLOCKED (armed 500ms)`, …).
+813
View File
@@ -0,0 +1,813 @@
# Jellyfin Web VR — Quest 3 / WebXR MVP
给 Jellyfin Web 播放页注入一个「VR 播放」按钮,在 Meta Quest 3 浏览器里直接进入
VR180 / VR360 沉浸播放,不用再手动切全屏 + 选投影。
---
## 1. 当前部署状态
已注入到 `192.168.123.199` 上的 Jellyfin 容器 `Jellyfin`(当前版本 `mvp-7616c9`,`NS.version = hud-5`,含 ADDENDUM 6 + 7 + 7b):
| 项 | 值 |
|---|---|
| 脚本 | `/jellyfin/jellyfin-web/jfvr-mvp.js`(容器内) |
| 注入点 | `/jellyfin/jellyfin-web/index.html` 的 `</body>` 之前 |
| 备份 | `/vol2/jellyfin/config/_vr_backup/index.html.orig` |
| 两个入口都生效 | `http://192.168.123.199:8096/web/` 与 `https://jellyfin.cat-shark.xyz/web/` |
> ⚠️ 注入写在**容器内文件**里,`jellyfin/jellyfin:latest` 一升级就没了,重跑 `deploy.py` 即可。
> 正式版应改用 **JavaScript Injector 插件**(`n00bcodr/jellyfin-plugins` 的 `12/manifest.json`,targetAbi 12.0.0.0)
> 或 **File Transformation**,这样升级不丢、可开关。
---
## 2. Quest 3 实测数据(已用 ADB + CDP 在真机验证)
### 2.0 设备与能力
```
Quest 3 Android 14 (SDK 34) build UP1A.231005.007.A1
Quest Browser 152.1.0.34.52.1079628208 (Chromium 152.0.7977.83)
UA Mozilla/5.0 (X11; Linux x86_64; Quest 3) ... Chrome/152 ... VR Safari/537.36
```
| 检测项 | `http://192.168.123.199:8096` | `https://jellyfin.cat-shark.xyz` |
|---|---|---|
| `isSecureContext` | false | **true** |
| `navigator.xr` | false | **true** |
| `immersive-vr` | — | **true** |
| **`XRMediaBinding.createEquirectLayer`** | — | **true** ← 首选路径可用 |
| `XRWebGLBinding` | — | true ← 回退路径可用 |
| `MAX_TEXTURE_SIZE` | 8192 | 8192 |
| `MAX_RENDERBUFFER_SIZE` | 16384 | 16384 |
| 8K HEVC `decodingInfo` | supported / powerEfficient | **supported:true, powerEfficient:true** |
结论:**HTTPS 下 Quest 浏览器提供全部所需 WebXR 能力,8K HEVC 走硬件解码。**
### 2.1 网络实测:HTTPS 域名在 Quest 上就是内网直连
```
Quest 3 IPv6 : 2409:8a14:44a:e360:70b9:446e:d5f7:10a7/64
NAS IPv6 : 2409:8a14:44a:e360:520:8aa2:7736:4f12/64 ← 同一 /64
domain AAAA → NAS 自己的局域网 IPv6
服务端连接表 : [NAS]:443 ← [Quest]:35860 (ESTAB, IPv6)
```
60 MB 文件在 Quest 浏览器上实测:
| Run | 吞吐 | TTFB |
|---|---|---|
| 1 | 450 Mbps | 11 ms |
| 2 | 620 Mbps | 7 ms |
| 3 | **702 Mbps** | 12 ms |
对比:从局域网 PC 用 curl 走 IPv6 只有 4.29 MB/s(34 Mbps),走 IPv4 阿里云中转只有 94 KB/s。
> 慢的根因:NAS 上跑着 **`npc` 内网穿透客户端**,域名的 **A 记录 `39.105.149.197` 是阿里云上的 nps 服务端**,
> 走 IPv4 就是 NAS → 阿里云 → 你。**走 IPv6 则完全不出内网。**
>
> 所以 **不需要为了速度退回 HTTP**。8K 片源码率仅 21 Mbps,702 Mbps 有 30 倍余量。
### 2.2 如果仍想用 `192.168.123.199:8096`
WebXR 要求「安全上下文」。实测结果:
| 入口 | `isSecureContext` | `navigator.xr` | WebXR |
|---|---|---|---|
| `http://192.168.123.199:8096` | **false** | **false** | ❌ 需先做 §2.1 |
| `https://jellyfin.cat-shark.xyz` | true | true | ✅ 直接可用 |
用内网 HTTP 地址 —— Quest 浏览器开 flags
Meta Quest 2/3/3S(**OS 79+**)浏览器支持在 HTTP 模式下使用 WebXR,需显式授权:
1. Quest 浏览器地址栏输入 `chrome://flags`
2. 搜索 `insecure`
3. 启用 **`unsafely-treat-insecure-origin-as-secure`**
4. 在下方输入框填:`http://192.168.123.199:8096`
(**必须带 `http://` 和端口号**;多个用逗号分隔)
5. 点空白处失焦 → 屏幕底部出现 **Relaunch** → 点它
6. 回到 `chrome://flags` 确认 flag 仍启用且 URL 已保存
代价:这台设备专属、浏览器更新后可能需要重设。
### 2.3 兜底:让局域网内解析到内网 IPv4
如果某天 IPv6 不可用,且 Quest 落到了 IPv4(阿里云中转)导致慢,最一劳永逸的修法是在
**OpenWrt 路由器(192.168.123.1)** 上加一条静态解析,让局域网内所有设备都解析到内网 IP:
- LuCI:`网络 → DHCP/DNS → 一般设置 → 地址`,添加 `/jellyfin.cat-shark.xyz/192.168.123.199`
- 或 SSH 到路由器:
```sh
uci add_list dhcp.@dnsmasq[0].address='/jellyfin.cat-shark.xyz/192.168.123.199'
uci commit dhcp && /etc/init.d/dnsmasq restart
```
之后 `https://jellyfin.cat-shark.xyz` 在局域网内全程走内网,证书依然有效,所有设备(不只 Quest)都受益。
---
## 3. 实操:如何在真机上验证(ADB + CDP)
```bash
ADB=/opt/SideQuest/resources/platform-tools/adb # SideQuest 自带的 adb
# 1) 看设备(SideQuest 的无线 ADB)
$ADB devices -l
# 2) 把 Quest 浏览器的 DevTools 端口转发出来
$ADB forward tcp:9222 localabstract:chrome_devtools_remote
curl -s http://127.0.0.1:9222/json/version
# 3) 让 Quest 打开某个页面
$ADB shell am start -a android.intent.action.VIEW \
-d "http://192.168.123.199:8096/web/index.html" \
-n com.oculus.browser/.OculusLauncherActivity
# 4) 截图 / OCR
$ADB exec-out screencap -p > /tmp/quest/shot.png # 4128x2208
tesseract /tmp/quest/shot.png - --psm 6
```
拿到 `9222` 后可以直接用 `agent_browser --cdp http://127.0.0.1:9222` 接管 Quest 浏览器,
在页面上跑 `Runtime.evaluate`、读网络、发输入事件。
> 注:Quest 浏览器**不把 `console.log` 转发到 logcat**,所以诊断要走 CDP `Runtime.evaluate` 或页面 UI。
---
## 4. 使用步骤(Quest 3 里)
1. 打开 Jellyfin(§2.1 或 §2.2 任一入口)
2. 进媒体库,播放一个 VR 视频
3. 右下角出现蓝色 **「VR 播放」** 按钮 → 点它
4. 面板里确认 **投影**(180°/360°)和 **立体**(单目/左右/上下)
- 自动按文件名 + 分辨率判断;**8192×4096(2:1)会默认选 180° + 左右**
5. 点 **「进入 VR」** → 进入沉浸模式;摘头显即退出
进不去时展开面板里的 **「诊断信息」**,把内容贴回来。
---
## 5. 在 VR 里修掉的两个关键缺陷
真机(Quest 3 + CDP)实测发现并修复:
### 5.1 卡顿 —— 帧率不匹配(3:2 pulldown)
| 项 | 实测值 |
|---|---|
| 视频帧率 | **59.94 fps** |
| XR 会话默认 | **90 Hz**(实测 87.9 fps) |
| `session.supportedFrameRates` | `[72, 80, 90, 120]` |
| `session.frameRate`(初始) | **0**(未指定) |
90 ÷ 59.94 = 1.5 → 视频帧在显示上被交替重复 1 次、2 次 —— 经典 3:2 pulldown 抖动。
2D 播放时系统会自动匹配刷新率,所以「web 端流畅、进 VR 就卡」。
**修复**:进 VR 后调用 `session.updateTargetFrameRate(120)`。120 ÷ 60 = 2,每帧均匀显示两次。
```
修复前 fps : 87.9
修复后 fps : 114.9 (20 秒稳定 111–115,丢帧 0.000%)
```
### 5.2 字幕丢失 —— libass 被冻结
两个独立原因叠加:
**(a) 字幕画布不在 XR 场景里。** Jellyfin Web 12 用 libass 渲染 ASS 到独立的
`canvas.libassjs-canvas`,和 `<video>` 是两个元素。只把 `<video>` 放进 XR layer,字幕自然没有。
**(b) 进了 VR,libass 就停止渲染。** 实测:
```
window.requestAnimationFrame : 6 秒内 0 次调用 ← 完全停摆(XR 接管了帧循环)
timeupdate : 22 次 ✅
requestVideoFrameCallback : 288 次 ✅
```
Jellyfin 的字幕渲染由 rAF 驱动 → 画布冻结在进 VR 前的最后一帧。
**修复**:安装一个 rAF 代理 —— 进 VR 后把 `requestAnimationFrame` 的回调收进自己的队列,
用 `video.requestVideoFrameCallback` 驱动(它跟着视频帧走,正好是字幕需要的节奏);
不在 VR 时直接透传原生实现,2D 播放完全不受影响。
**字幕层**:用 `XRWebGLBinding.createEquirectLayer` 建第二个动态等距柱状层
(同一立体几何),把画布拷进 `getSubImage(layer, frame).colorTexture` 后经 GL 合成。
### 5.3 字幕分辨率
libass 按屏幕上的视频框尺寸渲染(**1416×708**,在 180° 球面上只有约 **4 px/度**,
而 Quest 3 约 **19 px/度** → 非常糊)。
必须**在进入 VR 之前**放大:XR 会话运行期间 libass 不会重算画布尺寸。
所以放大动作挂在「打开 VR 面板」这个点击上(进 VR 本来就要先开面板)。
```
放大前 : 1416x708
放大后 : 4320x2160 (仍是精确 2:1,约 24 px/度)
```
### 5.4 你的字幕文件本身就是为 SBS 设计的
```
Title: VR Dual-Eye Subtitle
PlayResX: 3840 PlayResY: 1920 ← 2:1,与视频同比例
Style: LeftEye , Alignment=8, MarginL=50 , MarginR=1920, MarginV=700
Style: RightEye, Alignment=8, MarginL=1920 , MarginR=50 , MarginV=700
956 条对白,每条成对出现(LeftEye + RightEye)
```
实测墨迹分布 `left=30, right=30`(完全对称)—— 左半屏给左眼、右半屏给右眼,
正好匹配 `stereo-left-right` 层。**不需要任何转换。**
---
## 6. 硬性约束:必须由用户手势触发
WebXR 规范原文:
> Wait for the window to have *transient activation* …
> If the request was not made while the global object has transient activation, return `false`.
所以 `navigator.xr.requestSession()` **必须同步写在点击处理函数里**,前面不能有任何 `await`
(否则 user activation 丢失,被 UA 拒绝)。代码里这一点是硬性保证的,改动时不要破坏。
---
## 7. 渲染路径
| 路径 | 条件 | 说明 |
|---|---|---|
| **A. `XRMediaBinding.createEquirectLayer`**(首选) | 浏览器支持 WebXR Media Layers | Meta 官方推荐:视频直接交给系统合成器,不经 WebGL 采样。**8K 下唯一靠谱的路** |
| **B. WebGL2 等距柱状光线重建**(回退) | 上面不可用 | 每帧 `texImage2D(video)` 上传整帧,8K 会吃力,仅兜底 |
投影参数:
- VR180 SBS → `layout:'stereo-left-right'`, `centralHorizontalAngle:π`
- VR360 SBS → `layout:'stereo-left-right'`, `centralHorizontalAngle:2π`
- 上下 3D → `layout:'stereo-top-bottom'`
---
## 8. 修改 / 部署 / 回滚
```bash
# 改 jfvr-mvp.js 后重新部署
JF_PW='<ssh密码>' python3 deploy.py
# 完全回滚(恢复原始 index.html 并删除脚本)
JF_PW='<ssh密码>' python3 deploy.py rollback
```
`deploy.py` 用文件 md5 前 6 位作为版本号写进 script 标签的 query,自动破缓存。
---
## 9. Meta 官方规格(决定性依据)
出处:`developers.meta.com/horizon/documentation/web/browser-video/`
- 支持的容器:**MP4 / Ogg / WebM**(**MKV 不在列表**)
- 支持的编码:h.264 / **h.265** / vp8 / vp9 / AV1
- > **For all formats, Browser supports 4K and supports 8K on h.265 and AV1.**
- > **WebXR Media Layers 是显示沉浸式视频最高效的方式。如果可能,围绕这个 API 设计你的体验。**
- Quest 浏览器还内置 2D 页面全屏重投影(可手动选 360/180 + 单目/左右/上下),
但 2D 路径最大显示尺寸 ≤ 7680×3840,且每次要手动选 —— 所以走 WebXR 才有意义。
WebXR 规范(`immersive-web/webxr` index.bs):
> Wait for the window to have **transient activation** …
> If the request was not made while the global object has transient activation, return `false`.
所以 `navigator.xr.requestSession()` 必须**同步**写在点击处理里,前面不能有 `await` ——
代码里是硬保证,改动时不要破坏。
---
## 10. 片源情况(实测)
```
/mnt1/123 → 61 个 VR 文件,全部:
8192 × 4096 | HEVC Main / level 6.1 | ~60 fps | 8–25 Mbps | .mp4 (faststart 已确认)
```
即 **VR180 SBS**,每眼 4096×4096 的 180° 等距柱状。
> ⚠️ **必须保持 Direct Play**。服务器的 Pentium Gold 8505 + UHD 核显编不了 8K
> (Alder Lake-N 编码上限约 4K),一旦 Jellyfin 触发转码,分辨率腰斩且 CPU 打满。
> 诊断面板里的 `playbackMode` 会显示当前是 DirectPlay 还是 HLS。
---
## 11. 相关现成实现(备查)
| 项目 | 技术 | 备注 |
|---|---|---|
| `allexxex/jellyfinvr` | Three.js + WebXR | VR180/360/SBS/OU;每帧上传纹理,8K 下是瓶颈 |
| `Mix1C/Jellyfin-360VR-Player` | A-Frame | 仅 360°,无 180 |
| `Lqlsoftware/Jellyfin-vr-plugin` | C# 插件 | 独立播放页 `/VR/Video/{id}/Play` |
| `Toastyice/DeoVRDeeplink` | C# 插件 + `deovr://` | 可完全自动唤起 Quest 上的 DeoVR(deeplink 走系统,不受 user gesture 限制) |
| `IAmParadox27/jellyfin-plugin-file-transformation` | C# 插件 | 正式版注入通道 |
| `n00bcodr/jellyfin-plugins` → JavaScript Injector | 插件 | `12/manifest.json`,targetAbi 12.0.0.0 |
---
# ADDENDUM 2 — in-VR UX pass
Feedback from the first headset test, and what changed.
## Fixed
| Feedback | Fix |
|---|---|
| The progress bar was permanently visible | **Hidden by default.** `show()` appears on any pinch/trigger and auto-hides after `ctlAutoHideMs` (default 4 s, slider 1–10 s). While hidden: no draw, no texture upload, and the layer is dropped from the render state. |
| No guide line for hand/controller input | **Laser beam + cursor.** A thin quad beam (height 6 mm) runs from the input source to the panel, bright at the origin fading to transparent, plus a ring/dot cursor drawn at the exact hit point. |
| Subtitles should sit below the visual centre | Default `subPlacement.y` moved `-0.34 → -0.42` (quad centre, negative = below eye level). |
| Subtitle opacity and position should be adjustable | 4 live sliders in the panel: opacity, vertical position, distance, width. |
Input is now **one code path for hands and controllers**: both expose
`XRInputSource.targetRaySpace`, so each is ray-cast against the panel plane.
Gaze (view centre) is only the fallback when no ray reaches the panel.
## Beam orientation — corrected convention
The first spec said "orient the quad so its local `-Z` follows the ray". That is
geometrically wrong: a quad's local `-Z` is its plane **normal**, so the quad's
plane would end up perpendicular to the ray and its `width` would lie *across*
the beam — a thin face-on bar, not a laser.
Correct:
```
position = origin + dir * (L/2) // L = hit distance, else 1.2 m
width = L, height = 0.006
local +X -> dir // beam axis lies IN the quad plane
local -Z -> perpendicular to dir, biased up so it is never edge-on
```
Verified on device against two different rays — the measured beam midpoint
matched `origin + dir·L/2` to 1 cm, and the beam layer is added/removed with the
pointer.
## UI bug found and fixed this round
Adding 5 sliders grew the settings panel to 784 px while the viewport was 708 px,
so `进入 VR` ended up **off-screen and unclickable** (`elementFromPoint` returned
`null`). The panel is now a flex column with a scrolling body and a **sticky
footer** holding the status line and the CTA, and `max-height: calc(100vh - 104px)`.
## Verified on Quest 3 after this pass
```
layers : [equirect, quad(op=0.85)] ← control bar correctly absent
sub.placement : [-0.42, -1.45, 2.0]
sub.opacity : 0.85 (applied to layer.opacity)
ctl.hidden : true ← default hidden
fps : 110.9 targetFrameRate: 120
errors : []
```
Beam check (pointer frozen so main's per-frame update could not overwrite it):
```
controller ray -> beam w=1.22 pos=(0.13,-0.46,-0.86) matches origin+dir·L/2
hand ray -> beam w=1.45 pos=(-0.12,-0.47,-0.89) (different dir, follows)
pointer null -> beam removed (layers 4 -> 3, beamOn false)
```
---
# ADDENDUM 3 — aiming guide + system gestures
## Root cause of "no guide line" AND "cursor click does nothing"
Both symptoms were one chain of three bugs:
1. `applyBeamVisible()` required `S.shown` — the beam only existed while the
control bar was visible. But the bar only appears on activation, so the user
could never aim at it before pinching. **Chicken-and-egg.**
2. `hook()` early-returned on `!S.shown`, so `presentBeam()` never ran and the
beam texture was never uploaded — the beam could not render even when flagged
visible. **Second, independent blocker.**
3. Only the `select` event was handled. Meta documents hand pinch as
"implemented in the emulated Gamepad attribute for the hand as a **button
press**", which does not reliably raise `select`. So on hands **nothing fired
at all** — no bar, no beam, no click.
Also: `pointer.visible` used to require an actual panel hit, so the aiming guide
would only appear once you were already on target. Now `visible` means "an input
source is pointing" and `hit` is separate (may be null).
## Use the platform's gesture recognition — do not reimplement it
| Platform facility | What it gives us |
| --- | --- |
| `XRInputSource.gamepad` emulated buttons | **hand pinch** (per Meta's WebXR Hands doc) |
| `selectstart` / `select` / `selectend` | controller trigger, and pinch on some builds |
| `targetRaySpace` | one ray API for hands **and** controllers |
So `90-main.js` now polls gamepad buttons for a rising edge **and** listens for
`select`, with a 250 ms dedupe so both paths cannot double-fire. No thumb↔index
joint-distance math is done anywhere.
System-reserved gesture to avoid: **palm pinch** (left hand = menu button, leaves
the session). Ordinary point-and-pinch is ours.
Activation now requires a real ray hit (`pointer.visible && pointer.hit`), so a
stray pinch can no longer fall back to gaze-centre and seek to 50%.
## Verified on device after the fix
```
pointing, NOT hitting the panel -> beamOn: true, ctlHidden: true
beam quad present (h=0.006, w=1.2)
```
## Feasibility: auto-adjusting subtitle depth from scene content
Measured on a real 8K VR180 frame (SBS pair, 320x160 downsample, SAD block
matching +-24 px):
| Band | median disparity | match residual | disparity IQR |
| --- | --- | --- | --- |
| mid frame (y 40-88) | 1 px | 6.1 | — |
| **subtitle band (y 96-144)** | -5 .. -8 px | **13.5-15.3** | **[-16, +1]** |
- Disparity IS extractable and temporally stable (mid band held 1 px across 6 frames).
- But the **subtitle band is the least reliable region**: 2.3x the matching
residual and a 17 px inter-quartile spread — i.e. several depths coexist there,
so no single depth represents it.
- Disparity -> metres needs the VR180 rig's focal length and baseline, which are
not known a priori.
- Compute is NOT the blocker: the naive JS test cost ~250 ms/frame, but a WebGL2
SAD shader would be ~1-2 ms. Not the limiting factor.
Prior art (the depth question is studied; the answer is not "match scene depth"):
- Sidenmark, Kiefer & Gellersen, *Subtitles in Interactive VR: Using Gaze to
Address Depth Conflicts* (IEEE VR'19, Lancaster): depth conflicts between
environment and subtitles cause **occlusion and strain**; they propose
**gaze-based** techniques.
- Shimizu et al. (IEEE 2021): four dynamic subtitle placement methods, balancing
vertical/horizontal/**depth** of eye movement.
- *Gaze-Adaptive Subtitles for 360° Videos in VR* (IEEE 2025): adaptive placement
reduces visual fatigue versus fixed below-the-FOV placement.
**But**: the literature's best technique needs **eye tracking**, and Meta does
not expose `eye-tracking` to WebXR — our negotiated `enabledFeatures` were
`[local, viewer, bounded-floor, layers, hand-tracking, webxr, local-floor]`.
So gaze-adaptive placement is not available to us.
Recommendation if this is pursued: option **A** — use the *mid-band* (reliable)
disparity to bias the depth toward the scene, blended ~30/70 with the comfortable
default, clamped to 1.4-3.0 m, smoothed with a 2-3 s time constant, and scale the
quad proportionally so the **angular text size stays constant** (otherwise the
text visibly pulses). Default OFF. Chasing per-frame scene depth (option C) is
not recommended: disparity noise plus vergence conflict produces discomfort.
---
# ADDENDUM 4 — in-VR subtitle placement
Adjust the subtitles **without leaving VR**. Verified on device this round.
## Grab & drag the strip
Point at the subtitle text, pinch, and move:
- **up / down** → vertical position (drag on a fixed reference plane)
- **pull toward / push away** → depth (the hand's `origin.z` in viewer space;
forward is `-Z`, so pulling back brings the strip closer — like a real object)
- release → the new position is saved
Both the ray and the strip live in **viewer space**, so no world-space maths is
needed. The reference plane is fixed at grab time so the drag stays stable.
### Two deliberate restrictions (both needed)
The strip sits in the lower centre of the view by default and the control bar is
0.30 m tall at `y = -0.28`, so the two **overlap in y**. Without restrictions the
strip would swallow every aim in the lower half of the view.
1. Grabbing only happens when the control bar is **not already shown** — a pinch
while operating the bar never yanks the subtitles.
2. Only the **central 40 %** of the strip counts as grabbable (`nx,ny ∈ 0.30–0.70`),
which is where the text actually is (the crop is letterboxed into the texture).
## In-VR buttons on the control bar
The expanded bar grew from 1.1 × 0.22 m to **1.1 × 0.30 m** and gained a subtitle
row. Verified reachable layout (grid scan, no ambiguous overlaps):
```
y 0.25-0.45 seek track (full width)
y 0.50-0.70 [<<<<][PPPP][>>>>][ZZZZ] back10 / playpause / fwd10 / subs
y 0.85-1.00 [aa][AA][oo][OO][RR] size −/+ · opacity −/+ · reset
```
Actions: `sub_smaller`, `sub_bigger`, `sub_opaque_down`, `sub_opaque_up`, `sub_reset`.
## Persistence
`localStorage['jfvr.settings.v1']`, debounced 400 ms, loaded at boot with range
validation and silent fallback on corruption.
```json
{"subPlacement":{"y":-0.78,"z":-1.45,"width":2.8},"subOpacity":0.45,
"subMode":"hud","subAlpha":"premultiplied","projection":"180","stereo":"sbs",
"subtitles":true,"controls":true,"ctlAutoHideMs":4000}
```
## Verified this round
```
live setPlacement -> layer [-0.7, -2.1, 2.6] immediately
live setOpacity -> 0.35
resetPlacement() -> [-0.42, -1.45, 2]
slider -> localStorage written, and after a page reload the values come back
errors: []
```
Note: the four subtitle sliders in the 2D panel and the in-VR buttons now share
one code path (`setSubPlacementLive` / `setSubOpacityLive` / `setSubModeLive`),
so both save automatically.
---
# ADDENDUM 6 — subtitle depth vs size decoupling (angular-size lock)
## The problem, as the owner put it
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
Fixation is a **vergence** problem: an object at 0.4 m and subtitles at 1.45 m cannot
both be looked at comfortably at the same time. Eye tracking is not available to
WebXR on Quest (`enabledFeatures` has no `eye-tracking`), so the depth cannot be
driven by gaze — see ADDENDUM 3 above, which already concluded the gaze-adaptive
literature is out of reach for us.
What IS in reach is making **depth a pure depth knob**: keeping the strip's angular
size (and its angular position) constant while it moves in depth. That is exactly
the "scale the quad proportionally so the angular text size stays constant" note
from ADDENDUM 3, applied to the manual control.
Measured behaviour of the old control (free mode), which is why it did not work:
| depth | width | strip angular width |
| --- | --- | --- |
| 1.45 m | 2.00 m | 69° |
| 0.70 m | 2.00 m | 110° ← text blows up |
| 0.35 m | 2.00 m | 141° ← text leaves the view |
## What shipped
`state.subLockAngle = true`, `state.subAngleDeg = 69.2` (defaults). While locked:
```
width = 2 · |z| · tan(subAngleDeg / 2) constant angular width
y = (y0 / z0) · z constant angular height
```
so the strip stays at the same spot on the retina and only its **vergence
distance** changes. 69.2° is exactly the old `2.0 m @ 1.45 m`, so the default look
is unchanged. Depth range extended: near clamp **0.5 m → 0.35 m**.
In-VR subtitle row grew from five to nine buttons (the bar itself is unchanged —
`PANEL_M.expanded` = 0.72 × 0.21 m, canvas 1440 × 420 px). The row now carries
vertical position as well as depth, and the caption always leads with the depth:
```
y 0.80-1.00 [↑][↓][<<][>>][aa][AA][oo][OO][RR] 上 下 · 近 远 · 小 大 · 暗 亮 · 复位
caption 字幕 距1.45m · 69° 85%
```
`上`/`下` step the vertical position ±0.04 m (that control did not exist in the bar
before). `近`/`远` step depth ±0.15 m. `小`/`大` step the locked angle ∓5° (or
`width` ∓0.15 m when unlocked). Grab & drag carries `y` by the grab-start y/z
ratio while locked, so a strip pulled close does not slide below the view.
## Two bugs fixed in the same pass
1. **The CC (字幕) button never showed its state.** `Controls.readSubtitles()` only
read the controls module's own `S.subtitlesOn`, which `90-main.js` never sets —
it calls `setVideo(video)` with no `getExtra`. It now falls back to the shared
`state.subtitles`. The button is also unambiguous now: accent pill + `字幕开`
when on, slashed CC glyph + `字幕关` when off.
2. **There was no vertical-position control in VR.** Only grab & drag could move
the strip. The row gained `上`/`下`. (The 2D panel's `垂直位置` slider was
verified working — slider → `-0.70 m`, distance → `-0.80 m` re-derived
`y = -0.385` — so the gap was VR-only.)
## Verification (logic, offline)
The module was driven headlessly in node (state + maths only, no WebGL), holding
`subAngleDeg` at 69.2° and stepping depth:
```
z=-1.45 -> width 2.00, y -0.42 (69.2°)
z=-0.70 -> width 0.97, y -0.205 (69.2°)
z=-0.35 -> width 0.48, y -0.100 (69.2°)
z=-2.60 -> width 3.59, y -0.745 (69.2°)
```
and the 9-button row was grid-scanned through `Controls.hitTest` — 9 distinct
actions, no dead zone between pills. `Controls.draw` renders
`字幕开 · 字幕 距1.45m · 69°` while subtitles are on and locked, `字幕关` when
off, and `字幕 距1.45m · 2.0m` when unlocked. `build.py --check` passes.
Then verified in the **live Quest 3 browser** over CDP after deploying:
```
window.__JFVR.version -> 'hud-2'
state.stats.subLockAngle -> true
state.stats.subAngleDeg -> 69.2
state.errors -> []
Sub.setPlacement({z:-0.70}) -> width 0.97, y -0.20 (still 69.2°)
Sub.setPlacement({z:-0.35}) -> width 0.48, y -0.10 (still 69.2°)
Sub.setPlacement({z:-9}) -> width 3.59, y -0.745 (clamped to -2.6, still 69.2°)
Sub.setAngleDeg(50) -> width 2.42 (size control)
```
## Literature for the default value (not yet acted on)
All of these measure the **character height** angular size; our control is the
whole strip's angular width, so they need `stats.subTextDeg` (new) to compare.
- Kojić et al., *User Experience of Reading in Virtual Reality* (arXiv 2004.01545):
preferred angular size ≈ **16–18 dmm for medium/long text** (0.9–1.0° character
height), 27–32 dmm for 2-word text, vs Google's guidelines of 32 dmm (title) and
40 dmm (headline). Participants also rated **too-large** angular size among the
worst experiences.
- Dingler et al., *VR Reading UIs* (CHI'18 EA): **41 ± 14 dmm** (≈2.35° ± 0.8°).
- The default stays 69.2° deliberately: it is the on-device-proven look. Read
`stats.subTextDeg` in the headset, then decide.
## Persistence
`jfvr.settings.v1` gains `subLockAngle` and `subAngleDeg`; while locked the stored
`subPlacement.width` is re-derived from `subAngleDeg` + `z` on load.
```json
{"subPlacement":{"y":-0.42,"z":-0.35,"width":0.48},
"subLockAngle":true,"subAngleDeg":69.2,"subOpacity":0.85,
"subMode":"hud","subAlpha":"premultiplied","projection":"180","stereo":"sbs",
"controls":true,"ctlAutoHideMs":4000}
```
---
# ADDENDUM 7 — summon gating, lock badge, pre-VR panel trim
## The bar appeared on every pinch
Two independent causes, both removed:
1. `session.addEventListener('selectstart', onSelectStart)` called `Controls.show()`
unconditionally — any pinch anywhere in the view-summoned the bar.
2. `pointerDown()` called `Controls.show()` *before* it looked at the hit.
Now the bar is summoned only when the pointing ray lands on (or near) the bar:
```
bar hidden + aim on/near the bar -> show(), consume the press, arm for 500 ms
bar hidden + aim elsewhere -> nothing
bar shown + within 500 ms -> ignored
bar shown + aim on the bar -> handleSelect (and show() to extend auto-hide)
```
While LOCKED the same rule applies, except that what comes up is the padlock panel
alone — see ADDENDUM 7b below.
The summon window is wider than the hit window (`panelHit` padding 0.5/0.9 vs
0.25/0.6), because a ray that grazes the bar should still be able to bring it up —
the 500 ms arm is what prevents the accidental button press, not pixel precision.
**Superseded by ADDENDUM 7b:** the padlock was originally a separate quad layer to the
right of the bar. That did not work on device — the hand ray cannot reach that far
right. The padlock is now a normal panel button and "locked" is a third panel mode.
## Pre-VR settings panel trimmed
Sliders set on a flat screen do not predict what reads well in a headset, so the
subtitle section is now just 开 / 关. Removed: 模式, 透明合成, 不透明度, 垂直位置,
距离, 宽度, 距离与大小 (lock toggle), 视角宽度, 手部指向. `subAngleDeg`, placement
and opacity still persist — they are written by the in-VR controls.
The 手部指向 option (system ray vs index-finger ray) is gone together with its code
path; the system ray is the only mode (an index-finger ray injects fingertip jitter
into the cursor — see HANDOFF §6.6).
## "VR 播放" no longer covers the player's own controls
`.jfvr-btn` moved from `bottom: 24px` to `bottom: 120px`; at the viewport corner it
sat on Jellyfin's OSD fullscreen button. `.jfvr-panel` moved to `bottom: 180px`
(`max-height: calc(100vh - 200px)`) so the panel still stacks above the button.
## Verification
`build.py --check` passes. A local harness (a `file://` page with a fake 8192x4096
`<video>`, a fake libass canvas and stub `XRWebGLBinding`/`XRXRigidTransform`/
`createQuadLayer` classes) exercised the real bundle without a headset:
```
Controls.create(stubSession) -> ok, 4 quad layers (bar, beam x2, lock)
Controls.getLockLayer() -> the lock quad
Controls.setLock(true) -> stats.ctlLock = true, no throw
runFrameHooks(fakeFrame) -> presentLock/present ran, errors: []
panel DOM -> 画面: 投影/立体 · 字幕: 显示 · 控制: 控制条/自动收起
.jfvr-btn computed bottom -> 120px
drawLockBadge() -> 16 canvas ops, no throw (locked and unlocked)
```
Live Quest 3 over CDP after deploy: `__JFVR.version = 'hud-4'`,
`stats.subLockAngle = true`, `errors: []`.
---
# ADDENDUM 7b — padlock as a panel button + the "locked" panel mode
## The failure on device
> 我有新发现,在面板唤出状态时其无法点击,但面板消失后,其可以点击。
ADDENDUM 7's padlock was its own quad layer at `(0.45, -0.34, -1.45)` — 0.45 m right of
the panel centre, 0.09 m right of the panel's own right edge. Two things were wrong:
1. **It was outside every reachable ray.** Aiming is the binding constraint with hands:
a hand ray plus the cursor's acceptance window (`panelHit` 0.25/0.6 on a 0.72 m panel)
reaches x ≈ 0.45 — the panel's right edge. Nothing could aim 0.09 m past it.
2. **Its hit test used the panel's layer geometry.** Whether it fired therefore depended
on the panel's own size/position, which is why it behaved differently shown vs hidden.
## The fix: make it an ordinary button
The padlock is now the fifth button in the media row and uses the same code path as
everybutton that already worked (cursor → `hitTest` → `handleSelect` → `onAction`):
```
expanded: [<<<] [>||] [>>>] [CC] [padlock]
0.18 0.36 0.54 0.72 0.90 (hit pill half width 0.092)
```
## The behavior the owner asked for
> 想要的是:唤出面板,然后可以上锁,上锁后锁头图标和面板一样自动隐藏,下次触发面板
> 唤出时,只展示锁本身,不展示进度条等能调整的项目。(解锁后的0.5s的动做也忽略)
A third panel mode does exactly that:
| mode | size (m) | canvas | drawn / hit-testable |
| --- | --- | --- | --- |
| `expanded` | 0.72 × 0.21 | 1440 × 420 | everything |
| `locked` | 0.26 × 0.10 | 520 × 200 | the padlock + 解锁 label ONLY |
```
summon (unlocked) -> expanded panel, auto-hides after ctlAutoHideMs
press 上锁 -> panel becomes padlock-only IN PLACE, auto-hide timer restarts
(so the padlock disappears on its own, like the panel always did)
summon (locked) -> padlock alone; no progress bar, no rows, nothing adjustable
press 解锁 -> full panel returns, and 500 ms of presses are ignored
```
Because `hide()` no longer resets the mode, the layer keeps the size it will be shown
at: the summon window stays the full panel rect (0.72 × 0.21 unlocked, 0.26 × 0.10
locked) and re-showing never pops a resize. `hitTest()` takes the mode explicitly now
(`collapsed` and `locked` share no safe aspect-ratio invariant), and `handleSelect()`
no longer falls back to pressing the panel centre when there is no ray hit.
## Verification (offline, real bundle, no headset)
A temporary `file://` harness fed the real `dist/jfvr.js` a fake 8K `<video>`, a fake
libass canvas and stub `XRWebGLBinding` / `XRXRigidTransform` / `createQuadLayer`
classes. The harness file was deleted afterwards.
```
Controls.create(stubSession) -> ok, layers: bar + beam x2
expanded media row (5 centres) -> back10|playpause|fwd10|subs|lock, no dead zone
expanded subtitle row (9 centres) -> 9 distinct actions
locked panel: centre -> {type:'lock'} -> bottom edge / left edge -> null
draw() locked -> draws ONLY '解锁'
draw() expanded -> 上锁 + 字幕 距1.45m · 69° + 00:10 (all intact)
setLock(true) -> stats.ctlMode=locked, layer 0.72x0.21 -> 0.26x0.10, canvas resized
setLock(false) -> stats.ctlMode=expanded, layer back to 0.72x0.21
hide() -> mode KEPT (summon window stays the panel rect)
runFrameHooks() after each switch -> no errors, stats.hookErr / ctlErr null
handleSelect() with no ray hit -> null (was: silently pressed the centre)
```
Also standalone (`node`): the padlock hit region and the mode table above were asserted
directly against `Controls.hitTest` / `Controls.draw` with mock 2D contexts.
Live Quest 3 over CDP after deploy: `__JFVR.version = 'hud-5'`, `stats.ctlMode =
'expanded'`, `stats.ctlLock = false`, `errors: []`.
## Still unverified
The *feel* — how easily the padlock is aimed with the hand ray, and whether 0.5 s is the
right arm delay — can only be judged in the headset.
+82
View File
@@ -0,0 +1,82 @@
#!/usr/bin/env python3
"""
Concatenate src/*.js (in filename order) into dist/jfvr.js.
Each module is already a self-contained IIFE that attaches to window.__JFVR,
so the build is a straight concatenation with a banner. No minification: the
deployed file stays readable in DevTools, which matters because on-device
debugging happens over CDP.
Usage:
python3 build.py # build
python3 build.py --check # build + syntax check every module
"""
import hashlib
import os
import subprocess
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
SRC = os.path.join(HERE, "src")
DIST = os.path.join(HERE, "dist")
OUT = os.path.join(DIST, "jfvr.js")
ORDER = ["00-core.js", "10-subtitles.js", "20-controls.js", "30-hands.js", "90-main.js"]
def sources():
"""Return (path, text) for every module, in load order."""
found = sorted(f for f in os.listdir(SRC) if f.endswith(".js"))
ordered = [f for f in ORDER if f in found]
ordered += [f for f in found if f not in ORDER]
if not ordered:
sys.exit("no sources in %s" % SRC)
return [(f, open(os.path.join(SRC, f), encoding="utf-8").read()) for f in ordered]
def syntax_check():
bad = 0
for name, _ in sources():
p = os.path.join(SRC, name)
r = subprocess.run(["node", "--check", p], capture_output=True, text=True)
status = "ok" if r.returncode == 0 else "FAIL"
print(" %-18s %s" % (name, status))
if r.returncode != 0:
print(r.stderr.strip())
bad += 1
return bad
def main():
if "--check" in sys.argv:
print("syntax check:")
sys.exit(1 if syntax_check() else 0)
mods = sources()
parts = [
"/*!",
" * JFVR (Jellyfin Web VR) — built bundle. DO NOT EDIT.",
" * Edit src/*.js and run: python3 build.py",
" * Modules: " + ", ".join(n for n, _ in mods),
" */",
]
for name, text in mods:
parts.append("\n/* ===== %s ===== */\n" % name)
parts.append(text.rstrip("\n"))
parts.append("\n")
bundle = "\n".join(parts)
os.makedirs(DIST, exist_ok=True)
with open(OUT, "w", encoding="utf-8") as fh:
fh.write(bundle)
digest = hashlib.md5(bundle.encode()).hexdigest()
print("built %s" % OUT)
print(" modules : %d (%s)" % (len(mods), ", ".join(n for n, _ in mods)))
print(" bytes : %d" % len(bundle))
print(" md5 : %s" % digest)
print(" version : %s" % digest[:6])
if __name__ == "__main__":
main()
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env python3
"""
Deploy jfvr-mvp.js into the Jellyfin container and inject the script tag
into jellyfin-web/index.html.
Usage:
JF_PW='<ssh-password>' python3 deploy.py # deploy current jfvr-mvp.js
JF_PW='<ssh-password>' python3 deploy.py rollback # restore original index.html
Env:
JF_HOST (default 192.168.123.199)
JF_USER (default admin)
JF_PW (required) SSH password
JF_CONTAINER (default Jellyfin)
"""
import base64
import hashlib
import os
import subprocess
import sys
import paramiko
HOST = os.environ.get("JF_HOST", "192.168.123.199")
USER = os.environ.get("JF_USER", "admin")
PW = os.environ.get("JF_PW")
CONTAINER = os.environ.get("JF_CONTAINER", "Jellyfin")
HERE = os.path.dirname(os.path.abspath(__file__))
BUILT_JS = os.path.join(HERE, "dist", "jfvr.js")
LEGACY_JS = os.path.join(HERE, "legacy", "jfvr-mvp-v6-working.js")
REMOTE_JS = "/tmp/jfvr-mvp.js"
WEB_DIR = "/jellyfin/jellyfin-web"
BACKUP_DIR = "/vol2/jellyfin/config/_vr_backup"
TAG = '<script plugin="JellyfinVR" version="%s" src="jfvr-mvp.js?v=%s" defer></script>'
def build():
"""Concatenate src/*.js into dist/jfvr.js. Aborts on failure."""
r = subprocess.run(
[sys.executable, os.path.join(HERE, "build.py")],
capture_output=True, text=True, cwd=HERE,
)
print(r.stdout.strip() or r.stderr.strip())
if r.returncode != 0:
sys.exit("build failed")
r = subprocess.run(
[sys.executable, os.path.join(HERE, "build.py"), "--check"],
capture_output=True, text=True, cwd=HERE,
)
if r.returncode != 0:
print(r.stdout)
sys.exit("syntax check failed")
def run(client, script, timeout=180):
"""Run a bash script as root on the remote host."""
b64 = base64.b64encode(script.encode()).decode()
wrapped = (
"echo '%s' | sudo -S -p '' bash -c "
"'echo %s | base64 -d > /tmp/_jfvr.sh; bash /tmp/_jfvr.sh'" % (PW, b64)
)
_, out, err = client.exec_command(wrapped, timeout=timeout)
text = out.read().decode("utf-8", "replace")
errs = err.read().decode("utf-8", "replace")
errs = "\n".join(
line for line in errs.splitlines()
if "password for" not in line and not line.startswith("[sudo]")
)
return text, errs
def main():
if not PW:
sys.exit("JF_PW env var is required")
rollback = len(sys.argv) > 1 and sys.argv[1] == "rollback"
legacy = "--legacy" in sys.argv
if not rollback:
if legacy:
BUILT_JS_LOCAL = LEGACY_JS
print("deploying LEGACY bundle")
else:
build()
BUILT_JS_LOCAL = BUILT_JS
else:
BUILT_JS_LOCAL = None
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(
HOST, username=USER, password=PW, timeout=15,
look_for_keys=False, allow_agent=False,
)
try:
if rollback:
script = f"""
set -e
if [ ! -f "{BACKUP_DIR}/index.html.orig" ]; then
echo "no backup found at {BACKUP_DIR}/index.html.orig"; exit 1
fi
docker cp "{BACKUP_DIR}/index.html.orig" {CONTAINER}:{WEB_DIR}/index.html
docker exec {CONTAINER} rm -f {WEB_DIR}/jfvr-mvp.js
echo "rolled back. tail:"
docker exec {CONTAINER} tail -c 200 {WEB_DIR}/index.html
"""
out, err = run(client, script)
print(out)
if err.strip():
print("--- STDERR ---\n" + err)
return
data = open(BUILT_JS_LOCAL, "rb").read()
md5 = hashlib.md5(data).hexdigest()
version = "mvp-%s" % md5[:6]
tag = TAG % (version, md5[:6])
sftp = client.open_sftp()
with sftp.open(REMOTE_JS, "wb") as fh:
fh.write(data)
sftp.close()
print("uploaded %d bytes, md5=%s, tag=%s" % (len(data), md5, version))
script = f"""
set -e
mkdir -p "{BACKUP_DIR}"
[ -f "{BACKUP_DIR}/index.html.orig" ] || docker cp {CONTAINER}:{WEB_DIR}/index.html "{BACKUP_DIR}/index.html.orig"
docker cp {REMOTE_JS} {CONTAINER}:{WEB_DIR}/jfvr-mvp.js
docker exec {CONTAINER} sed -i -E 's|<script plugin="JellyfinVR"[^>]*></script>||g' {WEB_DIR}/index.html
docker exec {CONTAINER} sed -i 's|</body>|{tag}</body>|' {WEB_DIR}/index.html
echo "--- tail of index.html ---"
docker exec {CONTAINER} tail -c 300 {WEB_DIR}/index.html
echo
echo "--- md5 inside container ---"
docker exec {CONTAINER} md5sum {WEB_DIR}/jfvr-mvp.js
"""
out, err = run(client, script)
print(out)
if err.strip():
print("--- STDERR ---\n" + err)
finally:
client.close()
if __name__ == "__main__":
main()
+5315
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
Executable
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
# Reconnect to the Quest 3 via SideQuest's ADB and expose its browser CDP on :9222
set -u
ADB=${ADB:-/opt/SideQuest/resources/platform-tools/adb}
CDP_PORT=${CDP_PORT:-9222}
if ! [ -x "$ADB" ]; then echo "adb not found at $ADB" >&2; exit 1; fi
echo "== devices =="
$ADB devices -l
if ! $ADB devices | grep -qE '\bdevice$'; then
echo "!! no authorized device. Reconnect SideQuest (wireless ADB) and retry." >&2
exit 2
fi
echo "== forward tcp:$CDP_PORT =="
$ADB forward --remove-all >/dev/null 2>&1
$ADB forward "tcp:$CDP_PORT" localabstract:chrome_devtools_remote
echo "== verify =="
for i in 1 2 3 4 5; do
v=$(curl -s --max-time 3 "http://127.0.0.1:$CDP_PORT/json/version" || true)
if [ -n "$v" ]; then echo "$v"; echo "OK: CDP ready on 127.0.0.1:$CDP_PORT"; exit 0; fi
sleep 1
done
echo "!! CDP not answering on $CDP_PORT" >&2
exit 3
+343
View File
@@ -0,0 +1,343 @@
/*!
* JFVR core — shared namespace, state, helpers, registries.
* Loaded first. Everything else attaches to window.__JFVR.
*
* See INTERFACES.md for the binding contract.
*/
(function () {
'use strict';
if (window.__JFVR && window.__JFVR.__core) return; // idempotent
var NS = window.__JFVR = window.__JFVR || {};
NS.version = 'hud-5'; // hud-4 = ADDENDUM 7; hud-5 = ADDENDUM 7b (padlock in the panel + locked mode)
NS.__core = true;
/* ------------------------------------------------------------------ *
* logging
* ------------------------------------------------------------------ */
NS.log = function () {
var a = Array.prototype.slice.call(arguments);
try {
console.log.apply(console, ['%c[JFVR ' + NS.version + ']', 'color:#4ea1ff;font-weight:bold'].concat(a));
} catch (e) {
try { console.log('[JFVR]', a.join(' ')); } catch (e2) {}
}
};
/* ------------------------------------------------------------------ *
* shared state (modules read/write; integrator owns lifecycle)
* ------------------------------------------------------------------ */
var S = NS.state = {
video: null,
session: null,
refSpace: null,
viewerSpace: null,
projection: '180', // '180' | '360'
stereo: 'sbs', // 'mono' | 'sbs' | 'ou'
subtitles: true,
ctlLock: false, // ADDENDUM 7: control-bar lock badge (per VR session)
subMode: 'hud', // 'hud' | 'sphere' | 'off'
subAlpha: 'premultiplied', // 'premultiplied' | 'straight'
subOpacity: 0.85, // 0..1, applied to the subtitle layer live
// y is the quad CENTRE; negative = below eye level. -0.42 keeps the strip
// clearly under the visual centre rather than crowding it.
subPlacement: { y: -0.42, z: -1.45, width: 2.0 },
// ADDENDUM 6 — depth / size decoupling. With subLockAngle ON, changing the
// strip's DEPTH holds its angular size (subAngleDeg, the strip's full
// angular width in degrees) and its angular height (the y/z ratio), so the
// user can pull the subtitles to the depth of a nearby object without the
// text blowing up or dropping out of the lower view. 69.2 deg is exactly
// the pre-ADDENDUM-6 look: 2.0 m wide at 1.45 m.
subLockAngle: true,
subAngleDeg: 69.2,
controls: true,
ctlAutoHideMs: 4000, // in-VR control bar auto-hide
// written every XR frame by 90-main.js, read by modules
pointer: null, // { visible, source, origin:{x,y,z}, dir:{x,y,z}, hit:{nx,ny}|null }
stats: {},
errors: []
};
// live handles (mirrors of state, convenient for modules)
Object.defineProperty(NS, 'session', {
get: function () { return S.session; },
set: function (v) { S.session = v; }
});
Object.defineProperty(NS, 'video', {
get: function () { return S.video; },
set: function (v) { S.video = v; }
});
Object.defineProperty(NS, 'refSpace', {
get: function () { return S.refSpace; },
set: function (v) { S.refSpace = v; }
});
Object.defineProperty(NS, 'viewerSpace', {
get: function () { return S.viewerSpace; },
set: function (v) { S.viewerSpace = v; }
});
NS.recordError = function (where, e) {
var msg = where + ': ' + ((e && e.message) || e);
S.errors.push(msg);
if (S.errors.length > 40) S.errors.shift();
return msg;
};
/* ------------------------------------------------------------------ *
* DOM helpers
* ------------------------------------------------------------------ */
NS.makeEl = function (tag, className, text) {
var el = document.createElement(tag);
if (className) el.className = className;
if (text != null) el.textContent = text;
return el;
};
NS.clamp = function (v, lo, hi) { return v < lo ? lo : (v > hi ? hi : v); };
/* ------------------------------------------------------------------ *
* GL helpers
* ------------------------------------------------------------------ */
NS.compileShader = function (gl, type, src) {
var sh = gl.createShader(type);
gl.shaderSource(sh, src);
gl.compileShader(sh);
if (!gl.getShaderParameter(sh, gl.COMPILE_STATUS)) {
var info = gl.getShaderInfoLog(sh) || 'shader compile failed';
gl.deleteShader(sh);
throw new Error(info);
}
return sh;
};
// Fullscreen triangle in clip space. Fragment shader gets `in vec2 vUv`
// with vUv.y already flipped so (0,0) is the top-left of the source image.
NS.QUAD_VS = [
'#version 300 es',
'in vec2 aPos;',
'out vec2 vUv;',
'void main(){',
' vUv = vec2(aPos.x * 0.5 + 0.5, 0.5 - aPos.y * 0.5);',
' gl_Position = vec4(aPos, 0.0, 1.0);',
'}'
].join('\n');
NS.bindFullscreenTriangle = function (gl, program) {
var buf = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buf);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 3, -1, -1, 3]), gl.STATIC_DRAW);
var loc = gl.getAttribLocation(program, 'aPos');
gl.enableVertexAttribArray(loc);
gl.vertexAttribPointer(loc, 2, gl.FLOAT, false, 0, 0);
return loc;
};
NS.linkProgram = function (gl, vsSrc, fsSrc) {
var prog = gl.createProgram();
gl.attachShader(prog, NS.compileShader(gl, gl.VERTEX_SHADER, vsSrc));
gl.attachShader(prog, NS.compileShader(gl, gl.FRAGMENT_SHADER, fsSrc));
gl.linkProgram(prog);
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) {
var info = gl.getProgramInfoLog(prog) || 'program link failed';
throw new Error(info);
}
gl.useProgram(prog);
return prog;
};
NS.makeTexture = function (gl) {
var t = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, t);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
return t;
};
NS.xrGl = function (alpha) {
var c = document.createElement('canvas');
return c.getContext('webgl2', {
xrCompatible: true,
alpha: alpha !== false,
premultipliedAlpha: true,
antialias: false,
depth: false,
desynchronized: true
});
};
// 4x4 matrix inverse (column-major, as used by WebXR).
NS.invert4 = function (m) {
var a00 = m[0], a01 = m[1], a02 = m[2], a03 = m[3],
a10 = m[4], a11 = m[5], a12 = m[6], a13 = m[7],
a20 = m[8], a21 = m[9], a22 = m[10], a23 = m[11],
a30 = m[12], a31 = m[13], a32 = m[14], a33 = m[15],
b00 = a00 * a11 - a01 * a10, b01 = a00 * a12 - a02 * a10,
b02 = a00 * a13 - a03 * a10, b03 = a01 * a12 - a02 * a11,
b04 = a01 * a13 - a03 * a11, b05 = a02 * a13 - a03 * a12,
b06 = a20 * a31 - a21 * a30, b07 = a20 * a32 - a22 * a30,
b08 = a20 * a33 - a23 * a30, b09 = a21 * a32 - a22 * a31,
b10 = a21 * a33 - a23 * a31, b11 = a22 * a33 - a23 * a32,
det = b00 * b11 - b01 * b10 + b02 * b09 + b03 * b08 - b04 * b07 + b05 * b06;
if (!det) return new Float32Array(16);
det = 1.0 / det;
return new Float32Array([
(a11 * b11 - a12 * b10 + a13 * b09) * det,
(a02 * b10 - a01 * b11 - a03 * b09) * det,
(a31 * b05 - a32 * b04 + a33 * b03) * det,
(a22 * b04 - a21 * b05 - a23 * b03) * det,
(a12 * b08 - a10 * b11 - a13 * b07) * det,
(a00 * b11 - a02 * b08 + a03 * b07) * det,
(a32 * b02 - a30 * b05 - a33 * b01) * det,
(a20 * b05 - a22 * b02 + a23 * b01) * det,
(a10 * b10 - a11 * b08 + a13 * b06) * det,
(a01 * b08 - a00 * b10 - a03 * b06) * det,
(a30 * b04 - a31 * b02 + a33 * b00) * det,
(a21 * b02 - a20 * b04 - a23 * b00) * det,
(a11 * b07 - a10 * b09 - a12 * b06) * det,
(a00 * b09 - a01 * b07 + a02 * b06) * det,
(a31 * b01 - a30 * b03 - a32 * b00) * det,
(a20 * b03 - a21 * b01 + a22 * b00) * det
]);
};
/* ------------------------------------------------------------------ *
* XR layer + frame-hook registry
*
* Layers are owned by modules; main.js owns pushing the render state.
* ------------------------------------------------------------------ */
var layers = [];
var hooks = [];
var pushRenderState = null; // set by main.js once a session exists
NS.setRenderStatePusher = function (fn) { pushRenderState = fn; };
NS.addLayer = function (layer) {
if (!layer || layers.indexOf(layer) >= 0) return layer;
layers.push(layer);
NS.log('layer added (' + layers.length + ' total)');
if (pushRenderState) { try { pushRenderState(); } catch (e) { NS.recordError('addLayer', e); } }
return layer;
};
NS.removeLayer = function (layer) {
var i = layers.indexOf(layer);
if (i < 0) return;
layers.splice(i, 1);
NS.log('layer removed (' + layers.length + ' left)');
if (pushRenderState) { try { pushRenderState(); } catch (e) { NS.recordError('removeLayer', e); } }
};
NS.getLayers = function () { return layers.slice(); };
NS.clearLayers = function () {
layers.length = 0;
if (pushRenderState) { try { pushRenderState(); } catch (e) {} }
};
NS.addFrameHook = function (fn) {
if (typeof fn !== 'function' || hooks.indexOf(fn) >= 0) return fn;
hooks.push(fn);
return fn;
};
NS.removeFrameHook = function (fn) {
var i = hooks.indexOf(fn);
if (i >= 0) hooks.splice(i, 1);
};
NS.runFrameHooks = function (frame, time, session) {
for (var i = 0; i < hooks.length; i++) {
try { hooks[i](frame, time, session); }
catch (e) {
var msg = NS.recordError('frameHook', e);
if (!S.stats.hookErr) S.stats.hookErr = msg;
}
}
};
NS.clearFrameHooks = function () { hooks.length = 0; };
/* ------------------------------------------------------------------ *
* stereo / projection helpers shared by modules
* ------------------------------------------------------------------ */
NS.stereoLayout = function () {
return S.stereo === 'sbs' ? 'stereo-left-right'
: S.stereo === 'ou' ? 'stereo-top-bottom'
: 'mono';
};
NS.centralAngle = function () {
return S.projection === '360' ? Math.PI * 2 : Math.PI;
};
/* ------------------------------------------------------------------ *
* capabilities probe (safe at load time: no user gesture needed)
* ------------------------------------------------------------------ */
NS.caps = {
ua: navigator.userAgent,
secureContext: !!window.isSecureContext,
hasXR: !!navigator.xr,
immersiveVR: null,
hasXRMediaBinding: typeof window.XRMediaBinding !== 'undefined',
hasEquirectLayer: false,
hasXRWebGLBinding: typeof window.XRWebGLBinding !== 'undefined',
hasHandTracking: typeof window.XRHand !== 'undefined',
hasXRJointSpace: typeof window.XRJointSpace !== 'undefined',
hasGetJointPose: typeof window.XRFrame !== 'undefined' && typeof XRFrame.prototype.getJointPose === 'function',
maxTextureSize: null,
hevc: {},
decoding8k: null
};
(function probe() {
try {
var mb = window.XRMediaBinding;
if (mb && mb.prototype) NS.caps.hasEquirectLayer = typeof mb.prototype.createEquirectLayer === 'function';
var wg = window.XRWebGLBinding;
if (wg && wg.prototype) NS.caps.hasQuadLayer = typeof wg.prototype.createQuadLayer === 'function';
} catch (e) {}
try {
var gl = document.createElement('canvas').getContext('webgl2') ||
document.createElement('canvas').getContext('webgl');
if (gl) NS.caps.maxTextureSize = gl.getParameter(gl.MAX_TEXTURE_SIZE);
} catch (e) {}
try {
['video/mp4; codecs="hev1.1.6.L183.B0"',
'video/mp4; codecs="hvc1.1.6.L183.B0"',
'video/mp4; codecs="hev1"',
'video/mp4; codecs="hvc1"'].forEach(function (cs) {
NS.caps.hevc[cs] = (typeof MediaSource !== 'undefined' && MediaSource.isTypeSupported(cs)) ||
document.createElement('video').canPlayType(cs) || 'no';
});
} catch (e) {}
if (navigator.xr && navigator.xr.isSessionSupported) {
navigator.xr.isSessionSupported('immersive-vr')
.then(function (v) { NS.caps.immersiveVR = v; })
.catch(function (e) { NS.caps.immersiveVR = 'err:' + e.message; });
}
})();
NS.capsDiag = function () {
var c = NS.caps;
return [
'origin : ' + location.origin,
'secureContext : ' + c.secureContext,
'immersive-vr : ' + c.immersiveVR,
'XRMediaBinding : ' + c.hasXRMediaBinding,
'XRWebGLBinding : ' + c.hasXRWebGLBinding,
'createQuadLayer : ' + c.hasQuadLayer,
'hand tracking API : ' + c.hasHandTracking + ' / jointSpace=' + c.hasXRJointSpace + ' / getJointPose=' + c.hasGetJointPose,
'MAX_TEXTURE_SIZE : ' + c.maxTextureSize,
'HEVC : ' + JSON.stringify(c.hevc)
].join('\n');
};
})();
+1022
View File
File diff suppressed because it is too large Load Diff
+1940
View File
File diff suppressed because it is too large Load Diff
+569
View File
@@ -0,0 +1,569 @@
/* 30-hands.js — __JFVR.Hands
*
* Draws the official Meta generic-hand models (the @webxr-input-profiles GLBs)
* so the user sees their real hands in VR, posed from the 25 WebXR joint poses
* the runtime reports. This replaces the hand-rolled laser beam / cursor.
*
* Three.js draws the models, but it does NOT own the XR session. This page
* already composes several layers (an XRMediaBinding equirect video layer plus
* quad layers for subtitles and the control panel); calling
* renderer.xr.setSession() would install a baseLayer, which is illegal next to
* `layers`. So instead:
* - we build our own xrCompatible WebGL2 context,
* - create a projection layer with XRWebGLBinding and RETURN it, so
* 90-main.js can put it in updateRenderState({ layers }),
* - drive renderer.render() by hand from a __JFVR frame hook, binding the
* layer's swapchain framebuffer and each view's viewport ourselves.
*
* Interface (INTERFACES.md): init() / create(session) / destroy() / isReady() /
* getLayer(). Nothing here reads document, the network or WebXR at load time;
* only init()/create() do.
*/
(function () {
'use strict';
var NS = (typeof window !== 'undefined' && window.__JFVR) ? window.__JFVR : null;
if (!NS || NS.Hands) return; // core missing, or already installed
/* ------------------------------------------------------------------ *
* constants
* ------------------------------------------------------------------ */
var CDN = 'https://cdn.jsdelivr.net/npm/';
var THREE_VERSION = 'three@0.160.0';
var THREE_URL = CDN + THREE_VERSION + '/build/three.module.js';
var ADDONS_URL = CDN + THREE_VERSION + '/examples/jsm/';
var IMPORT_MAP = {
imports: {
'three': THREE_URL,
'three/addons/': ADDONS_URL
}
};
var HAND_URL = {
left: CDN + '@webxr-input-profiles/assets@1.0/dist/profiles/generic-hand/left.glb',
right: CDN + '@webxr-input-profiles/assets@1.0/dist/profiles/generic-hand/right.glb'
};
// The 25 WebXR hand joints. The generic-hand GLB names its joint nodes after
// these exact strings, so a normalised name lookup maps bone -> joint.
var JOINT_NAMES = [
'wrist',
'thumb-metacarpal', 'thumb-phalanx-proximal', 'thumb-phalanx-distal', 'thumb-tip',
'index-finger-metacarpal', 'index-finger-phalanx-proximal', 'index-finger-phalanx-intermediate', 'index-finger-phalanx-distal', 'index-finger-tip',
'middle-finger-metacarpal', 'middle-finger-phalanx-proximal', 'middle-finger-phalanx-intermediate', 'middle-finger-phalanx-distal', 'middle-finger-tip',
'ring-finger-metacarpal', 'ring-finger-phalanx-proximal', 'ring-finger-phalanx-intermediate', 'ring-finger-phalanx-distal', 'ring-finger-tip',
'pinky-finger-metacarpal', 'pinky-finger-phalanx-proximal', 'pinky-finger-phalanx-intermediate', 'pinky-finger-phalanx-distal', 'pinky-finger-tip'
];
/* ------------------------------------------------------------------ *
* state
* ------------------------------------------------------------------ */
var S = {
// asset state: survives sessions, so a re-entry reuses the loaded GLBs
inited: false,
ready: false,
THREE: null,
models: { left: null, right: null },
// session state: torn down by destroy()
session: null,
gl: null,
binding: null,
projectionLayer: null,
fbo: null,
renderer: null,
scene: null,
camera: null,
hookFn: null,
frames: 0
};
/* ------------------------------------------------------------------ *
* helpers
* ------------------------------------------------------------------ */
function log() {
try {
if (NS && typeof NS.log === 'function') { NS.log.apply(NS, arguments); return; }
} catch (e) { /* fall through */ }
try { console.log.apply(console, ['[JFVR hands]'].concat(Array.prototype.slice.call(arguments))); } catch (e2) {}
}
function msg(e) {
return (e && e.message) ? e.message : String(e);
}
function stats() {
if (!NS.state) NS.state = {};
if (!NS.state.stats) NS.state.stats = {};
return NS.state.stats;
}
function setStat(k, v) {
try { stats()[k] = v; } catch (e) {}
}
function setErr(text) {
setStat('handsErr', text);
}
function refSpaceOf() {
var proj = S.projectionLayer;
if (proj && proj.space) return proj.space; // layers with a space
if (NS.refSpace) return NS.refSpace;
return (NS.state && NS.state.refSpace) || null;
}
function normalizeName(n) {
return String(n == null ? '' : n).toLowerCase().replace(/_/g, '-');
}
/* ------------------------------------------------------------------ *
* asset loading (async, kicked off by init())
* ------------------------------------------------------------------ */
// 90-main.js has no import map, so Three's modules cannot resolve their bare
// `three` import until we install one. Must happen before the first dynamic
// import of a bare specifier, which is why init() does it up front.
function ensureImportMap() {
if (!document || !document.head) return;
if (document.querySelector('script[type="importmap"]')) return;
var im = document.createElement('script');
im.type = 'importmap';
im.textContent = JSON.stringify(IMPORT_MAP);
document.head.appendChild(im);
}
function buildModel(gltf, handedness) {
var root = gltf && (gltf.scene || (gltf.scenes && gltf.scenes[0]));
if (!root) { log('hands[' + handedness + ']: GLB has no scene'); return null; }
// The mesh is skinned: its bind-pose bounds are wrong once the joints move,
// so frustum culling would make the hand pop in and out. Never cull it.
root.traverse(function (o) {
if (o.isMesh || o.isSkinnedMesh) o.frustumCulled = false;
});
var byName = {};
root.traverse(function (o) {
if (o.name) byName[normalizeName(o.name)] = o;
});
var bones = {};
var missing = [];
for (var i = 0; i < JOINT_NAMES.length; i++) {
var joint = JOINT_NAMES[i];
var bone = byName[normalizeName(joint)];
if (bone) bones[joint] = bone;
else missing.push(joint);
}
// Tolerate a renamed/partial skeleton: report once, pose what we can.
if (missing.length) {
log('hands[' + handedness + ']: no GLB bone for ' + missing.join(', ') + ' (joint skipped)');
}
return { root: root, bones: bones, missing: missing };
}
function loadModels(THREE, GLTFLoader) {
var loader = new GLTFLoader();
function one(handedness) {
return new Promise(function (resolve, reject) {
loader.load(HAND_URL[handedness], function (gltf) {
var model = buildModel(gltf, handedness);
if (!model) { reject(new Error(handedness + ': empty scene')); return; }
S.models[handedness] = model;
if (S.scene) S.scene.add(model.root); // layer may already be rendering
resolve();
}, undefined, function (e) {
reject(new Error(handedness + ': ' + msg(e)));
});
});
}
return Promise.all([one('left'), one('right')]);
}
function load() {
return import('three').then(function (THREE) {
S.THREE = THREE;
return import('three/addons/loaders/GLTFLoader.js').then(function (mod) {
return loadModels(THREE, mod.GLTFLoader);
});
}).then(function () {
S.ready = true;
setStat('handsReady', true);
setErr(null);
ensureRenderer(); // if create() already ran, start drawing the models
log('hand models ready');
}).catch(function (e) {
setErr('load: ' + msg(e));
log('hand model load failed:', msg(e));
});
}
function init() {
if (S.inited) return;
S.inited = true;
setStat('handsReady', false);
setStat('handsErr', null);
setStat('handsSources', 'none');
setStat('handsFrames', 0);
try {
ensureImportMap();
var p = load();
if (p && typeof p.catch === 'function') {
p.catch(function (e) { setErr('init: ' + msg(e)); }); // belt and braces
}
} catch (e) {
// Never propagate: a missing import map / document must not kill boot.
setErr('init: ' + msg(e));
log('hands init failed:', msg(e));
}
}
/* ------------------------------------------------------------------ *
* three.js + scene (built lazily, once THREE and a session are available)
* ------------------------------------------------------------------ */
function makeGl() {
if (!document || !document.createElement) return null;
var canvas = document.createElement('canvas');
// A dedicated context: the other modules keep their own for the quad
// layers, and the projection layer must render into its own swapchain.
return canvas.getContext('webgl2', {
xrCompatible: true,
alpha: true,
premultipliedAlpha: true,
antialias: true,
depth: true,
stencil: false
}) || null;
}
function ensureRenderer() {
if (S.renderer || !S.gl || !S.THREE) return S.renderer;
try {
var THREE = S.THREE;
var renderer = new THREE.WebGLRenderer({
canvas: S.gl.canvas,
context: S.gl,
alpha: true,
antialias: true
});
// NEVER renderer.xr.setSession(): that sets a baseLayer and would clash
// with the page's `layers` array. We bind the projection framebuffer and
// render manually from the frame hook instead.
renderer.autoClear = true;
renderer.setClearColor(0x000000, 0); // transparent: the video shows through
var scene = new THREE.Scene();
// The generic-hand GLBs use PBR materials; without a light they render
// black, so give them a hemisphere fill plus a key light.
scene.add(new THREE.HemisphereLight(0xffffff, 0x445566, 2.5));
var key = new THREE.DirectionalLight(0xffffff, 1.4);
key.position.set(0.3, 0.8, 0.5);
scene.add(key);
var camera = new THREE.PerspectiveCamera();
// The hook writes this camera's matrices straight from the XR view, so
// stop three from recomputing matrixWorld from position/quaternion.
camera.matrixAutoUpdate = false;
camera.matrixWorldAutoUpdate = false;
S.renderer = renderer;
S.scene = scene;
S.camera = camera;
['left', 'right'].forEach(function (h) {
if (S.models[h]) scene.add(S.models[h].root);
});
log('three renderer ready for hands');
} catch (e) {
setErr('renderer: ' + msg(e));
log('renderer setup failed:', msg(e));
S.renderer = null;
S.scene = null;
S.camera = null;
}
return S.renderer;
}
/* ------------------------------------------------------------------ *
* XR projection layer + manual render
* ------------------------------------------------------------------ */
// Resolve the framebuffer + viewport to render `view` into.
//
// Two shapes are accepted, in order:
// 1. a projection layer that exposes `framebuffer` + `getViewport(view)`;
// 2. the WebXR Layers spec path, XRWebGLBinding.getViewSubImage(layer, view),
// which returns this view's viewport plus the swapchain colour/depth
// textures; we attach those to a framebuffer of our own. (The spec gives
// XRProjectionLayer no framebuffer and no getViewport(), so on a real
// runtime branch 2 is the one that runs.)
function bindViewTarget(view) {
var gl = S.gl;
var proj = S.projectionLayer;
var binding = S.binding;
if (!gl || !proj) return null;
if (proj.framebuffer && typeof proj.getViewport === 'function') {
var direct = null;
try { direct = proj.getViewport(view); } catch (e) { direct = null; }
if (direct) {
gl.bindFramebuffer(gl.FRAMEBUFFER, proj.framebuffer);
return { framebuffer: proj.framebuffer, viewport: direct };
}
}
if (binding && typeof binding.getViewSubImage === 'function') {
var sub = null;
try { sub = binding.getViewSubImage(proj, view); } catch (e) { sub = null; }
if (sub && sub.colorTexture) {
if (!S.fbo) S.fbo = gl.createFramebuffer();
gl.bindFramebuffer(gl.FRAMEBUFFER, S.fbo);
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, sub.colorTexture, 0);
if (sub.depthStencilTexture) {
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.DEPTH_ATTACHMENT, gl.TEXTURE_2D, sub.depthStencilTexture, 0);
}
if (sub.viewport) return { framebuffer: S.fbo, viewport: sub.viewport };
}
}
return null;
}
// Blank the layer while Three has not finished loading, so the compositor
// does not show stale content.
function clearLayer(pose) {
var gl = S.gl;
var target = bindViewTarget(pose.views[0]);
if (!target) return;
var vp = target.viewport;
gl.viewport(Math.round(vp.x), Math.round(vp.y), Math.round(vp.width), Math.round(vp.height));
gl.clearColor(0, 0, 0, 0);
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
}
function renderFrame(frame) {
var space = refSpaceOf();
if (!space) return;
var pose = null;
try { pose = frame.getViewerPose(space); } catch (e) { pose = null; }
if (!pose || !pose.views || !pose.views.length) return;
var renderer = S.renderer || ensureRenderer();
if (!renderer) { clearLayer(pose); return; }
var gl = S.gl;
var camera = S.camera;
// The layer colour texture holds BOTH eyes in one framebuffer, so a bare
// gl.clear() would erase the eye already drawn. autoClear stays on (it
// clears to the transparent clear colour above) and the scissor box keeps
// each clear + draw inside that eye's viewport.
gl.enable(gl.SCISSOR_TEST);
for (var i = 0; i < pose.views.length; i++) {
var view = pose.views[i];
var target = bindViewTarget(view);
if (!target) break;
var vp = target.viewport;
var x = Math.round(vp.x), y = Math.round(vp.y);
var w = Math.round(vp.width), h = Math.round(vp.height);
if (w <= 0 || h <= 0) continue;
gl.viewport(x, y, w, h);
gl.scissor(x, y, w, h);
camera.projectionMatrix.fromArray(view.projectionMatrix);
camera.projectionMatrixInverse.copy(camera.projectionMatrix).invert();
camera.matrixWorldInverse.fromArray(view.transform.inverse.matrix);
camera.matrixWorld.copy(camera.matrixWorldInverse).invert();
renderer.render(S.scene, camera);
}
gl.disable(gl.SCISSOR_TEST);
}
/* ------------------------------------------------------------------ *
* joint posing
* ------------------------------------------------------------------ */
function poseHand(frame, handedness, hand, space) {
var model = S.models[handedness];
if (!model || !model.bones || !space) return;
for (var i = 0; i < JOINT_NAMES.length; i++) {
var joint = JOINT_NAMES[i];
var bone = model.bones[joint];
if (!bone) continue; // unmatched bone: logged once at load
var jointSpace = null;
try { jointSpace = hand.get(joint); } catch (e) { jointSpace = null; }
if (!jointSpace) continue;
var jp = null;
try { jp = frame.getJointPose(jointSpace, space); } catch (e) { jp = null; }
if (!jp || !jp.transform) continue;
// The generic-hand skeleton is FLAT: every joint node is a direct child
// of the armature root, which sits at identity. A joint's pose in the
// reference space is therefore also its local transform, so feeding the
// matrix to the bone's TRS and letting three propagate is exact. The
// model is metre-scaled (hand span ~0.2 m), so jp.radius is NOT applied:
// the mesh already matches the joint scale.
bone.matrix.fromArray(jp.transform.matrix);
bone.matrix.decompose(bone.position, bone.quaternion, bone.scale);
}
}
function poseHands(frame, session) {
var left = false, right = false;
var sources = (session && session.inputSources) || null;
var space = refSpaceOf();
if (sources) {
for (var i = 0; i < sources.length; i++) {
var src = sources[i];
var hand = src && src.hand;
if (!hand || hand.size !== 25) continue; // XRHand: exactly 25 joints
var handedness = src.handedness;
if (handedness !== 'left' && handedness !== 'right') continue;
if (handedness === 'left') left = true; else right = true;
poseHand(frame, handedness, hand, space);
}
}
// Show a hand only while its input source is present.
if (S.models.left && S.models.left.root) S.models.left.root.visible = left;
if (S.models.right && S.models.right.root) S.models.right.root.visible = right;
setStat('handsSources', (left || right)
? (left ? 'left' : '') + (left && right ? ',' : '') + (right ? 'right' : '')
: 'none');
}
/* ------------------------------------------------------------------ *
* frame hook
* ------------------------------------------------------------------ */
function hook(frame, time, session) {
if (!S.projectionLayer || !frame) return;
S.frames++;
setStat('handsFrames', S.frames);
poseHands(frame, session);
renderFrame(frame);
void time;
}
function registerHook() {
if (S.hookFn) return;
S.hookFn = hook;
if (typeof NS.addFrameHook === 'function') NS.addFrameHook(hook);
}
/* ------------------------------------------------------------------ *
* lifecycle
* ------------------------------------------------------------------ */
function create(session) {
try {
if (S.projectionLayer && S.session === session) return S.projectionLayer; // idempotent
if (!session) { log('hands.create: no session'); return null; }
destroy(); // drop anything left from a previous session
if (typeof XRWebGLBinding === 'undefined') {
setErr('no XRWebGLBinding');
log('hands.create: XRWebGLBinding unavailable');
return null;
}
var gl = makeGl();
if (!gl) { setErr('no webgl2 context'); log('hands.create: no webgl2 context'); return null; }
var binding = new XRWebGLBinding(session, gl);
var proj = binding.createProjectionLayer({
textureType: 'texture',
colorFormat: gl.RGBA8,
depthFormat: gl.DEPTH_COMPONENT24,
scaleFactor: 1,
alpha: true
});
if (!proj) { setErr('createProjectionLayer returned null'); log('hands.create: no projection layer'); return null; }
S.session = session;
S.gl = gl;
S.binding = binding;
S.projectionLayer = proj;
S.fbo = null;
S.frames = 0;
ensureRenderer(); // no-op until THREE has loaded; the hook retries
registerHook();
setErr(null);
log('hands projection layer created (models ' + (S.ready ? 'ready' : 'loading') + ')');
return proj;
} catch (e) {
setErr('create: ' + msg(e));
log('hands.create failed:', msg(e));
try { destroy(); } catch (e2) { void e2; }
return null;
}
}
function destroy() {
try {
if (S.hookFn && typeof NS.removeFrameHook === 'function') NS.removeFrameHook(S.hookFn);
} catch (e) { void e; }
S.hookFn = null;
try {
if (S.projectionLayer && typeof NS.removeLayer === 'function') NS.removeLayer(S.projectionLayer);
} catch (e) { void e; }
try { if (S.gl && S.fbo) S.gl.deleteFramebuffer(S.fbo); } catch (e) { void e; }
S.fbo = null;
var renderer = S.renderer;
S.renderer = null;
if (renderer) {
// Drop the context with the session: browsers cap live WebGL contexts,
// and the next create() builds a bright new one. Only our own resources
// are touched - the loaded models are kept for reuse.
try { if (typeof renderer.forceContextLoss === 'function') renderer.forceContextLoss(); } catch (e) { void e; }
try { renderer.dispose(); } catch (e) { void e; }
}
S.scene = null;
S.camera = null;
S.binding = null;
S.projectionLayer = null;
S.gl = null;
S.session = null;
S.frames = 0;
setStat('handsFrames', 0);
setStat('handsSources', 'none');
}
function isReady() {
// Models loaded AND a live projection layer + renderer: genuinely drawing.
return !!(S.ready && S.renderer && S.projectionLayer);
}
function getLayer() {
return S.projectionLayer || null;
}
/* ------------------------------------------------------------------ *
* public API
* ------------------------------------------------------------------ */
NS.Hands = {
init: init,
create: create,
destroy: destroy,
isReady: isReady,
getLayer: getLayer
};
})();
+1412
View File
File diff suppressed because it is too large Load Diff