Injects a 「VR 播放」 button into Jellyfin Web's video player page and plays
VR180 / VR360 stereo content on Meta Quest 3 directly in the browser, with
DirectPlay 8K HEVC, in-VR subtitles and an in-VR control panel.
Modules (src/, concatenated in filename order into dist/jfvr.js):
00-core.js namespace, shared state, GL helpers, layer/frame-hook registry
10-subtitles.js libass canvas -> head-locked quad layer, auto-cropped to ink,
with depth/size decoupling (angular-size lock)
20-controls.js in-VR control panel: progress/seek, transport, subtitle rows,
aiming beam, padlock, pure 2D canvas -> XR quad layer
30-hands.js system hand model (Three.js + generic-hand GLB) via a manually
driven projection layer
90-main.js boot, XR session lifecycle, rAF broker (rAF stops during XR),
input dispatch, settings panel, localStorage persistence
Key on-device findings are documented in HANDOFF.md §6 and README.md addenda:
requestAnimationFrame stops during an XR session, 3:2 pulldown at 90 Hz needs
updateTargetFrameRate(120), libass must be resized before entering VR, a quad's
local -Z is its plane normal, and hand aiming must use the gaze ray because the
emulated targetRaySpace points sideways. Quest does not expose eye tracking.
Docs: INTERFACES.md (module contract + addenda), README.md (measurements),
HANDOFF.md (handover notes). Credentials are NOT in the repo: deploy.py reads
the SSH password from the JF_PW environment variable.
814 lines
35 KiB
Markdown
814 lines
35 KiB
Markdown
# Jellyfin Web VR — Quest 3 / WebXR MVP
|
||
|
||
给 Jellyfin Web 播放页注入一个「VR 播放」按钮,在 Meta Quest 3 浏览器里直接进入
|
||
VR180 / VR360 沉浸播放,不用再手动切全屏 + 选投影。
|
||
|
||
---
|
||
|
||
## 1. 当前部署状态
|
||
|
||
已注入到 `192.168.123.199` 上的 Jellyfin 容器 `Jellyfin`(当前版本 `mvp-7616c9`,`NS.version = hud-5`,含 ADDENDUM 6 + 7 + 7b):
|
||
|
||
| 项 | 值 |
|
||
|---|---|
|
||
| 脚本 | `/jellyfin/jellyfin-web/jfvr-mvp.js`(容器内) |
|
||
| 注入点 | `/jellyfin/jellyfin-web/index.html` 的 `</body>` 之前 |
|
||
| 备份 | `/vol2/jellyfin/config/_vr_backup/index.html.orig` |
|
||
| 两个入口都生效 | `http://192.168.123.199:8096/web/` 与 `https://jellyfin.cat-shark.xyz/web/` |
|
||
|
||
> ⚠️ 注入写在**容器内文件**里,`jellyfin/jellyfin:latest` 一升级就没了,重跑 `deploy.py` 即可。
|
||
> 正式版应改用 **JavaScript Injector 插件**(`n00bcodr/jellyfin-plugins` 的 `12/manifest.json`,targetAbi 12.0.0.0)
|
||
> 或 **File Transformation**,这样升级不丢、可开关。
|
||
|
||
---
|
||
|
||
## 2. Quest 3 实测数据(已用 ADB + CDP 在真机验证)
|
||
|
||
### 2.0 设备与能力
|
||
|
||
```
|
||
Quest 3 Android 14 (SDK 34) build UP1A.231005.007.A1
|
||
Quest Browser 152.1.0.34.52.1079628208 (Chromium 152.0.7977.83)
|
||
UA Mozilla/5.0 (X11; Linux x86_64; Quest 3) ... Chrome/152 ... VR Safari/537.36
|
||
```
|
||
|
||
| 检测项 | `http://192.168.123.199:8096` | `https://jellyfin.cat-shark.xyz` |
|
||
|---|---|---|
|
||
| `isSecureContext` | false | **true** |
|
||
| `navigator.xr` | false | **true** |
|
||
| `immersive-vr` | — | **true** |
|
||
| **`XRMediaBinding.createEquirectLayer`** | — | **true** ← 首选路径可用 |
|
||
| `XRWebGLBinding` | — | true ← 回退路径可用 |
|
||
| `MAX_TEXTURE_SIZE` | 8192 | 8192 |
|
||
| `MAX_RENDERBUFFER_SIZE` | 16384 | 16384 |
|
||
| 8K HEVC `decodingInfo` | supported / powerEfficient | **supported:true, powerEfficient:true** |
|
||
|
||
结论:**HTTPS 下 Quest 浏览器提供全部所需 WebXR 能力,8K HEVC 走硬件解码。**
|
||
|
||
### 2.1 网络实测:HTTPS 域名在 Quest 上就是内网直连
|
||
|
||
```
|
||
Quest 3 IPv6 : 2409:8a14:44a:e360:70b9:446e:d5f7:10a7/64
|
||
NAS IPv6 : 2409:8a14:44a:e360:520:8aa2:7736:4f12/64 ← 同一 /64
|
||
|
||
domain AAAA → NAS 自己的局域网 IPv6
|
||
服务端连接表 : [NAS]:443 ← [Quest]:35860 (ESTAB, IPv6)
|
||
```
|
||
|
||
60 MB 文件在 Quest 浏览器上实测:
|
||
|
||
| Run | 吞吐 | TTFB |
|
||
|---|---|---|
|
||
| 1 | 450 Mbps | 11 ms |
|
||
| 2 | 620 Mbps | 7 ms |
|
||
| 3 | **702 Mbps** | 12 ms |
|
||
|
||
对比:从局域网 PC 用 curl 走 IPv6 只有 4.29 MB/s(34 Mbps),走 IPv4 阿里云中转只有 94 KB/s。
|
||
|
||
> 慢的根因:NAS 上跑着 **`npc` 内网穿透客户端**,域名的 **A 记录 `39.105.149.197` 是阿里云上的 nps 服务端**,
|
||
> 走 IPv4 就是 NAS → 阿里云 → 你。**走 IPv6 则完全不出内网。**
|
||
>
|
||
> 所以 **不需要为了速度退回 HTTP**。8K 片源码率仅 21 Mbps,702 Mbps 有 30 倍余量。
|
||
|
||
### 2.2 如果仍想用 `192.168.123.199:8096`
|
||
|
||
WebXR 要求「安全上下文」。实测结果:
|
||
|
||
| 入口 | `isSecureContext` | `navigator.xr` | WebXR |
|
||
|---|---|---|---|
|
||
| `http://192.168.123.199:8096` | **false** | **false** | ❌ 需先做 §2.1 |
|
||
| `https://jellyfin.cat-shark.xyz` | true | true | ✅ 直接可用 |
|
||
|
||
用内网 HTTP 地址 —— Quest 浏览器开 flags
|
||
|
||
Meta Quest 2/3/3S(**OS 79+**)浏览器支持在 HTTP 模式下使用 WebXR,需显式授权:
|
||
|
||
1. Quest 浏览器地址栏输入 `chrome://flags`
|
||
2. 搜索 `insecure`
|
||
3. 启用 **`unsafely-treat-insecure-origin-as-secure`**
|
||
4. 在下方输入框填:`http://192.168.123.199:8096`
|
||
(**必须带 `http://` 和端口号**;多个用逗号分隔)
|
||
5. 点空白处失焦 → 屏幕底部出现 **Relaunch** → 点它
|
||
6. 回到 `chrome://flags` 确认 flag 仍启用且 URL 已保存
|
||
|
||
代价:这台设备专属、浏览器更新后可能需要重设。
|
||
|
||
### 2.3 兜底:让局域网内解析到内网 IPv4
|
||
|
||
如果某天 IPv6 不可用,且 Quest 落到了 IPv4(阿里云中转)导致慢,最一劳永逸的修法是在
|
||
**OpenWrt 路由器(192.168.123.1)** 上加一条静态解析,让局域网内所有设备都解析到内网 IP:
|
||
|
||
- LuCI:`网络 → DHCP/DNS → 一般设置 → 地址`,添加 `/jellyfin.cat-shark.xyz/192.168.123.199`
|
||
- 或 SSH 到路由器:
|
||
```sh
|
||
uci add_list dhcp.@dnsmasq[0].address='/jellyfin.cat-shark.xyz/192.168.123.199'
|
||
uci commit dhcp && /etc/init.d/dnsmasq restart
|
||
```
|
||
|
||
之后 `https://jellyfin.cat-shark.xyz` 在局域网内全程走内网,证书依然有效,所有设备(不只 Quest)都受益。
|
||
|
||
---
|
||
|
||
## 3. 实操:如何在真机上验证(ADB + CDP)
|
||
|
||
```bash
|
||
ADB=/opt/SideQuest/resources/platform-tools/adb # SideQuest 自带的 adb
|
||
|
||
# 1) 看设备(SideQuest 的无线 ADB)
|
||
$ADB devices -l
|
||
|
||
# 2) 把 Quest 浏览器的 DevTools 端口转发出来
|
||
$ADB forward tcp:9222 localabstract:chrome_devtools_remote
|
||
curl -s http://127.0.0.1:9222/json/version
|
||
|
||
# 3) 让 Quest 打开某个页面
|
||
$ADB shell am start -a android.intent.action.VIEW \
|
||
-d "http://192.168.123.199:8096/web/index.html" \
|
||
-n com.oculus.browser/.OculusLauncherActivity
|
||
|
||
# 4) 截图 / OCR
|
||
$ADB exec-out screencap -p > /tmp/quest/shot.png # 4128x2208
|
||
tesseract /tmp/quest/shot.png - --psm 6
|
||
```
|
||
|
||
拿到 `9222` 后可以直接用 `agent_browser --cdp http://127.0.0.1:9222` 接管 Quest 浏览器,
|
||
在页面上跑 `Runtime.evaluate`、读网络、发输入事件。
|
||
|
||
> 注:Quest 浏览器**不把 `console.log` 转发到 logcat**,所以诊断要走 CDP `Runtime.evaluate` 或页面 UI。
|
||
|
||
---
|
||
|
||
## 4. 使用步骤(Quest 3 里)
|
||
|
||
1. 打开 Jellyfin(§2.1 或 §2.2 任一入口)
|
||
2. 进媒体库,播放一个 VR 视频
|
||
3. 右下角出现蓝色 **「VR 播放」** 按钮 → 点它
|
||
4. 面板里确认 **投影**(180°/360°)和 **立体**(单目/左右/上下)
|
||
- 自动按文件名 + 分辨率判断;**8192×4096(2:1)会默认选 180° + 左右**
|
||
5. 点 **「进入 VR」** → 进入沉浸模式;摘头显即退出
|
||
|
||
进不去时展开面板里的 **「诊断信息」**,把内容贴回来。
|
||
|
||
---
|
||
|
||
## 5. 在 VR 里修掉的两个关键缺陷
|
||
|
||
真机(Quest 3 + CDP)实测发现并修复:
|
||
|
||
### 5.1 卡顿 —— 帧率不匹配(3:2 pulldown)
|
||
|
||
| 项 | 实测值 |
|
||
|---|---|
|
||
| 视频帧率 | **59.94 fps** |
|
||
| XR 会话默认 | **90 Hz**(实测 87.9 fps) |
|
||
| `session.supportedFrameRates` | `[72, 80, 90, 120]` |
|
||
| `session.frameRate`(初始) | **0**(未指定) |
|
||
|
||
90 ÷ 59.94 = 1.5 → 视频帧在显示上被交替重复 1 次、2 次 —— 经典 3:2 pulldown 抖动。
|
||
2D 播放时系统会自动匹配刷新率,所以「web 端流畅、进 VR 就卡」。
|
||
|
||
**修复**:进 VR 后调用 `session.updateTargetFrameRate(120)`。120 ÷ 60 = 2,每帧均匀显示两次。
|
||
|
||
```
|
||
修复前 fps : 87.9
|
||
修复后 fps : 114.9 (20 秒稳定 111–115,丢帧 0.000%)
|
||
```
|
||
|
||
### 5.2 字幕丢失 —— libass 被冻结
|
||
|
||
两个独立原因叠加:
|
||
|
||
**(a) 字幕画布不在 XR 场景里。** Jellyfin Web 12 用 libass 渲染 ASS 到独立的
|
||
`canvas.libassjs-canvas`,和 `<video>` 是两个元素。只把 `<video>` 放进 XR layer,字幕自然没有。
|
||
|
||
**(b) 进了 VR,libass 就停止渲染。** 实测:
|
||
|
||
```
|
||
window.requestAnimationFrame : 6 秒内 0 次调用 ← 完全停摆(XR 接管了帧循环)
|
||
timeupdate : 22 次 ✅
|
||
requestVideoFrameCallback : 288 次 ✅
|
||
```
|
||
|
||
Jellyfin 的字幕渲染由 rAF 驱动 → 画布冻结在进 VR 前的最后一帧。
|
||
|
||
**修复**:安装一个 rAF 代理 —— 进 VR 后把 `requestAnimationFrame` 的回调收进自己的队列,
|
||
用 `video.requestVideoFrameCallback` 驱动(它跟着视频帧走,正好是字幕需要的节奏);
|
||
不在 VR 时直接透传原生实现,2D 播放完全不受影响。
|
||
|
||
**字幕层**:用 `XRWebGLBinding.createEquirectLayer` 建第二个动态等距柱状层
|
||
(同一立体几何),把画布拷进 `getSubImage(layer, frame).colorTexture` 后经 GL 合成。
|
||
|
||
### 5.3 字幕分辨率
|
||
|
||
libass 按屏幕上的视频框尺寸渲染(**1416×708**,在 180° 球面上只有约 **4 px/度**,
|
||
而 Quest 3 约 **19 px/度** → 非常糊)。
|
||
|
||
必须**在进入 VR 之前**放大:XR 会话运行期间 libass 不会重算画布尺寸。
|
||
所以放大动作挂在「打开 VR 面板」这个点击上(进 VR 本来就要先开面板)。
|
||
|
||
```
|
||
放大前 : 1416x708
|
||
放大后 : 4320x2160 (仍是精确 2:1,约 24 px/度)
|
||
```
|
||
|
||
### 5.4 你的字幕文件本身就是为 SBS 设计的
|
||
|
||
```
|
||
Title: VR Dual-Eye Subtitle
|
||
PlayResX: 3840 PlayResY: 1920 ← 2:1,与视频同比例
|
||
Style: LeftEye , Alignment=8, MarginL=50 , MarginR=1920, MarginV=700
|
||
Style: RightEye, Alignment=8, MarginL=1920 , MarginR=50 , MarginV=700
|
||
956 条对白,每条成对出现(LeftEye + RightEye)
|
||
```
|
||
|
||
实测墨迹分布 `left=30, right=30`(完全对称)—— 左半屏给左眼、右半屏给右眼,
|
||
正好匹配 `stereo-left-right` 层。**不需要任何转换。**
|
||
|
||
---
|
||
|
||
## 6. 硬性约束:必须由用户手势触发
|
||
|
||
WebXR 规范原文:
|
||
|
||
> Wait for the window to have *transient activation* …
|
||
> If the request was not made while the global object has transient activation, return `false`.
|
||
|
||
所以 `navigator.xr.requestSession()` **必须同步写在点击处理函数里**,前面不能有任何 `await`
|
||
(否则 user activation 丢失,被 UA 拒绝)。代码里这一点是硬性保证的,改动时不要破坏。
|
||
|
||
---
|
||
|
||
## 7. 渲染路径
|
||
|
||
| 路径 | 条件 | 说明 |
|
||
|---|---|---|
|
||
| **A. `XRMediaBinding.createEquirectLayer`**(首选) | 浏览器支持 WebXR Media Layers | Meta 官方推荐:视频直接交给系统合成器,不经 WebGL 采样。**8K 下唯一靠谱的路** |
|
||
| **B. WebGL2 等距柱状光线重建**(回退) | 上面不可用 | 每帧 `texImage2D(video)` 上传整帧,8K 会吃力,仅兜底 |
|
||
|
||
投影参数:
|
||
|
||
- VR180 SBS → `layout:'stereo-left-right'`, `centralHorizontalAngle:π`
|
||
- VR360 SBS → `layout:'stereo-left-right'`, `centralHorizontalAngle:2π`
|
||
- 上下 3D → `layout:'stereo-top-bottom'`
|
||
|
||
---
|
||
|
||
## 8. 修改 / 部署 / 回滚
|
||
|
||
```bash
|
||
# 改 jfvr-mvp.js 后重新部署
|
||
JF_PW='<ssh密码>' python3 deploy.py
|
||
|
||
# 完全回滚(恢复原始 index.html 并删除脚本)
|
||
JF_PW='<ssh密码>' python3 deploy.py rollback
|
||
```
|
||
|
||
`deploy.py` 用文件 md5 前 6 位作为版本号写进 script 标签的 query,自动破缓存。
|
||
|
||
---
|
||
|
||
## 9. Meta 官方规格(决定性依据)
|
||
|
||
出处:`developers.meta.com/horizon/documentation/web/browser-video/`
|
||
|
||
- 支持的容器:**MP4 / Ogg / WebM**(**MKV 不在列表**)
|
||
- 支持的编码:h.264 / **h.265** / vp8 / vp9 / AV1
|
||
- > **For all formats, Browser supports 4K and supports 8K on h.265 and AV1.**
|
||
- > **WebXR Media Layers 是显示沉浸式视频最高效的方式。如果可能,围绕这个 API 设计你的体验。**
|
||
- Quest 浏览器还内置 2D 页面全屏重投影(可手动选 360/180 + 单目/左右/上下),
|
||
但 2D 路径最大显示尺寸 ≤ 7680×3840,且每次要手动选 —— 所以走 WebXR 才有意义。
|
||
|
||
WebXR 规范(`immersive-web/webxr` index.bs):
|
||
|
||
> Wait for the window to have **transient activation** …
|
||
> If the request was not made while the global object has transient activation, return `false`.
|
||
|
||
所以 `navigator.xr.requestSession()` 必须**同步**写在点击处理里,前面不能有 `await` ——
|
||
代码里是硬保证,改动时不要破坏。
|
||
|
||
---
|
||
|
||
## 10. 片源情况(实测)
|
||
|
||
```
|
||
/mnt1/123 → 61 个 VR 文件,全部:
|
||
8192 × 4096 | HEVC Main / level 6.1 | ~60 fps | 8–25 Mbps | .mp4 (faststart 已确认)
|
||
```
|
||
|
||
即 **VR180 SBS**,每眼 4096×4096 的 180° 等距柱状。
|
||
|
||
> ⚠️ **必须保持 Direct Play**。服务器的 Pentium Gold 8505 + UHD 核显编不了 8K
|
||
> (Alder Lake-N 编码上限约 4K),一旦 Jellyfin 触发转码,分辨率腰斩且 CPU 打满。
|
||
> 诊断面板里的 `playbackMode` 会显示当前是 DirectPlay 还是 HLS。
|
||
|
||
---
|
||
|
||
## 11. 相关现成实现(备查)
|
||
|
||
| 项目 | 技术 | 备注 |
|
||
|---|---|---|
|
||
| `allexxex/jellyfinvr` | Three.js + WebXR | VR180/360/SBS/OU;每帧上传纹理,8K 下是瓶颈 |
|
||
| `Mix1C/Jellyfin-360VR-Player` | A-Frame | 仅 360°,无 180 |
|
||
| `Lqlsoftware/Jellyfin-vr-plugin` | C# 插件 | 独立播放页 `/VR/Video/{id}/Play` |
|
||
| `Toastyice/DeoVRDeeplink` | C# 插件 + `deovr://` | 可完全自动唤起 Quest 上的 DeoVR(deeplink 走系统,不受 user gesture 限制) |
|
||
| `IAmParadox27/jellyfin-plugin-file-transformation` | C# 插件 | 正式版注入通道 |
|
||
| `n00bcodr/jellyfin-plugins` → JavaScript Injector | 插件 | `12/manifest.json`,targetAbi 12.0.0.0 |
|
||
|
||
---
|
||
|
||
# ADDENDUM 2 — in-VR UX pass
|
||
|
||
Feedback from the first headset test, and what changed.
|
||
|
||
## Fixed
|
||
|
||
| Feedback | Fix |
|
||
|---|---|
|
||
| The progress bar was permanently visible | **Hidden by default.** `show()` appears on any pinch/trigger and auto-hides after `ctlAutoHideMs` (default 4 s, slider 1–10 s). While hidden: no draw, no texture upload, and the layer is dropped from the render state. |
|
||
| No guide line for hand/controller input | **Laser beam + cursor.** A thin quad beam (height 6 mm) runs from the input source to the panel, bright at the origin fading to transparent, plus a ring/dot cursor drawn at the exact hit point. |
|
||
| Subtitles should sit below the visual centre | Default `subPlacement.y` moved `-0.34 → -0.42` (quad centre, negative = below eye level). |
|
||
| Subtitle opacity and position should be adjustable | 4 live sliders in the panel: opacity, vertical position, distance, width. |
|
||
|
||
Input is now **one code path for hands and controllers**: both expose
|
||
`XRInputSource.targetRaySpace`, so each is ray-cast against the panel plane.
|
||
Gaze (view centre) is only the fallback when no ray reaches the panel.
|
||
|
||
## Beam orientation — corrected convention
|
||
|
||
The first spec said "orient the quad so its local `-Z` follows the ray". That is
|
||
geometrically wrong: a quad's local `-Z` is its plane **normal**, so the quad's
|
||
plane would end up perpendicular to the ray and its `width` would lie *across*
|
||
the beam — a thin face-on bar, not a laser.
|
||
|
||
Correct:
|
||
|
||
```
|
||
position = origin + dir * (L/2) // L = hit distance, else 1.2 m
|
||
width = L, height = 0.006
|
||
local +X -> dir // beam axis lies IN the quad plane
|
||
local -Z -> perpendicular to dir, biased up so it is never edge-on
|
||
```
|
||
|
||
Verified on device against two different rays — the measured beam midpoint
|
||
matched `origin + dir·L/2` to 1 cm, and the beam layer is added/removed with the
|
||
pointer.
|
||
|
||
## UI bug found and fixed this round
|
||
|
||
Adding 5 sliders grew the settings panel to 784 px while the viewport was 708 px,
|
||
so `进入 VR` ended up **off-screen and unclickable** (`elementFromPoint` returned
|
||
`null`). The panel is now a flex column with a scrolling body and a **sticky
|
||
footer** holding the status line and the CTA, and `max-height: calc(100vh - 104px)`.
|
||
|
||
## Verified on Quest 3 after this pass
|
||
|
||
```
|
||
layers : [equirect, quad(op=0.85)] ← control bar correctly absent
|
||
sub.placement : [-0.42, -1.45, 2.0]
|
||
sub.opacity : 0.85 (applied to layer.opacity)
|
||
ctl.hidden : true ← default hidden
|
||
fps : 110.9 targetFrameRate: 120
|
||
errors : []
|
||
```
|
||
|
||
Beam check (pointer frozen so main's per-frame update could not overwrite it):
|
||
|
||
```
|
||
controller ray -> beam w=1.22 pos=(0.13,-0.46,-0.86) matches origin+dir·L/2
|
||
hand ray -> beam w=1.45 pos=(-0.12,-0.47,-0.89) (different dir, follows)
|
||
pointer null -> beam removed (layers 4 -> 3, beamOn false)
|
||
```
|
||
|
||
---
|
||
|
||
# ADDENDUM 3 — aiming guide + system gestures
|
||
|
||
## Root cause of "no guide line" AND "cursor click does nothing"
|
||
|
||
Both symptoms were one chain of three bugs:
|
||
|
||
1. `applyBeamVisible()` required `S.shown` — the beam only existed while the
|
||
control bar was visible. But the bar only appears on activation, so the user
|
||
could never aim at it before pinching. **Chicken-and-egg.**
|
||
2. `hook()` early-returned on `!S.shown`, so `presentBeam()` never ran and the
|
||
beam texture was never uploaded — the beam could not render even when flagged
|
||
visible. **Second, independent blocker.**
|
||
3. Only the `select` event was handled. Meta documents hand pinch as
|
||
"implemented in the emulated Gamepad attribute for the hand as a **button
|
||
press**", which does not reliably raise `select`. So on hands **nothing fired
|
||
at all** — no bar, no beam, no click.
|
||
|
||
Also: `pointer.visible` used to require an actual panel hit, so the aiming guide
|
||
would only appear once you were already on target. Now `visible` means "an input
|
||
source is pointing" and `hit` is separate (may be null).
|
||
|
||
## Use the platform's gesture recognition — do not reimplement it
|
||
|
||
| Platform facility | What it gives us |
|
||
| --- | --- |
|
||
| `XRInputSource.gamepad` emulated buttons | **hand pinch** (per Meta's WebXR Hands doc) |
|
||
| `selectstart` / `select` / `selectend` | controller trigger, and pinch on some builds |
|
||
| `targetRaySpace` | one ray API for hands **and** controllers |
|
||
|
||
So `90-main.js` now polls gamepad buttons for a rising edge **and** listens for
|
||
`select`, with a 250 ms dedupe so both paths cannot double-fire. No thumb↔index
|
||
joint-distance math is done anywhere.
|
||
|
||
System-reserved gesture to avoid: **palm pinch** (left hand = menu button, leaves
|
||
the session). Ordinary point-and-pinch is ours.
|
||
|
||
Activation now requires a real ray hit (`pointer.visible && pointer.hit`), so a
|
||
stray pinch can no longer fall back to gaze-centre and seek to 50%.
|
||
|
||
## Verified on device after the fix
|
||
|
||
```
|
||
pointing, NOT hitting the panel -> beamOn: true, ctlHidden: true
|
||
beam quad present (h=0.006, w=1.2)
|
||
```
|
||
|
||
## Feasibility: auto-adjusting subtitle depth from scene content
|
||
|
||
Measured on a real 8K VR180 frame (SBS pair, 320x160 downsample, SAD block
|
||
matching +-24 px):
|
||
|
||
| Band | median disparity | match residual | disparity IQR |
|
||
| --- | --- | --- | --- |
|
||
| mid frame (y 40-88) | 1 px | 6.1 | — |
|
||
| **subtitle band (y 96-144)** | -5 .. -8 px | **13.5-15.3** | **[-16, +1]** |
|
||
|
||
- Disparity IS extractable and temporally stable (mid band held 1 px across 6 frames).
|
||
- But the **subtitle band is the least reliable region**: 2.3x the matching
|
||
residual and a 17 px inter-quartile spread — i.e. several depths coexist there,
|
||
so no single depth represents it.
|
||
- Disparity -> metres needs the VR180 rig's focal length and baseline, which are
|
||
not known a priori.
|
||
- Compute is NOT the blocker: the naive JS test cost ~250 ms/frame, but a WebGL2
|
||
SAD shader would be ~1-2 ms. Not the limiting factor.
|
||
|
||
Prior art (the depth question is studied; the answer is not "match scene depth"):
|
||
|
||
- Sidenmark, Kiefer & Gellersen, *Subtitles in Interactive VR: Using Gaze to
|
||
Address Depth Conflicts* (IEEE VR'19, Lancaster): depth conflicts between
|
||
environment and subtitles cause **occlusion and strain**; they propose
|
||
**gaze-based** techniques.
|
||
- Shimizu et al. (IEEE 2021): four dynamic subtitle placement methods, balancing
|
||
vertical/horizontal/**depth** of eye movement.
|
||
- *Gaze-Adaptive Subtitles for 360° Videos in VR* (IEEE 2025): adaptive placement
|
||
reduces visual fatigue versus fixed below-the-FOV placement.
|
||
|
||
**But**: the literature's best technique needs **eye tracking**, and Meta does
|
||
not expose `eye-tracking` to WebXR — our negotiated `enabledFeatures` were
|
||
`[local, viewer, bounded-floor, layers, hand-tracking, webxr, local-floor]`.
|
||
So gaze-adaptive placement is not available to us.
|
||
|
||
Recommendation if this is pursued: option **A** — use the *mid-band* (reliable)
|
||
disparity to bias the depth toward the scene, blended ~30/70 with the comfortable
|
||
default, clamped to 1.4-3.0 m, smoothed with a 2-3 s time constant, and scale the
|
||
quad proportionally so the **angular text size stays constant** (otherwise the
|
||
text visibly pulses). Default OFF. Chasing per-frame scene depth (option C) is
|
||
not recommended: disparity noise plus vergence conflict produces discomfort.
|
||
|
||
---
|
||
|
||
# ADDENDUM 4 — in-VR subtitle placement
|
||
|
||
Adjust the subtitles **without leaving VR**. Verified on device this round.
|
||
|
||
## Grab & drag the strip
|
||
|
||
Point at the subtitle text, pinch, and move:
|
||
|
||
- **up / down** → vertical position (drag on a fixed reference plane)
|
||
- **pull toward / push away** → depth (the hand's `origin.z` in viewer space;
|
||
forward is `-Z`, so pulling back brings the strip closer — like a real object)
|
||
- release → the new position is saved
|
||
|
||
Both the ray and the strip live in **viewer space**, so no world-space maths is
|
||
needed. The reference plane is fixed at grab time so the drag stays stable.
|
||
|
||
### Two deliberate restrictions (both needed)
|
||
|
||
The strip sits in the lower centre of the view by default and the control bar is
|
||
0.30 m tall at `y = -0.28`, so the two **overlap in y**. Without restrictions the
|
||
strip would swallow every aim in the lower half of the view.
|
||
|
||
1. Grabbing only happens when the control bar is **not already shown** — a pinch
|
||
while operating the bar never yanks the subtitles.
|
||
2. Only the **central 40 %** of the strip counts as grabbable (`nx,ny ∈ 0.30–0.70`),
|
||
which is where the text actually is (the crop is letterboxed into the texture).
|
||
|
||
## In-VR buttons on the control bar
|
||
|
||
The expanded bar grew from 1.1 × 0.22 m to **1.1 × 0.30 m** and gained a subtitle
|
||
row. Verified reachable layout (grid scan, no ambiguous overlaps):
|
||
|
||
```
|
||
y 0.25-0.45 seek track (full width)
|
||
y 0.50-0.70 [<<<<][PPPP][>>>>][ZZZZ] back10 / playpause / fwd10 / subs
|
||
y 0.85-1.00 [aa][AA][oo][OO][RR] size −/+ · opacity −/+ · reset
|
||
```
|
||
|
||
Actions: `sub_smaller`, `sub_bigger`, `sub_opaque_down`, `sub_opaque_up`, `sub_reset`.
|
||
|
||
## Persistence
|
||
|
||
`localStorage['jfvr.settings.v1']`, debounced 400 ms, loaded at boot with range
|
||
validation and silent fallback on corruption.
|
||
|
||
```json
|
||
{"subPlacement":{"y":-0.78,"z":-1.45,"width":2.8},"subOpacity":0.45,
|
||
"subMode":"hud","subAlpha":"premultiplied","projection":"180","stereo":"sbs",
|
||
"subtitles":true,"controls":true,"ctlAutoHideMs":4000}
|
||
```
|
||
|
||
## Verified this round
|
||
|
||
```
|
||
live setPlacement -> layer [-0.7, -2.1, 2.6] immediately
|
||
live setOpacity -> 0.35
|
||
resetPlacement() -> [-0.42, -1.45, 2]
|
||
slider -> localStorage written, and after a page reload the values come back
|
||
errors: []
|
||
```
|
||
|
||
Note: the four subtitle sliders in the 2D panel and the in-VR buttons now share
|
||
one code path (`setSubPlacementLive` / `setSubOpacityLive` / `setSubModeLive`),
|
||
so both save automatically.
|
||
|
||
---
|
||
|
||
# ADDENDUM 6 — subtitle depth vs size decoupling (angular-size lock)
|
||
|
||
## The problem, as the owner put it
|
||
|
||
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
|
||
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
|
||
|
||
Fixation is a **vergence** problem: an object at 0.4 m and subtitles at 1.45 m cannot
|
||
both be looked at comfortably at the same time. Eye tracking is not available to
|
||
WebXR on Quest (`enabledFeatures` has no `eye-tracking`), so the depth cannot be
|
||
driven by gaze — see ADDENDUM 3 above, which already concluded the gaze-adaptive
|
||
literature is out of reach for us.
|
||
|
||
What IS in reach is making **depth a pure depth knob**: keeping the strip's angular
|
||
size (and its angular position) constant while it moves in depth. That is exactly
|
||
the "scale the quad proportionally so the angular text size stays constant" note
|
||
from ADDENDUM 3, applied to the manual control.
|
||
|
||
Measured behaviour of the old control (free mode), which is why it did not work:
|
||
|
||
| depth | width | strip angular width |
|
||
| --- | --- | --- |
|
||
| 1.45 m | 2.00 m | 69° |
|
||
| 0.70 m | 2.00 m | 110° ← text blows up |
|
||
| 0.35 m | 2.00 m | 141° ← text leaves the view |
|
||
|
||
## What shipped
|
||
|
||
`state.subLockAngle = true`, `state.subAngleDeg = 69.2` (defaults). While locked:
|
||
|
||
```
|
||
width = 2 · |z| · tan(subAngleDeg / 2) constant angular width
|
||
y = (y0 / z0) · z constant angular height
|
||
```
|
||
|
||
so the strip stays at the same spot on the retina and only its **vergence
|
||
distance** changes. 69.2° is exactly the old `2.0 m @ 1.45 m`, so the default look
|
||
is unchanged. Depth range extended: near clamp **0.5 m → 0.35 m**.
|
||
|
||
In-VR subtitle row grew from five to nine buttons (the bar itself is unchanged —
|
||
`PANEL_M.expanded` = 0.72 × 0.21 m, canvas 1440 × 420 px). The row now carries
|
||
vertical position as well as depth, and the caption always leads with the depth:
|
||
|
||
```
|
||
y 0.80-1.00 [↑][↓][<<][>>][aa][AA][oo][OO][RR] 上 下 · 近 远 · 小 大 · 暗 亮 · 复位
|
||
caption 字幕 距1.45m · 69° 85%
|
||
```
|
||
|
||
`上`/`下` step the vertical position ±0.04 m (that control did not exist in the bar
|
||
before). `近`/`远` step depth ±0.15 m. `小`/`大` step the locked angle ∓5° (or
|
||
`width` ∓0.15 m when unlocked). Grab & drag carries `y` by the grab-start y/z
|
||
ratio while locked, so a strip pulled close does not slide below the view.
|
||
|
||
## Two bugs fixed in the same pass
|
||
|
||
1. **The CC (字幕) button never showed its state.** `Controls.readSubtitles()` only
|
||
read the controls module's own `S.subtitlesOn`, which `90-main.js` never sets —
|
||
it calls `setVideo(video)` with no `getExtra`. It now falls back to the shared
|
||
`state.subtitles`. The button is also unambiguous now: accent pill + `字幕开`
|
||
when on, slashed CC glyph + `字幕关` when off.
|
||
2. **There was no vertical-position control in VR.** Only grab & drag could move
|
||
the strip. The row gained `上`/`下`. (The 2D panel's `垂直位置` slider was
|
||
verified working — slider → `-0.70 m`, distance → `-0.80 m` re-derived
|
||
`y = -0.385` — so the gap was VR-only.)
|
||
|
||
## Verification (logic, offline)
|
||
|
||
The module was driven headlessly in node (state + maths only, no WebGL), holding
|
||
`subAngleDeg` at 69.2° and stepping depth:
|
||
|
||
```
|
||
z=-1.45 -> width 2.00, y -0.42 (69.2°)
|
||
z=-0.70 -> width 0.97, y -0.205 (69.2°)
|
||
z=-0.35 -> width 0.48, y -0.100 (69.2°)
|
||
z=-2.60 -> width 3.59, y -0.745 (69.2°)
|
||
```
|
||
|
||
and the 9-button row was grid-scanned through `Controls.hitTest` — 9 distinct
|
||
actions, no dead zone between pills. `Controls.draw` renders
|
||
`字幕开 · 字幕 距1.45m · 69°` while subtitles are on and locked, `字幕关` when
|
||
off, and `字幕 距1.45m · 2.0m` when unlocked. `build.py --check` passes.
|
||
|
||
Then verified in the **live Quest 3 browser** over CDP after deploying:
|
||
|
||
```
|
||
window.__JFVR.version -> 'hud-2'
|
||
state.stats.subLockAngle -> true
|
||
state.stats.subAngleDeg -> 69.2
|
||
state.errors -> []
|
||
|
||
Sub.setPlacement({z:-0.70}) -> width 0.97, y -0.20 (still 69.2°)
|
||
Sub.setPlacement({z:-0.35}) -> width 0.48, y -0.10 (still 69.2°)
|
||
Sub.setPlacement({z:-9}) -> width 3.59, y -0.745 (clamped to -2.6, still 69.2°)
|
||
Sub.setAngleDeg(50) -> width 2.42 (size control)
|
||
```
|
||
|
||
## Literature for the default value (not yet acted on)
|
||
|
||
All of these measure the **character height** angular size; our control is the
|
||
whole strip's angular width, so they need `stats.subTextDeg` (new) to compare.
|
||
|
||
- Kojić et al., *User Experience of Reading in Virtual Reality* (arXiv 2004.01545):
|
||
preferred angular size ≈ **16–18 dmm for medium/long text** (0.9–1.0° character
|
||
height), 27–32 dmm for 2-word text, vs Google's guidelines of 32 dmm (title) and
|
||
40 dmm (headline). Participants also rated **too-large** angular size among the
|
||
worst experiences.
|
||
- Dingler et al., *VR Reading UIs* (CHI'18 EA): **41 ± 14 dmm** (≈2.35° ± 0.8°).
|
||
- The default stays 69.2° deliberately: it is the on-device-proven look. Read
|
||
`stats.subTextDeg` in the headset, then decide.
|
||
|
||
## Persistence
|
||
|
||
`jfvr.settings.v1` gains `subLockAngle` and `subAngleDeg`; while locked the stored
|
||
`subPlacement.width` is re-derived from `subAngleDeg` + `z` on load.
|
||
|
||
```json
|
||
{"subPlacement":{"y":-0.42,"z":-0.35,"width":0.48},
|
||
"subLockAngle":true,"subAngleDeg":69.2,"subOpacity":0.85,
|
||
"subMode":"hud","subAlpha":"premultiplied","projection":"180","stereo":"sbs",
|
||
"controls":true,"ctlAutoHideMs":4000}
|
||
```
|
||
|
||
---
|
||
|
||
# ADDENDUM 7 — summon gating, lock badge, pre-VR panel trim
|
||
|
||
## The bar appeared on every pinch
|
||
|
||
Two independent causes, both removed:
|
||
|
||
1. `session.addEventListener('selectstart', onSelectStart)` called `Controls.show()`
|
||
unconditionally — any pinch anywhere in the view-summoned the bar.
|
||
2. `pointerDown()` called `Controls.show()` *before* it looked at the hit.
|
||
|
||
Now the bar is summoned only when the pointing ray lands on (or near) the bar:
|
||
|
||
```
|
||
bar hidden + aim on/near the bar -> show(), consume the press, arm for 500 ms
|
||
bar hidden + aim elsewhere -> nothing
|
||
bar shown + within 500 ms -> ignored
|
||
bar shown + aim on the bar -> handleSelect (and show() to extend auto-hide)
|
||
```
|
||
|
||
While LOCKED the same rule applies, except that what comes up is the padlock panel
|
||
alone — see ADDENDUM 7b below.
|
||
|
||
The summon window is wider than the hit window (`panelHit` padding 0.5/0.9 vs
|
||
0.25/0.6), because a ray that grazes the bar should still be able to bring it up —
|
||
the 500 ms arm is what prevents the accidental button press, not pixel precision.
|
||
|
||
**Superseded by ADDENDUM 7b:** the padlock was originally a separate quad layer to the
|
||
right of the bar. That did not work on device — the hand ray cannot reach that far
|
||
right. The padlock is now a normal panel button and "locked" is a third panel mode.
|
||
|
||
## Pre-VR settings panel trimmed
|
||
|
||
Sliders set on a flat screen do not predict what reads well in a headset, so the
|
||
subtitle section is now just 开 / 关. Removed: 模式, 透明合成, 不透明度, 垂直位置,
|
||
距离, 宽度, 距离与大小 (lock toggle), 视角宽度, 手部指向. `subAngleDeg`, placement
|
||
and opacity still persist — they are written by the in-VR controls.
|
||
|
||
The 手部指向 option (system ray vs index-finger ray) is gone together with its code
|
||
path; the system ray is the only mode (an index-finger ray injects fingertip jitter
|
||
into the cursor — see HANDOFF §6.6).
|
||
|
||
## "VR 播放" no longer covers the player's own controls
|
||
|
||
`.jfvr-btn` moved from `bottom: 24px` to `bottom: 120px`; at the viewport corner it
|
||
sat on Jellyfin's OSD fullscreen button. `.jfvr-panel` moved to `bottom: 180px`
|
||
(`max-height: calc(100vh - 200px)`) so the panel still stacks above the button.
|
||
|
||
## Verification
|
||
|
||
`build.py --check` passes. A local harness (a `file://` page with a fake 8192x4096
|
||
`<video>`, a fake libass canvas and stub `XRWebGLBinding`/`XRXRigidTransform`/
|
||
`createQuadLayer` classes) exercised the real bundle without a headset:
|
||
|
||
```
|
||
Controls.create(stubSession) -> ok, 4 quad layers (bar, beam x2, lock)
|
||
Controls.getLockLayer() -> the lock quad
|
||
Controls.setLock(true) -> stats.ctlLock = true, no throw
|
||
runFrameHooks(fakeFrame) -> presentLock/present ran, errors: []
|
||
panel DOM -> 画面: 投影/立体 · 字幕: 显示 · 控制: 控制条/自动收起
|
||
.jfvr-btn computed bottom -> 120px
|
||
drawLockBadge() -> 16 canvas ops, no throw (locked and unlocked)
|
||
```
|
||
|
||
Live Quest 3 over CDP after deploy: `__JFVR.version = 'hud-4'`,
|
||
`stats.subLockAngle = true`, `errors: []`.
|
||
|
||
---
|
||
|
||
# ADDENDUM 7b — padlock as a panel button + the "locked" panel mode
|
||
|
||
## The failure on device
|
||
|
||
> 我有新发现,在面板唤出状态时其无法点击,但面板消失后,其可以点击。
|
||
|
||
ADDENDUM 7's padlock was its own quad layer at `(0.45, -0.34, -1.45)` — 0.45 m right of
|
||
the panel centre, 0.09 m right of the panel's own right edge. Two things were wrong:
|
||
|
||
1. **It was outside every reachable ray.** Aiming is the binding constraint with hands:
|
||
a hand ray plus the cursor's acceptance window (`panelHit` 0.25/0.6 on a 0.72 m panel)
|
||
reaches x ≈ 0.45 — the panel's right edge. Nothing could aim 0.09 m past it.
|
||
2. **Its hit test used the panel's layer geometry.** Whether it fired therefore depended
|
||
on the panel's own size/position, which is why it behaved differently shown vs hidden.
|
||
|
||
## The fix: make it an ordinary button
|
||
|
||
The padlock is now the fifth button in the media row and uses the same code path as
|
||
everybutton that already worked (cursor → `hitTest` → `handleSelect` → `onAction`):
|
||
|
||
```
|
||
expanded: [<<<] [>||] [>>>] [CC] [padlock]
|
||
0.18 0.36 0.54 0.72 0.90 (hit pill half width 0.092)
|
||
```
|
||
|
||
## The behavior the owner asked for
|
||
|
||
> 想要的是:唤出面板,然后可以上锁,上锁后锁头图标和面板一样自动隐藏,下次触发面板
|
||
> 唤出时,只展示锁本身,不展示进度条等能调整的项目。(解锁后的0.5s的动做也忽略)
|
||
|
||
A third panel mode does exactly that:
|
||
|
||
| mode | size (m) | canvas | drawn / hit-testable |
|
||
| --- | --- | --- | --- |
|
||
| `expanded` | 0.72 × 0.21 | 1440 × 420 | everything |
|
||
| `locked` | 0.26 × 0.10 | 520 × 200 | the padlock + 解锁 label ONLY |
|
||
|
||
```
|
||
summon (unlocked) -> expanded panel, auto-hides after ctlAutoHideMs
|
||
press 上锁 -> panel becomes padlock-only IN PLACE, auto-hide timer restarts
|
||
(so the padlock disappears on its own, like the panel always did)
|
||
summon (locked) -> padlock alone; no progress bar, no rows, nothing adjustable
|
||
press 解锁 -> full panel returns, and 500 ms of presses are ignored
|
||
```
|
||
|
||
Because `hide()` no longer resets the mode, the layer keeps the size it will be shown
|
||
at: the summon window stays the full panel rect (0.72 × 0.21 unlocked, 0.26 × 0.10
|
||
locked) and re-showing never pops a resize. `hitTest()` takes the mode explicitly now
|
||
(`collapsed` and `locked` share no safe aspect-ratio invariant), and `handleSelect()`
|
||
no longer falls back to pressing the panel centre when there is no ray hit.
|
||
|
||
## Verification (offline, real bundle, no headset)
|
||
|
||
A temporary `file://` harness fed the real `dist/jfvr.js` a fake 8K `<video>`, a fake
|
||
libass canvas and stub `XRWebGLBinding` / `XRXRigidTransform` / `createQuadLayer`
|
||
classes. The harness file was deleted afterwards.
|
||
|
||
```
|
||
Controls.create(stubSession) -> ok, layers: bar + beam x2
|
||
expanded media row (5 centres) -> back10|playpause|fwd10|subs|lock, no dead zone
|
||
expanded subtitle row (9 centres) -> 9 distinct actions
|
||
locked panel: centre -> {type:'lock'} -> bottom edge / left edge -> null
|
||
draw() locked -> draws ONLY '解锁'
|
||
draw() expanded -> 上锁 + 字幕 距1.45m · 69° + 00:10 (all intact)
|
||
setLock(true) -> stats.ctlMode=locked, layer 0.72x0.21 -> 0.26x0.10, canvas resized
|
||
setLock(false) -> stats.ctlMode=expanded, layer back to 0.72x0.21
|
||
hide() -> mode KEPT (summon window stays the panel rect)
|
||
runFrameHooks() after each switch -> no errors, stats.hookErr / ctlErr null
|
||
handleSelect() with no ray hit -> null (was: silently pressed the centre)
|
||
```
|
||
|
||
Also standalone (`node`): the padlock hit region and the mode table above were asserted
|
||
directly against `Controls.hitTest` / `Controls.draw` with mock 2D contexts.
|
||
|
||
Live Quest 3 over CDP after deploy: `__JFVR.version = 'hud-5'`, `stats.ctlMode =
|
||
'expanded'`, `stats.ctlLock = false`, `errors: []`.
|
||
|
||
## Still unverified
|
||
|
||
The *feel* — how easily the padlock is aimed with the hand ray, and whether 0.5 s is the
|
||
right arm delay — can only be judged in the headset.
|