README gets a section on the watchdog; HANDOFF gets ADDENDUM 12 with the full
investigation: the failing file is HEVC Main 10 while every other 8K file is
Main, Jellyfin Web builds its device profile from canPlayType('hvc1.2.4.L183')
which over-claims, the server has no force-transcode switch, and the player UI
(including the quality menu) is only created after 'playbackstart' -- which is
why a stuck video has no quality menu to click.
Also records that src/00-core.js only ever probed Main-profile codec strings, so
the documented "8K HEVC supported" result was never true for 10-bit.
1122 lines
49 KiB
Markdown
1122 lines
49 KiB
Markdown
# Jellyfin Web VR — Quest 3 / WebXR MVP
|
||
|
||
给 Jellyfin Web 播放页注入一个「VR 播放」按钮,在 Meta Quest 3 浏览器里直接进入
|
||
VR180 / VR360 沉浸播放,不用再手动切全屏 + 选投影。
|
||
|
||
---
|
||
|
||
## 1. 当前部署状态
|
||
|
||
已注入到 `192.168.123.199` 上的 Jellyfin 容器 `Jellyfin`(当前版本 `mvp-2e32e6`,`NS.version = hud-6`,含 ADDENDUM 6 + 7 + 7b + 8):
|
||
|
||
| 项 | 值 |
|
||
|---|---|
|
||
| 脚本 | `/jellyfin/jellyfin-web/jfvr-mvp.js`(容器内) |
|
||
| 注入点 | `/jellyfin/jellyfin-web/index.html` 的 `</body>` 之前 |
|
||
| 备份 | `/vol2/jellyfin/config/_vr_backup/index.html.orig` |
|
||
| 两个入口都生效 | `http://192.168.123.199:8096/web/` 与 `https://jellyfin.cat-shark.xyz/web/` |
|
||
|
||
> ⚠️ 老路子(`deploy.py`):注入写在**容器内文件**里,`jellyfin/jellyfin:latest` 一升级就没了,
|
||
> 得重跑 `deploy.py`。**现在已经有插件版了,见下面 §1.1,优先用插件。**
|
||
|
||
### 1.1 插件化(推荐,已实现)
|
||
|
||
JFVR 同时也是一个标准的 **Jellyfin 插件**,通过自建 Gitea 仓库分发,走 Dashboard 的
|
||
「添加存储库 → Catalog 安装」,**不再改容器内任何文件**。
|
||
|
||
| 项 | 值 |
|
||
|---|---|
|
||
| 插件仓库地址 | `https://git.cat-shark.xyz/cat-shark/jellyfin_vr_web_plugin/raw/branch/main/manifest.json` |
|
||
| 插件名 / GUID | `Jellyfin Web VR` / `7d33ef59-f9b0-412b-93bd-5cd5353018de` |
|
||
| 目标 ABI | `12.0.0.0`(Jellyfin 只保留 `targetAbi <= 服务器版本` 的条目,所以一份包覆盖整个 12.x) |
|
||
| 插件代码 | `plugin/Jellyfin.Plugin.JFVR/`(C#,net10.0,对 `Jellyfin.Controller 12.1.0`) |
|
||
| 打包 / 发布 | `python3 pack.py` / `python3 pack.py --bump --publish` |
|
||
| 脚本端点 | `/JFVR/jfvr.js`(bundle 内嵌在 DLL 里,URL 带内容哈希做缓存键) |
|
||
| 状态查询 | `/JFVR/status` |
|
||
| 设置页 | Dashboard → 插件 → Jellyfin Web VR(开关注入 / 覆盖脚本 URL) |
|
||
|
||
**安装**:Dashboard → 插件 → 目录 → ⚙️ → ➕ → 粘贴上面的仓库地址 → 保存 →
|
||
在 Catalog 里找到 **Jellyfin Web VR** → 安装 → **重启 Jellyfin**。
|
||
|
||
**工作方式**:`ScriptInjectionStartupFilter` 在**请求时**把
|
||
`<script plugin="JFVR" src="../JFVR/jfvr.js?v=<hash>" defer></script>`
|
||
插到 `index.html` 的 `</body>` 之前。磁盘上的 `/jellyfin/jellyfin-web/` 一个字节都不动,
|
||
所以升级 Jellyfin 不会丢。如果检测到老 `deploy.py` 留下的 `<script plugin="JellyfinVR" ...>`,
|
||
会**自动清掉**,不会重复加载 bundle。
|
||
|
||
**改完 web 代码后的流程**:
|
||
|
||
```bash
|
||
python3 pack.py --bump --publish # 构建 → 打包 → 建 Release → 推 manifest
|
||
# 然后在 Jellyfin 里:插件页点更新(或重装),重启,刷新浏览器
|
||
```
|
||
|
||
> 老路子仍然可用,作为插件的兜底:`JF_PW=... python3 deploy.py`。
|
||
> 但**两者不要同时开** —— 装了插件就别再跑 `deploy.py`(跑完插件也会把那个标签清掉)。
|
||
|
||
### 1.2 播放看门狗(卡住检测)
|
||
|
||
`src/40-watchdog.js`。针对一种很坑的失败:**某些设备/浏览器放不出来时,Jellyfin 的播放器界面根本不会被创建**。
|
||
|
||
原因(已从 Jellyfin Web 源码确认):播放器 UI 只在 `playbackstart` 事件之后构建,而 `playbackstart` 要等媒体真的起来了才会触发。加载顺序是:
|
||
|
||
```
|
||
HtmlVideoPlayer.play() htmlVideoPlayer/plugin.js:502
|
||
createMediaElement() :1802 转圈 + <div.videoPlayerContainer> + <video>
|
||
(此时还没 src)
|
||
setCurrentSrc() :593 elem.src = url
|
||
→ loadedmetadata → canplay → play → 'playbackstart'
|
||
→ OSD 控制器(apps/legacy/controllers/playback/video/index.js:591)
|
||
```
|
||
|
||
媒体起不来 → `playbackstart` 不触发 → **码率菜单根本不存在**,页面上只剩一个转圈和一个 `videoWidth === 0` 的 `<video>`。所以「卡住之后调不了码率」不是菜单藏起来了,是菜单还没被创建。
|
||
|
||
看门狗就盯这个状态(`.videoPlayerContainer` 里的 `<video>` 仍然是 `videoWidth === 0`):
|
||
|
||
| | 阈值 |
|
||
|---|---|
|
||
| 媒体报错(`video.error`) | 立即弹 |
|
||
| 直连没画面 | 15 秒 |
|
||
| 转码(src 已经是 hls) | 45 秒(服务器要先出片) |
|
||
| 已经有画面 | 立即退出,不打扰 |
|
||
| VR 会话中 | 完全不跑 |
|
||
|
||
弹窗会写明**具体格式**(走 Jellyfin API 查 MediaStreams),并给 20/8/3 Mbps 三档 + 「继续等待」。选一档后写入 Jellyfin 自己那套 localStorage 键然后重载:
|
||
|
||
```
|
||
maxbitrate-Video-<true|false> = 20000000
|
||
enableautobitratebitrate-Video-<true|false> = false
|
||
```
|
||
|
||
> 注意 `enableautobitratebitrate` 里 `bitrate` 确实出现了两次 —— 这是 Jellyfin 上游的拼写,照抄的。
|
||
> 之所以不用 `playbackManager.setMaxStreamingBitrate()`:那个模块没暴露到 window(已实测 `window.playbackManager === undefined`,`window.Emby` 只剩 `Page`),所以只能写设置 + 重载。
|
||
|
||
选过之后 120 秒内不再打扰(记在 sessionStorage)。
|
||
|
||
---
|
||
|
||
## 2. Quest 3 实测数据(已用 ADB + CDP 在真机验证)
|
||
|
||
### 2.0 设备与能力
|
||
|
||
```
|
||
Quest 3 Android 14 (SDK 34) build UP1A.231005.007.A1
|
||
Quest Browser 152.1.0.34.52.1079628208 (Chromium 152.0.7977.83)
|
||
UA Mozilla/5.0 (X11; Linux x86_64; Quest 3) ... Chrome/152 ... VR Safari/537.36
|
||
```
|
||
|
||
| 检测项 | `http://192.168.123.199:8096` | `https://jellyfin.cat-shark.xyz` |
|
||
|---|---|---|
|
||
| `isSecureContext` | false | **true** |
|
||
| `navigator.xr` | false | **true** |
|
||
| `immersive-vr` | — | **true** |
|
||
| **`XRMediaBinding.createEquirectLayer`** | — | **true** ← 首选路径可用 |
|
||
| `XRWebGLBinding` | — | true ← 回退路径可用 |
|
||
| `MAX_TEXTURE_SIZE` | 8192 | 8192 |
|
||
| `MAX_RENDERBUFFER_SIZE` | 16384 | 16384 |
|
||
| 8K HEVC `decodingInfo` | supported / powerEfficient | **supported:true, powerEfficient:true** |
|
||
|
||
结论:**HTTPS 下 Quest 浏览器提供全部所需 WebXR 能力,8K HEVC 走硬件解码。**
|
||
|
||
### 2.1 网络实测:HTTPS 域名在 Quest 上就是内网直连
|
||
|
||
```
|
||
Quest 3 IPv6 : 2409:8a14:44a:e360:70b9:446e:d5f7:10a7/64
|
||
NAS IPv6 : 2409:8a14:44a:e360:520:8aa2:7736:4f12/64 ← 同一 /64
|
||
|
||
domain AAAA → NAS 自己的局域网 IPv6
|
||
服务端连接表 : [NAS]:443 ← [Quest]:35860 (ESTAB, IPv6)
|
||
```
|
||
|
||
60 MB 文件在 Quest 浏览器上实测:
|
||
|
||
| Run | 吞吐 | TTFB |
|
||
|---|---|---|
|
||
| 1 | 450 Mbps | 11 ms |
|
||
| 2 | 620 Mbps | 7 ms |
|
||
| 3 | **702 Mbps** | 12 ms |
|
||
|
||
对比:从局域网 PC 用 curl 走 IPv6 只有 4.29 MB/s(34 Mbps),走 IPv4 阿里云中转只有 94 KB/s。
|
||
|
||
> 慢的根因:NAS 上跑着 **`npc` 内网穿透客户端**,域名的 **A 记录 `39.105.149.197` 是阿里云上的 nps 服务端**,
|
||
> 走 IPv4 就是 NAS → 阿里云 → 你。**走 IPv6 则完全不出内网。**
|
||
>
|
||
> 所以 **不需要为了速度退回 HTTP**。8K 片源码率仅 21 Mbps,702 Mbps 有 30 倍余量。
|
||
|
||
### 2.2 如果仍想用 `192.168.123.199:8096`
|
||
|
||
WebXR 要求「安全上下文」。实测结果:
|
||
|
||
| 入口 | `isSecureContext` | `navigator.xr` | WebXR |
|
||
|---|---|---|---|
|
||
| `http://192.168.123.199:8096` | **false** | **false** | ❌ 需先做 §2.1 |
|
||
| `https://jellyfin.cat-shark.xyz` | true | true | ✅ 直接可用 |
|
||
|
||
用内网 HTTP 地址 —— Quest 浏览器开 flags
|
||
|
||
Meta Quest 2/3/3S(**OS 79+**)浏览器支持在 HTTP 模式下使用 WebXR,需显式授权:
|
||
|
||
1. Quest 浏览器地址栏输入 `chrome://flags`
|
||
2. 搜索 `insecure`
|
||
3. 启用 **`unsafely-treat-insecure-origin-as-secure`**
|
||
4. 在下方输入框填:`http://192.168.123.199:8096`
|
||
(**必须带 `http://` 和端口号**;多个用逗号分隔)
|
||
5. 点空白处失焦 → 屏幕底部出现 **Relaunch** → 点它
|
||
6. 回到 `chrome://flags` 确认 flag 仍启用且 URL 已保存
|
||
|
||
代价:这台设备专属、浏览器更新后可能需要重设。
|
||
|
||
### 2.3 兜底:让局域网内解析到内网 IPv4
|
||
|
||
如果某天 IPv6 不可用,且 Quest 落到了 IPv4(阿里云中转)导致慢,最一劳永逸的修法是在
|
||
**OpenWrt 路由器(192.168.123.1)** 上加一条静态解析,让局域网内所有设备都解析到内网 IP:
|
||
|
||
- LuCI:`网络 → DHCP/DNS → 一般设置 → 地址`,添加 `/jellyfin.cat-shark.xyz/192.168.123.199`
|
||
- 或 SSH 到路由器:
|
||
```sh
|
||
uci add_list dhcp.@dnsmasq[0].address='/jellyfin.cat-shark.xyz/192.168.123.199'
|
||
uci commit dhcp && /etc/init.d/dnsmasq restart
|
||
```
|
||
|
||
之后 `https://jellyfin.cat-shark.xyz` 在局域网内全程走内网,证书依然有效,所有设备(不只 Quest)都受益。
|
||
|
||
---
|
||
|
||
## 3. 实操:如何在真机上验证(ADB + CDP)
|
||
|
||
```bash
|
||
ADB=/opt/SideQuest/resources/platform-tools/adb # SideQuest 自带的 adb
|
||
|
||
# 1) 看设备(SideQuest 的无线 ADB)
|
||
$ADB devices -l
|
||
|
||
# 2) 把 Quest 浏览器的 DevTools 端口转发出来
|
||
$ADB forward tcp:9222 localabstract:chrome_devtools_remote
|
||
curl -s http://127.0.0.1:9222/json/version
|
||
|
||
# 3) 让 Quest 打开某个页面
|
||
$ADB shell am start -a android.intent.action.VIEW \
|
||
-d "http://192.168.123.199:8096/web/index.html" \
|
||
-n com.oculus.browser/.OculusLauncherActivity
|
||
|
||
# 4) 截图 / OCR
|
||
$ADB exec-out screencap -p > /tmp/quest/shot.png # 4128x2208
|
||
tesseract /tmp/quest/shot.png - --psm 6
|
||
```
|
||
|
||
拿到 `9222` 后可以直接用 `agent_browser --cdp http://127.0.0.1:9222` 接管 Quest 浏览器,
|
||
在页面上跑 `Runtime.evaluate`、读网络、发输入事件。
|
||
|
||
> 注:Quest 浏览器**不把 `console.log` 转发到 logcat**,所以诊断要走 CDP `Runtime.evaluate` 或页面 UI。
|
||
|
||
---
|
||
|
||
## 4. 使用步骤(Quest 3 里)
|
||
|
||
1. 打开 Jellyfin(§2.1 或 §2.2 任一入口)
|
||
2. 进媒体库,播放一个 VR 视频
|
||
3. 右下角出现蓝色 **「VR 播放」** 按钮 → 点它
|
||
4. 面板里确认 **投影**(180°/360°)和 **立体**(单目/左右/上下)
|
||
- 自动按文件名 + 分辨率判断;**8192×4096(2:1)会默认选 180° + 左右**
|
||
5. 点 **「进入 VR」** → 进入沉浸模式;摘头显即退出
|
||
|
||
进不去时展开面板里的 **「诊断信息」**,把内容贴回来。
|
||
|
||
---
|
||
|
||
## 5. 在 VR 里修掉的两个关键缺陷
|
||
|
||
真机(Quest 3 + CDP)实测发现并修复:
|
||
|
||
### 5.1 卡顿 —— 帧率不匹配(3:2 pulldown)
|
||
|
||
| 项 | 实测值 |
|
||
|---|---|
|
||
| 视频帧率 | **59.94 fps** |
|
||
| XR 会话默认 | **90 Hz**(实测 87.9 fps) |
|
||
| `session.supportedFrameRates` | `[72, 80, 90, 120]` |
|
||
| `session.frameRate`(初始) | **0**(未指定) |
|
||
|
||
90 ÷ 59.94 = 1.5 → 视频帧在显示上被交替重复 1 次、2 次 —— 经典 3:2 pulldown 抖动。
|
||
2D 播放时系统会自动匹配刷新率,所以「web 端流畅、进 VR 就卡」。
|
||
|
||
**修复**:进 VR 后调用 `session.updateTargetFrameRate(120)`。120 ÷ 60 = 2,每帧均匀显示两次。
|
||
|
||
```
|
||
修复前 fps : 87.9
|
||
修复后 fps : 114.9 (20 秒稳定 111–115,丢帧 0.000%)
|
||
```
|
||
|
||
### 5.2 字幕丢失 —— libass 被冻结
|
||
|
||
两个独立原因叠加:
|
||
|
||
**(a) 字幕画布不在 XR 场景里。** Jellyfin Web 12 用 libass 渲染 ASS 到独立的
|
||
`canvas.libassjs-canvas`,和 `<video>` 是两个元素。只把 `<video>` 放进 XR layer,字幕自然没有。
|
||
|
||
**(b) 进了 VR,libass 就停止渲染。** 实测:
|
||
|
||
```
|
||
window.requestAnimationFrame : 6 秒内 0 次调用 ← 完全停摆(XR 接管了帧循环)
|
||
timeupdate : 22 次 ✅
|
||
requestVideoFrameCallback : 288 次 ✅
|
||
```
|
||
|
||
Jellyfin 的字幕渲染由 rAF 驱动 → 画布冻结在进 VR 前的最后一帧。
|
||
|
||
**修复**:安装一个 rAF 代理 —— 进 VR 后把 `requestAnimationFrame` 的回调收进自己的队列,
|
||
用 `video.requestVideoFrameCallback` 驱动(它跟着视频帧走,正好是字幕需要的节奏);
|
||
不在 VR 时直接透传原生实现,2D 播放完全不受影响。
|
||
|
||
**字幕层**:用 `XRWebGLBinding.createEquirectLayer` 建第二个动态等距柱状层
|
||
(同一立体几何),把画布拷进 `getSubImage(layer, frame).colorTexture` 后经 GL 合成。
|
||
|
||
### 5.3 字幕分辨率
|
||
|
||
libass 按屏幕上的视频框尺寸渲染(**1416×708**,在 180° 球面上只有约 **4 px/度**,
|
||
而 Quest 3 约 **19 px/度** → 非常糊)。
|
||
|
||
必须**在进入 VR 之前**放大:XR 会话运行期间 libass 不会重算画布尺寸。
|
||
所以放大动作挂在「打开 VR 面板」这个点击上(进 VR 本来就要先开面板)。
|
||
|
||
```
|
||
放大前 : 1416x708
|
||
放大后 : 4320x2160 (仍是精确 2:1,约 24 px/度)
|
||
```
|
||
|
||
### 5.4 你的字幕文件本身就是为 SBS 设计的
|
||
|
||
```
|
||
Title: VR Dual-Eye Subtitle
|
||
PlayResX: 3840 PlayResY: 1920 ← 2:1,与视频同比例
|
||
Style: LeftEye , Alignment=8, MarginL=50 , MarginR=1920, MarginV=700
|
||
Style: RightEye, Alignment=8, MarginL=1920 , MarginR=50 , MarginV=700
|
||
956 条对白,每条成对出现(LeftEye + RightEye)
|
||
```
|
||
|
||
实测墨迹分布 `left=30, right=30`(完全对称)—— 左半屏给左眼、右半屏给右眼,
|
||
正好匹配 `stereo-left-right` 层。**不需要任何转换。**
|
||
|
||
---
|
||
|
||
## 6. 硬性约束:必须由用户手势触发
|
||
|
||
WebXR 规范原文:
|
||
|
||
> Wait for the window to have *transient activation* …
|
||
> If the request was not made while the global object has transient activation, return `false`.
|
||
|
||
所以 `navigator.xr.requestSession()` **必须同步写在点击处理函数里**,前面不能有任何 `await`
|
||
(否则 user activation 丢失,被 UA 拒绝)。代码里这一点是硬性保证的,改动时不要破坏。
|
||
|
||
---
|
||
|
||
## 7. 渲染路径
|
||
|
||
| 路径 | 条件 | 说明 |
|
||
|---|---|---|
|
||
| **A. `XRMediaBinding.createEquirectLayer`**(首选) | 浏览器支持 WebXR Media Layers | Meta 官方推荐:视频直接交给系统合成器,不经 WebGL 采样。**8K 下唯一靠谱的路** |
|
||
| **B. WebGL2 等距柱状光线重建**(回退) | 上面不可用 | 每帧 `texImage2D(video)` 上传整帧,8K 会吃力,仅兜底 |
|
||
|
||
投影参数:
|
||
|
||
- VR180 SBS → `layout:'stereo-left-right'`, `centralHorizontalAngle:π`
|
||
- VR360 SBS → `layout:'stereo-left-right'`, `centralHorizontalAngle:2π`
|
||
- 上下 3D → `layout:'stereo-top-bottom'`
|
||
|
||
---
|
||
|
||
## 8. 修改 / 部署 / 回滚
|
||
|
||
### 8.1 插件(推荐)
|
||
|
||
```bash
|
||
python3 pack.py --bump --publish # 构建 + 打包 + 建 Release + 推 manifest.json
|
||
```
|
||
|
||
然后在 Jellyfin 里更新插件并重启,刷新浏览器。
|
||
|
||
### 8.2 老路子:直接注入容器内文件(兜底)
|
||
|
||
```bash
|
||
# 改 jfvr-mvp.js 后重新部署
|
||
JF_PW='<ssh密码>' python3 deploy.py
|
||
|
||
# 完全回滚(恢复原始 index.html 并删除脚本)
|
||
JF_PW='<ssh密码>' python3 deploy.py rollback
|
||
```
|
||
|
||
`deploy.py` 用文件 md5 前 6 位作为版本号写进 script 标签的 query,自动破缓存。
|
||
插件走的是同一套破缓存思路,只是 hash 由 DLL 内嵌 bundle 的 md5 算出来。
|
||
|
||
---
|
||
|
||
## 9. Meta 官方规格(决定性依据)
|
||
|
||
出处:`developers.meta.com/horizon/documentation/web/browser-video/`
|
||
|
||
- 支持的容器:**MP4 / Ogg / WebM**(**MKV 不在列表**)
|
||
- 支持的编码:h.264 / **h.265** / vp8 / vp9 / AV1
|
||
- > **For all formats, Browser supports 4K and supports 8K on h.265 and AV1.**
|
||
- > **WebXR Media Layers 是显示沉浸式视频最高效的方式。如果可能,围绕这个 API 设计你的体验。**
|
||
- Quest 浏览器还内置 2D 页面全屏重投影(可手动选 360/180 + 单目/左右/上下),
|
||
但 2D 路径最大显示尺寸 ≤ 7680×3840,且每次要手动选 —— 所以走 WebXR 才有意义。
|
||
|
||
WebXR 规范(`immersive-web/webxr` index.bs):
|
||
|
||
> Wait for the window to have **transient activation** …
|
||
> If the request was not made while the global object has transient activation, return `false`.
|
||
|
||
所以 `navigator.xr.requestSession()` 必须**同步**写在点击处理里,前面不能有 `await` ——
|
||
代码里是硬保证,改动时不要破坏。
|
||
|
||
---
|
||
|
||
## 10. 片源情况(实测)
|
||
|
||
```
|
||
/mnt1/123 → 61 个 VR 文件,全部:
|
||
8192 × 4096 | HEVC Main / level 6.1 | ~60 fps | 8–25 Mbps | .mp4 (faststart 已确认)
|
||
```
|
||
|
||
即 **VR180 SBS**,每眼 4096×4096 的 180° 等距柱状。
|
||
|
||
> ⚠️ **必须保持 Direct Play**。服务器的 Pentium Gold 8505 + UHD 核显编不了 8K
|
||
> (Alder Lake-N 编码上限约 4K),一旦 Jellyfin 触发转码,分辨率腰斩且 CPU 打满。
|
||
> 诊断面板里的 `playbackMode` 会显示当前是 DirectPlay 还是 HLS。
|
||
|
||
---
|
||
|
||
## 11. 相关现成实现(备查)
|
||
|
||
| 项目 | 技术 | 备注 |
|
||
|---|---|---|
|
||
| `allexxex/jellyfinvr` | Three.js + WebXR | VR180/360/SBS/OU;每帧上传纹理,8K 下是瓶颈 |
|
||
| `Mix1C/Jellyfin-360VR-Player` | A-Frame | 仅 360°,无 180 |
|
||
| `Lqlsoftware/Jellyfin-vr-plugin` | C# 插件 | 独立播放页 `/VR/Video/{id}/Play` |
|
||
| `Toastyice/DeoVRDeeplink` | C# 插件 + `deovr://` | 可完全自动唤起 Quest 上的 DeoVR(deeplink 走系统,不受 user gesture 限制) |
|
||
| `IAmParadox27/jellyfin-plugin-file-transformation` | C# 插件 | 正式版注入通道 |
|
||
| `n00bcodr/jellyfin-plugins` → JavaScript Injector | 插件 | `12/manifest.json`,targetAbi 12.0.0.0 |
|
||
|
||
---
|
||
|
||
# ADDENDUM 2 — in-VR UX pass
|
||
|
||
Feedback from the first headset test, and what changed.
|
||
|
||
## Fixed
|
||
|
||
| Feedback | Fix |
|
||
|---|---|
|
||
| The progress bar was permanently visible | **Hidden by default.** `show()` appears on any pinch/trigger and auto-hides after `ctlAutoHideMs` (default 4 s, slider 1–10 s). While hidden: no draw, no texture upload, and the layer is dropped from the render state. |
|
||
| No guide line for hand/controller input | **Laser beam + cursor.** A thin quad beam (height 6 mm) runs from the input source to the panel, bright at the origin fading to transparent, plus a ring/dot cursor drawn at the exact hit point. |
|
||
| Subtitles should sit below the visual centre | Default `subPlacement.y` moved `-0.34 → -0.42` (quad centre, negative = below eye level). |
|
||
| Subtitle opacity and position should be adjustable | 4 live sliders in the panel: opacity, vertical position, distance, width. |
|
||
|
||
Input is now **one code path for hands and controllers**: both expose
|
||
`XRInputSource.targetRaySpace`, so each is ray-cast against the panel plane.
|
||
Gaze (view centre) is only the fallback when no ray reaches the panel.
|
||
|
||
## Beam orientation — corrected convention
|
||
|
||
The first spec said "orient the quad so its local `-Z` follows the ray". That is
|
||
geometrically wrong: a quad's local `-Z` is its plane **normal**, so the quad's
|
||
plane would end up perpendicular to the ray and its `width` would lie *across*
|
||
the beam — a thin face-on bar, not a laser.
|
||
|
||
Correct:
|
||
|
||
```
|
||
position = origin + dir * (L/2) // L = hit distance, else 1.2 m
|
||
width = L, height = 0.006
|
||
local +X -> dir // beam axis lies IN the quad plane
|
||
local -Z -> perpendicular to dir, biased up so it is never edge-on
|
||
```
|
||
|
||
Verified on device against two different rays — the measured beam midpoint
|
||
matched `origin + dir·L/2` to 1 cm, and the beam layer is added/removed with the
|
||
pointer.
|
||
|
||
## UI bug found and fixed this round
|
||
|
||
Adding 5 sliders grew the settings panel to 784 px while the viewport was 708 px,
|
||
so `进入 VR` ended up **off-screen and unclickable** (`elementFromPoint` returned
|
||
`null`). The panel is now a flex column with a scrolling body and a **sticky
|
||
footer** holding the status line and the CTA, and `max-height: calc(100vh - 104px)`.
|
||
|
||
## Verified on Quest 3 after this pass
|
||
|
||
```
|
||
layers : [equirect, quad(op=0.85)] ← control bar correctly absent
|
||
sub.placement : [-0.42, -1.45, 2.0]
|
||
sub.opacity : 0.85 (applied to layer.opacity)
|
||
ctl.hidden : true ← default hidden
|
||
fps : 110.9 targetFrameRate: 120
|
||
errors : []
|
||
```
|
||
|
||
Beam check (pointer frozen so main's per-frame update could not overwrite it):
|
||
|
||
```
|
||
controller ray -> beam w=1.22 pos=(0.13,-0.46,-0.86) matches origin+dir·L/2
|
||
hand ray -> beam w=1.45 pos=(-0.12,-0.47,-0.89) (different dir, follows)
|
||
pointer null -> beam removed (layers 4 -> 3, beamOn false)
|
||
```
|
||
|
||
---
|
||
|
||
# ADDENDUM 3 — aiming guide + system gestures
|
||
|
||
## Root cause of "no guide line" AND "cursor click does nothing"
|
||
|
||
Both symptoms were one chain of three bugs:
|
||
|
||
1. `applyBeamVisible()` required `S.shown` — the beam only existed while the
|
||
control bar was visible. But the bar only appears on activation, so the user
|
||
could never aim at it before pinching. **Chicken-and-egg.**
|
||
2. `hook()` early-returned on `!S.shown`, so `presentBeam()` never ran and the
|
||
beam texture was never uploaded — the beam could not render even when flagged
|
||
visible. **Second, independent blocker.**
|
||
3. Only the `select` event was handled. Meta documents hand pinch as
|
||
"implemented in the emulated Gamepad attribute for the hand as a **button
|
||
press**", which does not reliably raise `select`. So on hands **nothing fired
|
||
at all** — no bar, no beam, no click.
|
||
|
||
Also: `pointer.visible` used to require an actual panel hit, so the aiming guide
|
||
would only appear once you were already on target. Now `visible` means "an input
|
||
source is pointing" and `hit` is separate (may be null).
|
||
|
||
## Use the platform's gesture recognition — do not reimplement it
|
||
|
||
| Platform facility | What it gives us |
|
||
| --- | --- |
|
||
| `XRInputSource.gamepad` emulated buttons | **hand pinch** (per Meta's WebXR Hands doc) |
|
||
| `selectstart` / `select` / `selectend` | controller trigger, and pinch on some builds |
|
||
| `targetRaySpace` | one ray API for hands **and** controllers |
|
||
|
||
So `90-main.js` now polls gamepad buttons for a rising edge **and** listens for
|
||
`select`, with a 250 ms dedupe so both paths cannot double-fire. No thumb↔index
|
||
joint-distance math is done anywhere.
|
||
|
||
System-reserved gesture to avoid: **palm pinch** (left hand = menu button, leaves
|
||
the session). Ordinary point-and-pinch is ours.
|
||
|
||
Activation now requires a real ray hit (`pointer.visible && pointer.hit`), so a
|
||
stray pinch can no longer fall back to gaze-centre and seek to 50%.
|
||
|
||
## Verified on device after the fix
|
||
|
||
```
|
||
pointing, NOT hitting the panel -> beamOn: true, ctlHidden: true
|
||
beam quad present (h=0.006, w=1.2)
|
||
```
|
||
|
||
## Feasibility: auto-adjusting subtitle depth from scene content
|
||
|
||
Measured on a real 8K VR180 frame (SBS pair, 320x160 downsample, SAD block
|
||
matching +-24 px):
|
||
|
||
| Band | median disparity | match residual | disparity IQR |
|
||
| --- | --- | --- | --- |
|
||
| mid frame (y 40-88) | 1 px | 6.1 | — |
|
||
| **subtitle band (y 96-144)** | -5 .. -8 px | **13.5-15.3** | **[-16, +1]** |
|
||
|
||
- Disparity IS extractable and temporally stable (mid band held 1 px across 6 frames).
|
||
- But the **subtitle band is the least reliable region**: 2.3x the matching
|
||
residual and a 17 px inter-quartile spread — i.e. several depths coexist there,
|
||
so no single depth represents it.
|
||
- Disparity -> metres needs the VR180 rig's focal length and baseline, which are
|
||
not known a priori.
|
||
- Compute is NOT the blocker: the naive JS test cost ~250 ms/frame, but a WebGL2
|
||
SAD shader would be ~1-2 ms. Not the limiting factor.
|
||
|
||
Prior art (the depth question is studied; the answer is not "match scene depth"):
|
||
|
||
- Sidenmark, Kiefer & Gellersen, *Subtitles in Interactive VR: Using Gaze to
|
||
Address Depth Conflicts* (IEEE VR'19, Lancaster): depth conflicts between
|
||
environment and subtitles cause **occlusion and strain**; they propose
|
||
**gaze-based** techniques.
|
||
- Shimizu et al. (IEEE 2021): four dynamic subtitle placement methods, balancing
|
||
vertical/horizontal/**depth** of eye movement.
|
||
- *Gaze-Adaptive Subtitles for 360° Videos in VR* (IEEE 2025): adaptive placement
|
||
reduces visual fatigue versus fixed below-the-FOV placement.
|
||
|
||
**But**: the literature's best technique needs **eye tracking**, and Meta does
|
||
not expose `eye-tracking` to WebXR — our negotiated `enabledFeatures` were
|
||
`[local, viewer, bounded-floor, layers, hand-tracking, webxr, local-floor]`.
|
||
So gaze-adaptive placement is not available to us.
|
||
|
||
Recommendation if this is pursued: option **A** — use the *mid-band* (reliable)
|
||
disparity to bias the depth toward the scene, blended ~30/70 with the comfortable
|
||
default, clamped to 1.4-3.0 m, smoothed with a 2-3 s time constant, and scale the
|
||
quad proportionally so the **angular text size stays constant** (otherwise the
|
||
text visibly pulses). Default OFF. Chasing per-frame scene depth (option C) is
|
||
not recommended: disparity noise plus vergence conflict produces discomfort.
|
||
|
||
---
|
||
|
||
# ADDENDUM 4 — in-VR subtitle placement
|
||
|
||
Adjust the subtitles **without leaving VR**. Verified on device this round.
|
||
|
||
## Grab & drag the strip
|
||
|
||
Point at the subtitle text, pinch, and move:
|
||
|
||
- **up / down** → vertical position (drag on a fixed reference plane)
|
||
- **pull toward / push away** → depth (the hand's `origin.z` in viewer space;
|
||
forward is `-Z`, so pulling back brings the strip closer — like a real object)
|
||
- release → the new position is saved
|
||
|
||
Both the ray and the strip live in **viewer space**, so no world-space maths is
|
||
needed. The reference plane is fixed at grab time so the drag stays stable.
|
||
|
||
### Two deliberate restrictions (both needed)
|
||
|
||
The strip sits in the lower centre of the view by default and the control bar is
|
||
0.30 m tall at `y = -0.28`, so the two **overlap in y**. Without restrictions the
|
||
strip would swallow every aim in the lower half of the view.
|
||
|
||
1. Grabbing only happens when the control bar is **not already shown** — a pinch
|
||
while operating the bar never yanks the subtitles.
|
||
2. Only the **central 40 %** of the strip counts as grabbable (`nx,ny ∈ 0.30–0.70`),
|
||
which is where the text actually is (the crop is letterboxed into the texture).
|
||
|
||
## In-VR buttons on the control bar
|
||
|
||
The expanded bar grew from 1.1 × 0.22 m to **1.1 × 0.30 m** and gained a subtitle
|
||
row. Verified reachable layout (grid scan, no ambiguous overlaps):
|
||
|
||
```
|
||
y 0.25-0.45 seek track (full width)
|
||
y 0.50-0.70 [<<<<][PPPP][>>>>][ZZZZ] back10 / playpause / fwd10 / subs
|
||
y 0.85-1.00 [aa][AA][oo][OO][RR] size −/+ · opacity −/+ · reset
|
||
```
|
||
|
||
Actions: `sub_smaller`, `sub_bigger`, `sub_opaque_down`, `sub_opaque_up`, `sub_reset`.
|
||
|
||
## Persistence
|
||
|
||
`localStorage['jfvr.settings.v1']`, debounced 400 ms, loaded at boot with range
|
||
validation and silent fallback on corruption.
|
||
|
||
```json
|
||
{"subPlacement":{"y":-0.78,"z":-1.45,"width":2.8},"subOpacity":0.45,
|
||
"subMode":"hud","subAlpha":"premultiplied","projection":"180","stereo":"sbs",
|
||
"subtitles":true,"controls":true,"ctlAutoHideMs":4000}
|
||
```
|
||
|
||
## Verified this round
|
||
|
||
```
|
||
live setPlacement -> layer [-0.7, -2.1, 2.6] immediately
|
||
live setOpacity -> 0.35
|
||
resetPlacement() -> [-0.42, -1.45, 2]
|
||
slider -> localStorage written, and after a page reload the values come back
|
||
errors: []
|
||
```
|
||
|
||
Note: the four subtitle sliders in the 2D panel and the in-VR buttons now share
|
||
one code path (`setSubPlacementLive` / `setSubOpacityLive` / `setSubModeLive`),
|
||
so both save automatically.
|
||
|
||
---
|
||
|
||
# ADDENDUM 6 — subtitle depth vs size decoupling (angular-size lock)
|
||
|
||
## The problem, as the owner put it
|
||
|
||
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
|
||
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
|
||
|
||
Fixation is a **vergence** problem: an object at 0.4 m and subtitles at 1.45 m cannot
|
||
both be looked at comfortably at the same time. Eye tracking is not available to
|
||
WebXR on Quest (`enabledFeatures` has no `eye-tracking`), so the depth cannot be
|
||
driven by gaze — see ADDENDUM 3 above, which already concluded the gaze-adaptive
|
||
literature is out of reach for us.
|
||
|
||
What IS in reach is making **depth a pure depth knob**: keeping the strip's angular
|
||
size (and its angular position) constant while it moves in depth. That is exactly
|
||
the "scale the quad proportionally so the angular text size stays constant" note
|
||
from ADDENDUM 3, applied to the manual control.
|
||
|
||
Measured behaviour of the old control (free mode), which is why it did not work:
|
||
|
||
| depth | width | strip angular width |
|
||
| --- | --- | --- |
|
||
| 1.45 m | 2.00 m | 69° |
|
||
| 0.70 m | 2.00 m | 110° ← text blows up |
|
||
| 0.35 m | 2.00 m | 141° ← text leaves the view |
|
||
|
||
## What shipped
|
||
|
||
`state.subLockAngle = true`, `state.subAngleDeg = 69.2` (defaults). While locked:
|
||
|
||
```
|
||
width = 2 · |z| · tan(subAngleDeg / 2) constant angular width
|
||
y = (y0 / z0) · z constant angular height
|
||
```
|
||
|
||
so the strip stays at the same spot on the retina and only its **vergence
|
||
distance** changes. 69.2° is exactly the old `2.0 m @ 1.45 m`, so the default look
|
||
is unchanged. Depth range extended: near clamp **0.5 m → 0.35 m**.
|
||
|
||
In-VR subtitle row grew from five to nine buttons (the bar itself is unchanged —
|
||
`PANEL_M.expanded` = 0.72 × 0.21 m, canvas 1440 × 420 px). The row now carries
|
||
vertical position as well as depth, and the caption always leads with the depth:
|
||
|
||
```
|
||
y 0.80-1.00 [↑][↓][<<][>>][aa][AA][oo][OO][RR] 上 下 · 近 远 · 小 大 · 暗 亮 · 复位
|
||
caption 字幕 距1.45m · 69° 85%
|
||
```
|
||
|
||
`上`/`下` step the vertical position ±0.04 m (that control did not exist in the bar
|
||
before). `近`/`远` step depth ±0.15 m. `小`/`大` step the locked angle ∓5° (or
|
||
`width` ∓0.15 m when unlocked). Grab & drag carries `y` by the grab-start y/z
|
||
ratio while locked, so a strip pulled close does not slide below the view.
|
||
|
||
## Two bugs fixed in the same pass
|
||
|
||
1. **The CC (字幕) button never showed its state.** `Controls.readSubtitles()` only
|
||
read the controls module's own `S.subtitlesOn`, which `90-main.js` never sets —
|
||
it calls `setVideo(video)` with no `getExtra`. It now falls back to the shared
|
||
`state.subtitles`. The button is also unambiguous now: accent pill + `字幕开`
|
||
when on, slashed CC glyph + `字幕关` when off.
|
||
2. **There was no vertical-position control in VR.** Only grab & drag could move
|
||
the strip. The row gained `上`/`下`. (The 2D panel's `垂直位置` slider was
|
||
verified working — slider → `-0.70 m`, distance → `-0.80 m` re-derived
|
||
`y = -0.385` — so the gap was VR-only.)
|
||
|
||
## Verification (logic, offline)
|
||
|
||
The module was driven headlessly in node (state + maths only, no WebGL), holding
|
||
`subAngleDeg` at 69.2° and stepping depth:
|
||
|
||
```
|
||
z=-1.45 -> width 2.00, y -0.42 (69.2°)
|
||
z=-0.70 -> width 0.97, y -0.205 (69.2°)
|
||
z=-0.35 -> width 0.48, y -0.100 (69.2°)
|
||
z=-2.60 -> width 3.59, y -0.745 (69.2°)
|
||
```
|
||
|
||
and the 9-button row was grid-scanned through `Controls.hitTest` — 9 distinct
|
||
actions, no dead zone between pills. `Controls.draw` renders
|
||
`字幕开 · 字幕 距1.45m · 69°` while subtitles are on and locked, `字幕关` when
|
||
off, and `字幕 距1.45m · 2.0m` when unlocked. `build.py --check` passes.
|
||
|
||
Then verified in the **live Quest 3 browser** over CDP after deploying:
|
||
|
||
```
|
||
window.__JFVR.version -> 'hud-2'
|
||
state.stats.subLockAngle -> true
|
||
state.stats.subAngleDeg -> 69.2
|
||
state.errors -> []
|
||
|
||
Sub.setPlacement({z:-0.70}) -> width 0.97, y -0.20 (still 69.2°)
|
||
Sub.setPlacement({z:-0.35}) -> width 0.48, y -0.10 (still 69.2°)
|
||
Sub.setPlacement({z:-9}) -> width 3.59, y -0.745 (clamped to -2.6, still 69.2°)
|
||
Sub.setAngleDeg(50) -> width 2.42 (size control)
|
||
```
|
||
|
||
## Literature for the default value (not yet acted on)
|
||
|
||
All of these measure the **character height** angular size; our control is the
|
||
whole strip's angular width, so they need `stats.subTextDeg` (new) to compare.
|
||
|
||
- Kojić et al., *User Experience of Reading in Virtual Reality* (arXiv 2004.01545):
|
||
preferred angular size ≈ **16–18 dmm for medium/long text** (0.9–1.0° character
|
||
height), 27–32 dmm for 2-word text, vs Google's guidelines of 32 dmm (title) and
|
||
40 dmm (headline). Participants also rated **too-large** angular size among the
|
||
worst experiences.
|
||
- Dingler et al., *VR Reading UIs* (CHI'18 EA): **41 ± 14 dmm** (≈2.35° ± 0.8°).
|
||
- The default stays 69.2° deliberately: it is the on-device-proven look. Read
|
||
`stats.subTextDeg` in the headset, then decide.
|
||
|
||
## Persistence
|
||
|
||
`jfvr.settings.v1` gains `subLockAngle` and `subAngleDeg`; while locked the stored
|
||
`subPlacement.width` is re-derived from `subAngleDeg` + `z` on load.
|
||
|
||
```json
|
||
{"subPlacement":{"y":-0.42,"z":-0.35,"width":0.48},
|
||
"subLockAngle":true,"subAngleDeg":69.2,"subOpacity":0.85,
|
||
"subMode":"hud","subAlpha":"premultiplied","projection":"180","stereo":"sbs",
|
||
"controls":true,"ctlAutoHideMs":4000}
|
||
```
|
||
|
||
---
|
||
|
||
# ADDENDUM 7 — summon gating, lock badge, pre-VR panel trim
|
||
|
||
## The bar appeared on every pinch
|
||
|
||
Two independent causes, both removed:
|
||
|
||
1. `session.addEventListener('selectstart', onSelectStart)` called `Controls.show()`
|
||
unconditionally — any pinch anywhere in the view-summoned the bar.
|
||
2. `pointerDown()` called `Controls.show()` *before* it looked at the hit.
|
||
|
||
Now the bar is summoned only when the pointing ray lands on (or near) the bar:
|
||
|
||
```
|
||
bar hidden + aim on/near the bar -> show(), consume the press, arm for 500 ms
|
||
bar hidden + aim elsewhere -> nothing
|
||
bar shown + within 500 ms -> ignored
|
||
bar shown + aim on the bar -> handleSelect (and show() to extend auto-hide)
|
||
```
|
||
|
||
While LOCKED the same rule applies, except that what comes up is the padlock panel
|
||
alone — see ADDENDUM 7b below.
|
||
|
||
The summon window is wider than the hit window (`panelHit` padding 0.5/0.9 vs
|
||
0.25/0.6), because a ray that grazes the bar should still be able to bring it up —
|
||
the 500 ms arm is what prevents the accidental button press, not pixel precision.
|
||
|
||
**Superseded by ADDENDUM 7b:** the padlock was originally a separate quad layer to the
|
||
right of the bar. That did not work on device — the hand ray cannot reach that far
|
||
right. The padlock is now a normal panel button and "locked" is a third panel mode.
|
||
|
||
## Pre-VR settings panel trimmed
|
||
|
||
Sliders set on a flat screen do not predict what reads well in a headset, so the
|
||
subtitle section is now just 开 / 关. Removed: 模式, 透明合成, 不透明度, 垂直位置,
|
||
距离, 宽度, 距离与大小 (lock toggle), 视角宽度, 手部指向. `subAngleDeg`, placement
|
||
and opacity still persist — they are written by the in-VR controls.
|
||
|
||
The 手部指向 option (system ray vs index-finger ray) is gone together with its code
|
||
path; the system ray is the only mode (an index-finger ray injects fingertip jitter
|
||
into the cursor — see HANDOFF §6.6).
|
||
|
||
## "VR 播放" no longer covers the player's own controls
|
||
|
||
`.jfvr-btn` moved from `bottom: 24px` to `bottom: 120px`; at the viewport corner it
|
||
sat on Jellyfin's OSD fullscreen button. `.jfvr-panel` moved to `bottom: 180px`
|
||
(`max-height: calc(100vh - 200px)`) so the panel still stacks above the button.
|
||
|
||
## Verification
|
||
|
||
`build.py --check` passes. A local harness (a `file://` page with a fake 8192x4096
|
||
`<video>`, a fake libass canvas and stub `XRWebGLBinding`/`XRXRigidTransform`/
|
||
`createQuadLayer` classes) exercised the real bundle without a headset:
|
||
|
||
```
|
||
Controls.create(stubSession) -> ok, 4 quad layers (bar, beam x2, lock)
|
||
Controls.getLockLayer() -> the lock quad
|
||
Controls.setLock(true) -> stats.ctlLock = true, no throw
|
||
runFrameHooks(fakeFrame) -> presentLock/present ran, errors: []
|
||
panel DOM -> 画面: 投影/立体 · 字幕: 显示 · 控制: 控制条/自动收起
|
||
.jfvr-btn computed bottom -> 120px
|
||
drawLockBadge() -> 16 canvas ops, no throw (locked and unlocked)
|
||
```
|
||
|
||
Live Quest 3 over CDP after deploy: `__JFVR.version = 'hud-4'`,
|
||
`stats.subLockAngle = true`, `errors: []`.
|
||
|
||
---
|
||
|
||
# ADDENDUM 7b — padlock as a panel button + the "locked" panel mode
|
||
|
||
## The failure on device
|
||
|
||
> 我有新发现,在面板唤出状态时其无法点击,但面板消失后,其可以点击。
|
||
|
||
ADDENDUM 7's padlock was its own quad layer at `(0.45, -0.34, -1.45)` — 0.45 m right of
|
||
the panel centre, 0.09 m right of the panel's own right edge. Two things were wrong:
|
||
|
||
1. **It was outside every reachable ray.** Aiming is the binding constraint with hands:
|
||
a hand ray plus the cursor's acceptance window (`panelHit` 0.25/0.6 on a 0.72 m panel)
|
||
reaches x ≈ 0.45 — the panel's right edge. Nothing could aim 0.09 m past it.
|
||
2. **Its hit test used the panel's layer geometry.** Whether it fired therefore depended
|
||
on the panel's own size/position, which is why it behaved differently shown vs hidden.
|
||
|
||
## The fix: make it an ordinary button
|
||
|
||
The padlock is now the fifth button in the media row and uses the same code path as
|
||
everybutton that already worked (cursor → `hitTest` → `handleSelect` → `onAction`):
|
||
|
||
```
|
||
expanded: [<<<] [>||] [>>>] [CC] [padlock]
|
||
0.18 0.36 0.54 0.72 0.90 (hit pill half width 0.092)
|
||
```
|
||
|
||
## The behavior the owner asked for
|
||
|
||
> 想要的是:唤出面板,然后可以上锁,上锁后锁头图标和面板一样自动隐藏,下次触发面板
|
||
> 唤出时,只展示锁本身,不展示进度条等能调整的项目。(解锁后的0.5s的动做也忽略)
|
||
|
||
A third panel mode does exactly that:
|
||
|
||
| mode | size (m) | canvas | drawn / hit-testable |
|
||
| --- | --- | --- | --- |
|
||
| `expanded` | 0.72 × 0.21 | 1440 × 420 | everything |
|
||
| `locked` | 0.26 × 0.10 | 520 × 200 | the padlock + 解锁 label ONLY |
|
||
|
||
```
|
||
summon (unlocked) -> expanded panel, auto-hides after ctlAutoHideMs
|
||
press 上锁 -> panel becomes padlock-only IN PLACE, auto-hide timer restarts
|
||
(so the padlock disappears on its own, like the panel always did)
|
||
summon (locked) -> padlock alone; no progress bar, no rows, nothing adjustable
|
||
press 解锁 -> full panel returns, and 500 ms of presses are ignored
|
||
```
|
||
|
||
Because `hide()` no longer resets the mode, the layer keeps the size it will be shown
|
||
at: the summon window stays the full panel rect (0.72 × 0.21 unlocked, 0.26 × 0.10
|
||
locked) and re-showing never pops a resize. `hitTest()` takes the mode explicitly now
|
||
(`collapsed` and `locked` share no safe aspect-ratio invariant), and `handleSelect()`
|
||
no longer falls back to pressing the panel centre when there is no ray hit.
|
||
|
||
## Verification (offline, real bundle, no headset)
|
||
|
||
A temporary `file://` harness fed the real `dist/jfvr.js` a fake 8K `<video>`, a fake
|
||
libass canvas and stub `XRWebGLBinding` / `XRXRigidTransform` / `createQuadLayer`
|
||
classes. The harness file was deleted afterwards.
|
||
|
||
```
|
||
Controls.create(stubSession) -> ok, layers: bar + beam x2
|
||
expanded media row (5 centres) -> back10|playpause|fwd10|subs|lock, no dead zone
|
||
expanded subtitle row (9 centres) -> 9 distinct actions
|
||
locked panel: centre -> {type:'lock'} -> bottom edge / left edge -> null
|
||
draw() locked -> draws ONLY '解锁'
|
||
draw() expanded -> 上锁 + 字幕 距1.45m · 69° + 00:10 (all intact)
|
||
setLock(true) -> stats.ctlMode=locked, layer 0.72x0.21 -> 0.26x0.10, canvas resized
|
||
setLock(false) -> stats.ctlMode=expanded, layer back to 0.72x0.21
|
||
hide() -> mode KEPT (summon window stays the panel rect)
|
||
runFrameHooks() after each switch -> no errors, stats.hookErr / ctlErr null
|
||
handleSelect() with no ray hit -> null (was: silently pressed the centre)
|
||
```
|
||
|
||
Also standalone (`node`): the padlock hit region and the mode table above were asserted
|
||
directly against `Controls.hitTest` / `Controls.draw` with mock 2D contexts.
|
||
|
||
Live Quest 3 over CDP after deploy: `__JFVR.version = 'hud-5'`, `stats.ctlMode =
|
||
'expanded'`, `stats.ctlLock = false`, `errors: []`.
|
||
|
||
## Still unverified
|
||
|
||
The *feel* — how easily the padlock is aimed with the hand ray, and whether 0.5 s is the
|
||
right arm delay — can only be judged in the headset.
|
||
|
||
---
|
||
|
||
# ADDENDUM 8 — panning the 180° video window about the eye
|
||
|
||
## The requirement
|
||
|
||
> 我希望画面可以不是固定在我的正前方180度,我希望能调整画面内容在整个vr空间中以我眼睛为
|
||
> 旋转点进行上下左右的水平移动(为了解决一些场景需要物理低头才能看到的问题,我不想真的
|
||
> 低头去看)
|
||
|
||
The window was hard-wired dead ahead, so anything staged below the horizon cost a
|
||
physical head-down.
|
||
|
||
## Why a pure rotation is the right mechanism
|
||
|
||
The video layer is an `XREquirectLayer` with `radius` 0, i.e. the content is **at
|
||
infinity**. That single fact decides everything:
|
||
|
||
| Property of radius-0 equirect | Consequence for this feature |
|
||
| --- | --- |
|
||
| Only the orientation is visible; translation has no effect | Panning = rotating `layer.transform`, nothing else |
|
||
| No parallax | The picture does not swim when the head moves |
|
||
| It is a sphere "around everywhere" | The rotation is about the **eye point** — literally what was asked |
|
||
| No new texture / no re-upload | 8K HEVC DirectPlay, 120 Hz and the ~112 fps budget are untouched |
|
||
|
||
Only the video layer rotates. The control panel is at a fixed pose in `refSpace`, the
|
||
subtitles are head-locked in `viewerSpace`, so **the UI does not move with the
|
||
picture**.
|
||
|
||
Rejected alternatives:
|
||
|
||
| Alternative | Why not |
|
||
| --- | --- |
|
||
| `refSpace.getOffsetReferenceSpace()` | Rotates the panel, the hands and every ray too — the UI would leave the user's view |
|
||
| Moving the layer's position | An infinite-radius sphere shows no visible change (and would add parallax/swim) |
|
||
| Changing `centralHorizontalAngle` | That changes FOV/distortion, not where the window points |
|
||
|
||
## Controls
|
||
|
||
`state.videoYaw` (+ = picture right) and `state.videoPitch` (+ = picture up):
|
||
|
||
```
|
||
q = qy(-yaw) * qx(pitch) // pitch in the content frame, then yaw about world Y
|
||
aim = (cos p·sin yaw, sin p, -cos p·cos yaw) // the window's aim direction
|
||
```
|
||
|
||
The yaw sign is flipped so the two numbers read the way a user thinks about them, and
|
||
every value shown in the panel is a plain "moved this many degrees right/up".
|
||
Pitch is limited only by a gimbal guard (±(π/2 − 0.02)) per the owner's "不限,看用户拖动
|
||
幅度来定"; yaw wraps to [−π, π]. **Reset to 0/0 on every VR session** — not persisted.
|
||
|
||
Interaction is a single affordance that lives in the panel, above the progress bar
|
||
(the owner's requested placement):
|
||
|
||
```
|
||
⠿ 拖动调整画面 偏航 +12° 俯仰 -8°
|
||
连点 3 次复位 0° / 0° = 正前方
|
||
```
|
||
|
||
- Pinch it and drag — the picture follows the hand 1:1.
|
||
- Three taps within 1.6 s reset to dead ahead; taps are counted on **release**, so a
|
||
drag cannot be mistaken for one.
|
||
- The live angles are printed on the right at all times.
|
||
|
||
The drag uses the **ray's direction**, not a cursor position on the panel:
|
||
|
||
```
|
||
yaw' = wrap(yaw0 + (rayYaw - rayYaw0)), pitch' = clamp(pitch0 + (rayPitch - rayPitch0))
|
||
rayYaw = atan2(dx, -dz), rayPitch = asin(dy)
|
||
```
|
||
|
||
A panel-space cursor leaves the panel after a few degrees; ray rotation has no range
|
||
limit, and the content point under the cursor stays under the cursor — which is what
|
||
"grab the picture" has to feel like. `Controls.hold(true)` freezes the panel auto-hide
|
||
for the duration, otherwise the panel would auto-hide mid-drag.
|
||
|
||
## The one runtime unknown
|
||
|
||
> **SUPERSEDED by ADDENDUM 8b** — this probe compared quaternion components (sign- and
|
||
> normalisation-sensitive) and so could report "read-only" on a perfectly good runtime.
|
||
> The gesture also changed: tap = reset, hold 0.5 s then drag = pan.
|
||
|
||
`XREquirectLayer.transform` is settable per spec but a runtime may ignore the write. It
|
||
is probed once per session (write a known rotation, read it back):
|
||
|
||
```
|
||
stats.videoRotMode = 'live' -> a transform write per frame, free
|
||
= 'rebuild' -> recreate the media layer, but only on
|
||
creation / release / reset (never per frame)
|
||
```
|
||
|
||
`stats.videoRebuilds` counts rebuilds. **This probe needs a real XR session, so it is
|
||
the one thing that cannot be checked offline.**
|
||
|
||
## Offline verification (real bundle, no headset)
|
||
|
||
Quaternion / drag maths, extracted from `src/90-main.js` and driven directly:
|
||
|
||
```
|
||
yaw=0 pitch=0 -> aim (0, 0, -1) dead ahead
|
||
yaw=+20 -> aim x = +0.342 picture moved RIGHT
|
||
yaw=-20 -> aim x = -0.342 picture moved LEFT
|
||
pitch=+20 -> aim y = +0.342 picture moved UP
|
||
pitch=-20 -> aim y = -0.342 picture moved DOWN
|
||
yaw=30 pitch=40 -> unit direction, azimuth 30.000° and elevation 40.000° recovered
|
||
wrapYaw(±3π/2) -> ∓π/2 wraps into [-π, π]
|
||
clampPitch(±9) -> ±(π/2-0.02) gimbal guard holds
|
||
drag -> the aimed point ends up under the cursor (yaw and pitch)
|
||
```
|
||
|
||
Panel geometry via `Controls.hitTest` on the 1440×520 canvas:
|
||
|
||
```
|
||
y 0.05 / 0.18 -> view_drag (the whole strip)
|
||
y 0.25 -> null (time readout: intentionally inert)
|
||
y 0.40 -> seek
|
||
y 0.60 -> media row (back10|playpause|fwd10|subs|lock)
|
||
y 0.90 -> subtitle row (9 actions)
|
||
x sweep 0.04..0.96 at y 0.05 -> no dead column
|
||
```
|
||
|
||
`Controls.draw` renders `拖动调整画面` / `连点 3 次复位` / `偏航 +12° 俯仰 -8°` /
|
||
`0° / 0° = 正前方`, and `hold(true|false)` is reflected in `stats.ctlHold`.
|
||
|
||
Live Quest 3 over CDP after deploy: `__JFVR.version = 'hud-6'`,
|
||
`state.videoYaw = state.videoPitch = 0`, `errors: []`.
|
||
|
||
---
|
||
|
||
# ADDENDUM 8b — live pan, the crash, and the real gesture
|
||
|
||
## Why the picture only moved on release
|
||
|
||
ADDENDUM 8 gated the transform write on a probe that compared **quaternion components**:
|
||
|
||
```js
|
||
live = Math.abs(readBack.orientation.y - q.y) < 1e-4 && Math.abs(readBack.orientation.w - q.w) < 1e-4;
|
||
```
|
||
|
||
`q` and `−q` are the same rotation, and a runtime may return a renormalised quaternion,
|
||
so this could answer "read-only" on a runtime that would have accepted the write. When it
|
||
did, the drag fell into the rebuild branch — which was throttled to release-only — so the
|
||
picture did not move until the finger came up.
|
||
|
||
The probe now compares the **position** (exact, no sign ambiguity):
|
||
|
||
```
|
||
layer.transform = XRRigidTransform({0.123, -0.456, 0.789}, identity)
|
||
writable <=> layer.transform.position equals that marker
|
||
```
|
||
|
||
## Two mechanisms, mutually exclusive
|
||
|
||
| `stats.videoRotMode` | Mechanism | Cost per change |
|
||
| --- | --- | --- |
|
||
| `transform` | `layer.transform = q` | one write |
|
||
| `space` | `layer.space = refSpace.getOffsetReferenceSpace(q)` | one space alloc |
|
||
| `rebuild` | recreate the media layer (manual escape hatch) | ~8/second max |
|
||
|
||
`space` exists because if the transform write is genuinely ignored,
|
||
`getOffsetReferenceSpace()` is the recentering path every WebXR app relies on, and a
|
||
layer's pose is recomputed from its space each frame. The two are never combined:
|
||
`createVideoLayer()` only bakes the transform in for the `rebuild` strategy, otherwise the
|
||
probe's write would double the aim. `stats.videoTransformWritable` shows the raw probe
|
||
result; the 2D panel's 「画面旋转: 自动 / 兼容」 can force the rebuild path.
|
||
|
||
## The crash, and every fix around it
|
||
|
||
The reset rebuilt the layer and created the replacement **before** destroying the old
|
||
one — two `XRMediaBinding` equirect layers on one `<video>` element at once, then a
|
||
second rebuild microseconds later (release rebuilt, then reset rebuilt again) took the
|
||
browser down.
|
||
|
||
A rebuild now: runs in a `setTimeout` **outside** the XR frame callback, **destroys
|
||
first**, is throttled to 120 ms, and never queues more than one. The `transform`/`space`
|
||
paths never rebuild at all, so the normal case cannot reach the crash.
|
||
|
||
Two robustness holes found while testing:
|
||
|
||
- `videoSpaceFor()` let a throwing `getOffsetReferenceSpace()` escape into the frame
|
||
loop, where an exception stops it for good (`requestAnimationFrame` is only re-armed at
|
||
the end of the callback) — a frozen session. It is caught, and the strategy demotes to
|
||
`rebuild`.
|
||
- The whole per-frame body is now wrapped in a try/catch for the same reason.
|
||
|
||
## The gesture
|
||
|
||
```
|
||
press the drag strip
|
||
├─ released before 500 ms with no movement -> RESET to dead ahead
|
||
├─ held past 500 ms, then moved -> PAN
|
||
└─ held past 500 ms, never moved -> RESET
|
||
```
|
||
|
||
The pan reference is taken **when the hold threshold passes**, not at the press, so the
|
||
hold itself cannot fling the picture; a ~1° dead zone absorbs hand jitter. The strip reads
|
||
`点一下复位 · 按住 0.5s 再拖动`.
|
||
|
||
## Verification
|
||
|
||
Everything below was asserted offline against the real sources with a stubbed ray, stubbed
|
||
WebXR layer objects and a stubbed clock (`build.py --check` passes as well):
|
||
|
||
```
|
||
test-gesture quick tap -> reset; hold-without-move -> reset; movement inside the hold
|
||
window ignored; after arming, +10°/+5° ray -> exactly +10°/+5°; a pan does
|
||
not reset; dead zone respected; pitch guard holds
|
||
test-mech probe true for a writable transform, false for an ignored one;
|
||
transform strategy: writes, no rebuild; space strategy: assigns the offset
|
||
space, leaves the transform alone, no rebuild; unchanged angle reuses the
|
||
cached space; rebuild strategy: one pending max, observed sequence
|
||
removeLayer -> createVideoLayer -> addLayer; throwing/absent
|
||
getOffsetReferenceSpace -> reported + permanent fallback
|
||
test-ad8 quaternion aim/wrap/guard maths, panel bands on the 1440x520 canvas,
|
||
the strip's drawn readout, hold()
|
||
```
|
||
|
||
Deployed as `mvp-5217a3` (hud-7) and confirmed live on the Quest.
|
||
|
||
## Still to confirm on device
|
||
|
||
Which of `transform` / `space` this runtime actually honours — `stats.videoRotMode` and
|
||
`stats.videoTransformWritable` say so once a session is running.
|