ADDENDUM 9 — two independent defects found on device.
1. VR subtitles froze on whatever line was on screen when the session started.
Jellyfin 12.x renders with SubtitlesOctopus, which starts ONE long-lived rAF
loop when the video loads; entering immersive-vr stops native rAF and destroys
the callback still pending in the native scheduler, and because the instance's
rafId stays non-zero it is never restarted (measured live: rafId=1150 native,
oneshotState.eventStart frozen ~7 min behind the playhead, zero oneshot-render
requests). The broker now ALWAYS queues rAF (native driver outside XR, rVFC +
50 ms idle fallback inside), keeps exactly one native callback in flight, and
no longer clears the queue at session start. A first version re-armed during
the drain and doubled the driver every frame, freezing the page; that is now
fixed and locked down by a Node harness (maxConcurrentNative = 1 over 300
frames, self-rescheduling loop survives a simulated session start).
2. Only dual-eye ASS subtitles worked; a normal subtitle (e.g. the CN SRT track)
is centred across the whole 2:1 frame and was cut at the eye seam because the
HUD always sampled the left half. state.subLayout ('auto'|'dual'|'mono',
default auto, persisted, panel row) plus per-sample detection: seam ink -> mono,
one half only -> dual/mono, both halves with an empty seam -> column-profile
copy test. Dual-eye is NOT a pixel mirror (same text at mirrored positions, no
glyph flip): measured dual copy distance 0.02-0.08 vs mono 1.00, seam ink
0 vs 50. M.eyeW (half for dual, full for mono) replaces the hard-coded halfW.
Docs: INTERFACES.md ADDENDUM 9, HANDOFF.md §13 (including the Jellyfin service
worker cache trap that made the first deploy look like a no-op).
The final build was not re-checked in the headset: the device left the worn state
mid-verification and the browser could not be relaunched. Verified on device: the
loop survives XR (rafId in the plugin range, eventStart tracking, subUploads 4->7
over 20 s in VR) and the layout metrics above; the final decision wiring is
covered by the offline suites.
1072 lines
49 KiB
Markdown
1072 lines
49 KiB
Markdown
# JFVR module interfaces (contract)
|
||
|
||
Everything lives under a single global namespace `window.__JFVR`, created by
|
||
`src/00-core.js` before any other module runs. Modules are plain
|
||
`(function(){ ... })()` IIFEs that attach themselves to `__JFVR`.
|
||
|
||
Load order (build script concatenates in this order):
|
||
|
||
```
|
||
00-core.js -> __JFVR core: state, helpers, layer/frame-hook registry
|
||
10-subtitles.js -> __JFVR.Subtitles
|
||
20-controls.js -> __JFVR.Controls
|
||
90-main.js -> boot, DOM scan, XR session lifecycle, settings panel, wiring
|
||
```
|
||
|
||
Nothing may read `document` / `navigator.xr` at load time except `00-core.js`
|
||
and `90-main.js`. The two module files must only *define* functions and are
|
||
called later with a session/frame.
|
||
|
||
---
|
||
|
||
## 00-core.js — always available as `window.__JFVR`
|
||
|
||
```js
|
||
__JFVR.state // shared mutable state object (see below)
|
||
__JFVR.log(...args) // console.log with [JFVR] prefix + colour
|
||
__JFVR.version // string
|
||
|
||
// --- helpers ---
|
||
__JFVR.makeEl(tag, className?, text?) // document.createElement wrapper
|
||
__JFVR.compileShader(gl, type, source) // throws with the info log on failure
|
||
__JFVR.invert4(mat4ArrayLike) // -> Float32Array(16)
|
||
__JFVR.QUAD_VS // vertex shader source: fullscreen triangle,
|
||
// `in vec2 aPos; out vec2 vUv;`
|
||
// vUv = vec2(aPos.x*.5+.5, .5 - aPos.y*.5)
|
||
__JFVR.bindFullscreenTriangle(gl, program) // creates the VBO, enables aPos, returns attribLoc
|
||
|
||
// --- XR layer registry (owned by 90-main.js) ---
|
||
__JFVR.addLayer(layer) // register + push updateRenderState if a session is live
|
||
__JFVR.removeLayer(layer) // unregister + push
|
||
__JFVR.addFrameHook(fn) // fn(frame, time, session) every XR frame
|
||
__JFVR.removeFrameHook(fn)
|
||
|
||
// --- capabilities (filled by 00-core.js at boot) ---
|
||
__JFVR.caps = {
|
||
secureContext, hasXR, immersiveVR, hasXRMediaBinding, hasEquirectLayer,
|
||
hasXRWebGLBinding, hasXRHand, hasXRJointSpace, maxTextureSize,
|
||
hevc: { ... }, ua
|
||
}
|
||
|
||
// --- live handles ---
|
||
__JFVR.session // current XRSession or null
|
||
__JFVR.video // current <video> element or null
|
||
__JFVR.refSpace // 'local' XRReferenceSpace for the running session
|
||
__JFVR.viewerSpace // 'viewer' XRReferenceSpace (head-locked layers)
|
||
```
|
||
|
||
### `__JFVR.state` fields that modules may use
|
||
|
||
```js
|
||
state.video // <video> or null
|
||
state.session // XRSession or null
|
||
state.refSpace // local space
|
||
state.viewerSpace // viewer space (head-locked)
|
||
state.projection // '180' | '360'
|
||
state.stereo // 'mono' | 'sbs' | 'ou'
|
||
state.subtitles // bool — render subtitles in VR
|
||
state.subMode // 'hud' (head-locked quad, default)
|
||
// 'sphere'(on the equirect layer, legacy)
|
||
state.subAlpha // 'premultiplied' | 'straight'
|
||
state.subPlacement // { y, z, width } metres, hud mode only
|
||
state.stats // diagnostics bag, free to add keys
|
||
state.gl // { refSpace, session } set by main during a session
|
||
```
|
||
|
||
### Layer / frame-hook lifecycle a module must follow
|
||
|
||
1. `create(session, glContextOrNull)` — build GL resources + the XR layer,
|
||
call `__JFVR.addLayer(layer)`, `__JFVR.addFrameHook(hook)`.
|
||
Return the layer, or `null` when it cannot run (missing canvas, missing API…).
|
||
**Never throw** — return null and `__JFVR.log()` the reason.
|
||
2. `hook(frame, time, session)` — per-frame work. Must be cheap; skip work when
|
||
nothing changed.
|
||
3. `destroy()` — remove the frame hook, remove the layer, delete GL resources,
|
||
drop canvas references. Must be safe to call twice and safe to call when
|
||
`create()` returned null.
|
||
|
||
---
|
||
|
||
## 10-subtitles.js — `__JFVR.Subtitles`
|
||
|
||
Renders Jellyfin's libass subtitle canvas into VR.
|
||
|
||
```js
|
||
__JFVR.Subtitles = {
|
||
// Called once at boot, before any session.
|
||
init(): void,
|
||
|
||
// BEFORE entering VR. Jellyfin sizes the libass canvas to the on-screen video
|
||
// box (~1250x625), which is far too small for VR. Blowing up the player box
|
||
// makes libass re-render bigger. Must be called while NOT in an immersive
|
||
// session (libass does not recompute its backing store during XR).
|
||
// Idempotent: calling twice must not double-apply.
|
||
prepare(): void,
|
||
|
||
// Undo prepare(). Safe when prepare() was never called.
|
||
restore(): void,
|
||
|
||
// True when a libass canvas exists and has non-zero size.
|
||
isReady(): boolean,
|
||
|
||
// Current libass canvas size, for diagnostics: { w, h } or null
|
||
canvasSize(): { w, h } | null,
|
||
|
||
// --- session lifecycle ---
|
||
create(session): XRLayer | null,
|
||
destroy(): void,
|
||
|
||
// --- settings, safe to call before or during a session ---
|
||
setMode(mode), // 'hud' | 'sphere' | 'off'
|
||
setAlphaMode(mode), // 'premultiplied' | 'straight'
|
||
setPlacement({ y, z, width }), // partial update, hud only
|
||
getMode(): string
|
||
};
|
||
```
|
||
|
||
### HUD mode requirements (the default, and the point of this module)
|
||
|
||
- A **head-locked quad layer** in `__JFVR.viewerSpace`
|
||
(`XRWebGLBinding.createQuadLayer({ space: viewerSpace, layout: 'mono', ... })`).
|
||
- Source pixels come from `canvas.libassjs-canvas`. That canvas holds a
|
||
**stereo left-right** image: the left half is the `LeftEye` subtitle, the
|
||
right half the `RightEye` one. For a mono HUD use **one half only**
|
||
(the left half is correct for this content).
|
||
- **Auto-crop to the ink bounding box** so the strip is tight and the text is
|
||
as large as possible. Recompute the bbox at a low rate (≈4 Hz) on a
|
||
downscaled copy — never `getImageData` the full-resolution canvas every frame
|
||
(a 4320x2160 full read is ~37 MB and has already crashed a renderer once).
|
||
Add hysteresis / smoothing so the strip does not jitter between frames.
|
||
- Upload only when the source changed (or at most ~30 Hz) — the earlier
|
||
implementation burned a 4K texture upload every frame.
|
||
- Target ≈30 px/degree. A 2.0 m wide strip at 1.45 m is ≈66° → texture width
|
||
2048 is comfortable on Quest 3.
|
||
- Respect `state.subAlpha`:
|
||
- `'premultiplied'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true)`
|
||
- `'straight'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false)`
|
||
- Draw with `XRWebGLBinding.getSubImage(layer, frame)` → bind
|
||
`sub.colorTexture` to a framebuffer → viewport to
|
||
`sub.colorTextureWidth/Height` → clear to `(0,0,0,0)` → draw.
|
||
`getSubImage` takes **two** arguments.
|
||
|
||
### `prepare()` detail
|
||
|
||
The measured behaviour: setting the player box to `7680x3840` makes libass
|
||
produce a **4320x2160** canvas (it does not go higher). The previous bug was
|
||
that this was only applied on "open panel" and then restored on session end
|
||
while the panel stayed open, so a re-entry got a 1250x625 canvas.
|
||
`prepare()` must therefore be idempotent and re-callable.
|
||
|
||
Target DOM: `document.querySelector('.videoPlayerContainer')` and the `<video>`.
|
||
Save/restore their inline `style.cssText` exactly.
|
||
|
||
---
|
||
|
||
## 20-controls.js — `__JFVR.Controls`
|
||
|
||
A head-locked control strip: progress bar + seek, play/pause, skip ±10 s,
|
||
subtitle toggle. **Pure 2D canvas rendering + a hit-test function, plus a thin
|
||
XR quad layer wrapper.**
|
||
|
||
```js
|
||
__JFVR.Controls = {
|
||
init(): void,
|
||
|
||
// --- pure rendering, no WebXR, must be unit-testable in isolation ---
|
||
// Draws the bar into `ctx` sized `w` x `h` (CSS pixels).
|
||
// `view` = {
|
||
// playing, currentTime, duration, buffered, subtitlesOn,
|
||
// mode: 'collapsed' | 'expanded',
|
||
// hover: null | 'playpause' | 'back10' | 'fwd10' | 'subs' | 'track'
|
||
// }
|
||
draw(ctx, w, h, view): void,
|
||
|
||
// Maps a point in normalized panel coords (0..1, origin top-left) to an action.
|
||
// Returns one of:
|
||
// { type: 'seek', value: 0..1 }
|
||
// { type: 'playpause' | 'back10' | 'fwd10' | 'subs' }
|
||
// null
|
||
hitTest(nx, ny, w, h): action | null,
|
||
|
||
// --- XR lifecycle ---
|
||
create(session): XRLayer | null,
|
||
destroy(): void,
|
||
|
||
// --- state / input ---
|
||
setVideo(videoEl, getExtra | null): void, // main provides time/duration/buffered readers
|
||
onAction(cb): void, // cb(action) — main performs the media op
|
||
show(autoHideMs?), hide(): void, // collapsed <-> expanded + auto-hide
|
||
isVisible(): boolean,
|
||
handleSelect(inputSource, frame): void // called on XR `select`
|
||
};
|
||
```
|
||
|
||
### Panel design requirements
|
||
|
||
- **Two visual states**: `collapsed` = a slim progress bar only (≈0.9 m x 0.07 m,
|
||
centre-bottom of view). `expanded` = progress row + buttons
|
||
(≈1.1 m x 0.22 m).
|
||
- Hit targets must be **generous**: the progress bar should be visually ~1 cm
|
||
tall but hit-testable over ~3–4 cm. Meta's docs warn that the emulated hand
|
||
target ray for hand tracking is only a coarse forward pointer, so **do not
|
||
rely on pixel-precise aiming**.
|
||
- Colours must read on top of arbitrary video: dark translucent pill background,
|
||
white text, accent `#4ea1ff` (matches the existing in-page panel).
|
||
- Everything must be drawn from scratch on a canvas — no DOM, no images, no
|
||
external fonts (use a generic stack: `'system-ui, -apple-system, sans-serif'`).
|
||
- `draw()` must be deterministic and side-effect free (no timers inside).
|
||
- `hitTest` must be a pure geometry function of `(nx, ny, w, h)`.
|
||
|
||
### Input model (main.js supplies)
|
||
|
||
- **Gaze**: the panel is head-locked, so the reticle is the view centre.
|
||
`main.js` converts that into `(nx, ny)` and calls `hitTest`.
|
||
- **Controller ray**: `main.js` intersects the controller ray with the panel
|
||
plane and converts to `(nx, ny)`.
|
||
- Both end in `handleSelect()` → the action from `hitTest` → `onAction` callback.
|
||
|
||
---
|
||
|
||
## 90-main.js — owned by the integrator
|
||
|
||
Boot, DOM scan, the in-page `VR 播放` button + settings panel, the XR session
|
||
lifecycle, the rAF broker (see below), input dispatch, and the settings UI for
|
||
subtitle mode / alpha mode / placement.
|
||
|
||
### rAF broker (already proven, keep it)
|
||
|
||
While an immersive XR session is running `window.requestAnimationFrame` is
|
||
**completely stopped** (measured: 0 callbacks in 6 s), and Jellyfin renders
|
||
libass subtitles from an rAF loop. `90-main.js` intercepts
|
||
`requestAnimationFrame`/`cancelAnimationFrame`, holds callbacks while in XR and
|
||
drives them from `video.requestVideoFrameCallback`. Outside XR it delegates to
|
||
the native implementation.
|
||
|
||
---
|
||
|
||
# ADDENDUM 2 — in-VR UX pass (after first on-device test)
|
||
|
||
Owner feedback from the first headset test:
|
||
|
||
1. **The control bar is permanently visible — wrong.** It must be hidden by
|
||
default and only appear on interaction, then auto-hide.
|
||
2. **Hand/controller input has no guide line.** There is no visible ray and no
|
||
cursor, so the user cannot tell where they are aiming.
|
||
3. **Subtitles must sit below the visual centre** by default, and both the
|
||
**position and the opacity must be user-adjustable**.
|
||
|
||
## New shared state (owned by 90-main.js, read by modules)
|
||
|
||
```js
|
||
state.subOpacity // 0..1, default 0.85
|
||
state.subPlacement // { y, z, width } — y is the quad CENTRE, negative = below eye level
|
||
state.ctlAutoHideMs // default 4000
|
||
state.pointer // { visible, source, origin:{x,y,z}, dir:{x,y,z}, hit:{nx,ny}|null }
|
||
// written every XR frame by 90-main.js; modules must treat it as read-only
|
||
```
|
||
|
||
## 10-subtitles.js — additions
|
||
|
||
```js
|
||
setOpacity(a) // 0..1; applied to the XR layer (layer.opacity or the
|
||
// shader's alpha), NOT by rebuilding the layer
|
||
getOpacity(): number
|
||
```
|
||
- Default `state.subPlacement` becomes `{ y: -0.42, z: -1.45, width: 2.0 }`.
|
||
- `setPlacement()` must take effect live (update `layer.transform` if the runtime
|
||
allows mutating it, otherwise rebuild just this layer and keep the frame hook).
|
||
- Keep the existing tight-crop behaviour and the 4320x2160 canvas preparation.
|
||
|
||
## 20-controls.js — additions
|
||
|
||
```js
|
||
setPointer(p) // p as in state.pointer; called every frame by main.
|
||
// Draw a beam from origin along dir and, when hit != null, a
|
||
// cursor dot at (hit.nx, hit.ny) on the panel.
|
||
setOpacity(a)
|
||
getOpacity(): number
|
||
```
|
||
|
||
- **Default state is HIDDEN.** `create()` must not make the bar visible.
|
||
- `show(autoHideMs)` → visible + expanded, auto-hides after `autoHideMs`
|
||
(default `state.ctlAutoHideMs`).
|
||
- `hide()` → fully hidden (no bar, no beam, no cursor) until `show()` again.
|
||
- `isVisible()` → true only while shown.
|
||
- The beam must be a **separate, always-updated visual** that is only visible
|
||
while the bar is shown. Implement it as a thin quad layer whose transform is
|
||
placed at `origin + dir * (L/2)` and oriented so its local -Z follows `dir`
|
||
(build the quaternion from the direction; L = distance to the hit, or a fixed
|
||
1.2 m when there is no hit). Height ≈ 0.006 m, width = L, with a soft gradient
|
||
texture (bright at the origin end, fading out) so it reads as a laser.
|
||
- The cursor: when `hit` is non-null, draw a ring/dot on the control panel canvas
|
||
at `(hit.nx, hit.ny)` so the user sees exactly what will be activated.
|
||
- Do not re-upload either texture more than ~30 Hz; skip everything when hidden.
|
||
- Add stats keys `ctlPointer`, `ctlHidden`, `ctlBeamOn`.
|
||
|
||
## Input model change (90-main.js) — unified pointing
|
||
|
||
Hand tracking and controllers both expose `targetRaySpace`. Use one code path:
|
||
|
||
- For **every** input source with `targetRayMode === 'tracked-pointer'`
|
||
(controllers *and* hands), intersect its ray with the control quad plane and
|
||
produce `hit`.
|
||
- If no source produces a hit, fall back to gaze: `hit = {nx:0.5, ny:0.5}` centred.
|
||
- `state.pointer.source` records which one won.
|
||
- Show the beam for the winning source; hide it when the controls are hidden.
|
||
|
||
Meta's docs note the hand target ray is an *emulated* forward-pointing ray, so
|
||
hit targets stay generous — that is already the case.
|
||
|
||
### Beam orientation — CORRECTED convention (my first spec was wrong)
|
||
|
||
Do **not** map the quad's local `-Z` onto the ray direction. A quad's local `-Z`
|
||
is its plane **normal**, so that would put the quad's plane perpendicular to the
|
||
ray and lay its `width` extent across the beam — a thin face-on bar, not a beam.
|
||
|
||
Correct:
|
||
|
||
```
|
||
position = origin + dir * (L / 2) // L = hit distance, else 1.2 m
|
||
width = L (the long axis)
|
||
height = 0.006
|
||
local +X -> dir // the beam axis lies IN the plane
|
||
local -Z -> perpendicular to dir, biased up so the ribbon is never edge-on
|
||
gradient -> bright at the origin end, fading to transparent at the hit end
|
||
```
|
||
|
||
Assertion for tests: `R(q)·(1,0,0) ≈ dir` and `R(q)·(0,0,-1) ⟂ dir`.
|
||
|
||
---
|
||
|
||
# ADDENDUM 4 — in-VR subtitle placement (grab & drag)
|
||
|
||
Owner requirement: adjust the subtitles **without leaving VR**. Chosen model:
|
||
|
||
- **Position**: grab the subtitle strip with the laser and drag it in space.
|
||
- **Reset** button restores the defaults.
|
||
- **Size** and **opacity** also adjustable in VR.
|
||
- Values **persist in localStorage** (per browser, no server changes).
|
||
|
||
## Grab & drag model (owned by 90-main.js)
|
||
|
||
The strip is a head-locked quad in viewer space, so both the ray and the strip
|
||
live in the same space — no world-space math needed.
|
||
|
||
```
|
||
on grab start:
|
||
grabActive = true
|
||
grabOrigin0 = ray.origin // viewer space
|
||
grabHit0 = ray ∩ plane(z = placement.z)
|
||
grabPlace0 = { y, z, width }
|
||
|
||
per frame while grabbed:
|
||
P = currentRay ∩ plane(z = grabPlace0.z) // fixed reference plane
|
||
y' = grabPlace0.y + (P.y - grabHit0.y) // drag up/down
|
||
z' = grabPlace0.z + (ray.origin.z - grabOrigin0.z) // pull toward / push away
|
||
clamp y' to [-0.95, -0.05], z' to [-2.6, -0.7]
|
||
Subtitles.setPlacement({ y: y', z: z', width: current })
|
||
|
||
on grab end: clear grab state, persist
|
||
```
|
||
|
||
Rationale for `z`: in viewer space forward is `-Z`, so pulling the hand back
|
||
(increasing `origin.z`) makes `z'` less negative, i.e. the strip comes **closer**
|
||
— which matches how pulling a real object behaves.
|
||
|
||
Grab **must not** steal the normal control-bar interaction: only start a grab when
|
||
the ray actually hits the subtitle quad; otherwise fall through to the bar.
|
||
|
||
## Input: press **and release** are now needed
|
||
|
||
The existing activation path in `90-main.js` only detects a rising edge. Grabbing
|
||
needs the held state too, so track per input source:
|
||
|
||
- `selectstart` / `selectend` session events, **and**
|
||
- `inputSource.gamepad` button press/release (hand pinch; see ADDENDUM 3)
|
||
|
||
Both feed one `pointerDown(src)` / `pointerUp(src)` pair, deduped the same way.
|
||
|
||
## 10-subtitles.js — additions
|
||
|
||
```js
|
||
getLayer(): XRLayer | null // so main can ray-cast against the strip
|
||
resetPlacement(): void // back to { y: -0.42, z: -1.45, width: 2.0 }
|
||
```
|
||
- `width` **is** the size control: `setPlacement({ width })` already rescales the
|
||
quad while keeping the angular text height consistent with the texture aspect.
|
||
- All of `setPlacement` / `setOpacity` / `resetPlacement` stay live during a session.
|
||
|
||
## 20-controls.js — additions
|
||
|
||
The expanded control bar gains a subtitle row with large −/+ buttons:
|
||
|
||
| Action returned by `hitTest` | Meaning |
|
||
| --- | --- |
|
||
| `sub_smaller` / `sub_bigger` | width by ±0.15 m, clamped to [0.8, 4.0] |
|
||
| `sub_opaque_down` / `sub_opaque_up` | opacity by ∓0.05, clamped to [0.1, 1.0] |
|
||
| `sub_reset` | restore all subtitle defaults |
|
||
|
||
- Add `view.sub = { width, opacity }` so `draw()` can render the current values.
|
||
- Keep hit targets generous; the rows may overlap visually but must not overlap
|
||
in hit-test order (buttons before track).
|
||
- Add a new expanded height that fits the extra row.
|
||
|
||
## 90-main.js — persistence
|
||
|
||
```
|
||
localStorage key: 'jfvr.settings.v1'
|
||
{ subPlacement: {y,z,width}, subOpacity, subMode, subAlpha,
|
||
projection, stereo, ctlAutoHideMs }
|
||
```
|
||
Load at boot (merge over defaults, validate ranges), save debounced ~400 ms after
|
||
any change. A corrupt/absent value must fall back to defaults silently.
|
||
|
||
---
|
||
|
||
# ADDENDUM 5 — gaze pointing + lazy-follow panel + panel size
|
||
|
||
## Why (measured on device)
|
||
|
||
The hand's `targetRaySpace` is an **emulated** ray (Meta documents this). Measured
|
||
on a real Quest 3 while the panel was straight ahead:
|
||
|
||
```
|
||
hand:right origin=(-0.317,-0.054,-0.248) dir=(-0.874, 0.461, -0.154)
|
||
^ 87 % sideways, 15 % forward
|
||
```
|
||
|
||
Aiming with it lands somewhere the user is not looking. Gaze + pinch is Meta's
|
||
documented recommendation for hands and needs no ray accuracy at all.
|
||
|
||
## Changes
|
||
|
||
| # | Change |
|
||
| --- | --- |
|
||
| 1 | Pointer rays are computed in **LOCAL space** (`__JFVR.refSpace`), not viewer space. |
|
||
| 2 | Hands aim with the **gaze ray** (head forward in local space). No beam - a guide line from the hand would be misleading because the hand is not aiming. The on-panel reticle is the indicator. |
|
||
| 3 | Controllers keep their own `targetRaySpace` ray **and** the laser beam; a controller ray is accurate and a laser is expected. Selected by `pointer.beam`. |
|
||
| 4 | The control panel layer's **space is `refSpace`** and holds a stable world pose. A head-locked panel would always put the view centre on its own centre, so no button could ever be targeted by gaze. |
|
||
| 5 | 90-main.js drives a **lazy follow**: it only re-centres the panel once the head has turned > 26 deg away from it, easing with a 0.45 s time constant. |
|
||
| 6 | `subtitles` is **no longer persisted**. Persisting it meant a session where the user toggled subtitles off came back with the layer never created - which looked like a load failure. Always ON at boot; the toggle is per-session. |
|
||
| 7 | Subtitle **distance** range extended: the near clamp is now **0.5 m** (was 0.7 m). |
|
||
|
||
---
|
||
|
||
# ADDENDUM 6 — subtitle depth vs size decoupling (angular-size lock)
|
||
|
||
## Why (owner requirement)
|
||
|
||
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
|
||
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
|
||
|
||
Eye tracking is **not** exposed to WebXR on Quest (HANDOFF §4), so the strip's depth
|
||
cannot follow the user's gaze automatically. What matters instead is that the DEPTH
|
||
control must not change the strip's apparent size: otherwise pulling the subtitles to
|
||
a nearby object makes the text huge (and, with the head-locked quad, drops it out of
|
||
the lower view). See HANDOFF §8 for the literature this is based on.
|
||
|
||
## Model
|
||
|
||
`state.subLockAngle` (default `true`) + `state.subAngleDeg` (default `69.2`).
|
||
|
||
While **locked**, a depth change holds TWO things:
|
||
|
||
```
|
||
width = 2 * |z| * tan(subAngleDeg / 2) angular width (theta)
|
||
y = (y0 / z0) * z angular height (the y/z ratio)
|
||
```
|
||
|
||
so the strip keeps both its angular size and its screen position — only depth
|
||
changes. The default pair is bit-for-bit the pre-ADDENDUM-6 look (2.0 m at 1.45 m →
|
||
69.2°), so nothing moves for anyone who does not touch the control.
|
||
|
||
While **unlocked** everything behaves exactly as at ADDENDUM 4: `width` and `z` are
|
||
independent ("pull closer = bigger").
|
||
|
||
Ranges: `theta` ∈ [20, 120]°, `z` ∈ [-2.6, -0.35] m (near clamp moved from 0.5 m to
|
||
0.35 m — the whole point is matching very close objects). Under the lock the width
|
||
floor drops to 0.05 m (the angle clamp is the real limit); free mode keeps 0.2 m.
|
||
|
||
## 10-subtitles.js — API additions
|
||
|
||
```js
|
||
setLockAngle(on) / getLockAngle(): boolean
|
||
setAngleDeg(deg) / getAngleDeg(): number // locked angular width, degrees
|
||
getPlacement(): { y, z, width } // the resolved, clamped triple
|
||
```
|
||
|
||
- `setPlacement({ y, z, width })` under the lock:
|
||
- `z` without `y` → depth-only: `y` is re-derived from the held y/z ratio and
|
||
`width` from `subAngleDeg`;
|
||
- `width` without `z` → size change: `subAngleDeg` is re-derived from the requested
|
||
width at the current depth (the derived width then wins);
|
||
- `y` + `z` (grab & drag) → the caller's `y` wins, only `width` is derived.
|
||
- `resetPlacement()` restores `{ y:-0.42, z:-1.45, width:2.0 }` **and**
|
||
`subAngleDeg = 69.2`; the lock flag is left alone.
|
||
- New diagnostics: `stats.subAngleDeg`, `stats.subLockAngle`, `stats.subYOverZ`, and
|
||
`stats.subTextDeg` — the ACTUAL angular height of the drawn subtitle ink, in
|
||
degrees. arXiv 2004.01545 measures preferred angular size (character height) at
|
||
~16–18 dmm ≈ 0.9–1.0° for medium/long text and ~32–40 dmm ≈ 1.8–2.3° for
|
||
titles/headlines; Google's dmm is 1 mm of character height at 1 m. Use
|
||
`subTextDeg` to pick a `subAngleDeg` default from real measurements later; the
|
||
shipped default is deliberately the on-device-proven look.
|
||
|
||
## 20-controls.js — subtitle row is now NINE buttons
|
||
|
||
```
|
||
sub_up sub_down sub_near sub_far sub_smaller sub_bigger sub_opaque_down sub_opaque_up sub_reset
|
||
上 下 近 远 小 大 暗 亮 复位
|
||
```
|
||
|
||
- The vertical-position pair (up/down) and the depth pair (near/far) are the
|
||
ADDENDUM-6 additions.
|
||
- `view.sub` gains `angleDeg`, `lock` and `dist` (= `|z|`). The caption ALWAYS
|
||
leads with the depth in metres — `字幕 距1.45m · 69°` while locked,
|
||
`字幕 距1.45m · 2.0m` while free — because the depth is the number the user has
|
||
to match against the object being watched. Opacity stays right-aligned.
|
||
- **`readSubtitles()` bug fixed**: it only read the module-local `S.subtitlesOn`,
|
||
which `90-main.js` never set (it calls `setVideo(video)` with no `getExtra`), so
|
||
the CC button always rendered as OFF. It now falls back to the shared
|
||
`state.subtitles`. The button also got an unambiguous on/off treatment: an
|
||
accent pill + `字幕开` when on, a slashed CC glyph + `字幕关` when off.
|
||
- Hit pills: pitch 0.10, half width 0.052 (a hair of overlap → no dead zone);
|
||
visual width 0.092; caption font 0.048·h, button label font 0.040·h. The bar
|
||
itself is unchanged (`PANEL_M.expanded` = 0.72 × 0.21 m, canvas 1440 × 420 px).
|
||
|
||
## 90-main.js
|
||
|
||
- Persists `subLockAngle` / `subAngleDeg` in `jfvr.settings.v1`; while locked the
|
||
stored `subPlacement.width` is re-derived on load, so `subAngleDeg` + `z` is what
|
||
restores the user's look.
|
||
- In-page panel: a `距离与大小` toggle (锁定视角尺寸 / 自由), and a `视角宽度`
|
||
slider in degrees shown while locked / the old `宽度` slider in metres while free.
|
||
All sliders re-read their value whenever a setter runs, so VR-bar and grab changes
|
||
show up in an already-open panel.
|
||
- `sub_up` / `sub_down` step `y` by ±0.04 m. `sub_near` / `sub_far` step `z` by
|
||
±0.15 m. `sub_smaller` / `sub_bigger` step `subAngleDeg` by ∓5° while locked,
|
||
else `width` by ∓0.15 m.
|
||
- Grab & drag: while locked, a depth drag also carries `y` by the grab-start y/z
|
||
ratio (vertical hand motion still adds on top), so the strip cannot slide out of
|
||
the lower view when it is pulled close.
|
||
|
||
---
|
||
|
||
# ADDENDUM 7 — summon gating, lock badge, pre-VR panel trim
|
||
|
||
Owner feedback after the ADDENDUM 6 pass:
|
||
|
||
> 控制面版的唤出太灵敏了,只有当指示线指向面板附近时,面板再显示出来即可;在 vr 空间
|
||
> 右侧边缘增加一个锁的图标,当选中上锁后,面板不会启动;在刚唤出面板时,0.5s 内的操作
|
||
> 应该忽略,否则可能误触。在进入"vr 模式"时,字幕相关有很多滑块配置,移除哪些配置,
|
||
> 在没有在真实 vr 空间中的调整,都是不准确的,没必要在进入前指定。视角宽度保持默认即可。
|
||
> 字幕相关只保留开启和关闭即可。"vr 播放"这个按钮,不要挡住原有播放器的选项,现在似乎
|
||
> 挡住了原播放器的全屏按钮;手部指向选项直接移除吧,没人会只使用食指作为指向。
|
||
|
||
## 1. Summon gating (90-main.js)
|
||
|
||
There were **two** independent reasons the bar appeared on *every* pinch, and both
|
||
are gone:
|
||
|
||
| Was | Now |
|
||
| --- | --- |
|
||
| `onSelectStart` called `Controls.show()` unconditionally | listener removed; summoning is decided in `pointerDown()` from the actual aim |
|
||
| `pointerDown()` called `Controls.show()` before checking the hit | `show()` is called only when the ray lands on / near the bar |
|
||
|
||
```
|
||
bar hidden + aim ON the bar -> show(), record barShownAt, return (the summoning press never clicks)
|
||
bar hidden + aim NEAR the bar-> same (padded window, see below)
|
||
bar hidden + aim elsewhere -> nothing
|
||
bar hidden + locked -> nothing
|
||
bar shown + < PANEL_ARM_MS(500) since it appeared -> ignore
|
||
bar shown + aim ON the bar -> show() (extends auto-hide) + handleSelect
|
||
```
|
||
|
||
Barely-visible: the summoning press is *consumed* — it does not also press a button,
|
||
and for 500 ms after the bar appears every activation is ignored. Otherwise the hand
|
||
travelling onto the bar right after the summon fires a random button.
|
||
|
||
The summon window is deliberately wider than the button window: `pointer.hit` (used
|
||
for the cursor and for button hits) uses `panelHit(..., 0.25, 0.6)`, while summoning
|
||
falls back to `panelHit(..., 0.5, 0.9)`. A ray that merely grazes the bar can still
|
||
bring it up; the 500 ms arm is what protects against mis-clicks.
|
||
|
||
Stats: `stats.pressAt` now reads `SUMMON (armed 500ms)`, `IGNORED (arming)`,
|
||
`MISS (not aiming at the panel)` or `MISS (panel locked)`.
|
||
|
||
## 2. Lock badge (20-controls.js + 90-main.js)
|
||
|
||
An **always-visible** small quad (`LOCK_X/Y/Z = 0.86 / -0.34 / -1.45`, `LOCK_M = 0.075`,
|
||
texture 128²) placed in the **same `refSpace`** as the control bar — to the right of
|
||
it. While `state.ctlLock` is true, nothing summons the bar; the badge stays visible
|
||
and hit-testable so it can be unlocked again.
|
||
|
||
Why `refSpace` and not a head-locked HUD corner: on this setup the only pointing
|
||
device is the hand's ray, and a head-locked badge could never be aimed at. "Right-hand
|
||
edge of the VR space" is therefore implemented as "right of the control bar, in the
|
||
same world-locked space".
|
||
|
||
- Unlocked: dim (60 % white) padlock with the shackle open.
|
||
- Locked: accent (`#4ea1ff`) filled padlock, shackle closed, keyhole punched out.
|
||
- Hit window: `panelHit(..., 0.6, 0.6)` ≈ 6.5° across at 1.45 m — the badge itself is
|
||
only ~3°, so it is easy to hit with a coarse hand ray.
|
||
- The lock is **per VR session**: `enterVR()` sets `ctlLock = false`, `onSessionEnd()`
|
||
resets it too. A stuck lock would look exactly like a broken plugin, so it is never
|
||
restored from `localStorage`.
|
||
|
||
New API:
|
||
|
||
```js
|
||
Controls.getLockLayer(): XRLayer | null
|
||
Controls.setLock(on) / getLock(): boolean
|
||
```
|
||
|
||
Stats: `stats.ctlLock`.
|
||
|
||
## 3. Pre-VR settings panel trimmed (90-main.js)
|
||
|
||
Sliders set on a flat screen do not transfer to a headset, so the only subtitle
|
||
setting left before entering VR is **开 / 关**:
|
||
|
||
| Removed | Why |
|
||
| --- | --- |
|
||
| 模式 (hud/sphere/off) | legacy A/B debug toggle |
|
||
| 透明合成, 不透明度 | judge-able only in VR |
|
||
| 垂直位置, 距离, 宽度 | now fully replaced by the in-VR row |
|
||
| 距离与大小 (lock toggle) | the lock is always ON; it is the whole point |
|
||
| 视角宽度 | keeps its default; in-VR `小/大` is the control |
|
||
| 手部指向 | the index-ray mode was removed outright |
|
||
|
||
`视角宽度` / `subAngleDeg` still **persists** — the in-VR `小/大` buttons write it,
|
||
and the panel just no longer offers a second, less accurate way to set it. Same for
|
||
placement / opacity: set in VR, remembered next time.
|
||
|
||
Removed with it: `setSubModeLive()`, `setSubLockLive()`, `handRay()` and every
|
||
`handRayMode` read/write.
|
||
|
||
## 4. "VR 播放" button position (90-main.js CSS)
|
||
|
||
`.jfvr-btn` moved from `bottom: 24px` to `bottom: 120px` — at the viewport's
|
||
bottom-right it sat on top of Jellyfin's own OSD, covering its fullscreen button.
|
||
`.jfvr-panel` moved to `bottom: 180px` and `max-height: calc(100vh - 200px)` so it
|
||
still stacks above the button.
|
||
|
||
---
|
||
|
||
# ADDENDUM 7b — the padlock is a PANEL BUTTON, not its own layer
|
||
|
||
## What was wrong with ADDENDUM 7's badge
|
||
|
||
On device the badge could not be selected while the panel was shown, but could be
|
||
while it was hidden. ADDENDUM 7 had given the padlock **its own quad layer** parked at
|
||
`(0.45, -0.34, -1.45)` — 0.45 m to the right of the panel centre, i.e. 0.09 m to the
|
||
right of the panel's own right edge.
|
||
|
||
Aiming is the binding constraint and it was underestimated: the only pointing device
|
||
here is the hand ray, and the cursor's own acceptance window (`panelHit` padding
|
||
0.25/0.6 on a 0.72 m panel) reaches `x ≈ 0.45` — the panel's right edge. The badge was
|
||
**outside every reachable ray**. Worse, its hit test ran on the *panel's* layer
|
||
geometry, so whether it fired depended on where the panel happened to be, which is
|
||
exactly the "works when hidden, not when shown" report.
|
||
|
||
## What it is now
|
||
|
||
An ordinary button on the panel canvas — the `lock` action, fifth in the media row:
|
||
|
||
```
|
||
expanded: [<<<] [>||] [>>>] [CC] [padlock]
|
||
0.18 0.36 0.54 0.72 0.90 (hit pill half width 0.092)
|
||
```
|
||
|
||
It travels the same code path as every button that already works: cursor → `hitTest`
|
||
→ `handleSelect` → `onAction` → `90-main.js handleControlAction('lock')`.
|
||
|
||
## The "locked" panel mode
|
||
|
||
`show()` picks the mode from the lock state, so locking answers "what does a summon
|
||
look like now?" with a third mode:
|
||
|
||
| mode | size (m) | canvas | drawn / hit-testable |
|
||
| --- | --- | --- | --- |
|
||
| `expanded` | 0.72 × 0.21 | 1440 × 420 | everything |
|
||
| `locked` | 0.26 × 0.10 | 520 × 200 | padlock + 解锁 label only |
|
||
| `collapsed` | 0.60 × 0.05 | 1200 × 100 | (legacy; no longer entered) |
|
||
|
||
`hitTest(nx, ny, w, h, modeHint)` now takes the mode explicitly, because "collapsed" (a
|
||
thin strip) and "locked" (a small square-ish pill) share no aspect-ratio invariant.
|
||
`handleSelect` passes `S.mode`; the aspect-ratio `modeFor()` remains only as a fallback
|
||
for external callers.
|
||
|
||
`hide()` **no longer resets the mode**. The layer therefore keeps the size it will be
|
||
shown at, so the summon window that `90-main.js` ray-casts is the full panel rect
|
||
(0.72 × 0.21 unlocked, 0.26 × 0.10 locked) instead of a 5 cm strip — and re-showing
|
||
never needs a resize pop.
|
||
|
||
## Lifecycle (all of it in 90-main.js `toggleControlsLock`)
|
||
|
||
```
|
||
lock: S.ctlLock = true; Controls.setLock(true) -> mode 'locked'
|
||
Controls.show() -> padlock-only panel, auto-hide restarts
|
||
barShownAt = now
|
||
unlock: S.ctlLock = false; Controls.setLock(false) -> mode 'expanded'
|
||
Controls.show() -> full panel
|
||
barShownAt = now -> 0.5 s ignored, so the unlocking
|
||
press cannot land on a button
|
||
```
|
||
|
||
Summoning while locked is allowed (it is the only way back out): `pointerDown` no longer
|
||
early-returns on `S.ctlLock`, and the summon shows the padlock panel alone.
|
||
|
||
## Also fixed
|
||
|
||
`handleSelect()` used to fall back to `(nx, ny) = (0.5, 0.5)` when there was no pointer
|
||
hit and no reticle — which silently pressed whatever button lives at the centre. It now
|
||
returns `null` and performs nothing.
|
||
|
||
## API
|
||
|
||
```js
|
||
Controls.setLock(on) / getLock(): boolean // setLock also swaps the panel in place
|
||
Controls.hitTest(nx, ny, w, h, mode?) // mode: 'expanded' | 'locked' | 'collapsed'
|
||
Controls.PANEL_M / PANEL_PX // now include .locked
|
||
```
|
||
|
||
Stats: `stats.ctlMode` (`expanded` | `locked`), `stats.ctlLock`,
|
||
`stats.pressAt` (`SUMMON (locked panel)`, `LOCKED`/`UNLOCKED (armed 500ms)`, …).
|
||
|
||
---
|
||
|
||
# ADDENDUM 8 — panning the video window (yaw / pitch about the eye)
|
||
|
||
## Why
|
||
|
||
> 我在观看过程中,有时想调整180度视频的播放位置,让其更适合我的环境,我希望画面可以不是
|
||
> 固定在我的正前方180度,我希望能调整画面内容在整个vr空间中以我眼睛为旋转点进行上下左右
|
||
> 的水平移动(为了解决一些场景需要物理低头才能看到的问题,我不想真的低头去看)
|
||
|
||
The 180° window was hard-wired dead ahead. Scenes staged below the horizon could only
|
||
be watched by physically lowering the head.
|
||
|
||
## Model
|
||
|
||
The video layer is an `XREquirectLayer` with `radius` 0 — the content sits **at
|
||
infinity**. Two consequences drive the whole design:
|
||
|
||
- Only the layer's **orientation** decides which direction is visible. Translating it
|
||
does nothing, and there is no parallax, so the picture does not swim when the head
|
||
moves.
|
||
- So panning the picture is *purely* `layer.transform`'s rotation, which rotates the
|
||
content **about the eye point** — exactly what was asked for. No decode, no texture
|
||
upload, no extra GPU work: 8K HEVC DirectPlay is untouched.
|
||
|
||
Only the video rotates. The panel lives at a fixed pose in `refSpace` and the
|
||
subtitles are head-locked in `viewerSpace`, so **the UI stays in front of the user**
|
||
while the picture moves.
|
||
|
||
```js
|
||
state.videoYaw // radians, + = the picture has been moved RIGHT
|
||
state.videoPitch // radians, + = the picture has been moved UP
|
||
q = qy(-yaw) * qx(pitch) // pitch in the content frame, then yaw about world Y
|
||
layer.transform = XRRigidTransform({0,0,0}, q)
|
||
```
|
||
|
||
Because of that composition the window's aim direction is exactly
|
||
`(cos p·sin yaw, sin p, −cos p·cos yaw)`, so yaw is a pure azimuth and pitch a pure
|
||
elevation — no gimbal weirdness in the useful range.
|
||
|
||
Limits: pitch ±(π/2 − 0.02) — i.e. effectively unlimited, per the owner's
|
||
"看用户拖动幅度来定"; the guard only avoids the degenerate pole. Yaw wraps to
|
||
[−π, π]. **Not persisted**: every VR session starts dead ahead.
|
||
|
||
## Interaction — one affordance, in the panel
|
||
|
||
A **drag strip above the progress bar** (the owner's requested placement), part of the
|
||
panel canvas like every other control:
|
||
|
||
```
|
||
┌─ 画面 → the drag strip (panel y 0.00–0.20) ────────────────────────┐
|
||
│ ⠿ 拖动调整画面 偏航 +12° 俯仰 -8° │
|
||
│ 连点 3 次复位 0° / 0° = 正前方 │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
- **Pinch the strip and drag** → the picture follows the hand 1:1.
|
||
- **Three taps inside 1.6 s** → reset to dead ahead. Taps are counted on RELEASE, so a
|
||
real drag can never be mistaken for one.
|
||
- The right side always shows the live angles, so the state is readable even when you
|
||
are not moving anything.
|
||
|
||
The drag is driven by the **pointer ray's own direction**, not by a position on the
|
||
panel:
|
||
|
||
```
|
||
yaw' = wrap(yaw0 + (rayYaw - rayYaw0))
|
||
pitch' = clamp(pitch0 + (rayPitch - rayPitch0)) rayYaw = atan2(dx, -dz)
|
||
rayPitch = asin(dy)
|
||
```
|
||
|
||
A panel-space cursor would run off the panel's edge after a few degrees of movement;
|
||
ray rotation has no range limit. It also makes the content point under the cursor stay
|
||
under the cursor, which is what "grab the picture" should feel like.
|
||
|
||
`Controls.hold(true/false)` freezes the panel auto-hide for the duration of the drag —
|
||
otherwise the panel would vanish mid-drag because the timer `show()` armed keeps
|
||
running.
|
||
|
||
## Runtime risk and how it is handled
|
||
|
||
> **SUPERSEDED by ADDENDUM 8b** — the probe below compared quaternion components,
|
||
> which is sign-sensitive and produced a false "read-only" verdict; and the gesture is
|
||
> now tap = reset / hold 0.5 s = drag. See the addendum at the end of this file.
|
||
|
||
`XRCompositionLayer.transform` / `XREquirectLayer.transform` are settable in the spec,
|
||
but a runtime may ignore the write. Since the whole interaction depends on it, the
|
||
module probes once per session:
|
||
|
||
```js
|
||
probeVideoAngleMutable(layer) // write a known rotation, read it back
|
||
-> stats.videoRotMode = 'live' | 'rebuild'
|
||
```
|
||
|
||
- `live` — a transform write per frame while dragging. Free.
|
||
- `rebuild` — the media layer is recreated. Far too expensive per frame, so rebuilds
|
||
happen only on **creation / release / reset**; during a drag the state updates and
|
||
the picture catches up when the finger comes up. `stats.videoRebuilds` counts them.
|
||
|
||
## 20-controls.js — changes
|
||
|
||
- The expanded panel grew from **0.72 × 0.21 m to 0.72 × 0.26 m** (canvas 1440 × 420 →
|
||
1440 × 520) to make room for the strip; every row was re-proportioned.
|
||
- New hit band `viewHitY0/Y1 = 0..0.20` returning `{type:'view_drag'}`, spanning the
|
||
panel width. It does not overlap the seek band (0.30–0.52) or the button bands.
|
||
- `view.view = { yawDeg, pitchDeg }` feeds the readout (`drawViewStrip`), and is part
|
||
of the redraw signature.
|
||
- New `hold(on)` for the auto-hide pause; `stats.ctlHold`.
|
||
|
||
## 90-main.js — changes
|
||
|
||
- `state.videoYaw/videoPitch` reset to 0 at session start and at session end.
|
||
- `S.videoLayer` handle (was local), `applyVideoAngle(force)` / `rebuildVideoLayer()`,
|
||
`resetVideoAngle()`, `videoQuat()`, `rayAngles()`, `beginViewDrag/updateViewDrag/
|
||
endViewDrag`, and `viewTaps` for the triple-tap reset.
|
||
- The frame loop calls `updateViewDrag(frame)`; `pointerUp()` calls `endViewDrag()`.
|
||
- `handleControlAction('view_drag')` starts the drag using the press context
|
||
(`lastPressSrc` / `lastPressFrame`) captured in `pointerDown`.
|
||
- Stats: `videoYaw`, `videoPitch`, `videoAngleLive`, `videoRotMode`, `videoRebuilds`,
|
||
`videoReset`, `viewDrag`, `viewTaps`.
|
||
|
||
---
|
||
|
||
# ADDENDUM 8b — live picture pan: two mechanisms, the crash, and the real gesture
|
||
|
||
Owner feedback on ADDENDUM 8:
|
||
|
||
> 触发复位后直接崩溃了;另外,我拖动后,我希望画面立刻就有反应,而不是我松手之后才改变。
|
||
> 按住0.5s后拖动才是拖动,否则算复位
|
||
|
||
## 1. Why the picture only moved on release
|
||
|
||
ADDENDUM 8 decided whether it could write `layer.transform` with this probe:
|
||
|
||
```js
|
||
layer.transform = new XRRigidTransform(origin, q);
|
||
var o = layer.transform.orientation;
|
||
live = Math.abs(o.y - q.y) < 1e-4 && Math.abs(o.w - q.w) < 1e-4;
|
||
```
|
||
|
||
Comparing **quaternion components** is sign-sensitive — `q` and `−q` are the same
|
||
rotation — and a runtime is free to hand back a renormalised quaternion. So the probe
|
||
could report "read-only" for a runtime that was perfectly capable of the write. When it
|
||
did, the drag path took the rebuild branch, and because that branch was throttled to
|
||
"on release only", the picture sat still until the finger came up. That is exactly what
|
||
was observed.
|
||
|
||
The probe now compares the **position**, which is exact and has no sign ambiguity:
|
||
|
||
```js
|
||
layer.transform = new XRRigidTransform({x:0.123, y:-0.456, z:0.789}, identity);
|
||
writable = layer.transform.position matches that marker;
|
||
```
|
||
|
||
## 2. Two mechanisms, chosen automatically
|
||
|
||
```js
|
||
videoRotStrategy() -> 'transform' // preferred: one cheap write per change
|
||
| 'space' // layer.space = refSpace.getOffsetReferenceSpace(q)
|
||
| 'rebuild' // manual escape hatch ("兼容" in the 2D panel)
|
||
```
|
||
|
||
- `transform` — post-creation `layer.transform` writes are honoured.
|
||
- `space` — they are not, so the aim lives in an **offset reference space** instead. A
|
||
layer's pose is recomputed from its space every frame, and `getOffsetReferenceSpace()`
|
||
is the recentering path every WebXR app uses, so this is the more likely of the two to
|
||
be fully implemented.
|
||
- The two are **mutually exclusive**: `createVideoLayer()` must NOT bake the transform in
|
||
while the probe is about to write it, or the aim would be applied twice. Only the
|
||
`rebuild` strategy bakes it in at creation.
|
||
- `layer.transform` is never written on the `space` path, and vice versa.
|
||
|
||
Switching is automatic per session; `stats.videoRotMode` reports what was chosen and
|
||
`stats.videoTransformWritable` reports the raw probe result. The 2D panel has
|
||
「画面旋转: 自动 / 兼容」 to force the rebuild path without a redeploy.
|
||
|
||
## 3. The crash
|
||
|
||
The reset path rebuilt the video layer, and it created the replacement **before**
|
||
destroying the old one. Two `XRMediaBinding` equirect layers bound to the same `<video>`
|
||
element at the same time, plus a second rebuild microseconds later (the release path
|
||
also rebuilt, then the reset rebuilt again), took the browser down.
|
||
|
||
Every rebuild now:
|
||
|
||
1. runs in a `setTimeout` **outside the XR frame callback** (never inline in a frame),
|
||
2. **destroys the old layer first**, then creates the replacement,
|
||
3. is throttled to at most one per `REBUILD_MIN_MS` (120 ms),
|
||
4. never queues more than one pending rebuild.
|
||
|
||
The `space`/`transform` strategies rebuild **not at all**, so in the normal case the
|
||
crash path is not even reachable.
|
||
|
||
Two more robustness fixes found while testing this:
|
||
|
||
- `videoSpaceFor()` used to let a throwing `getOffsetReferenceSpace()` escape. It runs
|
||
inside the XR frame loop, where an exception would stop the loop and freeze the
|
||
session. It is caught now, and the strategy falls back to `rebuild`.
|
||
- The whole per-frame body (`updatePointer` / `pollGamepads` / `updatePanelPose` /
|
||
`updateGrabFrame` / `updateViewDrag` / frame hooks) is wrapped in a try/catch. A single
|
||
bad frame could previously kill the loop: `requestAnimationFrame` is only re-armed at
|
||
the end of the callback, so any throw froze the session permanently.
|
||
|
||
## 4. The real gesture
|
||
|
||
```
|
||
press the drag strip
|
||
├─ release before 500 ms, no movement -> RESET to dead ahead
|
||
├─ held past 500 ms, then moved -> PAN (picture follows the hand)
|
||
└─ held past 500 ms, never moved -> RESET
|
||
```
|
||
|
||
- The pan reference (`a0`) is captured **at the moment the hold threshold passes**, not
|
||
at the press. Otherwise the half second the user spends holding would fling the picture
|
||
the instant dragging armed.
|
||
- A ~1° dead zone keeps hand jitter from turning a reset tap into a tiny pan.
|
||
- The strip now reads `点一下复位 · 按住 0.5s 再拖动` instead of the retired triple-tap hint.
|
||
|
||
`stats.viewDrag` walks through it: `pressed (hold 500ms to pan)` →
|
||
`armed (drag to pan)` → `ended (moved)` / `reset (tap)`.
|
||
|
||
## 5. Verification
|
||
|
||
Three offline suites, all against the real sources (`build.py --check` passes too):
|
||
|
||
`test-ad8` — quaternion + panel geometry: aim for ±yaw/±pitch, unit norm, exact
|
||
azimuth/elevation recovery, yaw wrap, pitch guard, "aimed point stays under the cursor";
|
||
panel bands on the 1440×520 canvas (strip 0–0.20, inert readout 0.20–0.30, seek
|
||
0.30–0.52, media 0.52–0.74, subtitle 0.84–1.0, no dead column); the strip's drawn
|
||
readout; `hold()`.
|
||
|
||
`test-gesture` — the state machine above, driven with a stubbed ray: quick tap resets;
|
||
hold-without-movement resets; movement inside the hold window is ignored (no fling);
|
||
after arming a +10°/+5° ray move yields exactly +10°/+5°; a real pan does not reset; the
|
||
dead zone is respected; pitch cannot pass the guard.
|
||
|
||
`test-mech` — the mechanism: the position-based probe returns true for a writable
|
||
transform and false for an ignored one; the transform strategy writes and schedules no
|
||
rebuild; the space strategy assigns an offset space, leaves the transform alone (no
|
||
double rotation) and schedules no rebuild; unchanged angles reuse the cached space; the
|
||
rebuild strategy queues at most one, and its observed order is
|
||
`removeLayer -> createVideoLayer -> addLayer`; a throwing/absent
|
||
`getOffsetReferenceSpace()` is reported and permanently falls back to `rebuild`.
|
||
|
||
# ADDENDUM 9 — subtitle pipeline survival + normal (mono) subtitles
|
||
|
||
Two independent defects, found on device with the VR session left in its broken state.
|
||
|
||
## 1. The rAF broker lost Jellyfin's subtitle loop at session start
|
||
|
||
`90-main.js` no longer just *forwards* to native `requestAnimationFrame` outside XR. It
|
||
**always** holds callbacks in its own queue and drives them with one of two drivers:
|
||
|
||
- outside XR: a native rAF driver that keeps **exactly one** native callback in flight
|
||
(`nativeRafStep` re-arms *before* draining and cancels itself when the queue empty);
|
||
- inside XR: `video.requestVideoFrameCallback`, plus a 50 ms idle fallback so a paused /
|
||
stalled / ended video still redraws (paused seek, subtitle track switch).
|
||
|
||
Why: `requestSession('immersive-vr')` stops native rAF completely. Jellyfin 12.x renders
|
||
subtitles with SubtitlesOctopus, whose main thread starts **one long-lived rAF loop**
|
||
(`rafId`) when the video loads, draws the canvas from it and requests the next prerender
|
||
from the worker (`oneshot-render`). A callback still pending in the *native* scheduler when
|
||
the session starts is destroyed with it, and because `rafId` stays non-zero the loop is
|
||
never restarted.
|
||
|
||
Evidence (live instance, via CDP closure inspection):
|
||
|
||
```
|
||
rafId : 1150 <- native id, not the plugin's 1e9 range
|
||
oneshotState.eventStart: 945.99 <- canvas frozen ~7 min behind the playhead
|
||
renderRequested : false
|
||
oneshot-render requests during 15 s of playback: 0
|
||
```
|
||
|
||
After the fix, in a live VR session: `rafId = 1000028035`, `eventStart` follows
|
||
`video.currentTime`, and `stats.subUploads` climbs while subtitles change.
|
||
|
||
`startRafPump()` must **not** clear the queue (that callback is the one being rescued) and
|
||
must drop the stale `nativeId`. `stopRafPump()` re-arms the native driver.
|
||
|
||
## 2. Auto dual-eye / mono subtitle layout
|
||
|
||
`10-subtitles.js` used to assume the libass canvas is a stereo left/right pair and always
|
||
sampled the **left half**. A normal subtitle (e.g. the `CN` SRT track) is centred across the
|
||
whole 2:1 frame, so half of it was cut off.
|
||
|
||
`state.subLayout` (`'auto' | 'dual' | 'mono'`, default `'auto'`, persisted) selects the
|
||
region; `stats.subLayout / subDual / subEyeW / subCopyND / subCenterInk` report it.
|
||
|
||
Per sample (on the 320×160 downsample) the module builds the ink box and column count of
|
||
each half plus the ink in a ±40 source-px band around the seam, then:
|
||
|
||
```
|
||
only the left half has ink -> dual (sample the left half)
|
||
only the right half has ink -> mono (sample the whole width)
|
||
ink crosses the seam -> mono
|
||
both halves, seam empty -> columnCopyND < COPY_TOL ? dual : mono
|
||
```
|
||
|
||
**Dual-eye is NOT a pixel mirror.** Both eyes carry the *same* text at mirrored *positions*;
|
||
the glyphs are not flipped, so `left[x]` vs `right[W-1-x]` fails badly (measured mean delta
|
||
258/1020 → "not a mirror"). What is true is that the two halves' ink **column profiles**
|
||
match under a small horizontal shift, because they are the same line. Measured on device:
|
||
|
||
| layout | centre-band ink | column-copy distance |
|
||
|---|---|---|
|
||
| dual-eye (`CN_dual_eye`) | 0 | 0.02 – 0.08 |
|
||
| mono (`CN` SRT) | 50 / frame | 1.00 |
|
||
|
||
`COPY_TOL = 0.25`. A layout change is debounced (two consecutive samples) because it
|
||
rebuilds the layer.
|
||
|
||
`M.eyeW` is one eye's region width (half for dual, full for mono); `computeTexDims()`,
|
||
`drawHud()`'s crop/clamp and the texture scale all use it instead of the old hard-coded
|
||
`halfW`.
|
||
|
||
### API additions
|
||
|
||
```js
|
||
__JFVR.Subtitles.setLayout('auto' | 'dual' | 'mono') // also state.subLayout
|
||
__JFVR.Subtitles.getLayout(): string
|
||
```
|
||
|
||
## 3. Verification
|
||
|
||
- Broker logic replicated exactly in a Node harness against the built bundle:
|
||
`maxConcurrentNative = 1` over 300 frames with a self-rescheduling loop (the first
|
||
version re-armed during the drain and doubled the driver every frame — that froze the
|
||
page); simulating session start (native callbacks dropped) leaves the loop in the queue
|
||
and it keeps firing from the XR driver.
|
||
- `columnCopyND` extracted from the built bundle and exercised with the on-device
|
||
measurements: dual 0.00–0.08 → `dual`, mono centre-ink → `mono`, no-copy word-gap
|
||
profile 0.41 → `mono`.
|
||
- On device (before the UI became unreachable): the broker change was observed to carry
|
||
SubtitlesOctopus's loop through `immersive-vr` (`rafId` in the plugin's id range,
|
||
`eventStart` tracking, `subUploads` 4 → 7 over 20 s in VR).
|