Injects a 「VR 播放」 button into Jellyfin Web's video player page and plays
VR180 / VR360 stereo content on Meta Quest 3 directly in the browser, with
DirectPlay 8K HEVC, in-VR subtitles and an in-VR control panel.
Modules (src/, concatenated in filename order into dist/jfvr.js):
00-core.js namespace, shared state, GL helpers, layer/frame-hook registry
10-subtitles.js libass canvas -> head-locked quad layer, auto-cropped to ink,
with depth/size decoupling (angular-size lock)
20-controls.js in-VR control panel: progress/seek, transport, subtitle rows,
aiming beam, padlock, pure 2D canvas -> XR quad layer
30-hands.js system hand model (Three.js + generic-hand GLB) via a manually
driven projection layer
90-main.js boot, XR session lifecycle, rAF broker (rAF stops during XR),
input dispatch, settings panel, localStorage persistence
Key on-device findings are documented in HANDOFF.md §6 and README.md addenda:
requestAnimationFrame stops during an XR session, 3:2 pulldown at 90 Hz needs
updateTargetFrameRate(120), libass must be resized before entering VR, a quad's
local -Z is its plane normal, and hand aiming must use the gaze ray because the
emulated targetRaySpace points sideways. Quest does not expose eye tracking.
Docs: INTERFACES.md (module contract + addenda), README.md (measurements),
HANDOFF.md (handover notes). Credentials are NOT in the repo: deploy.py reads
the SSH password from the JF_PW environment variable.
735 lines
33 KiB
Markdown
735 lines
33 KiB
Markdown
# JFVR module interfaces (contract)
|
||
|
||
Everything lives under a single global namespace `window.__JFVR`, created by
|
||
`src/00-core.js` before any other module runs. Modules are plain
|
||
`(function(){ ... })()` IIFEs that attach themselves to `__JFVR`.
|
||
|
||
Load order (build script concatenates in this order):
|
||
|
||
```
|
||
00-core.js -> __JFVR core: state, helpers, layer/frame-hook registry
|
||
10-subtitles.js -> __JFVR.Subtitles
|
||
20-controls.js -> __JFVR.Controls
|
||
90-main.js -> boot, DOM scan, XR session lifecycle, settings panel, wiring
|
||
```
|
||
|
||
Nothing may read `document` / `navigator.xr` at load time except `00-core.js`
|
||
and `90-main.js`. The two module files must only *define* functions and are
|
||
called later with a session/frame.
|
||
|
||
---
|
||
|
||
## 00-core.js — always available as `window.__JFVR`
|
||
|
||
```js
|
||
__JFVR.state // shared mutable state object (see below)
|
||
__JFVR.log(...args) // console.log with [JFVR] prefix + colour
|
||
__JFVR.version // string
|
||
|
||
// --- helpers ---
|
||
__JFVR.makeEl(tag, className?, text?) // document.createElement wrapper
|
||
__JFVR.compileShader(gl, type, source) // throws with the info log on failure
|
||
__JFVR.invert4(mat4ArrayLike) // -> Float32Array(16)
|
||
__JFVR.QUAD_VS // vertex shader source: fullscreen triangle,
|
||
// `in vec2 aPos; out vec2 vUv;`
|
||
// vUv = vec2(aPos.x*.5+.5, .5 - aPos.y*.5)
|
||
__JFVR.bindFullscreenTriangle(gl, program) // creates the VBO, enables aPos, returns attribLoc
|
||
|
||
// --- XR layer registry (owned by 90-main.js) ---
|
||
__JFVR.addLayer(layer) // register + push updateRenderState if a session is live
|
||
__JFVR.removeLayer(layer) // unregister + push
|
||
__JFVR.addFrameHook(fn) // fn(frame, time, session) every XR frame
|
||
__JFVR.removeFrameHook(fn)
|
||
|
||
// --- capabilities (filled by 00-core.js at boot) ---
|
||
__JFVR.caps = {
|
||
secureContext, hasXR, immersiveVR, hasXRMediaBinding, hasEquirectLayer,
|
||
hasXRWebGLBinding, hasXRHand, hasXRJointSpace, maxTextureSize,
|
||
hevc: { ... }, ua
|
||
}
|
||
|
||
// --- live handles ---
|
||
__JFVR.session // current XRSession or null
|
||
__JFVR.video // current <video> element or null
|
||
__JFVR.refSpace // 'local' XRReferenceSpace for the running session
|
||
__JFVR.viewerSpace // 'viewer' XRReferenceSpace (head-locked layers)
|
||
```
|
||
|
||
### `__JFVR.state` fields that modules may use
|
||
|
||
```js
|
||
state.video // <video> or null
|
||
state.session // XRSession or null
|
||
state.refSpace // local space
|
||
state.viewerSpace // viewer space (head-locked)
|
||
state.projection // '180' | '360'
|
||
state.stereo // 'mono' | 'sbs' | 'ou'
|
||
state.subtitles // bool — render subtitles in VR
|
||
state.subMode // 'hud' (head-locked quad, default)
|
||
// 'sphere'(on the equirect layer, legacy)
|
||
state.subAlpha // 'premultiplied' | 'straight'
|
||
state.subPlacement // { y, z, width } metres, hud mode only
|
||
state.stats // diagnostics bag, free to add keys
|
||
state.gl // { refSpace, session } set by main during a session
|
||
```
|
||
|
||
### Layer / frame-hook lifecycle a module must follow
|
||
|
||
1. `create(session, glContextOrNull)` — build GL resources + the XR layer,
|
||
call `__JFVR.addLayer(layer)`, `__JFVR.addFrameHook(hook)`.
|
||
Return the layer, or `null` when it cannot run (missing canvas, missing API…).
|
||
**Never throw** — return null and `__JFVR.log()` the reason.
|
||
2. `hook(frame, time, session)` — per-frame work. Must be cheap; skip work when
|
||
nothing changed.
|
||
3. `destroy()` — remove the frame hook, remove the layer, delete GL resources,
|
||
drop canvas references. Must be safe to call twice and safe to call when
|
||
`create()` returned null.
|
||
|
||
---
|
||
|
||
## 10-subtitles.js — `__JFVR.Subtitles`
|
||
|
||
Renders Jellyfin's libass subtitle canvas into VR.
|
||
|
||
```js
|
||
__JFVR.Subtitles = {
|
||
// Called once at boot, before any session.
|
||
init(): void,
|
||
|
||
// BEFORE entering VR. Jellyfin sizes the libass canvas to the on-screen video
|
||
// box (~1250x625), which is far too small for VR. Blowing up the player box
|
||
// makes libass re-render bigger. Must be called while NOT in an immersive
|
||
// session (libass does not recompute its backing store during XR).
|
||
// Idempotent: calling twice must not double-apply.
|
||
prepare(): void,
|
||
|
||
// Undo prepare(). Safe when prepare() was never called.
|
||
restore(): void,
|
||
|
||
// True when a libass canvas exists and has non-zero size.
|
||
isReady(): boolean,
|
||
|
||
// Current libass canvas size, for diagnostics: { w, h } or null
|
||
canvasSize(): { w, h } | null,
|
||
|
||
// --- session lifecycle ---
|
||
create(session): XRLayer | null,
|
||
destroy(): void,
|
||
|
||
// --- settings, safe to call before or during a session ---
|
||
setMode(mode), // 'hud' | 'sphere' | 'off'
|
||
setAlphaMode(mode), // 'premultiplied' | 'straight'
|
||
setPlacement({ y, z, width }), // partial update, hud only
|
||
getMode(): string
|
||
};
|
||
```
|
||
|
||
### HUD mode requirements (the default, and the point of this module)
|
||
|
||
- A **head-locked quad layer** in `__JFVR.viewerSpace`
|
||
(`XRWebGLBinding.createQuadLayer({ space: viewerSpace, layout: 'mono', ... })`).
|
||
- Source pixels come from `canvas.libassjs-canvas`. That canvas holds a
|
||
**stereo left-right** image: the left half is the `LeftEye` subtitle, the
|
||
right half the `RightEye` one. For a mono HUD use **one half only**
|
||
(the left half is correct for this content).
|
||
- **Auto-crop to the ink bounding box** so the strip is tight and the text is
|
||
as large as possible. Recompute the bbox at a low rate (≈4 Hz) on a
|
||
downscaled copy — never `getImageData` the full-resolution canvas every frame
|
||
(a 4320x2160 full read is ~37 MB and has already crashed a renderer once).
|
||
Add hysteresis / smoothing so the strip does not jitter between frames.
|
||
- Upload only when the source changed (or at most ~30 Hz) — the earlier
|
||
implementation burned a 4K texture upload every frame.
|
||
- Target ≈30 px/degree. A 2.0 m wide strip at 1.45 m is ≈66° → texture width
|
||
2048 is comfortable on Quest 3.
|
||
- Respect `state.subAlpha`:
|
||
- `'premultiplied'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true)`
|
||
- `'straight'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false)`
|
||
- Draw with `XRWebGLBinding.getSubImage(layer, frame)` → bind
|
||
`sub.colorTexture` to a framebuffer → viewport to
|
||
`sub.colorTextureWidth/Height` → clear to `(0,0,0,0)` → draw.
|
||
`getSubImage` takes **two** arguments.
|
||
|
||
### `prepare()` detail
|
||
|
||
The measured behaviour: setting the player box to `7680x3840` makes libass
|
||
produce a **4320x2160** canvas (it does not go higher). The previous bug was
|
||
that this was only applied on "open panel" and then restored on session end
|
||
while the panel stayed open, so a re-entry got a 1250x625 canvas.
|
||
`prepare()` must therefore be idempotent and re-callable.
|
||
|
||
Target DOM: `document.querySelector('.videoPlayerContainer')` and the `<video>`.
|
||
Save/restore their inline `style.cssText` exactly.
|
||
|
||
---
|
||
|
||
## 20-controls.js — `__JFVR.Controls`
|
||
|
||
A head-locked control strip: progress bar + seek, play/pause, skip ±10 s,
|
||
subtitle toggle. **Pure 2D canvas rendering + a hit-test function, plus a thin
|
||
XR quad layer wrapper.**
|
||
|
||
```js
|
||
__JFVR.Controls = {
|
||
init(): void,
|
||
|
||
// --- pure rendering, no WebXR, must be unit-testable in isolation ---
|
||
// Draws the bar into `ctx` sized `w` x `h` (CSS pixels).
|
||
// `view` = {
|
||
// playing, currentTime, duration, buffered, subtitlesOn,
|
||
// mode: 'collapsed' | 'expanded',
|
||
// hover: null | 'playpause' | 'back10' | 'fwd10' | 'subs' | 'track'
|
||
// }
|
||
draw(ctx, w, h, view): void,
|
||
|
||
// Maps a point in normalized panel coords (0..1, origin top-left) to an action.
|
||
// Returns one of:
|
||
// { type: 'seek', value: 0..1 }
|
||
// { type: 'playpause' | 'back10' | 'fwd10' | 'subs' }
|
||
// null
|
||
hitTest(nx, ny, w, h): action | null,
|
||
|
||
// --- XR lifecycle ---
|
||
create(session): XRLayer | null,
|
||
destroy(): void,
|
||
|
||
// --- state / input ---
|
||
setVideo(videoEl, getExtra | null): void, // main provides time/duration/buffered readers
|
||
onAction(cb): void, // cb(action) — main performs the media op
|
||
show(autoHideMs?), hide(): void, // collapsed <-> expanded + auto-hide
|
||
isVisible(): boolean,
|
||
handleSelect(inputSource, frame): void // called on XR `select`
|
||
};
|
||
```
|
||
|
||
### Panel design requirements
|
||
|
||
- **Two visual states**: `collapsed` = a slim progress bar only (≈0.9 m x 0.07 m,
|
||
centre-bottom of view). `expanded` = progress row + buttons
|
||
(≈1.1 m x 0.22 m).
|
||
- Hit targets must be **generous**: the progress bar should be visually ~1 cm
|
||
tall but hit-testable over ~3–4 cm. Meta's docs warn that the emulated hand
|
||
target ray for hand tracking is only a coarse forward pointer, so **do not
|
||
rely on pixel-precise aiming**.
|
||
- Colours must read on top of arbitrary video: dark translucent pill background,
|
||
white text, accent `#4ea1ff` (matches the existing in-page panel).
|
||
- Everything must be drawn from scratch on a canvas — no DOM, no images, no
|
||
external fonts (use a generic stack: `'system-ui, -apple-system, sans-serif'`).
|
||
- `draw()` must be deterministic and side-effect free (no timers inside).
|
||
- `hitTest` must be a pure geometry function of `(nx, ny, w, h)`.
|
||
|
||
### Input model (main.js supplies)
|
||
|
||
- **Gaze**: the panel is head-locked, so the reticle is the view centre.
|
||
`main.js` converts that into `(nx, ny)` and calls `hitTest`.
|
||
- **Controller ray**: `main.js` intersects the controller ray with the panel
|
||
plane and converts to `(nx, ny)`.
|
||
- Both end in `handleSelect()` → the action from `hitTest` → `onAction` callback.
|
||
|
||
---
|
||
|
||
## 90-main.js — owned by the integrator
|
||
|
||
Boot, DOM scan, the in-page `VR 播放` button + settings panel, the XR session
|
||
lifecycle, the rAF broker (see below), input dispatch, and the settings UI for
|
||
subtitle mode / alpha mode / placement.
|
||
|
||
### rAF broker (already proven, keep it)
|
||
|
||
While an immersive XR session is running `window.requestAnimationFrame` is
|
||
**completely stopped** (measured: 0 callbacks in 6 s), and Jellyfin renders
|
||
libass subtitles from an rAF loop. `90-main.js` intercepts
|
||
`requestAnimationFrame`/`cancelAnimationFrame`, holds callbacks while in XR and
|
||
drives them from `video.requestVideoFrameCallback`. Outside XR it delegates to
|
||
the native implementation.
|
||
|
||
---
|
||
|
||
# ADDENDUM 2 — in-VR UX pass (after first on-device test)
|
||
|
||
Owner feedback from the first headset test:
|
||
|
||
1. **The control bar is permanently visible — wrong.** It must be hidden by
|
||
default and only appear on interaction, then auto-hide.
|
||
2. **Hand/controller input has no guide line.** There is no visible ray and no
|
||
cursor, so the user cannot tell where they are aiming.
|
||
3. **Subtitles must sit below the visual centre** by default, and both the
|
||
**position and the opacity must be user-adjustable**.
|
||
|
||
## New shared state (owned by 90-main.js, read by modules)
|
||
|
||
```js
|
||
state.subOpacity // 0..1, default 0.85
|
||
state.subPlacement // { y, z, width } — y is the quad CENTRE, negative = below eye level
|
||
state.ctlAutoHideMs // default 4000
|
||
state.pointer // { visible, source, origin:{x,y,z}, dir:{x,y,z}, hit:{nx,ny}|null }
|
||
// written every XR frame by 90-main.js; modules must treat it as read-only
|
||
```
|
||
|
||
## 10-subtitles.js — additions
|
||
|
||
```js
|
||
setOpacity(a) // 0..1; applied to the XR layer (layer.opacity or the
|
||
// shader's alpha), NOT by rebuilding the layer
|
||
getOpacity(): number
|
||
```
|
||
- Default `state.subPlacement` becomes `{ y: -0.42, z: -1.45, width: 2.0 }`.
|
||
- `setPlacement()` must take effect live (update `layer.transform` if the runtime
|
||
allows mutating it, otherwise rebuild just this layer and keep the frame hook).
|
||
- Keep the existing tight-crop behaviour and the 4320x2160 canvas preparation.
|
||
|
||
## 20-controls.js — additions
|
||
|
||
```js
|
||
setPointer(p) // p as in state.pointer; called every frame by main.
|
||
// Draw a beam from origin along dir and, when hit != null, a
|
||
// cursor dot at (hit.nx, hit.ny) on the panel.
|
||
setOpacity(a)
|
||
getOpacity(): number
|
||
```
|
||
|
||
- **Default state is HIDDEN.** `create()` must not make the bar visible.
|
||
- `show(autoHideMs)` → visible + expanded, auto-hides after `autoHideMs`
|
||
(default `state.ctlAutoHideMs`).
|
||
- `hide()` → fully hidden (no bar, no beam, no cursor) until `show()` again.
|
||
- `isVisible()` → true only while shown.
|
||
- The beam must be a **separate, always-updated visual** that is only visible
|
||
while the bar is shown. Implement it as a thin quad layer whose transform is
|
||
placed at `origin + dir * (L/2)` and oriented so its local -Z follows `dir`
|
||
(build the quaternion from the direction; L = distance to the hit, or a fixed
|
||
1.2 m when there is no hit). Height ≈ 0.006 m, width = L, with a soft gradient
|
||
texture (bright at the origin end, fading out) so it reads as a laser.
|
||
- The cursor: when `hit` is non-null, draw a ring/dot on the control panel canvas
|
||
at `(hit.nx, hit.ny)` so the user sees exactly what will be activated.
|
||
- Do not re-upload either texture more than ~30 Hz; skip everything when hidden.
|
||
- Add stats keys `ctlPointer`, `ctlHidden`, `ctlBeamOn`.
|
||
|
||
## Input model change (90-main.js) — unified pointing
|
||
|
||
Hand tracking and controllers both expose `targetRaySpace`. Use one code path:
|
||
|
||
- For **every** input source with `targetRayMode === 'tracked-pointer'`
|
||
(controllers *and* hands), intersect its ray with the control quad plane and
|
||
produce `hit`.
|
||
- If no source produces a hit, fall back to gaze: `hit = {nx:0.5, ny:0.5}` centred.
|
||
- `state.pointer.source` records which one won.
|
||
- Show the beam for the winning source; hide it when the controls are hidden.
|
||
|
||
Meta's docs note the hand target ray is an *emulated* forward-pointing ray, so
|
||
hit targets stay generous — that is already the case.
|
||
|
||
### Beam orientation — CORRECTED convention (my first spec was wrong)
|
||
|
||
Do **not** map the quad's local `-Z` onto the ray direction. A quad's local `-Z`
|
||
is its plane **normal**, so that would put the quad's plane perpendicular to the
|
||
ray and lay its `width` extent across the beam — a thin face-on bar, not a beam.
|
||
|
||
Correct:
|
||
|
||
```
|
||
position = origin + dir * (L / 2) // L = hit distance, else 1.2 m
|
||
width = L (the long axis)
|
||
height = 0.006
|
||
local +X -> dir // the beam axis lies IN the plane
|
||
local -Z -> perpendicular to dir, biased up so the ribbon is never edge-on
|
||
gradient -> bright at the origin end, fading to transparent at the hit end
|
||
```
|
||
|
||
Assertion for tests: `R(q)·(1,0,0) ≈ dir` and `R(q)·(0,0,-1) ⟂ dir`.
|
||
|
||
---
|
||
|
||
# ADDENDUM 4 — in-VR subtitle placement (grab & drag)
|
||
|
||
Owner requirement: adjust the subtitles **without leaving VR**. Chosen model:
|
||
|
||
- **Position**: grab the subtitle strip with the laser and drag it in space.
|
||
- **Reset** button restores the defaults.
|
||
- **Size** and **opacity** also adjustable in VR.
|
||
- Values **persist in localStorage** (per browser, no server changes).
|
||
|
||
## Grab & drag model (owned by 90-main.js)
|
||
|
||
The strip is a head-locked quad in viewer space, so both the ray and the strip
|
||
live in the same space — no world-space math needed.
|
||
|
||
```
|
||
on grab start:
|
||
grabActive = true
|
||
grabOrigin0 = ray.origin // viewer space
|
||
grabHit0 = ray ∩ plane(z = placement.z)
|
||
grabPlace0 = { y, z, width }
|
||
|
||
per frame while grabbed:
|
||
P = currentRay ∩ plane(z = grabPlace0.z) // fixed reference plane
|
||
y' = grabPlace0.y + (P.y - grabHit0.y) // drag up/down
|
||
z' = grabPlace0.z + (ray.origin.z - grabOrigin0.z) // pull toward / push away
|
||
clamp y' to [-0.95, -0.05], z' to [-2.6, -0.7]
|
||
Subtitles.setPlacement({ y: y', z: z', width: current })
|
||
|
||
on grab end: clear grab state, persist
|
||
```
|
||
|
||
Rationale for `z`: in viewer space forward is `-Z`, so pulling the hand back
|
||
(increasing `origin.z`) makes `z'` less negative, i.e. the strip comes **closer**
|
||
— which matches how pulling a real object behaves.
|
||
|
||
Grab **must not** steal the normal control-bar interaction: only start a grab when
|
||
the ray actually hits the subtitle quad; otherwise fall through to the bar.
|
||
|
||
## Input: press **and release** are now needed
|
||
|
||
The existing activation path in `90-main.js` only detects a rising edge. Grabbing
|
||
needs the held state too, so track per input source:
|
||
|
||
- `selectstart` / `selectend` session events, **and**
|
||
- `inputSource.gamepad` button press/release (hand pinch; see ADDENDUM 3)
|
||
|
||
Both feed one `pointerDown(src)` / `pointerUp(src)` pair, deduped the same way.
|
||
|
||
## 10-subtitles.js — additions
|
||
|
||
```js
|
||
getLayer(): XRLayer | null // so main can ray-cast against the strip
|
||
resetPlacement(): void // back to { y: -0.42, z: -1.45, width: 2.0 }
|
||
```
|
||
- `width` **is** the size control: `setPlacement({ width })` already rescales the
|
||
quad while keeping the angular text height consistent with the texture aspect.
|
||
- All of `setPlacement` / `setOpacity` / `resetPlacement` stay live during a session.
|
||
|
||
## 20-controls.js — additions
|
||
|
||
The expanded control bar gains a subtitle row with large −/+ buttons:
|
||
|
||
| Action returned by `hitTest` | Meaning |
|
||
| --- | --- |
|
||
| `sub_smaller` / `sub_bigger` | width by ±0.15 m, clamped to [0.8, 4.0] |
|
||
| `sub_opaque_down` / `sub_opaque_up` | opacity by ∓0.05, clamped to [0.1, 1.0] |
|
||
| `sub_reset` | restore all subtitle defaults |
|
||
|
||
- Add `view.sub = { width, opacity }` so `draw()` can render the current values.
|
||
- Keep hit targets generous; the rows may overlap visually but must not overlap
|
||
in hit-test order (buttons before track).
|
||
- Add a new expanded height that fits the extra row.
|
||
|
||
## 90-main.js — persistence
|
||
|
||
```
|
||
localStorage key: 'jfvr.settings.v1'
|
||
{ subPlacement: {y,z,width}, subOpacity, subMode, subAlpha,
|
||
projection, stereo, ctlAutoHideMs }
|
||
```
|
||
Load at boot (merge over defaults, validate ranges), save debounced ~400 ms after
|
||
any change. A corrupt/absent value must fall back to defaults silently.
|
||
|
||
---
|
||
|
||
# ADDENDUM 5 — gaze pointing + lazy-follow panel + panel size
|
||
|
||
## Why (measured on device)
|
||
|
||
The hand's `targetRaySpace` is an **emulated** ray (Meta documents this). Measured
|
||
on a real Quest 3 while the panel was straight ahead:
|
||
|
||
```
|
||
hand:right origin=(-0.317,-0.054,-0.248) dir=(-0.874, 0.461, -0.154)
|
||
^ 87 % sideways, 15 % forward
|
||
```
|
||
|
||
Aiming with it lands somewhere the user is not looking. Gaze + pinch is Meta's
|
||
documented recommendation for hands and needs no ray accuracy at all.
|
||
|
||
## Changes
|
||
|
||
| # | Change |
|
||
| --- | --- |
|
||
| 1 | Pointer rays are computed in **LOCAL space** (`__JFVR.refSpace`), not viewer space. |
|
||
| 2 | Hands aim with the **gaze ray** (head forward in local space). No beam - a guide line from the hand would be misleading because the hand is not aiming. The on-panel reticle is the indicator. |
|
||
| 3 | Controllers keep their own `targetRaySpace` ray **and** the laser beam; a controller ray is accurate and a laser is expected. Selected by `pointer.beam`. |
|
||
| 4 | The control panel layer's **space is `refSpace`** and holds a stable world pose. A head-locked panel would always put the view centre on its own centre, so no button could ever be targeted by gaze. |
|
||
| 5 | 90-main.js drives a **lazy follow**: it only re-centres the panel once the head has turned > 26 deg away from it, easing with a 0.45 s time constant. |
|
||
| 6 | `subtitles` is **no longer persisted**. Persisting it meant a session where the user toggled subtitles off came back with the layer never created - which looked like a load failure. Always ON at boot; the toggle is per-session. |
|
||
| 7 | Subtitle **distance** range extended: the near clamp is now **0.5 m** (was 0.7 m). |
|
||
|
||
---
|
||
|
||
# ADDENDUM 6 — subtitle depth vs size decoupling (angular-size lock)
|
||
|
||
## Why (owner requirement)
|
||
|
||
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
|
||
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
|
||
|
||
Eye tracking is **not** exposed to WebXR on Quest (HANDOFF §4), so the strip's depth
|
||
cannot follow the user's gaze automatically. What matters instead is that the DEPTH
|
||
control must not change the strip's apparent size: otherwise pulling the subtitles to
|
||
a nearby object makes the text huge (and, with the head-locked quad, drops it out of
|
||
the lower view). See HANDOFF §8 for the literature this is based on.
|
||
|
||
## Model
|
||
|
||
`state.subLockAngle` (default `true`) + `state.subAngleDeg` (default `69.2`).
|
||
|
||
While **locked**, a depth change holds TWO things:
|
||
|
||
```
|
||
width = 2 * |z| * tan(subAngleDeg / 2) angular width (theta)
|
||
y = (y0 / z0) * z angular height (the y/z ratio)
|
||
```
|
||
|
||
so the strip keeps both its angular size and its screen position — only depth
|
||
changes. The default pair is bit-for-bit the pre-ADDENDUM-6 look (2.0 m at 1.45 m →
|
||
69.2°), so nothing moves for anyone who does not touch the control.
|
||
|
||
While **unlocked** everything behaves exactly as at ADDENDUM 4: `width` and `z` are
|
||
independent ("pull closer = bigger").
|
||
|
||
Ranges: `theta` ∈ [20, 120]°, `z` ∈ [-2.6, -0.35] m (near clamp moved from 0.5 m to
|
||
0.35 m — the whole point is matching very close objects). Under the lock the width
|
||
floor drops to 0.05 m (the angle clamp is the real limit); free mode keeps 0.2 m.
|
||
|
||
## 10-subtitles.js — API additions
|
||
|
||
```js
|
||
setLockAngle(on) / getLockAngle(): boolean
|
||
setAngleDeg(deg) / getAngleDeg(): number // locked angular width, degrees
|
||
getPlacement(): { y, z, width } // the resolved, clamped triple
|
||
```
|
||
|
||
- `setPlacement({ y, z, width })` under the lock:
|
||
- `z` without `y` → depth-only: `y` is re-derived from the held y/z ratio and
|
||
`width` from `subAngleDeg`;
|
||
- `width` without `z` → size change: `subAngleDeg` is re-derived from the requested
|
||
width at the current depth (the derived width then wins);
|
||
- `y` + `z` (grab & drag) → the caller's `y` wins, only `width` is derived.
|
||
- `resetPlacement()` restores `{ y:-0.42, z:-1.45, width:2.0 }` **and**
|
||
`subAngleDeg = 69.2`; the lock flag is left alone.
|
||
- New diagnostics: `stats.subAngleDeg`, `stats.subLockAngle`, `stats.subYOverZ`, and
|
||
`stats.subTextDeg` — the ACTUAL angular height of the drawn subtitle ink, in
|
||
degrees. arXiv 2004.01545 measures preferred angular size (character height) at
|
||
~16–18 dmm ≈ 0.9–1.0° for medium/long text and ~32–40 dmm ≈ 1.8–2.3° for
|
||
titles/headlines; Google's dmm is 1 mm of character height at 1 m. Use
|
||
`subTextDeg` to pick a `subAngleDeg` default from real measurements later; the
|
||
shipped default is deliberately the on-device-proven look.
|
||
|
||
## 20-controls.js — subtitle row is now NINE buttons
|
||
|
||
```
|
||
sub_up sub_down sub_near sub_far sub_smaller sub_bigger sub_opaque_down sub_opaque_up sub_reset
|
||
上 下 近 远 小 大 暗 亮 复位
|
||
```
|
||
|
||
- The vertical-position pair (up/down) and the depth pair (near/far) are the
|
||
ADDENDUM-6 additions.
|
||
- `view.sub` gains `angleDeg`, `lock` and `dist` (= `|z|`). The caption ALWAYS
|
||
leads with the depth in metres — `字幕 距1.45m · 69°` while locked,
|
||
`字幕 距1.45m · 2.0m` while free — because the depth is the number the user has
|
||
to match against the object being watched. Opacity stays right-aligned.
|
||
- **`readSubtitles()` bug fixed**: it only read the module-local `S.subtitlesOn`,
|
||
which `90-main.js` never set (it calls `setVideo(video)` with no `getExtra`), so
|
||
the CC button always rendered as OFF. It now falls back to the shared
|
||
`state.subtitles`. The button also got an unambiguous on/off treatment: an
|
||
accent pill + `字幕开` when on, a slashed CC glyph + `字幕关` when off.
|
||
- Hit pills: pitch 0.10, half width 0.052 (a hair of overlap → no dead zone);
|
||
visual width 0.092; caption font 0.048·h, button label font 0.040·h. The bar
|
||
itself is unchanged (`PANEL_M.expanded` = 0.72 × 0.21 m, canvas 1440 × 420 px).
|
||
|
||
## 90-main.js
|
||
|
||
- Persists `subLockAngle` / `subAngleDeg` in `jfvr.settings.v1`; while locked the
|
||
stored `subPlacement.width` is re-derived on load, so `subAngleDeg` + `z` is what
|
||
restores the user's look.
|
||
- In-page panel: a `距离与大小` toggle (锁定视角尺寸 / 自由), and a `视角宽度`
|
||
slider in degrees shown while locked / the old `宽度` slider in metres while free.
|
||
All sliders re-read their value whenever a setter runs, so VR-bar and grab changes
|
||
show up in an already-open panel.
|
||
- `sub_up` / `sub_down` step `y` by ±0.04 m. `sub_near` / `sub_far` step `z` by
|
||
±0.15 m. `sub_smaller` / `sub_bigger` step `subAngleDeg` by ∓5° while locked,
|
||
else `width` by ∓0.15 m.
|
||
- Grab & drag: while locked, a depth drag also carries `y` by the grab-start y/z
|
||
ratio (vertical hand motion still adds on top), so the strip cannot slide out of
|
||
the lower view when it is pulled close.
|
||
|
||
---
|
||
|
||
# ADDENDUM 7 — summon gating, lock badge, pre-VR panel trim
|
||
|
||
Owner feedback after the ADDENDUM 6 pass:
|
||
|
||
> 控制面版的唤出太灵敏了,只有当指示线指向面板附近时,面板再显示出来即可;在 vr 空间
|
||
> 右侧边缘增加一个锁的图标,当选中上锁后,面板不会启动;在刚唤出面板时,0.5s 内的操作
|
||
> 应该忽略,否则可能误触。在进入"vr 模式"时,字幕相关有很多滑块配置,移除哪些配置,
|
||
> 在没有在真实 vr 空间中的调整,都是不准确的,没必要在进入前指定。视角宽度保持默认即可。
|
||
> 字幕相关只保留开启和关闭即可。"vr 播放"这个按钮,不要挡住原有播放器的选项,现在似乎
|
||
> 挡住了原播放器的全屏按钮;手部指向选项直接移除吧,没人会只使用食指作为指向。
|
||
|
||
## 1. Summon gating (90-main.js)
|
||
|
||
There were **two** independent reasons the bar appeared on *every* pinch, and both
|
||
are gone:
|
||
|
||
| Was | Now |
|
||
| --- | --- |
|
||
| `onSelectStart` called `Controls.show()` unconditionally | listener removed; summoning is decided in `pointerDown()` from the actual aim |
|
||
| `pointerDown()` called `Controls.show()` before checking the hit | `show()` is called only when the ray lands on / near the bar |
|
||
|
||
```
|
||
bar hidden + aim ON the bar -> show(), record barShownAt, return (the summoning press never clicks)
|
||
bar hidden + aim NEAR the bar-> same (padded window, see below)
|
||
bar hidden + aim elsewhere -> nothing
|
||
bar hidden + locked -> nothing
|
||
bar shown + < PANEL_ARM_MS(500) since it appeared -> ignore
|
||
bar shown + aim ON the bar -> show() (extends auto-hide) + handleSelect
|
||
```
|
||
|
||
Barely-visible: the summoning press is *consumed* — it does not also press a button,
|
||
and for 500 ms after the bar appears every activation is ignored. Otherwise the hand
|
||
travelling onto the bar right after the summon fires a random button.
|
||
|
||
The summon window is deliberately wider than the button window: `pointer.hit` (used
|
||
for the cursor and for button hits) uses `panelHit(..., 0.25, 0.6)`, while summoning
|
||
falls back to `panelHit(..., 0.5, 0.9)`. A ray that merely grazes the bar can still
|
||
bring it up; the 500 ms arm is what protects against mis-clicks.
|
||
|
||
Stats: `stats.pressAt` now reads `SUMMON (armed 500ms)`, `IGNORED (arming)`,
|
||
`MISS (not aiming at the panel)` or `MISS (panel locked)`.
|
||
|
||
## 2. Lock badge (20-controls.js + 90-main.js)
|
||
|
||
An **always-visible** small quad (`LOCK_X/Y/Z = 0.86 / -0.34 / -1.45`, `LOCK_M = 0.075`,
|
||
texture 128²) placed in the **same `refSpace`** as the control bar — to the right of
|
||
it. While `state.ctlLock` is true, nothing summons the bar; the badge stays visible
|
||
and hit-testable so it can be unlocked again.
|
||
|
||
Why `refSpace` and not a head-locked HUD corner: on this setup the only pointing
|
||
device is the hand's ray, and a head-locked badge could never be aimed at. "Right-hand
|
||
edge of the VR space" is therefore implemented as "right of the control bar, in the
|
||
same world-locked space".
|
||
|
||
- Unlocked: dim (60 % white) padlock with the shackle open.
|
||
- Locked: accent (`#4ea1ff`) filled padlock, shackle closed, keyhole punched out.
|
||
- Hit window: `panelHit(..., 0.6, 0.6)` ≈ 6.5° across at 1.45 m — the badge itself is
|
||
only ~3°, so it is easy to hit with a coarse hand ray.
|
||
- The lock is **per VR session**: `enterVR()` sets `ctlLock = false`, `onSessionEnd()`
|
||
resets it too. A stuck lock would look exactly like a broken plugin, so it is never
|
||
restored from `localStorage`.
|
||
|
||
New API:
|
||
|
||
```js
|
||
Controls.getLockLayer(): XRLayer | null
|
||
Controls.setLock(on) / getLock(): boolean
|
||
```
|
||
|
||
Stats: `stats.ctlLock`.
|
||
|
||
## 3. Pre-VR settings panel trimmed (90-main.js)
|
||
|
||
Sliders set on a flat screen do not transfer to a headset, so the only subtitle
|
||
setting left before entering VR is **开 / 关**:
|
||
|
||
| Removed | Why |
|
||
| --- | --- |
|
||
| 模式 (hud/sphere/off) | legacy A/B debug toggle |
|
||
| 透明合成, 不透明度 | judge-able only in VR |
|
||
| 垂直位置, 距离, 宽度 | now fully replaced by the in-VR row |
|
||
| 距离与大小 (lock toggle) | the lock is always ON; it is the whole point |
|
||
| 视角宽度 | keeps its default; in-VR `小/大` is the control |
|
||
| 手部指向 | the index-ray mode was removed outright |
|
||
|
||
`视角宽度` / `subAngleDeg` still **persists** — the in-VR `小/大` buttons write it,
|
||
and the panel just no longer offers a second, less accurate way to set it. Same for
|
||
placement / opacity: set in VR, remembered next time.
|
||
|
||
Removed with it: `setSubModeLive()`, `setSubLockLive()`, `handRay()` and every
|
||
`handRayMode` read/write.
|
||
|
||
## 4. "VR 播放" button position (90-main.js CSS)
|
||
|
||
`.jfvr-btn` moved from `bottom: 24px` to `bottom: 120px` — at the viewport's
|
||
bottom-right it sat on top of Jellyfin's own OSD, covering its fullscreen button.
|
||
`.jfvr-panel` moved to `bottom: 180px` and `max-height: calc(100vh - 200px)` so it
|
||
still stacks above the button.
|
||
|
||
---
|
||
|
||
# ADDENDUM 7b — the padlock is a PANEL BUTTON, not its own layer
|
||
|
||
## What was wrong with ADDENDUM 7's badge
|
||
|
||
On device the badge could not be selected while the panel was shown, but could be
|
||
while it was hidden. ADDENDUM 7 had given the padlock **its own quad layer** parked at
|
||
`(0.45, -0.34, -1.45)` — 0.45 m to the right of the panel centre, i.e. 0.09 m to the
|
||
right of the panel's own right edge.
|
||
|
||
Aiming is the binding constraint and it was underestimated: the only pointing device
|
||
here is the hand ray, and the cursor's own acceptance window (`panelHit` padding
|
||
0.25/0.6 on a 0.72 m panel) reaches `x ≈ 0.45` — the panel's right edge. The badge was
|
||
**outside every reachable ray**. Worse, its hit test ran on the *panel's* layer
|
||
geometry, so whether it fired depended on where the panel happened to be, which is
|
||
exactly the "works when hidden, not when shown" report.
|
||
|
||
## What it is now
|
||
|
||
An ordinary button on the panel canvas — the `lock` action, fifth in the media row:
|
||
|
||
```
|
||
expanded: [<<<] [>||] [>>>] [CC] [padlock]
|
||
0.18 0.36 0.54 0.72 0.90 (hit pill half width 0.092)
|
||
```
|
||
|
||
It travels the same code path as every button that already works: cursor → `hitTest`
|
||
→ `handleSelect` → `onAction` → `90-main.js handleControlAction('lock')`.
|
||
|
||
## The "locked" panel mode
|
||
|
||
`show()` picks the mode from the lock state, so locking answers "what does a summon
|
||
look like now?" with a third mode:
|
||
|
||
| mode | size (m) | canvas | drawn / hit-testable |
|
||
| --- | --- | --- | --- |
|
||
| `expanded` | 0.72 × 0.21 | 1440 × 420 | everything |
|
||
| `locked` | 0.26 × 0.10 | 520 × 200 | padlock + 解锁 label only |
|
||
| `collapsed` | 0.60 × 0.05 | 1200 × 100 | (legacy; no longer entered) |
|
||
|
||
`hitTest(nx, ny, w, h, modeHint)` now takes the mode explicitly, because "collapsed" (a
|
||
thin strip) and "locked" (a small square-ish pill) share no aspect-ratio invariant.
|
||
`handleSelect` passes `S.mode`; the aspect-ratio `modeFor()` remains only as a fallback
|
||
for external callers.
|
||
|
||
`hide()` **no longer resets the mode**. The layer therefore keeps the size it will be
|
||
shown at, so the summon window that `90-main.js` ray-casts is the full panel rect
|
||
(0.72 × 0.21 unlocked, 0.26 × 0.10 locked) instead of a 5 cm strip — and re-showing
|
||
never needs a resize pop.
|
||
|
||
## Lifecycle (all of it in 90-main.js `toggleControlsLock`)
|
||
|
||
```
|
||
lock: S.ctlLock = true; Controls.setLock(true) -> mode 'locked'
|
||
Controls.show() -> padlock-only panel, auto-hide restarts
|
||
barShownAt = now
|
||
unlock: S.ctlLock = false; Controls.setLock(false) -> mode 'expanded'
|
||
Controls.show() -> full panel
|
||
barShownAt = now -> 0.5 s ignored, so the unlocking
|
||
press cannot land on a button
|
||
```
|
||
|
||
Summoning while locked is allowed (it is the only way back out): `pointerDown` no longer
|
||
early-returns on `S.ctlLock`, and the summon shows the padlock panel alone.
|
||
|
||
## Also fixed
|
||
|
||
`handleSelect()` used to fall back to `(nx, ny) = (0.5, 0.5)` when there was no pointer
|
||
hit and no reticle — which silently pressed whatever button lives at the centre. It now
|
||
returns `null` and performs nothing.
|
||
|
||
## API
|
||
|
||
```js
|
||
Controls.setLock(on) / getLock(): boolean // setLock also swaps the panel in place
|
||
Controls.hitTest(nx, ny, w, h, mode?) // mode: 'expanded' | 'locked' | 'collapsed'
|
||
Controls.PANEL_M / PANEL_PX // now include .locked
|
||
```
|
||
|
||
Stats: `stats.ctlMode` (`expanded` | `locked`), `stats.ctlLock`,
|
||
`stats.pressAt` (`SUMMON (locked panel)`, `LOCKED`/`UNLOCKED (armed 500ms)`, …).
|