Files
jellyfin_vr_web_plugin/INTERFACES.md
T
cat-shark 22dd482d69 hud-9: render native (SRT) subtitles in the HUD; stop persisting the layout override
ADDENDUM 10. Two more independent subtitle problems found on device.

1. Normal (non-ASS) subtitles never reached the HUD. Jellyfin's
   htmlVideoPlayer-plugin only routes ssa/ass through SubtitlesOctopus
   (`if (!['ssa','ass'].includes(codec))` -> pgssub/vobsub/custom-element/native);
   subrip falls through to the browser's own TextTrack rendering, so there is no
   libass canvas at all (measured with the CN SRT selected: 0 libass canvases,
   0 .videoSubtitles overlays, textTracks[0] = manualTrack/mode=showing/315 cues).
   No layout setting could have made that work.

   updateSource() now picks the source every XR frame: the libass canvas when it
   exists, otherwise the active TextTrack cue (fallback: the .videoSubtitlesInner
   overlay). The native text is cleaned, wrapped and painted white-on-black into a
   private 1920x1080 canvas that flows through the unchanged crop/upload/HUD path.
   prepare() is skipped when there is no libass canvas. New diagnostics:
   stats.subSource / stats.subNativeText, Subtitles.getSourceKind().

2. ADDENDUM 9 persisted state.subLayout, and a stale 'mono' in jfvr.settings.v1
   silently forced the mono branch on the default dual-eye content (dual:false,
   eyeW:4320, crop across the seam -> both eyes side by side). The override is now
   per-session: not saved, not restored, 'auto' at boot.

Verified on device with hud-9:
  CN_dual_eye (ass)   -> subSource=libass, dual=true, eyeW=2160,
                         crop=[1040,768,1176,924], copyND=0.055
  CN (subrip)         -> subSource=native, dual=false, eyeW=1920, text="进去了呢",
                         crop 256x88, uploads climbing
The private cue canvas was read back and OCR'd (cue "怎么样?好好地" -> "怎么样? 好好地"),
and injecting/removing a fake libass canvas flips subSource with no errors.
2026-10-12 19:53:10 +08:00

1138 lines
52 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# JFVR module interfaces (contract)
Everything lives under a single global namespace `window.__JFVR`, created by
`src/00-core.js` before any other module runs. Modules are plain
`(function(){ ... })()` IIFEs that attach themselves to `__JFVR`.
Load order (build script concatenates in this order):
```
00-core.js -> __JFVR core: state, helpers, layer/frame-hook registry
10-subtitles.js -> __JFVR.Subtitles
20-controls.js -> __JFVR.Controls
90-main.js -> boot, DOM scan, XR session lifecycle, settings panel, wiring
```
Nothing may read `document` / `navigator.xr` at load time except `00-core.js`
and `90-main.js`. The two module files must only *define* functions and are
called later with a session/frame.
---
## 00-core.js — always available as `window.__JFVR`
```js
__JFVR.state // shared mutable state object (see below)
__JFVR.log(...args) // console.log with [JFVR] prefix + colour
__JFVR.version // string
// --- helpers ---
__JFVR.makeEl(tag, className?, text?) // document.createElement wrapper
__JFVR.compileShader(gl, type, source) // throws with the info log on failure
__JFVR.invert4(mat4ArrayLike) // -> Float32Array(16)
__JFVR.QUAD_VS // vertex shader source: fullscreen triangle,
// `in vec2 aPos; out vec2 vUv;`
// vUv = vec2(aPos.x*.5+.5, .5 - aPos.y*.5)
__JFVR.bindFullscreenTriangle(gl, program) // creates the VBO, enables aPos, returns attribLoc
// --- XR layer registry (owned by 90-main.js) ---
__JFVR.addLayer(layer) // register + push updateRenderState if a session is live
__JFVR.removeLayer(layer) // unregister + push
__JFVR.addFrameHook(fn) // fn(frame, time, session) every XR frame
__JFVR.removeFrameHook(fn)
// --- capabilities (filled by 00-core.js at boot) ---
__JFVR.caps = {
secureContext, hasXR, immersiveVR, hasXRMediaBinding, hasEquirectLayer,
hasXRWebGLBinding, hasXRHand, hasXRJointSpace, maxTextureSize,
hevc: { ... }, ua
}
// --- live handles ---
__JFVR.session // current XRSession or null
__JFVR.video // current <video> element or null
__JFVR.refSpace // 'local' XRReferenceSpace for the running session
__JFVR.viewerSpace // 'viewer' XRReferenceSpace (head-locked layers)
```
### `__JFVR.state` fields that modules may use
```js
state.video // <video> or null
state.session // XRSession or null
state.refSpace // local space
state.viewerSpace // viewer space (head-locked)
state.projection // '180' | '360'
state.stereo // 'mono' | 'sbs' | 'ou'
state.subtitles // bool — render subtitles in VR
state.subMode // 'hud' (head-locked quad, default)
// 'sphere'(on the equirect layer, legacy)
state.subAlpha // 'premultiplied' | 'straight'
state.subPlacement // { y, z, width } metres, hud mode only
state.stats // diagnostics bag, free to add keys
state.gl // { refSpace, session } set by main during a session
```
### Layer / frame-hook lifecycle a module must follow
1. `create(session, glContextOrNull)` — build GL resources + the XR layer,
call `__JFVR.addLayer(layer)`, `__JFVR.addFrameHook(hook)`.
Return the layer, or `null` when it cannot run (missing canvas, missing API…).
**Never throw** — return null and `__JFVR.log()` the reason.
2. `hook(frame, time, session)` — per-frame work. Must be cheap; skip work when
nothing changed.
3. `destroy()` — remove the frame hook, remove the layer, delete GL resources,
drop canvas references. Must be safe to call twice and safe to call when
`create()` returned null.
---
## 10-subtitles.js — `__JFVR.Subtitles`
Renders Jellyfin's libass subtitle canvas into VR.
```js
__JFVR.Subtitles = {
// Called once at boot, before any session.
init(): void,
// BEFORE entering VR. Jellyfin sizes the libass canvas to the on-screen video
// box (~1250x625), which is far too small for VR. Blowing up the player box
// makes libass re-render bigger. Must be called while NOT in an immersive
// session (libass does not recompute its backing store during XR).
// Idempotent: calling twice must not double-apply.
prepare(): void,
// Undo prepare(). Safe when prepare() was never called.
restore(): void,
// True when a libass canvas exists and has non-zero size.
isReady(): boolean,
// Current libass canvas size, for diagnostics: { w, h } or null
canvasSize(): { w, h } | null,
// --- session lifecycle ---
create(session): XRLayer | null,
destroy(): void,
// --- settings, safe to call before or during a session ---
setMode(mode), // 'hud' | 'sphere' | 'off'
setAlphaMode(mode), // 'premultiplied' | 'straight'
setPlacement({ y, z, width }), // partial update, hud only
getMode(): string
};
```
### HUD mode requirements (the default, and the point of this module)
- A **head-locked quad layer** in `__JFVR.viewerSpace`
(`XRWebGLBinding.createQuadLayer({ space: viewerSpace, layout: 'mono', ... })`).
- Source pixels come from `canvas.libassjs-canvas`. That canvas holds a
**stereo left-right** image: the left half is the `LeftEye` subtitle, the
right half the `RightEye` one. For a mono HUD use **one half only**
(the left half is correct for this content).
- **Auto-crop to the ink bounding box** so the strip is tight and the text is
as large as possible. Recompute the bbox at a low rate (≈4 Hz) on a
downscaled copy — never `getImageData` the full-resolution canvas every frame
(a 4320x2160 full read is ~37 MB and has already crashed a renderer once).
Add hysteresis / smoothing so the strip does not jitter between frames.
- Upload only when the source changed (or at most ~30 Hz) — the earlier
implementation burned a 4K texture upload every frame.
- Target ≈30 px/degree. A 2.0 m wide strip at 1.45 m is ≈66° → texture width
2048 is comfortable on Quest 3.
- Respect `state.subAlpha`:
- `'premultiplied'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true)`
- `'straight'`: `gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false)`
- Draw with `XRWebGLBinding.getSubImage(layer, frame)` → bind
`sub.colorTexture` to a framebuffer → viewport to
`sub.colorTextureWidth/Height` → clear to `(0,0,0,0)` → draw.
`getSubImage` takes **two** arguments.
### `prepare()` detail
The measured behaviour: setting the player box to `7680x3840` makes libass
produce a **4320x2160** canvas (it does not go higher). The previous bug was
that this was only applied on "open panel" and then restored on session end
while the panel stayed open, so a re-entry got a 1250x625 canvas.
`prepare()` must therefore be idempotent and re-callable.
Target DOM: `document.querySelector('.videoPlayerContainer')` and the `<video>`.
Save/restore their inline `style.cssText` exactly.
---
## 20-controls.js — `__JFVR.Controls`
A head-locked control strip: progress bar + seek, play/pause, skip ±10 s,
subtitle toggle. **Pure 2D canvas rendering + a hit-test function, plus a thin
XR quad layer wrapper.**
```js
__JFVR.Controls = {
init(): void,
// --- pure rendering, no WebXR, must be unit-testable in isolation ---
// Draws the bar into `ctx` sized `w` x `h` (CSS pixels).
// `view` = {
// playing, currentTime, duration, buffered, subtitlesOn,
// mode: 'collapsed' | 'expanded',
// hover: null | 'playpause' | 'back10' | 'fwd10' | 'subs' | 'track'
// }
draw(ctx, w, h, view): void,
// Maps a point in normalized panel coords (0..1, origin top-left) to an action.
// Returns one of:
// { type: 'seek', value: 0..1 }
// { type: 'playpause' | 'back10' | 'fwd10' | 'subs' }
// null
hitTest(nx, ny, w, h): action | null,
// --- XR lifecycle ---
create(session): XRLayer | null,
destroy(): void,
// --- state / input ---
setVideo(videoEl, getExtra | null): void, // main provides time/duration/buffered readers
onAction(cb): void, // cb(action) — main performs the media op
show(autoHideMs?), hide(): void, // collapsed <-> expanded + auto-hide
isVisible(): boolean,
handleSelect(inputSource, frame): void // called on XR `select`
};
```
### Panel design requirements
- **Two visual states**: `collapsed` = a slim progress bar only (≈0.9 m x 0.07 m,
centre-bottom of view). `expanded` = progress row + buttons
(≈1.1 m x 0.22 m).
- Hit targets must be **generous**: the progress bar should be visually ~1 cm
tall but hit-testable over ~3–4 cm. Meta's docs warn that the emulated hand
target ray for hand tracking is only a coarse forward pointer, so **do not
rely on pixel-precise aiming**.
- Colours must read on top of arbitrary video: dark translucent pill background,
white text, accent `#4ea1ff` (matches the existing in-page panel).
- Everything must be drawn from scratch on a canvas — no DOM, no images, no
external fonts (use a generic stack: `'system-ui, -apple-system, sans-serif'`).
- `draw()` must be deterministic and side-effect free (no timers inside).
- `hitTest` must be a pure geometry function of `(nx, ny, w, h)`.
### Input model (main.js supplies)
- **Gaze**: the panel is head-locked, so the reticle is the view centre.
`main.js` converts that into `(nx, ny)` and calls `hitTest`.
- **Controller ray**: `main.js` intersects the controller ray with the panel
plane and converts to `(nx, ny)`.
- Both end in `handleSelect()` → the action from `hitTest` → `onAction` callback.
---
## 90-main.js — owned by the integrator
Boot, DOM scan, the in-page `VR 播放` button + settings panel, the XR session
lifecycle, the rAF broker (see below), input dispatch, and the settings UI for
subtitle mode / alpha mode / placement.
### rAF broker (already proven, keep it)
While an immersive XR session is running `window.requestAnimationFrame` is
**completely stopped** (measured: 0 callbacks in 6 s), and Jellyfin renders
libass subtitles from an rAF loop. `90-main.js` intercepts
`requestAnimationFrame`/`cancelAnimationFrame`, holds callbacks while in XR and
drives them from `video.requestVideoFrameCallback`. Outside XR it delegates to
the native implementation.
---
# ADDENDUM 2 — in-VR UX pass (after first on-device test)
Owner feedback from the first headset test:
1. **The control bar is permanently visible — wrong.** It must be hidden by
default and only appear on interaction, then auto-hide.
2. **Hand/controller input has no guide line.** There is no visible ray and no
cursor, so the user cannot tell where they are aiming.
3. **Subtitles must sit below the visual centre** by default, and both the
**position and the opacity must be user-adjustable**.
## New shared state (owned by 90-main.js, read by modules)
```js
state.subOpacity // 0..1, default 0.85
state.subPlacement // { y, z, width } — y is the quad CENTRE, negative = below eye level
state.ctlAutoHideMs // default 4000
state.pointer // { visible, source, origin:{x,y,z}, dir:{x,y,z}, hit:{nx,ny}|null }
// written every XR frame by 90-main.js; modules must treat it as read-only
```
## 10-subtitles.js — additions
```js
setOpacity(a) // 0..1; applied to the XR layer (layer.opacity or the
// shader's alpha), NOT by rebuilding the layer
getOpacity(): number
```
- Default `state.subPlacement` becomes `{ y: -0.42, z: -1.45, width: 2.0 }`.
- `setPlacement()` must take effect live (update `layer.transform` if the runtime
allows mutating it, otherwise rebuild just this layer and keep the frame hook).
- Keep the existing tight-crop behaviour and the 4320x2160 canvas preparation.
## 20-controls.js — additions
```js
setPointer(p) // p as in state.pointer; called every frame by main.
// Draw a beam from origin along dir and, when hit != null, a
// cursor dot at (hit.nx, hit.ny) on the panel.
setOpacity(a)
getOpacity(): number
```
- **Default state is HIDDEN.** `create()` must not make the bar visible.
- `show(autoHideMs)` → visible + expanded, auto-hides after `autoHideMs`
(default `state.ctlAutoHideMs`).
- `hide()` → fully hidden (no bar, no beam, no cursor) until `show()` again.
- `isVisible()` → true only while shown.
- The beam must be a **separate, always-updated visual** that is only visible
while the bar is shown. Implement it as a thin quad layer whose transform is
placed at `origin + dir * (L/2)` and oriented so its local -Z follows `dir`
(build the quaternion from the direction; L = distance to the hit, or a fixed
1.2 m when there is no hit). Height ≈ 0.006 m, width = L, with a soft gradient
texture (bright at the origin end, fading out) so it reads as a laser.
- The cursor: when `hit` is non-null, draw a ring/dot on the control panel canvas
at `(hit.nx, hit.ny)` so the user sees exactly what will be activated.
- Do not re-upload either texture more than ~30 Hz; skip everything when hidden.
- Add stats keys `ctlPointer`, `ctlHidden`, `ctlBeamOn`.
## Input model change (90-main.js) — unified pointing
Hand tracking and controllers both expose `targetRaySpace`. Use one code path:
- For **every** input source with `targetRayMode === 'tracked-pointer'`
(controllers *and* hands), intersect its ray with the control quad plane and
produce `hit`.
- If no source produces a hit, fall back to gaze: `hit = {nx:0.5, ny:0.5}` centred.
- `state.pointer.source` records which one won.
- Show the beam for the winning source; hide it when the controls are hidden.
Meta's docs note the hand target ray is an *emulated* forward-pointing ray, so
hit targets stay generous — that is already the case.
### Beam orientation — CORRECTED convention (my first spec was wrong)
Do **not** map the quad's local `-Z` onto the ray direction. A quad's local `-Z`
is its plane **normal**, so that would put the quad's plane perpendicular to the
ray and lay its `width` extent across the beam — a thin face-on bar, not a beam.
Correct:
```
position = origin + dir * (L / 2) // L = hit distance, else 1.2 m
width = L (the long axis)
height = 0.006
local +X -> dir // the beam axis lies IN the plane
local -Z -> perpendicular to dir, biased up so the ribbon is never edge-on
gradient -> bright at the origin end, fading to transparent at the hit end
```
Assertion for tests: `R(q)·(1,0,0) ≈ dir` and `R(q)·(0,0,-1) ⟂ dir`.
---
# ADDENDUM 4 — in-VR subtitle placement (grab & drag)
Owner requirement: adjust the subtitles **without leaving VR**. Chosen model:
- **Position**: grab the subtitle strip with the laser and drag it in space.
- **Reset** button restores the defaults.
- **Size** and **opacity** also adjustable in VR.
- Values **persist in localStorage** (per browser, no server changes).
## Grab & drag model (owned by 90-main.js)
The strip is a head-locked quad in viewer space, so both the ray and the strip
live in the same space — no world-space math needed.
```
on grab start:
grabActive = true
grabOrigin0 = ray.origin // viewer space
grabHit0 = ray ∩ plane(z = placement.z)
grabPlace0 = { y, z, width }
per frame while grabbed:
P = currentRay ∩ plane(z = grabPlace0.z) // fixed reference plane
y' = grabPlace0.y + (P.y - grabHit0.y) // drag up/down
z' = grabPlace0.z + (ray.origin.z - grabOrigin0.z) // pull toward / push away
clamp y' to [-0.95, -0.05], z' to [-2.6, -0.7]
Subtitles.setPlacement({ y: y', z: z', width: current })
on grab end: clear grab state, persist
```
Rationale for `z`: in viewer space forward is `-Z`, so pulling the hand back
(increasing `origin.z`) makes `z'` less negative, i.e. the strip comes **closer**
— which matches how pulling a real object behaves.
Grab **must not** steal the normal control-bar interaction: only start a grab when
the ray actually hits the subtitle quad; otherwise fall through to the bar.
## Input: press **and release** are now needed
The existing activation path in `90-main.js` only detects a rising edge. Grabbing
needs the held state too, so track per input source:
- `selectstart` / `selectend` session events, **and**
- `inputSource.gamepad` button press/release (hand pinch; see ADDENDUM 3)
Both feed one `pointerDown(src)` / `pointerUp(src)` pair, deduped the same way.
## 10-subtitles.js — additions
```js
getLayer(): XRLayer | null // so main can ray-cast against the strip
resetPlacement(): void // back to { y: -0.42, z: -1.45, width: 2.0 }
```
- `width` **is** the size control: `setPlacement({ width })` already rescales the
quad while keeping the angular text height consistent with the texture aspect.
- All of `setPlacement` / `setOpacity` / `resetPlacement` stay live during a session.
## 20-controls.js — additions
The expanded control bar gains a subtitle row with large −/+ buttons:
| Action returned by `hitTest` | Meaning |
| --- | --- |
| `sub_smaller` / `sub_bigger` | width by ±0.15 m, clamped to [0.8, 4.0] |
| `sub_opaque_down` / `sub_opaque_up` | opacity by ∓0.05, clamped to [0.1, 1.0] |
| `sub_reset` | restore all subtitle defaults |
- Add `view.sub = { width, opacity }` so `draw()` can render the current values.
- Keep hit targets generous; the rows may overlap visually but must not overlap
in hit-test order (buttons before track).
- Add a new expanded height that fits the extra row.
## 90-main.js — persistence
```
localStorage key: 'jfvr.settings.v1'
{ subPlacement: {y,z,width}, subOpacity, subMode, subAlpha,
projection, stereo, ctlAutoHideMs }
```
Load at boot (merge over defaults, validate ranges), save debounced ~400 ms after
any change. A corrupt/absent value must fall back to defaults silently.
---
# ADDENDUM 5 — gaze pointing + lazy-follow panel + panel size
## Why (measured on device)
The hand's `targetRaySpace` is an **emulated** ray (Meta documents this). Measured
on a real Quest 3 while the panel was straight ahead:
```
hand:right origin=(-0.317,-0.054,-0.248) dir=(-0.874, 0.461, -0.154)
^ 87 % sideways, 15 % forward
```
Aiming with it lands somewhere the user is not looking. Gaze + pinch is Meta's
documented recommendation for hands and needs no ray accuracy at all.
## Changes
| # | Change |
| --- | --- |
| 1 | Pointer rays are computed in **LOCAL space** (`__JFVR.refSpace`), not viewer space. |
| 2 | Hands aim with the **gaze ray** (head forward in local space). No beam - a guide line from the hand would be misleading because the hand is not aiming. The on-panel reticle is the indicator. |
| 3 | Controllers keep their own `targetRaySpace` ray **and** the laser beam; a controller ray is accurate and a laser is expected. Selected by `pointer.beam`. |
| 4 | The control panel layer's **space is `refSpace`** and holds a stable world pose. A head-locked panel would always put the view centre on its own centre, so no button could ever be targeted by gaze. |
| 5 | 90-main.js drives a **lazy follow**: it only re-centres the panel once the head has turned > 26 deg away from it, easing with a 0.45 s time constant. |
| 6 | `subtitles` is **no longer persisted**. Persisting it meant a session where the user toggled subtitles off came back with the layer never created - which looked like a load failure. Always ON at boot; the toggle is per-session. |
| 7 | Subtitle **distance** range extended: the near clamp is now **0.5 m** (was 0.7 m). |
---
# ADDENDUM 6 — subtitle depth vs size decoupling (angular-size lock)
## Why (owner requirement)
> 当前 vr 视频中可能出现 vr 空间中物体可能距离我很近,如果字幕还很远,我将无法在
> 看清物体的同时看清字幕。我希望可以让字幕的空间距离和物体的距离尽量接近。
Eye tracking is **not** exposed to WebXR on Quest (HANDOFF §4), so the strip's depth
cannot follow the user's gaze automatically. What matters instead is that the DEPTH
control must not change the strip's apparent size: otherwise pulling the subtitles to
a nearby object makes the text huge (and, with the head-locked quad, drops it out of
the lower view). See HANDOFF §8 for the literature this is based on.
## Model
`state.subLockAngle` (default `true`) + `state.subAngleDeg` (default `69.2`).
While **locked**, a depth change holds TWO things:
```
width = 2 * |z| * tan(subAngleDeg / 2) angular width (theta)
y = (y0 / z0) * z angular height (the y/z ratio)
```
so the strip keeps both its angular size and its screen position — only depth
changes. The default pair is bit-for-bit the pre-ADDENDUM-6 look (2.0 m at 1.45 m →
69.2°), so nothing moves for anyone who does not touch the control.
While **unlocked** everything behaves exactly as at ADDENDUM 4: `width` and `z` are
independent ("pull closer = bigger").
Ranges: `theta` ∈ [20, 120]°, `z` ∈ [-2.6, -0.35] m (near clamp moved from 0.5 m to
0.35 m — the whole point is matching very close objects). Under the lock the width
floor drops to 0.05 m (the angle clamp is the real limit); free mode keeps 0.2 m.
## 10-subtitles.js — API additions
```js
setLockAngle(on) / getLockAngle(): boolean
setAngleDeg(deg) / getAngleDeg(): number // locked angular width, degrees
getPlacement(): { y, z, width } // the resolved, clamped triple
```
- `setPlacement({ y, z, width })` under the lock:
- `z` without `y` → depth-only: `y` is re-derived from the held y/z ratio and
`width` from `subAngleDeg`;
- `width` without `z` → size change: `subAngleDeg` is re-derived from the requested
width at the current depth (the derived width then wins);
- `y` + `z` (grab & drag) → the caller's `y` wins, only `width` is derived.
- `resetPlacement()` restores `{ y:-0.42, z:-1.45, width:2.0 }` **and**
`subAngleDeg = 69.2`; the lock flag is left alone.
- New diagnostics: `stats.subAngleDeg`, `stats.subLockAngle`, `stats.subYOverZ`, and
`stats.subTextDeg` — the ACTUAL angular height of the drawn subtitle ink, in
degrees. arXiv 2004.01545 measures preferred angular size (character height) at
~16–18 dmm ≈ 0.9–1.0° for medium/long text and ~32–40 dmm ≈ 1.8–2.3° for
titles/headlines; Google's dmm is 1 mm of character height at 1 m. Use
`subTextDeg` to pick a `subAngleDeg` default from real measurements later; the
shipped default is deliberately the on-device-proven look.
## 20-controls.js — subtitle row is now NINE buttons
```
sub_up sub_down sub_near sub_far sub_smaller sub_bigger sub_opaque_down sub_opaque_up sub_reset
上 下 近 远 小 大 暗 亮 复位
```
- The vertical-position pair (up/down) and the depth pair (near/far) are the
ADDENDUM-6 additions.
- `view.sub` gains `angleDeg`, `lock` and `dist` (= `|z|`). The caption ALWAYS
leads with the depth in metres — `字幕 距1.45m · 69°` while locked,
`字幕 距1.45m · 2.0m` while free — because the depth is the number the user has
to match against the object being watched. Opacity stays right-aligned.
- **`readSubtitles()` bug fixed**: it only read the module-local `S.subtitlesOn`,
which `90-main.js` never set (it calls `setVideo(video)` with no `getExtra`), so
the CC button always rendered as OFF. It now falls back to the shared
`state.subtitles`. The button also got an unambiguous on/off treatment: an
accent pill + `字幕开` when on, a slashed CC glyph + `字幕关` when off.
- Hit pills: pitch 0.10, half width 0.052 (a hair of overlap → no dead zone);
visual width 0.092; caption font 0.048·h, button label font 0.040·h. The bar
itself is unchanged (`PANEL_M.expanded` = 0.72 × 0.21 m, canvas 1440 × 420 px).
## 90-main.js
- Persists `subLockAngle` / `subAngleDeg` in `jfvr.settings.v1`; while locked the
stored `subPlacement.width` is re-derived on load, so `subAngleDeg` + `z` is what
restores the user's look.
- In-page panel: a `距离与大小` toggle (锁定视角尺寸 / 自由), and a `视角宽度`
slider in degrees shown while locked / the old `宽度` slider in metres while free.
All sliders re-read their value whenever a setter runs, so VR-bar and grab changes
show up in an already-open panel.
- `sub_up` / `sub_down` step `y` by ±0.04 m. `sub_near` / `sub_far` step `z` by
±0.15 m. `sub_smaller` / `sub_bigger` step `subAngleDeg` by ∓5° while locked,
else `width` by ∓0.15 m.
- Grab & drag: while locked, a depth drag also carries `y` by the grab-start y/z
ratio (vertical hand motion still adds on top), so the strip cannot slide out of
the lower view when it is pulled close.
---
# ADDENDUM 7 — summon gating, lock badge, pre-VR panel trim
Owner feedback after the ADDENDUM 6 pass:
> 控制面版的唤出太灵敏了,只有当指示线指向面板附近时,面板再显示出来即可;在 vr 空间
> 右侧边缘增加一个锁的图标,当选中上锁后,面板不会启动;在刚唤出面板时,0.5s 内的操作
> 应该忽略,否则可能误触。在进入"vr 模式"时,字幕相关有很多滑块配置,移除哪些配置,
> 在没有在真实 vr 空间中的调整,都是不准确的,没必要在进入前指定。视角宽度保持默认即可。
> 字幕相关只保留开启和关闭即可。"vr 播放"这个按钮,不要挡住原有播放器的选项,现在似乎
> 挡住了原播放器的全屏按钮;手部指向选项直接移除吧,没人会只使用食指作为指向。
## 1. Summon gating (90-main.js)
There were **two** independent reasons the bar appeared on *every* pinch, and both
are gone:
| Was | Now |
| --- | --- |
| `onSelectStart` called `Controls.show()` unconditionally | listener removed; summoning is decided in `pointerDown()` from the actual aim |
| `pointerDown()` called `Controls.show()` before checking the hit | `show()` is called only when the ray lands on / near the bar |
```
bar hidden + aim ON the bar -> show(), record barShownAt, return (the summoning press never clicks)
bar hidden + aim NEAR the bar-> same (padded window, see below)
bar hidden + aim elsewhere -> nothing
bar hidden + locked -> nothing
bar shown + < PANEL_ARM_MS(500) since it appeared -> ignore
bar shown + aim ON the bar -> show() (extends auto-hide) + handleSelect
```
Barely-visible: the summoning press is *consumed* — it does not also press a button,
and for 500 ms after the bar appears every activation is ignored. Otherwise the hand
travelling onto the bar right after the summon fires a random button.
The summon window is deliberately wider than the button window: `pointer.hit` (used
for the cursor and for button hits) uses `panelHit(..., 0.25, 0.6)`, while summoning
falls back to `panelHit(..., 0.5, 0.9)`. A ray that merely grazes the bar can still
bring it up; the 500 ms arm is what protects against mis-clicks.
Stats: `stats.pressAt` now reads `SUMMON (armed 500ms)`, `IGNORED (arming)`,
`MISS (not aiming at the panel)` or `MISS (panel locked)`.
## 2. Lock badge (20-controls.js + 90-main.js)
An **always-visible** small quad (`LOCK_X/Y/Z = 0.86 / -0.34 / -1.45`, `LOCK_M = 0.075`,
texture 128²) placed in the **same `refSpace`** as the control bar — to the right of
it. While `state.ctlLock` is true, nothing summons the bar; the badge stays visible
and hit-testable so it can be unlocked again.
Why `refSpace` and not a head-locked HUD corner: on this setup the only pointing
device is the hand's ray, and a head-locked badge could never be aimed at. "Right-hand
edge of the VR space" is therefore implemented as "right of the control bar, in the
same world-locked space".
- Unlocked: dim (60 % white) padlock with the shackle open.
- Locked: accent (`#4ea1ff`) filled padlock, shackle closed, keyhole punched out.
- Hit window: `panelHit(..., 0.6, 0.6)` ≈ 6.5° across at 1.45 m — the badge itself is
only ~3°, so it is easy to hit with a coarse hand ray.
- The lock is **per VR session**: `enterVR()` sets `ctlLock = false`, `onSessionEnd()`
resets it too. A stuck lock would look exactly like a broken plugin, so it is never
restored from `localStorage`.
New API:
```js
Controls.getLockLayer(): XRLayer | null
Controls.setLock(on) / getLock(): boolean
```
Stats: `stats.ctlLock`.
## 3. Pre-VR settings panel trimmed (90-main.js)
Sliders set on a flat screen do not transfer to a headset, so the only subtitle
setting left before entering VR is **开 / 关**:
| Removed | Why |
| --- | --- |
| 模式 (hud/sphere/off) | legacy A/B debug toggle |
| 透明合成, 不透明度 | judge-able only in VR |
| 垂直位置, 距离, 宽度 | now fully replaced by the in-VR row |
| 距离与大小 (lock toggle) | the lock is always ON; it is the whole point |
| 视角宽度 | keeps its default; in-VR `小/大` is the control |
| 手部指向 | the index-ray mode was removed outright |
`视角宽度` / `subAngleDeg` still **persists** — the in-VR `小/大` buttons write it,
and the panel just no longer offers a second, less accurate way to set it. Same for
placement / opacity: set in VR, remembered next time.
Removed with it: `setSubModeLive()`, `setSubLockLive()`, `handRay()` and every
`handRayMode` read/write.
## 4. "VR 播放" button position (90-main.js CSS)
`.jfvr-btn` moved from `bottom: 24px` to `bottom: 120px` — at the viewport's
bottom-right it sat on top of Jellyfin's own OSD, covering its fullscreen button.
`.jfvr-panel` moved to `bottom: 180px` and `max-height: calc(100vh - 200px)` so it
still stacks above the button.
---
# ADDENDUM 7b — the padlock is a PANEL BUTTON, not its own layer
## What was wrong with ADDENDUM 7's badge
On device the badge could not be selected while the panel was shown, but could be
while it was hidden. ADDENDUM 7 had given the padlock **its own quad layer** parked at
`(0.45, -0.34, -1.45)` — 0.45 m to the right of the panel centre, i.e. 0.09 m to the
right of the panel's own right edge.
Aiming is the binding constraint and it was underestimated: the only pointing device
here is the hand ray, and the cursor's own acceptance window (`panelHit` padding
0.25/0.6 on a 0.72 m panel) reaches `x ≈ 0.45` — the panel's right edge. The badge was
**outside every reachable ray**. Worse, its hit test ran on the *panel's* layer
geometry, so whether it fired depended on where the panel happened to be, which is
exactly the "works when hidden, not when shown" report.
## What it is now
An ordinary button on the panel canvas — the `lock` action, fifth in the media row:
```
expanded: [<<<] [>||] [>>>] [CC] [padlock]
0.18 0.36 0.54 0.72 0.90 (hit pill half width 0.092)
```
It travels the same code path as every button that already works: cursor → `hitTest`
→ `handleSelect` → `onAction` → `90-main.js handleControlAction('lock')`.
## The "locked" panel mode
`show()` picks the mode from the lock state, so locking answers "what does a summon
look like now?" with a third mode:
| mode | size (m) | canvas | drawn / hit-testable |
| --- | --- | --- | --- |
| `expanded` | 0.72 × 0.21 | 1440 × 420 | everything |
| `locked` | 0.26 × 0.10 | 520 × 200 | padlock + 解锁 label only |
| `collapsed` | 0.60 × 0.05 | 1200 × 100 | (legacy; no longer entered) |
`hitTest(nx, ny, w, h, modeHint)` now takes the mode explicitly, because "collapsed" (a
thin strip) and "locked" (a small square-ish pill) share no aspect-ratio invariant.
`handleSelect` passes `S.mode`; the aspect-ratio `modeFor()` remains only as a fallback
for external callers.
`hide()` **no longer resets the mode**. The layer therefore keeps the size it will be
shown at, so the summon window that `90-main.js` ray-casts is the full panel rect
(0.72 × 0.21 unlocked, 0.26 × 0.10 locked) instead of a 5 cm strip — and re-showing
never needs a resize pop.
## Lifecycle (all of it in 90-main.js `toggleControlsLock`)
```
lock: S.ctlLock = true; Controls.setLock(true) -> mode 'locked'
Controls.show() -> padlock-only panel, auto-hide restarts
barShownAt = now
unlock: S.ctlLock = false; Controls.setLock(false) -> mode 'expanded'
Controls.show() -> full panel
barShownAt = now -> 0.5 s ignored, so the unlocking
press cannot land on a button
```
Summoning while locked is allowed (it is the only way back out): `pointerDown` no longer
early-returns on `S.ctlLock`, and the summon shows the padlock panel alone.
## Also fixed
`handleSelect()` used to fall back to `(nx, ny) = (0.5, 0.5)` when there was no pointer
hit and no reticle — which silently pressed whatever button lives at the centre. It now
returns `null` and performs nothing.
## API
```js
Controls.setLock(on) / getLock(): boolean // setLock also swaps the panel in place
Controls.hitTest(nx, ny, w, h, mode?) // mode: 'expanded' | 'locked' | 'collapsed'
Controls.PANEL_M / PANEL_PX // now include .locked
```
Stats: `stats.ctlMode` (`expanded` | `locked`), `stats.ctlLock`,
`stats.pressAt` (`SUMMON (locked panel)`, `LOCKED`/`UNLOCKED (armed 500ms)`, …).
---
# ADDENDUM 8 — panning the video window (yaw / pitch about the eye)
## Why
> 我在观看过程中,有时想调整180度视频的播放位置,让其更适合我的环境,我希望画面可以不是
> 固定在我的正前方180度,我希望能调整画面内容在整个vr空间中以我眼睛为旋转点进行上下左右
> 的水平移动(为了解决一些场景需要物理低头才能看到的问题,我不想真的低头去看)
The 180° window was hard-wired dead ahead. Scenes staged below the horizon could only
be watched by physically lowering the head.
## Model
The video layer is an `XREquirectLayer` with `radius` 0 — the content sits **at
infinity**. Two consequences drive the whole design:
- Only the layer's **orientation** decides which direction is visible. Translating it
does nothing, and there is no parallax, so the picture does not swim when the head
moves.
- So panning the picture is *purely* `layer.transform`'s rotation, which rotates the
content **about the eye point** — exactly what was asked for. No decode, no texture
upload, no extra GPU work: 8K HEVC DirectPlay is untouched.
Only the video rotates. The panel lives at a fixed pose in `refSpace` and the
subtitles are head-locked in `viewerSpace`, so **the UI stays in front of the user**
while the picture moves.
```js
state.videoYaw // radians, + = the picture has been moved RIGHT
state.videoPitch // radians, + = the picture has been moved UP
q = qy(-yaw) * qx(pitch) // pitch in the content frame, then yaw about world Y
layer.transform = XRRigidTransform({0,0,0}, q)
```
Because of that composition the window's aim direction is exactly
`(cos p·sin yaw, sin p, −cos p·cos yaw)`, so yaw is a pure azimuth and pitch a pure
elevation — no gimbal weirdness in the useful range.
Limits: pitch ±(π/2 − 0.02) — i.e. effectively unlimited, per the owner's
"看用户拖动幅度来定"; the guard only avoids the degenerate pole. Yaw wraps to
[−π, π]. **Not persisted**: every VR session starts dead ahead.
## Interaction — one affordance, in the panel
A **drag strip above the progress bar** (the owner's requested placement), part of the
panel canvas like every other control:
```
┌─ 画面 → the drag strip (panel y 0.00–0.20) ────────────────────────┐
│ ⠿ 拖动调整画面 偏航 +12° 俯仰 -8° │
│ 连点 3 次复位 0° / 0° = 正前方 │
└──────────────────────────────────────────────────────────────────┘
```
- **Pinch the strip and drag** → the picture follows the hand 1:1.
- **Three taps inside 1.6 s** → reset to dead ahead. Taps are counted on RELEASE, so a
real drag can never be mistaken for one.
- The right side always shows the live angles, so the state is readable even when you
are not moving anything.
The drag is driven by the **pointer ray's own direction**, not by a position on the
panel:
```
yaw' = wrap(yaw0 + (rayYaw - rayYaw0))
pitch' = clamp(pitch0 + (rayPitch - rayPitch0)) rayYaw = atan2(dx, -dz)
rayPitch = asin(dy)
```
A panel-space cursor would run off the panel's edge after a few degrees of movement;
ray rotation has no range limit. It also makes the content point under the cursor stay
under the cursor, which is what "grab the picture" should feel like.
`Controls.hold(true/false)` freezes the panel auto-hide for the duration of the drag —
otherwise the panel would vanish mid-drag because the timer `show()` armed keeps
running.
## Runtime risk and how it is handled
> **SUPERSEDED by ADDENDUM 8b** — the probe below compared quaternion components,
> which is sign-sensitive and produced a false "read-only" verdict; and the gesture is
> now tap = reset / hold 0.5 s = drag. See the addendum at the end of this file.
`XRCompositionLayer.transform` / `XREquirectLayer.transform` are settable in the spec,
but a runtime may ignore the write. Since the whole interaction depends on it, the
module probes once per session:
```js
probeVideoAngleMutable(layer) // write a known rotation, read it back
-> stats.videoRotMode = 'live' | 'rebuild'
```
- `live` — a transform write per frame while dragging. Free.
- `rebuild` — the media layer is recreated. Far too expensive per frame, so rebuilds
happen only on **creation / release / reset**; during a drag the state updates and
the picture catches up when the finger comes up. `stats.videoRebuilds` counts them.
## 20-controls.js — changes
- The expanded panel grew from **0.72 × 0.21 m to 0.72 × 0.26 m** (canvas 1440 × 420 →
1440 × 520) to make room for the strip; every row was re-proportioned.
- New hit band `viewHitY0/Y1 = 0..0.20` returning `{type:'view_drag'}`, spanning the
panel width. It does not overlap the seek band (0.30–0.52) or the button bands.
- `view.view = { yawDeg, pitchDeg }` feeds the readout (`drawViewStrip`), and is part
of the redraw signature.
- New `hold(on)` for the auto-hide pause; `stats.ctlHold`.
## 90-main.js — changes
- `state.videoYaw/videoPitch` reset to 0 at session start and at session end.
- `S.videoLayer` handle (was local), `applyVideoAngle(force)` / `rebuildVideoLayer()`,
`resetVideoAngle()`, `videoQuat()`, `rayAngles()`, `beginViewDrag/updateViewDrag/
endViewDrag`, and `viewTaps` for the triple-tap reset.
- The frame loop calls `updateViewDrag(frame)`; `pointerUp()` calls `endViewDrag()`.
- `handleControlAction('view_drag')` starts the drag using the press context
(`lastPressSrc` / `lastPressFrame`) captured in `pointerDown`.
- Stats: `videoYaw`, `videoPitch`, `videoAngleLive`, `videoRotMode`, `videoRebuilds`,
`videoReset`, `viewDrag`, `viewTaps`.
---
# ADDENDUM 8b — live picture pan: two mechanisms, the crash, and the real gesture
Owner feedback on ADDENDUM 8:
> 触发复位后直接崩溃了;另外,我拖动后,我希望画面立刻就有反应,而不是我松手之后才改变。
> 按住0.5s后拖动才是拖动,否则算复位
## 1. Why the picture only moved on release
ADDENDUM 8 decided whether it could write `layer.transform` with this probe:
```js
layer.transform = new XRRigidTransform(origin, q);
var o = layer.transform.orientation;
live = Math.abs(o.y - q.y) < 1e-4 && Math.abs(o.w - q.w) < 1e-4;
```
Comparing **quaternion components** is sign-sensitive — `q` and `−q` are the same
rotation — and a runtime is free to hand back a renormalised quaternion. So the probe
could report "read-only" for a runtime that was perfectly capable of the write. When it
did, the drag path took the rebuild branch, and because that branch was throttled to
"on release only", the picture sat still until the finger came up. That is exactly what
was observed.
The probe now compares the **position**, which is exact and has no sign ambiguity:
```js
layer.transform = new XRRigidTransform({x:0.123, y:-0.456, z:0.789}, identity);
writable = layer.transform.position matches that marker;
```
## 2. Two mechanisms, chosen automatically
```js
videoRotStrategy() -> 'transform' // preferred: one cheap write per change
| 'space' // layer.space = refSpace.getOffsetReferenceSpace(q)
| 'rebuild' // manual escape hatch ("兼容" in the 2D panel)
```
- `transform` — post-creation `layer.transform` writes are honoured.
- `space` — they are not, so the aim lives in an **offset reference space** instead. A
layer's pose is recomputed from its space every frame, and `getOffsetReferenceSpace()`
is the recentering path every WebXR app uses, so this is the more likely of the two to
be fully implemented.
- The two are **mutually exclusive**: `createVideoLayer()` must NOT bake the transform in
while the probe is about to write it, or the aim would be applied twice. Only the
`rebuild` strategy bakes it in at creation.
- `layer.transform` is never written on the `space` path, and vice versa.
Switching is automatic per session; `stats.videoRotMode` reports what was chosen and
`stats.videoTransformWritable` reports the raw probe result. The 2D panel has
「画面旋转: 自动 / 兼容」 to force the rebuild path without a redeploy.
## 3. The crash
The reset path rebuilt the video layer, and it created the replacement **before**
destroying the old one. Two `XRMediaBinding` equirect layers bound to the same `<video>`
element at the same time, plus a second rebuild microseconds later (the release path
also rebuilt, then the reset rebuilt again), took the browser down.
Every rebuild now:
1. runs in a `setTimeout` **outside the XR frame callback** (never inline in a frame),
2. **destroys the old layer first**, then creates the replacement,
3. is throttled to at most one per `REBUILD_MIN_MS` (120 ms),
4. never queues more than one pending rebuild.
The `space`/`transform` strategies rebuild **not at all**, so in the normal case the
crash path is not even reachable.
Two more robustness fixes found while testing this:
- `videoSpaceFor()` used to let a throwing `getOffsetReferenceSpace()` escape. It runs
inside the XR frame loop, where an exception would stop the loop and freeze the
session. It is caught now, and the strategy falls back to `rebuild`.
- The whole per-frame body (`updatePointer` / `pollGamepads` / `updatePanelPose` /
`updateGrabFrame` / `updateViewDrag` / frame hooks) is wrapped in a try/catch. A single
bad frame could previously kill the loop: `requestAnimationFrame` is only re-armed at
the end of the callback, so any throw froze the session permanently.
## 4. The real gesture
```
press the drag strip
├─ release before 500 ms, no movement -> RESET to dead ahead
├─ held past 500 ms, then moved -> PAN (picture follows the hand)
└─ held past 500 ms, never moved -> RESET
```
- The pan reference (`a0`) is captured **at the moment the hold threshold passes**, not
at the press. Otherwise the half second the user spends holding would fling the picture
the instant dragging armed.
- A ~1° dead zone keeps hand jitter from turning a reset tap into a tiny pan.
- The strip now reads `点一下复位 · 按住 0.5s 再拖动` instead of the retired triple-tap hint.
`stats.viewDrag` walks through it: `pressed (hold 500ms to pan)` →
`armed (drag to pan)` → `ended (moved)` / `reset (tap)`.
## 5. Verification
Three offline suites, all against the real sources (`build.py --check` passes too):
`test-ad8` — quaternion + panel geometry: aim for ±yaw/±pitch, unit norm, exact
azimuth/elevation recovery, yaw wrap, pitch guard, "aimed point stays under the cursor";
panel bands on the 1440×520 canvas (strip 0–0.20, inert readout 0.20–0.30, seek
0.30–0.52, media 0.52–0.74, subtitle 0.84–1.0, no dead column); the strip's drawn
readout; `hold()`.
`test-gesture` — the state machine above, driven with a stubbed ray: quick tap resets;
hold-without-movement resets; movement inside the hold window is ignored (no fling);
after arming a +10°/+5° ray move yields exactly +10°/+5°; a real pan does not reset; the
dead zone is respected; pitch cannot pass the guard.
`test-mech` — the mechanism: the position-based probe returns true for a writable
transform and false for an ignored one; the transform strategy writes and schedules no
rebuild; the space strategy assigns an offset space, leaves the transform alone (no
double rotation) and schedules no rebuild; unchanged angles reuse the cached space; the
rebuild strategy queues at most one, and its observed order is
`removeLayer -> createVideoLayer -> addLayer`; a throwing/absent
`getOffsetReferenceSpace()` is reported and permanently falls back to `rebuild`.
# ADDENDUM 9 — subtitle pipeline survival + normal (mono) subtitles
Two independent defects, found on device with the VR session left in its broken state.
## 1. The rAF broker lost Jellyfin's subtitle loop at session start
`90-main.js` no longer just *forwards* to native `requestAnimationFrame` outside XR. It
**always** holds callbacks in its own queue and drives them with one of two drivers:
- outside XR: a native rAF driver that keeps **exactly one** native callback in flight
(`nativeRafStep` re-arms *before* draining and cancels itself when the queue empty);
- inside XR: `video.requestVideoFrameCallback`, plus a 50 ms idle fallback so a paused /
stalled / ended video still redraws (paused seek, subtitle track switch).
Why: `requestSession('immersive-vr')` stops native rAF completely. Jellyfin 12.x renders
subtitles with SubtitlesOctopus, whose main thread starts **one long-lived rAF loop**
(`rafId`) when the video loads, draws the canvas from it and requests the next prerender
from the worker (`oneshot-render`). A callback still pending in the *native* scheduler when
the session starts is destroyed with it, and because `rafId` stays non-zero the loop is
never restarted.
Evidence (live instance, via CDP closure inspection):
```
rafId : 1150 <- native id, not the plugin's 1e9 range
oneshotState.eventStart: 945.99 <- canvas frozen ~7 min behind the playhead
renderRequested : false
oneshot-render requests during 15 s of playback: 0
```
After the fix, in a live VR session: `rafId = 1000028035`, `eventStart` follows
`video.currentTime`, and `stats.subUploads` climbs while subtitles change.
`startRafPump()` must **not** clear the queue (that callback is the one being rescued) and
must drop the stale `nativeId`. `stopRafPump()` re-arms the native driver.
## 2. Auto dual-eye / mono subtitle layout
`10-subtitles.js` used to assume the libass canvas is a stereo left/right pair and always
sampled the **left half**. A normal subtitle (e.g. the `CN` SRT track) is centred across the
whole 2:1 frame, so half of it was cut off.
`state.subLayout` (`'auto' | 'dual' | 'mono'`, default `'auto'`, persisted) selects the
region; `stats.subLayout / subDual / subEyeW / subCopyND / subCenterInk` report it.
Per sample (on the 320×160 downsample) the module builds the ink box and column count of
each half plus the ink in a ±40 source-px band around the seam, then:
```
only the left half has ink -> dual (sample the left half)
only the right half has ink -> mono (sample the whole width)
ink crosses the seam -> mono
both halves, seam empty -> columnCopyND < COPY_TOL ? dual : mono
```
**Dual-eye is NOT a pixel mirror.** Both eyes carry the *same* text at mirrored *positions*;
the glyphs are not flipped, so `left[x]` vs `right[W-1-x]` fails badly (measured mean delta
258/1020 → "not a mirror"). What is true is that the two halves' ink **column profiles**
match under a small horizontal shift, because they are the same line. Measured on device:
| layout | centre-band ink | column-copy distance |
|---|---|---|
| dual-eye (`CN_dual_eye`) | 0 | 0.02 – 0.08 |
| mono (`CN` SRT) | 50 / frame | 1.00 |
`COPY_TOL = 0.25`. A layout change is debounced (two consecutive samples) because it
rebuilds the layer.
`M.eyeW` is one eye's region width (half for dual, full for mono); `computeTexDims()`,
`drawHud()`'s crop/clamp and the texture scale all use it instead of the old hard-coded
`halfW`.
### API additions
```js
__JFVR.Subtitles.setLayout('auto' | 'dual' | 'mono') // also state.subLayout
__JFVR.Subtitles.getLayout(): string
```
## 3. Verification
- Broker logic replicated exactly in a Node harness against the built bundle:
`maxConcurrentNative = 1` over 300 frames with a self-rescheduling loop (the first
version re-armed during the drain and doubled the driver every frame — that froze the
page); simulating session start (native callbacks dropped) leaves the loop in the queue
and it keeps firing from the XR driver.
- `columnCopyND` extracted from the built bundle and exercised with the on-device
measurements: dual 0.00–0.08 → `dual`, mono centre-ink → `mono`, no-copy word-gap
profile 0.41 → `mono`.
- On device (before the UI became unreachable): the broker change was observed to carry
SubtitlesOctopus's loop through `immersive-vr` (`rafId` in the plugin's id range,
`eventStart` tracking, `subUploads` 4 → 7 over 20 s in VR).
# ADDENDUM 10 — normal (non-ASS) subtitles: Jellyfin does not use libass for them
## 1. Why normal subtitles were invisible
The HUD samples `canvas.libassjs-canvas`. That canvas only exists when Jellyfin
routes the subtitle through **SubtitlesOctopus (libass)**. From
`htmlVideoPlayer-plugin` (`renderTracksEvents`):
```js
a = (n.Codec || '').toLowerCase();
if (!X.includes(a)) { // X = ['ssa','ass']
if (a === 'pgssub') -> renderPgs
if (Y.includes(a)) -> renderVobSub // Y = ['dvdsub','vobsub']
if (!u(h.fe)) -> renderSubtitlesWithCustomElement // DOM overlay
/* else falls through to the native TextTrack path */
}
return renderSsaAss(...); // ssa/ass ONLY
```
So `subrip`/`vtt`/… never reach libass. Observed on device with the `CN` SRT track
selected: `canvas.libassjs-canvas` count **0**, `.libassjs-canvas-parent` 0,
`.videoSubtitles` 0 — and `video.textTracks[0] = { kind:'subtitles', label:'manualTrack',
mode:'showing', cues:315 }`, i.e. the **browser** is rendering the cues through
`::cue`. There was nothing for the plugin to read, at any layout setting.
## 2. What the module does now
`updateSource()` runs every XR frame and picks the source:
- a `canvas.libassjs-canvas` with a non-zero size → `srcKind = 'libass'` (unchanged path);
- otherwise, if the player has a showing TextTrack or a `.videoSubtitlesInner`
overlay → `srcKind = 'native'`. The active text comes from
`textTracks[i].activeCues` (fallback: the DOM overlay's `innerText`), is cleaned
of ASS/HTML tags and bidi marks, wrapped, and painted — white with a black outline,
centred — into a private 1920×1080 canvas. That canvas is fed through the exact
same crop/bbox/upload code as the libass one.
- neither → `srcKind = 'none'` and the strip is cleared once.
`prepare()` is a no-op without a libass canvas (blowing up the player box is only
needed so libass re-renders bigger; the native canvas is ours).
New diagnostics: `stats.subSource` (`'libass' | 'native' | 'none'`),
`stats.subNativeText`; `__JFVR.Subtitles.getSourceKind()`.
## 3. The persisted layout override was a trap
ADDENDUM 9 persisted `state.subLayout`. A stale `'mono'` in `jfvr.settings.v1`
silently forced the mono branch on the default **dual-eye** content: `dual:false`,
`eyeW:4320`, and a crop spanning the seam, so the HUD showed both eyes' text side by
side. Debugging that cost more time than the detection itself.
`subLayout` is therefore **not persisted** (same reasoning as `subtitles` and
`subMode:'off'` — a stored value that leaves the layer looking broken). It is
`'auto'` at boot, toggleable per session from the panel.
## 4. Verification (on device, build `hud-9`)
| selected track | `subSource` | result |
|---|---|---|
| `CN_dual_eye` (ass, idx 0) | `libass` | `dual:true`, `eyeW:2160`, `crop:[1040,768,1176,924]` (left half), `copyND:0.055`, uploads climbing |
| `CN` (subrip, idx 1) | `native` | `dual:false`, `eyeW:1920`, canvas 1920×1080, `text:"进去了呢"`, `crop` 256×88, uploads climbing |
The private cue canvas was read back and OCR'd ("怎么样? 好好地" for cue
"怎么样?好好地"), and injecting/removing a fake `canvas.libassjs-canvas` flips
`subSource` between `libass` and `native` with no errors.