selkies-ws-core

WebSocket streaming core: the page-side half of the WebSocket transport, started by selkies-core.js when the stored stream mode is websockets.

One socket at <route prefix>/api/websockets carries the whole session. It is read in a worker (SOCKET_WORKER_SRC, reached through WorkerWebSocket), which passes audio straight to the decoder and the playback worklet, routes video straight to the video worker while it decodes -- full frames through one decoder, the striped modes decoded per row and composited there, acked from the socket worker (received id for full frames, presented id for stripes) -- and carries the microphone and webcam encoders' frames out over their own ports: nothing occupying the page's thread can interrupt any of them. Binary messages are typed by their first byte. From the server: 0x01 audio (Opus, with the RED redundancy layout documented on extractOpusFrames), 0x03 a JPEG stripe (u8 reserved, u16 frame id, u16 stripe Y, JPEG data), 0x04 a video stripe or full frame (u8 codec and frame kind, u16 frame id, u16 stripe Y, u16 width, u16 height, u16 the id of the frame it predicts from -- its own id when it predicts from nothing or the encoder does not say -- then the coded data), and 0x05 a gzip-wrapped control text once the client advertised _gz,1. From the client: 0x02 microphone Opus, 0x06 webcam frames (startWebcamCapture), and 0x05 gzipped large text once the server echoed _gz,1. Text messages are control. The client sends SETTINGS,{json}, r,WxH,displayId, START_VIDEO, STOP_VIDEO, START_AUDIO, STOP_AUDIO, REQUEST_KEYFRAME, LOST_FRAME <id> (a frame the decoder dropped, which the encoder then predicts past), CLIENT_FRAME_ACK <id> <heldMs>, cr, REQUEST_CLIPBOARD, the chunked clipboard upload of lib/clipboard-worker-bridge.js, cmd,<command>, SET_NATIVE_CURSOR_RENDERING,<0|1>, vp,<originX>,<originY>,<scaleX>,<scaleY> (this page's stream box on the user's desktop, relayed to the other displays) and the input verbs of lib/input.js. The server sends MODE websockets, AUTH_SUCCESS,{json}, ROLE_UPDATE,{json}, MK_ACCESS,<0|1>, VIDEO_STARTED, VIDEO_STOPPED, AUDIO_STARTED, AUDIO_STOPPED, AUDIO_DISABLED, MICROPHONE_DISABLED, WEBCAM_DISABLED, WEBCAM_KEYFRAME, CAPTURE_DEMAND <subject> <0|1>, PIPELINE_RESETTING <display>, DISPLAY_CONFIG_UPDATE,{json}, cursor,{json}, system,{json}, KILL <reason>, the clipboard family (clipboard,, clipboard_binary,, clipboard_start,, clipboard_data,, clipboard_finish, clipboard_reply,), and JSON objects typed server_settings, server_apps, pipeline_status, stream_resolution, stream_info and stream_stats (lib/stream-stats.js).

Video is decoded with WebCodecs: a JPEG stripe through ImageDecoder, an H.264 stripe through a VideoDecoder per row offset -- in the video worker while the divert holds, on the page as the fallback -- and a full frame -- controller or shared viewer alike -- in the video worker or through the row-0 stripe decoder. Decoded frames reach the screen through the first sink available: a track generator feeding a <video>, the worker's OffscreenCanvas, or the page canvas, with the striped modes composited on a back-buffer and blitted whole at frame boundaries. Audio is decoded in a worker and played through an AudioWorklet, the microphone is encoded to Opus in a worker, and the webcam is lib/webcam-capture.js.

Dashboards talk to the core over same-origin window messages. The core handles setVolume, setMute, setScaleLocally, setSynth, showVirtualKeyboard, setUseCssScaling, setAntiAliasing, setUseBrowserCursors, setRawPointerMotion, setManualResolution, resetResolutionToWindow, settings, getStats, clipboardUpdateFromUI, clipboardImageUpdate, pipelineStatusUpdate, pipelineControl, audioDeviceSelected, gamepadControl, requestFullscreen, command, touchinput:trackpad, touchinput:touch, sidebarVisibilityChanged and statsOpen, and posts pipelineStatusUpdate, sidebarButtonStatusUpdate, serverSettings, systemApps, stats (to the parent window), clientRoleUpdate, effectiveCursorState, scalingDpiFollowed, trackpadModeUpdate, clipboardContentUpdate, the clipboard preview of lib/clipboard-sync.js, fileUpload, toggleDashboard and toggleTouchGamepad. The window globals it publishes for the dashboards and the tests are webrtcInput (the Input handler), fps, videoChunksReceived, videoDivertOn, videoStripeRows (the row layout the video worker is decoding), webcamCodec, stream_info, stream_client and stream_stats (lib/stream-stats.js), currentAudioBufferSize, currentAudioBufferDuration, currentAudioLevel, currentAudioUnderrunSamples, currentAudioWorkletDropped, currentAudioDropped, manual_resolution, enable_resize, streamResolutionDiverged, isAudioInitializing, isFallingBack, isCleaningUp, applyTimestamp and selkiesTransport (the page-side handle on the session socket, which itself runs in a worker), plus one window[key] per server setting mirrored by sanitizeAndStoreSettings.

Settings are read from localStorage at init with fallbacks only and persist nothing, so a fresh profile keeps every key unset and server-pushed defaults stay re-pushable; only genuine user actions, and sanitizeAndStoreSettings for keys the user already overrode, write localStorage. Keys in PER_DISPLAY_SETTINGS carry a _display2 suffix on the secondary display.

Functions

audioTsNewer()

function audioTsNewer(a, b): boolean;

Defined in: selkies-ws-core.js:173

32-bit wrap-safe comparison of audio timestamps.

Parameters

Parameter	Type	Description
`a`	`number`	-
`b`	`number`	-

Parameter	Type	Description
`data`	`string` \| `ArrayBuffer` \| `Blob`	-
`mime?`	`string`	-
`onSkip?`	`Function`	Told why nothing was sent.

Parameter	Type	Description
`key`	`string`	-
`value`	`string`	-

Parameter	Type
`key`	`any`
`default_value`	`any`

Parameter	Type
`key`	`any`
`value`	`any`