PipelineBridge
A bridge to asynchronously pass data between Media and the RTC pipeline.
maxsize selects the buffering policy: depth 1 is latest-wins (video wants the freshest frame), a deeper bound acts as a short drop-oldest FIFO (audio wants continuity so a brief consumer stall doesn't silently drop samples).
A drop happens upstream of RTP: no sequence number is spent, so the
receiver sees no gap and never asks for a keyframe, while every delta
frame behind the drop references a picture it never received. With
request_keyframe bound the bridge keeps the wire decodable the way the
websockets relay does. A frame that names what it predicts from is
dropped with a word to the encoder (invalidate_reference), which then
predicts past it, and only the frames predicting from a dropped one are
held back, so the stream resumes on the next frame without a keyframe.
The ids let go are forgotten once a frame the encoder began after the
latest word goes out: frames reach the bridge in encode order, so nothing
coded before the word can follow it, while an id recurs every 65536
frames, which an infinite GOP outlasts. A
second frame held back that way although its encode began after the word
went out says the encoder never heard it (a word sent while the capture is
still starting is lost) and will keep predicting from what the bridge let
go, so a keyframe is asked for, as for a closed gate, until one arrives.
The first such frame proves nothing, since the encoder reads its words
just before it stamps an encode, and the frames it coded before the word
are held without one however far the loop lags. A frame without an encode
instant counts instead, past LOST_CHAIN_FRAMES in a row. A
frame that names nothing closes a gate that holds delta frames back, a
keyframe is asked for until one arrives and reopens it, and a queued
keyframe is never evicted by a delta frame. A gate no keyframe answers
within GATE_TIMEOUT_S reopens on its own.
A bridge given its display's steered rate (set_budget) also holds the
stream to it. An encoder meets a rate by coarser quantizers, down to the
coarsest it has, and content the coarsest cannot fit (noise, film grain)
overshoots, which a path sized for the rate queues and loses, and
whatever repairs the loss adds more. The frames let through fill a bucket
that drains at the rate, and while it holds more than BUDGET_WINDOW_S of
the rate a delta frame that names what it predicts from is dropped with
the word to the encoder, so the frame rate falls instead of the latency
rising, by no more than BUDGET_RUN_MAX frames in a row from an encoder that
has sent no anchor: one predicting from its last frames only codes a key
frame past a longer run (x264 from its last eight), and the key frame
overshoots the rate again, so the stream is thinned, not cut. Keyframes pass uncharged: charged, a large one would drop the
delta frames behind it for as long as it takes on a slow path. Neither is
a frame FRAME_NUM_WRAP frames or a multiple of it past a keyframe dropped:
an H.264 decoder that misses the frame whose frame_num wraps to 0 cannot
be predicted past the gap, so the encoder answers such a drop with a
keyframe. Queued, such an H.264 frame is not let go for a newer frame
either: the newer one is, as behind a keyframe.
Attributes
attributedropped= 0attributeinvalidated= 0attributepassed= 0attributeover_budget= 0Functions
constructor__init__(maxsize=1, request_keyframe=None, clock=time.monotonic, invalidate_reference=None) -> NoneInitializes the bridge.
Source Code
def __init__(self, maxsize: int = 1,
request_keyframe: Optional[Callable[[], None]] = None,
clock: Callable[[], float] = time.monotonic,
invalidate_reference: Optional[Callable[[int], None]] = None) -> None:
"""Initializes the bridge.
Args:
maxsize: Queue depth; 1 means latest-wins, larger is drop-oldest.
request_keyframe: Asks the display's encoder for a keyframe, on
the loop thread, at most once per IDR_REQUEST_FLOOR_S while
the gate is closed. None leaves every item ungated: audio
samples are self-contained.
clock: Monotonic time source.
invalidate_reference: Tells the display's encoder a frame id was
dropped, so the frames after it stop predicting from it. None
gates every drop behind a keyframe.
Attributes:
dropped: Items discarded since construction, evicted for a newer
one or held back behind a closed gate. None of them spent a
sequence number, so they appear in no loss statistic on either
side; this counter is what separates a lagging sender from a
lossy link.
invalidated: The share of those the encoder was told to predict
past, the rest being frames that predicted from one already
dropped and so needed no word of their own.
"""
self._queue: asyncio.Queue = asyncio.Queue(maxsize=maxsize)
self._request_keyframe = request_keyframe
self._invalidate = invalidate_reference
self._clock = clock
self._queued_keyframe = False
self._gated_at: Optional[float] = None
self._last_request: Optional[float] = None
# Frame ids dropped recently, which nothing delivered may predict from.
self._lost: deque = deque(maxlen=LOST_FRAME_MEMORY)
# When the encoder was last told of a drop, on the clock the capture stamps
# encode instants with, and the frames held since then that it coded after
# that or that carry no encode instant.
self._told_at: Optional[float] = None
self._held = 0
self.dropped = 0
self.invalidated = 0
self._budget_bps: Optional[float] = None
self._budget_bytes = 0.0
# Budget drops in a row, and whether the encoder has sent an anchor (`BUDGET_RUN_MAX`).
self._budget_run = 0
self._anchored = False
# Delta frames naming their reference let through, beside `over_budget`, the ones the budget dropped.
self.passed = 0
self._budget_at = 0.0
self.over_budget = 0
# Frames the encoder coded since the last keyframe, as they arrive.
self._since_key = 0parammaxsizeint= 1Queue depth; 1 means latest-wins, larger is drop-oldest.
paramrequest_keyframeOptional[Callable[[], None]]= NoneAsks the display's encoder for a keyframe, on the loop thread, at most once per IDR_REQUEST_FLOOR_S while the gate is closed. None leaves every item ungated: audio samples are self-contained.
paramclockCallable[[], float]= time.monotonicMonotonic time source.
paraminvalidate_referenceOptional[Callable[[int], None]]= NoneTells the display's encoder a frame id was dropped, so the frames after it stop predicting from it. None gates every drop behind a keyframe.
Returns
Nonefuncset_budget(bps) -> NoneHold the frames let through to bps, the display's steered rate;
None lets every frame through.
Source Code
def set_budget(self, bps: Optional[float]) -> None:
"""Hold the frames let through to `bps`, the display's steered rate;
None lets every frame through."""
if not bps:
self._budget_bps = None
self._budget_bytes = 0.0
return
if self._budget_bps is None:
self._budget_at = self._clock()
self._budget_bps = float(bps)parambpsOptional[float]Returns
Nonefunc_budget_level() -> floatThe bytes the frames let through still stand at, drained to now.
Source Code
def _budget_level(self) -> float:
"""The bytes the frames let through still stand at, drained to now."""
now = self._clock()
self._budget_bytes = max(0.0, self._budget_bytes - (now - self._budget_at) * self._budget_bps / 8.0)
self._budget_at = now
return self._budget_bytesReturns
floatfunc_budget_full() -> boolWhether the frames let through stand past BUDGET_WINDOW_S of the rate.
Source Code
def _budget_full(self) -> bool:
"""Whether the frames let through stand past BUDGET_WINDOW_S of the rate."""
return (self._budget_bps is not None
and self._budget_level() > self._budget_bps / 8.0 * BUDGET_WINDOW_S)Returns
boolfunc_charge(item) -> NoneSource Code
def _charge(self, item: Any) -> None:
if self._budget_bps is not None:
self._budget_bytes = self._budget_level() + _frame_bytes(item)paramitemAnyReturns
Nonefuncset_data(data, keyframe=True) -> NoneEnqueue an item, dropping the oldest one when the queue is full.
Synchronous, no lock: the checks and the put have no await, so the single-threaded loop runs them without interleaving (all access is on the loop thread). A full queue means the consumer is lagging.
Source Code
def set_data(self, data: Any, keyframe: bool = True) -> None:
"""Enqueue an item, dropping the oldest one when the queue is full.
Synchronous, no lock: the checks and the put have no await, so the
single-threaded loop runs them without interleaving (all access is on
the loop thread). A full queue means the consumer is lagging.
Args:
data: The item.
keyframe: Whether the item decodes on its own. A delta frame needs
the frame it predicts from on the wire: one that names it is
held back only while that frame was dropped here, and one that
does not is held back behind any drop until the next keyframe.
"""
queue = self._queue
if self._request_keyframe is None:
if queue.full():
queue.get_nowait()
self.dropped += 1
queue.put_nowait(data)
return
if keyframe:
if queue.full():
queue.get_nowait()
self.dropped += 1
queue.put_nowait(data)
self._queued_keyframe = True
self._gated_at = None
self._lost.clear()
self._held = 0
self._since_key = 0
self._budget_run = 0
return
self._since_key += 1
dependency = data.dependency if self._invalidate is not None else None
if dependency is not None:
frame_id, reference = dependency
if reference in self._lost:
self._hold(frame_id, getattr(data, "timing", None))
return
kept = (str(getattr(data, "codec", "") or "").lower() == "video/h264"
and not self._since_key % FRAME_NUM_WRAP)
self._anchored = self._anchored or bool(getattr(data, "anchor", False))
if (self._since_key % FRAME_NUM_WRAP and self._budget_full()
and (self._anchored or self._budget_run < BUDGET_RUN_MAX)):
self.over_budget += 1
self._budget_run += 1
self._drop(data)
return
self._budget_run = 0
if queue.full():
if self._queued_keyframe:
self._drop(data)
return
self._drop(queue.get_nowait())
if reference in self._lost:
self._hold(frame_id, getattr(data, "timing", None))
return
queue.put_nowait(data)
self._charge(data)
self.passed += 1
self._queued_keyframe = kept
self._held = 0
if self._lost and self._told_at is not None:
encoded = _encode_start(getattr(data, "timing", None))
if encoded is not None and encoded > self._told_at:
self._lost.clear()
return
now = self._clock()
if self._gated_at is not None:
if now - self._gated_at < GATE_TIMEOUT_S:
self.dropped += 1
self._ask(now)
return
logger.warning("Video bridge: no keyframe within %.1fs of a drop; "
"sending delta frames again", GATE_TIMEOUT_S)
self._gated_at = None
if queue.full():
self.dropped += 1
if not self._queued_keyframe:
queue.get_nowait()
self.dropped += 1
self._gated_at = now
self._ask(now)
return
queue.put_nowait(data)
self._queued_keyframe = FalseparamdataAnyThe item.
paramkeyframebool= TrueWhether the item decodes on its own. A delta frame needs the frame it predicts from on the wire: one that names it is held back only while that frame was dropped here, and one that does not is held back behind any drop until the next keyframe.
Returns
Nonefunc_ask(now) -> NoneSource Code
def _ask(self, now: float) -> None:
if self._last_request is not None and now - self._last_request < IDR_REQUEST_FLOOR_S:
return
self._last_request = now
self._request_keyframe()paramnowfloatReturns
Nonefunc_drop(item) -> NoneLet a frame go and tell the encoder, so nothing later predicts from it.
Source Code
def _drop(self, item: Any) -> None:
"""Let a frame go and tell the encoder, so nothing later predicts from it."""
self.dropped += 1
self.invalidated += 1
self._held = 0
self._told_at = self._clock()
self._lost.append(item.dependency[0])
self._invalidate(item.dependency[0])paramitemAnyReturns
Nonefunc_hold(frame_id, timing) -> NoneHold back a frame predicting from one already let go, and ask for a keyframe when the encoder coded it after it was told of the drop.
Source Code
def _hold(self, frame_id: int, timing: Optional[tuple]) -> None:
"""Hold back a frame predicting from one already let go, and ask for a
keyframe when the encoder coded it after it was told of the drop."""
self.dropped += 1
self._lost.append(frame_id)
encoded = _encode_start(timing)
if encoded is None:
self._held += 1
unheard = self._held > LOST_CHAIN_FRAMES
elif self._told_at is not None and encoded > self._told_at:
self._held += 1
unheard = self._held > 1
else:
unheard = False
if unheard:
self._ask(self._clock())paramframe_idintparamtimingOptional[tuple]Returns
Nonefuncempty() -> boolSource Code
def empty(self) -> bool:
return self._queue.empty()Returns
boolfuncget_data() -> AnyWait until an item is available in the queue and return it.
Source Code
async def get_data(self) -> Any:
"""Wait until an item is available in the queue and return it."""
return await self._queue.get()Returns
typing.Any