Struct NvencEncoder 

pub struct NvencEncoder {

    encoder_session: *mut c_void,
    cuda_context: CUcontext,
    cuda_device: CUdevice,
    device_name: String,
    egl_display: *const c_void,
    codec: Codec,
    fullcolor: bool,
    bit_depth: u32,
    asked: (bool, bool),
    width: u32,
    height: u32,
    current_qp: u32,
    peak: Option<PeakBucket>,
    open_rc: NV_ENC_RC_PARAMS,
    last_avg_qp: u32,
    last_capped: bool,
    deferred: Option<BlockPlan>,
    refused_bind: Option<u32>,
    held_qp: Option<u32>,
    held_band: Option<(f64, f64)>,
    slice_layout: (u32, u32),
    hold_refused: bool,
    band_log: VecDeque<(u64, f64, f64)>,
    lost_cleanup: bool,
    cap: Option<usize>,
    refit: Option<f64>,
    held_bytes: usize,
    qp_map: Vec<i8>,
    refinement: Option<Refinement>,
    delta_mapped: bool,
    refine_refused: bool,
    last_quality: Option<u32>,
    last_bytes: Option<usize>,
    encode_config: NV_ENC_CONFIG,
    init_params: NV_ENC_INITIALIZE_PARAMS,
    input_device_ptr: CUdeviceptr,
    input_pitch: usize,
    input_format: NV_ENC_BUFFER_FORMAT,
    registered_input_resource: NV_ENC_REGISTERED_PTR,
    mapped_input_buffer: NV_ENC_INPUT_PTR,
    bitstream_buffers: Vec<NV_ENC_OUTPUT_PTR>,
    current_buffer_idx: usize,
    dmabuf_cache: HashMap<i32, CachedDmaBuf>,
    external_inputs: Vec<ExternalInput>,
    pinned_hosts: HashMap<usize, usize>,
    cuda: Arc<CudaFunctions>,
    egl: Option<Arc<EglFunctions>>,
    _nvenc_lib: Arc<NvencLibrary>,
    nvenc_funcs: NV_ENCODE_API_FUNCTION_LIST,
    omit_stripe_headers: bool,
    node_index: i32,
    pin_uploads: bool,
    direct_dmabuf: bool,
    csc: Option<ChromaConvert>,
    dpb: u32,
    engines: Option<i32>,
    references: Option<ReferenceWindow>,
    last_reference: Reference,
    frame_num: WideFrameNum,
    blit_semaphore: CUexternalSemaphore,
    band_hash: Option<Option<BandHash>>,
    upload_hash: Option<u32>,
    upload_words: Option<usize>,
    upload_hashes: Option<Vec<u64>>,
}

A live NVENC encoder session with its CUDA context and interop resources.

One instance owns a CUDA context bound to a specific GPU plus an NVENC session and everything the two input paths need:

• Packed path: a pitched device buffer (input_device_ptr / input_pitch) registered and mapped as the NVENC input (registered_input_resource / mapped_input_buffer) in the byte order input_format names (re-registered in place when a source of the other order arrives), fed either by a host→device upload or by the copy arm of the dmabuf path.
• Zero-copy dmabuf path: dmabuf_cache memoizes each fd’s EGLImage → CUDA import, keyed by fd but validated against the buffer’s DmaBufIdentity so a recycled fd re-imports; an import whose plane NVENC can take as it is — pitch-linear memory or a packed 8-bit CUDA array — is registered in place (DmaBufInput::Direct) unless direct_dmabuf was switched off, anything else is copied into the packed input each frame.

bitstream_buffers is a ring of BITSTREAM_BUFFERS output buffers (current_buffer_idx cycles it). pinned_hosts maps each page-locked host upload source’s base pointer to its registered length, with a 0 length recording a failed registration so that address is never re-pinned. codec and fullcolor name the session’s codec and negotiated chroma; current_qp tracks the live ConstQP so a paint-over reconfigure is skipped when unchanged. encode_config and init_params are retained so in-place reconfigure can resubmit them. omit_stripe_headers drops the wire header, and node_index is the effective CUDA device this session is bound to — a reuse across captures that now targets a different device must rebuild rather than reconfigure.

Fields

encoder_session: *mut c_void

cuda_context: CUcontext

cuda_device: CUdevice

device_name: String

The GPU’s marketing name, for the one line that says which device encodes.

egl_display: *const c_void

codec: Codec

fullcolor: bool

bit_depth: u32

The bits per sample the session codes.

asked: (bool, bool)

Whether the session was opened asking for 4:4:4 and for 10 bits, which a device without them codes at 4:2:0 and 8: a resize keeps the session for that ask or for what it codes, either of which opens the same session again.

width: u32

height: u32

current_qp: u32

peak: Option<PeakBucket>

A capped constant quantizer’s peak (peak_of), and the rate control it opened with, which a release restores.

open_rc: NV_ENC_RC_PARAMS

last_avg_qp: u32

The mean quantizer, in the device’s own domain, the last frame was coded at; 0 unknown.

last_capped: bool

Whether the last frame was coded bound to the peak (coded_capped), and the blocks the next frame holds at the coarsest quantizer (defer_blocks).

deferred: Option<BlockPlan>

refused_bind: Option<u32>

The peak’s rate, in bit/s, a bind to it was last refused at (bind_refused).

held_qp: Option<u32>

The quality index the next frame is held at whatever the rate control (hold_quantizer), the band of it that quantizer covers, and whether the driver has refused one, said once.

held_band: Option<(f64, f64)>

slice_layout: (u32, u32)

The (sliceMode, sliceModeData) of an HEVC session’s frames outside a band frame’s, which set_band_layout puts back.

hold_refused: bool

band_log: VecDeque<(u64, f64, f64)>

The bands held in recent frames, by the timestamp the device knows each frame by, and whether a frame that held none left some of them out of every prediction since the cleanup last asked (take_lost_cleanup).

lost_cleanup: bool

cap: Option<usize>

The bytes the next frame may come to (cap), and where the last was taken back for coming out past them and coded again, how far its held band then reached, or 0 (refit).

refit: Option<f64>

held_bytes: usize

Bytes of the last held frame, and the QP delta map a held band is coded through.

qp_map: Vec<i8>

refinement: Option<Refinement>

The blocks the next frame of the rate control’s refines (refine_blocks), whether the session’s rate control takes a QP delta map, and whether the driver refused one, said once, which ends refinement on the session.

delta_mapped: bool

refine_refused: bool

last_quality: Option<u32>

The quality index the rate control last coded a frame at (held frames aside), from the driver’s average quantizer.

last_bytes: Option<usize>

Bytes of the last frame the rate control coded, held frames aside.

encode_config: NV_ENC_CONFIG

init_params: NV_ENC_INITIALIZE_PARAMS

input_device_ptr: CUdeviceptr

input_pitch: usize

input_format: NV_ENC_BUFFER_FORMAT

registered_input_resource: NV_ENC_REGISTERED_PTR

mapped_input_buffer: NV_ENC_INPUT_PTR

bitstream_buffers: Vec<NV_ENC_OUTPUT_PTR>

current_buffer_idx: usize

dmabuf_cache: HashMap<i32, CachedDmaBuf>

external_inputs: Vec<ExternalInput>

pinned_hosts: HashMap<usize, usize>

cuda: Arc<CudaFunctions>

egl: Option<Arc<EglFunctions>>

The EGL entry points a dmabuf import takes, loaded for a session given an EGL display.

_nvenc_lib: Arc<NvencLibrary>

nvenc_funcs: NV_ENCODE_API_FUNCTION_LIST

omit_stripe_headers: bool

node_index: i32

pin_uploads: bool

Resolved once at init from PIXELFLUX_NVENC_PIN: page-lock the host upload sources so the copy is a direct pinned DMA rather than a pageable copy staged through a bounce buffer.

direct_dmabuf: bool

Resolved once at init from PIXELFLUX_NVENC_DIRECT: register pitch-linear dmabuf imports with NVENC in place instead of copying them into the packed input each frame.

csc: Option<ChromaConvert>

The 4:2:0 chroma convert, where the driver took the kernel and the session is not 4:4:4. None leaves NVENC’s own conversion in place.

dpb: u32

The decoded picture buffer the session declares, in frames, which a resize lowers where the new level admits fewer and never raises.

engines: Option<i32>

The device’s NVENC engines, which split_mode reads at every geometry.

references: Option<ReferenceWindow>

The frames the decoder holds, so a lost one can be left out of the predictions and each frame can name what it predicts from; None where the device cannot invalidate a reference, and nothing is tracked.

last_reference: Reference

frame_num: WideFrameNum

The H.264 stream’s frame_num a byte wider, where the session tracks its references: the frame where the counter wraps, which a loss covering it costs a key frame for, comes once in 65536 frames instead of 256.

blit_semaphore: CUexternalSemaphore

The X server’s blit semaphore imported into this context (blit_semaphore_fd), which encode_after_blit queues each frame’s wait on; null until one is made.

band_hash: Option<Option<BandHash>>

The band hash (band_hashes), loaded by the first frame that asks for one; Some(None) where the driver refused it.

upload_hash: Option<u32>

The bands of rows the next host upload is hashed in on the device (hash_next_upload), the words that hash left for the frame being coded, and its hashes once read back (take_upload_hashes).

upload_words: Option<usize>

upload_hashes: Option<Vec<u64>>

Implementations

impl NvencEncoder

fn load_egl() -> Result<EglFunctions, String>

Resolve EGL at runtime rather than link against it, so one binary boots even on hosts without EGL — it is needed only by the zero-copy dmabuf path — and reach the eglCreateImageKHR / eglDestroyImageKHR entry points through eglGetProcAddress because they are KHR extensions the base libEGL is not obliged to export as plain symbols. Erroring when the library or either extension is missing lets the caller fall back to another encoder instead of crashing at the first dmabuf import.

fn load_cuda() -> Result<CudaFunctions, String>

Resolve the CUDA driver library (libcuda.so.1, or nvcuda.dll on Windows) at runtime so the crate links against no CUDA SDK and still runs wherever a driver is installed, binding every cu* entry point up front so the per-frame hot path is plain indirect calls with no repeated symbol lookups. A missing symbol errors with its name, turning an ABI mismatch into a legible message instead of a later null-pointer call.

fn load_nvenc() -> Result<NvencLibrary, String>

Resolve libnvidia-encode at runtime for the same reason as CUDA — no SDK to link, runs against whatever driver ships — binding NvEncodeAPICreateInstance as the sole entry point every later encode call is reached through. NvEncodeAPIGetMaxSupportedVersion is kept optional and bound only when present, because very old drivers lack it; negotiation then falls back to probing create_instance acceptance directly rather than failing to load.

unsafe fn get_error_string(cuda: &CudaFunctions, err: CUresult) -> String

Turn a CUresult into the driver’s own error name via cuGetErrorName so a failure logs something diagnosable (e.g. CUDA_ERROR_OUT_OF_MEMORY) instead of a bare integer, falling back to the numeric code only when the name is unavailable.

unsafe fn probe_devices(cuda: &CudaFunctions)

Log the CUDA devices CUDA can enumerate — a debug aid when session init fails to find or bind the expected GPU.

unsafe fn device_name_of(cuda: &CudaFunctions, dev: CUdevice) -> String

The name CUDA gives a device, or a placeholder when it will not say.

fn get_pci_bus_id(render_index: i32) -> Option<String>

The PCI bus ID of the GPU behind /dev/dri/renderD<128+index>, read from the sysfs device symlink, so CUDA can bind to the same physical GPU the capture render node lives on.

pub fn new( settings: &CaptureSettings, egl_display: *const c_void, ) -> Result<Self, String>

Build a live NVENC session for the settings’ codec: bind CUDA to the target GPU, open and configure the encoder, and allocate its input and output buffers.

The sequence:

1. Load and negotiate: dlopen EGL / CUDA / NVENC, then nvenc_negotiate resolves the API version against the driver (set-once) before any struct is version-tagged. The multi-GPU GET_ATTACHED_IDS ioctl filter is installed after the NVIDIA libraries are loaded — so their GOTs can be patched — and before cuInit enumerates devices (a no-op unless a host GPU is hidden from this container). The three library Arcs are leaked once per process so the resolved function pointers stay valid for the program’s life.
2. Bind the device: cuInit, then bind by the render node’s PCI bus ID (encode_node_index, with auto <0 meaning device 0), falling back to CUDA device 0, and retain the device’s primary CUDA context — shared and refcounted across every session on that device rather than a fresh 100-300 MiB context each — pushing it current.
3. Allocate input: a pitched ARGB device buffer (cuMemAllocPitch, 16-byte element alignment) that the chroma convert, or hardware CSC, turns into YUV.
4. Open the session and query caps: create the function-list instance, open the session with the negotiated apiVersion, refuse a codec the device lists no engine for, and query nvEncGetEncodeCaps so init degrades rather than fails — a 4:4:4 request on a GPU or codec without it drops to 4:2:0, and a capture beyond the encoder’s max dimensions returns Err so the caller falls back to software. Then pull a preset config (P3, ultra-low-latency); a failed preset lookup logs the driver’s error string and proceeds with the zeroed default rather than aborting.
5. Configure the stream (mutating the returned preset config, whose version word is re-stamped while its embedded rcParams keeps the version the preset fill set): the codec’s 4:2:0 or 4:4:4 profile; CBR (two-pass quarter-resolution for tighter per-frame rate adherence, VBV sizing, optional min/max QP clamps in the codec’s quantizer domain) or ConstQP; infinite GOP (gopLength / idrPeriod = 0xFFFFFFFF); zeroReorderDelay and, for H.264, a bitstream-restriction VUI (max_num_reorder_frames=0) so no-reorder decoders don’t buffer; an explicit level from nvenc_level pinned from frame 1 so the level never bumps mid-stream; BT.709 primaries and transfer for the sRGB source, at limited range, with the matrix whichever conversion the session uses produces; a decoded picture buffer of as many frames as the level admits, so a frame a client lost can be left out of the predictions (invalidate_reference) with earlier frames still there to predict from; repeated parameter sets on every key frame; H.264 CABAC; no AUD; one AV1 tile; strict GOP target; and lookahead disabled for real-time latency.
6. Initialize with resize headroom: maxEncodeWidth / maxEncodeHeight are raised to at least HEADROOM_WIDTH x HEADROOM_HEIGHT so reconfigure_resolution can grow in place, but never past the driver’s reported maximum; this costs a few hundred MiB of device memory, so a failed init reopens the session and retries at the exact size (in-place resize then falls back to a rebuild).
7. Register, map, and buffer: register and map the packed input surface (as ARGB; set_input_format re-registers it for an RGBA source), and create a 4-deep ring of bitstream output buffers.

Every failure after the CUDA allocation unwinds the resources created so far — buffers, session, context — before returning Err. EGL is only needed by the zero-copy dmabuf path, so callers on the host-ARGB path pass a null egl_display. The retained init_params.encodeConfig raw pointer is nulled before the struct is returned (it points at a local config about to move); the reconfigure paths repoint it at self.encode_config when they resubmit.

pub(crate) fn new_tuned( settings: &CaptureSettings, egl_display: *const c_void, tuning: NvencTuning, ) -> Result<Self, String>

new with the preset, rate-control passes, and adaptive quantization named, for the tuning bench; production sessions take NvencTuning::default.

unsafe fn device_encodes( funcs: &NV_ENCODE_API_FUNCTION_LIST, session: *mut c_void, codec: &GUID, ) -> bool

Whether the open session’s device lists an encode engine for codec.

fn configure_codec( config: &mut NV_ENC_CONFIG, codec: Codec, fullcolor: bool, bit_depth: u32, level: u32, dpb: u32, anchors: usize, tuning: &NvencTuning, )

Program the codec-specific arm of config: level, an infinite IDR period, the chroma format, 8-bit input and output, parameter sets repeated on every key frame, and the color description every session converts with.

The whole description is BT.709 at limited range. Primaries and transfer describe the source, sRGB desktop pixels, which shares both with BT.709; the matrix is what ChromaConvert produces, and NVENC’s own conversion — the fallback, and the 4:4:4 sessions — follows the matrix declared here at the limited range it emits, which gpu_hardware_conversion_matches_the_declared_matrix holds it to. H.264 additionally restricts reordering in its VUI so no-reorder decoders don’t buffer, and codes CABAC. H.264 frames carry SLICES_PER_FRAME slices and HEVC frames one, an HEVC band frame its own (set_band_layout), and an HEVC session filters across no slice boundary, which is what keeps a band’s slice from softening the rows refined before it; AV1 asks for one tile, since tiles cost bitrate and buy no quality, and tier 0, the only tier NVENC takes for it. The driver honors a 1x1 request below 1986 pixels of picture height; above that it forces a second tile row whatever is asked for, in the low-latency presets this one is among but not from P5 up, so a 4K AV1 session codes two tile rows.

HEVC declares the tier h265_tier names for its level, High: NVENC validates a CBR target against the MaxBR of the pinned level, and the Main-tier ceiling of the 5.x levels (40 Mbit/s at 5.1) is one a 4K desktop session reaches, where a Main-tier open is refused and a live rate change past it is declined. The tier is a signaled cap, not a coding tool; it does change the codec string a client derives from the SPS (H153 for L153).

fn declared_level(&self) -> u32

The level the live config declares, read from its codec arm.

fn level_for(&self, width: u32, height: u32, fps: u32) -> u32

The level a width x height stream at fps takes at the live config’s peak bitrate and HEVC tier, and the driver’s AV1 share (NVENC_AV1_WEIGHTED).

fn set_level(&mut self, width: u32, height: u32, fps: u32)

Write the level for a new geometry or frame rate into the live config’s codec arm.

fn dpb_frames_at(&self, width: u32, height: u32, fps: u32) -> u32

The decoded picture buffer, in frames, the level a width x height stream at fps takes admits for this codec.

pub fn last_reference(&self) -> Reference

The frame the last encoded frame predicted from.

pub fn invalidate_reference(&mut self, frame_id: u16) -> bool

Leave frame frame_id and every frame after it out of the predictions. False when the device cannot or refuses, and the caller codes a key frame instead.

pub fn acknowledge_reference(&mut self, frame_id: u16, held: bool)

Every consumer holds frame frame_id, or where not held was sent it (ReferenceWindow::acknowledge).

pub fn take_report(&mut self, report: ReferenceReport) -> bool

Apply what the consumers say of a frame (FrameEncoder::take_report); false where a key frame is coded instead.

fn retract_last(&mut self) -> bool

Leave the frame just encoded out of the predictions as a frame the client lost is, for a frame coded again in its place (ReferenceWindow::retract); false where the window or the device cannot.

pub fn split_summary(&self) -> String

How the session splits a frame across the device’s encode engines (split_mode), for the line that says which device encodes.

pub fn device_name(&self) -> &str

The codec the session emits. The name CUDA gives the GPU this session encodes on.

pub fn codec(&self) -> Codec

pub fn is_full_range(&self) -> bool

Every NVENC session converts to and declares limited range.

pub fn is_fullcolor(&self) -> bool

Whether the session negotiated 4:4:4 chroma.

pub fn bit_depth(&self) -> u32

The bits per sample the session codes.

pub fn reconfigure_resolution( &mut self, settings: &CaptureSettings, ) -> Result<bool, String>

Follow a capture restart on the live session, folding in the current rate / QP / fps, without tearing it down.

The NVENC session, CUDA context, and bitstream buffers survive, so a restart costs a few milliseconds instead of a full rebuild. Flow:

1. Reject the unchangeable: a different encode device or codec, a flip of the chroma format or depth asked for (4:4:4, 10-bit; not of what a device without them codes), an RC-mode flip, or dimensions of zero or beyond the init-time maxEncode headroom all return Err so the caller rebuilds.
2. Keep the stream at unchanged dimensions: the reference chain, the input surface, and the dmabuf imports stay as they are, so the restart costs no IDR and no reset. Only the pinned hosts are dropped – the restart recreates the source buffers, often at the same addresses – and the rate, frame rate, and wire framing the restart carries are folded in, as reconfigure_rate does. Returns Ok(false).
3. Release geometry-dependent state under the pushed CUDA context: unmap / unregister / free the packed input surface, every cached dmabuf import (with the NVENC registration a direct import holds), and every pinned host. The dmabuf imports are re-created lazily by the encode path; pinned hosts are dropped because the source shm segments are recreated on resize and may reuse the same base addresses.
4. Reconfigure the session: update the level for the new size, the decoded picture buffer where that level admits fewer frames (the driver takes a smaller one at the forced IDR and refuses to raise it again), the CBR bitrate + VBV or the ConstQP, and the new dimensions / DAR / frame rate, then NvEncReconfigureEncoder with resetEncoder and forceIDR so the stream restarts cleanly at the new size. Driver rejection returns Err.
5. Reallocate the packed input at the new size and register + map it as init does, in the byte order the session was last fed.

On a resize the next encoded frame is a reset-RC IDR and Ok(true) is returned.

unsafe fn pin_host_source(&mut self, base: usize, len: usize)

Page-lock one host upload source’s base address once, under the already-current CUDA context, so the copy is a direct pinned DMA instead of a pageable copy staged through a driver bounce buffer. A 0-length entry records a failed registration so the address is never re-probed; the persistent, bounded shm / reused planar sources make this a one-time cost.

pub fn release_pinned_hosts(&mut self)

Drop every page-locked host registration, under the pushed CUDA context.

Called when the capture’s shm segments are recreated at unchanged dimensions, and when a Wayland encode thread hands its session on, before its pool is freed: new buffers often reuse the old base addresses, so a stale registration would alias fresh memory, and one freed while registered leaves CUDA the old length at that address. Subsequent uploads re-pin lazily. A 0-length entry marks a registration that failed and so is not unregistered.

unsafe fn release_dmabuf_import(&self, cache: CachedDmaBuf)

Tear down one cached dmabuf import under the already-current CUDA context, inner handle first: the NVENC mapping and registration a direct import holds, then the CUDA graphics resource, then the EGLImage it was built from. The encode that last read the import has completed (submit_frame waits for the bitstream), so nothing is still in flight on it.

unsafe fn set_input_format( &mut self, format: NV_ENC_BUFFER_FORMAT, ) -> Result<(), String>

Register the packed input surface with NVENC in the byte order format names, when it is not already: the surface memory is unchanged, only its registration (and mapping) is replaced, so a session fed first from one source order and then the other keeps one surface. Runs under the already-current CUDA context; a failed re-registration leaves the surface unregistered and returns Err, so the caller’s encode fails visibly instead of encoding swapped channels.

unsafe fn reconfigure(&mut self, reset: bool, force_idr: bool) -> NVENCSTATUS

Hand the live config to nvEncReconfigureEncoder, resetting the encoder and forcing an IDR as asked.

unsafe fn reconfigure_if_needed(&mut self, crf: u32) -> bool

Reconfigure the live session’s ConstQP when the quantizer the session quality index crf selects differs from the current one, returning whether a reconfigure actually happened.

A no-op in CBR mode (bitrate-controlled, so QP-based paint-over does not apply), save a capped constant quantizer bound to its peak, whose floor follows (peak_of), and when the QP is unchanged. When it does apply, the three constQP fields are updated and the session is reconfigured without a forced IDR: a lower-QP P-frame refines the static image against the existing reference chain (paint-over) with no intra-frame bitrate spike, so the GOP continues seamlessly across the reconfigure.

pub fn capped(&self) -> bool

Whether the constant quantizer runs under a peak (peak_of).

pub fn peak_bound(&self) -> bool

Whether the session is bound to that peak, coding through its constant rate.

pub fn coded_capped(&self) -> bool

Whether the last frame was coded bound to the peak.

pub fn idle(&mut self)

A tick that codes no frame (FrameEncoder::idle): the peak’s bucket drains, and a session bound to it counts the tick at its quantizer, nothing being coded coarser.

pub fn defer_blocks(&mut self, plan: Option<&BlockPlan>)

Hold the blocks plan names at the coarsest quantizer in the next frame of the rate control’s, through the QP delta map (FrameEncoder::defer_blocks): a block whose reference already holds it codes as a skip there.

pub fn last_quality(&self) -> Option<u32>

The quality index the rate control last coded a frame at, where the driver reports its quantizer; a held frame (hold_quantizer) is not the rate control’s and leaves it.

pub fn last_size(&self) -> Option<usize>

The bytes of the last frame the rate control coded; a held frame leaves it.

pub fn hold_quantizer(&mut self, crf: u32, band: Option<(f64, f64)>)

Encode the next frame at the constant quantizer the quality index crf selects, whatever the rate control, and leave the session’s own rate control and quantizer as they were for the frame after: the cleanup of a still screen. A held key frame of a constant-rate session that comes out past HELD_KEY_BUDGET_S of the target is coded again, as a key frame, at the coarser quantizer held_key_retry picks, and so is a held frame of the whole picture past HELD_REFRESH_LIMIT_S, predicted from the frame before the first attempt, which the device leaves out as it does a frame the client lost (retract_last). band, the share of the picture from and to in raster order, confines crf to the blocks it covers (band_size), the rest of the frame held at the coarsest quantizer, or for AV1 at the coarsest the delta map’s 128 steps reach above the band: a still region is left as it is at any quantizer, and a change the frame carries before its damage is known (X11 under Turbo hashes a frame beside its encode) costs what the rate control’s own frame would.

pub fn refines(&self) -> bool

Whether the session refines blocks through a QP delta map under its rate control (refine_blocks): H.264, HEVC, and AV1, until the driver refuses the map.

pub fn refine_blocks(&mut self, refinement: &Refinement)

Code the blocks refinement names in the next frame of the rate control’s at its target quality, through the frame’s QP delta map: each map unit (a 16x16 macroblock, a 32x32 coding tree block, a 64x64 superblock) holding a refined block takes the delta from the quantizer the rate control last coded at to the target, the rest 0. The rate control is put on the delta map mode the first time, by the reconfigure that takes the rate-control mode without a reset, and stays there.

unsafe fn block_map( &mut self, refinement: Option<&Refinement>, deferred: Option<&BlockPlan>, ) -> bool

Fill qp_map for the blocks the next frame refines (refine_blocks), each map unit holding one at the delta from the quantizer the rate control last coded at to the target, and for the blocks a refresh leaves for later frames (defer_blocks), each at the delta to the codec’s coarsest quantizer from the session’s; the rest 0. Returns whether the session codes through it, which a refused reconfigure ends for good.

unsafe fn delta_map_mode(&mut self) -> bool

Put the rate control on the QP delta map mode, by the reconfigure that takes the rate-control mode without a reset, the first time; false where the driver refused, which ends refining for good.

pub fn cap(&mut self, bytes: usize)

Take the next frame back where it comes out past bytes and code it again within them: a held band in a shorter one, to REFIT_SHARE of them at the bytes the band came to, and a frame of the rate control’s at the coarsest quantizer, which on a still screen refines nothing. The frame taken back is left out of the predictions as a frame a client lost is (retract_last); a key frame stands as coded.

pub fn refit(&self) -> Option<f64>

Where the last frame was taken back and coded again (cap): how far its held band then reached, or 0 for a frame of the rate control’s.

pub fn take_lost_cleanup(&mut self) -> bool

Whether a frame left a held band out of every prediction since this was last asked (an anchor predicting from the newest frame every client was sent, past a band coded since), where the frame held none to take it back into: the cleanup has to run again.

fn forget_bands(&mut self, pts: u64)

The frames from timestamp pts on have just been left out of every prediction: a band one of them held is coded again in the band this frame holds, which then reaches back to it, and is lost to a frame that holds none (take_lost_cleanup).

fn set_band_layout(&mut self, band: Option<usize>)

Keep the deblocking filter off the rows the bands before refined, on a frame holding a band whose first block is band (None on any other frame): at the edges a band shares with blocks held at the coarsest quantizer, the filter works at their mean quantizer and softens those rows (at 60 fps 0.5 Mbit/s a 1080p row from 44.5 dB to 33.9). H.264 turns the filter off for the frame, which its slice header carries. NVENC’s HEVC codes the same bytes whatever its switch says on a reconfigure, so its band frame starts a slice at the band instead, across whose boundary the session never filters (configure_codec), where that takes at most BAND_SLICES_MAX even slices; a band starting before them keeps the frame’s own. AV1 has neither.

pub fn band_size(&self) -> Option<usize>

The bytes of the last held frame (0 before one), where the session holds a band: H.264, HEVC and AV1, whose QP delta map is one entry a macroblock, a 32x32 coding tree block, or a 64x64 superblock.

unsafe fn hold_rate( &mut self, q: u32, band: Option<usize>, ) -> Option<NV_ENC_RC_PARAMS>

Put the session on constant quantizer q for the picture about to be submitted, with the picture’s QP delta map on top where it holds a band starting at block band (fill_band_map), answering the rate control to restore once it is encoded; None where the session is already there or the driver refused, which leaves the picture to the rate control.

The rate-control mode is one of the parameters nvEncReconfigureEncoder takes without a reset, so a constant-rate session keeps its HRD state across the held picture instead of starting over after it.

pub fn reconfigure_rate(&mut self, settings: &CaptureSettings) -> bool

Apply a runtime rate-control / frame-rate change to the live session, and report whether the session carries it afterwards.

In CBR mode the target bitrate, max bitrate, VBV, and its initial delay are updated (the VBV is ignored outside CBR); the target fps is updated in either mode. The session is reconfigured only when one of these actually changed — no RC reset, and a forced IDR only where a target past the declared level’s bitrate ceiling, or a frame rate past its macroblock rate, raises the level, which the decoder learns from the sequence header a key frame carries; a level a lower target or frame rate no longer needs stays, since a level only ever has to be high enough and the driver refuses one below the decoded picture buffer the session declared — so calling it every frame is cheap. A reconfigure the driver refuses leaves the session encoding at its previous rate, logged with the driver’s reason.

unsafe fn submit_frame( &mut self, mapped_buffer: NV_ENC_INPUT_PTR, buffer_format: NV_ENC_BUFFER_FORMAT, frame_number: u64, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode one mapped input picture and return its bitstream bytes behind the wire header.

The shared tail of all three encode paths:

1. Pick an output buffer from the ring (current_buffer_idx advances modulo the ring length) and submit the picture with nvEncEncodePicture; force_idr sets the force-IDR pic flag.
2. Lock the bitstream (nvEncLockBitstream, blocking) to read the encoded bytes. The lock has to block: on Linux a doNotWait lock answers an unfinished encode with NV_ENC_SUCCESS and an empty bitstream, not NV_ENC_ERR_LOCK_BUSY, so polling it would hand out empty frames.
3. Frame the output: unless omit_stripe_headers is set, prepend the wire header with the frame kind derived from the actual encoded pictureType (IDR = key, I = intra, else delta) rather than the force_idr request.
4. Emit: append the encoded bytes, unlock the bitstream, and return the framed buffer.

unsafe fn follow_peak(&mut self, bytes: usize, own: bool)

Fold the frame just coded, bytes long, into a capped constant quantizer’s peak (peak_of), and bind the session to the peak or release it where the frame says to; own is false for a held frame. A frame refining blocks through the QP map counts as the rate control’s: bound to a peak its content overflows, NVENC’s constant rate holds such a frame’s mean quantizer over the floor (none of 140 refining every block 25 steps finer came out at it, on an L4), and a session refining on every bound frame would otherwise never count one toward its release.

fn bind_refused(&self) -> bool

Whether a bind to the peak was refused at the peak’s rate as it stands (switch_peak), which is not asked for again until the peak moves.

unsafe fn switch_peak(&mut self, bind: bool)

Bind a capped constant quantizer to its peak, or release it to its constant quantizer (peak_of); a switch the driver refuses leaves the session as it was. A peak past the level the session declared releases it rather than raise the level, a key frame. A bind refused either way is not tried again at that peak (bind_refused).

fn level_for_peak(&self) -> u32

The level the live config takes at the session’s frame rate and size.

unsafe fn encode_picture( &mut self, pic_params: &mut NV_ENC_PIC_PARAMS, output_bitstream: NV_ENC_OUTPUT_PTR, frame_number: u64, held: bool, anchor: Option<u8>, ) -> Result<Vec<u8>, String>

Submit one picture and read its bitstream back behind the wire header: steps 1 to 4 of submit_frame past the choice of output buffer. A held picture leaves last_quality to the rate control’s own frames.

pub fn encode( &mut self, dmabuf: &Dmabuf, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a dmabuf frame zero-copy, by importing it through EGL into CUDA and, where the driver allows, handing the mapped plane to NVENC as its input.

Applies any pending ConstQP change, then works under the pushed CUDA context:

1. Import once, cache by fd with an identity check: the cache is keyed by the dmabuf fd but each entry stores the buffer’s DmaBufIdentity; an entry whose identity no longer matches (a recycled fd) is released first. On a miss, build an EGLImageKHR from the dmabuf’s fd / offset / pitch / modifier, register it as a CUDA graphics resource, map it to a CUeglFrame, and settle how it feeds the encoder: a first plane that direct_plane accepts (and direct_dmabuf on) is registered with NVENC in place — a pitch-linear plane as a CUDA device pointer at its own pitch, a four-channel 8-bit CUDA array as a CUDA array — in the byte order the dmabuf fourcc names, and mapped once (DmaBufInput::Direct); any other plane, or a registration the driver refuses, takes DmaBufInput::Copy. The result is memoized so a recurring capture buffer pays the import cost only once. Each failure destroys what it created and pops the context.
2. Feed the encoder: a direct import is submitted as it is — no copy at all. A copy import is copied with cuMemcpy2DAsync on the default stream — the array plane or the pitch-linear plane, per frame_type — into the packed input surface, re-registered in the dmabuf’s byte order when it differs; NVENC processes its input on that same stream, so the copy is ordered before the encode without a host wait.
3. Submit via submit_frame, then pop the context.

The dmabuf fd is read out before the context is pushed so an early ? return cannot leave the CUDA context stack imbalanced.

pub fn encode_after_blit( &mut self, dmabuf: &Dmabuf, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

encode, for a buffer the X server is still blitting into: the frame’s work is queued behind a wait on the semaphore the server signals once the blit has landed (blit_semaphore_fd), so it runs on the GPU right behind the blit instead of after the CPU has learnt of it. NVENC reads its input on the default stream, as the copy and the convert do, so one wait there orders all of it.

pub fn blit_semaphore_fd(&mut self) -> Result<OwnedFd, String>

A new semaphore for the X server to signal after a blit and encode_after_blit to wait on: made on this session’s GPU, imported into its context in place of any earlier one, and returned as the fd the server imports.

fn encode_frame( &mut self, dmabuf: &Dmabuf, frame_number: u64, crf: u32, force_idr: bool, after_blit: bool, ) -> Result<Vec<u8>, String>

unsafe fn register_direct_input( &mut self, frame: &CUeglFrame, plane: DirectPlane, format: NV_ENC_BUFFER_FORMAT, ) -> DmaBufInput

Register the first plane of a mapped dmabuf frame with NVENC in place — as a pitch-linear CUDA device pointer or as a CUDA array, per plane — and map it as an input, under the already-current CUDA context. Either step failing falls back to DmaBufInput::Copy — the per-frame copy then serves that import for as long as it is cached, so a driver that declines the direct path costs one failed registration, not a failed frame.

pub fn encode_cpu_argb( &mut self, argb: &[u8], src_stride: usize, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a host BGRA frame (B,G,R,A in memory, NVENC’s word-ordered ARGB — the layout an XShm grab or the pixman framebuffer yields) through encode_cpu_packed.

pub fn encode_cpu_packed( &mut self, pixels: &[u8], src_stride: usize, rgba_input: bool, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a host packed-pixel frame by uploading it straight into the packed input surface, with no CPU-side color conversion: the surface is either the chroma convert’s source or, where the kernel did not load, NVENC’s own conversion’s; a host prepass would cost this path its copy-free property.

rgba_input names the byte order — false for B,G,R,A (X11 XShm, the pixman framebuffer), true for R,G,B,A (a GLES readback) — and the input surface is registered with NVENC in that order (ARGB / ABGR, re-registered in place when it changes), so both arrive at the hardware CSC untouched. src_stride is the source row stride in bytes (>= width*4). Steps, under the pushed CUDA context after any pending QP change:

1. Bounds-check the source against stride × (rows-1) + width*4, erroring rather than reading past a short buffer.
2. Pin the source once: unless pinning was disabled at init (PIXELFLUX_NVENC_PIN=0, read once into pin_uploads), page-lock each distinct source base address via pin_host_source so the upload is a direct DMA from the caller’s buffer instead of a pageable copy staged through a driver bounce buffer. The persistent, bounded shm / pool sources make this a one-time bounded cost.
3. Upload and submit: cuMemcpy2DAsync the rows into the input surface on the default stream honoring src_stride, then submit_frame. NVENC processes its input on that same stream, so the upload is ordered before the encode without a host wait, and the blocking bitstream lock inside submit_frame (or the stream sync on its error path) guarantees the upload has finished reading pixels by the time this returns — the caller may reuse the buffer immediately.

pub fn encode_client_packed( &mut self, pixels: &[u8], src_stride: usize, rgba_input: bool, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

encode_cpu_packed for rows a client owns, such as a wl_shm pool’s, uploaded without page-locking them. The client unmaps its pool whenever it likes, and a registration outlives the mapping: a pool mapped later over the same addresses would be read from the old pool’s pages, or fail to upload where it straddles them.

unsafe fn convert_texture( &self, tex: CUtexObject, rgba_input: bool, ) -> Result<Option<(NV_ENC_INPUT_PTR, NV_ENC_BUFFER_FORMAT)>, String>

Run the chroma convert over an array-typed import through tex, answering the input NVENC should be handed. None where the session has no convert, or where the driver gave no texture for the import and NVENC’s own conversion stands in, which a 10-bit session, initialized for the convert’s surface, has none of.

unsafe fn convert_packed( &self, src: CUdeviceptr, src_pitch: usize, rgba_input: bool, ) -> Result<Option<(NV_ENC_INPUT_PTR, NV_ENC_BUFFER_FORMAT)>, String>

Run the chroma convert over a packed surface, answering the input NVENC should be handed, or None where the session has no convert and encodes the packed surface itself. The caller holds the CUDA context current.

pub fn width(&self) -> u32

The geometry the session is currently initialized for.

pub fn height(&self) -> u32

The geometry the session is currently initialized for.

pub(crate) fn push_context(&self) -> bool

Make this session’s CUDA context current on the calling thread, so a capture source can allocate its frames in the very context the encoder reads them from — the whole basis of a zero-copy hand-over. Paired with NvencEncoder::pop_context.

pub(crate) fn pop_context(&self)

Give back the context NvencEncoder::push_context made current.

pub fn release_external_input(&mut self)

Drop the registration of an externally-owned input surface, under the pushed CUDA context.

The caller owns the memory behind it, so this must run before that owner releases it: a registration outliving its buffer leaves the driver holding a mapping of freed video memory.

unsafe fn unmap_external_inputs(&mut self)

Unmap and unregister every external input, with the CUDA context already current.

unsafe fn unmap_one_input(&self, ext: ExternalInput)

unsafe fn register_external_input( &mut self, device_ptr: CUdeviceptr, pitch: usize, format: NV_ENC_BUFFER_FORMAT, ) -> Result<NV_ENC_INPUT_PTR, String>

The mapped NVENC input for an externally-owned device pointer, registering it in place the first time and reusing that registration for every later frame from the same buffer.

A capture source hands back one buffer, or a few in turn (NvFBC alternates between two on recent drivers), for as long as its geometry holds, so the register and map cost is paid once per buffer rather than per frame. Up to EXTERNAL_INPUTS stay registered; a pitch, format or geometry that changes releases the registrations made for the old one.

pub fn encode_cuda_pitch( &mut self, device_ptr: CUdeviceptr, pitch: usize, rgba: bool, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a frame that already lives in video memory, reading it exactly where it was produced.

This is the zero-copy hand-over: device_ptr addresses packed pixels in this session’s own CUDA context — the X11 NvFBC capture buffer the NVIDIA driver composited the screen into — so no upload or cuMemcpy stands between the screen and the bitstream. The chroma convert reads that buffer where it lies, as it does for every other input, and a session without one hands the packed surface to NVENC’s own conversion instead, registered in place. rgba names the byte order (false for the B,G,R,A the driver’s native format delivers), and pitch is the buffer’s row stride in bytes.

The caller must keep the buffer alive and unmodified until this returns; the blocking bitstream lock inside means the encoder has finished reading by then, so the next capture may overwrite it.

pub fn hash_next_upload(&mut self, rows: u32) -> bool

Hash the next host frame’s bands of rows rows on the device as it encodes, for take_upload_hashes to read after: the change detection a Turbo frame’s paint-over reads, for the cost of a kernel beside the encode instead of a CPU pass over the frame. False where the driver refuses the kernel, which leaves the hash to the caller.

pub fn take_upload_hashes(&mut self) -> Option<Vec<u64>>

The band hashes of the host frame last coded where hash_next_upload asked for them.

pub fn band_hashes( &mut self, device_ptr: CUdeviceptr, pitch: usize, width: u32, height: u32, rows: u32, ) -> Option<Vec<u64>>

The hash of each band of rows rows of the widthxheight packed frame at device_ptr (pitch bytes a row, in this session’s context), computed where the frame lies: the change detection of a capture that reports none, NvFBC’s. None where the driver refuses the kernel.

Trait Implementations

impl Drop for NvencEncoder

Release every GPU resource the session holds, in the one teardown order the drivers tolerate, so nothing leaks and no still-referenced handle is ever freed out from under the driver.

The whole sequence runs with the owning CUDA context pushed current, because the cuMemFree / cuGraphicsUnregisterResource / cuMemHostUnregister calls each act on the current context — pop it first and the frees silently do nothing, leaking device memory. Within that, resources go inner-handle before the outer handle that owns it, since freeing an owner first orphans or faults on what still points into it: unmap the packed input before unregistering it, free its device buffer, destroy the bitstream buffers, and release every cached dmabuf import (release_dmabuf_import: its NVENC mapping and registration, then the CUDA resource and the EGLImage) — all session-owned — before the encoder session itself, and destroy that session before releasing the device’s primary CUDA context it was opened against (the retain is refcounted, so the context lives until the last session on that device releases it). The page-locked host sources are unpinned in the same pass, each only when its recorded length is non-zero (a 0 marks a registration that failed and so was never pinned).

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for NvencEncoder