Module nvenc 

NVIDIA NVENC hardware H.264 / HEVC / AV1 encoder loaded via runtime libcuda / libnvidia-encode. NVENC hardware H.264 / HEVC / AV1 encoder: CUDA-bound sessions that encode packed BGRA or RGBA frames from the X11 host and Wayland readback paths, or Wayland dmabufs in place.

The module dynamically loads libcuda, libnvidia-encode, and libEGL at runtime, negotiates the NVENC API version against the installed driver (set-once per process), and stamps every NVENCAPI struct with the exact NV_ENC_*_VER word the negotiated SDK defines, so one binary drives drivers from NVENC 10.0 (~R445) through 13.0. Frames reach the GPU two ways: a zero-copy dmabuf import (EGLImage → CUDA, the mapped plane registered with NVENC in place as pitch-linear memory or as a CUDA array), and a pinned host→device upload of packed BGRA / RGBA. Color is converted on the GPU either way. The codec is a session parameter: the same rate control, GOP, VUI, and latency posture is programmed into whichever of the three codec configurations the device offers, and a codec the device lacks (AV1 before Ada) is refused at open so the caller falls back. Sessions reconfigure resolution and rate control in place, so a resize or bitrate change costs a few milliseconds instead of a full rebuild.

Structs

BandHash 🔒

A hash per band of rows of a packed 32-bit frame in video memory (band_hash), computed where the frame lies and read back as one word a band and column segment.

CUeglFrame 🔒

A CUDA frame mapped from an EGLImage: the cuGraphicsResourceGetMappedEglFrame result describing the imported dmabuf’s plane pointers, geometry, pitch, and pixel format.

CachedDmaBuf 🔒

Cached CUDA import of a dmabuf, keyed by fd so a recurring capture buffer is imported once: the EGLImageKHR, the CUDA graphics resource it registers as, the mapped CUeglFrame, and how that frame reaches NVENC. Torn down on drop / reconfigure, or evicted when the fd it is keyed by no longer names the same buffer (see DmaBufIdentity).

CapsDecision 🔒

The chroma format and dimensional feasibility a session settles on given the driver’s reported capabilities, so init degrades cleanly instead of failing opaquely.

ChromaConvert 🔒

CudaFunctions 🔒

Dynamically loaded CUDA driver-API entry points (from libcuda) for context, device, memory, host-pin, and EGL-interop calls. _lib keeps the library resident for the pointers’ life.

EglFunctions 🔒

Dynamically loaded EGL entry points (from libEGL) for creating and destroying the EGLImageKHR that wraps a dmabuf. _lib keeps the library resident for the pointers’ life.

ExternalInput 🔒

An input surface the session does not own: a device pointer produced by another component on the same CUDA context — the NvFBC capture buffer — registered and mapped with NVENC in place so the encoder reads the captured frame where it was composited.

Negotiated 🔒

A struct as long as the negotiated API’s layout of it, which can run eight bytes past the pinned header’s: NV_ENC_INITIALIZE_PARAMS below 12.2, and so the reconfigure params that embed it, and NV_ENC_LOCK_BITSTREAM at 12.1. The tail is zeroed and pixelflux’s, so the driver reads and writes inside memory this owns.

NvencEncoder

A live NVENC encoder session with its CUDA context and interop resources.

NvencLibrary 🔒

Dynamically loaded NVENC entry points (from libnvidia-encode).

NvencTuning 🔒

The encoder-side quality knobs of a session: the preset, the rate-control passes of a CBR session, and adaptive quantization. Production sessions take the default, which the tuning bench (gpu_bench_tuning) measures against the alternatives: P3 encodes a 1080p H.264 frame in 3.7 ms where P4 takes 5.2 ms on a V100, for 0.001 of SSIM at the same bitrate, and the presets above P4 buy nothing; a single pass saves half a millisecond but overshoots a CBR target by five to twelve percent; adaptive quantization moves neither time nor quality measurably. The quarter-resolution first pass needs the 1.5-frame VBV vbv_bits gives it: on a one-frame buffer its miss on a scene cut runs an 8 Mbit/s H.264 session at 13.7 Mbit/s (gpu_bench_cbr_rate_control). P3 holds on real content as well (gpu_bench_tuning_on_content, scrolling text and a panned texture at 1080p to 2160p): on a GTX 1080 and an RTX 4090 P1 encodes 0.5 to 2.4 ms sooner and P2 less, at most about 5% of the glass to glass latency a session of that size measures (25, 41, and 81 ms over WebSockets), and both code the text worse, by up to 0.8 dB in H.264 and 2.4 dB in AV1.

Enums

ConvertLayout 🔒

The 4:2:0 convert that replaces NVENC’s own, because the hardware’s fixed-function conversion weights the two columns of a block 3:1 instead of averaging them, leaving half the color of a subpixel-antialiased glyph edge in the chroma plane where the software and VA-API converts leave none. It follows the matrix the session declares (gpu_hardware_conversion_matches_the_declared_matrix), so only the siting is at stake, and 4:4:4 — which subsamples nothing — keeps it.

DirectPlane 🔒

How the first plane of a mapped CUeglFrame registers with NVENC in place: the NV_ENC_REGISTER_RESOURCE resource type and the pitch word that type expects.

DmaBufInput 🔒

How a cached dmabuf import feeds the encoder.

NvStruct 🔒

The NVENCAPI structs this encoder must stamp with a per-SDK version word — enumerated here precisely because getting that word exactly right is the whole mechanism that lets one compiled binary satisfy every driver’s version check.

Constants

ARGB_TO_NV12_PTX 🔒

The ARGB/ABGR → NV12 convert and the band hash, as PTX the driver JIT-compiles at session open. PTX is the portable form: libcuda compiles it for whatever GPU is present, so nothing beyond the driver NVENC already needs has to be installed, and .version 3.1/.target sm_30 keeps every NVENC-capable GPU in range.

AV1_REFERENCES 🔒

The frames an AV1 session keeps, as many as a frame predicts from, LAST, LAST2, LAST3, and GOLDEN (numFwdRefs tops out at four). The device keeps eight as well, predicting past a loss from further back, but takes 112 MiB more a session at the resize headroom. With SDK 13 and LTR support, the anchors among them let an invalidation reach further back.

BAND_LOG_FRAMES 🔒

Recent frames whose held bands the session keeps (forget_bands): past how far back an invalidation of its own before an anchor reaches (reference::HOLD_MAX, 384).

BAND_SLICES_MAX 🔒

The most slices an HEVC band frame takes to start one at its band (set_band_layout). NVENC sizes a frame’s slices evenly, so a band starting within the first eighth of the picture would take more, each a header a low rate’s frame cannot spare: at 60 fps and 0.5 Mbit/s, 40 to 60 of them took most of a band frame’s bytes and left its band coarse.

BITSTREAM_BUFFERS 🔒

Output bitstream buffers per session: one, because submit_frame locks, copies, and unlocks each frame’s bitstream before it returns, so no second buffer is ever outstanding (the lock blocks; a doNotWait lock on Linux answers an unfinished encode with an empty bitstream rather than NV_ENC_ERR_LOCK_BUSY). The ring stays, so a pipelined depth is one constant away.

CU_AD_FORMAT_U8 🔒

CUeglFrame::cu_format of an 8-bit-per-channel plane (CU_AD_FORMAT_UNSIGNED_INT8).

CU_EGL_FRAME_TYPE_ARRAY 🔒

CUeglFrame::frame_type values: the mapped planes are CUDA arrays, or pitch-linear device memory.

CU_EGL_FRAME_TYPE_PITCH 🔒

CU_TRSF_READ_AS_INTEGER 🔒

CU_TRSF_READ_AS_INTEGER, a cuda.h macro the bindings do not carry: a texture fetch returns the stored bytes rather than normalized floats.

DMABUF_IMPORTS 🔒

Dmabuf imports a session keeps at once: a capture’s own targets and the swapchain of the client whose buffers it encodes in place come to a handful, and past this the one longest unused is released, so a client allocating buffers freely never grows the cache.

EGL_DMA_BUF_PLANE0_FD_EXT 🔒

EGL_DMA_BUF_PLANE0_MODIFIER_HI_EXT 🔒

EGL_DMA_BUF_PLANE0_MODIFIER_LO_EXT 🔒

EGL_DMA_BUF_PLANE0_OFFSET_EXT 🔒

EGL_DMA_BUF_PLANE0_PITCH_EXT 🔒

EGL_HEIGHT 🔒

EGL_LINUX_DMA_BUF_EXT 🔒

EGL_LINUX_DRM_FOURCC_EXT 🔒

EGL_NONE 🔒

EGL_NO_IMAGE_KHR 🔒

EGL_WIDTH 🔒

EXTERNAL_INPUTS 🔒

External input registrations an encoder keeps at once (register_external_input).

HASH_SEGMENTS 🔒

HASH_THREADS 🔒

The threads of a band_hash block, as the kernel sizes its reduction, and the blocks a band is split into across its columns, so a frame’s few bands fill the GPU: a P100 hashed 1080p in 0.08 ms this way and in 0.12 with a block a band, its 34 bands leaving 22 of the 56 multiprocessors idle.

HEADROOM_FPS 🔒

The frame rate the level is read at when the session’s is lower, so a live change between the common rates keeps the level, as the geometry headroom keeps it across a resize.

HEADROOM_HEIGHT 🔒

HEADROOM_WIDTH 🔒

The geometry every session is initialized to hold, so reconfigure_resolution can grow into it in place. It spans both UHD and DCI 4K and is the largest picture the 5.x levels of all three codecs admit, which is what lets AV1 – pinned to this level – stay inside a level a decoder accepts; a taller resize rebuilds the session instead.

NVENC_AV1_RATE 🔒

The share of an AV1 level’s Annex A MaxBitrate the strictest driver lets a session declare at it, as a fraction: two thirds on 595.71.05, where 595.91.07 and 615.71.09 take the whole.

NV_ENC_H264_PROFILE_HIGH_444_GUID 🔒

GUID selecting the H.264 High 4:4:4 Predictive profile for full-color encoding.

NV_ENC_H264_PROFILE_HIGH_GUID 🔒

GUID selecting the H.264 High profile (4:2:0) for NV_ENC_CONFIG::profileGUID.

REFINE_FINER

Quality index steps finer than the paint-over one a refined block is coded at (FrameEncoder::refines): what brings it to the quality NVENC’s own cleanup of a still screen reaches. 1080p text beside a video, H.264 on an RTX 3060, against that cleanup’s 45.2 dB at 8 Mbit/s, 47.6 at 16, and 47.0 at a constant quality: 44.4, 45.5, and 46.8 at the paint-over quantizer, 45.9, 47.0, and 48.1 three steps finer, 48.2, 48.7, and 50.6 six steps finer, spending on the text what the video would have had.

REFIT_SHARE 🔒

The share of its cap a held band taken back for coming out past it is coded again to, at the bytes a block of it came to (NvencEncoder::cap): the next rows of a page can be denser.

SESSIONS_TAKEN 🔒

The probe’s answer where the device refuses a session for want of one to spare (the session cap of consumer boards, or its memory): nothing lasting, unlike its other refusals.

SLICES_PER_FRAME 🔒

Slices per H.264 frame, the count the VA-API (SLICES; one for H.264 on AMD’s VCE) and OpenH264 (SM_FIXEDSLCNUM_SLICE) sessions emit too: a client decoding in software threads a frame across its slices, and more than four upsets some Chromium decoders. What the slices cost at a fixed quantizer is measured by gpu_bench_slices. An HEVC frame is one slice, since no decoder a client has threads HEVC’s plain slices (FFmpeg threads its wavefronts and tiles), and its band frames take theirs where a band starts (set_band_layout). AV1 partitions by tiles instead, asked for as 1x1 in configure_codec.

SLICE_MODE_COUNT 🔒

sliceMode = 3: sliceModeData is the number of slices in the picture, which the driver divides evenly; the other modes count macroblocks, bytes, or rows and drift with geometry.

Statics

NVENC_AV1_WEIGHTED 🔒

Whether the process’s driver holds an AV1 level to NVENC_AV1_RATE of its Annex A rate: set where it refused a session or a rate change at the Annex A level and took the weighted one, so later sessions declare the weighted level without the refusal.

NVENC_NEG_VER 🔒

Negotiated NVENC API version (major, minor), resolved once per process. None until nvenc_negotiate runs; every struct-version word and the session apiVersion derive from it.

NVENC_SINGLE_PASS 🔒

Whether the process’s driver refused a CBR session’s quarter-resolution first pass, as Kepler’s does, taking only a full-resolution one: later sessions code CBR in a single pass without the refusal. On a Tesla K80 the full-resolution pass encodes a 1080p H.264 frame in 12.7 ms where the single pass takes 7.3 ms and overshoots its target by 4%.

Functions

anchor_count 🔒

Persistent anchors only where the negotiated API exposes them, the device can invalidate references and the DPB leaves two recent pictures beside them. AV1 gained the LTR fields in SDK 13; drivers negotiated down to an older API keep the unanchored path. AV1 keeps one, a frame every forty-eight, unless the consumer acknowledged what its clients hold and the device has room for two: then the newest every client holds is pinned beside the one the schedule marks. H.264 keeps one whatever the consumer says: OpenH264’s decoder, Firefox’s for WebRTC H.264 where it has no FFmpeg that decodes it, finds a long-term picture of a CABAC stream by a number no marking sets, so of two it reaches only the first, and decodes a frame naming the second against another picture without an error; and a key frame marked long-term leaves a second index past MaxLongTermFrameIdx besides.

cbr_bps 🔒

The CBR target of a session at settings, in bits per second, held to the codec’s nvenc_rate_ceiling.

cbr_vbv 🔒

The VBV of a CBR session at bps, from the session’s frame rate, key-frame interval, and explicit multiplier.

codec_guid 🔒

The NVENC codec GUID of a codec the encoder serves, or None for one NVENC has no engine for.

codes_ten_bit 🔒 ^⚠

Whether a session of codec codes 10 bits: HEVC and AV1 where the device reports it, at whichever API the session negotiated (set_bit_depth). The session converts its 8-bit RGB to 10-bit samples itself (ConvertLayout). H.264 stays 8-bit, which only the newest engines code at 10.

decide_caps 🔒

Resolve the requested chroma and geometry against the driver caps (None = unknown, ungated).

direct_plane 🔒

Whether NVENC can read a mapped CUeglFrame in place, and how.

egl_import_fourcc 🔒

The NVENC packed input format whose byte order is a dmabuf’s: the XR24 / AR24 family is B,G,R,A in memory (NVENC’s word-ordered ARGB), the XB24 / AB24 family R,G,B,A (ABGR). None for any other fourcc — nothing NVENC reads as packed 8-bit RGB. The fourcc a dmabuf is described with for CUDA’s EGL import: the alpha-carrying twin of an X-format, since the NVIDIA driver refuses XRGB/XBGR images there while the bytes lie identically and NVENC ignores the alpha channel of ARGB/ABGR input.

fill_band_map 🔒

Fill map, the QP delta map of a width x height picture of codec (one entry a 16x16 macroblock for H.264, a 32x32 coding tree block for HEVC, a 64x64 superblock for AV1, in raster order), with delta over the blocks the share of the picture from..to covers, rounded outward, and 0 elsewhere; returns the band’s first block.

fourcc_nvenc_format 🔒

guid_eq 🔒

Whether two GUIDs are the same.

keep_primary_context 🔒

Hold one reference on device’s primary CUDA context for the life of the process, once a session on it has opened. A capture restarted while a game fills the GPU’s memory then retains the context it had instead of creating one, which can fail there and leave the stream on the CPU. The cost is the idle context’s 106 MiB of VRAM on a device that has encoded once, kept also while no session runs. Its own copy of the driver’s entry points keeps libcuda loaded after the last session’s copy goes.

last_error 🔒 ^⚠

The driver’s message for the last failure on session, or a stand-in when it has none: NVENC clears the string once a session is torn down, so it is only meaningful read straight after the call that failed.

level_rate 🔒

The rate a codec level has to admit for a peak of bps: AV1’s weighed by NVENC_AV1_RATE where the driver holds a level to that share.

load_kernels 🔒 ^⚠

JIT the kernels’ module into the current context; None where the driver refuses it.

log_held_rate 🔒

Say so where cbr_bps held the target settings asks for to the codec’s ceiling.

neg_api 🔒

The negotiated session apiVersion word (major | minor<<24) passed to NvEncOpenEncodeSessionEx — note the minor sits at bit 24 here, unlike the (major<<4)|minor packing that NvStruct::rev matches on.

nvenc_cur_ver 🔒

The process’s effective NVENC API version (major, minor): the negotiated value once nvenc_negotiate has run, otherwise the pinned NVENCAPI_VERSION decomposed (major in the low byte, minor at bit 24) as the pre-negotiation fallback.

nvenc_headroom 🔒

The in-place resize headroom for one axis: the requested size lifted to floor but never past the driver’s reported maximum, so initializing with headroom cannot itself exceed what the GPU supports.

nvenc_level 🔒

The level an NVENC session advertises for a width x height stream at fps carrying up to bitrate_bps (NV_ENC_LEVEL shares each codec’s own numbering: level_idc for H.264, general_level_idc for HEVC, seq_level_idx for AV1).

nvenc_negotiate 🔒

Resolve the process-wide NVENC API version once, by probing the driver newest-first and remembering the highest version it accepts.

nvenc_rate_ceiling 🔒

The highest CBR target an NVENC session of codec opens at, its top level’s bitrate ceiling: H.264 6.2’s 1 Gbit/s, HEVC 6.2’s 800 Mbit/s at the High tier production sessions declare, and NVENC_AV1_RATE of AV1 6.3’s Main-tier 160 Mbit/s. No level admits a rate past it, so the driver would refuse the session, or the change, as an invalid level.

nvenc_struct_ver 🔒

Assemble the NVENCAPI_STRUCT_VERSION word for struct s at API version (maj, min).

peak_of 🔒

The peak a capped constant quantizer runs under (RateSettings::capped), as NVENC’s constant rate takes it: the target held to the codec’s ceiling, and its buffer.

point_at_qp_map 🔒

Hand pic the QP delta map map at API api. Below 12.0 the codec union runs eight bytes shorter and the driver reads the map’s pointer and size where the pinned layout keeps reserved5.

probe_codecs 🔒

The video codecs the NVENC of the GPU behind encode_node_index has an engine for, read from a bare session’s GUID list the way a real session reads it (device_encodes): the driver is negotiated, the device bound by the render node’s PCI bus id (the first CUDA device where the node names none), and the session opened on its primary context with no input buffers, EGL, or encoder initialization. An error names the step that failed: no driver, no device, or a session that would not open, each of which a real session would fail on too. The CUDA and NVENC libraries stay loaded like a session’s, since the driver does not promise to survive libcuda being unloaded after cuInit.

probe_session_codecs 🔒 ^⚠

Open a bare NVENC session on a current CUDA context, list the codecs its device encodes and the formats past 8-bit 4:2:0 it encodes each in, and close it.

profile_guid 🔒

The profile of a session: the 4:2:0 profile of the codec at its depth, or its 4:4:4 one where the codec has it. HEVC’s range extensions carry 4:4:4 at either depth, and AV1’s main profile both depths.

query_cap 🔒 ^⚠

Query one NVENC capability of codec on an open session, returning the driver’s integer answer or None when the entry point is absent or the query fails — decide_caps reads None as “do not gate”, so a query failure never becomes a false refusal.

reach_back 🔒

The band a frame codes once the frames from timestamp pts on are left out of every prediction, their bands taken out of log: the one it holds (held) reaching back over theirs; and whether theirs are lost, a frame that holds no quantizer (holds) taking none of them back. A frame held whole codes them anyway.

reconfigure_params 🔒

The reconfigure params re-initializing with init at API (maj, min), resetting the encoder and forcing an IDR as asked; below 12.2 the driver reads the two flags from the tail.

session_refusal 🔒

Why the driver refused to open a session: SESSIONS_TAKEN for the statuses it answers when none is left, NV_ENC_ERR_INCOMPATIBLE_CLIENT_KEY at the cap of a consumer board and NV_ENC_ERR_OUT_OF_MEMORY when its memory runs out.

set_bit_depth 🔒

Stamp an HEVC or AV1 config with bit_depth as API api reads it. From 12.2 the session names its input and output depths; before it HEVC carries the output’s minus eight in the bitfield the pinned layout keeps as reserved3, its input following the buffer format, and AV1 (12.0 and 12.1) the input’s and the output’s in the bits 13.0 gave its LTR, SVC and HDR flags. The offsets count from the codec config’s first bitfield, at byte 16.

set_cbr_rate 🔒

Write a CBR target into rc: the average and peak rate, the VBV, and an initial delay of the whole buffer. That delay is the driver’s default (gpu_bench_cbr_policy finds the two identical), stated so the HRD’s starting point is explicit and a rate change restates it with the buffer it belongs to.

split_mode 🔒

The split-frame mode a codec session asks for at width x height on a device of engines NVENC engines under the negotiated api, as gpu_bench_tuning_on_content measured it. AV1 splits across the engines whatever the picture: a 1080p frame encodes in 1.36 ms rather than 2.07 on an RTX 4090, a 1440p one in 2.27 rather than 3.58, and both code as well or better, where the driver’s own choice splits only from 4K. HEVC splits from a 4K picture, which the driver’s choice splits on Ada but not on Pascal: a GTX 1080 encodes the 2160p frame in 10.3 ms rather than 16.1, more than 5% of the 81 ms such a session measures glass to glass, for 2.5 dB of PSNR on scrolling text. Below 4K a split saves HEVC about a millisecond for up to 2.3 dB, so the driver decides there, and H.264 never splits. The forced mode cuts two strips on three engines as on two, so AV1 asks for three there: an L40S encodes a 1080p frame in 1.45 ms rather than 1.62 and a 2160p one in 4.2 rather than 5.3, coding better in five of six rows (up to 1.5 dB on text) and 0.5 dB worse in one, where HEVC’s third strip saves nothing at 4K and costs 0.1 to 0.2 dB. Before API 12.1 the field’s bits are another flag’s, so an older session leaves them clear.

sv 🔒

The struct-version word for s tagged with the process’s negotiated API version — the value every NVENCAPI struct literal assigns to its version field.

window_frames 🔒

The frames a reference window holds for a decoded picture buffer of dpb: an H.265 session keeping anchors still reaches, on Pascal’s NVENC, the frame the next frame’s reference picture set lets go, one past the buffer it declares. Turing’s, Ampere’s and Ada’s do not, and predict from the anchor before that frame where the window names it: a dependency the window states no older than the real one, so a client holding the frame named holds the one used.

Type Aliases

CUfunction 🔒

CUgraphicsResource 🔒

Opaque CUDA graphics-resource handle for the EGL interop path — an EGLImageKHR registered with CUDA maps to one of these.

CUmodule 🔒

Opaque CUDA module and kernel handles; the driver API’s own types, which the committed bindings do not carry.

EGLBoolean 🔒

EGLDisplay 🔒

EGL C-interop type aliases and the EGL_* attribute constants used to wrap a dmabuf as an EGLImageKHR for CUDA import.

EGLImageKHR 🔒

EGLenum 🔒

EGLint 🔒

EglCreateImageKhrFn 🔒

EglDestroyImageKhrFn 🔒

Unions

CUeglFrameUnion 🔒

The mapped frame’s plane pointers, as either CUDA arrays or pitch-linear device pointers — CUeglFrame::frame_type selects which arm of the union is valid.