Module computer_use 

The Anthropic Computer Use actions for AI agent desktop control. The Anthropic Computer Use actions on the captured desktop, and the recorder’s commands beside them, for whatever transport a host serves them over.

[action] runs one request body (screenshots, mouse/keyboard injection, scrolling, cursor position queries) against a [CuBackend], resolved per call: the Wayland compositor owned by this process when one is registered, otherwise the X server named by DISPLAY (XTEST injection on a private connection, no active capture required). [record_start], [record_stop], and [record_status] drive the built-in MP4 recorder; a recording the caller names is a file name inside PIXELFLUX_RECORD_DIR, the directory the operator gives callers, and nowhere else.

Nothing here listens, reads a credential, or knows who called: a caller drives the session with its owner’s full authority, so what serves these entry points answers for who may. The host’s own server does; server, behind the computer-use-server feature, is the Python package’s.

Structs

CuActionRequest 🔒

CuWaylandBackend

Wayland implementation: every primitive is a ThreadCommand on the compositor’s calloop channel, so injection and readback serialize naturally with rendering and encoding.

GpuReadback

A screenshot the GPU is still reading back into a ReadbackSlot.

InvalidRequest

A request body that is no Computer Use action, with the JSON reply saying why; a transport answers it as a bad request.

KeyResolver 🔒

Per-request literal-key resolver: batches keysym lookups against the backend’s active keymap and caches them for the request’s burst of key events (a fresh backend — and thus a fresh cache — is resolved per request, so a runtime layout switch is picked up by the next request).

ReadbackSlot 🔒

A pixel-pack buffer the compositor’s GL readback lands in, mapped persistently so the screenshot’s caller copies the pixels out on its own thread. The mapping is valid while mapped is set; the compositor clears it, under the lock a copy holds, before it deletes the buffer or its GL context goes away.

RecordStartRequest 🔒

Body of a record_start request. All fields are optional; unset ones fall back to the PIXELFLUX_RECORD_* environment variables and built-in defaults.

ScreenshotAdmission 🔒

Bound raw snapshots and PNG work together, including callers still compressing a reply. Two requests per output allow a readback while its previous response is compressed; four across all outputs bound concurrent frame copies without allocating worker threads.

ScreenshotBuffer

One screenshot’s pixels, returned to the kept buffers when dropped.

ScreenshotBuffers 🔒

The pixel buffers screenshots hand to their callers, kept when a caller is done with one so the next screenshot neither allocates nor faults in a whole frame (33 MB at 4K) on the compositor. Two are kept, as many as one display admits at once.

ScreenshotFrame

An owned copy of one rendered output, independent of the streaming buffer pool. Channel conversion and PNG compression run in the requesting thread.

ScreenshotPermit 🔒

Enums

CuButton

Semantic mouse-button vocabulary of the action layer; each backend maps it to its native codes (evdev BTN_ on Wayland, X core buttons 1/2/3 on X11).

ScreenshotPixelFormat

ScreenshotPixels

A screenshot’s pixels: in the caller’s buffer, or still on their way from the GPU.

Constants

BTN_LEFT 🔒

BTN_MIDDLE 🔒

BTN_RIGHT 🔒

KC_ALTGR 🔒

KC_SHIFT 🔒

Shift and default AltGr (pc105 right Alt / ISO_Level3_Shift) keycodes, held to reach a resolved key’s shift level: bit 0 = Shift, bit 1 = AltGr.

MAX_SCREENSHOT_REQUESTS 🔒

MAX_SCREENSHOT_REQUESTS_PER_DISPLAY 🔒

REPLY_TIMEOUT 🔒

Reply deadline for compositor round-trips (screenshot readback, cursor position, fb size). Bounded so a wedged render path turns into a JSON error instead of holding the caller, and every request waiting behind it, forever.

SCREENSHOT_BUFFERS_KEPT 🔒

TYPE_RUN_KEYSYMS 🔒

Distinct characters a type request binds and types at a time: each run’s binds then fit the seat’s text spares (keymap::TEXT_SPARE_KEYCODES; 9 free on a Japanese layout, 11 on most), on which every client types, where a bind past them lands above 255 and Chromium drops it.

Statics

CU_APP_WAYLAND_DISPLAY 🔒

Socket of the compositor apps run under when it is nested under pixelflux (labwc/kwin): keys injected into pixelflux’s own seat carry an overlay keymap the inner compositor never sees, so CU text is typed as a client of this socket instead. Set by selkies over the ScreenCapture ABI.

LAST_BACKEND 🔒

The backend the last action ran on, logged when it changes.

SCREENSHOT_ADMISSION 🔒

SCREENSHOT_BUFFERS 🔒

WAYLAND_BACKEND 🔒

Traits

CuBackend

The desktop-side primitives one Computer Use action decomposes into. Keycodes are X/xkb keycodes (evdev + 8) — the numbering both the smithay seat and XTEST consume — and coordinates are framebuffer/root pixels, already clamped by the action layer.

Functions

act

action for a caller inside the process, which reads why a body was refused from the reply as it reads any other error.

action

Run one Computer Use action: body is the JSON naming it, the reply the JSON it is answered with, an action that failed included. The backend and the framebuffer dimensions are resolved on every call, since the compositor can start, the stream can resize, or the X server can restart underneath the agent. It checks no token, so what calls it answers for who may.

app_wayland_socket_path 🔒

Resolve the app compositor socket PATH for CU typing, or None to type on the local seat. The ABI value selkies set wins; a standalone CU (no selkies) falls back to PIXELFLUX_APP_WAYLAND_DISPLAY. A value naming pixelflux’s own compositor means nothing is nested.

clamp 🔒

distinct_runs 🔒

syms cut into consecutive runs of at most max distinct keysyms.

encode_png_rgba

Turn a raw framebuffer into a PNG the Computer Use agent can actually look at.

error_json 🔒

The reply a request that could not run is answered with.

handle_action 🔒

handle_action_inner 🔒

Turn one parsed Computer Use request into a real action on the captured desktop — the bridge that lets an external AI agent see and drive the session.

is_modifier 🔒

keysym_for_key_spec 🔒

Keysym for one component of an xdotool-style key spec that is not a named key: a single literal character (a, 7, -, #) via the Unicode mapping, or a keysym NAME (minus, bracketleft, udiaeresis, …) via xkb’s name lookup (case-tolerant fallback so Minus still resolves). Returns 0 when the spec names no keysym.

level_modifiers 🔒

The level modifiers level requires beyond whatever is already held (held_mask uses the same bit layout), in press order. altgr is the backend’s AltGr keycode (CuBackend::altgr_keycode).

named_record_path 🔒

The path of a recording the caller names name: a plain file name inside dir, so a request can neither leave the operator’s directory nor name a file anywhere when none was given.

ok_json 🔒

record_start

Start a recording: body is the request’s JSON (RecordStartRequest; empty takes every default), the reply the recorder’s status or the error that kept it from starting.

record_status

The status of the recording under way or of the last one finished; {"active":false} before any.

record_stop

Stop the recording: its final status with stopped set, or the error where none runs.

register_wayland_backend

Make the Wayland compositor the preferred CU backend: once a live calloop sender is registered, every subsequent request routes to it instead of an X11 connection.

resolve_backend 🔒

Pick the backend for one request: the registered Wayland compositor when present, otherwise a fresh private connection to DISPLAY. The X11 connection is per-request so a restarted X server never leaves the CU thread holding a dead connection.

scancode_for_keyname 🔒

set_app_wayland_display

Set (or clear, with None/empty) the app compositor socket for CU text injection and for relative pointer motion into a nested KWin session.

wayland_command_sender 🔒

The registered compositor’s command channel, if a Wayland compositor is running in this process. The recorder uses it to attach to (or start) a capture without any Python client.