Input and audio

Keyboard, mouse, and touch input arrive as platform-independent events and accumulate into an InputState that answers held, pressed-this-frame, and released-this-frame for keys and mouse buttons, plus pointer position, pointer delta, scroll delta, and window focus. axis(negative, positive) returns -1, 0, or 1 and gives zero when both are held, so opposed movement keys cannot resolve by event order.

The editor routes all of it through the Game view during Play, in that view's own pixels, so a script reads the same position there as in the real build. A pointer outside the view is reported as gone rather than clamped to its edge, because a game told the person is pointing at somewhere they are not is worse than a game told they are not pointing at all.

Touch is held beside the mouse rather than folded into it, because they are different facts: a mouse has one position and is always somewhere, while fingers arrive and leave and there may be several. Fingers are bounded at ten, ordered by the id their host gave them so one keeps its place while it stays down, and let go of when a window loses focus — a finger cannot be reported as lifted once the window has stopped hearing about it. What unifies the two is a separate question and is answered where a game reads it: pointer_position follows the first active finger, retains its final position through the release frame, and otherwise follows the mouse. A parked mouse cannot mask touch on a hybrid device, and pointer_down(Left) is the left button or any finger.

Gamepads are read on desktop and in the browser (GamepadReader, over gilrs, behind the platform's default gamepad feature, which links libudev on Linux the way audio links ALSA). Every host that draws a window asks for them once a frame, and so does the editor's Play. A game reads pads by player slot rather than by device: a face button or Start on an unclaimed pad claims the first free slot, a slot stays claimed while its pad is connected and is given back when it goes, and joins and leaves are one-frame edges that never name two slots at once, so a second pad pressing in the same frame joins the next. The press that claims a slot is not reported through it, so joining does not also jump. Slot 0 reads every pad, for a game with one player. Buttons are named by position (south, right_bumper, dpad_up), sticks read in screen axes with a round dead zone taken out, and focus loss lets go of every button without dropping a player. Scripts read Gamepad; an action binding reads any pad as gamepad.south or gamepad.axis.left_x.

Input actions are declared by a scene's sindri.input.actions component, in the actions-document shape ActionMap reads. The script runner reads the declaration when it changes and every action's value from the step's input before any script runs; Action.held, pressed, released, axis and vector answer by name, and Action.bindings, Action.rebind and Action.last_pressed rebind one while the game runs, written back into the component. The platformer's hero runs and jumps by actions, and examples/input rebinds its boost (also covered by the browser smoke). Orbital Last Stand rebinds burst in its settings and saves the chosen binding through Save.text, restoring it when the next session opens.

Audio is a platform service rather than simulation state. Encoded WAV, Ogg, and MP3 assets are identified and validated by the asset layer, then registered with an AudioBackend. Native hosts use Rodio/CPAL; browser hosts decode each clip once into a Web Audio buffer and play it through one AudioContext, unlocked by the first key, click or touch — resumed inside the gesture's own handler, which iOS Safari requires, and on a finger lifting as well as landing, so a phone player holding the platformer's stick still hears it. Headless tests use a silent backend that records every request without needing a sound device. The shared boundary supports one-shot and looping playback plus per-voice stop and global pause, resume, and stop.

Scenes can author sindri.audio.source with a logical clip ID, autoplay, looping, and a normalized volume. The generic component inspector can add and edit it, and the project browser recognises WAV/Ogg/MP3 files as audio assets. Decay exposes typed Audio.play, loop, stop_all, pause_all, and resume_all calls while only emitting intent, so the language itself retains its no-I/O boundary. Gather exercises the path end to end with background, pickup, and victory sounds. The looping background music has been observed playing in a real browser through scripts/browser/smoke.mjs, which fails if a clip a page asked for did not play. The editor can audition a selected clip through its own audio backend.

AudioMixer in sindri-platform sits in front of any backend: it starts a voice at its own volume times its bus's times the master's, and set_bus_volume re-applies the gain to every live voice the bus reaches, which each backend supports (set_volume, is_active). Decay's Audio.play routes to effects, Audio.loop to music, play_on/loop_on to any bus, and Audio.set_volume and Audio.volume move and read a bus; an authored source names its bus or follows its looping. The game session routes scripts' requests and autoplay through the mixer. Orbital's pause screen has Master, Music and Effects sliders, saved between sessions, and examples/audio is the Sound Mixer feature example.

tools/sindri-chiptune (the Sindri Sound Lab) has a Sound FX designer beside its music workstation. Eight kinds — coin, jump, laser, explosion, power-up, hurt, blip and thud — are seeded regions of one synth voice: a wave, an attack/sustain/punch/decay envelope, a pitch path with slide, curve, vibrato, arpeggio and repeat, a square duty sweep, low- and high-pass filters and a sample-and-hold crush. The same kind and seed always render the same samples. Every parameter is a slider with a unit, a change replays the sound, and Mutate nudges each by a few percent.

Send to project writes assets/audio/<name>.wav and the editable <name>.sindrisfx beside it, adds the clip to sindri.toml's [assets] include (editing the manifest in place, comments kept), and offers Audio.play("audio/<name>.wav", 0.5); to paste. Without that include line a clip a script plays is found by the editor and silently left out of an export. The same delivery runs headless: sindri-chiptune sfx coin --seed 5 --project games/platformer --name coin.

The platformer's hero plays four sounds made this way — jump, landing, coin and the flag — and its scripted run to the flag asserts it heard each one and that each is listed for export. The tool's own tests re-render every committed .sindrisfx and compare it with the WAV beside it, so reopening one edits what ships.

The same Lab runs in a browser at /examples/sound-lab/, built by Pages with wasm-pack and drawn with WebGL2. There Send becomes Download of the same two files with the lines to add, and under 720 points wide the window folds into one column for a phone. scripts/browser/sound-lab.mjs opens it at desktop and phone sizes in both modes on every Pages build. The tool is not part of the editor.

New Sound Lab sessions use versioned Musical composition: a theme-derived Rise, modal eight-bar preparation/resolution, register-aware arpeggios that simplify under busy leads, and individual-voice chord inversion selection. Pad harmony follows bar roots instead of passing mid-bar bass tones. Preview and playback share the same deterministic voicing path. Saved sources and snapshots retain the composer version; missing fields select Legacy, which retains previous note and half-bar pad rules. The Song selector and CLI --generation legacy expose that path. Unknown versions are refused. Existing note locks preserve the composer version along with seed, key and mood. These rules have seed-sweep and compatibility tests, not a claim that music quality is objectively solved.

Sound Lab composes and edits a deterministic 32-bar score. Ensemble version 1 uses a separate seeded stream to choose coordinated instrument presets. All layers are enabled in every section by default, preserving the note composer's natural phrases. Section rests are manual choices; masks are saved explicitly, while legacy sources without masks retain their full-layer mix. Smooth gain changes preserve effect tails, and MIDI clips melodic notes at section rests without altering the editable source.

The workstation preserves notes, sound presets and layer participation independently. A/B candidates retain their own mixes; manual mixing and tuning survive generation. Song seeds accept the existing numeric values or UTF-8 phrases via fixed FNV-1a/64 hashing. Saved candidate A phrases are checked against their numeric seed; candidate B saves its exact incremented numeric seed. The random button uses OS entropy natively and browser Crypto on the web. The CLI also supports --random-seed and --classic, and --open preserves saved choices.

The standalone workstation also regenerates a selected bar range without changing surrounding notes, auditions coordinated ensembles while preserving mixes and section choices, inspects selected notes against the written harmony, and stores up to 16 named snapshots with the song. Snapshot restore and sound changes use the same undo/redo history as score edits. Phrase replacements follow edited bass roots; harmony inspection describes written notes rather than diagnosing the rendered audio.

Explicit musical development actions copy, sequence, invert or stretch motifs; arrange Arpeggio/Lead call-and-response pairs while preserving Bass; and shape seeded velocity, phrase dynamics and note lengths without moving onsets. Track locks and Undo/Redo protect edits. Source rhythm and melodic contours also seed related Exploration, Danger and Victory snapshots with adapted scales, tempo, dynamics and coordinated instruments. Family creation is atomic, uses existing snapshot persistence, and leaves the current candidates unchanged. Expression is quantized to the sixteenth-note grid; separate cues do not provide an adaptive music playback API.

These are standalone-tool capabilities. The parity row remains absent in the engine and editor until composition is available on those surfaces.

Pointer.delta and Input.Pointer.delta expose the existing platform mouse movement accumulator as a read-only Vec2, including component reads. Movement is in viewport pixels, positive right/down, spent once per fixed step; a first arrival/re-entry adds no jump. It is displacement rather than velocity, and independent of touch (Gesture remains the touch drag surface). Compiler and runtime reject writes, while a copied vector stays mutable. Bridge regressions exercise accumulated movement, reset, re-entry and both namespaces; LSP tests check completion/hover from the same host metadata.

Orbit Camera Lab uses it for right-button drag-to-look, with sensitivity and pitch limits in Decay. Its Session regression checks yaw/pitch changes, no repeated displacement on the next step and pitch clamping. Real WebGPU Chromium drove a right-button drag through the exported generic player; a console observer in a disposable copy recorded exactly 60 pixels right and 40 down, once. Assets loaded, the walker reached its goal and inspected captures showed the camera response without runtime/GPU errors. The flat-color lab used the same local eight-color smoke threshold as its earlier capture (fourteen colors drawn); the repository browser gate is unchanged. This is feature example evidence; native editor Play also exercises captured delta. Explorer game proof remains pending. Native/browser capture is described below.

The shared platform input state accepts PointerLockChanged as actual host capture feedback and PointerMotion as relative mouse displacement. These are platform events, not Decay APIs or capture requests. Relative motion is accepted only while captured and focused; cursor warps and leave events do not add motion or move/cancel UI presses during capture. Changing capture mode discards pending motion and rebases the next absolute position. Focus loss clears capture, pending motion and the absolute mouse position; focus regain does not recapture. Nonfinite motion and overflowing sums are rejected atomically.

crates/sindri-platform/tests/pointer_motion.rs exercises these transitions, invalid input and once-only consumption through the real EngineHost fixed-step loop, including frames with no step and catchup steps. Native/browser player hosts now emit this feedback and motion, as described below.

Pointer.lock() / Input.Pointer.lock() request capture; unlock() requests release and read-only locked reports actual state. No request changes state within an invocation. Successful invocation outputs collect requests in execution order (last request wins), Session queues the latest across fixed steps, and DesktopApp drains it once after update. Failed invocations do not forward their capture intent; checkpoint restore does not replay window commands.

The windowed host uses native Locked, with X11 confinement plus raw device motion as a fallback. Browser capture calls are caught at the JavaScript boundary and Promise rejection is consumed; actual document lock is polled before gameplay and relative motion. Delayed grants after explicit release are released again. Escape, focus loss, suspension, hidden pages, failure and normal exit release capture. Returning focus does not recapture. Motion uses native raw device counts or browser physical pixels, positive right/down, through the existing delta API.

Bridge regressions cover actual-state reads, both namespaces, read-only writes, invalid call signatures and failed-invocation intent. Orbit Camera Lab's Session regression covers request draining, relative camera motion and checkpoint restore. LSP tests cover read-only boolean metadata and capture-call completion; generated reference/metadata share the same typed host surface.

scripts/browser/pointer-lock.mjs runs the exported lab in real WebGPU Chromium: a trusted click captures, movement beyond the viewport reaches Decay as exactly 1,420 horizontal/60 vertical units once, U and Escape release, and a sandboxed frame denies capture without false success or interrupting the goal run. The captured-look screenshot was inspected. CI exports the same project and runs this regression, retaining its screenshot. Its flat-color fixture uses an eight-color minimum; the general browser smoke threshold remains unchanged. Native X11 verification runs the same generic player in an isolated Xvfb window on Vulkan, reaches the goal, and drives capture/raw motion/U/Escape/focus changes with XTest. Decay observes 4,000 horizontal/120 vertical raw units in one step, releases on each requested/focus transition and does not recapture on focus regain. The native window screenshot was inspected alongside Chromium. Native platforms other than this X11 run were not visually exercised here. Native editor Play also drains Session requests after stepping and editor transport/layout changes. Native window calls report success before input sees capture; missing windows cannot report false success. egui raw mouse events feed native counts without display scaling; the shared state ignores absolute warps. Unit regressions cover accumulation across frames, once-only consumption, release rebasing, stopping and absent windows. The actual X11 editor window runs Orbit Camera Lab to its goal, reports 6,000 horizontal/120 vertical raw units from an unbounded XTest move, and releases on U/Escape, pause/stop, focus loss and hiding the Game view. An external pointer-grab probe verifies actual capture and release; resume/focus regain/new Play do not automatically recapture. The editor screenshot with Decay motion and goal output was inspected. Other native editor platforms and Explorer game proof remain pending.