The job-spec vocabulary¶

The engine is driven by a structured job spec, not ffmpeg command-line strings. This is a deliberate choice (Why libav-direct): a typed surface that is exactly as capable as the engine, with no leaky partial-CLI illusion. The vocabulary is the compatibility contract between ffmpeg-wasi and afmpeg; it is versioned, and afmpeg pins a known-good engine + vocabulary version.

Operations¶

process — transcode / filter / mux¶

{
  "op": "process",
  "inputs":  [ { "path": "in/clip.mp4", "options": { } } ],
  "filter":  "[0:v]scale=1280:-2[v]",   // an ffmpeg filtergraph STRING (see below)
  "outputs": [ {
    "path": "out/clip.mp4",
    "map": ["[v]"],                     // graph pads / stream specifiers to mux
    "video_codec": "libx264",
    "audio_codec": "aac",
    "options": { "crf": "23", "movflags": "+faststart" }
  } ]
}

A job is bounded: 32 inputs, 8 outputs, 32 graph input pads, 16 graph output pads, 32 stream copies and 16 subtitle streams, all per job rather than per output. Crossing one fails the job — see limits.

Working examples (verified end-to-end):

// audio: WAV (pcm) → AAC in MP4 (no filter → passthrough graph)
{"op":"process","inputs":[{"path":"tone.wav"}],
 "outputs":[{"path":"out.mp4","audio_codec":"aac"}]}

// video: H.264 → scaled → H.264 (GPL variant, libx264)
{"op":"process","inputs":[{"path":"in.mp4"}],"filter":"[0:v]scale=160:120[vout]",
 "outputs":[{"path":"out.mp4","video_codec":"libx264","options":{"crf":"28"}}]}

// multi-input: combine a video + an audio file into one mp4
{"op":"process","inputs":[{"path":"clip.mp4"},{"path":"music.mp3"}],
 "filter":"[0:v]scale=1280:-2[vout];[1:a]anull[aout]",
 "outputs":[{"path":"out.mp4","video_codec":"libx264","audio_codec":"aac"}]}

// crossfade-concat two clips
{"op":"process","inputs":[{"path":"a.mp4"},{"path":"b.mp4"}],
 "filter":"[0:v][1:v]xfade=transition=fade:duration=0.4:offset=2[vout]",
 "outputs":[{"path":"out.mp4","video_codec":"libx264"}]}

// multi-output: split one source (asplit) into two files via `map`
{"op":"process","inputs":[{"path":"tone.wav"}],
 "filter":"[0:a]asplit=2[a1][a2];[a1]volume=0.9[loud];[a2]volume=0.1[quiet]",
 "outputs":[{"path":"loud.mp4","map":["[loud]"],"audio_codec":"aac"},
            {"path":"quiet.mp4","map":["[quiet]"],"audio_codec":"aac"}]}

// v2 — remux, no re-encode (both streams stream-copied)
{"op":"process","version":2,"inputs":[{"path":"in.mp4"}],
 "outputs":[{"path":"out.mkv","map":["0:v","0:a"],"video_codec":"copy","audio_codec":"copy"}]}

// v2 — mixed: copy the video, re-encode only the audio
{"op":"process","version":2,"inputs":[{"path":"in.mp4"}],"filter":"[0:a]loudnorm[aout]",
 "outputs":[{"path":"out.mp4","map":["0:v","[aout]"],"video_codec":"copy","audio_codec":"aac"}]}

// v2 — stream-copy join of like-codec segments via the concat demuxer
{"op":"process","version":2,"inputs":[{"concat":["a.ts","b.ts","c.ts"]}],
 "outputs":[{"path":"joined.ts","map":["0:v","0:a"],"video_codec":"copy","audio_codec":"copy"}]}

// v3 — clip extraction: 5s from 0:12.5, frame-accurate, re-encoded
{"op":"process","version":3,
 "inputs":[{"path":"in.mp4","seek":{"start":12.5,"mode":"accurate"}}],
 "filter":"[0:v]null[v]",
 "outputs":[{"path":"clip.mp4","map":["[v]"],"video_codec":"libopenh264","duration":5.0}]}

// v3 — keyframe-accurate copy-trim: cheap cut, no re-encode (fast seek + copy)
{"op":"process","version":3,
 "inputs":[{"path":"in.mp4","seek":{"start":12.5}}],
 "outputs":[{"path":"cut.mp4","map":["0:v","0:a"],"video_codec":"copy","audio_codec":"copy"}]}

// v4 — raw headerless input: forced demuxer + geometry, transcoded
{"op":"process","version":4,
 "inputs":[{"path":"frames.yuv","format":"rawvideo",
            "options":{"video_size":"1280x720","pixel_format":"yuv420p","framerate":"25"}}],
 "filter":"[0:v]null[v]",
 "outputs":[{"path":"out.mp4","map":["[v]"],"video_codec":"libopenh264"}]}

// v5 — mp4 → MPEG-TS remux, no re-encode (the h264_mp4toannexb BSF auto-inserts)
{"op":"process","version":5,"inputs":[{"path":"in.mp4"}],
 "outputs":[{"path":"out.ts","map":["0:v","0:a"],"video_codec":"copy","audio_codec":"copy"}]}

// v5 — HLS: one output entry writes segment files + a playlist to the fs (no network)
{"op":"process","version":5,"inputs":[{"path":"in.mp4"}],"filter":"[0:v]null[v]",
 "outputs":[{"path":"stream.m3u8","map":["[v]"],"format":"hls","video_codec":"libopenh264",
   "format_options":{"hls_time":"4","hls_segment_filename":"seg_%03d.ts","hls_list_size":"0"}}]}

// v7 — remux to mkv, set container tags, copy chapters, tag a stream's language/disposition
{"op":"process","version":7,"inputs":[{"path":"in.mp4"}],
 "outputs":[{"path":"out.mkv","map":["0:v","0:a"],"video_codec":"copy","audio_codec":"copy",
   "metadata":{"title":"My Title","artist":"Me"},
   "chapters":"copy",
   "stream_metadata":{"0:a":{"language":"eng","disposition":["default"]}}}]}

// v8 — convert a subtitle track to WebVTT (sidecar), and embed one as mov_text in mp4
{"op":"process","version":8,"inputs":[{"path":"sub.srt"}],
 "outputs":[{"path":"out.vtt","map":["0:s"],"subtitle_codec":"webvtt"}]}
{"op":"process","version":8,"inputs":[{"path":"in.mp4"},{"path":"sub.srt"}],
 "outputs":[{"path":"out.mp4","map":["0:v","0:a","1:s"],
   "video_codec":"copy","audio_codec":"copy","subtitle_codec":"mov_text"}]}

Subtitle streams (spec 0019). outputs[].subtitle_codec transcodes a subtitle track named in map by an N:s specifier (e.g. srt, webvtt, mov_text, ass), or "copy" to remux it unchanged. An output may carry subtitle_codec alone (a sidecar .srt/.vtt) or beside video + audio (an embedded track). Subtitles ride their own decode→encode lane — they do not traverse the filter graph — so they are mapped by stream specifier, never a graph-pad label.

Metadata (spec 0020). outputs[].metadata is a {key:value} tag map set on the output container; outputs[].chapters is a passthrough directive ("copy" carries the first input's chapters, an input index picks another, "none"/absent drops them); outputs[].stream_metadata maps a map entry to {language, disposition, tags} applied to that output stream. An attached_pic cover-art stream copied via map keeps its disposition automatically.

A segmenting output (hls/dash/segment) reports "segmented": true in its result entry; path is the playlist/manifest and the segment files sit beside it on the mounted filesystem by the requested pattern.

On success the engine prints what it wrote, one entry per output file, e.g. {"outputs":[{"path":"loud.mp4","streams":[{"type":"audio","codec":"aac"}]},{"path":"quiet.mp4","streams":[{"type":"audio","codec":"aac"}]}]}.

If the graph ran an analysis filter (cropdetect, blackdetect, silencedetect, ebur128, signalstats, astats, …), the reply also carries an analysis array of the lavfi.* measurements those filters attach to frames — objects of { "t": <seconds>, "key": <name>, "value": <string> }, consecutive-deduplicated per key (spec 0017 §Q). It is omitted when no analysis filter ran. (ebur128/astats need their metadata=1 option to populate it.) afmpeg surfaces it as ProcessResult.Analysis.

Progress side-channel¶

Progress (spec 0032). A top-level "progress": true on a process job turns on a one-way, best-effort progress emitter. As the engine muxes, it writes throttled NDJSON records (one per ~100 ms of media time, not per frame) to the device /dev/afmpeg-progress, plus a final record at completion:

{"frame":420,"out_time_us":16800000,"total_size":2097152,"duration_us":30000000}

frame counts muxed video packets, out_time_us is the furthest output timestamp reached, total_size the bytes muxed so far, and duration_us the job's target media duration (0 when unknown). The device is served by afmpeg's virtual filesystem — afmpeg's own runtime parses each line and surfaces it on the WithProgress channel (afmpeg spec 0031 phase B). It is purely additive and opt-in: if the device is absent or progress is unset, every emit is a no-op and the encode is never blocked, slowed, or failed by progress I/O.

probe — report stream information¶

{ "op": "probe", "inputs": [ { "path": "in/clip.mp4" } ] }

Reports container/stream metadata (format, duration, per-stream codec/type and dimensions/sample rate) as JSON on stdout. No outputs are written. For example, probing a WAV yields:

{"inputs":[{"path":"tone.wav","format":"wav","duration_sec":0.5,
  "streams":[{"index":0,"type":"audio","codec":"pcm_s16le","sample_rate":8000,"channels":1}]}]}

Since v7 (spec 0020) the reply additionally carries, where present, the container's tags object and chapters array (start/end in seconds + title), and on each stream its language, decoded disposition flag names, and tags — all additive, so an older consumer ignores them.

frames — extract still frames¶

{
  "op": "frames", "version": 6,
  "inputs": [ { "path": "in/clip.mp4" } ],   // exactly one video input
  "select": {                                 // exactly one selector
    "timestamp":  12.5,                        // (a) single frame, seconds
    "timestamps": [1.0, 5.0, 30.0],            // (b) explicit list
    "interval":   10.0,                        // (c) every N seconds
    "scene":      0.4                           // (d) scene-change threshold, or "thumbnail"
  },
  "path":  "out/frame_%03d.png",              // templated: one integer token (or none for a single frame)
  "codec": "png",                              // image encoder: png (default) | mjpeg | webp
  "scale": "320:-2",                           // optional ffmpeg scale args
  "count": 25                                  // optional cap on frames emitted
}

Pulls one or more stills from a video to templated image files (afmpeg spec 0021) — the bread-and-butter "poster at 5s / thumbnail strip / contact sheet" chore as typed fields rather than an fps/select graph into an image2 muxer. The seeking selectors (timestamp / timestamps / interval) fast-seek to the keyframe at-or-before each target and decode forward to the first frame ≥ target — cheap, not a full-stream decode; scene stream-decodes through the select='gt(scene,T)' or thumbnail filter (which need the intermediate profile). Each frame is optionally scaled, encoded, and written; the engine owns the naming and reports each:

{"frames":[{"path":"out/frame_000.png","index":0,"timestamp":12.5}],"count":1}

select is a one-of (zero or multiple is rejected). codec defaults to png; png and mjpeg are in every profile, webp needs the intermediate profile. path must contain zero or one integer conversion — %s, %n and any other conversion are refused outright, and a token-less path is only valid for a single frame.

The frame count is bounded twice. count caps how many frames are emitted; with count absent it defaults to 1000, so an uncapped interval over a long input cannot run away. Independently, the selector's target list is capped at 4096 timestamps, which count cannot raise — a finer-grained interval simply stops there. The scene selector streams rather than seeking, so count is its only bound. See limits.

version — report the vocabulary version¶

{ "op": "version" }

Prints the engine's highest supported vocabulary version + its FFmpeg build as JSON on stdout — the machine-readable channel a consumer preflights before running jobs. Needs no input, so it works before any media is mounted:

{"vocab_version":9,"ffmpeg_version":"n8.1.2"}

This query carries no version of its own, so even an engine older than the consumer answers it (it is the negotiation channel, exempt from the gate).

The filter field is ffmpeg's filtergraph syntax¶

The one place we deliberately don't invent our own language. libavfilter ships a complete graph parser; reinventing the [0:v]scale=…[v] mini-language would be folly, and your existing filtergraph knowledge transfers directly. Structured fields surround the graph (inputs, outputs, codecs, options); the graph itself is the standard string.

Transport¶

The spec is passed to the engine as argv[1] — one argument, the whole JSON document. There is no other channel: the engine does not read a spec from a file, from stdin, or from an environment variable. Results — the probe JSON, or the process status — come back on stdout; errors on stderr with a non-zero exit code. Only the media named inside the spec crosses afmpeg's filesystem bridge.

The full host contract is Driver invocation & ABI, and the failure messages are in errors & exit codes.

Versioning¶

The vocabulary carries a version that increments additively — once per landed vocabulary spec, in merge order — so a new field never has to be guessed at by an older engine:

The gate (two sides):

• Engine (runtime). Every process/probe spec is stamped by the consumer with the version it was written in. If that exceeds the engine's AFMPEG_VOCAB_VERSION, the engine rejects the whole spec — stderr message + a distinct exit code 3 (version-too-new, so a caller can tell "upgrade the engine" from the malformed-spec code 2) — rather than silently dropping the fields it doesn't understand.
• Consumer (preflight). afmpeg reads op:"version" once at construction and fails loudly if the module is a gated engine older than the vocabulary it emits, turning a would-be silent field-drop at first job into a clear startup error. A module that doesn't answer op:"version" (a pre-gate engine, or a generic non-ffmpeg-wasi module) is tolerated — it carries no vocabulary contract to check.

The engine's current version is in the --report output and via op:"version".