# Call Video Guide — H.264 over the native calls engine (BETA)

Companion to [`call-audio-implementation-guide.md`](./call-audio-implementation-guide.md).
The audio plane is production-validated; **video is BETA** until proven against a real
WhatsApp video call (the engine's media path still carries `NOT VALIDATED` markers).

The engine (`meowcaller` fork under `whatsmeow-private/calls`) carries **encoded H.264
Annex-B** over the WhatsApp relay. It does **not** encode or decode pixels — an external
codec (the browser's WebCodecs, ffmpeg, a hardware encoder) produces and consumes the
H.264.

---

## 1. Wire format

- **Codec:** H.264, **Annex-B** byte stream (NAL units prefixed with the start code
  `0x00 0x00 0x00 0x01`).
- **Framing:** one WebSocket **binary** message = one access unit (one frame's NALUs),
  reassembled on the RTP marker bit inbound.
- **Keyframes:** send an IDR/keyframe immediately after opening the socket and at a
  sensible GOP interval (~every 1–2 s). The peer cannot decode until it sees a keyframe;
  a decoder cannot configure until it sees the SPS.

---

## 2. Endpoints

| Method | Path | Purpose |
|---|---|---|
| `GET` (WS) | `/call/{call_id}/video/stream?token=…` | Bidirectional H.264 Annex-B |
| `GET` (SSE) | `/call/{call_id}/video/state?token=…` | Peer camera on/off + orientation |

Video is opened **in addition to** the audio socket (`/call/{call_id}/stream`) for a call
placed or answered with video (`dial { video: true }`).

### 2.1 `/video/stream` (WebSocket)

- **Server → client** (binary): the peer's video as Annex-B access units.
- **Client → server** (binary): your camera video as Annex-B access units; the engine
  forwards them to the peer. Do **not** send raw frames, JPEG, or WebRTC tracks.
- **Backpressure:** the server buffers outbound frames and **drops the oldest** when a slow
  client falls behind (newest-wins) — it never blocks the media loop. One video subscriber
  per call (last wins).
- **Outbound camera state:** on your first sent frame the engine announces your camera
  active to the peer (`<video>` state); it announces off when the socket closes. (BETA.)

### 2.2 `/video/state` (SSE)

Proper SSE framing (`data: <json>\n\n`) with a 15 s heartbeat comment and a
`retry: 3000` hint. Each event:

```json
{ "active": true, "upgrade": false, "orientation": 0, "raw": 1 }
```

- `active` — peer camera on (state 1).
- `upgrade` — mid-call audio→video upgrade (state 11).
- `orientation` — 0..3; rotate the rendered video by `orientation × 90°`.

The hub self-cleans: when the last SSE subscriber disconnects, the hub is removed and the
peer callback detached (no per-call leak).

---

## 3. Browser rendering (WebCodecs) — ready-made SDK helper

The dashboard SDK ships `ZZ.sdk.calls.video` (`static/dashboard/js/sdk/modules/calls.js`):

```js
// Feature-detect first.
if (!ZZ.sdk.calls.video.isSupported()) {
  // No WebCodecs — show an "unsupported browser" message.
}

// One call wires the whole session: WS + decode→canvas + camera→encode→WS + state SSE.
const session = await ZZ.sdk.calls.video.startSession({
  callId, token,
  remoteCanvas: document.getElementById("remoteVideoCanvas"), // peer video
  localVideo:   document.getElementById("localVideoPreview"),  // local camera
  sendCamera: true,
  onFirstFrame: () => console.log("peer video live"),
  onState: (st) => console.log("peer camera:", st.active),
});

// Later:
await session.setCameraEnabled(false); // toggle camera mid-call
session.stop();                          // tear down WS + SSE + camera
```

What the helper does:

- **Decode (peer → `<canvas>`):** a `VideoDecoder` fed Annex-B `EncodedVideoChunk`s. It
  derives the `avc1.PPCCLL` codec string from the **SPS** (profile/constraint/level bytes),
  **gates on the first keyframe** before configuring, and rotates by the peer's orientation.
- **Encode (camera → Annex-B):** a `VideoEncoder` configured `avc: { format: "annexb" }`,
  `latencyMode: "realtime"`. Frames come from `MediaStreamTrackProcessor` where available,
  else a `<canvas>` draw-loop + `new VideoFrame(...)`. A keyframe is forced ~every 3 s.

Lower-level building blocks are also exposed: `createDecoder({canvas})`,
`createEncoder({width,height,onChunk})`, `hasKeyframe(bytes)`, `codecFromAnnexB(bytes)`.

### 3.1 Browser support

WebCodecs is in current Chromium (Chrome/Edge) and Safari 16.4+. Where it is missing,
`isSupported()` returns `false` — surface an "unsupported browser" message. A full **MSE
fallback** would require an in-browser **Annex-B→fMP4 muxer** (not shipped) — documented
here as a known limitation, not a silently broken path.

---

## 4. Audio ↔ video parity

| Capability | Audio | Video |
|---|---|---|
| Dial / answer with media | ✅ | ✅ (`dial { video: true }`) |
| Two-way media socket | `/…/stream` (PCM) | `/…/video/stream` (H.264 Annex-B) |
| Peer state signal | mute events | `/…/video/state` SSE |
| Outbound state to peer | — | `<video>` active/off on send/close (BETA) |
| Incoming-media flag | media attr | `v1.call.offer.is_video` (webhook, additive) |

The `v1.call.offer` webhook now carries **`is_video`** (derived like the engine's own
detection: a `<video>` child or `media="video"`), so integrators can route "incoming video
call" without parsing `raw_xml`. Additive-only, under the `v1` key.

---

## 5. BETA status — honest diagnostic + what "validated" will mean

The video plane is meowcaller's **initial / experimental** video support (the upstream
`purpshell/meowcaller` engine calls it "initial video calls support"). It is **not** a
finished protocol implementation, and this fork does **not** reverse-engineer the missing
pieces. Treat every video path as **BETA** until proven on real devices.

### 5.1 Why video may not reach the peer (the honest version)

This is **not a bug in our sync or wiring** — it is an unfinished upstream feature:

- **Egress is real.** On the send side, `videoSender.send()` (`calls/engine_media.go`)
  genuinely packetizes the H.264 access unit into RTP, protects it with SRTP, and hands it
  to the relay channel (`vs.ch.Send`). Bytes leave the process.
- **The bridge is unproven.** meowcaller's video assumes WhatsApp bridges the **video SSRC**
  onto the same relay channel as audio. That assumption was **never captured or confirmed**.
  In real runs, audio RTP flows both ways but **no `first video RTP demuxed` line ever
  appears** — i.e. we never observe inbound video, which is the tell that the peer-side
  bridge isn't happening the way the initial implementation assumes.
- **Making it work = protocol reverse-engineering** (live capture of a real WhatsApp video
  call, same methodology as the `cstoken`/NCT-salt work). That is explicitly **out of scope**
  here. This doc records the gap honestly rather than shipping a silently-broken "feature".

### 5.2 Code-complete but `NOT VALIDATED`

The following compile and send/receive at the wire level, but are unproven against a real
WhatsApp video call. They keep their `NOT VALIDATED` markers until a live capture succeeds:

- inbound video RTP demux (`livecall.go` `ReceiveVideo` — `NOT VALIDATED`)
- outbound video send (`SendVideo` / `sendVideoFrame` — `NOT VALIDATED`)
- outbound video call placement (`CallVideo` / `placeCallVideo`, in `calls/video_fork.go` — `NOT VALIDATED`)
- outbound `<video>` state signaling (`SetVideoState` / `sendVideoState`, in `calls/video_fork.go` — `NOT VALIDATED`)
- accept-video (`engine.go` `sendAccept` answers an inbound video offer with a `<video>` node)

### 5.3 Validation runbook

To attempt end-to-end validation (and flip a path from BETA to validated):

1. Enable engine trace logging for the calls package (the engine logs via the injected
   `zerolog` logger; `diag.Recorder` also emits `keying` / `meta` events).
2. From the dashboard (or API), **dial a video call** (`dial { video: true }`, i.e.
   `CallVideo`) to a **second real phone** you control — or place an audio call from the
   phone and let the widget answer with the accept-video path.
3. **Answer on the other device** and turn its camera on.
4. Watch the logs for **`first video RTP demuxed`** (inbound video) and for the peer
   rendering our camera (outbound).
   - If it appears → inbound video actually bridges; capture the RTP vector and add it as a
     test fixture, then the corresponding `NOT VALIDATED` marker can be removed.
   - If it never appears (expected today) → you have hit the unproven bridge layer in §5.1;
     that is the reverse-engineering gap, not a regression.

Do **not** remove the `NOT VALIDATED` markers or the **BETA** labels until frames are
confirmed peer→screen and camera→peer on a real device.