Overview

These are notes from a task where I recorded a demo video of a 3D gallery built with three.js + WebXR (a Web app of the type where you walk through a virtual space and view works) from within the app, without relying on external screen-recording software. Recording itself can be written in a few lines with canvas.captureStream() and MediaRecorder, but when I actually tried to make it "a video you can show to people," there were several places where I stumbled more than I'd expected.

I'll leave a record of the pitfalls and remedies I found as far as I investigated, plus a way to automate "moving the camera automatically, recording, and saving the file — all unattended." I think the same discussion applies not just to three.js but to WebGL apps that draw to a canvas in general.

Premise: in-app canvas recording, or screen recording

There are broadly 2 ways to record a 3D app's video.

  • In-app canvas recording: record only the pixels of the WebGL canvas with canvas.captureStream(). The DOM — HUD, buttons, and so on — isn't captured, so only the 3D picture remains cleanly. No permission dialog appears.
  • Screen (tab) recording: record the screen or a tab with navigator.mediaDevices.getDisplayMedia(). DOM overlays are captured too, but a sharing-permission dialog appears every time, and the cursor and the browser frame can get in.

If you just want to "record the 3D viewing footage cleanly," the former is suited. On the other hand, if you want to capture a viewer implemented on the DOM side (discussed later), the former can't record it and the latter is needed. This article is mainly about the former (canvas recording).

Minimal implementation

First, a naive version looks like this.

function startRecording(canvas, fps = 30) {
  const stream = canvas.captureStream(fps);
  const recorder = new MediaRecorder(stream, {
    mimeType: "video/webm;codecs=vp9",
    videoBitsPerSecond: 12_000_000,
  });
  const chunks = [];
  recorder.ondataavailable = (e) => { if (e.data.size) chunks.push(e.data); };
  recorder.onstop = () => {
    const blob = new Blob(chunks, { type: "video/webm" });
    const a = document.createElement("a");
    a.href = URL.createObjectURL(blob);
    a.download = "demo.webm";
    a.click();
  };
  recorder.start();
  return recorder;
}

For three.js, the canvas to record is renderer.domElement. If you draw every frame with setAnimationLoop (the loop you normally use for WebXR support), that content rides straight onto the stream. This does record, but I stumbled on the following points.

Stumble 1: DOM overlays don't appear

What canvas.captureStream() records is only that canvas's pixels. HTML overlays (operation hints, the work title display, a crosshair cursor, etc.) are DOM on a separate layer, so they don't enter the video at all.

This is also the advantage that "you get a clean video without buttons showing," but it has the aspect of erasing information you want to show, too. Two things became particular problems this time.

Text like work titles disappears

I normally showed the work title with an HTML HUD, but that doesn't appear in the recording. As a remedy, I decided to draw the text as an object inside the 3D scene. I painted the characters with Canvas 2D onto a CanvasTexture, pasted it onto a Plane, and placed it a little in front of the camera.

// Place the board with text drawn on it in world coordinates, a little in front of and below the camera
const offset = new THREE.Vector3(0, -0.3, -0.9).applyQuaternion(camera.quaternion);
captionMesh.position.copy(camera.position).add(offset);
captionMesh.quaternion.copy(camera.quaternion);
scene.add(captionMesh);

One thing I got stuck on here was that making the board a child of the camera means it isn't drawn. Since renderer.render(scene, camera) draws by traversing scene, camera-child objects not in scene aren't in the traversal (they're drawn if you've added the camera to the scene, but I think usually you don't). Even though opacity and such were set correctly in the program, it didn't appear on screen, and I couldn't figure out the cause for a while. In the end I stopped making it a child of the camera and re-placed it "in the world coordinates in front of the camera" every frame.

As a lesson, I should have checked the recording's content by looking at the actually-written frames with my eyes. It was a mistake to judge "it should be captured" from just checking the DOM state (the element's opacity being 1, etc.); when I pulled a frame and looked, it wasn't there.

A viewer on a separate layer doesn't appear

If a high-resolution image viewer (the type that progressively loads tile images and zooms in) is overlaid on the DOM by a separate library, it too doesn't enter the canvas recording. If you want it in the video, it becomes one of:

  • record the whole tab with getDisplayMedia() (in exchange for a permission dialog),
  • composite that viewer's canvas into the WebGL side (heavy to implement), or
  • substitute a rendering of "in the 3D scene, move in on the target and swap in a high-resolution texture."

This time I took the third one, which completes within the recording (moving the camera in on the target while swapping the texture for a high-definition version). The framing is to switch to screen recording only when you want to show the real viewer UI.

Stumble 2: the resolution becomes huge

What captureStream() records is the canvas's backing-store resolution. On a Retina or other devicePixelRatio = 2 environment, even if the display is 1800px wide, the backing is 3600px, and the recording is that size too (precisely, renderer.getPixelRatio() times, which is devicePixelRatio clamped to an upper bound). In fact, the first thing I recorded was a 3600×2018 file.

For 3D viewing footage this is excessive — the file is large and playback becomes heavy. As a remedy, I made it lower the draw resolution only while recording. Keeping the aspect ratio, I lower pixelRatio so it doesn't exceed a height cap (e.g. 1440px), and restore it when recording ends.

// Only during recording, to about height 1440px. Keep the aspect ratio
const targetH = 1440;
const cur = renderer.getPixelRatio();
const dpr = Math.min(cur, targetH / window.innerHeight);
renderer.setPixelRatio(dpr);          // the canvas's backing store shrinks = recording resolution drops
composer.setPixelRatio(dpr);           // reflect it on the post-process render target too
// …after recording ends, setPixelRatio again with the original value to restore

I got stuck on one point here. EffectComposer (post-processing) holds the pixelRatio from construction time internally, and calling composer.setSize() doesn't update that value. To lower the resolution for recording you need to use composer.setPixelRatio(dpr). With renderer.setPixelRatio() alone, the canvas (= recording resolution) drops, but the post-process render target stays at the original size, wasting draw cost.

Note that when going through EffectComposer, the renderer's antialias: true doesn't take effect (because the draw destination becomes its own render target). In WebGL2 you can bring anti-aliasing back by making the render targets multisample.

if (renderer.capabilities.isWebGL2) {
  composer.renderTarget1.samples = 4;
  composer.renderTarget2.samples = 4;
}

Thin, high-contrast contours like frames and fixtures tend to stand out as flicker (shimmer) under video compression, so tightening this up made the post-compression appearance stable.

Stumble 3: choosing the codec (VP9 or H.264)

The default when you don't specify a mimeType for MediaRecorder depends on the environment, but it becomes WebM (VP8 in many environments, VP9 in some). There are a fair number of environments without hardware VP9 decode, in which case playback is on the CPU and becomes heavy, especially at high resolution. Also, WebM sometimes can't be played directly in some environments like QuickTime.

As a remedy, I made it prefer H.264 (MP4) when available. I confirm support with MediaRecorder.isTypeSupported() before choosing.

function pickMime() {
  const cands = [
    "video/mp4;codecs=h264",
    "video/mp4;codecs=avc1.42E01E",
    "video/webm;codecs=vp9",
    "video/webm;codecs=vp8",
    "video/webm",
  ];
  return cands.find((m) => MediaRecorder.isTypeSupported?.(m)) || "";
}

As far as I investigated, in recent Chrome-family browsers the number of environments where MediaRecorder can output MP4/H.264 is increasing (it seems limited to cases where the OS has a hardware encoder, and gating with isTypeSupported() is a prerequisite). Being able to choose this let it open directly in many playback environments and play lightly with hardware decode. It's also safe to decide the save extension from blob.type (always saving as .webm can mismatch the content). Deciding the bitrate to match resolution and fps (roughly around 0.1 bpp) wastes less than a fixed value.

Stumble 4: headless automated capture

I wanted to automate through capture, and tried to drive a headless Chrome via CDP (Chrome DevTools Protocol) to record, but this is where I struggled the most. Within what I observed, there were the following behaviors.

  • Trying to use the GPU headlessly can produce black frames. Even trying to enable the GPU with --use-angle=metal and such, headlessly the draw context is effectively ineffective, producing an almost pitch-black video (= an extremely small file since there's almost no information). To draw reliably, using a software renderer (SwiftShader: --use-gl=angle --enable-unsafe-swiftshader) was stable. Image quality is inferior to the GPU, but frames with actual content come out reliably. This is specific to headless / GPU-less environments, and SwiftShader is the official software path for that.
  • When the window isn't visible, the draw loop itself gets throttled. Throttling of requestAnimationFrame / setAnimationLoop is decided by visibility, not by focus. If the tab is in the background, the window is minimized, or it's completely covered by other windows (document.visibilityState === "hidden", so-called occluded), it's throttled. Conversely, as long as it's visible on screen — even if not frontmost — it basically isn't throttled. This time I launched a separate window and pushed it to the back, and captureStream kept duplicating the same frame, producing an almost-frozen video (about 100KB in a few tens of seconds).

To organize it, my sense was that unattended automated capture and high-resolution GPU quality are hard to satisfy at the same time. For usage, I settled on a two-tier approach:

  • run change confirmation and regression checks unattended → headless + software renderer (keep the resolution modest so CPU drawing doesn't drop frames)
  • the clean production footage → press the button on the frontmost tab I'm looking at (real GPU and visible, so not throttled)

Even with the software renderer, if I kept the resolution to around 720p, a video of continuous frames at about 30fps came out properly (as in "at 100KB for the same duration it was frozen, and when 25MB came out it was continuous" — file size is a rough gauge of continuity).

From launch to save with CDP

With CDP you can automate the whole thing: launch → page load → call the in-app tour function → record → save the file. I'll just list the key points.

  • Launch Chrome with --headless=new --remote-debugging-port=… --user-data-dir=… (a disposable profile), get the page's debugging WebSocket from the /json endpoint, and connect.
  • For saving, if you specify downloadPath with Browser.setDownloadBehavior, the app-side download (a.click()) writes out to that folder (the old Page.setDownloadBehavior is deprecated).
  • The app's modules can be called from Runtime.evaluate via a dynamic import(). One thing I got stuck on here is that top-level await can't be used in Runtime.evaluate expressions (in the default mode). You need to wrap it in (async () => { … })().
  • Before starting recording, waiting until content loading stabilizes was also important. This time it was built to fetch images from outside, so if I started recording "at the moment the first one appeared," most of the works would still be unloaded in the footage. I solved it by waiting until the element count stopped changing for a certain time (note that it's only a gauge of load completion, not a guarantee).
// Expression passed to Runtime.evaluate: top-level await isn't allowed, so wrap it in an IIFE
const expr = `(async () => {
  const tour = await import('./src/tour.js');
  await tour.startTour({ record: true });
  return true;
})()`;

Verification: check with your eyes / check with a program

Recording is prone to the failure of "it looks like it's working but isn't actually captured / is frozen," so having two ways to verify gave peace of mind.

  1. Pull out frames and look with your eyes. With mkdir -p frames && ffmpeg -i out.mp4 -vf fps=1 frames/%02d.png, extract an image per second (ffmpeg doesn't create the output directory, so create it first) and confirm the picture is as expected (text burned in, the move-in, the subject's position). The earlier "the text wasn't captured" issue was something I only noticed by doing this.
  2. Assert with a program. Properties hard to judge by appearance (e.g. whether the subject clips through a fixture, whether the camera goes outside a wall) are more reliable when sampled numerically every frame. This time I checked, across all frames, that the subject's coordinates weren't inside the collision region and that the camera's radius stayed inside the wall.
// Example: every frame, confirm the camera is inside the wall (radius wall)
const id = setInterval(() => {
  const r = Math.hypot(camera.position.x, camera.position.z);
  if (r > wall + 0.05) console.warn("camera outside wall", r);
}, 16);

avg_frame_rate becoming 0/0 (variable frame rate) is normal for MediaRecorder's WebM. If it bothers you, you can convert to a fixed frame rate with ffmpeg.

Presentation touches (bonus)

Things I added, after the content (recording method) had settled, to make it easier to watch as a demo. All are unassuming but effective.

  • Add an intro that surveys the space at the beginning. Rather than moving straight in on a work, first showing the whole space slowly conveys the situation.
  • Make travel time proportional to distance. With a fixed number of seconds per segment, near targets look slow and far targets look fast — unnatural. Clamping and using distance / speed gives a sense of constant speed.
  • Don't visit everything; narrow to a few points. Since it's a demo, thinning uniformly and touring only a few representative points keeps it well-paced.
  • Reuse the existing collision detection. When moving the camera or character automatically, reusing the movement resolution used in normal operation (the processing that slides per axis on collision) prevents clipping through fixtures. Writing your own separate movement logic tends to drop these existing considerations.

Summary

  • canvas recording (captureStream + MediaRecorder) is easy, but I first stumbled on 3 points: the DOM doesn't appear, the resolution becomes huge, and the codec affects playback compatibility.
  • Draw the information you want to show, such as text, inside the 3D scene. Placing it in front of the camera in world coordinates, rather than as a child of the camera, was reliable.
  • With adjustments like lowering the resolution only during recording (don't forget composer.setPixelRatio too) and preferring H.264, I could keep image quality while making the file manageable.
  • For automated capture, headless + software renderer is stable. But high resolution and GPU quality favor manual capture on the frontmost tab, and using each by purpose was realistic.
  • Verifying recording with both wheels — inspecting frames with your eyes and asserting with a program — reduced oversights.

My takeaway this time is that without adding special libraries, you can get this far within the range of the browser's standard APIs and CDP.