Dither

I keep coming back to John Provencher's work. He takes a photo, removes almost all the detail, and you can still tell what it is from just a few hundred marks.

I wanted to understand how it works well enough to control it myself, so I basically put everything on a slider. There are five ways to break a picture down, and they all share one pipeline in a single file with no dependencies.

How it works

Each one cuts the image into a grid, averages every cell down to one colour and one brightness, and then draws a mark in that cell based on those two values. The character map uses a letter, the mark grid uses a shape, halftone uses a dot size and ordered dither uses a colour from the palette. Only that last step changes, so they all sit behind one function, and switching between them feels like picking up a different tool.

Drawing the marks

My first go at the mosaic drew everything with fillText. I assumed a set of solid and hollow squares would cover most of what I needed, but it didn't. A letter sits wherever the font's metrics put it in the cell, so filled squares never quite meet. An outline has whatever stroke weight is built into the typeface, and a dot is always the same size. The text renderer controls all of that and you can't really get at it.

Drawing the marks as vectors got me all of that back. I kept the characters as names for the shapes, which also meant I could add triangles and diamonds without worrying about whether a font has them.

// The characters are tokens, not text. The glyph only has to be typeable
// and suggestive of what it draws.
const MARKS = {
  '■': { kind: 'fill', size: 1 },
  '□': { kind: 'ring', size: 1 },
  '●': { kind: 'dot',  size: 0.86 },
  '▲': { kind: 'tri',  size: 1, up: true },
}

// A filled mark tiles the whole cell, an outline holds a real stroke
// weight, and a dot's radius is free to track tone. Text metrics can do
// none of those.
const inner = cell - gutter * 2
drawMark(ctx, MARKS[token], mx, my, inner, lineWidth, colour)

Halftone

I sized the halftone dots so the radius matched how dark the cell was, which seems like the obvious way to do it but it's wrong. The midtones came out too heavy and the whole image went too dark. Ink coverage is about area, and area goes up with the square of the radius, so a cell that's half dark needs a radius of the square root of a half. Adding that square root fixed it.

const coverage = 1 - luminance / 255

// Area proportional to coverage, so the tonal ramp stays linear.
// Using coverage directly here is the classic mistake: it makes
// every midtone read a stop too dark.
const r = cell * 0.5 * Math.sqrt(coverage) * dotScale

Blur

Merging the halftone dots into blobs is done with a blur and then a threshold. The blur softens the dots so the ones next to each other merge, and the threshold turns them back into hard edges. My first version took the blur radius in pixels, which seemed fine until I made the screen coarser. With big spacing a fixed radius starts averaging across whole cells when it should only be merging dots next to each other, so the midtones lose their detail and the image ends up as a couple of big shapes.

Setting it as a fraction of the screen spacing fixed it, and now the control means the same thing at every dot size. I've started checking anything I measure in pixels at both ends of its slider because of this.

Video

Error diffusion passes each pixel's rounding error on to the pixels next to it, so every pixel depends on all the ones processed before it. If the input changes by even one value, everything after that point comes out differently. On a still image you'd never notice, but on video a flat wall flickers with noise, because two frames that look the same to me are slightly different in the actual pixel values.

Ordered dither and halftone don't have that problem. A pixel's threshold comes from its own position, so the same input in the same place gives the same result every frame and flat areas stay completely still. That's mostly why I use those two for motion work, and why I put a warning inside the error diffusion panel where you'd actually run into it. I'd only use Floyd-Steinberg on video if I actually wanted that texture.

Exporting

Recording the canvas live is one click, but it drops frames as soon as the filter can't keep up. Stepping through the clip frame by frame and saving a numbered sequence is exact but slow. I needed both at different times so I built both. The sequence goes straight into ffmpeg and comes out as ProRes.

Drawing tens of thousands of letters straight onto the on-screen canvas took fifteen seconds. Doing the same thing on a separate canvas and copying the finished frame across once took a second and a half, with exactly the same algorithm.

Oh yeah and I made it look like early Photoshop because why not?