Becoming Friends with Your Camera

Why the second one looks like me and the first one doesn’t

A pinhole camera puts a point of height h at depth d onto the sensor at height f·h/d. The important part is the division by d: the image of a three-dimensional object is not a uniform scaling of it, it is a scaling that gets weaker the further away a feature is. So the ratio of depths across the face is what decides its rendered shape.

That single number explains everything visible in shot A. The nose, lips and the front of the cheeks are the closest points, so they are magnified most; the ears are the furthest, so they are magnified least and disappear behind the cheekbones entirely. The face reads as if it were pushed outward from the middle, because geometrically it was.

In shot B the whole head sits inside a 5 % depth window, so every feature is scaled by almost the same factor. The projection is close to orthographic and the proportions survive.

The same arithmetic as the selfie, on a bigger object

The building has two faces meeting at a corner. Call the near corner depth d and the far end of the receding wall d + Δ, where Δ is fixed by the building itself (roughly its length, maybe 15 m) and has nothing to do with where I stand. Everything follows from the ratio again:

In A the far end of the wall is only 33 % smaller than the near corner, so the wall hardly tapers, the roofline is nearly horizontal, and the two façades keep similar widths. Layers of the scene that are genuinely far apart — the foreground trees, the wall, the sky — arrive at nearly the same magnification and therefore look glued together. That is perspective compression: it is not the telephoto lens flattening anything, it is the long shooting distance making all depth ratios approach 1.

In B I walked in until the near corner was a few metres away. Now the far end of the wall is 3.5× smaller than the near corner, so the wall visibly rushes to its vanishing point, the cornice dives down to the right, and the vertical edges splay because the camera is tilted up and no longer far enough away for the projection to be locally orthographic. The near corner dominates the frame the way the nose dominated shot A in Part 1.

What I shot, and why it isn’t a filmstrip

Doing this by hand needs the subject to stay registered to within a few pixels across every frame while both the tripod and the focal length move. Instead I shot a matched pair on a tripod-still phone — one frame with the subject on the table, one clean plate of the same scene with the subject removed — and then synthesised the entire sweep in code. Every intermediate frame you see below was computed, not photographed.

I want to be straightforward about the trade-off: this is a reconstruction of a dolly zoom rather than a physically executed one, and its depth model is a smooth radial approximation rather than a measured depth map. What it buys is a continuous, controllable sweep instead of eight discrete stills — which is what makes the slider below possible.

The geometry the renderer implements

A subject of height h at distance d covers f·h/d on the sensor. Holding the subject the same size therefore means holding f/d constant: if the camera retreats by a factor s, the focal length must grow by the same s. The background does not cooperate, because its extra depth Δ does not scale:

With a single photograph there is no Δ per pixel, so I approximate the scene's depth structure with a smooth radial profile: pixels near the locked centre are treated as near the subject's depth and are scaled least, pixels out towards the frame edge are treated as further and are scaled most. The background scale field is 1 + 0.65·(s−1)·(0.25 + 0.75·exp(−(r/0.68R)²)), and the subject gets a small counter-contraction 1 + 0.06t − 0.35t·(r/r_obj)^1.35 so its silhouette stays put while its own internal perspective changes very slightly. Both fields are radial, which is the fact the JavaScript port exploits.

Pipeline

1. Difference-of-Gaussian keypoints

   Five Gaussian blurs of the grayscale image, σ = 1.6 with k = 2^1/3, differenced into a four-level DoG stack. Non-maximum suppression over 3×3×3 neighbourhoods with a |D| threshold of 0.015 gives 842 keypoints. keypoints.py: build_dog_space, find_local_extrema

2. Patch descriptors

   A 9×9 window of the DoG level around each keypoint, flattened to 81 numbers, mean-subtracted and divided by its standard deviation — cheap invariance to brightness and contrast. keypoints.py: extract_descriptors

3. Ratio-test matching

   Euclidean distances between every pair of descriptors; a match is kept only if the best is closer than 0.8× the second best, so ambiguous wood-grain texture is rejected. 373 candidate matches survive. extractor.py: match_features

4. Homography by normalised DLT inside RANSAC

   3000 iterations, four random correspondences each, Hartley-normalised DLT solved by SVD, 3.5 px inlier threshold, then a re-fit on all inliers followed by three refinement rounds. 239 inliers. extractor.py: solve_homography, estimate_homography_ransac

5. Warp the plate into the subject's frame

   The estimated homography is inverted, normalised, and handed to a perspective resample so the two exposures share one coordinate system. main.py: warp_to_reference

6. Segment the subject

   Chroma seeds (a yellow test in RGB plus a saturation floor, intersected with the difference against the aligned plate), connected components to drop specks, then geodesic region growth constrained to a dilated band around the seed, hole filling, and a feathered alpha. main.py: foreground_mask

7. Radial field-of-view warp and composite

   Inverse-mapped radial resample of the plate and of the premultiplied subject with their two different scale fields, blended by the warped alpha. main.py: radial_warp, make_frames

Evidence

Porting the renderer to the browser

A direct transliteration of radial_warp would evaluate a square root, a division and two profile functions per pixel per frame, which is far too slow to drag. The saving grace is that every scale field in the renderer is a function of the radius and nothing else. The inverse warp needs r_src = r / s(r), so the whole per-pixel geometry collapses into a one-dimensional lookup table rebuilt once per frame — a few thousand entries instead of seven hundred thousand transcendental evaluations.

Two further things earn their keep. Compositing in premultiplied alpha means the warped foreground and the warped alpha go straight into bg·(1−α) + fg with no divide-then-multiply round trip. And because the subject only ever contracts towards the locked centre, the foreground fetch is skipped entirely outside the matte's bounding box, which is most of the frame. The result is a full inverse warp of both layers, per pixel, at interactive rates — the render time is printed in the panel.

Porting the rest of it too

The renderer alone still needed my Python to have run first, because it starts from three precomputed tensors. So pipeline.js ports everything upstream of it as well — all seven stages above — and the panel below the lab runs them on a pair of photographs you supply. Nothing is uploaded: the files never leave the browser.

Reproducing NumPy and SciPy honestly is most of the work. scipy.ndimage's Gaussian is matched exactly, down to its truncate=4.0 kernel radius and its reflecting boundary. Its binary morphology defaults to a 4-connected cross rather than a 3×3 square, so this does too, and binary_propagation becomes a breadth-first flood inside the permitted region. The one place I deliberately diverge is binary_dilation(seed, iterations=14): a cross dilated fourteen times reaches exactly the pixels within city-block distance fourteen, so a two-pass chamfer distance transform gives the identical result for a fraction of the work.

The linear algebra needed replacing outright, since there is no numpy.linalg in a browser. The normalised DLT wants the null vector of a 2N×9 design matrix; rather than pull in a library, the solver runs a one-sided Jacobi SVD — rotate pairs of columns until they are mutually orthogonal, and the shortest resulting column is the smallest singular value with the accumulated rotations as its singular vector. It is about thirty lines, it is numerically well behaved on Hartley-normalised coordinates, and it is fast enough to run inside all three thousand RANSAC iterations.

One more shortcut is worth naming because it changes the arithmetic. The descriptors are mean-subtracted and divided by their standard deviation, so every one of them satisfies ‖v‖² = 81 exactly. That makes ‖a−b‖² = 162 − 2a·b, so the entire matching stage reduces to a dot product per pair — the Euclidean distances the ratio test needs are recovered exactly from it, at roughly half the cost.

Code

• keypoints.py DoG scale space, extrema detection, normalised patch descriptors
• extractor.py ratio matching, normalised DLT homography, RANSAC and refinement
• main.py alignment, matting, the radial dolly-zoom renderer, GIF output
• export_web.py builds every figure and asset on this page
• docs/pipeline.js the JavaScript port of the detection, matching, alignment and matting stages
• docs/dolly-lab.js the JavaScript port of the renderer, and the interface for both panels