# Warp Fun

### About:

Given a set of coplanar points in world space and their projection onto two different 2D image planes (i.e.: as seen by two different cameras or from two different points of view), a 2D planar homograhpy is a mapping between these two images of the same points. This mapping is useful in a variety of vision and computational photography tasks involving planes, such as image recognition, image stitching (i.e.: panoramas) and image rectification.

[Figure 1. The homography drawing]

In this project I implemented a general approach to finding homographies between sets of corresponding points in images as well as a few applications of the resulting homographies.

### Finding a Homography between Corresponding Points in two Images:

From Figure 1 and assuming we have two images (a source $\mathbf{S}$ and a target $\mathbf{T}$) which depict the same world-space coplanar points from different points of view, we wish to find a homography $\mathbf{H} \in \mathbb{R}^{3 \times 3}$ such that we can reproject these points (pixel coordinates) in $\mathbf{S}$ to their counterparts in $\mathbf{T}$.

Additionally, since the homography operates on projective space:

• Source points need to be converted from euclidean to projective space before multiplying them by $\mathbf{H}$.
• The resulting target points need to be converted from projective to euclidean space in order to obtain their final 2D coordinates.

So we want $\mathbf{H}$ such that given source pixel locations $\mathbf{s}^{(i)}$ and target pixel locations $\mathbf{t}^{(i)}$:

Which can be expressed as a system of linear equations $\mathbf{A}\mathbf{h} = \mathbf{b}$. In this system $\mathbf{h} \in \mathbb{R}^9$ is the flattened homography matrix and $\mathbf{A} \in \mathbb{R}^{n \times 9}: n >= 9$ represents x and y-constraints (one per pixel coordinate; at least 4 pixel coordinates) plus a constant scale constraint which makes the system inhomogeneous and solvable via linear least squares.

The constraints are as follow:

• x: %

• y: %

• scale: %

From these constraints the system can be put together as:

And the solution to $\text{min}_\mathbf{h}\ \lVert\mathbf{A}\mathbf{h} - \mathbf{b}\rVert^2_2 = (\mathbf{A}^T \mathbf{A})^{-1} \mathbf{A}^T \mathbf{b}$ will be $\mathbf{H}$. Also, since 2D planar homographies are invertible, we have $\mathbf{H}_{S2T}^{-1} = \mathbf{H}_{T2S}$ which means that having computed the homography in one direction, we can invert it to reproject in the other direction.

### Warping Images:

Once a homography between source and target coordinates is computed, the whole source image can be warped by reprojecting all its pixel coordinates onto the target image’s space. This is referred to as forward warping.

One issue with forward warping is that integer pixel coordinates in the source will likely map to subpixel coordinates in the target, requiring some interpolation scheme to figure out the values of a number of neighboring target pixels for the result to look acceptable. The alternative, known as inverse warping, instead uses the inverse of $\mathbf{H}_{S2T} = \mathbf{H}_{T2S}$ so that the problem is reformulated as:

Which is easier to approach since it requires computing a single interpolated value given a subpixel lookup in the source. The applications in this project use inverse warping with bilinear interpolation other than to figure out the sizes of final images.

### Application: Fake Billboards:

This application reprojects a frontal image $\mathbf{S}$ onto several locations in an image $\mathbf{T}$, as if $\mathbf{S}$

TODO

TODO