#+title: mrcal - camera calibrations and more!
#+AUTHOR: Dima Kogan

#+OPTIONS: toc:nil H:2

#+LATEX_CLASS_OPTIONS: [presentation]
#+LaTeX_HEADER: \setbeamertemplate{navigation symbols}{}

# I want clickable links to be blue and underlined, as is custom
#+LaTeX_HEADER: \usepackage{letltxmacro}
#+LaTeX_HEADER: \LetLtxMacro{\hreforiginal}{\href}
#+LaTeX_HEADER: \renewcommand{\href}[2]{\hreforiginal{#1}{\color{blue}{\underline{#2}}}}
#+LaTeX_HEADER: \renewcommand{\url}[1]{\href{#1}{\tt{#1}}}

# I want a visible gap between paragraphs
#+LaTeX_HEADER: \setlength{\parskip}{\smallskipamount}

* Overview
** Where is all this?
Documentation, installation instructions and everything else are available here:

- http://mrcal.secretsauce.net/

This talk is a condensed version of the "tour of mrcal" page:

- http://mrcal.secretsauce.net/tour.html

Please see that page for a bit more detail, and for links to all the data and
commands in the studies I'll discuss

** mrcal: calibrations and other stuff
I couldn't find a set of tools to make my visual ranging work possible, so I
wrote my own:

- A set of C, Python libraries to work with
  - camera models
  - 3D geometry
  - images
- Lots of analysis, tools available
- Fancy lens models
- Lots of visualization facilities
- Many commandline tools available, so no coding required for many tasks

** Tour of mrcal
I will go through the "tour of mrcal" page:

- http://mrcal.secretsauce.net/tour.html

We follow a real-world data flow, from chessboard observations to stereo
processing. Images captured using

- Nikon D750 full-frame SLR. 6000x4000 imager
- /Very/ wide lens: Samyang 12mm F2.8 fisheye. 180deg field of view
  corner-corner
- Just one camera

* Corners
** Gathering corners

This is a wide lens, so we have a large chessboard:

- 10x10 point grid
- 7.7cm between adjacent points

Most observations take right in front of the lens, so depth of field is a
concern. Thus

- Images gathered outside
- F22

** Corner detections
mrgingham corner detections look like this for an arbitrary image:

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/calibration/mrgingham-results.png]]

* Calibrating opencv8
** Let's run a calibration!
This is a wide lens, so we need a lens model that can handle it. Let's use the
8-parameter OpenCV model: =LENSMODEL_OPENCV8=

#+begin_example
$ mrcal-calibrate-cameras --lensmodel LENSMODEL_OPENCV8 ...
...
RMS reprojection error: 0.8 pixels
Noutliers: 3 out of 18600 total points: 0.0% of the data
calobject_warp = [-0.00103983  0.00052493]
#+end_example

Let's examine the solution. We want the errors in the solve to follow the mrcal
noise model, and if they don't, we want to try to fix it.

** Noise model
mrcal assumes that

- The model (lens parameters, geometry, ...) accurately represents reality
- All errors (differences between the observations of the chessboard and what
  the model predicts) come from observation noise
- The errors are independent, gaussian and have the same variance everywhere

If all those assumptions are true, then the results of the least-squares
optimization (what the calibration routine is doing) are the maximum-likelihood
solution.

We will never satisfy all these assumptions, but we should try hard to do that.

** Does the solved geometry look right?

#+ATTR_LATEX: :width 0.8\linewidth
[[file:../external/figures/calibration/calibration-chessboards-geometry.pdf]]

Yes. That's how I danced.

** =LENSMODEL_OPENCV8= residuals histogram
What does the error distribution look like?

#+ATTR_LATEX: :width 0.9\linewidth
[[file:../external/figures/calibration/residuals-histogram-opencv8.pdf]]

** =LENSMODEL_OPENCV8= worst-observation residuals
The worst-fitting observations are a great way to see common issues such as:

- out-of focus images
- images with motion blur
- rolling shutter effects
- synchronization errors
- chessboard detector failures
- insufficiently-rich models (of the lens or of the chessboard shape or anything
  else)

Any of these would violate the assumptions of the noise model, so we want to fix
them

** =LENSMODEL_OPENCV8=: the worst image
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/calibration/worst-opencv8.png]]

** =LENSMODEL_OPENCV8=: the 3rd-worst image
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/calibration/worst-incorner-opencv8.png]]

** =LENSMODEL_OPENCV8=: residual directions
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/calibration/directions-opencv8.pdf]]

** =LENSMODEL_OPENCV8=: conclusions
The =LENSMODEL_OPENCV8= lens model does not fit our data in observable ways.

These unmodeled errors are small, but cause big problems when doing precision
work, for instance with long-range stereo.

Let's fix it.

* Calibrating splined models
** =LENSMODEL_SPLINED_STEREOGRAPHIC= definition
- We need a more flexible lens model to represent our lens.
- mrcal currently supports a /splined/ model that is configurable to be as rich
  as we like

We compute a normalized /stereographic/ projection:

\[ \vec u = \mathrm{project}_\mathrm{stereographic}\left(\vec p\right) \]

This maps a 3D direction vector to a 2D point $\vec u$. This works behind the
camera, so wide-angle lenses are supported well.

** =LENSMODEL_SPLINED_STEREOGRAPHIC= definition
Then use $\vec u$ to look-up an adjustment factor $\Delta \vec u$ using two
splined surfaces: one for each of the two elements of

\[ \Delta \vec u \equiv
\left[ \begin{aligned}
\Delta u_x \left( \vec u \right) \\
\Delta u_y \left( \vec u \right)
\end{aligned} \right] \]

We can then define the rest of the projection function:

\[\vec q =
 \left[ \begin{aligned}
 f_x \left( u_x + \Delta u_x \right) + c_x \\
 f_y \left( u_y + \Delta u_y \right) + c_y
\end{aligned} \right] \]

** Let's re-run the calibration
Let's re-process the same calibration data using this splined model. We run the
same command as before, but using the =LENSMODEL_SPLINED_STEREOGRAPHIC_= ...
=order=3_Nx=30_Ny=20_fov_x_deg=170= model. This is one long string.

#+begin_example
$ mrcal-calibrate-cameras
    --lensmodel LENSMODEL_SPLINED_STEREOGRAPHIC_ ...
    ... order=3_Nx=30_Ny=20_fov_x_deg=170 ...
...
RMS reprojection error: 0.6 pixels
Noutliers: 0 out of 18600 total points: 0.0% of the data
calobject_warp = [-0.00096895  0.00052931]
#+end_example

** =LENSMODEL_SPLINED_STEREOGRAPHIC= residuals histogram
This is promising. What does the histogram look like?

#+ATTR_LATEX: :width 0.9\linewidth
[[file:../external/figures/calibration/residuals-histogram-splined.pdf]]

** =LENSMODEL_SPLINED_STEREOGRAPHIC=: the worst image
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/calibration/worst-splined.png]]

** =LENSMODEL_SPLINED_STEREOGRAPHIC=: 3rd-worst image
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/calibration/worst-incorner-splined.png]]

** =LENSMODEL_SPLINED_STEREOGRAPHIC=: residual directions
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/calibration/directions-splined.png]]

* Differencing
** Differencing
We computed the calibration two different ways. How different are the two
models?

Let's compute the difference using an obvious algorithm:

Given a pixel $\vec q_0$,

- Unproject $\vec q_0$ to a fixed point $\vec p$ using lens 0
- Project $\vec p$ back to pixel coords $\vec q_1$ using lens 1
- Report the reprojection difference $\vec q_1 - \vec q_0$

#+ATTR_LATEX: :width 0.8\linewidth
[[file:../figures/diff-notransform.pdf]]

** Differencing
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/diff/diff-radius0-heatmap-splined-opencv8.png]]

** Differencing
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/diff/diff-radius0-vectorfield-splined-opencv8.pdf]]

** Differencing
So with a motion of the camera, we can make the errors disappear.

The issue is that each calibration produces noisy estimates of all the
intrinsics and all the coordinate transformations:

[[file:../figures/uncertainty.pdf]]

And the point $\vec p$ we were projecting wasn't truly fixed.

** Differencing
We want to add a step:

- Unproject $\vec q_0$ to a fixed point $\vec p_0$ using lens 0
- *Transform $\vec p_0$ from the coordinate system of one camera to the coordinate
  system of the other camera*
- Project $\vec p_1$ back to pixel coords $\vec q_1$ using lens 1
- Report the reprojection difference $\vec q_1 - \vec q_0$

[[file:../figures/diff-yestransform.pdf]]

** Differencing
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/diff/diff-splined-opencv8.png]]

** Differencing
/Much/ better. As expected, the two models agree relatively well in the center,
and the error grows as we move towards the edges.

This differencing method has numerous applications:

- evaluating the manufacturing variation of different lenses
- quantifying intrinsics drift due to mechanical or thermal stresses
- testing different solution methods
- underlying a cross-validation scheme

** Differencing
A big question:

- How much of the observed difference is random sampling error?

To answer this (an other) questions, mrcal can quantify the projection
uncertainty, so let's do that.

* Uncertainty
** Uncertainty
When we project a point $\vec p$ to a pixel $\vec q$, it would be /really/ nice
to get an uncertainty estimate $\mathrm{Var} \left(\vec q\right)$. The we could

- Propagate this uncertainty downstream to whatever uses the projection
  operation, for example to get the uncertainty of ranges from a triangulation
- Evaluate how trustworthy a given calibration is, and to run studies about how
  to do better
- Quantify overfitting effects
- Quantify the baseline noise level for informed interpretation of model
  differences

Since splined models can have 1000s of parameters (the one we just demoed has
1204), they are prone to overfitting, and it's critically important to gauge
those effects.

** Uncertainty
A grand summary of how we do this:

1. We are assuming a particular distribution of observation input noise
   $\mathrm{Var}\left( \vec q_\mathrm{ref} \right)$
2. We propagate it through the optimization to get the variance of the
   optimization state $\mathrm{Var}(\vec b)$
3. For any /fixed/ point, its projection $\vec q = \mathrm{project}\left(
   \mathrm{transform}\left( \vec p_\mathrm{fixed} \right)\right)$ depends on
   parameters of $\vec b$, whose variance we know. So

\[ \mathrm{Var}\left( \vec q \right) =
\frac{\partial \vec q}{\partial \vec b}
\mathrm{Var}\left( \vec b \right)
\frac{\partial \vec q}{\partial \vec b}^T
\]

** Uncertainty simulation
The mrcal test suite contains a simulation to validate the approach.

- 4 cameras
- Placed side by side + noise in pose
- =LENSMODEL_OPENCV4= lens model
- looking at 50 chessboard poses, with randomized pose

** Uncertainty simulation
The geometry looks like this:

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/simulated-uncertainty-opencv4--simulated-geometry.pdf]]

** Uncertainty simulation
Each camera sees this:

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/simulated-uncertainty-opencv4--simulated-observations.pdf]]

The red *$\ast$* is a point we will examine.

** Uncertainty simulation
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/simulated-uncertainty-opencv4--distribution-onepoint.pdf]]

** Uncertainty simulation
Let's look at the uncertainty everywhere in the imager

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/simulated-uncertainty-opencv4--uncertainty-wholeimage-noobservations.pdf]]

This confirms the expectation: the sweet spot of low uncertainty follows the
region where the chessboards were

** Uncertainty simulation
- The worst-uncertainty-at-*$\ast$* camera claims an uncertainty of 0.8 pixels.
  That's pretty low. But we had no chessboard observations there; is this
  uncertainty realistic? _No_

- =LENSMODEL_OPENCV4= is stiff, so the projection doesn't move much due to
  noise. And we interpreted that as low uncertainty. But that comes from our
  choice of model, and /not/ from the data. So

*Lean models always produce overly-optimistic uncertainty estimates*

Solution: use splined models! They are very flexible, and don't have this issue.

** Uncertainty simulation
Running the same simulation with a splined model, we see the /real/ projection
uncertainty:

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/simulated-uncertainty-splined--uncertainty-wholeimage-noobservations.pdf]]

So /only/ the first camera actually had usable projections.

** Uncertainty simulation
Let's overlay the observations:

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/simulated-uncertainty-splined--uncertainty-wholeimage-observations.pdf]]

** Uncertainty from previous calibrations
Computing the uncertainty map from the earlier =LENSMODEL_OPENCV8= calibration:

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/uncertainty-opencv8.pdf]]
** Uncertainty from previous calibrations
And from the =LENSMODEL_SPLINED_STEREOGRAPHIC_...= calibration:

#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/uncertainty/uncertainty-splined.pdf]]

** Uncertainty conclusion
The splined model promises double the uncertainty that =LENSMODEL_OPENCV8= does.

Conclusions:

- We have a usable uncertainty-quantification method
- It is over-optimistic when applied to lean models

So splined models have a clear benefit even for long lenses, where the lean
models are expected to fit

* Ranging note
** Ranging note
Let's revisit an important detail I glossed-over when talking about differencing
and uncertainties. Both computations begin with $\vec p =
\mathrm{unproject}\left( \vec q \right)$

But an unprojection is ambiguous in range, so *diffs and uncertainties are
defined as a function of range*

#+ATTR_LATEX: :width \linewidth
[[file:../figures/projection-scale-invariance.pdf]]

All the uncertainties reported so far, were at $\infty$

** The uncertainty figure
The uncertainty of our =LENSMODEL_OPENCV8= calibration at the center as a
function of range:

#+ATTR_LATEX: :width 0.8\linewidth
[[file:../external/figures/uncertainty/uncertainty-vs-distance-at-center.pdf]]

* Choreography
** Overview
We have a good way to estimate uncertainties, so let's study what kind of
chessboard dance is best. We

- set up a simulated world with some baseline geometry
- scan some parameter
- calibrate
- look at the uncertainty-vs-range plots as a function of that parameter

This is output of a tool included in the mrcal tree. See the [[http://mrcal.secretsauce.net/tour.html][tour of mrcal]] page
for the commands.

** How many chessboard observations should we get?
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/dance-study/dance-study-scan-Nframes.pdf]]

** How far should the chessboards be placed?
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/dance-study/dance-study-scan-range.pdf]]

** How much should we tilt the chessboards?
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/dance-study/dance-study-scan-tilt_radius.pdf]]

** How many cameras should be included in each calibration?
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/dance-study/dance-study-scan-Ncameras.pdf]]

** How dense should our chessboard be?
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/dance-study/dance-study-scan-object_width_n.pdf]]

** What should the chessboard corner spacing be?
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/dance-study/dance-study-scan-object_spacing.pdf]]

** Do we want tiny boards nearby or giant boards faraway?
#+ATTR_LATEX: :width \linewidth
[[file:../external/figures/dance-study/dance-study-scan-object_spacing-compensated-range.pdf]]

** Conclusions
- More frames are good
- Closeups are /extremely/ important
- Tilted views are good
- A smaller number of bigger calibration problems is good
- More chessboard corners is good, as long as the detector can find them
  reliably
- Tiny chessboards near the camera are better than giant far-off chessboards. As
  long as the camera can keep the chessboards /and/ the working objects in focus

#+ATTR_LATEX: :width 0.7\linewidth
[[file:../figures/observation-usefulness.pdf]]

* Stereo
** Overview
mrcal can do some basic stereo processing. At its core, it's the usual epipolar
geometry process:

1. Ingest two camera models
2. Ingest images captured by these two cameras
3. Transform the images to construct "rectified" images
4. Perform "stereo matching"

Each pair of corresponding rows in the rectified images represents a plane in
space:

#+ATTR_LATEX: :width 0.65\linewidth
[[file:../figures/rectification.pdf]]

** Input images
I used the lens I calibrated at the start to capture a pair of images in
downtown Los Angeles. The left image:

#+ATTR_LATEX: :width 0.75\linewidth
[[file:../external/figures/stereo/0.downsampled.jpg]]

We're on a catwalk between 2nd and 3rd, looking S over Figueroa St.

** Rectification
I then used mrcal's rectification function to produce the rectified image. The
left:

#+ATTR_LATEX: :width 0.7\linewidth
file:../external/figures/stereo/rectified0-splined.downsampled.jpg

** Disparity
And the resulting disparity, as computed by the OpenCV matcher:

#+ATTR_LATEX: :width 0.7\linewidth
file:../external/figures/stereo/disparity-splined.downsampled.png

** JPLV

What if we wanted to use JPLV stereo with splined models?

We can use mrcal to remap to another projection and feed /that/ to jplv. For
instance, let's

- Remap to a pinhole model (with some arbitrary zoom factor)
- Use jplv to compute the rectified image

** JPLV remapped-to-pinhole image
Remapped to a pinhole image with mrcal

#+ATTR_LATEX: :width 0.9\linewidth
file:../external/figures/stereo/0-reprojected-scale0.35.downsampled.jpg

** JPLV rectified image
Rectified with jplv

#+ATTR_LATEX: :width 0.9\linewidth
file:../external/figures/stereo/jplv-stereo-rect-left-scale0.35.downsampled.png

** JPLV stereo
Disparity from OpenCV

#+ATTR_LATEX: :width 0.9\linewidth
file:../external/figures/stereo/disparity-jplv-scale0.35.downsampled.png

** Narrow virtual cameras
Another way to do stereo processing of wide images using tools that aren't built
for it is to

- split the wide-angle stereo pair into a set of narrow-view stereo pairs

This generates a skewed geometry, but mrcal can still use it just fine. Due to a
bug, jplv cannot.

** Narrow virtual cameras
#+ATTR_LATEX: :width 0.9\linewidth
file:../external/figures/stereo/stereo-geometry-narrow.pdf

** Narrow virtual cameras
One of the resulting resampled /pinhole/ images:

#+ATTR_LATEX: :width 0.9\linewidth
file:../external/figures/stereo/narrow-left.downsampled.jpg

** Narrow virtual cameras
Rectified using mrcal

#+ATTR_LATEX: :width 0.9\linewidth
file:../external/figures/stereo/rectified0-narrow.downsampled.jpg

** Narrow virtual cameras
Disparity from OpenCV

#+ATTR_LATEX: :width 0.9\linewidth
file:../external/figures/stereo/disparity-narrow.downsampled.png

* Finally
** Conclusions
- We have a toolkit that can do lots of cool stuff

- There's much to do still, and there's a laundry list on the documentation page.

** Thanks!
Questions?
