#+TITLE: Towards mrcal 3.0
#+AUTHOR: Dima Kogan

#+OPTIONS: toc:nil H:2

#+LATEX_CLASS_OPTIONS: [presentation]

# Org adds this: \usepackage[T1]{fontenc} On my box this has the effect of
# asking for fonts that I don't have, which results in very ugly rendering with
# mupdf, where it uses bitmapped fonts, and scales them in ugly ways. Adding the
# below makes it pick the nice fonts
#+LaTeX_HEADER: \usepackage{lmodern}

# I don't want navigation buttons in the document. AND I do this only if using
# beamer; otherwise the latex fragments aren't generated right
#+LaTeX_HEADER: \IfClassLoadedTF{beamer}{ \setbeamertemplate{navigation symbols}{} }{}

# I want clickable links to be blue and underlined, as is custom
#+LaTeX_HEADER: \usepackage{letltxmacro}
#+LaTeX_HEADER: \LetLtxMacro{\hreforiginal}{\href}
#+LaTeX_HEADER: \renewcommand{\href}[2]{\hreforiginal{#1}{\color{blue}{\underline{#2}}}}
#+LaTeX_HEADER: \renewcommand{\url}[1]{\href{#1}{\tt{#1}}}

# I want a visible gap between paragraphs
#+LaTeX_HEADER: \setlength{\parskip}{\bigskipamount}

#+LATEX_HEADER: \usepackage[all,cmtip,color,matrix,arrow]{xy}
#+LATEX_HEADER: \usepackage{color}

* Overview
** What is mrcal?
mrcal is a toolkit for geometric vision. Applicable to anything that makes or
uses camera models. For instance:

- Calibration
- Triangulation
- Photogrammetry
- Stucture from motion
- Perception system core
- Design and analysis of any system that uses cameras for perception

** Why does mrcal exist?

Tooling that was precise-enough for my visual ranging work did not exist.

By necessity, mrcal is a complete re-design and re-implementation of camera
modeling tooling from the ground-up:

- Richer camera model available to precisely model lens behavior
- Uncertainty propagation and cross-validation techniques available to
  validate a calibration
- No implicit pinhole assumption anywhere: *no homogeneous coordinates*
- Fisheye-friendly stereo rectification

Also, /nice/ C and Python library, with lots of commandline tools, good
documentation, sane build and distribution, etc.

** Computing details                                               :noexport:

- mrcal is a library (C and Python)
- Lots of routines. Easy and common to use mix some library usage with other
  tools, for instance using a separate optimization routine for aligning
  non-camera sensors
- Many commandline tools available, so no coding required for common tasks
- /Thoroughly/ documented
- Open-source, available in stock Debian /today/
- Builds on a wide variety of architectures and platforms. If you need something
  not supported today, talk to me.

Contributions welcome!

** Where is it?                                                    :noexport:

Detailed documentation is available at https://mrcal.secretsauce.net/

- This talk is a repackaging of the documentation on that page

- See the docs for more detail, and for links to all the data and commands that
  produced the presented results

** Where are we now: calibration
[[https://mrcal.secretsauce.net/news-2.5.html][mrcal 2.5]] was just released. These tools are no longer experimental. They have
been used with great success in /lots/ of contexts. Some highlights when used
for /calibration/:

- mrcal was used to calibrate extremely wide lenses
- ... and extremely narrow lenses
- ... and joint systems with both at the same time
- ... with /many/ cameras; 10 cameras is the record thus far
- both visible and thermal cameras
- and [[https://github.com/dkogan/camera-lidar-calibration/][joint camera-LIDAR systems]]
- and [[https://www.github.com/dkogan/mrcal/blob/master/analyses/calibrate-camera-imu.py][joint camera-IMU systems]]
- mrcal is part of [[https://photonvision.org/][PhotonVision]], the toolkit used by teams in the [[https://www.firstinspires.org/robotics/frc][FIRST Robotics
  Competition]]

** Where are we now: pipelines
In many deployments the C API was a core part of the perception pipeline:

- Used in a traditional dense stereo workflow, with smarter rectification
- The new mrcal triangulated features were used for SFM to compute the world
  geometry on-line
- Stereo from moving cameras results in weird geometries: for instance moving
  along the optical axis. It all works just fine.

** Where are we now: analyses
- mrcal has been used with weird experimental setups employing custom
  calibration objects and single-view solves
- Used in countless ways for perception system design, analysis of correctness,
  stability, drift, and so on

** Where I want to go

*I want to be able to get /reliable/ and /precise/ calibrations /without/ a
chessboard.*

The obvious method: structure-from-motion

- moving camera observes a stationary scene
- no knowledge of observed world
- feature correspondences only; no ranges

This is photogrammetry

Plenty of existing methods, but all lack the uncertainty reporting and precision
you get from mrcal and chessboards.

** Why a chessboard-less calibration is hard
Because chessboard-less observations of matching features are information-poor.

Let's compare

- 2 cameras observing a 10x10 chessboard
- 2 cameras observing a set of 100 fixed points in space

In both cases we have 400 observations (100 points in space, 2 cameras, xy
measurements for each point)

- With a chessboard we have 6 extra degrees of freedom (the pose of the board)
- Without a chessboard we have 300 extra degrees-of-freedom (xyz coordinates of
  each point)

** Why a chessboard-less calibration is hard
This is still overdetermined (400 > 300), but much worse.

We thus need a /lot/ of data to achieve results of similar quality. How many
points is unknown: hundreds? thousands? millions? The mrcal uncertainty
reporting is /essential/ to give us this feedback.

** chessboard-less calibration high-level plan
- Implement uncertainty reporting
- Run simulations to see how many points in which geometry are needed for a
  confident calibration
- Validate in the real world

* Tour of mrcal
** Demo calibration
Let's start by looking at the "tour of mrcal":
https://mrcal.secretsauce.net/tour.html

We follow a real-world data flow, starting with chessboard observations. Images
captured using

- Sony Alpha 7 III full-frame SLR. 6000x3376 imager
- /Very/ wide lens: Samyang 12mm F2.8 fisheye. 180deg field of view
  corner-corner
- Just one camera (in this demo; mrcal supports multiple cameras)
- Outdoor images captured in downtown Los Angeles

** Sample image of the scene                                       :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo/0.downsampled.jpg]]

** Gathering and detecting chessboard corners
We capture images, and use [[https://github.com/dkogan/mrgingham/][mrgingham]] to detect chessboard corners. For an
arbitrary image:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/mrgingham-results.png]]

* Tour of mrcal: =LENSMODEL_OPENCV8=
** Let's run a calibration!
We find a set of geometry and lens parameters to most closely match the
observations.

We start with the 8-parameter OpenCV model: =LENSMODEL_OPENCV8=

#+begin_example
$ mrcal-calibrate-cameras --lensmodel LENSMODEL_OPENCV8 ...
...

RMS reprojection error: 0.4 pixels
Worst residual (by measurement): 1.8 pixels
Noutliers: 564 out of 16464 total points: 3.4% of the data
calobject_warp = [-0.00012726 -0.00014325]
#+end_example

** Why is the reprojection error > 0?                              :noexport:
We have two sources of error:

- *Sampling error*: chessboard observations are noisy. We can study this, but we
  cannot reduce it

- *Model error*: our model of the lens behavior, chessboard shape and everything
  else isn't perfect. We can and must suppress this as much as possible

We want the model error to be negligible. If it is and if the sampling error is
normal and i.i.d., then we get a bias-free maximum-likelihood calibration result

_Patterns_ in the residuals indicate the presence of model errors

** Does the solved geometry look right?                            :noexport:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/calibration-chessboards-geometry-crop.pdf]]

Yes. That's how I danced.

** =LENSMODEL_OPENCV8= residuals histogram                         :noexport:
What does the error distribution look like?

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/residuals-histogram-opencv8-crop.pdf]]

** =LENSMODEL_OPENCV8= worst-observation residuals                 :noexport:
The worst-fitting observations are a great way to see common issues such as:

- out-of focus images
- images with motion blur
- rolling shutter effects
- synchronization errors
- chessboard detector failures
- insufficiently-rich models (of the lens or of the chessboard shape or anything
  else)

We look for _patterns_ in the residuals: patterns indicate the presence of model
errors

** =LENSMODEL_OPENCV8=: the worst image                            :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-opencv8.png]]

** =LENSMODEL_OPENCV8=: the worst image in a corner                :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-incorner-opencv8.png]]

** =LENSMODEL_OPENCV8=: residual directions                        :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-opencv8-crop.pdf]]

** =LENSMODEL_OPENCV8=: conclusions                                :noexport:
We see clear patterns in the residuals, so

- =LENSMODEL_OPENCV8= does not fit our data

Let's fix it.

* Tour of mrcal: =LENSMODEL_SPLINED_STEREOGRAPHIC=
** =LENSMODEL_SPLINED_STEREOGRAPHIC= definition                    :noexport:
- We need a more flexible lens model to represent our lens.
- mrcal currently supports a /splined/ model that is configurable to be as rich
  as we like

We compute a normalized /stereographic/ projection:

\[ \vec u = \mathrm{project}_\mathrm{stereographic}\left(\vec p\right) \]

This maps a 3D direction vector to a 2D point $\vec u$. This works behind the
camera, so wide-angle lenses are supported well.

** =LENSMODEL_SPLINED_STEREOGRAPHIC= definition                    :noexport:
Then use $\vec u$ to look-up an adjustment factor $\Delta \vec u$ using two
splined surfaces: one for each of the two elements of

\[ \Delta \vec u \equiv
\left[ \begin{aligned}
\Delta u_x \left( \vec u \right) \\
\Delta u_y \left( \vec u \right)
\end{aligned} \right] \]

We can then define the rest of the projection function:

\[\vec q =
 \left[ \begin{aligned}
 f_x \left( u_x + \Delta u_x \right) + c_x \\
 f_y \left( u_y + \Delta u_y \right) + c_y
\end{aligned} \right] \]

** Doing it again with =LENSMODEL_SPLINED_STEREOGRAPHIC=           :noexport:
Let's re-process the same calibration data using the splined model. We run the
same command as before, but using the =LENSMODEL_SPLINED_STEREOGRAPHIC_= ...
=order=3_Nx=30_Ny=18_fov_x_deg=150= model. This is one long string.

This model has 1084 parameters.

#+begin_example
$ mrcal-calibrate-cameras
    --lensmodel LENSMODEL_SPLINED_STEREOGRAPHIC_ ...
    ... order=3_Nx=30_Ny=18_fov_x_deg=150 ...
...
RMS reprojection error: 0.2 pixels
Worst residual (by measurement): 1.3 pixels
Noutliers: 28 out of 16464 total points: 0.2% of the data
calobject_warp = [-1.26851438e-04 -8.03269701e-05]
#+end_example

** =LENSMODEL_OPENCV8= residuals histogram                         :noexport:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/residuals-histogram-opencv8-crop.pdf]]

** =LENSMODEL_SPLINED_...= residuals histogram                     :noexport:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/residuals-histogram-splined-crop.pdf]]

** =LENSMODEL_OPENCV8= worst-observation residuals                 :noexport:
The worst-fitting observations are a great way to see common issues such as:

- out-of focus images
- images with motion blur
- rolling shutter effects
- synchronization errors
- chessboard detector failures
- insufficiently-rich models (of the lens or of the chessboard shape or anything
  else)

We look for _patterns_ in the residuals: patterns indicate the presence of model
errors

** =LENSMODEL_OPENCV8=: the worst image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-opencv8.png]]

** =LENSMODEL_SPLINED_...=: the worst image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-splined.png]]

** =LENSMODEL_OPENCV8=: the worst image in a corner                :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-incorner-opencv8.png]]

** =LENSMODEL_SPLINED_...=: the worst image in a corner            :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-incorner-splined.png]]

** =LENSMODEL_OPENCV8=: residual directions                        :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-opencv8-crop.pdf]]
** =LENSMODEL_SPLINED_...=: residual directions                    :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-splined-crop.pdf]]

** Conclusion                                                      :noexport:
We have good evidence that =LENSMODEL_SPLINED_STEREOGRAPHIC= fits this lens much
better than =LENSMODEL_OPENCV8=

* Differencing
** Differencing
We computed the calibration two different ways. How different are the two
models?

A differencing method could be used for

- evaluating the manufacturing variation of different lenses
- quantifying intrinsics drift due to mechanical or thermal stresses
- testing different solution methods
- underlying a cross-validation scheme

** Differencing
The obvious algorithm:

Given a pixel $\vec q_0$,

- Unproject $\vec q_0$ to a fixed point $\vec p$ using lens 0
- Project $\vec p$ back to pixel coords $\vec q_1$ using lens 1
- Report the reprojection difference $\vec q_1 - \vec q_0$

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/diff-notransform.pdf]]

** Differencing
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-radius0-heatmap-splined-opencv8-crop.pdf]]

** Differencing
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-radius0-vectorfield-splined-opencv8-crop.pdf]]

** Differencing
The issue is that each calibration produces noisy estimates of all the
intrinsics and all the coordinate transformations:

[[file:../out/figures/uncertainty.pdf]]

And the point $\vec p$ we were projecting wasn't truly fixed.

** Differencing
We need to add a step:

- Unproject $\vec q_0$ to a fixed point $\vec p_0$ using lens 0
- *Transform $\vec p_0$ from the coordinate system of one camera to the coordinate
  system of the other camera*
- Project $\vec p_1$ back to pixel coords $\vec q_1$ using lens 1
- Report the reprojection difference $\vec q_1 - \vec q_0$

[[file:../out/figures/diff-yestransform.pdf]]

** Differencing
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-splined-opencv8-crop.pdf]]

** Differencing                                                    :noexport:
/Much/ better. As expected, the two models agree relatively well in the center,
and the error grows as we move towards the edges.

This differencing method has numerous applications:

- evaluating the manufacturing variation of different lenses
- quantifying intrinsics drift due to mechanical or thermal stresses
- testing different solution methods
- underlying a cross-validation scheme

* Cross-validation
** Cross-validation
We can use this method to quantify the quality of a calibration:

- gather two (or more) independent data sets
- diff the results

The samples give us a sense of the distribution of these random variables. A
tight distribution, observed as /small/ cross-validation diffs, indicates a
confident calibration.

** Cross-validation
Each calibration result is a sample of the camera parameters random variables

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/perfect-data-splined-diff-samples.pdf]]

** Cross-validation
In practice we often see /high/ cross-validation diffs, and we need to
investigate.

The sources of error:

- *Sampling error*: chessboard observations are noisy. We can study this, but we
  cannot reduce it

- *Model error*: our model of the lens behavior, chessboard shape and everything
  else isn't perfect. We can and must suppress this as much as possible

We want the model error to be negligible: the cross-validation and other mrcal
tools aid with this

The sampling error can be quantified and studied with a single solve, by
directly propagating the input noise

* Solve formulation
** Solve formulation
To talk about how this is done, an overview of the main optimization

mrcal uses traditional non-linear least squares

A hypothetical calibration is fully described by a /state/ vector $\vec b$, which
contains

- the poses of /all/ the cameras
- the poses of /all/ the chessboards
- the intrinsic parameters of /all/ the lenses
- the parameters describing the chessboard shape

Given $\vec b$, the solver can predict the observed pixel coordinates $\vec q$, and compare
with the observations $\vec q_{\mathrm{ref}}$. The solver then finds the $\vec
b$ to minimize the discrepancy, by minimizing some function $E \equiv \left
\Vert \vec x \left(\vec b\right)\right \Vert ^2$

** Solve formulation
We have 3 coordinate systems:

- The _frame_ or _board_ system: local to the chessboard
- The _camera_ coordinate system: local to a camera
- The _reference_ coordinate system: some fixed coordinate system that all frame
  and camera poses are linked to

** Solve formulation
With a "normal" chessboard-based calibration we have this data flow:

\[
\xymatrix @C=5em {
\vec q &
\vec p_\mathrm{cam}     \ar[l]^{\vec b_\mathrm{intrinsics} } &
\vec p_\mathrm{ref}     \ar[l]^{T_\mathrm{cr}} &
\vec p_\mathrm{board}   \ar[l]^{T_\mathrm{rf}}
}
\]

If we're solving with discrete points instead, the point coordinates $\vec
p_\mathrm{point}$ are defined in the reference coordinate system, and the flow
is slightly different:

\[
\xymatrix @C=5em {
\vec q &
\vec p_\mathrm{cam}     \ar[l]^{\vec b_\mathrm{intrinsics} } &
\vec p_\mathrm{point}   \ar[l]^{T_\mathrm{cr}}
}
\]

The transforms $T$ and intrinsics are tweaked by the optimizer as it tries to
match $\vec q$ to the observations $\vec q_{\mathrm{ref}}$.

* Uncertainty
** Uncertainty                                                     :noexport:
- All calibrations are based on observations of the calibration object
  (chessboard corners)
- These observations /always/ contain some noise (sampling error)
- A calibration result is trustworthy /only/ if it is insensitive to this noise

We quantify this sensitivity by computing a _projection uncertainty_

** Uncertainty
We can propagate the calibration input noise analytically to characterize the
calibration distribution /without sampling/.

With some reasonable assumptions we can compute the covariance of the
optimization state $\mathrm{Var}\left(\vec b\right)$

This isn't useful in itself

** Uncertainty
As with the differencing, we must consider extrinsics shifts /and/ intrinsics,
even though the intrinsics is what we care about here.

$\mathrm{Var}\left(\vec b\right)$ describes how $\vec b$ is likely to move. We derive $\vec q^+\left(\vec
b\right)$ to see how these shifts affect projection of a /fixed/ point in space.
Then

\[ \mathrm{Var}\left( \vec q^+ \right) = \frac{\partial \vec q^+}{\partial \vec b} \mathrm{Var}\left( \vec b \right) \frac{\partial \vec q^+}{\partial \vec b}^T \]

** mean-pcam uncertainty
The chessboard data flow is:

\[
\xymatrix @C=5em {
\vec q &
\vec p_\mathrm{cam}     \ar[l]^{\vec b_\mathrm{intrinsics} } &
\vec p_\mathrm{ref}     \ar[l]^{T_\mathrm{cr}} &
\vec p_\mathrm{board}   \ar[l]^{T_\mathrm{rf}}
}
\]

We perturb $\vec q^+_\mathrm{ref} = \vec
q_\mathrm{ref} + \Delta \vec q_\mathrm{ref}$ and reoptimize:

\[
\xymatrix @C=5em {
\vec q^+ &
\vec p^+_\mathrm{cam}    \ar[l]^{\vec b^+_\mathrm{intrinsics} } &
\vec p^+_\mathrm{ref}    \ar[l]^{T_\mathrm{c^+r^+}} &
\vec p^+_\mathrm{board}  \ar[l]^{T_\mathrm{r^+f^+}}
}
\]

All perturbed quantities are marked with a $+$ superscript.

We need to connect these two data flows.

** mean-pcam uncertainty
We need to connect the baseline and perturbed solves: I fix the point in board
coordinates: $\vec p^+_\mathrm{board}=\vec p_\mathrm{board}$. We then get this
flow:

\[
{\tiny
\xymatrix @=1.5em {
         & &                                                                  & \vec p^+_{\mathrm{camera}_0} \ar[dl]_{\mathrm{mean}} & \vec p^+_{\mathrm{reference}_0} \ar[l]_-{T^+_{\mathrm{c_0 r}}} & \vec p_{\mathrm{board}_0} \ar[l]_-{T^+_{\mathrm{rf_0}}} & \vec p_{\mathrm{reference}_0} \ar[l]_-{T_\mathrm{f_0 r}} \\
\vec q^+ & & \vec p^+_\mathrm{camera} \ar[ll]_-{\vec b_\mathrm{intrinsics}^+} & \vec p^+_{\mathrm{camera}_1} \ar[l] _{\mathrm{mean}} & \vec p^+_{\mathrm{reference}_1} \ar[l]_-{T^+_{\mathrm{c_1 r}}} & \vec p_{\mathrm{board}_1} \ar[l]_-{T^+_{\mathrm{rf_1}}} & \vec p_{\mathrm{reference}_1} \ar[l]_-{T_\mathrm{f_1 r}} & \vec p_\mathrm{camera} \ar[l]_-{T_\mathrm{r_1 c}} \ar[lu]_-{T_\mathrm{r_0 c}} \ar[ld]^-{T_\mathrm{r_2 c}} & & \vec q \ar[ll]_-{\vec b_\mathrm{intrinsics}} \\
         & &                                                                  & \vec p^+_{\mathrm{camera}_2} \ar[ul]^{\mathrm{mean}} & \vec p^+_{\mathrm{reference}_2} \ar[l]_-{T^+_{\mathrm{c_2 r}}} & \vec p_{\mathrm{board}_2} \ar[l]_-{T^+_{\mathrm{rf_2}}} & \vec p_{\mathrm{reference}_2} \ar[l]_-{T_\mathrm{f_2 r}} 
}
}
\]

This gives us $\vec q^+\left(\vec b\right)$, and we can thus compute

\[ \mathrm{Var}\left( \vec q^+ \right) = \frac{\partial \vec q^+}{\partial \vec b} \mathrm{Var}\left( \vec b \right) \frac{\partial \vec q^+}{\partial \vec b}^T \]

This is the _mean-pcam_ uncertainty method, originally implemented in mrcal

** Uncertainty                                                     :noexport:
When we project a point $\vec p$ to a pixel $\vec q$, it would be /really/ nice
to get an uncertainty estimate $\mathrm{Var} \left(\vec q\right)$. Then we could

- Propagate this uncertainty downstream to whatever uses the projection
  operation, for example to get the uncertainty of ranges from a triangulation
- Evaluate how trustworthy a given calibration is, and to run studies about how
  to do better
- Quantify overfitting effects
- Quantify the baseline noise level for informed interpretation of model
  differences

Since splined models can have 1000s of parameters (the one we just demoed has
1084), they are prone to overfitting, and it's critically important to gauge
those effects.

** Uncertainty                                                     :noexport:
A grand summary of how we do this:

1. We are assuming a particular distribution of observation input noise
   $\mathrm{Var}\left( \vec q_\mathrm{ref} \right)$
2. We propagate it through the optimization to get the variance of the
   optimization state $\mathrm{Var}(\vec b)$
3. For any /fixed/ point, its projection $\vec q = \mathrm{project}\left(
   \mathrm{transform}\left( \vec p_\mathrm{fixed} \right)\right)$ depends on
   parameters of $\vec b$, whose variance we know. So

\[ \mathrm{Var}\left( \vec q \right) =
\frac{\partial \vec q}{\partial \vec b}
\mathrm{Var}\left( \vec b \right)
\frac{\partial \vec q}{\partial \vec b}^T
\]

** Uncertainty from the DTLA data
And from the =LENSMODEL_SPLINED_STEREOGRAPHIC_...= calibration:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-splined-crop.pdf]]

** Uncertainty simulation                                          :noexport:
The mrcal test suite contains a simulation to validate the approach.

- 4 cameras
- Placed side by side + noise in pose
- =LENSMODEL_OPENCV4= lens model
- looking at 50 chessboard poses, with randomized pose

** Uncertainty simulation                                          :noexport:
The geometry looks like this:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--simulated-geometry-crop.pdf]]

** Uncertainty simulation                                          :noexport:
Each camera sees this:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--simulated-observations-crop.pdf]]

The red *$\ast$* is a point we will examine.

** Uncertainty simulation                                          :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--distribution-onepoint-crop.pdf]]

** Uncertainty simulation                                          :noexport:
Let's look at the uncertainty everywhere in the imager

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--uncertainty-wholeimage-noobservations-crop.pdf]]

This confirms the expectation: the sweet spot of low uncertainty follows the
region where the chessboards were

** Uncertainty simulation                                          :noexport:
- The worst uncertainty-at-*$\ast$* camera claims an uncertainty of 0.8 pixels.
  That's pretty low. But we had no chessboard observations there; is this
  uncertainty realistic? _No_

- =LENSMODEL_OPENCV4= is stiff, so the projection doesn't move much due to
  noise. And we interpreted that as low uncertainty. But that comes from our
  choice of model, and /not/ from the data. So

*Lean models always produce overly-optimistic uncertainty estimates*

Solution: use splined models! They are very flexible, and don't have this issue.

** Uncertainty simulation                                          :noexport:
Running the same simulation with a splined model, we see the /real/ projection
uncertainty:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-splined--uncertainty-wholeimage-noobservations-crop.pdf]]

So /only/ the first camera actually had usable projections.

** Uncertainty simulation
Let's overlay the observations:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-splined--uncertainty-wholeimage-observations-crop.pdf]]

** Uncertainty from the DTLA data                                  :noexport:
Computing the uncertainty map from the earlier =LENSMODEL_OPENCV8= calibration:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-opencv8-crop.pdf]]
** Uncertainty conclusion                                          :noexport:
The splined model promises double the uncertainty that =LENSMODEL_OPENCV8= does.

Conclusions:

- We have a usable uncertainty-quantification method
- It is over-optimistic when applied to lean models

So splined models have a clear benefit even for long lenses, where the lean
models are expected to fit

* Ranging note                                                     :noexport:
** Ranging note
Let's revisit an important detail I glossed-over when talking about differencing
and uncertainties. Both computations begin with $\vec p =
\mathrm{unproject}\left( \vec q \right)$

But an unprojection is ambiguous in range, so *diffs and uncertainties are
defined as a function of range*

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/projection-scale-invariance.pdf]]

All the uncertainties reported so far were at $\infty$

** The uncertainty figure
The uncertainty of our =LENSMODEL_OPENCV8= calibration at the center as a
function of range:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-vs-distance-at-center-crop.pdf]]

* Let's apply these techniques
** Let's apply these techniques                                    :noexport:
We described several analysis techniques:

- Visualizing the solve residuals
- Computing projection differences between two models
- Evaluating projection uncertainty

Let's use these to answer practical questions

** Optimal choreography overview
For many of the following analyses we study the effects of sampling error. We

- Set up a simulated world with some baseline geometry
- Scan some parameter
- Calibrate
- Look at the uncertainty-vs-range plots as a function of that parameter

* What kind of calibration object do we want?
** How dense should our chessboard be?                             :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-object_width_n-crop.pdf]]

** What should the chessboard corner spacing be?                   :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-object_spacing-crop.pdf]]

** Do we want tiny boards nearby or giant boards faraway?          :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-object_spacing-compensated-range-crop.pdf]]

** How far should the chessboards be placed?
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-range-crop.pdf]]

** How many chessboard observations should we get?
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-Nframes-crop.pdf]]

** What kind of calibration object do we want? Guidelines          :noexport:
- More data is good
  - More chessboard corners
  - More chessboard observations
- The chessboard should fill the imager
  - Close-ups
  - Big chessboards

Questions:

- So what kind of calibration object do we want? Are /chessboards/ the right
  choice?

- Should we place the chessboard immediately in front of the lens? Should we use
  a /giant/ chessboard?

** Chessboards? Circles? AprilTags? Charuco?                       :noexport:
mrcal doesn't care!

- Grids of circles (and possibly AprilTags) don't directly observe the point, so
  they could be biased. mrcal has a visual validation tool:
  =mrcal-reproject-to-chessboard= that produces a [[file:~/projects/mrcal/doc/out/external/figures/reprojected-to-chessboard/reprojected-to-chessboard.mp4][validation sequence]]
- Anything with AprilTags needs a high-resolution-enough image to resolve the
  AprilTag. This resolution could instead be used to cram extra chessboard
  squares into the image

I use chessboards with the mrgingham detector

** The downsides of extreme closeups                               :noexport:
*** Corners out of focus
- If the blur is unbiased and gaussian: this will increase the noise, but we can
  compensate by gathering more data
- It looks like the blur mostly /is/ unbiased and gaussian, but don't push it

*** Noncentral effects become significant
Core assumption of almost all camera modeling and processing:

- All rays of light intersect at a single point

*This is not a valid assumption near the lens*

** Noncentrality                                                   :noexport:
The size of the glass in the lens becomes non-negligible as we observe nearby
objects

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/noncentral.pdf]]

** Noncentrality                                                   :noexport:
- Most triangulation and stereo routines assume a central projection. This is
  true for non-closeups

- If necessary, noncentral behavior /can/ be modeled:
  - mrcal has partial support, which was critically important for some projects
  - CAHVORE is noncentral with most people throwing away the noncentrality when
    they use it

- We should try to calibrate and use the cameras beyond where noncentral effects
  are significant. mrcal cross-validation will tell you if you're too close.

** Noncentrality                                                   :noexport:
There's limited awareness that this is what's being done. I've seen people carry
around code like this to centralize CAHVORE unprojections:

#+begin_src c
  cmod_2d_to_3d(p_in, &(model.core), ow, nw, NULL, NULL);
  udmxs_(nw, nw, 1000000, 3);  // nw = 1000000*nw
  udadd_(nw, ow, nw, 3);       // nw = ow + nw
#+end_src

Don't do it: just set $\vec E = 0$

** The downsides of huge chessboards                               :noexport:
- Difficult to manufacture
- Expensive
- Unstable

mrcal has a simple /static/ deformation model: a parabolic deformation in x and
in y. *Usually this isn't enough to accurately represent foam boards*

** The downsides of huge chessboards                               :noexport:
Because *intrinsics are sensitive to chessboard shape errors*. Simulated
intrinsics calibration error due to a board shape error of 1mm in the center in
one direction, and 0.5mm in the center in the other direction. No other noise
present.

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unstable-chessboard-shape-diff-crop.pdf]]

** The downsides of huge chessboards                               :noexport:
I usually use an Aluminum-honeycomb-backed 1m x 1m square board. This works
well.

** What kind of calibration object do we want? Conclusions         :noexport:
- Chessboard as large as possible
- Placed as close to the camera as possible
- With as dense a chessboard grid as possible

Using the mrcal tools to verify that we didn't go too far

* How should we capture images?                                    :noexport:
** How should we capture images?
- Same physical settings as when using the system: aperture size, zoom, focus
- Features should be in-focus: trade-off with uncertainty requirements from
  before. Some out-of-focus blur is ok
- Self-consistent timing
  - No motion blur. Use a tripod or capture images with lots of light: doing it
    outside is great
  - No rolling shutter effects. If you have such a camera, use a tripod
  - If calibrating multiple cameras, hardware sync is a requirement

mrcal will clearly identify these issues, if they're present in the data

* How should we dance?
** How much should we tilt the chessboards?                        :noexport:
We already saw that we want

- Closeups
- Lots of images

Should we tilt the chessboards?

#+ATTR_LATEX: :width 0.9\textwidth :height 0.6\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-tilt_radius-crop.pdf]]

** How should we dance? Conclusions

#+ATTR_LATEX: :width 0.9\textwidth :height 0.4\textheight :options keepaspectratio
[[file:../out/figures/observation-usefulness.pdf]]

Use mrcal tools to validate

* Which model should we use for the lenses?                        :noexport:
** Which model should we use for the lenses?
Today mrcal supports

- OpenCV models with 4,5,8,12 "distortion" parameters
- CAHVOR, CAHVORE
- =LENSMODEL_SPLINED_STEREOGRAPHIC=: the rich, splined model

Unless you really need compatibility with a legacy system or you have low
accuracy requirements, *=LENSMODEL_SPLINED_STEREOGRAPHIC= is strongly
recommended* for all applications

No plans to support any other legacy models (Kannala-Brandt, FOV, etc), since
they're all strictly worse than what is here already. If you /really/ need
something and are willing to contribute it, talk to me.

* Calibrating                                                      :noexport:
** Computing the calibration
We are ready to compute the calibration!

- Run the =mrcal-calibrate-cameras= tool
- For cross-validation you want to split your data into 2 (or more) independent
  sets, and process those independently
- If using a lesser lens model, calibrate both with the model you selected, and
  with =LENSMODEL_SPLINED_STEREOGRAPHIC=. Use that to see how much error you get
  from your model choice

* Results interpretation                                           :noexport:
** Interpreting the calibration results

Once we have a calibration, we should see how well we did:

- We examine the projection uncertainty to make sure we have enough good data in
  the right places
- We examine the cross-validation diffs to confirm that the model fits
- If these diffs are too high, we examine the residuals to find the cause of our
  model errors

* Uncertainty                                                      :noexport:
** Projection uncertainty
- Projection uncertainty gauges the effect of sampling error
- This is directly affected by the quality of the data we gathered. Problems
  with the chessboard dance will show up here
- Lean lens models (anothing other than =LENSMODEL_SPLINED_STEREOGRAPHIC=) will
  produce an overly-optimistic uncertainty report
- *A low projection uncertainty is a necessary, but not sufficient condition for
  a good calibration*: uncertainty reporting samples the input pixel noise, but
  not the model noise

If the uncertainty is unacceptable, stop there, and fix that first.

** DTLA projection uncertainty: OPENCV8

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-opencv8-crop.pdf]]

** DTLA projection uncertainty: splined model

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-splined-crop.pdf]]

* Cross-validation                                                 :noexport:
** Cross-validation diffs
Now we look for model errors

- We split our input dataset, and process the subsets independently: this
  samples the model error
- We use the differencing method to compare the projection behaviors
- Unlike the uncertainty reporting, interpreting these requires some thought

** Cross-validation diffs: detecting model errors
I want to see

\[E_{\mathrm{uncertainty}_0} + E_{\mathrm{uncertainty}_1} \approx E_{\mathrm{crossvalidation}}\]

Let's look at the downtown LA data. We want to see a cross-validation diff of ~
0.2 pixels.

** DTLA cross-validation diffs: OPENCV8
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-opencv8-crop.pdf]]

** DTLA cross-validation diffs: splined model
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-splined-crop.pdf]]

** DTLA cross-validation diffs
- Clearly the =LENSMODEL_OPENCV8= result has issues
- But the =LENSMODEL_SPLINED_STEREOGRAPHIC= result has too-high errors too

Because I captured images from too close to the lens, and we're seeing
non-negligible noncentral behavior. Asking mrcal to model that behavior
produces:

** DTLA cross-validation diffs: splined model, noncentral
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-splined-noncentral-crop.pdf]]

** DTLA cross-validation diffs
- If this calibration was important, I would get a different dataset from
  further out

** DTLA cross-validation diffs
- Here the cross-validation diffs alerted us to the presense of a problem. They
  are /very/ good at that

- Finding the cause of the problem requires some intuition and experimentation

* Residuals                                                        :noexport:
** Residuals
- One technique is available to help diagnose problems: examining the solve
  residuals

** Residuals

We usually have a /lot/ of images and a /lot/ of residuals. I look at the few
worst-fitting images. Usually I only look at the residuals if

- I'm calibrating an unfamiliar system
- I don't trust something about the way the data was collected
- Something unknown is causing issues (we're seeing too-high cross-validation
  diffs), and we need to debug

Model errors are indicated with noise that is correlated or heteroscedastic, so
*we look for patterns in the residuals*.

Let's examine the residuals we get from common problems

** Residuals: poorly-fitting lens model
We saw this in the downtown Los Angeles data

- We looked at both the =LENSMODEL_OPENCV8= and
  =LENSMODEL_SPLINED_STEREOGRAPHIC= residuals

- The latter was much better, but still showed patterns

Earlier residual plots follow below

** Residuals: =LENSMODEL_OPENCV8=: the worst image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-opencv8.png]]

** Residuals: =LENSMODEL_OPENCV8=: residual directions
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-opencv8-crop.pdf]]

** Residuals: rolling shutter

Some cameras save money on memory by sending pixel data as it is captured. The
result: *rolling shutter cameras capture different parts of the image at
different times*.

This produces funky residuals

** Residuals: rolling shutter
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/rolling-shutter-0.pdf]]

** Residuals: rolling shutter
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/rolling-shutter-2.pdf]]

** Residuals: rolling shutter
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/rolling-shutter-3.pdf]]

** Residuals: syncronization errors
- In a multi-camera calibration we assume that sets of images were captured at
  the same instant in time
- This requires a shared physical wire that each camera uses to initiate image
  capture

If this doesn't work right we get the tell-tale residuals, and we can examine
the solution to find the smoking-gun images that prove the breakage

** Residuals: syncronization errors
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/sync-errors-0.pdf]]

** Residuals: syncronization errors
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/sync-errors-1.pdf]]

** Residuals: chessboard shape errors
- Errors in chessboard shape are difficult to disentangle from errors in
  intrinsics
- We can have static and dynamic shape errors:
  1. The chessboard is non-flat, but in a way not modeled by the solver
  2. The chessboard shape changes over the course of the chessboard dance

** Residuals: errors due to unmodeled chessboard shape             :noexport:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unmodeled-chessboard-shape.pdf]]

** Residuals: errors due to unstable chessboard shape

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unstable-chessboard-shape.pdf]]

** Residuals: chessboard shape errors. Conclusions
- These are hard to conclusively pick out from residual plots
- It's helpful to look at more than just 1 or 2 worst-case images
- The most tilted chessboard observations usually show very consistent residual
  vectors along the far edge of the chessboard

** Perfectly-corrupted solves
mrcal can report the errors from a solve containing /only one kind/ of
hypothetical error. This measures the effect of problems we think may exist

A board shape error of 1mm in the center in one direction, and 0.5mm in the
center in the other direction does this:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unstable-chessboard-shape-diff-crop.pdf]]

* Recipes                                                          :noexport:
** Camera stability
Let's switch gears, and look at some applications

Let's use the differencing method to gauge stability of intrinsics:

- If we stress a camera system (mechanically, thermally, etc), does its behavior
  change?

** Lens stability
As a baseline, once again here's the cross-validation diff from the downtown Los
Angeles dataset. This is the /difference between two subsequent solves without
touching anything/

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-splined-noncentral-crop.pdf]]

** Lens stability
- Then I moved the camera and tripod over by 2m or so, and gathered more
  chessboard images. Comparison from before:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/lens-stability/diff-dance34-splined-noncentral-crop.pdf]]

* Interoperating with other tools                                  :noexport:
** Interoperating with other tools
What if your system is built around CAHVOR, but you want to use
=LENSMODEL_SPLINED_STEREOGRAPHIC= for more accuracy?

- mrcal libraries are easy to use and integrate: it's not a lot of work to
  transition

But if you really don't want to....

** What is the cost of a too-lean model?
mrcal gives us tools to quantify this.

The usual process is:

- Calibrate using the production model
- Look at the projection uncertainty and cross-validation plots to estimate the
  expected calibration errors

** What is the cost of a too-lean model?
With the =LENSMODEL_SPLINED_STEREOGRAPHIC= we can go one step further:

- /Also/ calibrate using =LENSMODEL_SPLINED_STEREOGRAPHIC=
- Use the metrics to show that this fits very well
- Use =LENSMODEL_SPLINED_STEREOGRAPHIC= as the ground truth to get more
  informative diffs

** Having our cake and eating it too
What if we want the accuracy of =LENSMODEL_SPLINED_STEREOGRAPHIC= but we also
don't want to rearchitect our code? With a bit of extra computation we can.

- We have a $\left(\mathrm{model},\mathrm{image}\right)$ tuple that faithfully
  describes the world, if we use =LENSMODEL_SPLINED_STEREOGRAPHIC=
- We can generate another tuple
  $\left(\mathrm{model}_\mathrm{production},\mathrm{image}_\mathrm{reprojected}\right)$
  that reprojects the input image to one that follows the production model. We
  use this tuple everywhere, and it /also/ faithfully describes the world
- Pixel noise variances will need to be adjusted

* Model evaluation                                                 :noexport:
** Estimating ranging errors caused by calibration errors
- Projection errors aren't what we ultimately care about

- mrcal allows us to propagate these to what we care about

Propagating errors to triangulation:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/triangulation/sample--ellipses-crop.pdf]]

** Scene-aware error propagation
If the rough geometry of an observed scene is known beforehand, we can make a
rough expected-error map:

1. Compute $\frac{\partial\mathrm{range}}{\partial\mathrm{azimuth}}$ from the
   triangulation expression
2. Estimate $\Delta \mathrm{azimuth}$ by combining expected sampling error and
   calibration error
3. $\Delta \mathrm{range} \approx \frac{\partial\mathrm{range}}{\partial\mathrm{azimuth}} \Delta \mathrm{azimuth}$

** Scene-aware error propagation
In the downtown Los Angeles scene we observed this calibration error using
=LENSMODEL_OPENCV8=:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-splined-opencv8-crop.pdf]]

** Scene-aware error propagation
- Rough median of calibration error: 0.5 pixels per camera
- At worst: 1.0 pixels of calibration error
- Noise in stereo matching is ~ 0.3 pixels

So we assume a disparity error of 1.0 + 0.3 = 1.3 pixels

** Scene-aware error propagation: left-rectified image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo-range-sensitivity/0-rectified.downsampled.png]]

** Scene-aware error propagation: disparity
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo-range-sensitivity/disparity-crop.pdf]]

** Scene-aware error propagation: propagated range error
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo-range-sensitivity/sensitivity-crop.pdf]]

** Scene-aware error propagation
These errors are correlated and will /not/ average out. They should be
minimized.

* Conclusion                                                       :noexport:
** Conclusion
- mrcal solves many pervasive issues in traditional camera-modeling toolkits
- Allows many practical questions to be addressed directly
- Many improvements and extensions and applications planned and in development

* Stereo                                                           :noexport:
** Overview
mrcal can do some basic stereo processing. At its core, it's the usual epipolar
geometry process:

1. Ingest two camera models
2. Ingest images captured by these two cameras
3. Transform the images to construct "rectified" images
4. Perform "stereo matching"

Each pair of corresponding rows in the rectified images represents a plane in
space:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.4\textheight :options keepaspectratio
[[file:../out/figures/rectification.pdf]]

** Input images
I used the lens I calibrated at the start to capture a pair of images in
downtown Los Angeles. The left image:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo/0.downsampled.jpg]]

We're on a catwalk between 2nd and 3rd, looking S over Figueroa St.

** Rectification
I then used mrcal's rectification function to produce the rectified image. The
left:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/0-rectified.downsampled.png

** Disparity
And the resulting disparity, as computed by the OpenCV matcher:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/0-disparity.downsampled.png

** JPLV

What if we wanted to use JPLV stereo with splined models?

We can use mrcal to remap to another projection and feed /that/ to jplv. For
instance, let's

- Remap to a pinhole model (with some arbitrary zoom factor)
- Use jplv to compute the rectified image

** Narrow virtual cameras
Another way to do stereo processing of wide images using tools that aren't built
for it is to

- split the wide-angle stereo pair into a set of narrow-view stereo pairs

This generates a skewed geometry, but mrcal can still use it just fine. Due to a
bug, jplv cannot.

** Narrow virtual cameras
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/stereo-geometry-narrow.pdf

** Narrow virtual cameras
One of the resulting resampled /pinhole/ images:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/narrow-left.downsampled.jpg

** Narrow virtual cameras
Rectified using mrcal

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/rectified0-narrow.downsampled.jpg

** Narrow virtual cameras
Disparity from OpenCV

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/disparity-narrow.downsampled.png

* Cross-reprojection
** mean-pcam uncertainty shortcomings

This "mean-pcam" uncertainty method is not perfect, and is unsuitable for a
chessboard-less calibration.

- Chessboards are a hard requirement: the "chessboard" coordinate system is used
  explicitly in the method
- $T_{\mathrm{r}^+\mathrm{r}} = \mathrm{mean}_i \left( T_{\mathrm{r}^+\mathrm{f}_i} T_{\mathrm{f}_i\mathrm{r}} \right)$ is aphysical
- Pessimistic response to disparate observed chessboard ranges

** mean-pcam uncertainty shortcomings                   :noexport:
The far-away boards have a less well-defined pose

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/disparate-ranges.pdf]]

** mean-pcam uncertainty shortcomings
And since we use a mean of $\vec p_\mathrm{cam}$ to combine all the points, the far-away boards
make the uncertainty appear artificially poor

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-num-far-constant-num-near--mean-pcam.pdf]]

** Cross-reprojection uncertainty
For any perturbed solve we explicitly solve for $T_\mathrm{rr^+}$ by optimizing
the cross path in red. We solve /only/ for $T_\mathrm{rr^+}$. Everything else is
fixed.

\[
\small
\xymatrix @C=5em {
  {\color{red} \vec q}                                                                          &
  {\color{red} \vec p  _\mathrm{cam}}    \ar@[red][l]^{\color{red} \vec b_\mathrm{intrinsics} } &
  {\color{red} \vec p  _\mathrm{ref}}    \ar@[red][l]^{\color{red} T_\mathrm{cr}}               &
  \vec p  _\mathrm{board}  \ar[l]^{T_\mathrm{rf}}                                               \\
  \vec q^+                                                                                                      &
  \vec p^+_\mathrm{cam}    \ar[l]^{\vec b^+_\mathrm{intrinsics} }                                               &
  {\color{red} \vec p^+_\mathrm{ref}}    \ar[l]^{T_\mathrm{c^+r^+}}  \ar@[red][u]^{\color{red} T_\mathrm{rr^+}} &
  {\color{red} \vec p^+_\mathrm{board}}  \ar@[red][l]^{\color{red} T_\mathrm{r^+f^+}}
}
\]

** Cross-reprojection uncertainty             :noexport:
\[
\small
\xymatrix @C=5em {
  {\color{red} \vec q}                                                                          &
  {\color{red} \vec p  _\mathrm{cam}}    \ar@[red][l]^{\color{red} \vec b_\mathrm{intrinsics} } &
  {\color{red} \vec p  _\mathrm{point}}    \ar@[red][l]^{\color{red} T_\mathrm{cr}}             \\
  \vec q^+                                                                                                      &
  \vec p^+_\mathrm{cam}    \ar[l]^{\vec b^+_\mathrm{intrinsics} }                                               &
  {\color{red} \vec p^+_\mathrm{point}}    \ar[l]^{T_\mathrm{c^+r^+}}  \ar@[red][u]^{\color{red} T_\mathrm{rr^+}}
}
\]

** Cross-reprojection uncertainty
We propagate the input noise $\vec q_\mathrm{ref}$ to get $\mathrm{Var}\left( T_\mathrm{rr^+}
\right)$. We propagate /that/ through this new flow:

\[
{\tiny
\xymatrix @=3.5em {
         & &                                                               & \vec p^+_{\mathrm{cam}_0} \ar[dl] & \vec p^+_{\mathrm{ref}_0} \ar[l]_-{T^+_{\mathrm{c_0 r}}} & \vec p_{\mathrm{ref}_0} \ar[l]_-{T_\mathrm{r^+r}} \\
\vec q^+ & & \vec p^+_\mathrm{cam} \ar[ll]_-{\vec b_\mathrm{intrinsics}^+} & \vec p^+_{\mathrm{cam}_1} \ar[l]  & \vec p^+_{\mathrm{ref}_1} \ar[l]_-{T^+_{\mathrm{c_1 r}}} & \vec p_{\mathrm{ref}_1} \ar[l]_-{T_\mathrm{r^+r}} & \vec p_\mathrm{cam} \ar[l]_-{T_\mathrm{r_1 c}} \ar[lu]_-{T_\mathrm{r_0 c}} \ar[ld]^-{T_\mathrm{r_2 c}} & & \vec q \ar[ll]_-{\vec b_\mathrm{intrinsics}} \\
         & &                                                               & \vec p^+_{\mathrm{cam}_2} \ar[ul] & \vec p^+_{\mathrm{ref}_2} \ar[l]_-{T^+_{\mathrm{c_2 r}}} & \vec p_{\mathrm{ref}_2} \ar[l]_-{T_\mathrm{r^+r}} 
}
}
\]

and we get $\mathrm{Var}\left( \vec q^+ \right)$

** Cross-reprojection uncertainty
This has a /lot/ of complexity, but it is already implemented and well-tested in
many situations.

** Cross-reprojection uncertainty: response to far-away chessboards
With the mean-pcam uncertainty method:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-num-far-constant-num-near--mean-pcam.pdf]]

** Cross-reprojection uncertainty: response to far-away chessboards
With the new cross-reprojection uncertainty method:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-num-far-constant-num-near--cross-reprojection-rrp-Jfp.pdf]]

** Cross-reprojection uncertainty: next steps
There are a lot of thorny issues that I have mostly figured out. One problem is
still open:

How to merge the uncertainty from different camera poses

#+ATTR_LATEX: :width 0.6\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/moving-camera.pdf]]

This has the same issues as the mean-pcam uncertainty producing biased results
from mixed-range observations.

Using the /most-certain/ camera pose works, but it is slow

* Other ways of looking at this
** Other ways of looking at this
I've been looking at diffs and uncertainty of /intrinsics, compensating for
extrinsics shifts/

There are other approaches, I've barely explored. With a /pair/ of cameras, we can:

- compute an /extrinsics diff/, compensating for intrinsics shifts

- compute an intrinsics+extrinsics diff

** Other ways of looking at this
In both cases, we align the left camera of the two stereo pairs, and compare the
right pair

#+ATTR_LATEX: :width 0.6\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/stereo-pair-shifted.pdf]]


** Intrinsics+extrinsics diff
With a stereo /pair/, we can look at the shift in the left-right pixel
transform. This obviates the need for an implied transform.

This is implemented in the [[https://mrcal.secretsauce.net/mrcal-show-stereo-pair-diff.html][=mrcal-show-stereo-pair-diff= tool]]. From the same
sample data:

#+begin_src sh
mrcal-show-stereo-pair-diff                \
  --vectorfield                            \
  --vectorscale 0.1                        \
  --gridn 20 15                            \
  --cbmax 500                              \
  set1/camera-[01].cameramodel             \
  set2/camera-[01].cameramodel
#+end_src
#+begin_src sh :exports none :eval no-export
mkdir -p ~/projects/mrcal-doc-external/figures/stereo-pair-diff
D0=(~/data/2026-02-09*)
mrcal-show-stereo-pair-diff \
  --vectorfield --vectorscale 0.1 \
  --gridn 20 15 \
  --cbmax 500 \
  --title "" \
  --unset key                                                         \
  --hardcopy ~/projects/mrcal-doc-external/figures/stereo-pair-diff/stereo-pair-diff-manually-moved.svg \
  --terminal 'svg size 800,450 noenhanced solid dynamic font ",14"' \
  $D0/set1/results/camera-[01].cameramodel \
  $D0/set2/results/camera-[01].cameramodel
mrcal-show-stereo-pair-diff \
  --vectorfield --vectorscale 0.1 \
  --gridn 20 15 \
  --cbmax 500 \
  --title "" \
  --unset key                                                         \
  --hardcopy ~/projects/mrcal-doc-external/figures/stereo-pair-diff/stereo-pair-diff-manually-moved.pdf \
  --terminal 'pdf size 8in,6in       noenhanced solid color   font ",16"' \
  $D0/set1/results/camera-[01].cameramodel \
  $D0/set2/results/camera-[01].cameramodel
#+end_src

** Intrinsics+extrinsics diff

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures//stereo-pair-diff/stereo-pair-diff-manually-moved.pdf]]

** Extrinsics-only diff                                            :noexport:
The naive expression for the 0right-1right transform:

\[
T_{0\mathrm{right},0\mathrm{left}}
T_{1\mathrm{left},1\mathrm{right}}
\]

This doesn't work because of the required implied transform, as described in the
intrinsics differencing section. The full expression:

\[
T^\mathrm{implied}_{1\mathrm{right},0\mathrm{right}}
T_{0\mathrm{right},0\mathrm{left}}
T^\mathrm{implied}_{0\mathrm{left},1\mathrm{left}}
T_{1\mathrm{left},1\mathrm{right}}
\]

** Extrinsics-only diff
This is implemented in the experimental [[https://www.github.com/dkogan/mrcal/blob/master/analyses/extrinsics-stability.py][=analyses/extrinsics-stability.py=]] tool.

Example: I calibrated, moved one of the cameras, and calibrated again:

#+begin_example
$ analyses/extrinsics-stability.py \
  set1/camera-[01].cameramodel     \
  set2/camera-[01].cameramodel

translation: 151.45mm in the direction [0.78 0.06 0.63]
rotation:    16.626deg around the axis [-0.04 -0.99  0.16]
#+end_example

* Implicit point optimization
** Implicit point optimization
Confident calibration from SFM might require a gajillion points. Direct
optimization would produce a /huge/ optimization problem:

\[N_\mathrm{state} = 3 N_\mathrm{points} + ...\]

We can do this without explicitly including the point coordinates in the state
vector.

** Implicit point optimization
Each measurement using a point $\vec p_\mathrm{ref_k}$ is of the form

\[x_i = \mathrm{project}\left( T_\mathrm{c_j r} \vec p_\mathrm{ref_k} \right) - \vec q_\mathrm{ref} \]

For any two observations of the same point, we can use the mrcal triangulation
routines to compute the 3D point coordinate:

\[
\vec p^\mathrm{implicit}_\mathrm{cam} = \mathrm{triangulate}\left(
  \mathrm{unproject}\left( \vec q_\mathrm{ref_0},  T_\mathrm{c_0 r} \right),
  \mathrm{unproject}\left( \vec q_\mathrm{ref_1},  T_\mathrm{c_1 r} \right)
\right)
\]

** Implicit point optimization
We can use then use this implicit point in the optimization /without/ storing
the point coordinates in the optimization state.

Already implemented and used in several projects. Has lots of unresolved details
that will be addressed

* The end
** Thanks!
These new techniques are cool.

Let's finish them!
