#+TITLE: mrcal 3.0 release notes
#+OPTIONS: toc:nil

* New in mrcal 3.0

cross-reprojection

mrcal.drt_ref_refperturbed__dbpacked()
mrcal.compose_rt_tinyrt0_gradientrt0()
mrcal.compose_rt_tinyrt1_gradientrt1
compose_r_tinyr1_gradientr1

75893eea: Added initialy EARLY version of mrcal-show-stereo-pair-diff (this
needs to be tested, documented)

faster uncertainty computation:
  cecab50a CHOLMOD_factorization.solve_xt_JtJ_bt() can solve other problems
  9393ef79 Much faster uncertainty computation

mrcal.cameramodel() can take icam_extrinsics. To support creating and managing
a model object built from moving cameras

mrcal.show_geometry() and mrcal-show-geometry have icam_extrinsics_fromsolve and
--icam-extrinsics-from-solve. This is useful for visualizing moving cameras. If
given, instead of showing one set of camera axes per model given on the
commandline, we instead take the ONE model from the commandline, and use the
extrinsics in its optimization_inputs. The extrinsics we use are given in
--icam-extrinsics-from-solve as a ,-separated list of integers A and/or A-B
ranges. A value of -1 means "identity", useful to show cameras at the
optimization reference. A special value of "all" means "show ALL of the
extrinsics". This option is useful for moving-camera solves

I can save .pgm and .ppm images


* Migration notes 2.4 -> 3.0

* todo
- add notes about the triangulated-points merge

** triangulated features merge
*** =git grep -n Noutliers= Noutliers has change meaning: it's now =Nmeasurements_outliers=

   #+begin_src diff
   diff --git a/doc/c-api.org b/doc/c-api.org
   index 7d3c8939..2ae7d9d5 100644
   --- a/doc/c-api.org
   +++ b/doc/c-api.org
        /* The RMS error of the optimized fit at the optimum. Generally the residual */
        /* vector x contains error values for each element of q, so N observed pixels */
        /* produce 2N measurements: len(x) = 2*N. And the RMS error is */
        /*   sqrt( norm2(x) / N ) */
        double rms_reproj_error__pixels;

   -    /* How many pixel observations were thrown out as outliers. Each pixel */
   -    /* observation produces two measurements. Note that this INCLUDES any */
   -    /* outliers that were passed-in at the start */
   +    /* How many measurements were thrown out as outliers. Each pixel */
   +    /* observation of a chessboard point produces two measurements (x,y). */
   +    /* Note: this INCLUDES any outliers that were passed-in at the start */
        int Noutliers;
    } mrcal_stats_t;


   --- a/mrcal-calibrate-cameras
   +++ b/mrcal-calibrate-cameras
   @@ -745,13 +745,13 @@ Npoints_chessboard = args.object_width_n*args.object_height_n*Nobservations
    residuals = \
        stats['x'][:Npoints_chessboard*2]. \
        reshape(Nobservations, args.object_height_n, args.object_width_n, 2)
    worst_point_err = np.sqrt(np.max(nps.norm2( nps.clump(residuals, n=3) )))
    report += f"Worst residual (by measurement): {worst_point_err:.01f} pixels\n"
    if not args.skip_outlier_rejection:
   -    report += "Noutliers: {} out of {} total points: {:.01f}% of the data\n". \
   +    report += "Noutliers: {} out of {} total measurements: {:.01f}% of the data\n". \
            format(stats['Noutliers'],
                   args.object_height_n*args.object_width_n*len(observations),
                   100.0 * stats['Noutliers'] / (args.object_height_n*args.object_width_n*len(observations)))
    if calobject_warp is not None:
        report += f"calobject_warp = {calobject_warp}\n"
 
   #+end_src

*** divergent-rays-are outlier logic is weird
I declare and outlier on the first pass. That pass is for getting the threshold

** patches deferred for next release

#+begin_src diff
diff --git a/mrcal-show-projection-diff b/mrcal-show-projection-diff
index 572d701..6cb48dc 100755
--- a/mrcal-show-projection-diff
+++ b/mrcal-show-projection-diff
@@ -503,3 +503,7 @@ if not args.intrinsics_only and args.radius != 0 and \
 
 if args.hardcopy is None:
     plot.wait()
+
+
+# should --unset key be the default? And for the uncertainty plot?
+
diff --git a/mrcal-show-residuals-board-observation b/mrcal-show-residuals-board-observation
index 76ce4db..b8c17eb 100755
--- a/mrcal-show-residuals-board-observation
+++ b/mrcal-show-residuals-board-observation
@@ -365,3 +365,8 @@ The optimization inputs are available in the optimization_inputs dict
 for i in range(Nplots):
     os.waitpid(pids[i], 0)
 sys.exit()
+
+
+
+
+### add auto-vector-scale
#+end_src

** _propagate_calibration_uncertainty() needs to be exported in the API
** I should check the camera extrinsics uncertainty
If the camera geometry is very uncertain, the calibration isn't successful; even
if the variance in the other state variables compensates for these perfectly.
The _propagate_calibration_uncertainty() function can easily do this. I should
rename it. And I should expose it as part of the API. This code works to detect
uncertain extrinsics for a camera pair:

#+begin_src python

model_filename = sys.argv[1]
m = mrcal.cameramodel(model_filename)
optimization_inputs = m.optimization_inputs()

istate_extrinsics0 = mrcal.state_index_extrinsics(0, **optimization_inputs)
Nstate_extrinsics  = mrcal.num_states_extrinsics(    **optimization_inputs)

Nstate = mrcal.num_states( **optimization_inputs)

if Nstate_extrinsics != 6:
    raise Exception(f"Unexpected {Nstate_extrinsics=}")

dF_db = np.zeros((Nstate_extrinsics, Nstate), dtype=float)
dF_db[:,istate_extrinsics0:istate_extrinsics0+Nstate_extrinsics] = \
    np.eye(Nstate_extrinsics)

Var_rt_cam_ref = \
    mrcal.model_analysis._propagate_calibration_uncertainty('covariance',
                                                            dF_db = dF_db,
                                                            observed_pixel_uncertainty = 1.,
                                                            optimization_inputs = optimization_inputs)

print(f"stdev(rt_cam_ref) = {np.sqrt(np.diag(Var_rt_cam_ref))}")

#+end_src

** uncertainty regression
The triangulated-features merge caused the uncertainty reporting to be a bit
different for some reason. I need to chase it down to see what happened. I'm
looking at

~/projects/mrcal.old/out0.cameramodel

This command is returning slightly different results before/after the merge:

~/projects/mrcal.old/mrcal-show-projection-uncertainty out0.cameramodel --cbmax 30

** uncertainty strongly affected by regularization weight
Computing the uncertainty of the results of stationary-calibration.py can
produce wildly different output if I tweak the regularization weight

** regularization scaling
I should aim for specific number of pixels instead of for some ratio. This will
probably break loading optimization_inputs from model files: they'd need
reoptimization

** point range normalization
I removed it here: 0e727189. Do I want it back in some form? I do still require
point_min_range and point_max_range. Do I really need these?

** XyJax loaded in too many doc pages
I need it everywhere I use \xymatrix (currently uncertainty.org only). So that's
the only place I should use it. Loading it needlessly is slow

** mrcal-convert-lensmodel
This needs to support points:
- search for indices_point_camintrinsics_camextrinsics
- solving without --sampled fails with points: no logic to do point culling

** mrcal-cull-corners should be able to cull board edges
Need new option like =--cull-board-rowscols L,T,R,B=

Can hack it on the commandline:

#+begin_src sh
R=1; < $C vnl-filter --sub 'ii() { if(filename != prev(filename)) { i=0; return i; } return ++i; }' -p .,'i=ii()' | vnl-filter -p .,\!i,'i=int(i/14)',j='i % 14' | vnl-filter -p filename,x,y,level="(i<$R || i>=14-$R || j<$R || j>=14-$R) ? \"-\" : level" > /tmp/corners-board-edge-cut$R.vnl
#+end_src

** mrcal-stereo should have an anti-aliasing filter
When I downsample. Just before =mrcal.transform_image()= it should

#+begin_src python
for i in range(len(images)):
    images[i] = cv2.GaussianBlur(images[i],
                                 ksize=(0,0), # auto-select
                                 # sigmaX = 2 ^ -pixels_per_deg,
                                 sigmaX = 2 )
#+end_src

** I should support more lens models
Being compatible with at least ROS would be nice. Their models are:

- =plumb_bob=: This is =LENSMODEL_OPENCV5=
- =rational_polynomial=: This is =LENSMODEL_OPENCV8=
- =equidistant=: mrcal does not support this today. It should. This is
  [[https://docs.opencv.org/3.4/db/d58/group__calib3d__fisheye.html][cv::fisheye]]

** _mrcal_drt_ref_refperturbed__dbpacked() currently is hardcoded to use the Jfpcw formulation
Give it an argument to select the formulation. And rename the function. Or
something
** other stuff
- "pydoc3 mrcal" should show everything. It doesn't. "compose_rt" isn't there,
  for instance

- mrcal-stereo: during the rectification (or maybe disparity search) stage C-c
  doesn't work.

** Expose _options_heatmap_with_contours()
** mrcal.stereo_range() does uint16 based on qrect0 is None. It should look at the type
Currently a full-image float range image doesn't work right: it casts to uint16,
and we lose accuracy
** Port to mrcal 2.5
commit 317b4b904f1f1fa3c983e48d86104dca50893a6e
Author: Dima Kogan <dima@secretsauce.net>
Date:   Thu Aug 10 14:30:58 2023 -0700

    r_from_R_core() better implementation around th=180deg

 And mrcal-stereo --single-buffered

** document valid-intrinsics-region

** new observed_pixel_uncertainty
  - apply sqrt() factor. Documented in 68789474. git grep -n
    observed_pixel_uncertainty ..
  - Does too little data give smaller residuals? It should. Is this reflected in
    the sqrt() factor?

** test-projection-uncertainty improvements
"--fixed frames" and --moving-camera and --points should be consolidated into a
clearer set of things. I should be able to have the surveyed calibration
uncertainty test in test-projection-uncertainty, but that doesn't currently
work. The best-effort is the below patch:

#+begin_src diff
diff --git a/test/test-projection-uncertainty.py b/test/test-projection-uncertainty.py
index 8cc40962..2ac9d2c7 100755
--- a/test/test-projection-uncertainty.py
+++ b/test/test-projection-uncertainty.py
@@ -379,17 +379,72 @@ x_baseline_optimized = \
 
 lensmodel       = optimization_inputs_baseline['lensmodel']
 imagersizes     = optimization_inputs_baseline['imagersizes']
 intrinsics_true = nps.cat( *[m.intrinsics()[1] \
                              for m in models_true] )
 
-models_baseline = \
-    [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
-                         icam_intrinsics     = i) \
-      for i in range(args.Ncameras) ]
 
+print(f"rt_cam_ref.shape: {optimization_inputs_baseline['rt_cam_ref'].shape}")
+if optimization_inputs_baseline.get('indices_frame_camintrinsics_camextrinsics') is not None:
+    print(f"indices_frame_camintrinsics_camextrinsics.shape: {optimization_inputs_baseline['indices_frame_camintrinsics_camextrinsics'].shape}")
+else:
+    print(f"indices_point_camintrinsics_camextrinsics.shape: {optimization_inputs_baseline['indices_point_camintrinsics_camextrinsics'].shape}")
+
+
+if False:
+    ####### this can't be written in all cases:
+    # test/test-projection-uncertainty.py \
+    #   --fixed frames \
+    #   --model opencv4 \
+    #   --Ncameras 1 \
+    #   --range-to-boards 4 \
+    #   --points
+    # mrcal.c(3944): Cannot compute icam_extrinsics. I don't have a vanilla calibration problem (stationary cameras, cam0 is reference)
+    # Traceback (most recent call last):
+    #   File "/home/dima/projects/mrcal/test/test-projection-uncertainty.py", line 389, in <module>
+    #     [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+    #       ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    #                          icam_intrinsics     = i) \
+    #                          ^^^^^^^^^^^^^^^^^^^^^^^^
+    #   File "/home/dima/projects/mrcal/test/../mrcal/cameramodel.py", line 1249, in __init__
+    #     icam_extrinsics = mrcal.corresponding_icam_extrinsics(icam_intrinsics,
+    #                                                           **optimization_inputs)
+    # RuntimeError: mrcal-pywrap.c:3277 callback_corresponding_icam_extrinsics(): Error calling mrcal_corresponding_icam_extrinsics()
+    models_baseline = \
+        [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+                             icam_intrinsics     = i) \
+          for i in range(args.Ncameras) ]
+else:
+    # This path causes a crash:
+    # test/test-projection-uncertainty.py \
+    #   --fixed cam0 \
+    #   --model opencv4 \
+    #   --do-sample \
+    #   --observed-pixel-uncertainty 0.03 \
+    #   --reproject-perturbed cross-reprojection-rrp-Jfp \
+    #   --points
+    # Traceback (most recent call last):
+    #   File "/home/dima/projects/mrcal/test/test-projection-uncertainty.py", line 394, in <module>
+    #     [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+    #       ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    #                          icam_intrinsics = 0,
+    #                          ^^^^^^^^^^^^^^^^^^^^
+    #                          icam_extrinsics = i) \
+    #                          ^^^^^^^^^^^^^^^^^^^^
+    #   File "/home/dima/projects/mrcal/test/../mrcal/cameramodel.py", line 1254, in __init__
+    #     self.rt_cam_ref(optimization_inputs['rt_cam_ref'][icam_extrinsics])
+    models_baseline = \
+        [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+                             icam_intrinsics = 0,
+                             icam_extrinsics = i) \
+          for i in range(args.Ncameras) ]
+##### the extra test in Makefile.tests does not pass. I try to do
+##### fixed-points-only uncertainty. It should work. test-surveyed-calibration
+##### does the same thing (I think) and that does work. The only difference is
+##### that that test has only a single camera pose
 
 # I evaluate the projection uncertainty of this vector. In each camera. I'd like
 # it to be center-ish, but not AT the center. So I look at 1/3 (w,h). I want
 # this to represent a point in a globally-consistent coordinate system. Here I
 # have fixed frames, so using the reference coordinate system gives me that
 # consistency. Note that I look at q0 for each camera separately, so I'm going
@@ -2332,12 +2397,69 @@ else:                                                          reproject_perturb
 # "method" argument for mrcal.projection_uncertainty()
 if   re.match('^(mean-pcam|bestq)$', args.reproject_perturbed): method = args.reproject_perturbed
 elif re.match('cross-reprojection',  args.reproject_perturbed): method = 'cross-reprojection-rrp-Jfp'
 # default
 else:                                                           method = 'mean-pcam'
 
+
+
+
+
+
+
+
+
+distance = args.distances[0]
+# shape (Ncameras, 3)
+p_cam_baseline = mrcal.unproject(q0_baseline, lensmodel, intrinsics_baseline,
+                                 normalize = True) * distance
+
+# shape (Ncameras, 3)
+p_ref_baseline = \
+    mrcal.transform_point_rt( mrcal.invert_rt(extrinsics_baseline_mounted),
+                              p_cam_baseline )
+p_ref_query = p_ref_baseline
+p_cam_query = \
+    mrcal.transform_point_rt(rt_cam_ref_true, p_ref_query)
+q = mrcal.project(p_cam_query, lensmodel, intrinsics_true)
+print(q[0])
+
+import IPython
+IPython.embed()
+sys.exit()
+
+# THE PROBLEM:
+#
+# extrinsics_baseline_mounted should be the reoptimized rt_cam_ref_true, but
+# they don't even have the same shape:
+r'''
+In [5]: extrinsics_baseline_mounted.shape
+Out[5]: (50, 6)
+
+In [10]: extrinsics_baseline_mounted[0]
+Out[10]: array([ 0.05111173,  0.22532253,  0.07174262, -0.00834507, -0.03635452,  0.20792226])
+
+In [11]: rt_cam_ref_true[0]
+Out[11]: array([0., 0., 0., 0., 0., 0.])
+
+In [12]: rt_cam_ref_true.shape
+Out[13]: (1, 6)
+
+'''
+
+
+
+
+
+
+
+
+
+
+
+
 q0_true = dict()
 for distance in args.distances:
 
     # shape (Ncameras, 2)
     q0_true_here = \
         reproject_perturbed(q0_baseline,
@@ -2401,13 +2525,13 @@ for icam in (0,3):
                                       atinfinity = True,
                                       method     = method,
                                       observed_pixel_uncertainty = args.observed_pixel_uncertainty )
 
     # I move the extrinsics of a model, write it to disk, and make sure the same
     # uncertainties come back
-    if True:
+    if False:
         model_moved = mrcal.cameramodel(models_baseline[icam])
         model_moved.rt_cam_ref([1., 2., 3., 4., 5., 6.])
         model_moved.write(f'{workdir}/out.cameramodel')
         model_read = mrcal.cameramodel(f'{workdir}/out.cameramodel')
 
         icam_intrinsics_read = model_read.icam_intrinsics()
#+end_src

Running:

#+begin_src sh
test/test-projection-uncertainty.py \
  --fixed frames \
  --model opencv4 \
  --Ncameras 1 \
  --range-to-boards 4 \
  --points
#+end_src

Doesn't work. q_baseline diverges a LOT from q_true. Something isn't right. This
is already implemented and working in test-surveyed-calibration --do-sample. So
the functionality is validated, it's the test that needs to be improved.

* C stereo unproject

* triangulate() should report stuff in the ref coords, not camera0 coords
It doesn't make a whole lot of sense the way I'm doing it right now

* mrcal c api stuff should mostly be double* instead of mrcal_point3_t

* m-c-c should be able to do extrinsics-only solves better
- I need to pass in intrinsics. Currently I do that with --seed, which is bad
  because:
  - It's ,-separated which breaks tab-completion
  - I don't want to seed the extrinsics. I want to load the intrinsics and use
    the normal seeding algorithm for extrinsics
- In this case I should be able to have non-matching --lensmodel


* release checklist
These are notes to myself containing the steps needed to roll a new release

- docs: make sure all new python functions are described in python.org
- test building with clang
- new [[file:versions.org][versions]]
- new [[file:news-2.2.org][news]]
- [[file:~/projects/mrcal/Makefile::PROJECT_NAME := mrcal][Makefile ABI version]]
- package build and upload
- versioned docs:
  - on the server move =docs-latest-release= symlink. This controls what =make publish-docs= writes to
  - on the server populate the =docs-latest-release/external= symlink
  - on the server move =docs-default= symlink. This controls what
    https://mrcal.secretsauce.net sees
- git tag
