July if you say that nothing happened this month!

Horrible jokes aside, this month’s update contains some cool news both in the #App department (mostly account stuff), as well as the #Tech section (mostly sparse reconstruction stuff).

There are also things happening in the background which we can’t talk about yet… stay tuned 👀!

App

Most of the non-account-related things have been implemented last month, with this month being spent on polishing the changes and implemented the account-related features, namely password resetting and email verification, both of which require an email service. Luckily, there are relatively straightforward to implement, so a 0.3 release early next month is likely.

More UI changes for version 0.3.
Email verification UI coming in version 0.3.

The exact set of features that will be implemented in the next version (modulo minor things) can still be found in the roadmap section of this page.

#Tech

Sparse reconstruction improvements

Until now, we have been cutting corners when doing sparse reconstruction on the new sets. Most importantly, we have not been running bundle adjustment, which meant that the reconstruction precision was sub-par.

We have remedied this by using bundle adjustment on the to-be-registered images, which markedly improved the reconstruction quality. Here, for example, is the difference in the raw number of points1 with and without using bundle adjustment.

BA results for Crimp.
Number of reconstructed points for Crimp with/without bundle adjustment.

This is only one of the many improvements we could make in the sparse reconstruction process, but arguably the most significant one. Others include filtering out false keypoints from the original reconstruction and splitting the reconstruction into sub-reconstructions based on the matched keypoints.

SAM more

We are also continuing ways of using SAM in our reconstruction pipeline, in more ways than data augmentation – it turns out that it’s really good at detecting all objects in the images, but just doesn’t really know what it’s looking at.

We believe that combining SAM with YOLO (classification/segmentation) in some way, along with a healthy dose of heuristics for holds/volumes, will produce the best results. There are no benchmarks yet available, but we are looking into it!

Team Climbuddy

  1. While the raw number of points is definitely not the only metric to look at, the median reprojection error is comparable with/without BA, so the reconstruction quality is much better – more points means easier time for the dense reconstruction algorithms. ↩