FOUND: Foot Optimisation with Uncertain Normals for Surface Deformation using Synthetic Data

WACV 2024

Oliver Boyne Gwangbin Bae James Charles Roberto Cipolla

University of Cambridge

arXiv

Code

Synthetic dataset

Reconstruction dataset

At a glance

We produce a large photorealistic synthetic foot dataset using BlenderSynth.
We train a predictor to predict surface normals and uncertainties.
We use surface normals and uncertainty to fit the FIND foot model to multiview images
We beat SOTA on foot surface normal prediction, and COLMAP on multiview reconstruction, also requiring far fewer views.

Synthetic data

We create a synthetic dataset, SynFoot, of 50K images of feet, along with surface normals, keypoints and masks.

These were created using our custom library BlenderSynth, and are available for download.

Samples from synthetic dataset (drag the slider to view)

Surface normal prediction

We train a network to predict both normals and corresponding uncertainties.

Even though our synthetic dataset only has 8 foot scans, we find that, with aggressive data augmentation, our normal predictor achieves high quality surface normal predictions on in-the-wild images.

We obtain ground truth normals from our reconstruction dataset, and show that our method significantly outperforms COLMAP, and SOTA normal predictors.

In-the-wild predictions (drag the slider to view)

**Surface normal prediction, compared with state-of-the-art**

3D Reconstruction

We fit the parameters of the FIND model to the surface normals, using the uncertainty to weight the loss function.

We do this in a multiview setting, and produce better reconstructions than COLMAP, evaluated on our new benchmark foot reconstruction dataset, available for download.

We can do this accurately with as few as 3 views, whereas COLMAP needs 15+.

**3D reconstruction. Left: A single view of a multi-view reconstruction. Right: Our reconstructed model.**

Reconstruction results based on number of input views. We show that COLMAP fails with fewer than 15 views, whereas we can handle as few as 3. Also our normal loss, and use of normal uncertainty both increase our reconstruction quality.

Acknowledgements

We acknowledge the collaboration and financial support of Trya Srl.

If you make use of this project, please cite the following paper:

@inproceedings{boyne2024found,
            title={FOUND: {F}oot {O}ptimisation with {U}ncertain {N}ormals for Surface {D}eformation using Synthetic Data},
            author={Boyne, Oliver and Bae, Gwangbin and Charles, James and Cipolla, Roberto},
            booktitle={Winter Conference on Applications of Computer Vision (WACV)},
            year={2024}
        }