Geometry aware neural radiance fields for freehand ultrasound reconstruction

Abstract Reconstructing 3D volumes from 2D freehand ultrasound (US) is a challenging task. During reconstruction, the ensuing overlap between sweeps can cause multiple pixels to be assigned to the same voxel, so the accurate alignment of these sweeps is critical. In recent years, implicit representation methods, such as neural radiance fields (NeRFs), have been utilized for modeling the 3D scene as a continuous volumetric function learned from 2D B-mode images. However, NeRF-based methods are also highly sensitive to errors in camera or transducer poses, a challenge that is particularly pronounced in freehand US with misregistration between overlapping sweeps, leading to severe reconstruction artifacts. To address this challenge, we propose geometric aware US NeRF (GAU-NeRF) by introducing a gradient reweighting strategy to reduce gradient fluctuations from noisy poses during early training iterations and to stabilize the optimization process. GAU-NeRF results in improved accurate pose refinement and improved reconstruction quality. Our method significantly outperforms existing baseline models on both simulated and in vivo US datasets, achieving substantial gains across multiple metrics, including up to 132% increase in the peak signal-to-noise ratio, 133% improvement in the structural similarity index measure, and 350% reduction in the learned perceptual image patch similarity.