Enhancing 3-D Reconstruction Using Multi-Objective Loss Functions in Neural Radiance Fields for Broad-scale Outdoor Scenes
by Cameron Halaby, Agni Athreya, Kavya Mohan, Lia Mathews, and Cullen Owens
University of Georgia Small Satellite Research Laboratory
This study investigates the application of Neural Radiance Fields (NeRF) to generate continuous 3-D renderings from UAV-captured data. The research specifically explores the extent to which integrating a multi-objective loss function, through combining traditional pixel-level metrics with Learned Perceptual Image Patch Similarity (LPIPS), can improve the structural and perceptual quality of reconstructions. By evaluating four experimental conditions that varied input scope, frequency, and weight scheduling, we assessed the tradeoffs between mathematical pixel accuracy and human-centric visual fidelity. Our results demonstrate that periodically applying full-image LPIPS calculations (every ten iterations) significantly enhances edge preservation and structural coherence without sacrificing pixel-level precision. Additionally, though basic forms of NeRF involve computationally heavy operations (e.g., sampling thousands of points along each ray out of millions of rays in a single image), our research explores methods to reduce these demands while maintaining high-quality 3-D reconstructions. Use of perceptual similarity reduces the amount of ray sampling required in NeRF processing, creating a model that requires significantly less image overlap, geometric computation, and processing time. This model leads to more efficient data collection and richer feature reconstructions for UAV platforms, with the long-term goal of supporting onboard reconstruction capabilities for CubeSat missions. These findings suggest that prioritizing perceptual logic allows for more photorealistic and reliable 3-D models, ultimately providing a more robust framework for monitoring real-time changes in complex physical environments.
computer vision, image processing, UAV, neural networks
Introduction
Unmanned Aerial Vehicles (UAVs), autonomous aircraft capable of precise movement (e.g. drones), are crucial for environmental data collection due to their ability to provide high-resolution, precise aerial perspectives of complex terrains. A primary objective in UAV data processing is the generation of 3-D models from 2-D imagery, with the goal of improving terrain mapping and enabling geographic contour analysis. Traditional methods of obtaining accurate measurements and 3-D models from photographs rely on the identification of discrete key points (i.e. the same part of an object identified in separate images) to reconstruct geometry. However, when applied to UAV data, these methods frequently produce fragmented, sparse point-clouds with disconnected pieces instead of whole, continuous shapes. As an example, you can imagine trying to model a mountain range as a series of discrete points scattered in space as opposed to something like a smooth surface. These models are often the consequence of inconsistencies in exposure and shadows, resulting from differences between images captured at differing angles, or scene geometry exacerbating differences in lighting across time.1, 2, 6
To address the limitations of discrete reconstruction, Neural Radiance Fields (NeRF) have emerged as an alternative framework for representing 3-D scenes. NeRF utilizes a Multi-Layer Perceptron (MLP), a feedforward neural network architecture, to model the volumetric density and radiance of a scene as a continuous function. By encoding a scene implicitly within the weights of a neural network, NeRF facilitates the synthesis of novel views with high geometric smoothness. This approach replaces the reliance on fragmented point-clouds with a continuous representation, allowing for more comprehensive modeling of large-scale outdoor environments, like mountains and rivers. For each angle of the scene that you would like to view, the neural network offers a guess at the color of each pixel in the frame. In a sparse point-cloud on the other hand, if there is no point in space to account for a certain pixel, the spot will be left blank, meaning whole regions can be left empty. The accuracy of these guesses is then determined by the quality and quantity of the data used to train the network.1, 3, 8, 9
Despite the advantages of implicit representation, the accuracy of a NeRF model is also highly dependent on the objective functions used during the training phase. The standard optimization metric, Mean Squared Error (MSE), evaluates the difference in pixel intensity between rendered views and ground-truth images. While MSE is effective for achieving pixel-level correspondence, it lacks sensitivity to the structural and textural features necessary for realistic 3-D rendering.9, 10
This study evaluates the integration of a multi-objective loss function to improve the fidelity of UAV-based reconstructions. Specifically, the research explores the use of Learned Perceptual Image Patch Similarity (LPIPS) in conjunction with traditional pixel-level metrics (e.g. MSE, PSNR). Utilizing a dataset of drone imagery of Sanford Stadium in Athens, Georgia, this paper examines how varying the frequency and weight of perceptual loss calculations affects model stability and structural coherence in a complex environment with multiple height levels and reflective material at a large-scale. This environment thus provided a suitable baseline for how this technology would render the nuances of riverbeds and mountainscapes, with the advantage of yard lines allowing for the transfer of the scene to real-world scale. The analysis focuses on determining an optimal weight schedule to enhance edge preservation and overall reconstruction accuracy. It is argued that the implementation of perceptually aware loss functions provides a more robust strategy for the 3-D monitoring of real-world environments.
Computational Framework
An MLP consists of multiple layers of nodes that process data in a non-linear fashion to achieve pattern recognition. Within this network, weights indicate the relative importance of
specific input features, while biases are added to the summed weights of a layer to facilitate complex interactions between input variables. During the training process, the model undergoes backpropagation, where gradients are computed to adjust these weights and biases. This optimization typically utilizes an Adam Optimizer to minimize the loss value, a measure of model inaccuracy.4, 9
In this study, the MLP is utilized within the NeRF framework to represent scenes as continuous volumetric functions. NeRF models map 5-D coordinates, comprising spatial position (x, y, z) and viewing direction (θ, ϕ) to specific outputs: volume density and view-dependent RGB radiance.9 The density determines the opacity of a point in space, controlling how much light is emitted or absorbed. By stochastically sampling these points along camera rays and integrating the results, the model synthesizes novel views that were not present in the original training data. The efficacy of a NeRF model is determined by the loss function used to evaluate the discrepancy between rendered views and ground-truth images. The most common metric is MSE, which calculates the square of the difference between predicted and actual pixel intensities. As previously stated, the pixel-by-pixel computation of a standard NeRF model may achieve a high MSE value while producing reconstructions that appear blurred or structurally inaccurate to a human observer.5, 19
To improve structural fidelity, this research incorporates LPIPS as a secondary loss function, creating a multi-objective optimization strategy. Unlike MSE, which evaluates individual pixels, LPIPS employs deep features extracted from pre-trained convolutional neural networks to compare image patches. By calculating and summing the average spatial distances between deep feature activations across multiple network layers, LPIPS quantifies errors in edges, contours, and fine structural details.19 This approach aims to balance the mathematical precision of MSE with high structural coherence, particularly when processing inherently inconsistent aerial data.
Methodology
Data Acquisition and Preprocessing
The dataset utilized for this study consisted of 5,964 UAV TIFF images of Sanford Stadium, provided by the University of Georgia Department of Geography. The imagery was captured using an Alta X-Blue 2.0 drone mounted with Micasense RedEdge-P Sensor following a regular grid system at constant altitude, capturing images with 75% image overlap. The raw data contains minor inconsistencies in lighting due to shifting sun angles during image capture (60 minutes). These inconsistencies were exacerbated by long shadows cast by stadium height occlusions and required standardization before model training.
The Micasense RedEdge-P sensor contains six spectral bands, each from a different camera. However, the individual cameras are not mechanically aligned and require post-processing to align. For each image of the stadium, only three bands (red, green, and blue) were considered for image reconstruction. Misalignment between these bands was corrected using a phase correlation algorithm from the MicaSense software development kit (https://micasense.github.io/rededge-api/), which estimates the translational shift between channels by comparing frequency patterns. After preprocessing, a total of 994 combined RGB images were used for training.
Camera Pose Estimation
To establish the geometric relationship between the 2-D images and the 3-D scene, camera intrinsics and extrinsics were estimated using COLMAP.13 This process generated a sparse point cloud and a set of oriented camera poses. To ensure the reliability of the pose estimation, a baseline model was initially trained using an unmodified Nerfstudio implementation (https://github.com/nerfstudio-project/nerfstudio). This preliminary step confirmed that the data-loading and pose-estimation pipelines were functioning correctly before the introduction of experimental loss functions.
Experimental Design
The research employed a factorial design to evaluate how modifications to the LPIPS calculation influenced model stability and reconstruction quality. In its default configuration, LPIPS is applied to small image patches, which can introduce gradient fluctuations and lead to a loss of global spatial context. To address this, three variables of the LPIPS loss were modified. First, input scope: a comparison of the baseline patch-based method against full-images resized to 50% of their original spatial resolution via bilinear interpolation. Second, frequency: a comparison of LPIPS computation at every training iteration versus computation once every ten iterations to stabilize gradients and reduce computational overhead. Third, weight schedule: a comparison of a constant LPIPS weight (0.1) against a dynamic schedule that gradually increased the influence of the perceptual loss as the training progressed.
These variables were tested across four primary experimental conditions (Table 1). Each model was trained using the same dataset, initialization parameters, and random seed to ensure a controlled comparison.
Evaluation Metrics
To assess reconstruction quality, four quantitative error metrics were utilized. MSE and Peak Signal-to-Noise Ratio (PSNR) were used to evaluate pixel-level accuracy and noise levels.4 Structural Similarity Index Measure (SSIM) was employed to measure luminance, contrast, and structural correlation between the rendered and ground-truth images.5, 10 Finally, the LPIPS score was recorded to assess the perceptual distance between the model output and the reference imagery, with lower values indicating higher similarity.
Results and Analysis
To evaluate the impact of multi-objective loss functions on NeRF reconstruction, four experimental configurations (A–D) were compared against a standard MSE baseline. Each experiment utilized a 0.1 weight for the LPIPS component.
Quantitative Performance and Metric Comparison
The data in Table 2 indicates that the integration of a multi-objective loss function improves reconstruction quality across most measured metrics compared to the MSE baseline. The baseline model achieved an RGB loss of 0.0345 and a PSNR of 15.07. In contrast, all four experimental trials (A–D) achieved higher PSNR values and, with the exception of Experiment B, lower RGB loss values.
Experiment C, which utilized periodic full-image LPIPS calculations every ten iterations, yielded strong structural fidelity, as evidenced by an LPIPS score of 0.4808 and an SSIM of 0.4860, while also achieving the highest PSNR of 16.18. Experiment D, which implemented a dynamic weight schedule, produced the lowest LPIPS loss of 0.4788 and a PSNR of 15.98, along with an RGB loss of 0.03025. These results suggest a functional tradeoff between structural coherence and pixel-level intensity matching. While Experiment D better optimizes for the latter, Experiment C demonstrates superior performance in preserving the perceptual and geometric relationships within the scene.
NeRF results on Sanford Stadium (see Table 1) show consistent improvement in evaluation loss across Experiments A, C, and D. These experiments also exemplify substantial improvement in the LPIPS loss, with Experiment B still yielding a slight reduction compared to our baseline (0.6141). It is also evident that all four experimental trials (A-D) achieved a slightly higher PSNR value than the baseline model.
Experiment A revealed a significant reduction in perceptual error (LPIPS -21.4%) relative to baseline MSE, additionally indicating a moderate improvement in pixel-wise reconstruction quality (MSE -9.2%). PSNR gains (+0.16 dB) were limited, however. Experiment B highlighted the inefficacy of full-image LPIPS for every iteration. It generated a substantial increase in the evaluation loss, almost 10 times the loss displayed in the MSE-only model. Moreover, this destabilization of optimization caused a small reduction in structural similarity (-3.0%). Experiment C acquired the highest overall performance, significantly enhancing
reconstruction fidelity (PSNR +1.1 dB) while also exhibiting substantial perceptual gains (LPIPS -21.7%, MSE -13.0%) and structural similarity improvement (SSIM +2.4%). Experiment D, using a dynamic weight schedule, yielded the lowest LPIPS score (-22.0%). However, this was achieved at the expense of structural similarity, which faced a reduction of 6.8%.
Overall, the results show consistent improvement for LPIPS, with up to a 22.0% drop compared to our MSE-only baseline. While the application of LPIPS for every iteration of the feedforward network degrades optimization stability, reducing this feature to every 10 iterations (as reflected in Experiments C and D) produced great improvements across all metrics. Between C and D, we have determined that Experiment C showcases the best overall tradeoff, achieving the largest PSNR gain, preserving low LPIPS and MSE loss, and slightly improving SSIM. These scores support a balance between perceptual and pixel-level reconstruction quality.
Impact of Camera Pose Density on Reconstruction
The quality of the NeRF reconstruction is highly dependent on the spatial distribution of the input imagery. Figure 1 illustrates a render from Experiment C at 30,000 training iterations, with individual camera poses estimated by COLMAP marked with yellow boxes. A clear discrepancy in reconstruction quality is visible between the center portion accounted for by camera viewing angles, and the edges of the scene. The center, which is covered by estimated viewing angles (camera poses), exhibits defined geometry. Conversely, the outer portion, characterized by sparse camera coverage, appears blurred and lacks structural definition. The portion of the scene between these two regions, the periphery right on the edge of viewing angle coverage, exhibits a transition zone in structural coherence.
Training progression is shown in Figure 2. The 2×3 grid compares the evolution of Experiment C (top row) and the MSE baseline (bottom row) at 500, 10,000, and 30,000 iterations. In the early stages of training (500 iterations), both models struggle to represent areas with low coverage. However, as iterations increase, the models utilizing LPIPS integration (Experiment C) demonstrate a more rapid stabilization of edge regions compared to the MSE baseline.
Comparative Qualitative Analysis
A direct comparison of the final renders at 30,000 iterations provides insight into the practical benefits of perceptually aware loss functions. Figure 3 presents side-by-side renders of Experiment C and the MSE baseline. Specific regions of interest are highlighted with red rectangles to illustrate the differences in fine-detail preservation.
In the MSE-only model, the high-frequency features, such as text and specific edge boundaries, exhibit blurring artifacts. In contrast, the render from Experiment C demonstrates improved edge sharpness and structural continuity in the highlighted regions. In regions with little or no coverage, the Experiment C render retains sparser, white patches, as opposed to the MSE baseline, which fills in regions of lower coverage with increasingly sporadic noise. By utilizing a loss function that evaluates image patches through a pre-trained vision network, the model is better able to identify and preserve the boundaries of physical features, as well as avoid irregular pixel noise.
A qualitative comparison of Experiments C and D at 30,000 iterations is provided in Figure 4. While Experiment D achieved the highest PSNR, Experiment C resulted in the highest perceptual metrics (SSIM, LPIPS).
In summary, the combination of quantitative data and qualitative inspection confirms that the implementation of periodic, full-image LPIPS calculations (Experiment C) is the most effective strategy for enhancing 3-D reconstructions of aerial data.
Discussion
The results of this study demonstrate that integrating perceptual loss functions into the NeRF training loop significantly improves the structural fidelity of 3-D reconstructions of UAV imagery. The quantitative performance of Experiment C—specifically in its SSIM (0.4860) and LPIPS (0.4808) scores—indicates that periodic, full-image perceptual evaluations provide a more effective optimization signal than traditional MSE alone. This section evaluates the technical implications of these findings regarding perceptual weighting and the role of data coverage.
Perceptual Weighting and Structural Consistency
A primary finding of this research is the inverse relationship between mathematical signal-to-noise ratios and structural coherence. While Experiment D produced the highest PSNR (22), it did not achieve the highest structural metrics. This suggests that optimizing pixel-level intensity (MSE) and optimizing spatial structure (LPIPS) are distinct objectives that occasionally conflict. In high-resolution outdoor environments like Sanford Stadium, the preservation of boundaries and physical features is more critical than exact color intensity matching. The implementation of a multi-objective loss function allows the model to prioritize the geometric relationships between pixels, which mitigates the blurring artifacts observed in MSE-only reconstructions.
Furthermore, the success of the full-image input scope over the patch-based method highlights the importance of global context. By downsampling the full image and applying the loss function across the entire frame, the model maintains a broader understanding of the scene’s geometry, leading to better preservation of long-range structural features.
Computational Efficiency and Edge Deployment
A critical factor for the deployment of 3-D reconstruction technology is the computational overhead required for training. Perceptual loss functions like LPIPS are computationally intensive because they require an auxiliary pass through a pre-trained convolutional neural network. The results from Experiment C prove that continuous LPIPS calculation is not necessary to improve resultant outputs. Experiments C and D instead perform LPIPS computation every ten iterations, reducing the computational overhead and balancing pixel and perceptual objectives effectively. The model still maintains structural integrity while significantly lowering the memory and processing requirements. By utilizing this frequency reduction, our findings suggest that full-image LPIPS tends to outperform patch-based LPIPS when applied at reduced frequency. This result may suggest that spatial coherence across the whole image is especially valuable in NeRF reconstruction and feature mapping with LPIPS.
Reducing frequency is particularly relevant for the deployment of these models on lightweight edge devices. Real-time monitoring of structural changes, such as erosion control, construction progress, or crop health, requires models that can be updated rapidly in the field. The finding that periodic perceptual feedback is sufficient for high-fidelity reconstruction suggests a promising direction for more efficient, perceptually aware neural rendering pipelines.
Limitations and Future Directions
While this research demonstrates the benefits of LPIPS integration, several limitations remain, such as the lack of robust data partitioning and edge effect detection. Additionally, while the results show clear qualitative improvements, human insight is still required to confirm image fidelity, as current numerical metrics do not perfectly mirror human visual perception.
Future work should involve the use of revised datasets with explicit, varied overlap settings to test the limits of the model’s reconstructive capabilities. Furthermore, exploring other perceptual metrics beyond LPIPS (e.g. Laplacian Variance, Gradient Magnitude Similarity Deviation), or developing a custom loss function specifically for UAV data, may yield even higher degrees of accuracy for professional mapping applications.
Conclusion
This research evaluates the integration of multi-objective loss functions to enhance the structural fidelity of UAV-based 3-D reconstructions using Neural Radiance Fields. The results indicate that while Mean Squared Error is effective for achieving pixel-level correspondence, it is insufficient for maintaining the structural coherence and edge sharpness required for accurate 3-D reconstructions. The experimental trials demonstrate that the implementation of periodic, full-image LPIPS calculations (as in Experiment C) provides the most effective optimization strategy, achieving significant qualitative improvements in boundary definition.
References
Berra, E. F., & Peppa, M. V. (2020). Advances and Challenges of UAV SFM MVS Photogrammetry and Remote Sensing: Short Review. 2020 IEEE Latin American GRSS & ISPRS Remote Sensing Conference (LAGIRS), 533–538. https://doi.org/10.1109/LAGIRS48042.2020.9285975
Du, K., Liao, X., Xia, J., Guo, C., Gu, Y., Guan, Y., Wang, D., ShengHuang, & Wang, Z. (2025). UAVLight: A Benchmark for Illumination-Robust 3-D Reconstruction in Unmanned Aerial Vehicle (UAV) Scenes (arXiv:2511.21565). arXiv. https://doi.org/10.48550/arXiv.2511.21565
Hermann, M., Kwak, H., Ruf, B., & Weinmann, M. (2024). Leveraging Neural Radiance Fields for Large-Scale 3-D Reconstruction from Aerial Imagery. Remote Sensing, 16(24), 4655. https://doi.org/10.3390/rs16244655
Kingma, D. P., & Ba, J. (2014b, December 22). Adam: A method for stochastic optimization. arXiv.org. https://arxiv.org/abs/1412.6980
Lan, X., Zhou, M., Wei, X., Yan, J., Huang, Y., Pu, H., Luo, J., & Jia, W. (2026). Image Quality Assessment: Exploring Quality Awareness via Memory-driven Distortion Patterns Matching (arXiv:2602.18000). arXiv. https://doi.org/10.48550/arXiv.2602.18000
Li, Z., Ma, W., Wu, T., Zhu, J., Q, Z., Zhang, S., Ou, J., Ren, Y., Qi, W., Shen, G., Xiong, H., & Zhao, W. (2025). Beyond a Single Light: A Large-Scale Aerial Dataset for Urban Scene Reconstruction Under Varying Illumination (arXiv:2512.14200). arXiv. https://doi.org/10.48550/arXiv.2512.14200
Long, M., Jiangbin, D., Jiayao, Z., Xuhao, W., & Yangfan, P. (2023). Large-field Astronomical Image Restoration and Superresolution Reconstruction using Deep Learning. Publications of the Astronomical Society of the Pacific, 135(1053), 1–7.
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2020, August 5). NERF in the Wild: Neural radiance fields for unconstrained photo collections. arXiv.org. https://arxiv.org/abs/2008.02268
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020, March 19). NERF: Representing scenes as neural radiance fields for view synthesis. arXiv.org. https://arxiv.org/abs/2003.08934
Nilsson, J., & Akenine-Möller, T. (2020). Understanding SSIM (arXiv:2006.13846). arXiv. https://doi.org/10.48550/arXiv.2006.13846
Peat, C., Batchelor, O., Green, R., & Atlas, J. (2022, November 22). Zero NeRF: Registration with Zero Overlap. arXiv.org. https://arxiv.org/abs/2211.12544
Remondino, F., Karami, A., Yan, Z., Mazzacca, G., Rigon, S., & Qin, R. (2023). A critical analysis of NERF-Based 3-D reconstruction. Remote Sensing, 15(14), 3585. https://doi.org/10.3390/rs15143585
Schönberger, J. L., Frahm, J.-M., University of North Carolina at Chapel Hill, & Eidgenössische Technische Hochschule Zürich. (2021). Structure-from-Motion revisited. University of North Carolina at Chapel Hill.
https://openaccess.thecvf.com/content_cvpr_2016/papers/Schonberger_Structure-Fro m-Motion_Revisited_CVPR_2016_paper.pdf
Szeliski, R. (2022). Computer Vision: Algorithms and applications (2nd ed.). Springer. Wang, C., Wu, X., Guo, Y., Zhang, S., Tai, Y., & Hu, S. (2022). NeRF-SR: High Quality Neural Radiance Fields using Supersampling. Proceedings of the 30th ACM International Conference on Multimedia, 6445–6454. https://doi.org/10.1145/3503161.3547808
Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612. https://doi.org/10.1109/tip.2003.819861
Xu, N., Qin, R., Huang, D., & Remondino, F. (2024). Multi‐tiling neural radiance field (NeRF)—geometric assessment on large‐scale aerial datasets. The Photogrammetric Record, 39(188), 718–740. https://doi.org/10.1111/phor.12498
Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2020, December 3). pixelNeRF: Neural Radiance Fields from One or Few Images. arXiv.org. https://arxiv.org/abs/2012.02190 Zhang, R., Isola, P., Efros, A., Shechtman, E., & Wang, O. (2018, June 1). The unreasonable ef ectiveness of deep features as a perceptual metric. IEEE Conference Publication | IEEE Xplore. https://ieeexplore.ieee.org/document/8578166
Zhang, X., Su, Y., Lu, H., Zhang, C., & Liu, Y. (2025). Rethinking SSIM-Based Optimization in Neural Field Training. 2025 IEEE Intelligent Vehicles Symposium (IV), 828–834. https://doi.org/10.1109/IV64158.2025.11097545
Acknowledgments: Special thanks to Dr. Nancy O’Hare for collecting UAV data over Sanford Stadium, handling airspace restrictions, guidance and reviews of earlier versions of this manuscript. Dr. Deepak Mishra provided guidance and oversight in developing this research during the Fall 2025 and Spring 2026 VIPR SSRL classes. Evan Hammam and Ali Batu Ozdener provided technical support with server use and running NeRF models.
Citation Style: APA