top of page
3D Photo-Realistic Scene Rendering Using NeRF


Role: Summer Research Intern
Date: June 2023 - July 2023
A solution pipeline that implements fine-tuned cutting-edge Neural Radiance Field (NeRF) machine learning algorithms to render 3D photo-realistic scenes from limited image data of various data modalities.
So what was I trying to do?
How Do We Go From These 2D Images...



To this?

The solution lies in NeRF!
But what is NeRF?
Neural Radiance Fields (NeRF) are machine learning algorithms that compose of a fully connected network that takes as input a 5d camera pose (3d location and 2d viewing angle) and outputs a volume density and view dependent rgb value at that spatial location.

The views are synthesized by querying 5d camera poses and using volume rendering techniques to project their corresponding color and densities. The NeRF scene representation or network can be optimized to accurately render photorealistic scenes with limited image data and complex geometry.

But how do we generate accurate camera poses to begin with?
I used COLMAP, a tool that serves as a general purpose, end to end image based 3D reconstruction pipeline. COLMAP relies on its Structure From Motion (SFM) process to transform pixel coordinates into their world coordinate frame to capture each image's corresponding camera pose.


Structure From Motion (SFM) is typically used to estimate the 3D structure of a scene from 2D images. To do so, SFM relies on feature detection and matching algorithms to find key points that are common amongst the images. Properly matched keypoints allow for SFM to determine each image's camera pose with respect to one another.


So what sort of NeRF algorithms did I experiment with?
I used NeRF Studio, an API that provides access to various NeRF algorithms for training and testing. I specifically experimented rendering scenes using Nvidia's Instant-NGP method, NeRF Studio's recommended Nerfacto method, and its more complex version, Nerfacto-Big.

Through NeRF Studio's built-in visualizer and access to MIT Lincoln Lab's superclusters, I learned how to fine-tune my NeRF models, monitor my model's training process in real time, and optimize my workflow and model performance by taking advantage of using multiple compute nodes.

Ultimately, I see NeRF as the future of creating high-quality 3D scenes and objects. From real-time rendering to more efficient data storage, NeRF will (if it hasn't already) revolutionize augmented reality, virtual reality, media, autonomous vehicles, etc...
Takeaways from my experience working at MIT Lincoln Laboratory?
Working at Lincoln Labs gave me the perfect balance between an industry and academia setting. While still expected to finish work deliverables, I also had time to learn and grow my knowledge in Computer Vision, Photogrammetry, and Neural Radiance Fields. My team, supervisor, mentor, and fellow summer interns were all very collaborative and supportive; whether it be helping me tackle dependency issues, think through my code, or even spend quality time at group happy hours, they helped me grow as a person both professionally and personally.
** Please note all code is proprietary to MIT Lincoln Labs and therefore not publicly accessible **

bottom of page