top of page

3D Photo-Realistic Scene Rendering Using NeRF

NeRFMedia.jpg
ExternalLinkArrow_edited.png

Role: Summer Research Intern

Date: June 2023 - July 2023

A solution pipeline that implements fine-tuned cutting-edge Neural Radiance Field (NeRF) machine learning algorithms to render 3D photo-realistic scenes from limited image data of various data modalities.

So what was I trying to do?

How Do We Go From These 2D Images...

fern_3.JPG
fern_2.JPG
fern_1.JPG

To this?

The solution lies in NeRF!

But what is NeRF?

Neural Radiance Fields (NeRF) are machine learning algorithms that compose of a fully connected network that takes as input a 5d camera pose (3d location and 2d viewing angle) and outputs a volume density and view dependent rgb value at that spatial location. 

The views are synthesized by querying 5d camera poses and using volume rendering techniques to project their corresponding color and densities. The NeRF scene representation or network can be optimized to accurately render photorealistic scenes with limited image data and complex geometry

ExternalLinkArrow_edited.png

But how do we generate accurate camera poses to begin with?

I used COLMAP, a tool that serves as a general purpose, end to end image based 3D reconstruction pipeline. COLMAP relies on its Structure From Motion (SFM) process to transform pixel coordinates into their world coordinate frame to capture each image's corresponding camera pose. 

COLMAP_1.webp
ExternalLinkArrow_edited.png

Structure From Motion (SFM) is typically used to estimate the 3D structure of a scene from 2D images. To do so, SFM relies on feature detection and matching algorithms to find key points that are common amongst the images. Properly matched keypoints allow for SFM to determine each image's camera pose with respect to one another.  

COLMAP_2.png
ExternalLinkArrow_edited.png

So what sort of NeRF algorithms did I experiment with?

I used NeRF Studio, an API that provides access to various NeRF algorithms for training and testing. I specifically experimented rendering scenes using Nvidia's Instant-NGP method, NeRF Studio's recommended Nerfacto method, and its more complex version, Nerfacto-Big

nerfstudio_img.png

Through NeRF Studio's built-in visualizer and access to MIT Lincoln Lab's superclusters, I learned how to fine-tune my NeRF models, monitor my model's training process in real time, and optimize my workflow and model performance by taking advantage of using multiple compute nodes. 

ExternalLinkArrow_edited.png

Ultimately, I see NeRF as the future of creating high-quality 3D scenes and objects. From real-time rendering to more efficient data storage, NeRF will (if it hasn't already) revolutionize augmented reality, virtual reality, media, autonomous vehicles, etc...

Takeaways from my experience working at MIT Lincoln Laboratory? 

Working at Lincoln Labs gave me the perfect balance between an industry and academia setting. While still expected to finish work deliverables, I also had time to learn and grow my knowledge in Computer Vision, Photogrammetry, and Neural Radiance Fields. My team, supervisor, mentor, and fellow summer interns were all very collaborative and supportive; whether it be helping me tackle dependency issues, think through my code, or even spend quality time at group happy hours, they helped me grow as a person both professionally and personally

** Please note all code is proprietary to MIT Lincoln Labs and therefore not publicly accessible **

AEnB2Ur5juO1svp3OGX2fnqA1bsA2TWa04sgeF-7NF4k9fCx4z3cF7Yk6FgXOIE0PpuOy9L-52eXmzNwKtwe9CIvx-

Yuvanshu Agarwal @2023

bottom of page