Fourier Slice Photography Ren Ng Stanford University Conventional Photograph Light Field Photography Capture the light field inside the camera body

Hand-Held Light Field Camera Medium format digital camera Camera in-use 16 megapixel sensor Microlens array

Light Field in a Single Exposure Light Field in a Single Exposure Light Field Inside the Camera Body Digital Refocusing

Digital Refocusing Questions About Digital Refocusing What is the computational complexity? Are there efficient algorithms?

What are the limits on refocusing? How far can we move the focal plane? Overview Fourier Slice Photography Theorem

Fourier Refocusing Algorithm Theoretical Limits of Refocusing Previous Work

Integral photography Lippmann 1908, Ives 1930 Lots of variants, especially in 3D TV Okoshi 1976, Javidi & Okano 2002

Closest variant is plenoptic camera Adelson & Wang 1992 Fourier analysis of light fields Chai et al. 2000

Refocusing from light fields Isaksen et al. 2000, Stewart et al. 2003 Fourier Slice Photography Theorem In the Fourier domain, a photograph is a

2D slice in the 4D light field. Photographs focused at different depths correspond to 2D slices at different trajectories. Digital Refocusing by RayTracing x u

Lens Sensor Digital Refocusing by RayTracing x u

Imaginary film Lens Sensor Digital Refocusing by RayTracing x

u Imaginary film Lens Sensor Digital Refocusing by RayTracing x

u Imaginary film Lens Sensor Digital Refocusing by RayTracing x

u Imaginary film Lens Sensor Refocusing as Integral Projection

u x x u Imaginary film Lens

Sensor Refocusing as Integral Projection u x x u

Imaginary film Lens Sensor Refocusing as Integral Projection u x

x u Imaginary film Lens Sensor

Refocusing as Integral Projection u x x u Imaginary film

Lens Sensor Classical Fourier Slice Theorem Integral Projection 1D Fourier

Transform 2D Fourier Transform Slicing Classical Fourier Slice Theorem Integral

Projection 1D Fourier Transform 2D Fourier Transform Slicing

Classical Fourier Slice Theorem Integral Projection 1D Fourier Transform 2D Fourier Transform

Slicing Classical Fourier Slice Theorem Integral Projection Spatial Domain Fourier Domain

Slicing Classical Fourier Slice Theorem Integral Projection Spatial Domain Fourier Domain

Slicing Fourier Slice Photography Theorem Integral Projection Spatial Domain Fourier Domain

Slicing Fourier Slice Photography Theorem Integral Projection 4D Fourier Transform

Slicing Fourier Slice Photography Theorem Integral Projection 2D Fourier Transform

4D Fourier Transform Slicing Fourier Slice Photography Theorem Integral Projection

2D Fourier Transform 4D Fourier Transform Slicing Fourier Slice Photography Theorem

Integral Projection 2D Fourier Transform 4D Fourier Transform Slicing

Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing

Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing Theorem Limitations Film parallel to lens

Everyday camera, not view camera Aperture fully open Closing aperture requires spatial mask Overview

Fourier Slice Photography Theorem Fourier Refocusing Algorithm

Theoretical Limits of Refocusing Existing Refocusing Algorithms Existing refocusing algorithms are expensive O(N4)

All are variants on integral projection where light field has N samples in each dimension

Isaksen Vaish Levoy Ng et et et et

al. al. al. al. 2000 2004 2004 2005

Refocusing in Spatial Domain Integral Projection 2D Fourier Transform 4D Fourier

Transform Slicing Refocusing in Fourier Domain Integral Projection Inverse

2D Fourier Transform 4D Fourier Transform Slicing Refocusing in Fourier Domain

Integral Projection Inverse 2D Fourier Transform 4D Fourier Transform

Slicing Asymptotic Performance Fourier-domain slicing algorithm Pre-process: O(N4 log N)

Refocusing: O(N2 log N) Spatial-domain integration algorithm

Refocusing: O(N4) Resampling Filter Choice Triangle filter (quadrilinear) Kaiser-Bessel filter

(width 2.5) Gold standard (spatial integration) Overview Fourier Slice Photography Theorem

Fourier Refocusing Algorithm Theoretical Limits of Refocusing Problem Statement Assume a light field camera with

An f /A lens N x N pixels under each microlens If we compute refocused photographs from these light fields, over what range

can we move the focal plane? Analytical assumption Assume band-limited light fields Band-Limited Analysis Band-Limited Analysis

Band-width of measured light field Light field shot with camera Band-Limited Analysis Band-Limited Analysis

Band-Limited Analysis Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing Results of Band-Limited Analysis Assume a light field camera with

An f /A lens N x N pixels under each microlens From its light fields we can

Refocus exactly within depth of field of an f /(A N) lens In our prototype camera Lens is f /4

12 x 12 pixels under each microlens Theoretically refocus within depth of field of an f/48 lens Light Field Photo Gallery Stanford Quad Rodins Burghers of Calais

Palace of Fine Arts, San Francisco Palace of Fine Arts, San Francisco Waiting to Race Start of the Race

Summary of Main Contributions Formal theorem about relationship between light fields and photographs

Computational application gives asymptotically fast refocusing algorithm Theoretical application gives analytic solution for limits of refocusing Future Work

Apply general signal-processing techniques Cross-fertilization with medical imaging Thanks and Acknowledgments

Collaborators on camera tech report Marc Levoy, Mathieu Brdif, Gene Duval, Mark Horowitz and Pat Hanrahan Readers and listeners Ravi Ramamoorthi, Brian Curless, Kayvon Fatahalian, Dwight Nishimura,

Brad Osgood, Mike Cammarano, Vaibhav Vaish, Billy Chen, Gaurav Garg, Jeff Klingner Anonymous SIGGRAPH reviewers Funding sources

NSF, Microsoft Research Fellowship, Stanford Birdseed Grant Questions? Start of the race, Stanford University Avery Pool, July 2005