Reconstructing scenes across time, seasons, and conditions using Time Splatting.

Abstract

This project explores how to record, explore, and visualize long-term changes in an environment—at the scale of days, months, and even years—based on data that a single user can conveniently capture using the mobile phone they already carry. Our strategy involves making the data capture process as quick and convenient as possible so that it is easy to integrate into daily routines. This strategy yields large unstructured panoramic image datasets, which we process using novel registration and scene reconstruction approaches. Our central contribution lies in demonstrating pocket time-lapse as a novel application, made possible through several key technical contributions. These include a novel method for quickly and robustly registering thousands of unstructured panoramic images, a novel reconstruction technique for rendering time-lapse and performing state-of-the-art intrinsic image decom- position, and several large hand-captured datasets that span multiple years of data collection, totaling over 6k separate capture sessions and 50k images

Time Splatting Results: Interpolation Across Seasons

Video 1: Time Splatting through seasons on the Balcony 1 dataset successfully disentangles seasons and progress of construction from time of day.
Video 2: Time Splatting of a scene with construction.
Video 3: Time Splatting shows changes in leaves throughout the seasons.
Video 4: Another example of changing leaves through the seasons.
Video 5: Time Splatting through seasons on the Balcony 2 dataset successfully disentangles season from time of day.
Video 6: Another example of construction progress reconstructed with Time Splatting.
Video 7: Time Splatting is able to reconstruct water and heavy foliage as well.
Video 8: Time Splatting is able to reconstruct water and heavy foliage as well.

Time of Day Interpolation

Reconstructions (Left) and Shading Images (Right)

Video 9a: Timelapse of the baseball field dataset from sunrise to sunset over one day.
Video 9b: Shading component of the timelapse of the baseball field dataset from sunrise to sunset over one day.
Video 10a: Timelapse of the home garage dataset from sunrise to sunset over one day.
Video 10b: The shading component of the timelapse of the home garage dataset from sunrise to sunset over one day.
Video 11a: Timelapse of the glass building dataset from sunrise to sunset over one day.
Video 11b: The shading component of the timelapse of the glass building dataset from sunrise to sunset over one day.
Video 12a: Timelapse of the Balcony 1 dataset from sunrise to sunset over one day.
Video 12b: Shading component of the timelapse of the Balcony 1 dataset from sunrise to sunset over one day.

Linear Interpolation Results

Here we present results of linear interpolation on various datasets.

Video 13: Interpolation through seasons.
Video 14: Interpolation through seasons.
Video 15: Interpolation through construction.
Video 16: Melting snow.
Video 17: Interpolation through seasons.
Video 18: Interpolation through seasons.
Video 19: Mini-timelapse: Interpolation through time of day while keeping the date fixed.
Video 20: Interpolation through time of day.
Video 21: Interpolation through seasons.
Video 22: Interpolation through construction.
Video 23: Interpolation as tree petals bloom.
Video 24: Interpolation through construction.

Time Splatting and Linear Interpolation on a Traditional Time-Lapse

We compare Time Splatting and linear interpolation results on a traditional time-lapse. The sequence is taken from scene 10917 of the SkyFinder dataset [Mihail et al., 2016].

Because the traditional time lapse is sampled densely from a static camera, linear interpolation does a good job at reconstructing detail. However, unlike Time Splatting, it cannot disentangle lighting changes from seasonal changes. We demonstrate how Time Splatting is able to generate a sequence through seasons while maintaining constant lighting below.

Video 25: Time Splatting of a traditional time lapse with constant lighting.
Video 26: Linear interpolation of a traditional time lapse cannot disentangle lighting from seasonal changes.

Registration Comparison

Video 27: We perform a comparison between our alignment graph registration strategy and two baselines. All-to-one registration matches all captured sessions to one central node, and sequential registration registers each session to its immediate neighbors in time. All-to-one registration fails, due to its inability to account for dramatic changes in geometry during the captured construction. Sequential registration exhibits drift in registration accuracy, which wrongly causes the scene to be rotated 90 degrees. Our method on the other hand is able to register more captures successfully, with only two sessions returning no registration.