Synchronization and Rolling Shutter Compensation for Consumer Video Camera Arrays

Derek Bradley, Bradley Atcheson, Ivo Ihrke, Wolfgang Heidrich

IEEE International Workshop on Projector-Camera Systems (PROCAMS 2009)
2nd Best Paper Award!


Download:    PDF

Bibtex:
@INPROCEEDINGS{Bradley:2009,
   author = {Derek Bradley and Bradley Atcheson and Ivo Ihrke and Wolfgang Heidrich},
   title = {Synchronization and Rolling Shutter Compensation for Consumer Video Camera Arrays},
   journal = {International Workshop on Projector-Camera Systems (PROCAMS 2009)},
   year = {2009},
}


Abstract

Two major obstacles to the use of consumer camcorders in computer vision applications are the lack of synchronization hardware, and the use of a "rolling" shutter, which introduces a temporal shear in the video volume. We present two simple approaches for solving both the rolling shutter shear and the synchronization problem at the same time. The first approach is based on strobe illumination, while the second employs a subframe warp along optical flow vectors. In our experiments we have used the proposed methods to effectively remove temporal shear, and synchronize up to 16 consumer-grade camcorders in multiple geometric configurations.

Rolling Shutter Camera Model


In the rolling shutter camera model, just-in-time exposure and readout of the individual scanlines creates a shear of the exposure intervals along the time axis. The slope of this shear is a function of the camera frame rate and the period is determined by the number of scanlines in the video format. A fast rotating checkerboard captured with a rolling shutter camera has straight lines that are warped into curves.

Method 1: Stroboscope Illumination

Stroboscopes create instantaneous illumination for all scanlines of all cameras.
In rolling shutter cameras, consecutive frames that contain the instantly exposed scanlines are combined to make the final image.


Increasing the flash duration creates a virtual exposure time. The exposed scanlines overlap with a ramp up at the begining and ramp down at the end. Summing the frames in linear space creates the final image.


A non-continuous camera exposure results in scanlines with less than full intensity after the summation.

Method 2: Subframe Warping

Our subframe warping method removes the rolling shutter distortion and synchronizes multiple cameras by interpolating intermediate frames. We compute optical flow between adjacent frames, and then perform the interpolation using different offsets for each scanline.

    
Original (left) and corrected (right) frame from a handheld panning sequence. The red line shows how the vertical wall is displaced by as much as 8 pixels in the uncorrected version.

Experiments

Stroboscope synchronization experiment. Left: 3 consecutive frames (left to right) from 2 unsynchronized cameras (top and bottom). Right: 3 consecutive frames (left to right) from 2 cameras (top and bottom) synchronized by strobe lighting.


           
Subframe warping synchronization. Left: two consecutive overlaid fields from first camera. Center: closest integer frame aligned field from second camera. Right: warped field from first camera, synchronized to match the second.


Top left: a fast rotating checkerboard (1 rev. per second) in which straight lines are warped into curves. Top right: after rolling shutter correction by subframe warping. Bottom left: slower motion (half the speed) still produces noticeable distortion. Bottom right: the same scene captured with strobe illumination.


Noise vs. motion blur trade-off. A ball with fast constant rotation and a stationary checkerboard. Top left: short virtual exposure. Top right: long virtual exposure. Bottom row: zoom regions show noise in the short virtual exposure and motion blur in the long virtual exposure. The noise in the zoom regions is amplified by a factor of 2 for better visualization.