Event

The first part of the  dissertation addresses the synchronization of video/audio signals. We propose a new modality for audio and video synchronization based on the electric network frequency (ENF) signal naturally embedded in multimedia recordings. Synchronization of audio and video is achieved by aligning the ENF signals. The proposed method is a very different approach to tackling the audio/video synchronization problem from existing work, and offers a strong potential to address such difficult scenarios that are otherwise intractable.

Estimation of the ENF signal from video is a challenging task and has not been well addressed previously. To solve the problem of insufficient sampling rate of video, we propose to exploit the rolling shutter mechanism commonly adopted in CMOS camera sensors. Several techniques are designed to alleviate the distortions of motions and brightness changes in videos for ENF estimation.

We also address several challenges that are unique to the synchronization of digitized analog audio recordings. Speed offset often occurs in digitized analog audio recordings due to the inconsistency in the tape rolling speed. We show that the ENF signal captured by the original analog audio recording can be retained in the digitized version. The ENF signal is approximately considered as a single-tone signal and used as a reference to detect and correct speed offsets automatically.

The last part of the dissertation examines the quality assessment of local image features for efficient and robust spatial alignment. We propose a scheme to evaluate the quality of SIFT features in terms of their robustness and discriminability. A quality score is assigned to every SIFT feature based on its contrast value, scale and descriptor, using a quality metric kernel that is obtained in a one-time training phase. Feature selection is carried out by retaining features with high quality scores. The proposed approach is also applicable to other local image features such as the as the Speeded Up Robust Features (SURF).