Maryland's HMO for Video Cameras

Newsletter of the Department of Electrical Engineering

The University of Maryland’s Health Maintenance Organization for Video Cameras

Pictured: A mosaic of shots taken of the University of Maryland’s Memorial Chapel and surrounding area. These images have been stitched together from a video sequence using the image sequence stabilization techniques developed by Chellappa’s research group.

Most video cameras have health-related problems of one sort or another. Many have “the shakes.” Others have poor eyesight. Some suffer from tunnel vision.

Prof. Rama Chellappa’s research group in the Department of Electrical Engineering and the Center for Automation Research offers a comprehensive Health Maintenance Organization package for ailing video cameras–along with fast, efficient, and cost-effective treatments.

Patient Case #1 - Too Much Caffeine
Video cameras are shaky, especially when used either as hand-held devices or as devices mounted on unmanned ground and air vehicles. Several commercial technologies have been established to compensate for video stabilization, including Sony’s new “stable shot” technology (which is limited to small movements), and mechanical gyroscope systems (which are often bulky and costly).

Chellappa’s research group uses Electronic Digital Image Stabilization (EDIS) to treat tremulous video data. EDIS features a lower cost and greater flexibility than other shake-reducing methods, even giving the user “on-demand stabilization,” where the user specifies which component of video motion he or she wants to be stabilized.

“What we do is find the shift between two images and compensate for it using algorithms,” Chellappa explained. “But video is real-time, so the algorithms we use have to be fast. What’s more,” he explained, “these algorithms can be tailored to the user in terms of how much speed is needed.”

Chellappa’s EDIS system uses an image registration algorithm to compute the transformation between two image frames and minimize their spatial misalignment (assuming that the scene is rigid and the movement of the camera causes the misalignment). The EDIS system estimates the motion of the camera by modeling the motion of the brightness patterns in the image. The images are then aligned by compensating for the motion of the camera.
In many cases, the EDIS system can divide an image into regions describing different motion fields, so the system can select one of them to stabilize. In the output sequence created by the EDIS system, the selected region appears to be at the same position throughout each frame. Unless otherwise specified by the user, the largest or dominant background region is chosen and stabilized.

EDIS can be used for image analysis, or simply as a visualization tool. It could be used to tele-operate robotic vehicles, or to track independently moving objects from mobile platforms.

Patient Case #2 - Tunnel Vision
Video cameras capture data through only one tiny portal, limiting the breadth of data that can be viewed in any single frame. Chellappa’s research group has found a way to take the image data from a video camera that has panned right or left over a scene, whether zooming or fading, and tie these images together so that one larger scene or panoramic view is created.

“Looking through a video camera is like looking through a straw,” Chellappa explained. “We take images gathered from a camera and stitch them together to form a mosaic.” During this process, called Mosaicking, motion estimation techniques used for image sequence stabilization are used to create mosaic images. Simply put, this is done by superimposing the overlapping regions of multiple images sampled from a video camera. These overlapping regions are calculated using multi-resolution feature-based motion estimation schemes with subpixel precision. Motion-compensated successive frames are then aligned and superimposed to create a larger mosaic. This method is designed to be fast, robust, and accurate. It is designed for real-world, real-time video applications–such as surveillance.

Patient Case #3 - Better Glasses
Video images can be fuzzy, especially when a camera is in motion. Fuzziness can occur when a camera is tracking a moving object, or just panning to the left or right. Maryland researchers have come up with a method of creating “super-resolution,” or increasing a camera’s resolving power through image processing.
“When you’re taking video,” said Chellappa, “there is a lot of overlap between successive frames. You get to look at some points more than once. We take this information and make the video look better.”

“Each point or pixel is made of lower-resolution grids,” Chellappa explained. “We examine this information, look for repeating sets and define a final, higher-resolution grid.”

Many methods of image processing used to increase resolution are slow, or sacrifice quality for speed. But Chellappa’s group has developed a method of super-resolution that is accurate and fast. In many circumstances, a computer goes through many sets of algorithms when sampling image data to determine which is the best to use, and only then does it actually run the algorithm. But Chellappa’s group utilizes a closed-form solution, featuring a predetermined algorithm that yields quality results very quickly. This efficiency is important for real time video applications.

Super-resolution can be used to take images shot from two or more different cameras and fuse them together into one better, higher-resolution picture. This technique mimics the way the human eye focuses on objects and scenes.

“The human eyes practice super-resolution,” said Dr. Hassan Shekarforoush, who developed the current super-resolution scheme used by Chellappa’s group. “Both eyes,” he explained, “are constantly jittering, sampling images and resolving them at a super resolution in real time.”
Super-resolution can be used for the visual surveillance of both human and vehicular activities, whether in battlefield or civilian applications. Using this technology, a computer could monitor a parking lot and notify a guard or the appropriate authorities when it determines that there is suspicious activity underway.

Multiple Treatments
Many of the treatments described above can be used together, or in succession, for superior results. For example, if someone were trying to track an object with a video camera, EDIS could be used to stabilize the image, super-resolve it (to track an object), and stitch together several shots to view a larger area.

Members of Chellappa’s group who have contributed to the efforts described include: Dr. Phillipe Burlina, Dr. Carlos Morimoto, Dr. Yi-Shing Yao, Mr. Shridhar Srinivasan, Ms. Lydia Ng and Mr. Steve Balakirsky.
The work reported here is supported by the DARPA Image Understanding for Battlefield Awareness Program, and the Office of Naval Research MURI Program.
More information about Chellappa’s video camera HMO can be found at http://www.cfar.umd.edu./~rama/.