AUTOMATED VIDEO EXPLOITATION

Full-motion video has inherent advantages over still imagery for characterizing events and movement. Military and intelligence analysts currently view live video imagery from airborne and ground-based video platforms, but few tools are available for further exploitation of these streams of information. Consumers of video need object- and event-level indexing of video and its accompanying metadata, to be able to efficiently interact with and exploit streams of video.

SRI'S MVIEWS SYSTEM

SRI has developed a Modular Video Imagery Exploitation Work Station (MVIEWS), a demonstration system for annotating, indexing, extracting, and disseminating information from video streams for surveillance and intelligence applications. With MVIEWS , a single operator can view and annotate live video data. Aided by a set of intelligent software agents and automated tools, the operator can rapidly create and forward multimedia messages and reports. The MVIEWS demonstration system includes:

• A multimodal user interface for annotating the video via talking, pointing, and drawing
  
  
• Automatic detection and tracking of moving objects, such as vehicles in live video
  
  
• A map display showing the georeferenced positions of moving objects updated in real time
  
  
• Wireless transmission of the extracted information to remote PDAs
  
  
• Storage and replay of the video, indexed by the video events and associated multimedia streams.

MVIEWS Functions

A key feature of MVIEWS is the multimodal user interface, which accepts voice and pen input. Talking, pointing, and drawing are very natural ways for humans to convey information. These modes of communication are especially valuable for describing the contents of video. Pointing and drawing are ideal for specifying locations, paths, and spatial relationships. In combination with speech, descriptions of complex concepts can be formulated very efficiently. For example, an operator can circle an object in the video and identify it ("report T-72 tank"), and then draw a path ("headed in this direction"). The identity of the detected object, its location, and its probable path can be immediately dispatched in a message. A still image chip or short MPEG video clip of the object can also be extracted from the video. The MVIEWS system automatically records the operator's verbal comments and drawings on the video image, and associates them with specific frames in the video sequence. The verbal and drawing annotations are archived and can be accessed and replayed in a later phase of analysis.

Live Video Stream Presented to Operator for Annotation and Exploitation

Recognized Speech for Report Generation and Command Interpretation

Map of Surveillance Area

Timeline Enables Video Replay and Browsing Using Multimedia Indexing

SRI COMPONENT TECHNOLOGIES

The MVIEWS system is based on a number of SRI-developed technologies:

• Speech recognition--SRI's continuous, speaker-independent speech recognition toolkit (marketed by Nuance Communications) recognizes commands to the MVIEW system and key words in the analyst's description of the video. Based on SRI's research in speech technology, the toolkit can handle up to 40,000 words. The MVIEWS system also incorporates natural language understanding.
  
  
• Gesture Recognition--Understanding of hand-drawn gestures (for example, an arrow indicating the direction of movement, or a circle delineating the boundaries of an object or region) were developed as part of other multimodal user interfaces.
  
  
• Motion Detection and Tracking--MVIEWS uses two types of algorithms: a pixel-based method for change detection and a tracking method based on dynamically updated image structure models. Current SRI research efforts in this area include:
  
  
  • automated target recognition and automated surveillance
  • Small Vision Module
  • Extra Set of Eyes
    
    
• Distributed Agent Infrastructure--SRI's Open Agent Architecture, a framework for the interoperation of software agents, was used to integrate the MVIEW software components. This infrastructure permits the rapid integration of off-the-shelf software modules and supports the dynamic integration of distributed resources in a plug-and-play fashion.

APPLICATIONS

MVIEWS is a video exploitation testbed that can be adapted for a wide variety of surveillance, monitoring, and intelligence applications such as:

• Control, tasking, and management of multiple video sensors and mobile sensor platforms to provide information for situational awareness
  
  
• Automated target detection, tracking, and real-time geolocation
  
  
• Authoring of multimedia reports containing video-derived imagery and collateral data (such as maps or site models) for mission planning and damage assessment
  
  
• Analysis of surveillance video data recorded near a bombing site immediately preceeding the incident
  
  
• Real-time collaboration among multiple analysts who jointly view and annotate a video sequence for threat assessment

FOR ADDITIONAL INFORMATION, CONTACT:

Mr. Eric C. Rickard, SRI International,

1611 N. Kent St., Arlington, VA 22209

Voice: (703) 247-8519 Fax: (703) 247-8537

Email: rickard@wdc.sri.com