Multimodal sensors digital interfaces
1 / 22

Multimodal sensors digital interfaces - PowerPoint PPT Presentation

  • Updated On :

Multimodal sensors & digital interfaces . Credits. The original Multimodal Project developed by and credit for: Zhigang Zhu and Weihong Li (Integration of Laser Vibrometry with Infrared Video for Multimedia Surveillance Display). Outline. Multimodal System Overview Multimedia Sensors

Related searches for Multimodal sensors digital interfaces

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Multimodal sensors digital interfaces' - joben

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Credits l.jpg

  • The original Multimodal Project developed by and credit for:

  • Zhigang Zhu and Weihong Li (Integration of Laser Vibrometry with Infrared Video for Multimedia Surveillance Display)

Outline l.jpg

  • Multimodal System Overview

  • Multimedia Sensors

    • Infrared camera

    • The LDV sensor

    • PTZ camera

  • Multimodal System Components

  • System Design Concept

  • Design Issues

  • Integration Issues

Multimodal system overview l.jpg
Multimodal System Overview

  • The object of this system is to provide a multimodal integration of audio, visible, thermal for human signature detection.

  • The goal is to use the sensing technologies for perimeter surveillance.

    • Sensors, alarm, response.

  • Multimodal system interface

    • The environment, the sensors, and the events.

Multimedia sensors infrared camera l.jpg
Multimedia Sensors – Infrared Camera

  • Infrared camera

    • FLIR ThermoVision A40M IR camera

    • Temp Range of -20º to 500ºC, accuracy (% of reading) ± 2ºC or ± 2%

    • 320x240 Focal Plane Array

    • 24º FOV Lens

    • Firewire Output IEEE 1394

    • Video output – RS170 EIA/NTSC or CCIR/PAL composite video for monitoring on a TV screen

    • ThermoVision System Developers Kit (C++)

    • Each thermal image is built from 76,800 individual picture elements that are sampled 60 times per second by the camera's on-board electronics.

Multimedia sensors infrared camera6 l.jpg
Multimedia Sensors – Infrared Camera

  • Samples

Figure 1. A person sitting in dark room can be clearly seen in the IR image. The temperature at Sp1 on the face is 33.1ºC

Figure 2. Two IR images before and after a person standing at about 200 feet. The reading of the temperature at Sp1 changes from 11C to 27C.

Multimedia sensors infrared camera7 l.jpg
Multimedia Sensors – Infrared Camera

  • Thermographic measurement techniques

  • An infrared camera measures and images the emitted infrared radiation from an object.

  • The radiation measured by the camera does not only depend on the temperature of the objects but is also a function of the emissivity.

  • Radiation also originates from the surroundings and is reflected in the object

  • Radiation from the object and reflected radiation will also be influenced by the absorption of the atmosphere

  • Parameters need to take care:

    • The emissivity of the object

    • The reflected temperature

    • The distance between the object and the camera

    • The relative humidity

Multimedia sensors infrared camera8 l.jpg
Multimedia Sensors – Infrared Camera

  • Emissivity:

  • How much radiation is emitted from the object

  • Object materials and surface treatments exhibit emissivity ranging from approximately 0.1 – 0.95

  • Highly polished (mirror) surface < 0.1

  • Human skin exhibits an emissivity close to 1

  • Metal: low, increase with temperature

  • Non-metal: high, decrease with temperature

Multimedia sensors infrared camera9 l.jpg
Multimedia Sensors – Infrared Camera

  • Reflected ambient temperature:

  • To compensate for the radiation reflected in the object and the radiation emitted from the atmosphere between the camera and the object.

  • If the emissivity is low, the distance very long and the object temperature relatively close to that of the ambient it will be important to set and compensated for the ambient.

  • Distance

  • The distance between object and the front lens of the camera.

  • Relative Humidity

  • Normally, default 50%

Multimedia sensors infrared camera10 l.jpg
Multimedia Sensors – Infrared Camera

  • History of Infrared Technology

  • Sir William Herschel (1738-1822)

    • Discover of infrared spectrum

  • Marsilio Landriani (1746-1815)

    • As the blackened thermometer was moved slowly along the colors of the spectrum, the temperature readings showed a steady increase from the violet end to the red end.

  • Macedonio Melloni (1798-1854)

    • Rock salt (NaCl) (to be made into lenses and prisms) is remarkably transparent to the infrared.

  • Sir John Herschel

    • The first ‘heat-picture’ in 1840, thermograph

  • Samuel P. Langley (1834-1906)

    • Inventor of the bolometer (1880)

Multimedia sensors ldv sensor l.jpg
Multimedia Sensors – LDV sensor

  • Vibrometer types:

  • Single Point Vibrometers:

    • Measure the vibration of an object in the direction of laser beam

  • Differential Vibrometers: (dual beam)

    • Allow vibration measurement between two points vibrating relative to each other.

  • Rotational Vibrometers:

    • Measure angular vibrations on rotating structures.

  • In-plane Vibrometers:

    • measure continuous (DC) velocity and superimposed variable (AC) components perpendicular to the central axis of two converging laser beams.

  • 3D Vibrometers

Multimedia sensors ldv sensor12 l.jpg
Multimedia Sensors – LDV sensor

  • Laser Doppler Vibrometer (LDV)

  • Optical instruments for accurately measuring velocity and displacement of vibrating structures completely without contact.

  • Sensor head OFV-505

    • HeNe laser,  = 633.8 nm.

    • OFV-SLR lens (f=30mm) 1.8m – 200+m, auto focus

  • Controller OFV-5000 with a digital velocity decode card VD-6

    • RS232 interface for computer control

  • Telescope VIB-A-P05

    • ±1º vertical tilt and ± 1.5º horizontal tilt

Multimedia sensors ldv sensor13 l.jpg
Multimedia Sensors – LDV sensor

  • Measurement Principle

S is the light source

f is frequency

P is the moving with velocity v and reflects the light

O is the receiver (f + fD)

Resultant frequency shift

For vibrometers: S=O ("backscatter") 1= - 2, therefore,

Multimedia sensors ldv sensor14 l.jpg
Multimedia Sensors – LDV sensor

  • LDV schema

  • Velocity is directly obtained by demodulation:  = 2fm

  • Voice frequency f: 300Hz ~ 3000Hz

  • LDV can detect vibration at a magnitude as low as m = /2f = 1/(2*3.14*300) = 0.5µm

Multimedia sensors ptz camera l.jpg
Multimedia Sensors – PTZ camera

  • Pan/tilt/zoom (PTZ) camera

  • Human and other target detection at a large distance

  • Canon PTZ

  • 26X optical zoom lens & 12X digital zoom

  • Pan: ±100º, Tilt: +90º/-30º

  • Built-in IR light (effective up to 9 feet)

  • BNC video output

  • RS-232 computer control interface

Multimedia sensors ptz camera16 l.jpg
Multimedia Sensors – PTZ camera

  • PTZ Samples

Two images of a person at a distance of about 200 feet, captured by changing the zoom factors of the PTZ camera.

Multimodal system components l.jpg
Multimodal System Components

  • Three components

    • The IR/EO imaging video surveillance component

      • Human motion tracking, human face detection

      • Thermal Camera for daytime and nighttime

      • Visible camcorder (sony), Web cam (logitech)

    • The LDV audio surveillance component

      • Audio signal capture, voice recognition

    • The human-computer interaction component

      • Cognitive understanding of the environment, the sensors, and the events

System design concept l.jpg
System Design Concept

  • The overall goal the project is to design a human computer interface for human-centered multimodal (MM) surveillance.

Design issues l.jpg
Design Issues

  • Issues need to be considered

    • how to use EO camera tracking human motion;

    • how to incorporate IR imaging with existing EO captured image;

    • how to use IR imaging to help the laser Doppler vibrometer to select the appropriate targets;

    • how to select optimal viewpoint from audio detection.

Integration issues l.jpg
Integration Issues

  • Target detection and localization via IR/EO imaging

    • Set up an IR/EO imaging system with an IR camera and a PTZ camera for finding vibration targets for LDV listening

  • Registration between the IR/EO imaging system and the LDV system.

    • Two types of sensors need to be precisely aligned so that we can point the laser beam of the LDV to the target that the IR/EO imaging system has detected

  • Future research on automated targeting and focusing.

References l.jpg

  • Main multimodal system technical report


  • Polytec Laser Vibrometer


  • FLIR Systems Security ThermoVision Cameras


  • Paper:

    • Z Zhu, W Li, “Integrating LDV Audio and IR Video for Remote Multimodal Surveillance”

  • Others…