visicast 2002 technical audit n.
Skip this Video
Loading SlideShow in 5 Seconds..
ViSiCAST 2002 Technical Audit PowerPoint Presentation
Download Presentation
ViSiCAST 2002 Technical Audit

Loading in 2 Seconds...

play fullscreen
1 / 70

ViSiCAST 2002 Technical Audit - PowerPoint PPT Presentation

  • Uploaded on

ViSiCAST 2002 Technical Audit. 4 October 2002, Brussels Michele Wakefield - Project Manager, ITC. The ViSiCAST Project. Vi rtual Si gning C apture A nimation S torage and T ransmission. Aims of ViSiCAST Project.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'ViSiCAST 2002 Technical Audit' - chars

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
visicast 2002 technical audit
ViSiCAST 2002 Technical Audit

4 October 2002, Brussels

Michele Wakefield - Project Manager, ITC

the visicast project
The ViSiCAST Project




Animation Storage and


aims of visicast project
Aims of ViSiCAST Project

“…support improved access by deaf citizens to information and services in sign language”

  • by successfully developing signing systems for
    • broadcast, WWW & ‘over the counter’ type applications
    • user friendly methods to capture & generate signs
    • machine readable system to describe gestures
    • ... preferred medium is sign language


voor Doven



University of East Anglia





The Post Office

Institut National des


Royal National

Institute for

Deaf People

Institut für


ViSiCAST Consortium

project dimensions
Project Dimensions
  • Duration
    • Start : January 2000
    • Finish : December 2002
    • 36 months
  • Total Costs
    • 3770kECU total
    • 2876kECU funding from EC
visicast project highlights
ViSiCAST Project Highlights
  • Signing transmissions demonstrated at IBC 2002
    • MPEG-4 compliant INT-IRT demonstrator to deliver an open signing service for broadcast DTV
  • BBC demonstrator to deliver closed DTV signing service
  • Translate simple sentences in real time to sign animation
  • WWW Weather-forecaster launched in the Netherlands
  • Interactive sign language learning tool
  • 2nd trial of TESSA system now nationwide and RNID re-promoting ViSiCAST
    • after success of pilot at Science Museum, London
    • encouraging national media coverage








ViSiCAST Project Structure


User Application

Exploitation &Dissemination

presentations by core streams
Presentations by Core Streams
  • Technology: Animation & Linguistics
    • WP4 Animation
    • WP5 Linguistics
  • User: Applications
    • WP2 Sign Tutor
    • WP1 Broadcast
    • WP2 WWW
    • WP3 Face to Face
    • WP6 Usability
  • Exploitation & Dissemination
presentation by streams animation
Presentation by Streams - Animation
  • WP 4 Animation
    • Increased realism in sign generation
    • Enhanced signing experience
  • WP5 Sign Language Linguistics
    • Use of natural sign language
    • Synthesis of sign language gestures
animation work objectives
Animation Work: Objectives
  • WP4:
    • Develop Hi-Resolution Avatars + related capture, and animation
    • To enable and support application development in WPs 1-2-3 using WP4 (& WP5) Product
    • To further develop, compare and integrate both proprietary and standard solutions, where appropriate, in networked environments
technology wp4 animation
Technology: WP4, Animation
  • At start of Year:
  • Visia 2
    • Running in Mask 1
    • Using Motion Capture Data only
    • Reasonable animation, expression etc.
technology wp4 animation1
Technology: WP4, Animation
  • Visia 2 in MPEG-4
    • Mesh partitioned into anatomical segments
    • MPEG-4 compliant authoring tool
    • Animation editing tool
    • Server-client tool for TX of animation parameters
    • MPEG-4 SNHC player<25fps
    • Embedded within an MPEG-4 set-top box
technology wp4 animation2
Technology: WP4, Animation
  • Visia 3
    • Updated Virtual Human
    • Higher resolution & polygon count, more realistic photographic textures
    • Improved articulation
    • Mesh distortion applied to garments
    • Facial expression via skeleton manipulation & morphs
    • Speech Enabled
technology wp4 animation3
Technology: WP4, Animation
  • Visia 3
    • New host software - Mask TNG
    • Writing new Active X Controls
    • Superior functionality, lighting and Camera FX, image quality, frame rate, flexibility etc.
    • >75 FPS
technology wp4 animation4
Technology: WP4, Animation
  • Visia 3
  • Running in Mask TNG Graph
technology wp4 animation5
Technology: WP4, Animation
  • Facial Morphs
  • Created in Maya, exported to Mask TNG
  • Based on Sign Language expressions (BSL Dictionary)
  • Inter-operable
  • Variable weighting (<100%+)
  • May be used with Mo-Cap data or for synthetic sign
technology wp4 animation6
Technology: WP4, Animation
  • Facial Animation - Experimental Work
    • Tracking of Active Shape Models
    • Tracking of Active Appearance Models
technology wp4 animation7
Technology: WP4, Animation
  • Facial Animation - Experimental Work
  • Vision-based motion capture of facial expressions using MPEG-4 compliant templates.
wp4 synthetic animation introduction
WP4: Synthetic Animation - Introduction
  • Task:
    • Make avatar do signing synthetically
    • as specified by ViSiCAST’s Signing Gesture Markup Language - SiGML
  • Motive:
    • Synthetic animation is more flexible than animation via motion-capture - “just write some more SiGML”
    • Support Natural-Language-to-Animation strategy of WP4-5
    • In broadcasting applications: put synthetic player on receiver and transmit SiGML - very low bandwidth
wp4 synthetic animation context
WP4: Synthetic Animation - Context
  • Televirtual Avatar is a deformable textured Mesh
  • Mesh shape and position are determined by configuration of underlying Skeleton
    • skeleton configuration: a.k.a. “Bone-Set”
  • To animate avatar: need to generate stream of Bone-Sets - one per frame of animation
    • i.e. BAF data stream - BAF = “Bones Animation Format”
    • Data intensive: 4Kb per bone-set
wp4 synthetic animation technical approach
WP4: Synthetic Animation - Technical Approach
  • SiGML specifies gestures through:
    • Postures:
      • hand shape
      • hand orientation - palm and extended finger direction
      • position of hand(s) in signing space
    • Motions - straight-line, circular, zig-zag etc.
  • Synthetic Animation Engine:
    • specifies hand bone configuration for given posture
    • configures arm/shoulder bones using Inverse Kinematics
    • implements transition from one posture to next using non-linear interpolation - often via control system modelling
wp4 synthetic animation progress i
WP4: Synthetic Animation - Progress (i)
  • Initial Prototype (D4-2) delivered 2001-12
    • Supported most of manual SiGML
    • Implemented in Perl (interpreted scripting language)
    • BAF/VRML output to file - and then to avatar
    • Relatively slow - often < 15 fps
    • Perl module packaged as ActiveX control
      • relatively unwieldy architecture
  • Enhancements for 2002-02 (M5-11)
    • BAF data stream cached in memory-fed directly to avatar
    • Front-end(for WP5): HamNoSys input server, with built-in HamNoSys-to-SiGML translation
wp4 synthetic animation progress ii
WP4: Synthetic Animation - Progress (ii)
  • HamNoSys-to-Signing (Fast) 2002-06
    • Synthetic Animation Engine re-implemented in C++
      • 50 times faster - generates approx. 1000 fps, supporting real-time streamed input (e.g. Broadcast, WWW)
    • More flexible framework - basis for improved authenticity
    • Modular system architecture - supports flexible application development, scripting in WWW pages, etc.
  • Upgrade to Mask2 2002-09
    • Interface to new primitive Mask2 ActiveX control
      • allows better control of animation frame scheduling
    • BAF replaced by VBM (ViSiCAST Bones and Morphs) - provides framework for support of non-manual SiGML
presentation by streams linguistics
Presentation by Streams - Linguistics
  • WP 4 Animation
    • Increased realism in sign generation
      • Enhanced signing experience
  • WP5 Sign Language Linguistics
    • Use of natural sign language
    • Synthesis of sign language gestures
wp 5 language technology
WP 5: Language Technology
  • Goal within the project:
    • To provide semi-automatic translation from English into BSL, DGS, NGT
  • Can also be used to assist the user in monolingual language input
    • No writing system for sign languages established

Presentation by Streams

  • Animation and Linguistics
  • User Applications
  • Exploitation and Dissemination

Presentation by Streams - Sign Tutor

  • WP2 Sign Tutor
  • WP1 Television
    • Closed signing for Broadcast DTT
  • WP2 Internet
    • Information and Education for Sign Language Learners
  • WP3 Face to Face
    • High Street Post Office Counter Services
  • WP6 Comparison of virtual signing
    • with video-recorded Human Signing
presentation by streams television
Presentation by Streams - Television
  • WP2 Sign Tutor
  • WP1 Television
    • Closed signing for Broadcast DTT
    • Enhanced signing experience
    • Regulation and Standards
  • WP2 Internet
    • Information and Education for Sign Language Learners
  • WP3 Face to Face
  • WP6 Comparison of virtual signing
virtual humans on tv the advantages

Virtual Humans on TV: The Advantages

Low transmission rate < 25 kbit/s

Compatibility with signing on other media and sign languages

Precise, sharp representation of signer

Open display options

Compliance with international standards: MPEG, DVB


cost saving

allows vast no. of signed programmes

unified framework from video-based to VH signing

broadcast vh signing achievements

Broadcast VH Signing:Achievements

Integrated TX system for broadcast to STBs

Implementing virtual human s/w in STB

MPEG-2 delivery layer for maximum compliance:

with existing hardware

with MPEG & DVB standards

with proprietary formats

MPEG-4 Audio-Video codec and player

MPEG-4 compliant virtual human

MPEG-4 SNHC virtual human codec and player

MPEG-4 based closed signing service demonstrated

at IBC 2002











































Broadcast VH Signing:Functional architecture



broadcast vh signing system layer implementation









receiver card



Broadcast VH Signing:System layer implementation









broadcast vh signing perspectives

Broadcast VH Signing:Perspectives

Advanced TX system for broadcast to

MHP compliant STBs

Open, MPEG & DVB compliant architecture

Improved synchronisation layer

Integrating a compositing layer

Implementing an enriched MPEG-4 multimedia

authoring tool

Integrating SiGML stream

presentation by streams www web pages with signing field trials
Presentation by Streams -WWW - Web pages with signing Field trials
  • WP2 Sign Tutor
  • WP1 Television
    • Closed signing for Broadcast DTT
  • WP2 Internet
    • Information and Education for sign language learners
    • Web-pages with signing
  • WP3 Face to Face
    • High Street Post Office Counter Services
  • WP6 Comparison of virtual signing
weather forecast a pplication


‘play list’

content provider

web-browser + plug-in

forecast creation tool

Weather Forecast Application

1rst DEMO

2nd DEMO



weather signs


The field trials with Deaf users

  • Hosting at site of Dutch Deaf organisation Dovenschap:
  • Running from end-June until end-October
  • Deaf users can join the field trial by filling in a form on the website
  • CD-rom with necessary software sent to users
field trial promoted
Field Trial Promoted
  • 70 e-mails to webmasters of Deaf clubs, Deaf schools, Deaf organisations and private sites of Deaf persons
  • promotion on Teletext (T.V.)
  • on informative websites for Deaf people
  • visit at meeting of national Deaf organisation with 12 member organisations
  • article in magazine for sign language interpreters
  • 30 CD-roms sent to Deaf clubs and schools
trial feedback
Trial Feedback
  • Helpdesk, contacted by e-mail
  • Discussion page on website
  • Evaluation form: software and installation, included with receiving software
  • Evaluation form: avatar and sign language, will be sent end of October 2002
present situation
Present Situation
  • Field trial still running
  • News slowly spreading
  • Positive reactions
  • Results at the end of November
presentation by streams face to face
Presentation by Streams – Face to Face
  • WP2 Sign Tutor
  • WP1 Television
    • Closed signing for Broadcast DTT
  • WP2 Internet
  • WP3 Face to Face
    • High Street Post Office Counter Services
    • Close involvement with RNID
  • WP6 Comparison of virtual signing
    • with video-recorded Human Signing
wp3 overview
WP3 Overview
  • Evaluation – October 2001
  • New TESSA system – Mar 2002
  • Post Office Trial – May 2002 – Present
  • Sign Recognition – April 2002 – Present
evaluation october 2001
Evaluation – October 2001
  • Evaluation conducted at PO concept store using TESSA V3.
  • 10 Deaf People and 5 Counter Clerks participated over 10 days.
  • Mirror of previous evaluation + Some comparative tests of virtual signing with a video recorded human signer (full details in WP6 presentation)
evaluation observations
Evaluation – Observations
  • Clerks complained about the speed of transactions
  • Caused by :
    • Toggle switch for recogniser
    • Mis-recognitions caused by large vocabulary
    • Poor mapping from recognised speech to phrases
    • Cumbersome graphical interface
tessa v4 recognition system
Tessa V4 – Recognition System
  • ‘Bag of words’ language model.
  • Only words relevant to post office phrases recognised
  • Many fewer insertion errors
  • More resilient to external noise








tessa v4 phrase mapping
TESSA V4 – Phrase Mapping
  • Phrase mapping system derived from work on Automatic Call Routing
  • Represent each of the signed phrases and the test phrase as vectors in a co-occurrence matrix

Phrase N

Phrase 2

Phrase 3

Phrase 1

A 0 2 0 . . . 1

About 0 0 0 . . . 0

Access 1 0 0 . . . 0

Account 0 1 1 . . . 0

. . . . . . . .

. . . . . . . .

. . . . . . . .

You 0 0 0 . . . 0

you’ve 0 0 0 . . . 1

Your 1 0 0 . . . 0

tessa v4 phrase mapping1
TESSA V4 – Phrase Mapping
  • Weight the entry W(i,j) such that :
  • Calculate distance between vectors representing each canonical phrase and input phrase.

More details in

S. Cox. “Speech and Language Processing for a Constrained Speech Translation System”. In Proc. Int. Conf. On Spoken Language Processing. October 2002

M.Lincoln and S.Cox. “A Comparison of Language Processing Techniques for a Constrained Speech Translation System” (Submitted ICASSP 2003)

tessa v4 mapping evaluation
TESSA V4 - Mapping Evaluation
  • Subset of 155 phrases.
  • 5 Talkers, each asked to
    • write down another way of expressing the phrase
    • record speaker saying this phrase
  • Recognise speech (NB No Adaptation)
    • 75.1% Correct ; 49.8% Accurate
  • Test phrase mapping on both text and recognised speech
tessa v4 user interface
TESSA V4 – User Interface
  • Push to talk (automatic end of speech detection)
  • Larger Buttons
  • Common Phrases which don’t need to be spoken
  • Continually updated list of top 5 most used signs
post office trial set up
Post Office Trial - Set-up
  • Tessa V4 used
  • 5 Post Offices
    • London, Bristol, Derby, Liverpool, Wolverhampton
  • Known Deaf Communities In Each Area
  • 3 Months Duration
  • Equipment Given Health Safety Approval
  • Trained 19 Counter Clerks
  • Provided Help Desk Support
post office trial survey
Post Office Trial - Survey
  • Independent Survey Customers by RNID
  • Independent Survey of Counter Clerks
  • All Users Given RNID Questionnaire
  • All Counters Clerks Interviewed
post office trial publicity
Post Office Trial - Publicity
  • BBC See Hear – early Oct
  • Channel 4 – Documentary on BSL
  • Disability Times – 1 October 2002
  • BBC Worldwide – 24 August 2002
  • ITV London Tonight – 21 August 2002
  • Liverpool Echo – 1 August 2002
  • Camden Chronicle – 1 August 2002
  • Wolverhampton Chronicle – 25 July 2002
post office trial publicity1
Post Office Trial - Publicity
  • Bristol Evening Post – 22 July
  • Liverpool Echo – 19 July
  • Derby Evening Telegraph – 18 July
  • Wolverhampton Express and Star – 17 July
sign language recognition
Sign Language recognition
  • Preliminary investigation
    • 6 Gestures, 10 training and 5 testing examples
    • Single user
    • Motion captured data
    • HMM recognition system
  • Initial results – 95% accuracy
sign language recognition1
Sign Language Recognition
  • Comparison of recognition using motion captured data and video.
  • Collaboration with EU ‘WISDOM’ project.
  • Currently Recording and editing multiuser database.
    • 10 signs, 10 training and 5 testing examples
    • 5 users
    • Motion captured and video
  • RNID to make independent evaluation of recognition accuracy.
presentation by streams usability of virtual signing
Presentation by Streams – Usability of Virtual Signing
  • WP2 Sign Tutor
  • WP1 Television
    • Closed signing for Broadcast DTT
  • WP2 Internet
    • Information and Education for Deaf People
  • WP3 Face to Face
    • High Street Post Office Counter Services
  • WP6 Comparison of virtual signing
    • with video-recorded Human Signing
  • 60 phrases from the PO TESSA system signed by human interpreter on video
  • 120 phrases signed by the virtual human
  • 10 profoundly deaf people whose first language is BSL
  • Outcome measures:
    • Accuracy of identification
    • Subjective ratings for each phrase
    • Overall subjective ratings
subjective ratings
Subjective Ratings

Very difficult

Very easy



usability conclusions
Usability Conclusions
  • Higher accuracy of identification for human than virtual signed phrases (20%)
  • Some improvements in intelligibility of virtual signing required
  • Non-ceiling benchmark of accuracy determined
  • 60% virtual signed phrases judged as good as human signed phrases
  • Greater scope for improvements in terms of subjective views of virtual signing
  • Impressive results for virtual signing
exploitation and dissemination highlights
Exploitation and Dissemination Highlights
  • TESSA IT Awards & success in the community
  • WWW Weather Forecaster launched in 2 European Sign Languages & encouraging feedback
      • IvD & RNID host in UK and the Netherlands
  • Close Involvement of Deaf People
      • RNID promoting ViSiCAST nationally
  • BBC Collaboration for closed signing solution for broadcasting DTV for bandwidth efficiency
      • Increasing amount of in-vision signing disliked by hearing people
      • Impacts on DTT multiplexes where bit-rate is already at a premium
exploitation dissemination
Exploitation & Dissemination
  • UK Government 10 year target - 5%programmes on DTT services to be signed
  • Today, services use ‘open signing’
    • Hearing viewers can find distracting
    • Seldom transmitted at peak viewing times
  • Closed signing offers freedom
    • for viewers - to turn on and off
    • scheduling freedom for broadcasters
    • but needs extra transmission feed
  • ViSiCAST uses ‘virtual human’
    • reducing bandwidth needs by factor of ten compared to video
closed signing why an avatar based solution
Closed Signing – Why an avatar-based solution ?
  • MPEG2 coding (0.5-1Mbit/s)
    • only 1 service signed per multiplex if at all
  • MPEG4 coding (<350Kbit/s)
    • no more that 2 services signed per multiplex
    • more efficient compression, and ability to code non-rectangular objects
  • Animated Avatars (<100Kbit/s)
    • may be possible to sign all services in a multiplex
    • need new techniques to capture motion of real signers
closed signing requirements for the broadcaster
Closed Signing Requirements for the Broadcaster
  • Be compatible with existing studio, distribution & monitoring infrastructures
  • maintain freedom to schedule as needed
  • accommodate live signing and reactive scheduling
  • allow for regional content insertion and time-shifting &
  • cope with the variety of picture display formats
avatar signing developments for broadcasting
Avatar Signing developments for broadcasting
  • Motion capture needs to be efficient and signer-independent
    • enabling signing of live and reactive broadcast material
    • best suited for offline broadcasting today
  • Facial motion capture needs refinements
    • Increasing realism make avatars more acceptable



Video Server


Signing Capture - Studio Implementation

Original Programme

SDI with embedded Signing Data


Signing Data SDI inserter


Coding /






studio and distribution issues
Studio and distribution issues
  • Provision of television programme material with associated signing
  • Development of equipment for conveying signing data within studio infrastructure
    • We have developed hardware to add signing or motion capture data to a SDI video stream.
    • The main program video/audio, and the corresponding data can then be routed via standard studio infrastructure.
    • The combined A/V and signing data can also be stored on server or video tape
  • Development of DVB inserter agnostic of signing signal coding method
  • Development of end-to-end DT demonstrator