Andrew Perkis, Peter Svensson, Professors

1. Network Media Handling • A common platform and scenario based on Virtual worlds, gaming and quality Andrew Perkis, Peter Svensson, Professors Centre for Quantifiable Quality of Service in Communication Systems (Q2S) Department of Electronics andTelecommuniations Norwegian University of Science and Technology Andrew Perkis, NMH Virtual worlds, gaming and quality

2. Common architecture - common usage scenario Process Q2S research plan Networked Media handling Technology to present, manipulate and evaluate multi media content Focus on media representation, error protection, perceived quality, functional placement QoS mechanisms for dynamic networks Mechanisms that affect QoS, emphasis on heterogeneous and dynamic environment Focus on dependability, security, resource allocation Quality assessment and monitoring Measuring methods and models for QoS Measurement of perceived QoS, measurement methods and architecture, and traffic performance • Common usage scenario • Develop work on Virtual worlds and serious gaming • Align and coordinate research teams • Team • Andrew, Peter and Touradj • Postdocs • PhDs • Master students • Process • Coordination – weekly meetings, MindMap • Information and communication • Q2S research plan • Network Media Handling • Quality assessment and monitoring Andrew Perkis, NMH Virtual worlds, gaming and quality

3. Network Media Handling Focusing on the End-Points Functionality in media representation/distribution Error resilience/protection/concealment Media conversion and scalable/layered coding Multi-modal perception of audio and video Interaction between Networks & End-Points Optimized media handling in wired and wireless networks Congestion and admission control for multimedia traffic Objective network measurements for end-to-end QoS Quantifiable perceptual quality metrics and their use Application Studies Interactive two-way and multi-party communication High-quality streaming media services and applications Virtual acoustics / 3D audio systems Digital cinema Andrew Perkis, NMH Virtual worlds, gaming and quality

4. Multimodal Perceptual Quality Assessment Multimodal Perception Relationship between audio and video Perceptual effects of network impairments Relates to end-user expectation and context Developing Objective Models Study of realistic network environments Simulation or controlled laboratory experiments Laboratory for multimodal subjective experiments Design of quantifiable metrics for perceived quality Use of Objective Models Automated monitoring of end-user perceived quality Relationship to measurable end-2-end QoS parameters Using “Perceived QoS” to adapt and enhance system performance Andrew Perkis, NMH Virtual worlds, gaming and quality

5. Architecture Andrew Perkis, NMH Virtual worlds, gaming and quality

6. Research topics Andrew Perkis, NMH Virtual worlds, gaming and quality

7. Applications Andrew Perkis, NMH Virtual worlds, gaming and quality

8. Q2S resources and activities Andrew Perkis, NMH Virtual worlds, gaming and quality

9. Research results Compilations Andrew Perkis, NMH Virtual worlds, gaming and quality

10. Implementation End user quality assessment Congestion control with delivery of erroneous packets Classificationof data, UEP, FEC Low delay lossless compression for multichannel audio Computational quality measurement for IP-based audio Andrew Perkis, NMH Virtual worlds, gaming and quality

11. Creative Demonstration available during break Figure 2: Sample stereoscopic video contents use for measuring the impact of exposure affect on multimedia quality. Andrew Perkis, NMH Virtual worlds, gaming and quality

12. Subjective test methods 4 parameter categories; visual, audio, gameplay and network visual quality metric for HQ images NovelEvaluationMethod: DSAQFA StereoscopicQualityassessment Andrew Perkis, NMH Virtual worlds, gaming and quality

13. Models and assesment Economical and technology aspects of end-to-end Quality of Services provisioning in the Internet PEAQ ODG measurement Focus of Attention (FOA) based Visual Quality Assessment Andrew Perkis, NMH Virtual worlds, gaming and quality

14. FP7 Call 4 proposals Andrew Perkis, NMH Virtual worlds, gaming and quality

15. European proposals Andrew Perkis, NMH Virtual worlds, gaming and quality

16. EURO NF 35 partners Andrew Perkis, NMH Virtual worlds, gaming and quality

17. 3D Media Internet Services (3DMEDIA) - NoE • Leading WPs • European 3D Media Lab integration • 3D Media Quality of Experience modeling and evaluation • Common software, data sets and repositories • maintaininventoryofparticipating labs • enablestrongcollaboration • Developmethods for subjectiveassessments • Establishvirtual 3D MEDIA QoE test facility • Determine and measurequality features and models • Contribute to standardization Andrew Perkis, NMH Virtual worlds, gaming and quality

18. Quality of Experience in Networked Systems and Technologies (QUEST) - NoE • Leading WP • Develop an integrated QoE platform based on the datasets, measures and methodologies provided by other WPs. • Explore, design, and implement a certification mechanism for quality of experience based on the integrated platform. • Activeparticipation in otherWPs • Subjective and objective image, video and audio-visualqualityassessmentwithrespect to different services, applications and scenarios Andrew Perkis, NMH Virtual worlds, gaming and quality

19. ENd User quality of experieNCe In multi modAl neTworked mEdiaENUNCIATE : CSA Andrew Perkis, NMH Virtual worlds, gaming and quality

20. Conclusions • A good common scenario is under developments • Substantial involvement • Cross disciplinary • Needs to enhance network components • Has potential for being a platform for many integrated projects at Q2S Andrew Perkis, NMH Virtual worlds, gaming and quality

21. Research results Individual Andrew Perkis, NMH Virtual worlds, gaming and quality

22. High definition assesments • Digital Cinema • For very high qualities, PSNR slightly tends to be best objective metric so far, compared to structural similarity based metrics • No significant differences in correlation between metrics • Comparison: effects of source vs. source + channel distortion on perceived quality • At low source distortion levels, PSNR tends to overestimate quality with both source + channel distortion • With increasing source distortion levels, Δ between PSNR values of source and source + channel distortion gets smaller Ulrich Andrew Perkis, NMH Virtual worlds, gaming and quality

23. Novel Evaluation Method: DSAQFA • Compare different types of distortion • Usually: distortion expressed in terms of PSNR • Qualitative characteristics of source and channel distortion are different • PSNR doesn’t scale well Channel distortion Source distortion Ulrich Andrew Perkis, NMH Virtual worlds, gaming and quality

24. Ulrich Andrew Perkis, NMH Virtual worlds, gaming and quality

25. Results • PSNR • Is not a good measure when comparing different types of distortion • Especially for low source distortion (= HQ video) • Performance increases for higher source distortions (= LQ video) • Methodology • Well suited for comparing different types of artifacts • Minimum training nesseray • Fast, reliable, no rating scales Ulrich Andrew Perkis, NMH Virtual worlds, gaming and quality

26. Congestion control based on delivery of partly corrupted packets • In the presence of bit errors, network throughput can be improved and stabilized by allowing delivery of corrupted packets • Also the video quality is improved (in terms of PSNR), if appropriate resilience mechanisms are employed (example results below) High bit error rate Throughput Avg. PSNR Trad. 0.95 Mbit/s 32.62 MAC Lite 1.55 Mbit/s 34.30 Low bit error rate Throughput Avg. PSNR Trad. 1.41 Mbit/s 34.72 MAC Lite 1.57 Mbit/s 35.22 Jari Andrew Perkis, NMH Virtual worlds, gaming and quality

27. Modeling and services • Computational models for audio and multimedia quality assessment • Developed a priliminary version of a computational model for IP-based audio quality measurement • Service and business models • Developed a simple financial model demonstrating effectiveness of cooperation between competing providers for end-to-end quality of services provisiong • Based on real Internet topology, investigated current probability and potential evolution scenarios for end-to-end QoS provisioning in the Internet Eugene Andrew Perkis, NMH Virtual worlds, gaming and quality

28. PEAQ ODG measurement PEAQ ODG measurement results for AAC and Lame MP3 codecs with different bitrates and packet loss rates Eugene Andrew Perkis, NMH Virtual worlds, gaming and quality

29. Focus of Attention (FOA) based Visual Quality Assessment • Focus of attention (FOA) has a great influence on visual quality assessment. Compared with image quality assessment, the influence of FOA on video quality assessment is stronger. • Objective FOA model can be constructed based on bottom-up and top-down approaches. • Our experiments demonstrate that the correlation between PSNR and subjective measurements can be increased by about 0.33 by integrating FOA into quality assessment. • Based on certain quality features, a visual attention based quality metric (VAQM) is proposed and it outperforms the existing metrics, such as PSNR, SSIM, NTIA model. • The future work is to find and design more suitable quality features and consider the temporal attention as well as the audiovisual attention analysis in the joint audio-video quality assessment. Junyong Andrew Perkis, NMH Virtual worlds, gaming and quality

30. High definition quality evaluations The scatter plots of MOS versus model predictions Pearson’s correlations and their associated 95% confidence intervals for each metric Fitri Andrew Perkis, NMH Virtual worlds, gaming and quality

31. Content-creation • Creation of mixed-reality representations that can be used for interactive future 3D-TV applications: • Created contents for Stereoscopic projection • contents are currently created • Manipulating video sequences to study facial motion perception on quality. • Program and create interactive design and contents for augmented reality Wendy Ann Andrew Perkis, NMH Virtual worlds, gaming and quality

32. Analysis of the impacts of affective elements • Examine impact of affective elements in multimedia presentation on users perception of quality and user experience. The created contents will be used to access the impacts of the following variables: • Effects of familiarity in the in the perception of enjoyment and quality assessment in a stereoscopic environment. • currently planning to conduct an experiment to determine the impact. • Effect of moving Tatcher Illusion technique on perception of quality . • Effects Familiarity over Photometric Inconsistencies. This examines how to effectively combine natural and synthetic environment. • Research shows that people are excellent at identifying faces familiar to them, even from very low quality images, but are bad at recognizing, or even matching, faces that are unfamiliar (Hancock et al. 2000). • Write a paper adding “emotion-level” on the user`s perspective model that affects user perception and multimedia quality. Wendy Ann Andrew Perkis, NMH Virtual worlds, gaming and quality

33. Quantifying the strategy taken in an ensemble delayed hand-clappeing A parameter called compensation factor was suggested to capture the amount of early clapping which performers do to compensate the tempo decrease caused by latencyin a musical interaction. CF can explain Chafe effect: ”Moderate amounts of delay are beneficial to improve the collaboration“ If CF equals a certain amount, this would be the condition for which the trial will have a stable tempo. We call the achieved “delay time over measure duration” as critical CF: Compensation Factor increases linearly with delay Nima Andrew Perkis, NMH Virtual worlds, gaming and quality

34. Stereoscopic image quality metric • Incorporating crosstalk, depth and visual discomfort • Stereoscopic techniques provide a very effective depth perception sensation because of using the binocular cues. The principle behind is of displaying two slightly different perspective views for left and right eyes. • Crosstalk is probably one of the most annoying distortion in visualization stage by using stereoscopic techniques. • We propose a stereoscopic image quality metric which defines explicitly the positive and negative depth information in the crosstalk artefact psychometric function • Original psychometric function = frac{M-m}{1+e^{-( a+b*x)}} • Modified psychometric function = frac{M-m}{1+e^{-( a+b*x*y)}}+ c*y • Where x, y is the objective measurement of the crosstalk and depth stimulus intensity separately. Parameter a, b, c control the curve location and steepness. m is the minimum on the grading scale and M is the maximum. Liyuan Andrew Perkis, NMH Virtual worlds, gaming and quality

35. Stereoscopic image quality Liyuan Andrew Perkis, NMH Virtual worlds, gaming and quality

36. QoE in Internet television - Which network parameters can be extracted and quantized? - How can these parameters be related to the QoE? - How to effectively monitor the last mile (ISP access network)? - How can we handle various user equipment? Mathias Andrew Perkis, NMH Virtual worlds, gaming and quality

37. NTNU Recruitment Game • ApplicationofSeriousGaming • First Person Game on a Virtual Campus (mainbuilding) • Will be used as a recruitmenttool for highschool students in Norway • Containspresentationsofthedifferent studies at IME • The goal is to show students what NTNU looks like and whatyouareable to make after 5 yearsofstudy Håvard Andrew Perkis, NMH Virtual worlds, gaming and quality

38. Research plans Andrew Perkis, NMH Virtual worlds, gaming and quality

39. Computational quality measurement for IP-based audio • Computational models for audio and multimedia quality assessment for a) IP-based audio, b) Videoconferencing, IPTV. • Quantification of various impairments on audio of video quality • Quantification of parameters for various audio and video codecs, packet loss rates, packet loss concealment algorithms, etc. • Subjective studies to investigate (to confirm) assumptions about simultaneous effects of multiple impairment factors • Service and business models • Continue working on economical aspects of end-to-end Quality of Services provisioning in the Internet (SLAs, inter-provider revenue distribution, pricing strategies, etc.) • Risk assessment methods for IP-networking • Business models for new services provisioning (3D media applications, multimedia, quality effects) Eugene Andrew Perkis, NMH Virtual worlds, gaming and quality

40. 3D media Internet • 3D MEDIA related • Develop methods for AV subjective assessments • Determine and measure AV quality features and models • Contribute to standardization • PERCEVAL related • Experimental design for measuring cross-modal effects • Focus on low-attention and long-term quality assessment • AV quality objective metric design and validation Ulrich Andrew Perkis, NMH Virtual worlds, gaming and quality

41. UEP FEC • Rate control allowing delivery of partly corrupted packets • More simulating results with more realistic video sequences and larger network topologies • Analysis of perceptual significance of different data elements (media units) • Essential in unequal erasure protection schemes (UEP FEC, retransmissions) • Aim is to improve the methods for classifying data elements • Audiovisual quality assessment • Results to be used for optimizing the trade-off between audio and video distortion, as well as source and channel distortion • Subjective metrics to be turned into objective metrics Jari Andrew Perkis, NMH Virtual worlds, gaming and quality

42. Perceived audio-visual quality assessment for future media • Perceived quality assessment for multimodal media with complex scenes (HDTV, UHDTV, NHX, 3D, etc.) • Prepare appropriate media scenarios and dataset, design subjective quality methodologies. • Cognitive model for joint audio-visual quality assessment and corresponding objective metrics. • Spatial and temporal attention analysis, semantic multimedia content analysis and the relationship with the perceptual quality assessment. • Integrated platform for quality of experience (QoE) certificate • Define certificate mechanism for QoE methods for future media. Junyong Andrew Perkis, NMH Virtual worlds, gaming and quality

43. Virtual worlds • In the new age of virtual reality and mixed environment, users become part of the synthetic environment to generate new forms of experiences. How can we effectively maximize our perception of quality on these kinds of applications? • Synthetic or 3D contents are less expensive than natural representations because they are mostly pre-defined and can easily be computed in real-time. In real-time or live broadcast (e.g. news event, weather forecast, live sports) some regions are not so salient and therefore possible to substitute or combine with synthetic contents. How can we effectively combine natural and synthetic representations without sacrificing quality perception. Wendy ann Andrew Perkis, NMH Virtual worlds, gaming and quality

44. Stereoscopic visual quality • How does the artefact of coding or communication on different part of the data (left vs right multiview, or color vs depth rendering) affect the overall stereoscopic quality • Same artefect on the different position of left and right multiview, or on the color and depth. How it affects the people. • How about the artefacts on the different depth? Is there any difference? It seems that people will focus more on the foreground(less depth) than background(more depth). Liyuan Andrew Perkis, NMH Virtual worlds, gaming and quality

45. Multimodal Perceptual Quality Assessment of High Definition Content • Codec performance for digital cinema application Conducting subjective quality assessment using currently standardized methodology in DCI specified Digital Cinema environment • Developing subjective assessment methodology that take into account multimodal factor • Developing objective model that take into account multimodal factor Andrew Perkis, NMH Virtual worlds, gaming and quality

46. Extra slides Backups Andrew Perkis, NMH Virtual worlds, gaming and quality

47. Computational quality measurement for IP-based audio (1) • Quality measurement techniques: • Subjective (using human subjects) • Objective based on comparison with original non-degraded signal • Objective computational models (when no original signal is available for comparision) • Focus on objective computational quality assessment for IP-based audio • Previous work: computational quality model for narrowband (up to 4 kHz) and wideband (up to 7 kHz) speech • Work on a similar model for IP-based audio in general Eugene Andrew Perkis, NMH Virtual worlds, gaming and quality

48. Computational quality measurement for IP-based audio (2) • 100-point scale is used for quality measurement and impairment quantification • Impairments are percepually additive • Input: quantified qualtiy impairments • Output: uni-directional quality estimate; should be mapped to a subjective quality measurement scale • Impairments: codec, bit rate, packet loss rate, packet loss concealment, delay (for interactive applications), content dependency,noise, etc. Eugene Andrew Perkis, NMH Virtual worlds, gaming and quality

49. Computational quality measurement for IP-based audio (3) • Current work: • Start with quantification of encoding and packet loss impairments: Q = 100 – function (codec, bitrate) – function (loss rate, packet loss concealment) • Use software implementation of the objective accurate PEAQ algorithm, which is based on comparison of degraded and original versions of a signal • Qualitify the impairments using experimental PEAQ measurement results converting data to the perceptually additive 100-point scale. Eugene Andrew Perkis, NMH Virtual worlds, gaming and quality

50. Some pics Encoding impairment quantification Loss impairment quantification Eugene Andrew Perkis, NMH Virtual worlds, gaming and quality