WORD PROGRESS IN VIDEO AND AUDIO QUALITY EVALUATION AND CONTROL IN BROADCASTING SYSTEMS

1. WORD PROGRESS IN VIDEO AND AUDIO QUALITY EVALUATION AND CONTROL IN BROADCASTING SYSTEMS

2. The main goal - attainment of viewers and listeners satisfaction • The main effect on this satisfaction has program quality. It depends on video and audio quality and successively on technical parameters of transmission line as well on people propriety of hearing and sight sense (sensitiveness, spectrum properties, inertness of sight sense and resolution possibility)

3. Two kinds of measuring methods : objective and subjective Objective are based on signals parameters testing using measuring equipment. . Objective test results don’t give the full information about subscribers impression concerning program observed. Especially in digital television exact correlation between disturbance and the viewers sight impression maybe missed.The special sight and hearing phenomenon which make testing with signals. impossible. Subjective are based on observation the estimated program by groups of observers , who give adequate opinions about the quality of them and treating these opinions statistically. The nature of human judgment (very much complex and dependent on many factors).

4. Principles of subjective evaluation methods (ITU-R)- Rec.ITU-R BT 500 “Methodology for the assessment of the quality of television pictures” . the general common features and subjective assessment methodology. Two basic methods are described in detail. • The DSCQS (double stimulus continuous quality-scale), using a continuous quality scale. It can in particular provide accurate results for very small differences in quality, and where impairments have either a positive or negative effect on quality. • DSIS evaluation method (double stimulus impairment scale), agreed impairment scales for psycho-physical testing of television. The scales are used in a controlled and defined way, the results of tests obtained are valid, reliable, and consistent between tests.

5. Alternative methods • Single-stimulus (SS) methods, a single image or sequence is presented - an index on the entire presentation, contain numerical categorical judgement methods, non- categorical judgement methods and performance methods, • Stimulus comparison methods, two images or sequences are presented- an index of the relation between the two presentation. contain: adjectival categorical judgement methods, non-categorical judgement methods and performance methods. • Single stimulus continuous quality evaluation (SSCQE), digitally coded video ( scene-dependent and time varying) is measured continuously, with subjects viewing the material once, without the source reference. • Simultaneous double stimulus for continuous evaluation method (SDSCQE), in which the reference condition is introduce to SSCQE.

6. Selection of test methods • Assessment problem/Best method Measure the quality of systems relative to a reference Double stimulus continuous quality scale (DSCQS) method Measure the robustness of systems (i.e. failure characteristics) Double stimulus impairment scale (DSIS) method Quantify the quality of systems when no reference is available Ratio-scaling method or single stimulus categorical scaling

7. Compare the quality of alternative systems when no reference is available Method of direct comparison, ratio-scaling method or categorical scaling. Measure the fidelity between two impaired video sequences Simultaneous double stimulus for continuous evaluation method SDSCE) Compare different error resilience tools Simultaneous double stimulus for continuous evaluation method (SDSCE) Measure the quality of stereoscopic image coding Simultaneous double stimulus for continuous evaluation method (SDSCE)

8. BT 500 describes the general methods of subjective assessment, the details concerning the application to subjective assessment of measured systems in the connected recommendations: • BT 1128 on subjective assessment of conventional television systems, • BT 1129 on subjective assessment of standard definition digital television systems (SDTV) • BT 710 on subjective assessment methods for image quality in high-definition television • BT 1438 on subjective assessment methods for image quality in stereoscopic television • BT 812 on subjective assessment of the quality of alphanumeric and graphic pictures in Teletext and similar services • Test material to be used in the subjective assessment of picture quality - BT1210.

9. Audio quality evaluation Recommendations BS 1116, BS1285, BS1286 subjective assessment Recommendation BS 1387 objective measurements Multimedia systems( ITU-T SG 9). Recommendations P910 and P911 -similar methods, as: • Absolute category rating (ACR), • Absolute category rating with hidden reference (ACR-HR), • Degradation category rating (DCR), • Pair comparison method (PC), • Single stimulus continuous quality evaluation (SSCQE).

10. Digital television - new quality of service considerations, with complex relationships between objective parameter measurements and subjective picture quality. Objective measurements with good correlation to subjective quality assessment are desirable Objective measurements are not a direct replacement for subjective quality assessment. • Subjective quality assessments - designed procedures intended to determine the average opinion of human viewers to a specific set of video sequences for a given application. Results - valuable in basic system design and benchmark evaluations. Subjective quality assessments for a different application with different test conditions will still provide meaningful results; however, opinion scores for the same set of video sequences have different values. .

11. Objective measurements - intended for use in a broad set of applications producing the same results with a given set of video sequences. The choice of video sequences to use and the interpretation of the resulting objective measurements are the factors which vary for a specific application. • Objective measurements and subjective quality assessment are complementary rather than interchangeable. . • Subjective assessment is appropriate for research related purposes, objective measurements are required for equipment specifications and day-to-day system performance measurement and monitoring. .

12. „objective perceptual measurement" refers to the measurement of the performance of a programme chain by the use of programme-like pictures and objective (instrumental) measurement methods to obtain an indication that approximates the rating that would be obtained from a subjective assessment test. There are three basic methods to perform objective measurements: • • FR – A method applicable when the full reference video signal is available - double-ended method • • RR – A method applicable when only reduced video reference information is available - double-ended method. • • NR – A method applicable when no reference video signal or information is available.- single-ended method..

13. Studies of perceptual video quality measurements were conducted in informal group, Video Quality Experts Group (VQEG).: • The first task of VQEG was to assess the performance of proposed full reference perceptual video quality measurement algorithms FRTV. • The proponent models were checked in terms of: • • prediction accuracy (the model's ability to predict the subjective quality); • • prediction monotonicity (the degree to which the model's predictions agree with the rank • ordering of subjective quality ratings; • • prediction consistency (the degree to which the model maintains prediction accuracy over the range of video test sequences and video systems, i.e., that its response is robust with respect to a variety of video impairments).

14. VQEG's FRTV validation test Phase I did not provide sufficient evidence to identify a method to be recommended for objective assessment of perceptual video quality. VQEG's FRTV validation test Phase II produced results that indicate that four of the methods are appropriate for inclusion in the Normative part of Recommendation. • In Phase II, strict adherence to BT.500-11 procedures for the Double Stimulus Continuous Quality Scale (DSCQS) method was followed in the subjective evaluation. On the base of VQEG works were prepared the ITU-R Recommendations on objective perceptual video quality measurement technique for broadcasting applications in presence of full reference signal: :BT 1683 for SDTV, BT 1866 for low definition and objective perceptual video quality measurement technique for broadcasting applications with reduced bandwidth BT 1867 for low definition and BT 1885 for SDTV.

15. Next VQEG Projects completed: • • FRTV Phase I - completed • • FRTV Phase II – completed • • HDTV – completed • • Multimedia Phase I – completed • • RRNR – completed • Are the base for preparing draft new recommendations

16. Technical progress and new system developing results in necessity of developing new evaluation methods complying with complexity of factors affecting perceived picture quality in present and new broadcasting systems. • New assessment methodology ,which includes the cumulative influence of the different elements of the media delivery chain, and at any given point, the cumulative influence of quality and impairment factors

17. The actual VQEG Projects, concerning the new systems are the following: • • HYBRID Perceptual/Bitstream, in which hybrid models use as inputs both the coded bitstream information and the decoded video. No reference (NR) models will be tested. • • VQEG Joint-Effort Group (JEG), whose scope is the collaborative development of a hybrid no-reference video quality assessment model. • • Multimedia Phase II • • VQEG Quality Assessment for Recognition Tasks (QUART), whose scope of t is to develop subjective and objective methodologies for task-based video quality assessment • • VQEG 3DTV. The purpose of this project is to develop subjective and objective metrics for stereoscopic and auto-stereoscopic systems. The study of subjective methodologies for 3DTV, and crosstalk was identified as a major factor influencing user Quality of Experience (QoE) of 3DTV applications. • HDTV Phase 2

18. The ITU-R all Recommendations “in force” are revised. The Draft Revision of BT 1210, concerning test material was approved by SG 6. The preliminary Draft Recommendation concerning general viewing conditions for subjective assessment of quality of television pictures and Preliminary Draft Revision of Rec BT 500, concerning observers. were proposed • A drafting group on evaluation methodology for 3DTV began work and prepare working document towards the preliminary draft new recommendation on 3D TV subjective assessment, having regard to specific quality assessment factors. • The main evaluated parameter for 3D TV is Quality of experience (QoE).