Test for video quality assessment Proposed methodology

1. Test for video quality assessment Proposed methodology • There are several approaches for the assessment of video quality: • Subjective evaluation perceptual • Objective evaluation metric with perceptual (full reference, reduced-reference, No-reference)The subjective evaluation tests are the first step to create reference data to validate the metrics.

2. Test for video quality assessment Proposed methodology • We place ourselves in the first field of application: SQM • We propose a methodology to organize the mean test evaluation using real end user (no simulations). • A necessary condition for the success: the assessment must be reliable and replicableThree key points:The procedureVariable managementChoice of scale test

3. Test for video quality assessment Proposed methodology Step 1: Selection of test material • Generally provided by the client of the evaluation • Digital content has a significant influence on the test results • From 8 to 16 different content must be included in the test (i.e., www.vqeg.orgSusie, Waterfall, Tree, Tampete, Mobile & calendar ...)

4. Test for video quality assessment Proposed methodology Step 2: choice of testers • Experts vs. non-experts • All must pass an eligibility threshold (acuity, color perception) Snellen Eye Chart and Ishihara test plate. • Acceptable: 16 to 24 meters to get statistically acceptable • Experts only for tests aimed at developing algorithms. For a test that simulates the perception of an average consumer will choose non-expert testers.

5. Test for video quality assessment Proposed methodology Step 3: Design of experiments session • Decisions are variables to consider • Each subject should not watch more than 90 minutes in sessions of 30 minutes max. • Mode is selected: single-monitor, triple monitor, and physical parameters (inch screen, distance meters) • We must consider the environmental noise, brightness and calibration

6. Test for video quality assessment Proposed methodology Step 3: Design of experiments session • Definition of sessions warm up and re-set. • Generation of instructions to testers. • Choice of test Single or Double Stimulus • Choice of sequences of trial.

7. Test for video quality assessment Proposed methodology Step 4: Choosing the scale test Double stimulus impairment methodSubjective scale in 5 steps with respect to the variable considered: imperceptible, perceptible but not annoying, slightly annoying, annoying, very annoying. Double stimulus quality scale method Double continuous scale (0 to 100) with two votes. Comparison scales method

8. Test for video quality assessment Proposed methodology Step 5: data analysisWe obtain the detection averaged quality score over all test participants, for EACH test sequence and for EACH variable studied We analyze the statistics to evaluate which variable is significant or which is irrelevant to the perception of video quality

9. Test for video quality assessment Proposed methodology • Which variables?Psychological variables (interest and expectations of the user)Display properties (resolution, physical dimensions, size, viewing distance)Properties of coding / transmission (bit rate, encoder, video resolution)Viewing conditions (artificial lighting, contrast of the screen)Fidelity (due to compression, color saturation, natural)It also evaluates the sound? (synchronization, sound quality)Others, defined by the customer

10. Test for video quality assessment Proposed methodology • Timing of the testSet up of the room with the display to use.We define the sequences of the reference and those of trials.We obtain the total number of minutes of the test, the number of meters to be convened and programming sessions.Program the data analysis step at the end of the tests.

11. Test for video quality assessment Proposed methodology • Role of the EngineerSet up of the test and characterize it completely.Recruit testers and test their reliabilityTest and consolidate dataPresentation of statistical results with an analysis of their meaning and possible interpretation. • Role of the customer Providing video material to evaluate and agree upon upgrade hw / sw required Provide guidance on which variables to assess interest.

12. Exercise: image quality assessment • Adapt the proposedmethology to image qualityassessment. Organise the tasks in group of 10-12 students;