html5-img
1 / 31

Take Home Messages

Results of the ATIS/T1A1.1 Ad Hoc Group on Full-Reference Video Quality Metrics (FR-VQM) VSF Meeting October 3, 2001 John Pearson Sarnoff Corporation jpearson@sarnoff.com. Take Home Messages. Tariff’s can now include Visual Quality Metrics (Full Reference)

gad
Download Presentation

Take Home Messages

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Results of the ATIS/T1A1.1Ad Hoc Group on Full-Reference Video Quality Metrics (FR-VQM)VSF MeetingOctober 3, 2001John PearsonSarnoff Corporationjpearson@sarnoff.com

  2. Take Home Messages • Tariff’s can now include Visual Quality Metrics (Full Reference) • The basis for this is a family of 4 Technical Reports by ATIS/T1A1 • The T1A1 approach is extensible to additional Visual Quality Metrics, and does NOT establish a Standard

  3. Outline • Why is measuring Visual Quality important? • Why is measuring Visual Quality hard? • International Standards for VQM’s • T1A1 Technical Reports

  4. FR-VQM Needs of US Telecom Q-?? Q-A Q-B • Digital video processing can create objectionable noise • End-to-End QoS across the networks of multiple companies requires agreement on Quality at Transfer Points (Tariffs) • Tariff’s require ANSI sanctioned technical documentation Company Auses VQM-A Company Buses VQM-B Site of Video Origination(e.g., Denver) Transfer Between Network A & B Site of Video Consumption(e.g., Mexico City)

  5. Digital Video Creates “Patterned” Noise ... Human visual response to patterned noise highly non-linear ... Random “Analog” Noise Blocky “Digital” Noise MSE = 27.10 MSE = 21.26 Measures like MSE suitable for Analog noise no longer work for Digital noise

  6. Patterned noise in the sky much more perceptible even though much smaller in terms of pixel differences Source Frame Difference Map Codec Frame

  7. Visual Quality Metrics... correlate well across scene types, unlike MSE ... Visual Discrimination Model Mean-Squared Error Mean of 80 trials for 20 subjects Bars show 5% confidence intervals

  8. Vital Role of Subjective Database • Goal of VQMs is to approximate subjective quality assessments (SQA) • The relevance of the SQA depends on: • Test sequences (SRC’s) • Distortion generators (HRC’s) • Viewing conditions and testing protocols • Producing a relevant SQA is hard

  9. Three Kinds of VQM’s • Full Reference (FR) • a double-ended method and is the subject of this Technical Report. • Reduced Reference (RR) • only reduced video reference information is available. This is also a double-ended method. • No Reference (NR) • no reference video signal or information is available. This is a single-ended method. • It is generally believed that the FR method will provide the most accurate measurement results while the RR and NR methods will be more convenient for QoS monitoring. • The T1A1 Technical Reports concern FR methods

  10. Full-Reference VQM’s with Normalization

  11. International Standards Progress • VQEG may be several years from recommending a FR-VQM standard to ITU • Its possible that no single FR-VQM will be a clear “winner” • The FR-VQM field is young, and significant, steady improvements are expected over the next decade • It’s possible that several different FR-VQMs may gain industry acceptance

  12. T1A1.1 FR-VQM Strategy… an extensible family of TR’s for FR-VQ, enabling Industry to move ahead without Standards ... • Provide guidelines for how Industry can • specify its specific FR-VQM needs • assess the suitability of existing documented FR-VQMs • drive the development by FR-VQM proponents of new/improved FR-VQM algorithms and products • inter-operate with different FR-VQMs • Provide guidelines for how FR-VQMs can be • documented in algorithms, accuracy and limitations • quantitatively cross-calibrated to each another • Extensible framework enabling addition of FR-VQMs • Start by specifying two already disclosed FR-VQMs • Stimulates continued FR-VQM innovation

  13. Primary Contributors

  14. Family of Technical Reports • TR A1: Accuracy and Cross-Calibration (Mike Brill, Sarnoff) • defines accuracy (statistical analysis), limitations of a FR-VQM • defines transformation to common scale, for cross-calibration with other applicable FR-VQMs • TR A2: Normalization Methods (David Fibush, Tektronix) • applied to source and processed video before VQM calculation • e.g., spatial/temporal registration, gain/level offset calibration, ... • may utilize special test signals • TR A3: Peak Signal to Noise Ratio (Steve Wolf, NTIA) • Specify PSNR VQM, following TR A1 and TR A2 guidelines • TR A4: Objective Perceptual FR-VQM Using a JND-Based Full Reference Technique (David Fibush, Mike Brill) • Specify JND-based FR-VQM, following TR A1 and TR A2 guidelines

  15. TRA1 Defines Basic Methods: LIMITATIONS • “How to” specify VQM accuracy • with respect to subjective assessments • based on defined statistical analysis • “How to” specify VQM scope/limitations • type of scene content (“signal”) • high/low motion, color/b&w, interlaced/progressive • type/severity of artifacts (“noise”) • e.g., encoding techniques, bit-rates, blurring, blockiness • subjective testing characteristics • behavior with viewing distance, resolution, gamma, … • expert vs non-expert viewers • “How to” cross-calibrate VQMs • determination of mathematical transformation relating one VQM’s outputs to another’s SCOPE Works well, & has been well tested here

  16. VQEG Database: “SRC’s”

  17. VQEG Database: “HRC’s”

  18. JND/PQR & PSNR Limitations: no H.263

  19. Algorithm Documentation: JND/PQR

  20. Stripping for JND/PQR Registration

  21. Algorithm Documentation: PSNR

  22. Normalization Requirements JND/PQR PSNR

  23. VQEG data & Logistic-mapped PQR

  24. Logistic-mapped PQR for Common Scale… provides approach for cross-calibration...

  25. Accuracy -- 3 Methods • RMSE • Resolving Power • Classification of Errors

  26. Confidence vs.D-VQM: JND/PQR

  27. Confidence vs.D-VQM: PSNR

  28. RMSE • RMSE: root mean square error between subjective and objective normalized scores

  29. Classification of Errors

  30. Progress • T1A1.1 Ad Hoc Group created Feb. 2001, co-chairs John Grigg, John Pearson • Mail Ballot Approval August 2001 • Approved by T1A1.1 25 September 2001 • Approved at Plenary meeting of T1A1, 28 September 2001

More Related