1 / 53

The problem

Evaluation of packet video quality in real time using Random Neural Networks Samir Mohamed Gerardo Rubino IRISA, Rennes, FRANCE. stream of voice, music, video , multimedia, …. Source. Receiver. The problem.

nira
Download Presentation

The problem

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation ofpacket video qualityin real timeusingRandom Neural NetworksSamir MohamedGerardo RubinoIRISA, Rennes, FRANCE

  2. stream of voice, music, video, multimedia, … Source Receiver The problem • How to automatically quantify the quality of the stream, as perceived by the receiver? IP network Workshop RNN, ICANN’02, Madrid, 27/8/02

  3. OUTLINE • Subjective tests • Objective tests • Our approach in 5 steps • The obtained performances • ANN vs RNN • Application: analyzing parameters impact • A view of our demo tool • Related work in audio • Ongoing research Workshop RNN, ICANN’02, Madrid, 27/8/02

  4. 1: The “voie royale” • To quantify the quality at the receiver side … • just put a human there, • give her/him a scale (say, from 1 to 10), • and ask her/him to evaluate the qualityof a (short) part of the stream (say, some seconds). Workshop RNN, ICANN’02, Madrid, 27/8/02

  5. (1:) The “voie royale”(cont.) • Better: • put n humans at the receiver side, • ask them to evaluate a (short) given sequence, • take the average as the quantified quality. • Still better: • give to the set of humans the sequence to evaluateplus several other sequences, in order to allow each member to adjust her/his scale Workshop RNN, ICANN’02, Madrid, 27/8/02

  6. (1:) Subjective tests • The previous procedure is in fact standardized:see, for instance, the norm ITU-R BT.500-10 (March 2000). • Basically, the procedure is as previously described + a statistical filter of the results. • Goal of the statistical filter:to eliminate bad observers. • Bad observer:“one disagreeing with the majority” Workshop RNN, ICANN’02, Madrid, 27/8/02

  7. (1:) Drawbacks • The previous procedure gives “the truth”but doesn’t do the job automatically (so, a fortiori, not in real time neither). • Moreover, performing subjective tests is costly • in logistics (you need about 20 people with appropriate characteristics) • in resources (you need an appropriate place or environment to put your set of human subjects) • and in time to perform the tests. Workshop RNN, ICANN’02, Madrid, 27/8/02

  8. (1:) Drawbacks (cont.) • Suppose you decide to analyze the way some factor (the loss rate in the network, for instance) affects quality: • you must then evaluate the function q = f(lr) where q is the quality and lr is the loss rate in many points and for many sequences: VERY EXPENSIVE. • Suppose you want now to study q = g(lr,br,d) where br is the source bit rate and d the mean end-to-end delay: TOO EXPENSIVE TO BE DONE Workshop RNN, ICANN’02, Madrid, 27/8/02

  9. 2: Another possibility • The other way is obviously to look for an explicit formula (or algorithm) giving q as a function of the considered factors (lr, d, etc.) in, say, O(1) time. • This appears to be a formidable task, because of two factors: • no formal definition of what we want to quantify (quality) is available • the “intuitive concept”of quality depends a priori on too many factors and in a complex way Workshop RNN, ICANN’02, Madrid, 27/8/02

  10. (2:) Objective tests • There is an area called “objective tests” where different procedures are proposed to evaluate the degradation of a sequence. • They consist of specific metrics that compare the original and the degradated streams. • Even if this is not the goal here, the results of these objective tests are not satisfactory so far. • How do we know? by comparing the results obtained using these techniques with “the truth”, that is, with those coming from subjective tests. Workshop RNN, ICANN’02, Madrid, 27/8/02

  11. (2:) An example • The MNB metric: • developped at the US Dep. of Commerce, 1997 • for voice (VoIP applications) • based on a cognition model • has been shown to behave correctly in some cases • In next slide, some results of a test against subjective evaluations(from T. A. Hall, Objective speech quality measures for Internet telephony, Proc. of SPIE 2001). Workshop RNN, ICANN’02, Madrid, 27/8/02

  12. (2:) Performance of MNB 2 Workshop RNN, ICANN’02, Madrid, 27/8/02

  13. (2:) The only exception • There is one trial of measuring quality without referring to the original sequence: ITU “E-model”www.itu.int • It has been proposed for the specific case of VoIP applications. • However, the comparison with subjective evaluation says that the performance of this metric can be very poor (in next slide, some results from T. A. Hall, Objective speech quality measures for Internet telephony, Proc. of SPIE 2001). Workshop RNN, ICANN’02, Madrid, 27/8/02

  14. (2:) Performanceof the E-model Workshop RNN, ICANN’02, Madrid, 27/8/02

  15. (2:) An example in video:the ITS metric Source: S. Voran and S. Wolf, The development and evaluation of an objective video quality assesment system that emulates human viewing panels, in IBC 1992. Workshop RNN, ICANN’02, Madrid, 27/8/02

  16. 3a: Our approach: first step • Our goal is to take, in some sense, the best of subjective and objective approaches. • As a first step, we select a set of factors or parameters that we think are important to the final perceived quality. • Even if this is an a prioritask, it is not a difficult one. • For each parameter, we select a few representative or important values.Our goal here is to discretize the problem. Workshop RNN, ICANN’02, Madrid, 27/8/02

  17. (3a:) Our approach: first step (cont.) • In our case, we selected 5 parameters: • BR: Bit Rate.The normalized rate of the encoder.In our environments, we considered 4 possible values (256, 512, 768 and 1024 KBps), normalized with respect to the maximal value. • FR: Frame Rate, in fps (frames per sec).The rate at which the original stream is encoded.Four selected values: 6, 10, 15 and 30 fps. Workshop RNN, ICANN’02, Madrid, 27/8/02

  18. (3a:) Our approach: first step (cont.) • LR: loss rate, in % (loss probability).Selected values: 0, 1, 2, 4 and 8 %. • CLP: number of consecutive loss packets.We consider packets dropped in bursts of size 1, 2, 3, 4 or 5. • RA: ratio between intra macro-blocs to inter macro-blocs.It (indirectly) measures the redundancy in the sequence.We used for this parameter five values between 0.05 and 0.45. Workshop RNN, ICANN’02, Madrid, 27/8/02

  19. (3a:) Our approach: first step (cont.) • Observe that there can be interactions (in general, complex) between the parameters • for instance, here, RA is used by the encoder to protect the stream against losses (measured by LR) • It must also be underlined that our approach does not depend on the specific set of chosen parameters. • Observe that BR, FR and RA are source parameters, and that LR and CLP are network parameters. Workshop RNN, ICANN’02, Madrid, 27/8/02

  20. 3b: Our approach: second step • We have a set of parameters P = {1, 2, …, P} • For parameter i we have a set of possible values: i  {vi1, vi2, …} • Any vector of the form (v1i, v2j , …, vPk) is then called a configuration. Workshop RNN, ICANN’02, Madrid, 27/8/02

  21. (3b:) Our approach: second step (cont.) • In our experiments, we had P = 5 parameters • The # of selected values for them were respectively 4, 4, 4, 5, 5, leading to 44455 = 1600 configurations • Call Cthe set of all possible configurations. • The second step consists of selecting a reduced part of C, trying to have a “good coverage”of this set. Workshop RNN, ICANN’02, Madrid, 27/8/02

  22. (3b:) Our approach: second step (cont.) • In our experiments, we selected about 100 configurations. • The method we followed was not to use something like a low discrepancy sequence of points in a hypercube, but • to take into account the characteristics of the parameters • and the extreme values. • Call SCthe set of selected configurations Workshop RNN, ICANN’02, Madrid, 27/8/02

  23. 3c: Our approach: third step • For the third step, we must be able to reproduce an environment (source + network) where the selected parameters can be put in any configuration we want. • In our case, we achieved this using simulators and appropriated controlled (lab conditions) networks. • The third step consists of reproducing each configuration  from the set SC and to send a fixed original sequence  from source to receiver. Workshop RNN, ICANN’02, Madrid, 27/8/02

  24.   1 2 3 1 2 3 (3c:) Our approach: third step (cont.) SC = {1, 2, … } Workshop RNN, ICANN’02, Madrid, 27/8/02

  25. 3d: Our approach: fourth step • The result is a set of versions of , each having encountered different conditions at the source and in the network. • We have now a set of sequences (in our tests, about 100 sequences), and a configuration associated with each one. • The fourth step consists of performing a standard subjective test on each sequence, to build a value assumed to be, by definition, the quality of the sequence. Workshop RNN, ICANN’02, Madrid, 27/8/02

  26. (3d:) Our approach: fourth step (cont.) • In symbols, we have S sequences {1,…, S} and, associated with sequence i, the configuration (xi1, xi2 ,…, xiP). • In other words, xikis the value of the kth parameter which led to the sequence i at the receiver. • Together with this, we have the quality value of each version, coming from the subjective test;i is the value of i. Workshop RNN, ICANN’02, Madrid, 27/8/02

  27. (3d:) Our approach: fourth step (cont.) Workshop RNN, ICANN’02, Madrid, 27/8/02

  28. 3e: Our approach: fifth (last) step • The S sequences are now divided into two sets, randomly. For instance, our 100 sequences were divided into one set of about 80, and one set of about 20. • To simplify, assume we renumber things such that the first set is {1, …, K}. • The idea is then to train a Neural Network (NN) to learn “the”function with P inputs and 1 output, which associates with the input (xk1, xk2 ,…, xkP) the output k, for k = 1,…,K. Workshop RNN, ICANN’02, Madrid, 27/8/02

  29. (3e:) Our approach: fifth (last) step (cont.) • The second set of sequences is then used to validate the obtained NN (standard approach). • The hope is the following: if we give to the function an input (x1, x2 ,…, xP) which is not in the data base (taht is, not in SC), the output y should be close to the subjective evaluation of any sequence degradated through a system (source + network) where the configuration was (x1, x2 ,…, xP). Workshop RNN, ICANN’02, Madrid, 27/8/02

  30. (3e:) Our approach:key implicit assumption • Our approach implies the following implicit assumption: For any sequence  (or for any sequence  belonging to a given family or class),the subjective quality depends onlyon the values of the set of chosen characterizing parameters. Workshop RNN, ICANN’02, Madrid, 27/8/02

  31. (3e:) Our approach:key implicit assumption (cont.) • This means in particular that given, say, three video sequences perhaps with very different contents, if the system configuration is the same (source bit rate, loss rate, …) then the perceived quality will be roughly the same. Workshop RNN, ICANN’02, Madrid, 27/8/02

  32. stream of voice, music, video, multimedia, … Source Receiver (3e:) Our approach at work IP network NN asking the source for BR, FR, RA measuring LR, CLP Workshop RNN, ICANN’02, Madrid, 27/8/02

  33. Receiver BR source parameters module FR NN RA network parameters module LR CLP (3e:) Our approach at work (cont.) measure of quality Workshop RNN, ICANN’02, Madrid, 27/8/02

  34. 4: Our results in a nutshell • Once trained, the NN is supposed to behave like an average human observer face to a received sequence. • Observe that the performance of the method depends on the selected parameters, but that there is no restriction in such a choice. • Moreover, if a posteriorisome new parameter appears to be important, it can be easily added following the same procedure. Workshop RNN, ICANN’02, Madrid, 27/8/02

  35. (4:) Our results in a nutshell (cont.) • The trained NN (classical and RNN) correlated remarquably well against human evaluations. • They (obviously) run in negligible time. • They allow us to study the behavior of quality as a function of several parameters. • We did the same work for audio, with the same results. • We also developped an example of application using our tool for control purposes (in audio). Workshop RNN, ICANN’02, Madrid, 27/8/02

  36. (4:) Performances (RNN) during the training phase Workshop RNN, ICANN’02, Madrid, 27/8/02

  37. (4:) Another view Workshop RNN, ICANN’02, Madrid, 27/8/02

  38. (4:) Performances (RNN) during the validation phase Workshop RNN, ICANN’02, Madrid, 27/8/02

  39. (4:) Another view Workshop RNN, ICANN’02, Madrid, 27/8/02

  40. 5: ANN vs RNN • We used the toolbox of MATLAB implementing several standard Neural Networks techniques (Artificial Neural Networks, ANN) and a specific software for RNN. • We compared their performances mainly in the respective learning abilities. • We will present here some of the obtained results, which show that RNN behave better for our applications. Workshop RNN, ICANN’02, Madrid, 27/8/02

  41. (5:) An interesting example:left: RNN right: ANN Workshop RNN, ICANN’02, Madrid, 27/8/02

  42. (5:) RNN vs. ANN:# of hidden neurons Workshop RNN, ICANN’02, Madrid, 27/8/02

  43. (5:) RNN vs. ANN: # of hidden neurons Workshop RNN, ICANN’02, Madrid, 27/8/02

  44. 6: Analyzing parameters impact Workshop RNN, ICANN’02, Madrid, 27/8/02

  45. (6:) Analyzing parameters impact (cont.) Workshop RNN, ICANN’02, Madrid, 27/8/02

  46. (6:) MPQM (well-known metric) when applying losses Workshop RNN, ICANN’02, Madrid, 27/8/02

  47. (6:) ITS (well-known metric)when bit rate is varied Workshop RNN, ICANN’02, Madrid, 27/8/02

  48. 7: Our demo tool Workshop RNN, ICANN’02, Madrid, 27/8/02

  49. 8: Our work oncontrol schemes for audio Workshop RNN, ICANN’02, Madrid, 27/8/02

  50. BW needed by PCM BW needed by PCM BW needed by GSM BW needed by GSM (8:) Some simulation results Workshop RNN, ICANN’02, Madrid, 27/8/02

More Related