Effects of Network Congestion (Packet Loss) on Video Streaming – A User Study

1. Effects of Network Congestion (Packet Loss) on Video Streaming – A User Study Rahul Amin, France Jackson, Morris (Trey) Lee, Jim Martin, Juan Gilbert Last Updated: May 24, 2013

2. Objective • The goal of the study was to test the following hypotheses: • Is it beneficial for video content providers to start the video at a lower quality so as to set low user expectations and eventually move to a little higher quality? • If the user expectation is set low by a low video quality in the beginning, are the users ok with a sub-par quality for the rest of the video? • Do users have different video quality expectations if they are told that the video they are watching is online content vs. TV Cable provider content?

3. Methodology – Pilot Study • To determine the different levels of video quality, a pilot study with different packet loss settings was run • The goal of this study was to capture the packet loss settings where video quality degraded but no buffering screen message was received • At 12% packet loss, the video kept buffering every few seconds. So the the worst video quality loss setting had to be less than 12% packet loss. • After getting an opinion from several participants, the following video quality-packet loss mappings were determined • Bad Quality – 9% Packet Loss • OK Quality – 6% Packet Loss • Good Quality – 0% Packet Loss

4. Methodology – Actual Study • A 10-minute movie clip streamed using Netflix was recorded with 3 different network congestion settings • Setting 1 (Bad-to-Good): 9% Packet Loss for first 5 minutes, 0% Packet Loss for last 5 minutes • Setting 2 (Good-to-Bad): 0% Packet Loss for first 5 minutes, 9% Packet Loss for last 5 minutes • Setting 3 (Bad-to-OK): 9% Packet Loss for first 5 minutes, 6% Packet Loss for last 5 minutes

5. Demographic Breakdown

6. Mean Opinion Score Results – On Demand Study • Based on the results, bad-to-ok setting has the highest ‘overall’ MOS. So, starting the video quality low and then improving it a little satisfied most of the users. • Starting at low quality and bumping up the quality to maximum level (bad-to-good) does not help with the ‘overall’ MOS because the participant sees the actual difference on what he/she should have obtained (end compared to beginning) • Good-to-Bad setting also frustrates the viewer and results in about same ‘overall’ rating as Bad-to-Good setting

7. Mean Opinion Score Results – Online Study • Based on the results, bad-to-ok setting again has the highest ‘overall’ MOS. So again, starting the video quality low and then improving it a little satisfied most of the users. • The overall MOS for each setting is lower than that of the ‘On-Demand’ study as users have a pre-conceived notion of ‘online’ quality generally being not too good. This is more apparent in the ‘willingness to pay’ result presented in the next slide.

8. Mean Opinion Score Results – Willingness to Pay for the Content • ‘On-demand’ or ‘online’ content makes a huge difference in participant’s willingness to pay. If the content is online, the participants generally do not want to pay too much for it; whereas the participants are willing to pay a little more. • This clearly shows that viewer’s have different expectations for on-demand vs online content. This will have an effect on how the participants answered some of the questions.

9. Mean Opinion Score Results • There isn’t much variation in MOS for all the settings for this question. Perhaps once users see some degradation (beginning/end) which is the case for all settings, they are always going to be a little likely to change service providers.

10. Mean Opinion Score Results • This question was asked at the very end after asking questions about ‘video’ as well as ‘audio’ quality. • The bad-to-ok on-demand setting again has the highest MOS; however, bad-to-good on-demand/online settings have the lowest MOS!!

11. Oral Question Responses • Participants were asked an opinion question at the very end: would they prefer the quality of video be a little bad at the beginning or end? Or would any degradation be unacceptable to them? • Most of the participants said they would not tolerate any degradation (45% - None). • Majority of Participants who would tolerate a little bit of degradation would prefer the degradation in the beginning (33%) as opposed to the end (22%). • Remark: Some participants stated different responses based on if they were paying for the content. If they were paying for it, they would not accept any degradation; but if it was free, then some degradation in the beginning or end was acceptable (which is what is plotted in the pie chart)

12. General Conclusions • If the user expectation is set low by a low video quality in the beginning, bumping up the quality a little (bad-to-ok) setting results in a higher MOS than the case where the quality is bumped up all the way (bad-to-good) • Overall MOS results for ‘On-Demand’ are a little higher than ‘Online’ studies; perhaps due to the viewer expectation being low for online quality video