Subjective quality and HTTP adaptive streaming: a review of psychophysical studies

Transcription

1 Subjective quality and HTTP adaptive streaming: a review of psychophysical studies Francesca De Simone, Frédéric Dufaux ; Télécom ParisTech; CNRS LTCI

2 Content Basic concepts Quality of Service (QoS) vs Quality of Experience (QoE) Subjective vs objective assessment QoE and DASH Subjective QoE and DASH: what do we know Perception of stalling events Perception of quality switches Tradeoffs Conclusions 2

3 Basic Concepts 3

4 QoS vs QoE Example of Quality of Service (QoS) metrics Available bandwidth Round Trip Time (RTT) Packet loss rate (PLR) Etc.. Quality of Experience (QoE)? 4

5 QoS vs QoE Example of Quality of Service (QoS) metrics Available bandwidth Round Trip Time (RTT) Packet loss rate (PLR) Etc.. Quality of Experience (QoE)? [Qualinet WhitePaper 2012] Price Visual signal MULTIMEDIA SERVICE Audio signal Novelty Interface etc CONTEXT Expectations Human Visual System USER Emotions Auditory System Attention etc multimedia experience 5

6 Subjective vs objective assessment Subjective quality assessment In controlled environment standard methodologies for video designed for analog television [ITU Rec BT ] adapted for multimedia applications [ITU Rec. P.910] In uncontrolled environment crowdsourcing field experiments users feedback from real products 6

7 Subjective vs objective assessment Objective quality assessment Full-Reference (FR) Reduced-Reference (RR) No-Reference (NR) Input/Reference signal Image/ video processing Output/Processed signal FR METRIC Input/Reference signal Image/ video processing Output/Processed signal Feature extraction RR METRIC Input/Reference signal Image/ video processing Output/Processed signal NR METRIC Example of standardized solutions FR video quality metric for HDTV for digital cable television [ITU-T Rec. J.341] NR model for audiovisual quality assessment in IPTV [ITU-T Rec. P ] 7

8 QoE and DASH 128 kbps 500kbps 1Mbps HTTP server TCP transport layer DASH client Terminology Chunk Adaptation Set Realization Realization Realization 128 kbps 500kbps 1Mbps time (s)

9 QoE and DASH No packet losses Reliable transport layer Stalling events Playout interruptions due to video player buffer underflow Quality switches Quality variation due to switch between representations 9

10 QoE and HAS No packet losses Reliable transport layer Stalling events Playout interruptions due to video player buffer underflow Quality switches Quality variation due to switch between representations Not addressed in traditional video QoE research! 10

11 Goal of this talk Identify key questions concerning subjective video quality perception and DASH Review existing answers Perception of video stalling Perception of video quality switches Tradeoffs 11

12 Subjective QoE and DASH: what do we know 12

13 Perception of stalling events Stallings should be avoided at all times [Dobrian2011, Mok2011, Floris2012, Atzori2013, Hobfeld2013] but if they occur what is the impact of stalling frequency and duration? 13

14 Perception of stalling events Stallings should be avoided at all times [Dobrian2011, Mok2011, Floris2012, Atzori2013, Hobfeld2013] but if they occur what is the impact of stalling frequency and duration? Studies before DASH [Pastrana-Vidal2004, Staelens2010] Studies on DASH [Moorthy2011, vankester2011] 14

15 Perception of stalling events Conclusions Threshold to detect stalling: 80ms [Pastrana-Vidal2004] Subjects prefer a single stalling of longer duration compared to multiple short stallings [Moorthy2011] Subjects prefer regular stallings over irregular ones [Moorthy2011] Stallings up to 400ms can still be tolerated in the case of few stalling events and long video sequences [vankester2011] [Staelens2010] 15

16 Perception of quality switches Assumptions Each representation corresponds to a different bitrate Representations may differ in terms of spatial resolution, temporal resolution and encoding quantization settings Three kinds of switches are possible Encoding switch (ESW) Spatial switch (SSW) Temporal switch (TSW) 128 kbps 500kbps 1Mbps 16

17 Perception of quality switches What is the impact of switching amplitude and frequency? 17

18 Perception of quality switches What is the impact of switching amplitude and frequency? Question 1: shall the bitrate be reduced (increased) once with a big quality drop or gradually? Layer or Rate Layer or Rate time time 18

19 Perception of quality switches What is the impact of switching amplitude and frequency? Question 1: shall the bitrate be reduced (increased) once with a big quality drop or gradually? Layer or Rate Layer or Rate time time Studies on layer encoded videos and SVC [Zink 2003, Ni 2011] Studies on HAS [Moorthy 2012, Mok 2012, Yitong 2013] 19

20 Perception of quality switches Conclusions Smooth variations preferred to abrupt variations Amplitude of the variation must be kept as small as possible Examples of amplitude thresholds to deliver generally acceptable quality [Ni 2011] Quality differences should not exceed 4 QPs for ESW, one third of the original frame rate for TSW, half the original size for SSW 20

21 Perception of quality switches Conclusions Smooth variations preferred to abrupt variations Amplitude of the variation must be kept as small as possible Examples of amplitude thresholds to deliver generally acceptable quality [Ni 2011] Quality differences should not exceed 4 QPs for ESW, one third of the original frame rate for TSW, half the original size for SSW But Conclusion might not be generalizable to switches between levels exhibiting small quality separation [Moorthy 2012] 21

22 Perception of quality switches Question 2: Do users prefer shorter but more frequent variation or longer but less frequent variations? 22

23 Perception of quality switches Question 2: Do users prefer shorter but more frequent variation or longer but less frequent variations? [Zink 2003, Ni 2011] layer encoded videos and SVC [Moorthy 2012, Robinson 2012, Yitong 2013] Layer or Rate Layer or Rate time time 23

24 Perception of quality switches Conclusions Less frequent quality variations are preferred to more frequent variations Frequency of variations should be kept as small as possible For SNR or spatial resolution switches, low frequency can relieve the annoyance of quality switch [Ni 2011] 24

25 Perception of quality switches Conclusions Less frequent quality variations are preferred to more frequent variations Frequency of variations should be kept as small as possible For SNR or spatial resolution switches, low frequency can relieve the annoyance of quality switch [Ni 2011] But More frequent switches are preferred over fewer switches, if the subject is able to view the highest quality video for longer duration [Moorthy 2012] For temporal resolution switches, the frequency does not seem to have significant influence on QoE [Ni 2011] 25

26 Perception of quality switches Question 3: is it better to switch quality levels or to stay at a lower quality? i.e. to switch or not to switch? 26

27 Perception of quality switches Question 3: is it better to switch quality levels or to stay at a lower quality? i.e. to switch or not to switch? [Zink 2003, Ni 2011] layer encoded videos and SVC [Moorthy 2012, Robinson 2012, Mok 2012, Yitong 2013, Villa 2013, Rehman 2013, Tavakoli 2014] Layer or Rate Layer or Rate time time 27

28 Perception of quality switches Conclusions Constant quality preferred to varying quality Short-term spikes degrade QoE [Yitong 2013] Constant (lower) quality better than decreasing quality (from higher to lower) [Zink 2003] Constant or nearly constant quality is preferable to frequently varying quality (even if mean quality is lower) [Robinson 2012] [Ni 2011] Providing as high bitrate as possible does not necessarily lead to the highest QoE [Mok 2012] 28

29 Perception of quality switches But Preferable to switch to a higher rate, if the duration of the higher rate is at least half the duration of the lower rates [Moorthy 2012] If constant quality is too low, any adaptation is preferred [Tavakoli 2014] [Ni 2011] Maintaining a reasonable quality for longer duration results in a small bias towards better subjective experience [Rehman 2013] 29

30 Perception of quality switches Temporal trend The end quality of the video makes a definite impact on the perceived quality: all is well that ends well [Zink 2003][Moorthy 2012] [TavakoliSPIE2014] For increasing quality, overall results did not show any statistically significantly difference between different strategies. For the decreasing scenarios, there was a statistically significantly preference to gradual change with 10 seconds chunk compared to the other scenarios. Content dependency [NiMMsys2011] [RobinsonBellTJ2012] Effect of switch of spatial and temporal resolution varies on different content types, even for the same amplitude difficult to spot quality oscillations when frequent scene changes in the scene. Quality change is more noticeable in steady, pan, tracking shots and when there are number of hard edges. More complex texture details, more strongly affected by the loss of spatial resolution The lack of relative movement between objects in the scene may limit the visible effect of frame dropping. 30

31 Perception of quality switches When the bitrate has to be lowered (increased), for instance due to restricted (better) network conditions, Question 1: is it better to reduce (increase) encoded bitrate, frame rate, or resolution? 31

32 Perception of quality switches When the bitrate has to be lowered (increased), for instance due to restricted bandwidth (better network conditions), Question 1: is it better to reduce (increase) the encoded bitrate, the video frame rate, or the video resolution? Many subjective studies on optimal combinations (SNR, spatial resolution, temporal resolution) for scalable video coding [Lee 2012] H.264/SVC encoded versions at same bitrate Stronger quantization 1/16 of the original resolution upscaled 32

33 Perception of quality switches Conclusions Spatial quality more important than frame rate at low bit rates and viceversa at high bit rates Low vs high bit rate threshold higher for content with faster motion Among quality switch of SNR levels, spatial resolutions or temporal resolutions, SNR switch is the most recommended as less noticeable [Ni 2011] 33

34 Perception of quality switches Methodological findings: [RehmanQomex2013] Subjects are resistent in updating their opinions: when there is a small quality variation between consecutive scenes, subjects tend to keep their opinions or change only slightly. [RehmanQomex2013] [ChenIP2014] Subjects use asymmetric strategies in updating their opinions. A significant quality degradation between consecutive scenes results in a large penalty, as compared to the reward obtained by a similar quality improvement between consecutive scenes. Behavioral response of viewers to quality variation is more sensitive in low quality region than in high quality region. [ChenIP2014] The QoE of a video at a particular moment depends on the viewing experience before the moment: the current video quality can affect the perceived quality in the next 15 second [ChenIP2014] A viewer suffering a previous unpleasant viewing experience tends to penalize the perceived quality in the future [VillaNIK2013] Significant differences in terms of self-reported pleasure, arousal, and the degree of delight between the two usage scenarios (user alone vs group of users). All significantly higher in the first usage scenario. 34

35 Tradeoffs: initial delay vs stalling Stallings can be avoided by employing larger client buffers but this will affect the initial startup delay of the video, so Question 2: what is the impact of initial loading? Question 3: what is the impact of combined effect (stalling + initial loading)? 35

36 Tradeoffs: initial delay vs stalling Stallings can be avoided by employing larger client buffers but this will affect the initial startup delay of the video, so Question 2: what is the impact of initial loading? Question 3: what is the impact of combined effect (stalling + initial loading)? [Hobfeld2013, HoBfeldQomex2012, Staelens2010] 36

37 Tradeoffs: initial delay vs stalling Conclusions Initial startup delay does not significantly influence quality perception [Hoßfeld 2013] End-users are willing to tolerate larger startup delays, if this results in less video stallings [Staelens 2010] Different amounts of startup delay are tolerated, depending on the specific type of application [HoBfeld 2012] 37

38 Conclusions 38

39 Conclusions Agreed upon conclusions: Frequency of stalling is the main factor affecting QoE The amplitude of the quality switch is the most dominant factor for the perception of quality switch (more than the frequency) Issue: How generalizable are the findings of these studies? 39

40 Challenges Why is it difficult to design a subjective test for HAS? Temporal variation of quality Longer sequences may be needed: which test method to use? Variety of terminals in the real scenario Different network conditions Different users expectations 40

41 Thank You for your Attention! Thanks to: Marie-Neige Garcia and Alexander Raake (T-Labs TU Berlin), Nicolas Staelens (Ghent University), Samira Tavakoli (Universidad Politecnica de Madrid), Sebastian Egger (ftw), Kjell Brunnstrom (Acreo) 41

42 Further reading and resources 1. Qualinet White Paper on Definitions of Quality of Experience (2012), European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003), Patrick Le Callet, Sebastian Möller and Andrew Perkis, eds., Lausanne, Switzerland, Version 1.1, June 3, Zink et al., Subjective impression of variations in layer encoded videos, ACM R.R. Pastrana-Vidal, J.C. Gicquel, C. Colomes, and H. Cherifi, Sporadic frame dropping impact on quality perception, in Proceedings of SPIE, 2004, vol. 5292, p Q. Huynh-Thu and M. Ghanbari, Temporal Aspect of Perceived Quality in Mobile Video Broadcasting, IEEE Transactions on Broadcasting, vol.54, no.3, pp , Sept J.-S. Lee, F. De Simone, N. Ramzan, E. Izquierdo and T. Ebrahimi, Quality assessment of multidimensional video scalability, in IEEE Communications Magazine, vol. 50, num. 4, p , J.-S. Lee, F. De Simone and T. Ebrahimi, Subjective quality evaluation via paired comparison: application to scalable video coding, in IEEE Transactions on Multimedia, vol. 13, num. 5, p , Venkataraman et al., Effects of Internet path selection on video QoE, MMSys Chen et al., Quantifying QoS requirements of network services: a cheat-proof framework, MMSys R. Mok, E. Chan, and R. Chang. Measuring the quality of experience of HTTP video streaming. In Proc. IEEE/IFIP 10. IM (pre-conf.), R. Mok, E. Chan, X. Luo, and R. Chang., Inferring the QoE of HTTP Video Streaming from User-Viewing Activities, ACM W- MUST 11, Zhou Wang; Bovik, A.C., "Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures," Signal Processing Magazine, IEEE, vol.26, no.1, pp , Jan Kusching at el., An evaluation of TCP-based rate-control algorithms for adaptive internet streaming of H.264/SVC, MMSys Lederer at el., Dynamic adaptive streaming over HTTP dataset, MMSys K. Seshadrinathan and A. C. Bovik, Motion Tuned Spatio-temporal Quality Assessment of Natural Videos, vol. 19, no. 2, pp , IEEE Transactions on Image Processing, Feb NTIA VQM Yim et al., Evaluation of temporal variation of video quality in packet loss network, Signal Processing Image Communication, Singh, K.D.; Hadjadj-Aoul, Y.; Rubino, G.;, "Quality of experience estimation for adaptive HTTP/TCP video streaming using H.264/AVC," Consumer Communications and Networking Conference (CCNC), 2012 IEEE, vol., no., pp , Jan