Perceptual Video Metrics, a new vocabulary for QoE Jeremy Bennington Cheetah Technologies
Video Quality Management QoS QoE PQM How well is the network performing Did the packets get from point A to point B Content agnostic Quality of Service What is the network impact on the viewer Statistically estimated from QoS & encoding information Content aware, but does not measure the quality of the content Quality of Experience What is the viewer s perception of quality Measures content using the model of the human eye Network agnostic, can be used ingest, encode, network & client Perceptual Quality Monitoring
What is a Mean Opinion Score (V-Factor)? Source Issue Headend Issue Network Issue V-FACTOR V-FACTOR 5 Imperceptible 5 Imperceptible 4 Perceptible 4 Perceptible 3 Slightly Annoying 3 Slightly Annoying 2 Annoying 2 Annoying 1 Very Annoying 1 Very Annoying
Real World Case Studies Gradual Degradations 12 Hours 3 Hours V-Factor MOS shows a significant change in quality over 12hours Issue with the encoding from the programmer V-Factor MOS shows a significant change in quality over 3hours Issue with the program (like an old movie) Gradual degradations of visual quality must be trended over a longer period before alarming Only monitoring using perceptual metrics can identify these events
Real World Case Studies Sudden Degradation Sudden drops in quality are common and un-related to MPEG-TS or VCL These events should be alarms for operations These events (when seen at the ingest) should be used to screen trouble tickets & truck rolls
Quality Trends Week over Week with threshold violations Consistent week-over-week average quality Events are easy to find with a threshold violation Useful in determining problematic programs and reducing trouble tickets
Quality Trends Week over Week with periodic trend Consistent week-over-week periodic quality Events are hard to find with a threshold violation Correlation and moving average are easiest ways to find errors Pattern caused by different programming from midnight to 6am
Quality Trends East versus West Feeds Consistent east-to-west periodic quality Correlation is useful for determining errors in East feed Could be used to catch errors before they hit the west
So What Junk in, junk out If what your receive is poor quality only perceptual quality tools will see it...and your customers Poor quality passes thru the system to all the viewers Perceptual quality monitoring identifies content issues to compliment QoE/QoS Perceptual Quality Monitoring is essential to both traditional video delivery and newer streaming techniques
So What Working with Programmers Scorecards: Provide each a ranking of their channel(s) weekly & monthly Identify periodic quality issues: paid programming, shows, remotes Empowering Operations Is the issue the programmer or in our network? Is the issue the encoding or distribution? Is the issue visible to the subscriber? How often does this happen? Reducing Cost CSRs: We are aware of the problem CSRs: This is typical for this time of day CSRs & Operations: Reduction in trouble tickets
Summary Perceptual Quality Monitoring (PQM) Models the human eye to find visual artifacts Identifies baseline quality & trends; and alerts at ingest & encode Replaces manual methods of video monitoring at the Headend End to End Video Management QoS to determine network fidelity QoE to estimate QoS and MPEG impacts on subscribers PQM to objectively determine video quality QoS, QoE and PQM correlation achieves: Prioritization of issues based on impact Faster time-to-fix and isolation of issues Reduction in costs & increases customer satisfaction