Subjective Test Methodology Design for Perceptual Quality Optimization

Transcription

1 Subjective Test Methodology Design for Perceptual Quality Optimization

2 Table of Contents INTRODUCTION 3 SUBJECTIVE TEST DESIGN 4 PHYSICAL ENVIRONMENT 4 DATA PRESENTATION AND SCORING SCHEME 5 SELECTION OF SUBJECTIVE TEST CONTENT 7 REPRESENTATIVE CONTENT: CLASSIFYING BASED ON SEMANTIC CATEGORIES 7 REPRESENTATIVE CONTENT: CLASSIFYING BASED ON MOTION AND COMPLEXITY SECTORS8 QUANTIFYING MOTION AND COMPLEXITY SECTORS 9 SUITABLE CONTENT FOR SUBJECTIVE TESTING 10 DETERMINATION OF CHALLENGING PERFORMANCE POINTS USING VIDEO QUALITY BREAKDOWN BITRATE 12 DEFINING THE VIDEO QUALITY BREAKDOWN BITRATE 12 RELATIONSHIP OF VQBB TO SUBJECTIVE TEST SCORING 14 USING VQBB TO QUANTIFY COMPRESSION GAINS 16 POST-TEST ANALYSIS 18 SUBJECT RELIABILITY METRICS 18 CALCULATING COMPRESSION GAIN 19 SUMMARY 23 EuclidIQ White Paper 2

3 Requirements for Subjective Testing Subjective Test Methodology Design for Perceptual Quality Optimization Introduction EuclidIQ has been developing an innovative compression technology called IQ264 that applies perceptual quality optimization (PQO), integrating consideration of the human visual system (HVS), to improve standard H.264 encoding. Because IQ264 optimizes human perception, its gains are best measured through subjective testing using human subjects. Because of the time and expense associated with subjective testing, the development of objective metrics to approximate human perception remains an active research area; see, for example, the survey paper by Chikkerur i. However, no objective metric to date has captured human perception well enough to be universally agreed upon. To achieve the gold standard of subjective testing, and to avoid the inherent limitations of objective metrics, EuclidIQ has designed a practical subjective test methodology that produces meaningful results without being prohibitive in terms of time or cost. This whitepaper details the components of that subjective test methodology: the physical design of the test, the data presentation and scoring scheme, the selection of representative and suitable data, the determination of appropriate operating points, and the post-test analysis. An effective subjective test methodology needs to be credible, adhering closely enough to accepted standards in all essential areas; feasible, not prohibitive in terms of time or cost; and meaningful, run over a broad enough set of data and performance points that the conclusions of the test are definitive. To achieve credible subjective testing results, results that would be accepted as valid and believable in the video compression community, the methodology needs to adhere closely enough to accepted standards in all essential areas. Accepted standards documents include the ITU-R BT.500 recommendation ii for subjective video quality analysis (VQA) of television video, the ITU-T P.910 recommendation iii for subjective VQA in multimedia applications, and the ITU-T P.913 recommendation iv for subjective VQA of both Internet and distributed television video in any environment. Unlike the BT.500 and P.910 recommendations, which describe VQA for controlled environments (broadcast or pay TV signals over reliable networks transmitted to an immobile screen in a quiet and non-distracting environment), the P.913 recommendation recognizes a new paradigm of video watching that includes on-demand video over unreliable networks, transmitted to a variety of devices, many of which are mobile, in often-distracting environments. Thus, P.913 allows some flexibility in the test setup (including the physical environment, test type, and scoring method), depending on the purposes of the test. However, P.913 recommends at least 35 subjects for public environment tests, compared to 24 subjects for controlled environment tests. Thus, constructing a feasible subjective testing methodology given time and resource constraints requires a balance between stricter adherence to standards for EuclidIQ White Paper 3

4 controlled environment testing and more subjects for public environment testing. For subjective test results to be meaningful, the test must be run over a broad enough set of test videos and conditions for the conclusions of the test to be definitive. For EuclidIQ, the subjective tests must also occur frequently enough for its R&D team to analyze the subjective test results and refine the compression algorithms under evaluation. As one might observe, the credibility, feasibility, and meaningfulness requirements conflict, because setting up a fully credible subjective test that strictly adheres to accepted standards, whether in a controlled environment or a public environment, requires too much time and results in too few data points to provide the frequent feedback needed for algorithm development. Thus, EuclidIQ has developed two types of subjective test methodologies with different goals. External tests are more formal and less frequent, conducted in a controlled environment with close adherence to accepted standards and designed to produce results that can be credibly reported publicly. Internal tests are less formal and more frequent, constructed to approximate accepted standards and designed to produce results with sufficient frequency and breadth to aid algorithm development. In line with their respective goals, external tests are also designed to reflect public opinion and scoring by non-expert viewers, while the internal tests are designed to reflect the opinion and scoring of expert viewers. Subjective Test Design Physical Environment For external tests, EuclidIQ converted an interior room in the company offices in Concord, MA into a video viewing room (VVR) that met various requirements in the BT.500 and P.910 standards v. To help meet the various luminance requirements in the standards, the VVR s single glass window was blocked off and its walls painted gray, and two torch lights were purchased for the room. The torch lights were positioned to avoid any direct glare on the monitor screens. The monitor displays were then calibrated using a ColorMunki Display spectrophotometer vi to adhere to the ITU-R BT.709 recommendation vii. The ratio of ambient light and light behind the monitors to the peak luminance of the screen was set to be less than 0.15, as suggested in the BT.500 general viewing conditions in a laboratory environment viii. The monitors were selected to be Apple Thunderbolt displays with 27-inch diagonal screens and screen resolutions The chairs for viewing the videos were situated at 30 inches away from the screen, which represented 1.67 picture heights and ensured that the viewing angle from monitor to chair was no greater than 30. For internal tests, EuclidIQ makes use of its R&D team of developers, who are geographically dispersed throughout the United States. Instead of calibrating each individual environment, developers are given general guidelines for setting up their viewing areas: the room should be fairly dark, with little outside light coming in; the monitor should have 1080p resolution or better; and the viewing distance should reflect each developer s typical viewing distance when watching videos on their computer. For external tests, subjects go through a pre-test screening that includes both a vision test and a color blindness test, following the recommendations ix. For internal tests, EuclidIQ developers have proven over time that they possess the requisite vision and color vision by generating consistently good scores in post-test subject reliability metrics. EuclidIQ White Paper 4

5 Data Presentation and Scoring Scheme One of the most interesting decisions in subjective test methodology design is the presentation and scoring of the data. There are three main schemes, as summarized in the P.910 recommendation x : 3 = Slightly annoying 2 = Annoying 1 = Very annoying. And the PC method uses a seven-level relative impairment scale, comparing the second clip in a pair to the first: Absolute Category Rating (ACR), where each video clip is judged according to its quality, independently of other clips; Degradation Category Rating (DCR), also known as Double Stimulus Impairment Scale (DSIS), where each video clip is compared against a reference clip and judged according to how much impairment the viewer notices in comparison to the reference; Pairwise Comparison (PC), where video clips from different processes (systems or algorithms) are presented in pairs and judged relative to the other video in the pair. The ACR method usually uses a five-level quality scale: 5 = Excellent 4 = Good 3 = Fair 2 = Poor 1 = Bad. The DCR method uses a five-level impairment scale: 5 = Imperceptible 4 = Perceptible but not annoying -3 = Much Worse -2 = Worse -1 = Slightly worse 0 = Same 1 = Slightly better 2 = Better 3 = Much Better. The PC method seems on the surface to be the most natural way to compare one encoding type against another, and it is generally thought to be the best way to determine whether one method is better than another. However, the PC method is not well-suited to determine how much better one method is versus another, which requires multiple operating points (e.g., multiple bitrates for target bitrate tests or multiple QP values for QP-mode tests) evaluated on a common scale to form rate-distortion (R-D) curves and to measure gains from the R-D curves. The pairwise structure of the PC method enables relative evaluation of a video clip against its pairwise counterpart but not absolute evaluation against multiple operating points. The double stimulus or full reference nature of the DCR method, where every video clip is compared against an unimpaired reference stream, makes it well-suited for the EuclidIQ White Paper 5

6 generation of calibrated R-D curves. However, the process of viewing a reference stream together with every video clip doubles the testing time relative to a single stimulus method such as ACR. Creation of valid R-D curves requires a minimum of three operating points and two processing streams (a test encoding type and a reference encoding type), or six encodings for each video clip, and there are a wide range of video clips in each subjective test. Given the above considerations, the ACR method was chosen for internal tests and the DCR method for external tests. For internal tests, where it is most important to obtain subjective testing results useful for developing algorithms, ACR is the method that enables evaluation of the most videos in a given amount of testing time. However, to allow for some calibration to reference streams, the internal test methodology departs from the standards recommendations by playing a high-quality reference stream before each set of six corresponding to a given video clip. This partial reference setup strikes a good balance between the full-reference calibration of the DCR method and the no-reference speed of the (original) ACR method. For external tests, the full-reference DCR method provides the most consistent and most frequent calibration for untrained viewers and also the most credibility for a formal test, even though the testing time for each subject is longer. To present subjective test data, EuclidIQ created an application to conduct the subjective test as double blind, meaning that neither the subject nor the test presenter knows what is being presented and when. As noted above, six encodings are associated with each video clip. In the partial-reference internal tests, for each set of six encodings, the application plays a high-quality reference stream first, followed by a randomized presentation ordering of the six encodings. In the full-reference external tests, the presentation ordering of each set of six encodings is again randomized, but the high-quality reference stream is played before each encoding. After each encoding is played back, the subject is asked to score the encoding according to a nine-level scale. Subjects are asked to score according to either the DCR impairment scale or ACR quality scale descriptions noted above. In external tests, subjects view the video clips at full speed, without the ability to pause playback, and each encoding (and its high-quality reference) is played back twice before scoring. In internal tests, subjects are told to view the encodings at full speed but are given the ability to pause playback and restart the video, and a given encoding can be played back a number of times at the discretion of the subject before scoring. For internal tests, video clips have durations ranging from 10 to 30 seconds; for external tests, all video clips have duration 10 seconds, adhering to the guidelines from the P.913 recommendation xi. For both internal and external tests, in order to avoid visual fatigue, subjects are asked to take a five-minute break after 20 minutes of video viewing. For internal tests, the 20- minute viewing time generally allows for one full run of five videos clips with six encodings each, plus high-quality playbacks before each set of six encodings and a 3-second pause after every encoding. For external tests, the 20- minute viewing time generally allows for a full-reference run of three video clips with six encodings each, with each encoding and its corresponding high-quality reference played back twice, plus a 3-second pause after every clip. Both the internal and external tests were originally designed with replication, such that each subject would score the entire set of video clips in the test twice, with a different, randomized presentation ordering in each run. Replication reduces the effects of presentation bias, also known as the order effect, where scores for a given encoding may be affected by the score for the previous encoding. Replication also aids in post-test screening of EuclidIQ White Paper 6

7 subjects, as one can determine the variance of each subject s scores from run to run for identical encodings. Within-subject variance is one method of determining which subjects scores should be discarded as unreliable. For internal tests, after extensive verification of all developers scores as reliable, replication of runs was removed in favor of testing more videos in each test. For external tests, replication was removed in favor of fullreference testing, because putting both in place would make the testing time for each subject too long to evaluate a reasonable number of video clips. Selection of Subjective Test Content For the results of any encoder comparison whether subjective or objective to be meaningful, the test materials (the video clips in the test) must be representative. However, there are multiple criteria for defining representative test materials, including semantic categories, motion and complexity characteristics, and encoding difficulty. Additionally, the chosen video clips must be suitable for subjective evaluation, which brings in further considerations such as clip length, watchability, and noticeability of artifacts. EuclidIQ undertakes subjective testing in two different cases with different purposes: first, to prove the benefits of its IQ264 perceptual quality optimization (PQO) technology to current and potential customers; second, to measure the overall compression benefit of IQ264 as the PQO technology is developed and productized. In the first case, customers often evaluate encoding technologies on specific data sets with specific settings, and they have specific evaluation criteria with specific applications in mind. For example, if the encoder is to be applied to Blu-ray creation for motion picture distribution, then subjective tests should be run on video content that will match motion picture video in terms of source video acquisition, shot composition, editing style and effects, CGI content, titles, and quality requirements. In the second case, for general technology development and productization, choosing a good content means finding a set of test videos that accurately represents the broad range of video content consumed by viewers today. Not only should the videos cover the full content space, in terms of genres and modern editing styles, but they must also be amenable to subjective testing in a way that leads to meaningful test results. This whitepaper focuses on content selection for this second, general viewing case. Representative Content: Classifying Based On Semantic Categories In typical video quality analysis studies, representative test materials are classified based on the diversity of semantic categories, defined by the meaning and purpose of the video clips. For example, the Video Quality Experts Group (VQEG) stated in a recent test plan xii, The test material will be representative of a range of content and applications, selecting a set of eight semantic categories: Videoconferencing Movies and movie trailers Sports Music videos Advertisements Animation EuclidIQ White Paper 7

8 Broadcast news Home videos Semantic category lists, while they often cover a wide range of video types, are problematic because some of the categories are exceedingly broad and may involve fulllength videos that consist of many individual clips edited together. Movies, for example, contain many different scene types with varying characteristics, including long establishing shots of city skylines or panoramic natural vistas, juxtaposed against action scenes with fast motion and quick scene cuts, or even dialogue scenes with intense facial close-ups. Sports is itself a wide category: tennis, basketball, baseball, ice hockey, football, golf, and boxing are all considered sports, but the respective broadcasts of those sports contain very different types of scenes with very different types of characteristics (e.g., athlete motion, ball/puck motion, the size of the athletes relative to the overall playing field, and the rate of switching camera angles to properly follow the action). In addition, some of the categories often involve video clips that typically have one of two issues: they are either too difficult to watch and score or too simple to distinguish quality. Including these types of clips leads to inconclusive subjective test results. In videos with multiple, rapid scene cuts, it is difficult to focus on any particular area or subject of the video, making the overall video difficult to watch. A good example is movie trailers, especially fast paced CGI based trailers, which are meant to be flashy and awe-inspiring. Music videos, though not involving as much motion as movie trailers, also typically contain multiple scene cuts. Conversely, videoconferencing and broadcast news videos often contain a minimal amount of motion set against a stationary background, with most of the video comprised of a person talking to the camera. Such videos are easy to watch but difficult to use in subjective testing to distinguish encoding quality at typical bitrates, as most encoders will produce equally good quality for them. The bottom line is that, while some videos in these semantic categories may be appropriate for subjective testing, the majority are not, so requiring a full representation of videos in each semantic category is problematic. In addition, semantic lists provide no guidance to help select individual segments for subjective viewing once a particular long-form video is selected to represent a category. For example, if the subject football is selected to represent the sports category, which shots of a football game should be used in the test? Should it be the crowd scenes, the kickoff, the slow-motion replay, or the morose-looking player who has just been benched? Each choice contains characteristics that will garner both different encoding performance and subjective viewing response. Representative Content: Classifying Based On Motion and Complexity Sectors Test sets based solely on semantic category lists often are not representative data, so EuclidIQ believes it is better to classify videos according to the characteristics of their content rather than the (semantic) categories of their content. In particular, the motion and complexity characteristics of videos often directly correlate with their encoding difficulty. This whitepaper makes use of the following general, conceptual definitions for motion and complexity. Motion is defined as the temporal displacement in the video from frame to frame, while complexity is defined as the amount of high spatial frequency content in a given scene. EuclidIQ White Paper 8

9 Videos containing soft content, with no strong edges or texture, have low complexity visualize an empty blackboard or a uniformly gray sky while high complexity videos contain many regions with significant edge and texture information imagine a scene with highly detailed imagery of a spring forest. For the purpose of selecting representative data sets for subjective testing, one can classify video clips according to their motion (high, medium, or low) and their complexity (high, medium, or low) characteristics. This results in a total of nine motion/complexity sectors, as illustrated in Figure 1. Figure 1: Motion and Complexity Sectors Generally, it is difficult to distinguish encoder quality for videos in either the high-motion, high-complexity sector or the low-motion, low-complexity sector. Videos are more difficult to encode with good perceptual quality the closer they are to the upper right-hand corner, with high-motion, high-complexity videos the most difficult (all encoders will likely perform relatively poorly), but videos are easier to encode with good quality the closer they are to the lower left-hand corner, with low-motion, low-complexity videos the easiest (all encoders will likely perform relatively well). EuclidIQ White Paper 9

10 EuclidIQ believes that truly representative data sets should include a good distribution of video clips from all nine motion/complexity sectors. Quantifying Motion and Complexity Sectors One might ask how motion and complexity can be quantified so that videos can be placed in their proper motion/complexity sector. The ITU-T P.910 recommendation for subjective video quality analysis (VQA) in multimedia applications suggests encodingindependent measures xiii to compute the temporal and spatial characteristics of videos. While these measures have the advantage of not requiring an encoding to compute them, they are not well correlated with encoding quality. A more suitable way to quantify the amount of motion in a video is to perform a sample encoding and gather statistics for the magnitudes of the motion vectors in the encoding. One can calculate the average motion vector magnitude across the entire video or the median of the average motion vector magnitudes from individual frames. To measure complexity, the video is encoded with intra-frame (I- Frame) encoding and then PSNR or bits-per-pixel (bpp) statistics are gathered from the sample I-frame encoding, where a low overall PSNR (or high bpp) corresponds to high complexity and a high overall PSNR (or low bpp) corresponds to low complexity. Suitable Content for Subjective Testing Additional considerations come into play when selecting content for subjective evaluation of encoder performance. While finding videos that reside in each of the nine motion and complexity sectors ensures that the encoder is stressed for all combinations of motion and complexity, the test video clips need to be refined further to ensure that they are suitable for subjective viewing, both in terms of their watchability and the consistency of the resulting subjective scoring. Watchable videos are interesting and enjoyable to view during the test, and they are neither too long, nor too short. However, not all watchable videos produce consistent scoring within and across viewers. Videos can be difficult to score because they are chaotic and visually complex (e.g., a school of fish swimming, confetti falling from the sky, swirling water, etc.). Because these videos have no clear subject, subjective scoring of such videos will have greater variability and thus be less reliable, making it difficult to draw quantitative conclusions about encoder performance. The following factors are used by EuclidIQ in the refinement process for selecting subjective test content: the video clip length, the video watchability (including the technical attributes of the cinematography), and the likelihood that the video will produce consistent and reliable subjective scoring. Other than video clip length, these factors are not easy to quantify, so additional input from developers that have experience in video production can help identify appropriate clips. Because all video clips are subjectively scored by human subjects after real-time playback, the aforementioned P.910 recommendatio xiv, as well as the BT.500 recommendation xv for subjective VQA of television video, specify that test clips should be limited to 10 seconds in length. Clips shorter than 10 seconds often do not have enough content for humans to easily distinguish encoder performance. Clips longer than 25 seconds are problematic for two reasons. First, they often have multiple scenes with different performance characteristics, making overall evaluation of the clip more difficult. Second, it is often difficult to remember the relative degradation of an artifact that occurs near the start of the clip when the clip is long, and this EuclidIQ White Paper 10

11 makes quality scoring less accurate. EuclidIQ believes its expert viewers can accurately assess video clips anywhere between 10 to 25 seconds in length. Videos that are interesting and enjoyable to watch have high watchability and make good test material. For example, videos with human faces are highly watchable and important for subjective tests because of the typical viewer s sensitivity to distortions in faces. On the other hand, some video clips are not watchable because they are physically taxing to watch (e.g., videos taken by shaky or hand-held cameras, videos that capture motion that would normally cause motion-sickness like a point-of-view video from a rollercoaster, or videos with strobing and continually flashing lights). Additionally, sometimes short video segments do not make sense when taken out of the context of a longer video. These types of videos are confusing for viewers because they don t provide a good reference with which to make an assessment of video quality. Finally, some videos are not suitable because of inappropriate content (e.g., politically-charged or violent videos) that viewers would find objectionable. Another factor that affects the watchability of video content for subjective testing is the quality of the cinematography. Cinematography can be defined xvi as the science or art of motion-picture photography. For the purposes of subjective test content selection, one can skip the artistic notions of narrative and themes and concentrate on the technical aspects of cinematography, selecting videos that show good camera focus and tracking and good scene lighting. Video clips with poor camera work or bad lighting are not easy to watch and can confuse subjective test results because viewers mistakenly interpret poor original source media as compression artifacts. Additionally, subjective test content should include a variety of camera angles and camera motion, as these can alter encoding performance due to variations in motion and spatial frequency characteristics. The purpose of subjective testing is to derive numerical measures of video quality based on human scoring. As such, it is vital to test with video clips that produce consistent and meaningful mean opinion scores. Generally, watchable videos with easily identifiable subjects and background are easy for viewers to score consistently. These videos make up the majority of video content in motion pictures, television, advertisements, news, and most sports productions. However, there is also a class of video clips that are watchable but difficult to score. These clips might have rapid scene changes or contain visually dense content or show large variations in lighting and contrast. They are sometimes seen in movies, concert footage, and action dramas on TV. These clips are also important because they stress the limits of encoders and help to differentiate encoder performance. EuclidIQ believes that such difficult-to-score videos should be included as part of an overall encoder evaluation process, but they should not be a large factor in subjective testing. Such difficult-to-score videos should be considered as corner cases, to be evaluated outside of subjective testing by expert viewers. Other important considerations for content selection include the nature of the source acquisition and postproduction work such as color correction and video editing. The acquisition format (camera type, frame size, and frame rate) can have a significant effect on the video s look. For example, film-based content that has been digitized to video can often have high amounts of film grain. Similarly, the frame rate is important since lower frame rate video will contain higher amounts of motion blur during high-motion sequences. Post-production editing and color correction can drive encoder performance but are often confusing to viewers who watch short clips and do not have the full EuclidIQ White Paper 11

12 context of the longer video to give clues about the intent of the effects; these types of heavily-edited videos are generally consigned to corner case evaluation. Finally, it is important when selecting content to be aware of the compression format and compression ratio of the original source video. Evaluating encoder performance when the original video is highly compressed and shows noticeable artifacts is difficult because viewers cannot easily distinguish between artifacts present in the original video and additional artifacts introduced by the encoder. The effects of this are minimized to some extent by using a high-quality reference stream of the video during subjective testing, but it is preferable to use the original source video with the highest quality possible and avoid content that is overly soft or blocky. Determination of Challenging Performance Points using Video Quality Breakdown Bitrate The next step in subjective test methodology design is to select appropriate performance points for encoder evaluation. Encoded bitrate is fundamental to video compression and the dominant factor that affects quality, storage and transmission costs, and computational complexity. Subjective testing enables one to verify the effect of bitrate changes on video quality and, with followon analysis, to optimize the financial aspects of storage, transmission and CPU expenditures. To stress an encoder s compression efficiency, it is vital to select bitrates where the video quality starts to break down. At the breakdown point, the video quality transitions from acceptable to unacceptable, and any associated reductions in cost or speed are outweighed by complaints from customers. This transition point is termed here the Video Quality Breakdown Bitrate (VQBB). EuclidIQ s driving goal is to improve encoding technology at the most critical performance point where the video quality breaks down with conventional, non-pqo based encoders. The only means to determine encoder performance at the quality transition point is to test above, at (or near), and below the VQBB. Thus, the VQBB must be identified for each video, since the combination of content, acquisition format, motion, and spatial complexity uniquely determines the subjective quality of a video as bitrate is varied. Defining the Video Quality Breakdown Bitrate The VQBB is defined as the bitrate at which artifacts just become noticeable when a video is watched under ideal subjective viewing conditions (as described in the section above on the physical environment of the test). Since different observers perceive compression artifacts differently, there is no specific bitrate at which viewers will universally agree that a video s quality has degraded and artifacts have become noticeable. But there is a range of bitrates over which the quality will degrade until virtually all viewers will agree that artifacts are present. At high bitrates, the compressed video looks pristine and is considered visually lossless. These bitrates are above the VQBB. As bitrate decreases, subtle artifacts begin to occur that only very sharp-eyed or expert viewers will notice, and then only in optimal lighting conditions. Typical viewers might find these noticeable, but only in still frame comparison against the uncompressed source video. Once the bitrate drops below the VQBB, almost everyone will agree there are artifacts present, even during real-time playback in daylight or office lighting. The VQBB rests EuclidIQ White Paper 12

13 between the levels, where artifacts are only slightly noticeable and where there is near-widespread agreement that the quality is degraded. An illustration of video content above and below the VQBB is shown in Figure 2. This figure contains two cropped segments taken from a 1080p30 video encoded at 2.4 Mbits/s and 1.6 Mbits/s. Encoding at 2.4 Mbits/s, on the left, using H.264 with encoding settings typical for online streaming, there is some softening and loss of texture compared to the original source, but most casual viewers would find it difficult to differentiate between the original and encoded video. However, encoding at 1.6 Mbits/s, on the right, results in large areas of blocking on the face and skin, and a loss of visual coherence in the soft focus background. This is more apparent during real time playback of the video than the figure depicts, and the artifacts are strong enough to be noticed during playback in normal daylight viewing. Thus, the VQBB for this video lies somewhere between 1.6 and 2.4 Mbits/s. Figure 2: Example screenshots showing video above the VQBB (left, 2.4 Mbits/s) and below the VQBB (right, 1.6 Mbits/s). The VQBB lies somewhere between 2.4 and 1.6 Mbps. (Source video obtained used under license from Shutterstock.com) For a given video, determining the appropriate bitrates for subjective testing involves identifying the bitrates where artifacts become just barely noticeable (above the VQBB) and fully noticeable (below the VQBB) and then estimating the VQBB itself, which lies between these two bitrates. The process entails reviewing the video after it is encoded over a broad range of bitrates and categorizing the results according to the quality level distinctions provided in Table 1. EuclidIQ White Paper 13

14 Video Quality Level Typical Artifacts Noticeable Viewing Conditions Above VQBB Near VQBB Slight intra-flicker Loss of texture on fabrics, hair, and face Increased intra-flicker Increased softness Blocking minor to annoying Temporal shimmering Still frame review in optimal lighting conditions Real-time playback in subjective test environment lighting Below VQBB Significant intra-flicker Significant blocking large areas and easily seen Significant loss of texture Motion trails Real-time playback in daylight or office lighting Table 1: Description of Typical Artifacts Relative to VQBB Relationship of VQBB to Subjective Test Scoring It is interesting to observe how the three VQBB levels correspond to the five-level quality scale used in external subjective testing as described above. A series of external subjective tests undertaken by EuclidIQ measured observer responses for 14 HD ( p) video clips encoded above, near, and below the VQBB. The encodings for these tests were done with two-pass VBR, with VBV buffering constraints that are typical of adaptive bitrate streaming. To determine appropriate bitrates, the VQBB analysis process was applied by EuclidIQ expert viewers to the reference encoding type in the test, x264 (denoted x264/ref), to determine a unique range of VQBB-based bitrates for each test video EuclidIQ White Paper 14

15 clip. The VQBB-based bitrates, summarized over the 14 videos, are presented in Table 2. Because the test video clips covered a wide range of content, motion, and spatial complexity, there was approximately an order of magnitude difference between the highest and lowest bitrates used in the test. Above VQBB Near VQBB Below VQBB Max (kbits/s) Average (kbits/s) Min (kbits/s) Table 2: Bitrate Statistics for Reference Encoder Videos The test video clips were scored by 30 members of the general public (non-expert viewers) using the Absolute Category Rating (ACR) scheme. Error! Reference s ource not found. shows the average MOS at each VQBB level, averaged across the 30 viewers and 14 videos. Videos encoded at the Above VQBB level had an average MOS of 4.11 (with standard deviation = 0.31). Videos encoded at the Near VQBB level had an average MOS of 3.79 (std = 0.44). Videos encoded at the Below VQBB level had an average MOS of 2.89 (std = 0.56). These results match a common sense notion of the VQBB. When video quality is above the level where people start to see artifacts, the ACR scores should be somewhere between Good and Excellent. As the noticeability of artifacts increases, the scores should fall somewhere between Fair and Good. And once artifacts are seen by nearly all, the quality scores should lie between Fair and Poor. EuclidIQ White Paper 15

16 ACR Rating ACR Score Average MOS Score VQBB Range Excellent Above VQBB Good Near VQBB Fair Below VQBB Poor 2.0 Bad 1.0 Table 3: Correspondence between Absolute Category Rating and Average MOS for Reference Encoder Using VQBB to Quantify Compression Gains VQBB levels can be used to provide approximate qualitative compression improvements of one encoding type versus another by referencing the ACR scoring scale descriptions. In the aforementioned external subjective tests, EuclidIQ s IQ264 technology applied to the x264 encoder was evaluated against reference x264 encoding (x264/ref). Each video clip in the subjective tests was encoded at the three VQBB levels using the two encoding types (IQ264 and x264/ref), for a total of six encodings per clip. Figure 3 compares the MOS values, averaged over 30 viewers and 14 videos, for IQ264 and x264/ref as a function of the three VQBB levels. Videos encoded at the Below VQBB level were scored EuclidIQ White Paper 16

17 Average MOS between Poor and Fair (average scores between 2.0 and 3.0) for x264/ref but improved to scores between Fair and Good (average scores between 3.0 and 4.0) for IQ264. This represents a significant improvement for the bitrates that most stress the reference encoder, x264/ref. Quality improvements for IQ264 over x264/ref are also seen at the Near VQBB and Above VQBB levels as well. At the Near VQBB level, average MOS values improved from between Fair and Good (average scores between 3.0 and 4.0) for x264/ref to scores between Good and Excellent (average scores between 4.0 and 5.0). While VQBB levels can provide some idea of qualitative compression improvements, the next section provides a more video-specific way to calculate quantitative compression gains Average MOS by VQBB Level Below VQBB Near VQBB Above VQBB x264/ref IQ264 Figure 3: IQ264 improves video quality over x264/ref by applying perceptual quality optimization; the improvement is captured by average MOS values at the three different VQBB levels. EuclidIQ White Paper 17

18 Post-Test Analysis The final step in subjective test methodology design is to perform post-test analysis to quantify the results of the subjective tests. First, subject reliability metrics are applied to identify and remove those test subjects whose scores are likely unreliable. Then, the scores of the remaining (reliable) test subjects are aggregated into mean opinion scores, enabling the calculation of compression gain (of one encoding type versus another reference encoding type) as average bandwidth savings over a common quality interval. Subject Reliability Metrics Three reliable metrics are applied to identify which test subjects have produced scores that are likely unreliable. Recall that each video clip is encoded using two encoding types (referred to here without loss of generality as the test and reference encoding types) at three bitrates each, with the bitrates based on the video quality breakdown bitrate, as detailed above. This results in six total performance points per clip. The first metric is termed here switch percentage. For given video clip and encoding type, with all other encoding settings being the same, subjects should give a higher score to a higher-bitrate encoding; if this isn t the case, a switch has occurred. For example, for the reference encodings at 1 Mbits/s, 2 Mbits/s, and 3 Mbits/s for a given video clip, subjects should assign the highest score to the 3 Mbits/s encoding and the lowest score to the 1 Mbits/s encoding. For any set of three encodings (corresponding to three bitrates in a given video for a given encoding type), there are three possible switches: high bitrate with medium bitrate, medium with low, and high with low. A high-low switch may be considered more serious than the other two switch types. The switch percentage is then calculated as the number of switches divided by the number of possible switches. For external tests conducted with non-expert subjects, the maximum allowed switch percentage is 20%; subjects with higher switch percentages have their scores discarded. For internal tests, the EuclidIQ development team (most of which may be classified as expert viewers) consistently achieves switch percentages around 3 to 5%. The second metric is termed (intra-subject) variance percentage and applies to tests where all encodings are scored twice, in two separate runs. In this case, subjects should score the same encoding (a given video clip encoded using a given encoding type at a given bitrate) consistently from run to run. If the scores for the same encoding differ by more than 1 from run to run, a scoring variance has occurred. The variance percentage is then calculated as the number of variances divided by the number of possible variances. For external tests, the maximum allowed variance percentage is again 20%; subjects with higher variance percentages have their scores discarded. For internal tests, the EuclidIQ development team consistently achieves variance percentages around 3 to 5%. The third metric is termed (inter-subject) difference percentage and measures, for a given encoding (a given video clip, encoding type, and bitrate), the difference between an individual subject s score and the mean opinion score (MOS) for that encoding. If that difference is greater than 1, a scoring difference has occurred. The difference percentage is then calculated as the number of differences divided by the number of possible differences. The difference percentage, for both internal and external tests, is generally used to monitor whether a given subject is stricter or more lenient than other subjects in the test. A high difference percentage in itself will not disqualify a subject from the subjective test results, as long as the EuclidIQ White Paper 18

19 subject s scores are otherwise consistent (i.e., low switch and variance percentages). It should be noted that the above three metrics are variations of subject reliability metrics proposed in the ITU-R BT.500 recommendation xvii, where the metrics are termed local inversions (a variation of switch percentage) and systematic shifts (a variation of difference percentage). The metrics described here are simpler than those found in the standards and tailored to the EuclidIQ subjective test methodology. Calculating Compression Gain Once subject reliability metrics have been applied and unreliable subjects and their scores have been discarded, the remaining scores are averaged to obtain mean opinion scores (MOS). The MOS values are paired with the corresponding encoding bitrates to obtain rate-quality plots, a variation of the more well-known rate-distortion plots. For the bitrates in the rate-quality plots, instead of using the target (input) bitrate, the actual output bitrate of each encoding is measured from the size in bits of the bitstream and the elapsed time of the video clip (determined from the video clip s frame rate and number of frames). With two encoding types (a Reference encoding and a Test encoding) and three bitrates each, the subjective test results for a given video contains six total performance points, as illustrated in Table 4 and Figure 4. Reference Bitrate (kbits/s) Reference MOS Test Bitrate (kbits/s) Test MOS Table 4: Example Subjective Test Results EuclidIQ White Paper 19

20 Figure 4: Example Plot of Subjective Test Results The next step is to approximate the rate-quality curves underlying the six data points from the subjective test (three data points per encoding type). This is done by interpolation, under the (relatively safe) assumption that quality (MOS) monotonically increases with bitrate for a given encoding type, all other settings being equal. Experience has shown that polynomial interpolation often results in unrealistic rate-quality curves, as illustrated in Figure 5, where polynomial interpolation of the data from Table 4 produces a Test encoding curve that contains an unrealistic kink in the curve. Piecewise spline interpolation usually results in more realistic ratequality curves, as illustrated in Figure 6. EuclidIQ White Paper 20

21 Figure 5: Polynomial interpolation produces unrealistic Test encoding curve. Figure 6: Piecewise spline interpolation produces more realistic rate-quality curves. EuclidIQ White Paper 21

22 Given the interpolated curves from Figure 3, one can then calculate the compression gain of the Test encoding relative to the Reference encoding in terms of average bitrate savings over a common quality interval. This metric is termed Bjøntegaard delta bitrate, or BD-Rate for short, after Bjøntegaard, who originally xviii applied it to PSNR-based rate-quality curves. More recently, Hanhart and Ebrahimi extended xix the BD-Rate calculation to MOS-based rate-quality curves, as is done here. In the example from Figure 3, the quality (MOS) interval common to both the Reference and Test curves is bounded by the lowest MOS value of the Test curve (2.32) and the highest MOS value of the Reference curve (3.32), so the common quality interval is [2.32, 3.32]. To obtain the average bandwidth required by each encoding, one calculates the area to the left of the rate-quality curve, within the common quality interval, as illustrated in Figure 4 for the Test encoding curve (note that the horizontal bitrate axis has been extended all the way back to 0 in Figure 4, to illustrate the full area being calculated). This area can be calculated using simple integration techniques, such as the trapezoidal rule. Figure 7: Average bandwidth is calculated from the area to the left of the rate-quality curve. If the area to the left of the Reference encoding curve is given by AR and the area to the left of the Test encoding curve is given by AT, then the BD-Rate bandwidth savings is calculated as (AR AT) / AR. In the example of Figs. 4 7, the BD-Rate is calculated as 0.29, or 29%, meaning that the Test encoding produces an average of 29% EuclidIQ White Paper 22

23 bandwidth savings over the Reference encoding, over the MOS quality interval [2.32, 3.32]. The subjective test methodology presented here has been used to measure the compression gains of IQ264 (serving as the test encoding) relative to x264 (the reference encoding). Because IQ264 uses perceptual quality optimization that focuses on human perceptual considerations to improve H.264 encoding, its gains are best measured via subjective testing. In the subjective tests referenced in the previous section, where 14 video clips were scored by 30 subjects each under formal test conditions, the average MOS-based BD-Rate gain of IQ264 over x264 was measured to be 22.6%. This means that IQ264 was seen to provide 22.6% bandwidth savings relative to x264 for equivalent MOS quality. Summary This whitepaper has presented the subjective test methodology that was designed by EuclidIQ to quantify the compression gains from its IQ264 technology in a way that is both practical and meaningful, achieving the gold standard of subjective testing while avoiding the inherent limitations of objective metrics. The subjective test methodology includes several components: the set-up of the physical environment, the selection of the test type and scoring scheme, the selection of representative data that is suitable for subjective testing, the determination of challenging performance points (encoding bitrates), and the application of post-test metrics. Subjective testing that followed this methodology has enabled EuclidIQ to quantify the compression gains of IQ264 relative to reference x264 encoding at above 20% bandwidth savings at equivalent subjective quality. Nigel Lee is the Chief Science Officer at EuclidIQ. Dane Kottke is the Director of Software Development at EuclidIQ. Katie Cornog is a Senior Video Codec Analyst at EuclidIQ. Endnotes i (Chikkerur, Sundaram, Reisslein, & Karam, 2011) ii (ITU-R BT , 2012) iii (ITU-T P.910, 2008) iv (ITU-T P.913, 2014) v (ITU-R BT , 2012) (ITU-T P.910, 2008) vi (X-Rite Color Services, 2013) vii (ITU-R BT.709-5, 2009) viii (ITU-R BT , 2012, p. 3) ix (ITU-R BT , 2012, p. 8) (ITU-T P.910, 2008, p. 12) x (ITU-T P.910, 2008, pp. 6-9) xi (ITU-T P.913, 2014, p. 5) EuclidIQ White Paper 23

24 xii (VQEG Hybrid Perceptual/Bitstream Group Test Plan, 2012, pp ) xiii (ITU-T P.910, 2008, pp. 5,14-15) xiv (ITU-T P.910, 2008, p. 6) xv (ITU-R BT , 2012, p. 12) xvi (Wikipedia entry on "cinematography", 2016) xvii (ITU-R BT , 2012, pp. 26,35-38) xviii (Bjontegaard, 2001) xix (Hanhart & Ebrahimi, 2014) Bibliography Bjontegaard, G. (2001). Calculation of average PSNR differences between RD-curves. VCEG-M33. Austin, TX: ITU-T. Chikkerur, S., Sundaram, V., Reisslein, M., & Karam, L. (2011). Objective video quality assessment methods: a classification, review, and performance comparison. IEEE Trans. on Broadcasting, 57(2), Hanhart, P., & Ebrahimi, T. (2014). Calculation of average coding efficiency based on subjective quality scores. J. of Visual Communication and Image Representation, 25(3), ITU-R BT (2012). Methodology for the subjective assessment of the quality of television pictures. International Telecommunication Union. Retrieved from ITU-R BT (2009). Parameter values for the HDTV standards for production and international programme exchange. International Telecommunication Union. Retrieved from I/en ITU-T P.910. (2008). Subjective video quality assessment methods for multimedia applications. International Telecommunication Union. Retrieved from ITU-T P.913. (2014). Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment. International Telecommunication Union. Retrieved from (2012). VQEG Hybrid Perceptual/Bitstream Group Test Plan. Wikipedia entry on "cinematography". (2016). Retrieved from Wikipedia: X-Rite Color Services. (2013). Profiling with ColorMunki Display. Retrieved from EuclidIQ White Paper 24