Loudspeakers and headphones: The effects of playback systems on listening test subjects

Loudspeakers and headphones: The effects of playback systems on listening test subjects Richard L. King, Brett Leonard, and Grzegorz Sikora Citation: Proc. Mtgs. Acoust. 19, 035035 (2013); View online: https://doi.org/10.1121/1.4799550 View Table of Contents: http://asa.scitation.org/toc/pma/19/1 Published by the Acoustical Society of America

Proceedings of Meetings on Acoustics Volume 19, 2013 http://acousticalsociety.org/ ICA 2013 Montreal Montreal, Canada 2-7 June 2013 Musical Acoustics Session 2pMU: Musical Preference, Perception, and Processing 2pMU8. Loudspeakers and headphones: The effects of playback systems on listening test subjects Richard L. King*, Brett Leonard and Grzegorz Sikora *Corresponding author's address: Music Research, Schulich School of Music, McGill University, 555 Sherbrooke West, Montreal, H3A 1E3, Quebec, Canada, richard.king@mcgill.ca Many modern listening test designers disagree on the best playback system to be used when conducting tests. The preference often trends towards headphone-based monitoring in order to minimize the possibility of undesirable acoustical interactions with less than ideal testing environments. On the other hand, most recording and mixing engineers prefer to monitor on loudspeakers, citing a greater ability to make critical decisions on level balances and effects. While anecdotal evidence suggests that differences exist between systems, there is little quantified, perceptually-based data to guide both listening test designers and engineers in what differences to expect when alternating between monitoring systems. Controlled tests are conducted with highly trained subjects manipulating the level of solo musical elements against a backing track on both headphones and loudspeakers. This task serves to make the results equally applicable to critical mixing tasks and rigorous listening tests. The results from both playback systems are compared, showing a defined difference in the mean levels set on the two different monitoring systems. Likewise, the variance seen across subjects were larger when monitoring on headphones than on loudspeakers, lending credence to the hypothesis that tests conducted on one playback system may not be equally applicable to the other. Published by the Acoustical Society of America through the American Institute of Physics 2013 Acoustical Society of America [DOI: 10.1121/1.4799550] Received 21 Jan 2013; published 2 Jun 2013 Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 1

1. INTRODUCTION The intent of the research is to explore the various methods and techniques used by audio engineers, and to quantify the variance of certain adopted procedures. The goal of this specific research is to compare the variance shown in music mixing engineers while balancing mixes using headphone monitoring, as opposed to loudspeakers. As mixing engineers work on music recordings using both of these systems, they must compensate for the differences and create a product that is consistent, regardless of the monitoring environment used. The null hypothesis being tested therefore, is that both data sets came from distribution with an equal mean. 1.1 Motivation Headphone monitoring while mixing music is considered advantageous by many mixers, as it eliminates the mixing room's acoustic properties and possible faults, including background noise, strong reflections, and heavy resonances. Many engineers have adopted headphone monitoring as an affordable alternative to a properly designed mixing room incorporating expensive amplifiers and loudspeakers, and complete isolation from the outside world. The principal disadvantages of headphone monitoring are thus: - Since each channel is only heard by one ear (i.e. left channel information only received by the left ear), "phantom" center images can be difficult to localize [1]. - Sources panned to the extreme left will only be heard by the left ear, whereas a signal appearing in the left loudspeaker only will still be heard by both ears [2]. - The frequency response of the headphones themselves can play a significant role as well, and can affect the resulting mixes, but this is also the case with various studio loudspeakers. It should also be noted that headphones are not physically able to provide the same sensation in low frequency energy as a full range loudspeaker, since headphone playback is only heard while loudspeakers are heard and felt, as both an aural and visceral experience [3]. While engineers understand and regularly experience these differences, little has been done to quantify them. This is especially important, considering the use of one or the other in many research projects, but rarely both. It is the authors' hope that this study may serve to validate the use of one or both of the methods for future experimentation in the area of control room acoustics. The data obtained from these experiments is analyzed using various statistical models. 1.2 Historical Context Previous research has involved the study of mixing variances, i.e. how mixing engineers vary in certain mixing balances over time [4,5]. The data set from these trials has been established as a baseline against which future testing can be measured. The current research results are compared to the original baseline, and the results show some indication of the differences between headphone and loudspeaker monitoring during mixing. 2. TESTING Given the focus of this study on the day-to-day activities of the working audio professional, most standard testing methodologies are inadequate. While classic discriminability and ranking tasks often prove useful, a different approach is required to generate results that apply directly to those working in audio production. To ensure applicability to the intended audience, the proposed testing method must meet three criteria: testing should be completed in a high quality listening environment over a professional-quality playback system, the user interface should employ similar controls to those found on professional audio equipment and all testing material should be musical in nature and of sufficiently high quality [6]. To meet these criteria, the following methodology was adopted, as presented in [4] and [5]. Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 2

2.1 Testing Overview In order to best emulate the working environment and tasks of an audio engineer, the testing methodology asks subjects to complete a basic mixing task. Each subject is instructed to set the level of a stereo stem containing a lead instrument or vocal, as presented along with a stereo stem containing the instrumental accompaniment to the excerpt. The stems were extracted from the full mix, so that the solo element and the accompanying track are split apart yet each stem contains all processing inherent to the full mix. When both stereo stems are played back at the same nominal level, the original engineer's mix is reproduced exactly. This is to say that the vocal stem is very different from the raw vocal track, which would be un-equalized and have no reverberation or level changes that follow the final balancing or mixing. Each musical excerpt is 30 seconds long and is automatically looped, playing back up to four times while subjects perform their task. Subjects may confirm their choice of level whenever they are satisfied or at the end of the fourth playback. The restriction of a time limit for each trial emulates the real-life scenario of making quick decisions in order to complete productions on budget. Each subject completed the task 24 times, divided into two sets of 12 trials containing four excerpts from each genre. 2.1.1 Musical Material Testing utilizes three distinct samples in order to cover a range of genres and normal mixing styles. Both the stereo backing track and the lead stem are complete with effects and pre-written automation. Level automation and processing are included in the stems to ensure that a static level can be reached and set by each subject. While testing relies upon the individual preference of the subjects, the relative level of the lead stem set by the original engineer (nominally 0 db) is retained as a reference for later analysis of subjects responses. The first musical excerpt features a soprano voice with orchestra, taken from a commercial release. While this type of recording would normally present isolation problems, this particular recording included an overdubbed vocal element, allowing the orchestra and vocal stems to remain completely isolated. The second sample is a modern independent rock release, featuring ambient yet thickly layered guitars, bass and drums under a solo vocalist. As with most pop recordings, this track featured good isolation and provided energy across the full frequency range. The third excerpt is from an album featuring a jazz trumpet solo. This was a generally dry studio recording, providing a good contrast to the other examples. 2.1.2 Testing Apparatus Subjects control the level of the stereo stem through the use of a continuously variable rotary encoder, changing level in.25 db increments. Typical mixing tasks in the studio employ a linear fader, which provides visual and tactile feedback as opposed to an unmarked, continuous encoder, so the latter was chosen. This is coupled with a terse visual display which provides the subject a color-coded indication of how many loops of the excerpt are left until a level must be confirmed (e.g. yellow means two loops are left, red means one, etc.) and simple instructions regarding which keys to hit in order to confirm their selection. The combination of a simple control, minimal visual feedback and limited visual distraction force the subject to focus on listening while eliminating any opportunity to cheat the system. The initial level of the solo element in each trial is randomized, beginning around -20dB below normal, at ± 2.5db. Testing was conducted using a software package designed and implemented in Max MSP. The software controls playback, level adjustment and data capture from the encoder, as well as providing the visual display for the subjects. In addition, the software also provides real-time data on a subject's progress and their current level settings on a second remote display for viewing by the researchers. Playback order of the three excerpts is randomized within the software ensuring no repeated musical excerpts and minimizing repetition of excerpt order. The software tracks and records level adjustments made by subjects over time, the final level set and elapsed time for each trial. As part of the in-situ nature of the testing, a standard playback level was not fixed, since each mixing engineer tends to work at varying listening levels, and are generally the most comfortable with the mixing process when they are able to set the playback volume themselves. Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 3

King et al. 2.2 Loudspeaker Testing Loudspeaker testing was conducted in a small control room located on the second sub-basement of the Schulich School of Music (Figure 1). The room, designed for critical mixing, mastering and ear training applications, with a full-range monitoring system configured in the standard ITU configuration [7]. The room exhibits an RT60 of approximately 200ms across all frequency bands, with a slight bump below 100 Hz (Figure 2). In addition to desirable acoustics within the room, its location also provides exemplary isolation from outside noise. This reproduction system (in the given room) displays a flat frequency response, ± 3 db from 20 Hz to 18 khz at the listener's position. FIGURE 1. Studio 22 at McGill University, used for loudspeaker testing. During loudspeaker testing, and for acoustic reasons, the computer monitor was removed and replaced with an ipod for display to the subjects. 2.3 Headphone Testing A variety of headphones were considered for testing, including standard studio and audiophile models. Circumaural and supra-aural as well as both open and closed designs were tested and, while most opt for open, circum-aural designs for testing, a set of closed, supra-aural headphones were selected. This particular set of headphones provided the closest match to the loudspeaker playback system described above, including a similar brightness above 6kHz and flat low frequency response (Figure 2). The headphone response is an average of three measurements taken. FIGURE 2. Comparison of frequency response from headphone playback (in black) and loudspeakers/room (grey). Loudspeaker/room measurements taken at the listening position with a single, omnidirectional microphone while headphone response was taken with a Brüel & Kjaer Head and Torso Simulator (HATS). Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 4

2.4 Subjects The subject pool for the combined tests included 42 individuals, all of whom are active recording engineers, editors or producers. A total of 24 subjects participated in the loudspeaker portion of the experiment and 33 subjects participated in the headphone portion. This amount of data provides a rich source of information on both loudspeaker and headphone monitoring which is used for genre-based analysis. Certain analysis, however must be confined to only the subjects that were able to complete both portions of the experiment. This criteria leaves 10 subjects whose data is directly related between loudspeakers and headphones. The 10 subjects common to both testing scenarios were representative of the entire subject pool in both demographic data and in testing results. This common subject pool was composed of a mix of males and females between the ages of 24 and 45. All subjects are working professionals and/or students in McGill University's graduate-level Sound Recording program. Included in this group were faculty members from the university, a number of alumni of the program and one Grammy award winning recording engineer. While the subjects primarily identified themselves as recording engineers, two subjects identified themselves specifically as mixing engineers and one as a classical music producer. Subjects were split equally between pop/rock and classical/jazz as their main working genre. The subject pool also showed a strong musical background, averaging over 10 years of formal musical training. The average production experience of the subjects was somewhat less, with a mean of eight years. 3. ANALYSIS Statistical analysis was performed to compare the results obtained from the two reproduction systems. This analysis was focused on level and variance differences between genres (to investigate general preferred level and performance contrast) [7] and between subjects given the two different reproduction systems. Non-parametric analysis was applied, as the number of subjects was relatively low and no assumption about level preference distribution was made. 3.1 Inter-Genre Analysis When analyzing inter-genre relationships, the entirety of the data set is available. This means that, for each genre, roughly 240 data points were taken (there are very few missing values due to the accidental interface and / or user misbehavior). All six data-sets (two playback methods x three genres) were tested for normality. Using an established significance level of α =.05, five of the six sets show some statistically significant departures from normal distribution (Figure 3), hence the use of a non-parametric statistical model is justified. The one exception was the jazz excerpt presented over headphones. Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 5

FIGURE 3. Kernel density estimation by genre. 3.1.1 Classical Genre The median for all subjects for classical music over loudspeakers was -3.6 db and 0.6 db for the same over headphones (Figure 4 left). Non-parametrical hypothesis testing (with a null hypothesis that both data-sets came from distribution with an equal mean) showed the rejection of null (p << 0.05). Variance analysis yielded similar results. With a standard deviation of 2.98 db for loudspeaker reproduction and 3.6 db for headphones, an Ansari- Bradley test for equal variance [8] rejected null hypothesis that both data-sets came from distribution with equal variance. One-tailed testing showed that variance of experiment conducted on headphones produced significantly larger variance. FIGURE 4. Left: Kernel density estimation by for loudspeaker and headphone playback of classical music excerpt. Right: Kernel density estimation by for loudspeaker and headphone playback of jazz music excerpt Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 6

3.1.2 Jazz Genre The analysis of the jazz genre yielded slightly different results than the results for classical (Figure 4 right). However, despite of the fact that there is only 1.2 db difference between loudspeaker and headphone reproduction (0 db and -1.2 db accordingly) the Mann-Whitney U test rejected the null hypothesis. Subjects' performance showed a variance that was significantly higher for loudspeakers (std. dev. = 4.2 db) than for headphones (std. dev. = 3.6 db). 3.1.3 Rock Genre The rock music excerpt yielded similar results to those of the classical genre. Hypothesis testing rejects null hypothesis for equal means (with p << 0.05) and equal variances (p = 0.00004), where the variance of underlying distribution of headphone reproduction is significantly larger (Figure 6). In general, the rock data sets have the lowest standard deviations (2.37 db for loudspeakers and 3.24 db for headphones) and lowest mean deviation from the "0 db" level (1.97 db for loudspeakers and 2.52 db for headphones). FIGURE 6. Kernel density estimation by for loudspeaker and headphone playback of rock music excerpt.} 3.2 Inter-Subject Analysis As mentioned earlier, only 10 subjects took part in both portions of the experiment. As was illustrated in [4], the level preference (median) and performance (variance) of single subjects did not change over the period of one week. Therefore, we can conclude that the time gap between loudspeaker and headphone testing will have little effect on the results and the main factor causing changes in level preference and its dispersion will be the playback method. Table 1 shows a general overview of relevant statistical descriptors of the 10-subject group. 3.2.1 Level Preference Shift Significant preferred level difference between playback medium was observed for eight subjects in the classical genre and in six subjects for both the jazz and rock genre. No strong trend was observed when it comes to the direction of the shift across all subjects. However, most subjects exhibited one shift direction consistently across all genres. What is more, nine out of 10 subjects mixed the classical stem higher on headphones, seven subjects mixed the jazz stem lower on headphones and five subjects preferred a lower level on headphones in the rock genre. Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 7

Statistical Descriptor Classical LS Classical HP Jazz LS Jazz HP Rock LS Rock HP MEAN -3.6-0.015-0.54-1.56 0.94-2.45 MEDIAN -3.6 0.6 0-1.2 0.6-2.70 VARIANCE 8.85 12.93 17.62 12.66 5.63 10.48 STD 2.98 3.60 4.20 3.56 2.38 3.24 MIN. VALUE -12.6-11.4-13.8-9.6-3.0-13.2 MAX VALUE 6.0 8.4 11.4 9.0 7.8 6.6 RANGE 18.6 19.8 25.2 18.6 10.8 19.8 MEAN DEV. FROM 0dB 2.15 2.82 3.36 2.89 1.96 2.52 TABLE 1. Statistical descriptors with direct comparison of loudspeaker reproduction (LS) and headphone reproduction (HP) for each genre. 3.2.2 Subjects' Performance Ansari-Bradley test showed that 10 subjects produced variance difference between both experiments (two subjects in the classical genre, two subjects in jazz and four subjects in rock) (Figure 7). All variance changes were in favor of loudspeakers playback, as there was higher dispersion for headphones reproduction in all of these cases. FIGURE 7. Box plots for the 10-subject pool across all genres and reproduction systems. 4. CONCLUSION Statistical analysis showed significant mean level shift based on reproduction method in both the classical and rock genres. Preferred levels for the classical excerpt reproduced over loudspeakers were lower than when reproduced over headphones. Conversely, preferred levels for rock over loudspeakers were significantly higher than over headphones. Variance analysis found that headphone reproduction had significantly larger variance in classical and rock than the same genres over loudspeakers. In general, subjects produced more dispersed results in the jazz Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 8

genre, possibly due a lack of reverberation, which helps to create some sort of additional level anchor in the other genres. Graphical analysis of the approximate distributions supports this observation. Overall, distributions when mixing over loudspeakers are markedly tighter, especially for rock and classical music. 4.1 Future Work An obvious progression from this validity testing would involve longer periods of study employing a single genre of musical material. The resulting data demonstrates the relative difference between the two playback methods for this particular task, and provides a baseline to which future headphone mixing tasks can be compared, for instance the use of Binaural Room Simulation (BRS). ACKNOWLEDGMENTS This research was supported by the Social Sciences and Humanities Research Council of Canada. Additionally, the Authors would like to express their thanks for the technical support from Jon Hong, Harold Kilianski and Yves Méthot at the Centre for Interdisciplinary Research in Music, Media, and Technology (CIRMMT). REFERENCES 1. E. Vickers, Fixing the Phantom Center: Diffusing Acoustical Crosstalk, presented at the 127 th Convention of the Audio Eng. Soc., New York, USA, October 9 12 (2009). 2. B. Bauer, Improving Headphone Listening Comfort, J. Audio Eng. Soc., vol. 13, no. 4, October (1965). 3. F. Toole, The Acoustics and Psychoacoustics of Headphones, presented at the 2 nd International Conference of the Audio Eng. Soc., Anaheim, USA, May (1984). 4. R. King, B. Leonard and G. Sikora, Variance in Level Preference of Balance Engineers: A study of mixing preference and variance over time, presented at the 129 th Convention of the Audio Eng. Soc., San Francisco, USA, November 4 7 (2010). 5. R. King, B. Leonard, and G. Sikora. "The Practical Effects of Mixing in an Environment Closely Resembling a Home Listening Environment, presented at the 162 nd meeting of the Acoustical Society of America, San Diego, October 31 st (2011). 6. Shively, R.E & W.N. House, Listener Training and Repeatability for Automobiles, presented at the AES104 th Convention, Amsterdam, The Netherlands, May 16-19 (1998). 7. Multichannel Stereophonic Sound System with and without Accompanying Picture. ITU-R BS.775-2. 2 nd ed. Geneva: International Telecommunications Union,. Print (2006). 8. Ansari, A.R. and R. A. Bradley, Rank-sum tests for dispersion, Annals of Mathematical Statistics 31, 1174 1189 (1960). Proceedings of Meetings on Acoustics, Vol. 19, 035035 (2013) Page 9