Influence of display quality on radiologists performance in the detection of lung nodules on radiographs

Transcription

1 The British Journal of Radiology, 80 (2007), Influence of display quality on radiologists performance in the detection of lung nodules on radiographs N BULS, MSc, W SHABANA, MD, PhD, P VERBEEK, MD, P PEVENAGE, MD and J DE MEY, MD, PhD Universitair Ziekenhuis Brussel (UZ Brussel), Department of Radiology, Laarbeeklaan 101, B-1090, Belgium ABSTRACT. The purpose of this study was to evaluate the influence of display quality on radiologists performance in the detection of lung nodules. Display systems with various technical properties were considered based on their general availability in a radiology department. Their quality was assessed by physical tests. Multireadermulticase receiver operating characteristic (ROC) analysis was used to evaluate observer performance. The area under the curve (Az was used as a metric for detectability of simulated lung nodules with diameters of 5 mm and 10 mm, and peak contrast values ranging from 0.1 (subtle) to 0.4 (evident) that were digitally superimposed on normal chest radiographs. Three experienced radiologists interpreted a batch of 60 radiographs on five different display systems; four monitors (two liquid crystal display (LCD) and two cathode ray tube (CRT) monitors) and one printed hardcopy. The physical tests showed superior performance of the two LCD monitors. ROC analysis resulted in the following Az scores: LCD-5MP Az50.78, hardcopy Az50.77, LCDc-2MP Az50.75, CRT-5MP Az50.72 and CRTc-1MP Az Difference in Az scores between the LCD-5MP monitor and both the CRT-5MP (p50.04) and CRTc-1MP (p50.01) monitors was significant. The primary class CRT-5MP monitor that showed reduced observer performance failed to comply with physical acceptance requirements. Luminance response was particularly observed to be insufficient. The results indicate that a quality assurance program has the potential to detect non-optimised display systems that could otherwise result in reduced observer performance. Received 21 November 2006 Revised 12 February 2007 Accepted 22 February 2007 DOI: /bjr/ The British Institute of Radiology With the development of PACS (picture archiving and communication systems), digital image interpretation has come to play a major role in an ever-growing number of radiology departments. It is well acknowledged that the inherent advantages of digital radiological equipment can be fully exploited only when the primary image evaluation is also based on a digital format [1, 2]. Studies in various areas of diagnostic imaging have demonstrated that soft-copy image interpretation is at least as reliable as hard-copy interpretation [3, 4]. On the other hand, soft-copy reading introduces additional variables that can influence diagnostic accuracy. The specifications of the display device affect the displayed image quality, as do the clinical setting and the ambient lighting conditions. The results of previous studies demonstrate the influence of monitor resolution, luminance and ambient light level on observer performance [5 7]. Also, the influence of the more recently introduced liquid crystal display (LCD) systems has been evaluated in relation to the traditional cathode ray tube (CRT) systems [1, 2, 8, 9]. There is a potential for display variations that could result in non-optimized presentation of radiological images. Surveys that assessed monitor performance in various hospitals also demonstrate that display devices do not always operate at their optimal levels [10]. In a radiology department, image quality assurance should involve the whole imaging Address correspondence to: N Buls, MSc, Universitair Ziekenhuis Brussel (UZ Brussel), Department of Radiology, Laarbeeklaan 101, B-1090, Brussels, Belgium. Nico.Buls@az.vub.ac.be chain, from image acquisition to image interpretation on a medical display device, to ensure adequate and consistent image display. Recently, much effort has been made to address the quality control (QC) aspects of electronic display devices and to verify their physical display characteristics. The American Association of Physicists in Medicine (AAPM) issued standard guidelines in their report that address the issue of performance evaluation for both primary and secondary class display devices [11]. According to this report, primary or diagnostic display systems are those used for the interpretation of medical images, whereas secondary or clinical displays are used for viewing images by general medical staff and medical specialists other than radiologists and utilized after an interpretive report has been provided for the images. Differences between both classes are evident in engineering, performance, acceptance criteria and cost. The objective of this study was to determine the effect of physical image display quality on the accuracy of pulmonary nodule detection on posteroanterior (PA) chest radiographs. The detection of subtle lung nodules on chest radiographs is still one of the outstanding challenges in chest radiography. Approximately 40% of lung nodules are manifestations of either primary lung cancer or metastatic disease, and about 30% of lung nodules are missed at the first reading of chest radiographs, although they can be clearly identified retrospectively [12]. The effect of image display quality on the accuracy of pulmonary nodule detection was evaluated by a multiobserver study of digital chest radiographs 738 The British Journal of Radiology, September 2007

2 Influence of display quality on lung nodule detection with simulated solitary lung nodules viewed on five display types. Display systems were selected on the basis of their general availability in a typical radiology department and on their possible usage for the interpretation of chest radiographs. Both primary class (dedicated reporting workstations) and secondary class (non-dedicated review systems) display devices were considered. Their physical display characteristics were measured according to AAPM recommendations [11]. Methods and materials The use of clinical images was preferred over phantom images in order to include the influence of the anatomical background on detection performance. It has been shown that local anatomic variations surrounding and overlying a subtle lung nodule on a chest radiograph, created by the projection of anatomic features of the thorax, such as ribs and pulmonary vessels, can greatly influence the detection of nodules, altering nodule detectability (Az score) by as much as 28% [12]. The disadvantage of using clinical cases is that the standard of reference must be known, so only lung nodules that have CT validation or those that are clinically or histologically obvious can be included. Therefore, data are usually available in only a few patients, which limits the applicability of observer studies. In order to cope with this, solitary lung nodules were simulated on normal radiographs. Selection of normal radiographs 60 normal digital PA adult chest radiographs were randomly selected from our database of patients without pulmonary disease. Expert consensus by two experienced chest radiologists was used as the gold standard for normality. After selection, all patient-related information was digitally obscured. The radiographs were obtained by a FCR-9501 computed radiography CR system (Fuji Medical Systems, Tokyo, Japan) with cm 2 imaging plates(st- Va, Fuji Medical Systems). The plates were scanned with a 10 bit/pixel reading scale in a matrix with pixel pitch 0.2 mm. Image acquisition was performed with an automated exposure-controlled (AEC) high-frequency generator (LX-50; Siemens, Erlangen, Germany) at 120 kvp and 180 cm focus to image plate distance. Under AEC conditions, the entrance surface dose measured with a 15 cm 3 ionization chamber (Keithley Instruments, Cleveland, OH) on a standard chest phantom constructed with 265 cm acrylic slabs, a 5 cm air gap and a 1 mm aluminium sheet according to AAPM recommendations [13] was 0.23 mgy. Informed patient consent was waived by the ethical committee since analysis was performed on post-processed images and identification of patients was not possible. Nodule simulation Lung nodules with different subject-contrasts and diameters were simulated by digitally superimposing circular Gaussian profiles on 30 normal radiographs with Matlab software (Mathworks Inc, Natick). Two nodule diameters of 5 mm and 10 mm were considered by adjusting the profile full width at half maximum. The subject peak contrast (difference in pixel value between superimposed nodule and background divided by pixel value background) of the nodules was adjusted to match clinical cases. To do so, a set of superimposed nodules with peak contrast values ranging from 0.05 to 0.8 was evaluated by a board-certified radiologist with subspecialty in digital thoracic imaging who was not involved in the observer study. After evaluation, nodules were selected with peak contrasts ranging between 0.1 (subtle) and 0.4 (evident). The simulated nodules were superimposed over high-attenuation areas such as the mediastinum, the retrocardiac space or the chest wall, as well as over low-attenuating areas of the non-obscured lung fields. A batch of 60 radiographs was obtained that consisted of 30 normal radiographs and 30 solitary-nodule radiographs containing simulated nodules placed on various locations within the lung fields. Display systems and performance evaluation Five commercially available display types of varying performances were considered. They were selected on the basis of their availability in a typical radiology department and their potential usage for the interpretation of chest radiographs. We considered two primary class displays, one LCD and one CRT, two secondary class displays, one LCD and one CRT, and finally hardcopy images displayed on a view box (Table 1). All display systems were available in our radiology department. Both primary class displays (LCD-5MP and CRT-5MP) and hardcopy images were already used for chest radiography interpretation. The secondary class displays (LCDc-2MP and CRTc-1MP) were included in the study in order to evaluate the influence of using non-dedicated review systems on diagnostic performance. For display on a view box, images were printed on cm film with a 14 bit Kodak Dryview 8900 laser imager (Eastman Kodak, Rochester) using a 39 mm spot. The view box (Planilux, Warstein, Germany) had a luminance of 4300 cd m 22 measured in the centre (Light-O-Meter; Unfors Instruments, Billdal, Sweden) and a uniformity of 12%. After calibration, the characteristic curves of all displays agreed with the DICOM grayscale display function standard based on the barten model to ensure a consistent image appearance [14]. Luminance response was obtained with a telescopic photometer (Konica Minolta LS-100, Zaventem, Belgium). The minimum luminance was set as low as possible. The look-up table of the printer was adjusted to achieve image consistency with the softcopy displays when viewed on a light box. Physical performance of the displays used in the study was assessed on the basis of the recommendations of the AAPM report [11]. Not all the recommended tests of the report were considered. In order to achieve a quick protocol that could be used on a routine basis with a reasonable investment of QC equipment, a selection was made of the following properties: geometrical distortion, luminance response, luminance uniformity, resolution and veiling glare. The softcopy test patterns that were used to evaluate the display properties were obtained in DICOM format [15] and were displayed on the five calibrated display types. A free available DICOM viewer [16] was used for the display on the monitors. The display properties were obtained with a telescopic photometer The British Journal of Radiology, September

3 N Buls, W Shabana, P Verbeek, et al Table 1. Evaluated display systems. Primary class displays and hardcopy are used for image interpretation Display type Manufacturer and model Year of installation Chromaticity Resolution matrix Primary class LCD-5MP Totoku, ME511L 2004 Monochrome CRT-5MP Totoku, MD Monochrome Secondary class LCDc-2MP Viewsonic, VX Colour CRTc-1MP Dell, M782P 2004 Colour Printed hardcopy Kodak, DV Laser filmprinter LCD, liquid crystal display; CRT, cathode-ray tube; MP, megapixel. (Konica Minolta LS-100) to measure luminance response, luminance uniformity and veiling glare, a flexible ruler to measure geometrical distortion and a magnifying glass to assess resolution. The following test patterns were used: TG18-LN-01 to TG18-LN-18 for luminance response, TG18- UNL10 and TG18-UNL80 for luminance uniformity, TG18- QC for geometrical distortion and resolution, and TG18-GQ, TG18-GQB and TG18-GQN for veiling glare [15]. The obtained properties were tested against the quality requirements for monitors that are valid for primary diagnosis [11]. Image analysis and reading conditions Three board-certified radiologists, all experienced in digital chest radiography (years of experience, 8 10; mean 8.6 years), participated in the observer study. They independently assessed the set of images on each of the five displays. For each reviewer, both the order of display type and the display order of the images within each reading session were randomized. The reviewers were told that 60 chest radiographs were to be shown randomly and that 50% contained one nodule. They were asked to rank their level of confidence in the presence or absence of a nodule by using a continuous rating scale (05definitely no nodule, 1005nodule definitely present). They were also asked to record the position of each suspected nodule to verify that the identified nodule corresponded to the location of the simulated nodule. Prior to the reading sessions, a training session was carried out to acquaint the observers with the scoring forms and the display systems, including the controls for brightness, contrast and zoom. The interval between sessions was at least 3 weeks to obviate learning effects. The radiologists were allowed to change the window level and width on the monitors and viewing time was unrestricted. The observations were carried out in the same location; ambient light in the room was controlled to a maximum illuminance level of 3 lux (Light-O-Meter, Unfors Instruments). To avoid reduced contrast performance of the LCD monitors due to off-axis viewing, the viewing angle was directly on-axis in front of all monitors. Prior to each session, the display luminance response was recorded and compared with baseline to ensure consistent image appearance. Data analysis Multireader multicase receiver operating characteristic (ROC) analysis was used to evaluate observer performance. The area under the best-fit curve (Az) was used as a metric for detectability. A total of 900 observations (60 radiographs63 readers65 displays) were analysed. The obtained data were pooled for each display and significance between display performances was assessed by a univariate z-score test of the difference between the areas under the ROC curves by the jacknife analysis proposed by Dorfman et al [17]. This was carried out with the Rockit 0.9B program (Metz CE, The University of Chicago), which can obtain binormal ROC curves from scale rating data and has the ability to analyse partially paired data sets [18]. A p-value of less than 0.05 was considered to represent a statistically significant result. Interobserver variability was estimated by the Cohen weighted kappa statistic [19]. Results The results of the physical performance tests of the displays are shown in Table 2. The last row contains the acceptance criteria for primary class displays [11]. The best overall performance was observed for the two LCD monitors. They scored better on most evaluated properties than the CRT monitors. The LCD-5MP was the only monitor that achieved compliance with the acceptance criteria for all evaluated properties. The secondary class LCDc-2MP monitor failed only for luminance ratio (LR5127). The CRT-5MP monitor performed poorly and did not comply with any of the Table 2. Results of display performance test compared with AAPM acceptance criteria for primary class displays Display Geometrical distortion (%) Luminance ratio (LR) (%) Maximum luminance (cd/m 2 ) Luminance uniformity (%) Resolution (cx score) Veiling glare ratio LCD-5MP CRT-5MP LCDc-2MP CRTc-1MP Printed hardcopy n.a. n.a. AAPM limit [11],2 >250 >170 (30 0(Cx(4 >400 LCD, liquid crystal display; CRT, cathode-ray tube; MP, megapixel. 740 The British Journal of Radiology, September 2007

4 Influence of display quality on lung nodule detection Table 3. Observer performance for the detection of pulmonary nodules viewed on various display systems (95% confidence intervals provided in parentheses) Display Average reading time (min) Specificity Sensitivity Area under ROC curve Az p-value LCD-5MP (0.82, 0.95) 0.61 (0.51, 0.71) 0.78 (0.71, 0.84) CRT-5MP (0.81, 0.95) 0.52 (0.42, 0.63) 0.72 (0.64, 0.79) 0.04 LCDc-2MP (0.88, 0.98) 0.51 (0.41, 0.61) 0.75 (0.67, 0.82) 0.14 CRTc-1MP (0.82, 0.95) 0.48 (0.37, 0.58) 0.71 (0.64, 0.76) 0.01 Hardcopy (0.90, 0.94) 0.57 (0.46, 0.67) 0.77 (0.69, 0.83) 0.30 LCD, liquid crystal display; CRT, cathode-ray tube; MP, megapixel. tested acceptance criteria. In particular, luminance ratio (LR592) and maximum luminance (L max 5150 cd m 22 ) were observed to be insufficient. The CRT monitors also suffered from veiling glare: the observed glare ratios were only 94 for the CRT-5MP monitor and 37 for the CRTc-1MP monitor compared with a glare ratio acceptance limit of at least 400. The CRTc-1MP monitor failed on all evaluated properties except uniformity. It showed the poorest luminance performance of the group for both maximum luminance (L max 5123 cd m 22 ) and luminance ratio (LR574). The highest maximum luminance (L max 5728 cd m 22 ) and luminance ratio (LR5495) values were observed for the printed hardcopy film on the view box. Significant geometrical distortion was observed only for the CRT monitors; they both failed to achieve acceptance criteria. The time needed to collect the physical performance data in Table 2 did not exceed 20 min per monitor. The observer performance data are shown in Table 3. Data are expressed as average reading time, specificity, sensitivity, the area under the ROC curves (Az score) and the p-values from the analysis of variance for the Az values. The 95% confidence intervals are provided in parentheses. The ROC curves of all displays are shown in Figure 1. Figure 1a shows the curves for all false-positive fractions (up to 1); Figure 1b shows the same data up to a falsepositive fraction scale of 0.3. The data revealed that there were statistically reliable differences regarding lung nodule detection among the display types. The LCD- 5MP monitor showed the highest Az of This performance was observed to be significantly better than the performance of both CRT-5MP (p50.04) and the CRT- 1MP (p50.01) monitors. The performance with the LCD- 5MP monitor was also better than both the LCDc-2MP monitor and hardcopy images but this was not observed to be significant (p50.14 and p50.30, respectively). Interobserver variability showed that agreement was fair (kappa50.52). The average reading time on the different display systems was comparable (range min). A significantly reduced reading time was observed only for the hardcopy reading (average 38 min). Three examples of radiographs with simulated nodules are shown in Figure 2, together with their detection rate by the observers. Figure 2a represents a simulated nodule with a high subject contrast that was detected by almost all observers on all display types, Figure 2b,c represents more subtle nodules that showed reduced detection rates. Although still detected by two observers with the LCD-5MP monitor, the nodule in Figure 2b was not detected on hardcopy or the CRTc- 1MP monitor by any of the observers. Discussion The results indicate that the performance of radiologists in the detection of nodules in chest images can be affected by physical display quality. Both primary class and secondary class CRT monitors in the group showed reduced observer performance. Several of their properties also failed physical acceptance criteria during the QC test. The CRT-5MP monitor failed to comply with any of the evaluated criteria. In particular, the luminance ratio was observed to be very poor compared with the acceptance criteria and with the values of the other display systems. The poor physical performance of the CRT-5MP display is probably due to ageing. The LCD- 5MP, although meeting acceptance criteria, did not meet manufacturers technical specifications for new high-end LCD-5MP monitors, which should easily achieve a luminance ratio of about 350 and a maximum luminance of 400 cd m 22. However, manufacturers specifications do not automatically guarantee that display systems operate at optimal luminance levels. An assessment of monitor conditions in four hospitals by Wade and Brennan [10] demonstrated a significant intramanufacturer variation for maximum luminance up to a factor of two in the same hospital. Reports in the literature of the luminance of LCD displays often exceed the observed values of the LCD-5MP monitor used in this study. For example, in a recent performance assessment of 32 LCD- 3MP systems, Jung et al [20] observed a mean maximum luminance of 405 cd m 22 and a mean luminance ratio of 379. Luminance ratio is a critical property of monitors as it provides the contrast visualization over the full range of pixel values. A ratio of at least 250 maintains all contrast information in an image within a luminance ratio where the eye has reasonably good response [11]. Owing to its inadequate performance, the primary class CRT-5MP monitor was removed from the department. It is no longer used for image interpretation. Although hardcopy viewing does not benefit from the advantages of digital viewing tools such as magnification and window/levelling, overall nodule detection was not observed to be inferior compared with the display system with the best performance (LCD-5MP). This is probably due to its high luminance ratio and maximal luminance level which compensates for the lack of dynamic viewing. It did show, however, a reduced detection for lower false-positive fractions. For the LCD- 5MP and hardcopy images, the sensitivities at a falsepositive rate of 5% were 44% and 36%, respectively. This is also shown in Figure 1b by the ROC curves at low false-positive fractions. The British Journal of Radiology, September

5 N Buls, W Shabana, P Verbeek, et al Figure 1. (a) Receiver operating characteristic curves for five display systems. (b) The same data for a false-positive fraction scale up to 0.3. A significantly reduced performance was observed for CRT-5MP and CRTc-1MP. Figure 2. Examples of simulated nodules on PA chest radiographs and their detection on various displays by three observers. The reduced intrinsic resolution of the LCDc-2MP monitor compared with the LCD-5MP monitor did not result in significantly reduced observer performance (p50.14) and did not involve an increased observer reading time (52 min and 57 min, respectively). Although this study did not specifically address the influence of resolution, it may indicate that the resolution of a 2 megapixel monitor is adequate for primary diagnosis with posteroanterior chest images. This is in agreement with the studies of both Herron et al [7] and Otto et al [5], who observed that a resolution of 1 megapixel and a brightness of 260 cd m 22 should be sufficient for detecting nodules on chest images. Also Usami et al [9] found that an LCD monitor with a resolution higher than 1.3 megapixel can be sufficient for pulmonary nodule detection. It is acknowledged that luminance ratio plays a more important role in the detection of lung nodules than resolution with soft-copy reading. Although this study considered only five display systems, the fact that the primary class CRT-5MP display failed physical acceptance requirements during the QC test indicates that a quality assurance (QA) programme has the potential to detect non-optimised display systems that could result in reduced observer performance. The physical tests used in this study were straightforward and took less than 20 min per monitor. They required only an illuminance meter, a telescopic photometer, a flexible ruler, a magnifying-glass and softcopy patterns. This demonstrates that monitor QA is feasible in a busy radiology department at a negligible cost compared with 742 The British Journal of Radiology, September 2007

6 Influence of display quality on lung nodule detection the investment of the display system itself. Display systems were selected on the basis of their presence in a typical radiology department. It is important to note that their observed physical performance (Table 2) is not typical of display systems of their class. Performances were observed to be rather poor when compared with published data from recent display systems. In consequence, the observer performance data of this study (Table 3) should not be compared with results from studies that merely compare display types. For example, in a recent observer preference study by Balassy et al [2], the overall quality for chest radiography was found to be equal with CRT-5MP and LCD-3MP display. However, both monitors in their study had a maximum luminance of 300 cd m 22, which is significantly higher than the maximum luminance of the CRT-5MP monitor used in this study. Also, the observer performance data in this study using simulated nodules should not be compared with accuracy performances observed for clinical studies. The readers were forewarned that the objective of the test was to detect single pulmonary nodules and that half the cases were normal. The study did not assess other thoracic abnormalities encountered in clinical practice, in which diagnostic decisions are more complex than simply the detection of pulmonary nodules. Therefore, the performance of the observers was probably optimised compared with that in clinical routine. However, this effect should not have biased the results of this study in favour of or against a specific display system. Conclusion The evaluation of various display systems in a radiology department has shown that physical display quality can influence the performance of radiologists in the detection of simulated pulmonary nodules on chest radiographs. This was done by ROC analysis of simulated nodules on normal PA chest radiographs and by physical QC tests of the displays. A significantly reduced diagnostic performance was observed with two display systems that failed to comply with physical acceptance criteria. In particular, the luminance response of these displays, with both luminance ratio and maximum luminance, was observed to be insufficient when compared with acceptance criteria. Such inferior display systems could be easily detected by a QA programme in a radiology department. Image quality assurance should involve the whole imaging chain, from image acquisition to image interpretation on a medical display device to ensure adequate and consistent image display. This can be done by non-time-consuming straightforward tests, based on established protocols. References 1. Scharitzer M, Prokop M, Weber M, Fuchsjäger M, Oschatz E, Schaefer-Prokop C. Detectability of catheters on bedside chest radiographs: comparison between liquid crystal display and high-resolution cathode-ray tube monitors. Radiology 2005;234: Balassy C, Prokop M, Weber M, Sailer J, Herold CJ, Schaefer- Prokop C. Flat-panel display (LCD) versus high-resolution gray-scale display (CRT) for chest radiography: an observer preference study. Am J Roentgenol 2005;184: Kundel H, Polansky M, Dalinka M, Choplin R, Gefter W, Kneelend B, et al. Reliability of soft-copy versus hard-copy interpretation of emergency department radiographs: a prototype study. Am J Roentgenol 2001;177: Ishigaki T, Endo T, Ikeda M, Kono M, Yoshida S, Ikezoe J, et al. Subtle pulmonary disease: detection with computed radiography versus conventional chest radiography. Radiology 1996;201: Otto D, Bernhardt T, Rapp-Bernhardt U, Ludwig K, Kästner A, Liehr U, et al. Subtle pulmonary abnormalities: detection on monitors with varying spatial resolutions and maximum luminance levels compared with detection on storage phosphor radiographic hard copies. Radiology 1998;207: Goo J, Choi J, Im J, Lee H, Chung M, Han D, et al. Effect of monitor luminance and ambient light on observer performance in soft-copy reading of digital chest radiographs. Radiology 2004;232: Herron J, Bender T, Campbell W, Sumkin J, Rockette H, Gur D. Effects of luminance and resolution on observer performance with chest radiographs. Radiology 2000;215: Oschatz E, Prokop M, Scharitzer M, Weber M, Balassy C, Schaefer-Prokop. C. Comparison of liquid crystal versus cathode ray tube display for the detection of simulated chest lesions. Eur Radiol 2005;15: Usami H, Ikeda M, Ishigakil T, Fukushima H, Shimamoto K. The influence of liquid crystal display (LCD) monitors on observer performance for the detection of nodular lesions on chest radiographs. Eur Radiol 2006;16: Wade C, Brennan PC. Assessment of monitor conditions for the display of radiological diagnostic images and ambient lighting. Br J Radiol 2004;77: Samei E, Badano A, Chakraborty D, Compton K, Cornelius C, Corrigan K, et al. Assessment of display performance for medical imaging systems, Report of the American Association of Physicists in Medicine (AAPM), Medical Physics Publishing, Madison (USA), AAPM On-Line Report No Samei E, Flynn M, Peterson E, Eyler W. Subtle lung nodules: influence of anatomic variations on detection. Radiology 2003;228: Chu R, Fisher J, Archer B, Conway B, Goodsitt M, Glaze S, et al. Standardized methods for measuring diagnostic x-ray exposures. Report of the American Association of Physicists in Medicine (AAPM) 1991, American Institute of Physics, New York (USA), Report No Digital Imaging and Communications in Medicine (DICOM) Part 14: Greyscale Standard Display Function. National Electrical Manufacturers Association (NEMA) PS deckard.mc.duke.edu/,samei/tg18 [homepage on the internet]. Durham, USA: Duke University Medical Center [updated 2006 January 10; cited 2006 August 2]. Available from: Rasband W [homepage on the internet]. Bethesda, USA: National Institutes of Health [cited 2004 September 8]. Available from: Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jacknife method. Invest Radiol 1992;27: Metz CE, Herman BA, Shen JH. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med 1998;17: Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968;70: Jung H, Kim HJ, Kang WS, Yoo SK, Fujioka K, Hasegawa M, et al. Assessment of flat panel LCD primary class display performance based on AAPM TG 18 acceptance protocol. Med Phys 2004;31: The British Journal of Radiology, September