Image hacemet i the JPG Domai for People with Visio Impairmet Jisha Tag, Seior Member, Jeoghoo Kim, ad li Peli Abstract A image ehacemet algorithm for low-visio patiets was developed for images compressed usig the JPG stadard. The proposed algorithm ehaces the images i the Discrete Cosie Trasform (DCT) domai by weightig the quatizatio table i the decoder. Our specific implemetatio icreases the cotrast at all bads of frequecies by a equal factor. The ehacemet algorithm has four advatages: (1) low computatioal cost, (2) suitability for real time applicatio, (3) ease of adustmet by ed-users (for example, adustig a sigle parameter), ad (4) less severe block artifacts as compared with covetioal (post compressio) ehacemets. xperimets with visually impaired patiets show improved perceived image quality at moderate levels of ehacemet but reectio of artifacts caused by higher levels of ehacemet. Idex Terms DCT, DCT filterig, ehacemet, image quatizatio table, JPG, Televisio, visio impairmet, M I. INTRODUCTION ILLIONS of people are visually impaired, with the umber of people with disablig visual problems icreasig with the growig agig populatio. A Louis Harris survey foud that visio impairmet affects 17% of Americas 45 ad older, ad 26% of those 75 ad older [1]. Visually impaired people have difficulties readig small prit, watchig televisio, recogizig faces, etc. While much research ad rehabilitatio effort has bee aimed at improvig the readig ability of low-visio patiets [2, 3], the icreasig use of televisio ad persoal computers heightes the eed for image ehacemet particular to these domais as well. Previous work o image ehacemet as a low-visio aid [4-9] has bee carried out with ucompressed images. However, may images are ow hadled i compressed formats, e.g. i computers ad digital televisio, ad the expected growth of such applicatios is likely to icrease the eed for ehacemet that ca perform well without access to the ucompressed origial. This paper describes a image ehacemet approach for lowvisio viewers applied directly withi the compressio domai, based o aspects of the JPG stadard compressio protocol that are also applicable to MPG compressio for movig images [10]. Images Mauscript received October 6, 2003. This work was supported i part by NIH grats Y05957 ad Y12890 to P ad by a postdoctoral fellowship program from Korea Sciece & gieerig Foudatio (KOSF) to JK. Jisha Tag was with Schepes ye Research Istitute, Harvard Medical School, Bosto, MA 02114 USA. He is ow with the Departmet of lectrical ad Computer gieerig, Uiversity of Virgiia, Charlottesville, VA 22903 USA. Jeoghoo Kim was with Schepes ye Research Istitute, Harvard Medical School, Bosto, MA 02114 USA. li Peli is with Schepes ye Research Istitute, Harvard Medical School, Bosto, MA 02114 USA. (e-mail: eli@visio.eri.harvard.edu). delivered ultimately i a JPG format may be ehaced: prior to compressio, after decompressio, or withi the JPG domai (the method chose here). Pre-compressio ehacemet has the disadvatage of reducig the amout of compressio subsequetly possible as compared to the uehaced origial. For example, Hader et al [11] proposes to low-pass filter the image before compressio ad the ehace it after decompressio. I additio, as it is desiged to reduce high frequecy cotet, ay amout of compressio will couteract the ehacemet effect. O the other had, postcompressio ehacemet is more likely to icrease block artifacts: compressig a image creates block artifacts above, ear-to or (ideally) below their visibility thresholds, ad ehacemet is likely to icrease the visibility of the artifacts as well. Our approach of applyig image ehacemet withi the JPG domai helps to reduce this problem. Sice block artifacts are maily affected by the quatizatio of low-frequecy coefficiets, keepig these (ad the DC) coefficiets umodified or miimally adusted should reduce the severity of artifacts. The algorithm implemeted here has this property ad ideed reduces the appearace of block artifacts. To apply ad assess the visual effects of image ehacemet, oe requires a visually meaigful defiitio of cotrast [See, for example, a special issue of Visio Research, vol 37(23) 1998, coverig this topic]. Various cotrast measures have bee proposed, ad those defied i the spatial frequecy domai may be cosidered applicable for use i our proposed image ehacemet applicatio. I particular, Peli [12] defied cotrast for atural or complex images as the ratio of the bad-pass filtered image at a give frequecy bad to the low-pass filtered image oe octave below it. Similarly, Toet [13] defied cotrast as the ratios of low-pass versios of the image. The measure we propose is based o ratio of bad pass versios of the image redefied withi the discrete cosie trasform (DCT) domai that is used i JPG compressio [10]. As our goal is to improve everyday image viewig for low visio patiets, we chose to use TV type moitors (NTSC, iterlaced) i our subective evaluatio experimet rather tha computer progressive displays; although use of computer displays is also icreasig, idividuals i the agig populatio view televisio screes more ofte ad for loger periods tha computer screes. Our efforts as described here for still images are expected to be applied i a future study to movig images usig the (JPG-like) MPG format. II. IMAG NHANCMNT IN TH JPG DOMAIN A. JPG basics JPG is a image compressio ad decompressio stadard that is based o the DCT [14, 15]. I the compressio stage, a give image is first divided ito o-overlappig blocks of 8 8 pixels. The two-dimesioal DCT is the computed for each block. The 64 DCT coefficiets are subsequetly quatized usig a quatizatio table (a lossy operatio), ad thereafter they are losslessly coded ad trasmitted or stored together with the quatizatio table. I the decompressio stage, each block of the received compressed data is 1
decoded, dequatized usig the quatizatio table, ad iverse DCT trasformed ito a image block. The specific desig of a quatizatio table is importat because of its iflueces o both the compressio ratio ad recostructed image quality. A Basic Quatizatio table (see Figure 5 i [15]) is ofte used i JPG based image compressio, but other quatizatio tables ca be derived from it by adustig a quality factor [15]. Ay other quatizatio table is acceptable withi the JPG stadard sice the table is trasmitted or stored with the coded image. B. Cotrast Measure of Images i DCT Domia Image ehacemet methods may be classified ito those that ehace cotrast directly ad those that ehace cotrast idirectly. Direct cotrast ehacemet methods [16-18] measure the image cotrast before ehacemet. I this paper, we itroduce a ew direct cotrast ehacemet method based o a defiitio of image cotrast i the DCT domai. Let D be a 8 8 array of DCT coefficiets of a image block. The DCT coefficiets represet the spatial frequecy cotet of the image i a similar way to the coefficiets i oe quadrat of the two dimesioal Fourier domai. The d 00 coefficiet represets the DC level of the block, ad the other coefficiets represet spatial frequecies that icrease with their distace from d 00. For istace, coefficiets d 04 ad d 40 represet a spatial frequecy of 4 cycles per block i the horizotal ad vertical directios, respectively. A bad limited cotrast measure i the DCT domai ca be defied by C = ( 1 14, Z ), (2) where 1 d i, i+ = =, (3) N is the average amplitude over a spectral bad eclosed by a ellipse i (1) ad + 1 = 14 + 1 < 8 8 (1) N. (4) Note that the 14 bads defied i this way represet approximately equal spatial frequecies ad are cosistet with the zigzag structure of codig the blocks withi the JPG stadard. C. Image cotrast ehacemet i the DCT domai Let the cotrast of the various bads defied i the compressed/quatized DCT block be C = ( c1, c2,..., c14 ) ad the cotrast of the ehaced image be C = ( c1, c2,..., c14 ). If we ehace the cotrast by the same costat ehacemet factor, λ, for all bads of frequecies, the the relatioship betwee them ca be described by c = λc. (5) Thus, we have 1 = c = λc that ca be expaded as follows λ = λ λ 1 λ 1 = 1 = 2 1 1 2 2 = λ 2 = λ 0 = 2 0, (6), λ where 0 = 0. Usig equatio (3) ad (7), we ca obtai the ehaced DCT coefficiets d i as i+ d i = λ di, (8) that ca be realized by weightig the dequatizatio table. The ehacemet algorithm (Figure 1) shows that the modified dequatizatio table Q is obtaied by weightig the quatizatio table Q, trasmitted with the compressed image, by the followig equatio: Q = Λ Q, (9) where the otatio * is poit-wise multiplicatio of two matrices ad i+ Λ( i, ) = λ, (10) is the filterig matrix. The post-trasmissio ehacemet of a image i the DCT domai provides distict advatages over image ehacemet methods that either utilize spatial filterig after applicatio of a Iverse Discrete Cosie Trasform (IDCT) or filter the image before trasmissio. The preset method has miimal computatioal cost (oly 64 multiplicatios) ad uses the IDCT operatio already performed as part of the decompresssio. Implemetatio merely requires access to the quatizatio table employed to decode the image. Futhermore, the method allows a user to choose the desired filter itractively. For example, the user may cotiuously vary λ, ad view the result i real-time, i order to select the appearace meetig their idividual requiremets. This ca be easily acomplished i real time. (7) 2
Figure 1. Image hacemet i JPG Domai is achieved by weightig the quatizatio table with a appropriate filter (weightig array). Q is the modified quatizatio table obtaied by multiplyig (poit by poit) the weightig array with the quatizatio table Q, which ca be accessed from the JPG bit stream. D. Directioal Cotrast hacemet of Images i the DCT Domai Applyig ehacemet i the iterlaced video domai ormally results i a substatial icrease i iterlace artifacts. Such artifacts may be reduced by ehacig the horizotal directio cotrast (withi a sca lie) more tha the vertical directio cotrast. Usig the same formulatio as above, more cotrast ehacemet i the vertical directio frequecy i the DCT domai (the horizotal space domai) may be achieved by limitig the ehacemet to the upperright segmet of the filterig matrix usig: d i = i+ λ d di, i, i i > III. MTHODS. (11) A. Choice of quality factor for JPG Compressio For our experimets we eeded a level of compressio appropriate for good quality TV pictures, oe i which the compressed images were almost idistiguishable from the origials. I computer scree JPG-based compressio applicatios [15] the quality factor, q, is ofte set at 50. We tested 4 cadidate quality factors, oe of which is above ad two of which are below that value, q=15, 35, 50 or 60, ad applied subective testig by ormally sighted observers to compare the quality of the variously compressed versios with their ucompressed origials. Thirty aalog images were captured radomly from cable televisio broadcasts, ad each image was compressed usig the four differet quality factors. The decompressed images ad the ucompressed origial images were displayed o a 27-ich televisio i radom order. Subects with ormal visio were asked to rate the display images i terms of their quality as compared to a stadard TV image usig a graphics tablet ad mouse. The distace from the TV moitor to the subect was 36 iches (such short viewig distace is frequetly used by visually impaired observers). Subects were asked to rate the decompressed images as very bad, bad, acceptable, good, or excellet, ad their ratigs were scored o a correspodig scale of 1 to 5. ight subects (20 to 40 years old) with ormal, or corrected to ormal, visio participated i the experimet. The subects quality ratig scores (1 to 5) were used to calculate receiver operatig characteristic curves (ROCs) [19]; a o-stadard applicatio of ROC aalysis was used for a compariso betwee the ratigs for the variously compressed versios ad the ucompressed origial. (Typically i ROC aalysis, performace is compared agaist a kow groud truth see Sectio III-G below). The ROC curve was obtaied as follows: For the first data poit o the curve, the fractio of the subects givig a ratig of very bad (correspodig to a score of 1) for the origial image was plotted agaist the fractio of the subects givig the same ratig for a particular compressed image. The correspodig cumulative fractios were calculated similarly for the other poits of the curve, ad the data were fitted usig a biormal model [19]. The area uder the ROC, A z, was take as a measure of the relative quality of the compressed images. The level of correlatio betwee the resposes for the two compared coditios was used to determie the statistical sigificace of the differece betwee the areas uder the two ROC curves. (A p-value of less tha 0.05 level was cosidered to be sigificat [19].) The ROC curves are show i Figure 2. For q = 15, 35 ad 50, the quality of the uehaced compressed images was iferior to that of the ucompressed images as the A z was < 0.5 (p < 0.013). For q = 60, the perceived quality differece betwee the origial images ad the compressed images was ot statistically sigificat (p = 0.26). Therefore, i the followig experimets, q = 60 was used as the quality factor i the JPG compressio. True postive fractio 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Data for q=15 Fittig for q=15,p=0.0000 Data for q=35 Fittig for q=35,p=0.0000 Data for q=50 Fittig for q=50,p=0.0130 Data for q=60 Fittig for q=60,p=0.26 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 False positive fractio Figure 2. ROC aalysis for the choice of quality factor, q, of compressed images as compared to ucompressed origials. The area uder the ROC, A z, is take as a measure of the relative quality of the compressed images. The p- value idicates the statistical sigificace of the differece betwee two areas. Oly for q = 60 the perceived quality differece 3
TBM-00426-2003.R1 (a) (d) (b) (e) (c) (f) Figure 3. xamples of processed images: (a), (b) ad (c) are samples of ehaced images; (d), (e) ad (f) are differece images betwee the decompressed origial image ad the ehaced image i (a), (b) ad (c), respectively. haced images i (a) ad (b) were obtaied usig λ = 1.9. The ehaced image i (c) was obtaied usig λ = 1.5. Note, the effect of the ehacemet as see o the TV scree is more dramatic tha it appears i prit. betwee the origial images ad the compressed images was ot sigificat; we used images with this quality factor to evaluate our ehacemet techique. B. Directioal ehacemet I pilot experimets we oted that whe ehaced images obtaied by the o-directioal ehacemet method were displayed o the televisio moitor they produced a sigificat flickerig artifact. This flickerig did ot occur whe the same images were displayed o a computer display. The flickerig was a result of field iterlacig. While flickerig artifacts i a sigle iterlaced frame are well kow to occur due to image motio occurrig betwee the two fields, this artifact was also see i images with miimal or o motio. The flickerig artifact occurred whe the ehacemet resulted i two abuttig raster lie segmets with a large differece i brightess. As these segmets were alterately refreshed (repeatig at 30 Hz), they appear to move or flicker. The effect is eve stroger from the shorter observatio distace typically used by low visio persos (approximately 36 iches, 91 cm). Sice the artifact appeared to be associated oly with the ehacemet of vertical cotrast (i the spatial domai), we applied the directioal ehacemet method (Sectio II-D), which substatially reduced these flickerig artifacts. 4
C. Image processig The static TV images (see Sectio III- below) were compressed usig a stadard JPG compressio algorithm with the quality factor of 60. The correspodig quatizatio table is 13 10 11 11 Q = 14 19 39 58 9 10 10 14 18 28 51 74 11 13 18 30 44 62 76 13 15 19 23 45 51 70 78 The images were subsequetly decompressed. I the decompressio stage, the directioal ehacemet method (Sectio II-D) was applied to ehace the images. Oly the lumiace compoet was ehaced; the color compoets were ot modified. For each JPG image, we implemeted 16 differet processig levels, each correspodig to a differet value of λ, ragig from 0.1 to 1.9 (Table 1). The image with λ = 1 reproduced the origial decompressed uehaced image. The levels of λ = 1.1 to 1.9 produced ehaced images while λ = 0.1 to λ = 0.9 levels resulted i low pass filterig of the images ad thus produced degraded images. Degraded images were ecessary to verify that the subects were respodig to cotrast ehacemet, ad ot ust cotrast modificatio. 8 19 21 32 41 54 65 82 90 32 46 46 70 87 83 97 80 41 48 55 64 82 90 96 82 49 44 45 50 62 74 81 79 Table 1. The 16 image ehacemet levels used i procedure 1. The first 6 levels are images processed with values chose to produce degraded images (λ < 1.0). The 7 th level is the origial image ad the other 9 ehacemet levels are processed usig the JPG ehacemet algorithm described here with the idicated gai factors (λ > 1.0). Level No. Scale Factor or λ 1 0.1 2 0.3 3 0.5 4 0.7 5 0.8 6 0.9 7 1.0 8 1.1 9 1.2 10 1.3 11 1.4 12 1.5 13 1.6 14 1.7 15 1.8 16 1.9 (12) The iclusio of degraded images also prevets the origial images from always beig the lowest-cotrast images preseted. Images were preprocessed ad stored for presetatio durig the experimet. Figure 3 shows samples of ehaced images. The prited images are ot a valid represetatio of the displayed images. I particular, the prited image caot preset the flickerig effect associated with the iterlaced video used i the presetatio (this effect also caot be see whe the same image is preseted o a progressive display). D. hacemet evaluatio by low-visio patiets Two procedures were used to examie the low visio patiets appreciatio of the JPG ehaced images. I the first procedure, a subect was istructed to select the level (oe of sixtee choices of λ) that they cosidered to look the best. I the secod procedure, the subect compared the origial images (The origial images here are uehaced decompressed images. Because we are iterested i ehacig compressed images, we do t use the images before compressio i the followig experimets for low-visio patiets.) to the images that were processed usig the media of the levels selected i procedure 1, ad the subect raked the images o a scale of perceived quality (see detailed descriptio of procedures 1 ad 2 below). Prior to testig, each subect was asked about the size of his or her televisio at home, ad about how close to it he or she usually sits while watchig a program. Subects were the seated at a distace from our 27-ich televisio test moitor that approximated the visual agle they were accustomed to viewig their ow set at home. This distace was reduced if the subect could ot discer image chages as the ehacemet level varied. For our low-visio subects, the average seatig distace was 36.0 ± 14.2 iches from the televisio, while the stadard viewig distace of a 27-ich televisio would have bee 105 iches [20]. Because may of the subects were elderly, with little or o computer experiece, before the actual experimetal sessio they sometimes eeded a practice sessio to become comfortable with the graphics tablet ad mouse. The images used i these practice sessios were differet from those used to collect the image quality data. The room was dimly lit by recessed overhead icadescet lamps, ad the lumiace at the moitor surface was measured approximately 1 ftcadle. 1) Procedure 1: Selectig the Preferred JPG hacemet Level Subects were show a static image (draw from a set of 10 differet images) o the TV scree. By movig the mouse up ad dow o the blak graphics tablet, they could select which of the 16 predetermied levels of cotrast adustmet were applied to the image. ach subect was asked to fid the spot o the tablet correspodig to the image adustmet level where, you like the picture the best, where it is clearest for you, ad where you got the most detail out of the picture. Oce subects foud a image that looked the best to them, they recorded that settig by clickig o a mouse butto. After a respose, the ext image from the set of 10 was displayed. For each trial, the mappig of the active regio of the graphics tablet to the ehacemet level preseted was radomly shifted so that the subects were uable to associate a fixed mechaical positio with their choices. 2) Procedure 2: Perceived Image Quality The rouded media level [21] of procedure 1 was chose as the idividually preferred ehacemet level for use i procedure 2. Four versios of each of 50 images (a total of 200 images) were show to subects i a radomized sequece. The four versios for each image icluded: (1) origial image; (2) idividually chose ehacemet (based o procedure 1); (3) a degraded image (λ = 0.8); ad (4) a image ehaced by a secod arbitrarily selected ehacemet level. The 50 images used i procedure 2 did ot iclude the 10 used i procedure 1. The secod arbitrarily selected ehacemet level was selected to supply aother ehacemet which had a differet appearace from the level selected by the patiet. The secod arbitrarily selected ehacemet level was chose to be several levels above the idividually-selected ehacemet level for those who selected a low level of ehacemet, ad several levels below the idividuallyselected ehacemet level for patiets who selected a high level of ehacemet. Table 2 lists the possible idividually chose ehacemet levels ad the correspodig secod selected ehacemet levels. 5
By movig the mouse vertically o the graphics tablet, the subects rated the quality of each ehaced image agaist a uehaced versio of the same image. The subects were asked to rak each image as better, slightly better, typical, slightly worse, or worse tha the origial image, with these rakigs prited i large fot o the graphics tablet, with better ear the top of the tablet, ad worse ear the bottom. Before the computer accepted their score, the subects were required to compare the test image to the origial image at least oce by movig the mouse to a desigated sectio at the right edge of the tablet marked by a black stripe, thereby displayig the origial image. Oce this was viewed, the subect was allowed to grade the test image. If desired, subects could view the origial image ad compare it with the test image multiple times. Table 2. The available idividually chose ehacemet levels ad their correspodig secod arbitrarily selected ehacemet levels. Idividually chose ehacemet. Image Acquisitio Secod arbitrarily selected ehacemet Level 1 10 Level 2 10 Level 3 10 Level 4 11 Level 5 12 Level 6 12 Level 7 12 (Origial) Level 8 12 Level 9 12 Level 10 13 Level 11 13 Level 12 14 Level 13 16 Level 14 12 Level 15 12 Level 16 12 Sigle video frames (static images) were radomly grabbed from various shows o cable televisio chaels i Bosto, Massachusetts o Jue 26, 2000. The frames were captured usig a Video Toaster card [22] as 480 x 720 x 3 RGB bitmap images. They were coverted usig Matlab [23] to RTV format for presetatio o a TV moitor usig a SpeedRazor graphics card [24]. Of the 200 digitized images acquired, 127 were udged by 2 ormally-sighted observers to cotai little or o apparet motio due to differeces betwee the 2 iterlaced fields. Fifty of these were selected radomly for the study. F. Apparatus All processig, experimet cotrol, ad aalysis were doe usig a Itel PC ruig Widows NT 4.0 (Service Pack 6). Images were displayed o a 27-ich (diagoal) Soy Triitro NTSC format televisio moitor usig a Video Toaster image processig system [22], uder the cotrol of programs writte i Microsoft Visual Basic ad Matlab [23]. I procedure 1, subects moved the mouse over a 12-ich SummaSketch III [25] graphics tablet device to select ehacemet levels. The same tablet was used to grade the images i procedure 2, as described above. I both experimets, subects desigated their fial choice by pressig the mouse butto. G. Data Aalysis Data from procedure 2 was aalyzed usig the ROC sigal detectio approach [26] described above. The Rockit program [19,27] was used to determie the area uder the fitted ROC curve, A z [28]. Paired comparisos were made betwee resposes to the origial images ad a set of correspodig processed images. As there were three sets of processed images for each subect, three ROC curves were computed (see Figure 4), represetig the perceived image quality of each of the processig optios as compared with the origial. I traditioal ROC aalysis, system (e.g. subect) resposes to oise presetatios ad to oise-plus-sigal presetatios are compared. I our study, the origial images are treated as the oise presetatios, ad the processed versios are treated as the oiseplus-sigal presetatios. As ca be see i Figure 4, our raw data cosisted of multiple distributios alog the perceived image quality dimesio (for simplicity, Figure 4 oly shows data for 3 of the 4 Figure 4. Oe patiet s distributio of resposes to the origial, chose ehacemet, ad degraded images. For simplicity, the resposes to the secod ehacemet are ot icluded. The resposes are take from the graphics tablet readigs ad quatized to the rage 0 to 10, where 0 represets a score of worse ad 10 represets better i compariso to the origial image. These data were for a patiet who clearly preferred the ehacemet; the three distributios are clearly separated. These data were used to costruct the ROC curves show i pael (a) of Figure 5. image sets). Whe the perceived image quality of the processed images was higher tha the origial images (level 9 image set i Figure 5(a)), A z was greater tha 0.5. For the degraded image set, the perceived image quality distributio was lower tha that of the origial images, creatig a A z of less tha 0.5. Our ROC aalysis measures perceived relative image quality, ad ot ehacemet detectio, as might be doe i aother applicatio. Cosequetly, the traditioal labels of the axes of the ROC figure (e.g. true-positive fractio, or hit rate) do ot to represet our situatio. I our aalysis, the true-positive fractio dimesio is the proportio of the processed image set with a higher perceived image quality tha the origial images, while the false-positive fractio ( false-alarm rate) dimesio is the equivalet proportio for the origial images perceived as havig higher quality tha the processed images (a higher quality beig relative to the criterio used for the particular poit o the ROC curve). However, we use the traditioal axis labels as show i Figure 5. 6
While the graphics tablet gives a cotiuous respose measure, for some subects, the resposes were multi-modal, a cosequece of the large-fot guide words o the tablet (may subects did ot iterpolate betwee the five words). The data show i figure 4 has a slight tedecy towards this multi-modal respose patter. I additio, ofte the respose distributios were ot ormally distributed. ve so, the Rockit program appeared to give a reasoable fit to our data i most cases (Figure 5). The Rockit program provides cofidece limits for each ROC curve area [28], ad these were used to determie the sigificace of the resposes of idividual subects to a particular type of image processig. Sice the image ehacemet levels used i procedure 1 were ordered but the perceptual itervals were ot ecessarily equal, oparametric statistical tests were used for these comparisos. A z data distributios from procedure 2 were foud to be approximately ormally distributed, ad thus parametric statistical tests were used for these comparisos. as did all others, clearly reected the degraded images (A z = 0.01). (b) A more typical example i which oly slight ad ot statistically sigificat preferece was foud for the chose ehacemet: a patiet with Retiitis Pigmetosa (visual acuity 20/83). Here, preferece for the chose ehacemet (level = 8,A z = 0.61) was slightly but ot sigificatly higher tha for the origial. This subect showed o sigificat preferece for the secod ehacemet level = 12 (A z = 0.37), ad also reected the degraded image (level = 5, A z = 0.002). Note that the ROC data show i pael (a) is costructed from distributio of data show i Figure 4. H. Subects Patiets were recruited from cliical practices that cocetrated o retial diseases, ad most had cetral retial dysfuctio from such diseases as age related macular degeeratio. All patiets siged a subect coset form, approved by IRB committee. All icluded subects were at least 18 years of age, able to read ad uderstad the coset form, able to follow verbal istructios i glish, ad were ot sufferig from a coditio, such as arthritis, that might ihibit their ability to cotrol the mouse. We did ot recruit low-visio subects who use a telescope device to view televisio. Subects viewed the TV images with both eyes. Visual acuity was measured usig a BVAT (Model No. 22-4850, Metor O&O Ic). Visual fields were measured usig a Bausch & Lomb Auto-Plot Taget Scree (Cat. No. 71-54-41) to documet cetral field loss (CFL). Visual fields were measured moocularly, usig a 6 mm target at 1 meter, with the subect wearig habitual distace correctio (e.g. glasses). Some of the patiets did ot udergo the visual field tests but had a clear diagosis of macular lesios accoutig for their acuity loss ad thus were presumed to have CFL. Oe subect of the 48 total subects who were referred did ot meet the study iclusio criteria. The remaiig subects (Group A, N = 47) completed procedure 1. Due to cliical schedules ad physical costraits (such as age-related stamia), fewer subects also completed procedure 2 (Group B, N = 27). Table 3 shows the characteristics ad umbers of subects that completed the two portios of the experimet. (a) Table 3. The cliical characteristics of the participatig patiets. Due to schedule costraits ad other factors oly some of the patiets who completed procedure 1 (Group A) also completed procedure 2 (Group B). N is the umber of subects i each group, ad CFL is the umber of subects who had documeted cetral visual field loss i both eyes. Group N Age (years) Average ± SD Media (Rage) VA (Log MAR) Average ± SD Media (rage) CFL A 47 61.8 ± 20.0 68.5 (19.2-86.5) 0.93 ± 0.31 0.88 (0.50-2.10) 32 B 27 59.5 ± 20.4 68.1 (20.8-86.5) 0.99 ± 0.37 0.96 (0.50-1.20) 15 IV. RSULTS (b) Figure 5. The ROC fitted curves for two patiets. The thick lies are the fits to the filled triagular symbols (the chose ehacemet level) ad the thi lies are the fits to the ope square symbols (the secod ehacemet level tested). The dotted lies are the fits to the filled diamod symbols (degraded image). (a) A patiet with Optic Atrophy (visual acuity 20/250) who clearly favored the chose ehacemet (level = 9,A z = 0.83), ad showed o preferece for the secod ehacemet (level = 12, A z = 0.49). This patiet, A. Reductio of block artifacts by image ehacemet i JPG domai Figure 6 provides a compariso of JPG based ehacemet with stadard post-compressio ehacemet of a JPG compressed image. To better illustrate the effects i prit, we show a elarged partial image of the familiar Lea image. Figure 6(a) shows the origial (ucompressed) image, while Figure 6(b) is a JPG decompressed image without ehacemet with a Peak Sigal-to- 7
Noise Ratio (PSNR) of 35.4 db (q = 60) [20]. Oly mior degradatio is evidet with this level of compressio. Figure 6(c) shows the decompressed image ehaced usig covetioal postcompressio ehacemet with Pait Shop Pro TM [29], applyig the Image-Sharpe tool four times recursively. This produce similar cotrast ehacemet to the λ = 1.35 compariso images. As see i Figure 6(c) this processig results i a high pass filterig cotrast ehacig effect but also causes a obvious icrease i block artifacts. Figure 6(d) shows the result of the JPG based ehacemet applied i the decodig stage. hacemet with λ = 1.35 was used to produce a similar ehacemet effect to that of Figure 6(c), but, as is evidet i the figure, that level of ehacemet is achieved with less severe block artifacts. B. xperimetal results for low-visio patiets Figure 7 is the histogram of the selected preferred ehacemet level i procedure 1 (Group A, N = 47). The media preferred level selected was 8, correspodig to λ = 1.1(25% quartile: level 7, 75% quartile: level 10), which was sigificatly differet from the origial image level of 7 (Wilcoxo siged-rak test, Z 34 = 3.831, p < 0.0001). The ehacemet levels that the patiets selected were ot correlated with their visual acuities (r = 0.030, p = 0.842). Seve patiets preferred the origial image without ehacemet (λ = 1). leve of the 47 patiets actually selected degraded images, although oly mildly degraded images were selected (most chose λ = 0.9 with oly 2 patiets selectig λ = 0.8). It is likely that the patiets could ot differetiate these low level degradatios from the origial images. I respose to our questios ad ofte spotaeously, all of the patiets who selected degraded images reported that the ehaced images appeared the same or were ot as clear as the origial images. Most of the 29 (62%) subects who selected ehaced images reported, i respose to questios or spotaeously, that the ehaced images were clearer, sharper, ad easier to see tha the origial oes. Twety of the 27 subects who completed both procedures 1 ad 2 repeated procedure 1 after completig procedure 2. For this group, (a) (c) (b) (d) Figure 6. The effect of ehacemet o block artifacts. (a) The origial Lea image. Note that the image is a magified partial face image. (b) The JPG decompressed image without ehacemet (PSNR = 35.4 db, q = 60). Oly mior degradatio is evidet with this level of compressio. (c) The compressed image of (b) ehaced with covetioal graphics software (Pait Shop Pro TM ) (PSNR = 19.9 db). Note the clear visibility of block artifacts i the image. (d) Image ehaced with the proposed JPG based ehacemet, λ = 1.35 (PSNR = 21.1 db). Note the similar cotrast ehacemet compared to (c) with fewer block artifacts. 8
the media score of their first selectio was 8.5 ad the media selectio o repeat was 8.0, this differece was ot sigificat. The idividual selectios i the two repeats were highly correlated (r = 0.764, p < 0.0001). differet tha the A z with the secod ehacemet (paired sample T test, t = 6.745, p < 0.0001) ad the degraded ehacemet (t = 4.870, p < 0.0001), but the secod ehacemet was ot sigificatly statistically differet tha the A z with the degraded ehacemet (t = 0.802, p = 0.430). Figure 7. Preferred ehacemet level distributio foud i procedure 1 for Group A (N = 47) ad Group B (N 27). The two groups did ot sigificatly differ i their selectios of ehacemet level (25% level = 7 ad 6.5, respectively, media level = 8 for both, 75% level = 10 for both). Note that few patiets selected degraded images ad the oly images with slight degradatio were selected. I procedure 2, patiets viewed 50 images, each with their idividually-chose level of ehacemet, with a arbitrarily selected level of ehacemet (Table 2) ad a degraded versio of the image (level 5, λ = 0.8). The patiets compared each of these images to a uehaced versio ad idicated a comparative perceived image quality usig the graphics tablet. The measuremets for each of the three image versios were quatized to eleve levels ad coverted to ROC curves each with a associated area, A z. Figure 5 shows the results for two subects. Oe patiet clearly favored the images with the chose ehacemet (this was the oly patiet who had such a clear appreciatio of the ehacemet). The other patiet oly slightly preferred the ehaced image ad that effect was ot statistically sigificat. These latter resposes are similar to those of most of the patiets. The degraded (λ = 0.8) images were clearly reected by all patiets i this procedure. If the quality of ehaced images was udged to be superior to that of the origial images, A z (the area uder the ROC curve) would be larger tha 0.50. Seve subects of twety-seve had A z greater tha 0.5 for their idividually selected ehacemet level, but the differece was statistically sigificat for oly oe of these subects (A z = 0.83, Asymmetric 95% Cofidece Iterval (0.74, 0.90)). The average A z for group B was 0.38 ± 0.19. Although most subects idicated a preferece for a particular JPG ehacemet i procedure 1, most did ot fid idividuallyselected ehacemets to be much better tha the origial images. I procedure 2, subects also viewed a secod arbitrarily selected ehacemet alog with itetioally degraded images. This allowed us to ivestigate the validity of our psychophysical method. If our method was flawed, we might expect that the subects would ot report a differece i image quality for these other image sets. All of the subects did idicate that the secod ehacemet set had less image quality (A z = 0.20 ± 0.16), ad all idicated that the degraded images had lower quality scores (A z = 0.18 ± 0.15) tha the origial images, eve though a very modest level of degradatio was used. The A z for chose ehacemet was statistically sigificatly V. CONCLUSION A image ehacemet algorithm for low-visio patiets i the JPG domai has bee proposed ad implemeted. The algorithm, which was tested here o static images, is iteded for use with movig video sequeces ad ca be easily applied to MPG video formats that iclude a JPG-like codig. The proposed algorithm has umerous advatages. The computatioal cost is very low, sice it eeds oly to filter the quatizatio table i the decompressio stage (64 multiplicatios), allowig for a real-time implemetatio. Because the ehacemet is post-compressio ad could be implemeted i the user s TV receiver, it could be adusted maually by low-visio patiets via a remote cotrol uit. This would permit idividual ehacemet tued to the patiets visual loss ad would allow adustmet i respose to the differig spatial cotet of images. xperimetal results have show that most low visio patiets select a moderate level of ehacemet whe viewig still images displayed o a televisio moitor. Surprisigly i procedure 2, whe the patiets compared their idividually selected ehacemet to the origial, oly oe subect showed a sigificat level of preferece. The reasos for this dichotomy betwee the results of the two procedures are ot clear ad will require further ivestigatio. Patiets remarked that they preferred to see atural-lookig images, ad that the ehaced images were, to some extet, distorted. Wheever the patiets could otice the distortio as such, they reected it. The specific filter implemeted i this study (quatio 11) provides a uiform cotrast ehacemet for all frequecies (though aisotropic). While this cocept is simple ad the resultig filter has a elegatly simple (sigle parameter) structure, it may be a less tha optimal way of ehacig images for the visually impaired; a limited bad ehacemet filter might be more effective sice it reduces the distortios [30]. I additio, sice most low visio patiets are completely uable to see very high frequecies, it might be better to actually suppress these frequecies as they ca cause visible quatizatio artifacts, potetially without beefit. I additio the patiets foud the iterlace artifacts i our static images to be particularly bothersome. Whe compared to froze sigle frames, iterlace artifacts are much less oticeable i movig images; movig videos might ot be as obectioable with ehacemet. Implemetatio ad testig of this cocept with movig video will be the topic of further study. ACKNOWLDGMNT The authors thak Robert B Goldstei for programmig help ad Avi Vora for helpig to ru the patiet experimets. RFRNCS [1] Louis Harris ad Associates, I., The Lighthouse Natioal Survey o Visio Loss: The experiece, attitudes, ad kowledge of middle-aged ad older Americas. 1995, The Lighthouse, Ic. [2]. Peli,. M. Fie, ad K. Pisao, "Video ehacemet of text ad movies for the visually impaired," i Low Visio:Research ad New Developmets i Rehabilitatio, A. C. Kooima, P. L. Looiesti, J. A. Wellig, ad G. J. va der Wildt, eds. IOS Press, Amsterdam, pp. 191 198,1994. 9
[3] T. B. Lawto, "Image ehacemet filters sigificatly improve readig performace for low-visio observers," Ophthalmic ad Physiological Optics, vol. 12, pp 93-200, 1992. [4]. Peli, "Perceived quality of video ehaced for the visually impaired," i Visio Sciece ad Its Applicatios. Sata Fe, New Mexico, pp. 46-48, 1999. [5]. Peli ad T. Peli, "Image ehacemet for the visually impaired," Optical egieerig, vol. 23, pp. 47-51, 1984. [6]. Peli, "Simple 1-D image ehacemet for head-mouted low-visio aid," Visual Impairmet Research, vol. 1, pp. 3-10, 1999. [7] L. Iseberg, A. Luebker, ad G. Legge, "Image ehacemet of faces for ormal ad low visio," Ivest Ophthalmol Vis Sci, vol. 30 (suppl): 396, 1989. [8]. Peli, R. Goldstei, B. Youg, C. Trempe, S. Buzey, "Image ehacemet for the visually impaired: simulatios ad experimetal results," Ivest Ophthalmol Vis Sci, vol..32, pp. 2337-2350, 1991. [9]. Fie,. Peli, "hacemet of text for the visually impaired," J. Optical Soc. Am. A, vol. 12, pp. 1439-1447, 1995. [10] J. Kim, ad. Peli, "MPG based image ehacemet for people with low visio," i SID 2003. Baltimore, MD, 2003, pp. 1156-1159. [11] O. Hader, A.Ster ad R. Koresh, "hacemet of a image compressio algorithm by pre- ad post-filterig," Optical gieerig, vol. 40, pp. 193-199, 2001. [12]. Peli, "Cotrast i complex images," J Optical Soc Am A, vol. 7, pp. 2032-2040, 1990. [13] A. Toet, "Multiscale cotrast ehacemet with applicatio to image fusio," Optical gieerig, vol. 31, pp. 1026-1031, 1992. [14] V. Bhaskara ad K. Kostatiides, Image ad Video Compressio Stadards: Algorithms ad Architecture, Kluwer Academic Publishers, 1995. [15] R. Asari ad N. Memo, "The JPG lossy image compressio stadard," i Hadbook of Image ad Video Processig, Academic Press, pp. 513-526, 2000. [16] J. Tag,. Peli, ad S. Acto, " Image hacemet Usig A Cotrast Measure i the Compressed Domai," I Sigal Processig Letters, vol. 10, pp. 289-292, 2003. [17] A. Beghcladi ad A.L.Negrate, "Cotrast ehacemet techique based o local detectio of edges," Computer visio, Graphics, Image Processig, vol. 46, pp. 162-274, 1989. [18] W.M.Morrow ad et al, "Regio-based cotrast hacemet of Mammograms," I Tras. O Medical Imagig, vol. 11, pp. 392-406, 1992. [19] The software used i the ROC aalysis is available at http://www-radiology.uchicago.edu/krl/toppage11.htm. [20] M. Ardito et al., "Preferred viewig distace ad display parameters," i MOSAIC Hadbook, pp. 165-181, 1996. [21] J. Freud, ad B. Perles, "A New Look at Quartiles of Ugrouped Data," The America Statisticia, vol. 41, pp. 200-203, 1987. [22] NewTek, VideoToaster. 1998, NewTek: Sa Atoio, TX. [23] MathWorks, MatLab Image Processig Toolbox. 1997, MathWorks: Natick, MA. [24] I:Syc, Speed Razor. 1999, I: Syc: Bethesda, MD. [25] GTO CalComp, I., SummaSketch III. 1989, GTO CalComp, Ic. Columbia, MD. [26] N. Macmilla ad C. Creelma, Detectio theory: A user's guide, Cambridge Uiversity Press, 1991. [27] C. Metz, P. Wag, ad H. Kroma, "A ew approach for testig the sigificace of differeces betwee ROC curves measured from correlated data," i Proceedigs of the VIII Coferece o Iformatio Processig i Medical Imagig. The Hague, 1983. [28] J. Haley, B. McNeil, "The meaig ad use of the area uder a receiver operatig characteristic (ROC) curve," Radiology, vol. 143, pp. 29-36,1982. [29] "http://www.asc.com/products/psp/". [30]. Peli, "Limitatios of image ehacemet for the visually impaired," Optometry ad Visual Sciece, vol. 69, pp. 15-24, 1992. Jisha Tag received his Ph.D. degree from Beiig Uiversity of Posts ad Telecommuicatios, P.R. Chia, i 1998. From September 1998 to Jue 2000, he worked as a ivited researcher i ATR Media Itegratio ad Commuicatios Research Laboratories (MIC), Kyoto, Japa. I Jue 2000, he wet to Harvard Medical School to work o image ehacemet techology. Sice Dec. 2001, Dr. Tag has bee workig as a research scietist i Departmet of lectrical ad Computer gieerig, Uiversity of Virgiia. His research iterests are image ehacemet for low-visio, cotet based image retrieval, cell detectio ad trackig, face detectio ad recogitio. Dr. Jisha Tag is a seior member of I. Jeoghoo Kim received his MS ad PhD degrees i lectroics from the Yosei Uiversity, Seoul, Korea i 1989 ad 2000, respectively. Sice 1989 he has bee with the Sigal Processig ceter at Samsug lectroics as a seior research egieer. From 2002 to 2003 he worked as a Postdoctoral researcher at Schepes ye Resarch Istitute, Harvard Medical School, Bosto. He is a assistat professor i the departmet of lectroic Commuicatio at the Shiheug College, Korea. His iterests are visio rehabilitatio, digital image processig, 3-D processig ad digital ASIC desig. li Peli received a BS, cum laude, ad MS from the Techio-Israel Istitute of Techology, ad his doctorate from New glad College of Optometry i Bosto. Dr. Peli is Seior Scietist ad the Moakley Scholar i Agig ye Research at The Schepes ye Research Istitute, ad Professor of Ophthalmology at Harvard Medical School. He also serves o the faculty of the New glad College of Optometry ad Tufts Uiversity School of Medicie. Sice 1983 he has bee carig for visually impaired patiets as the director of the Visio Rehabilitatio Service at the New glad Medical Ceter Hospitals i Bosto. Dr. Peli is a Fellow of the America Academy of Optometry, a Fellow of the Optical Society of America, ad a Fellow of the SID (Society for Iformatio Display). His pricipal research iterests are image processig i relatio to visual fuctio ad cliical psychophysics i low visio rehabilitatio, image uderstadig ad evaluatio of display-visio iteractio. He also maitais a iterest i oculomotor cotrol ad biocular visio. 10