Smartolor: Disambiguation Framework for the olorblind Ken Wakita Tokyo Institute of Technology Department of Mathematical and omputing Science 2-12-1 Ookayama, Meguro-ku Tokyo 152-8552, Japan +81 3 5734 3493 wakita@is.titech.ac.jp Kenta Shimamura Tokyo Institute of Technology Department of Mathematical and omputing Science 2-12-1 Ookayama, Meguro-ku Tokyo 152-8552, Japan +81 3 5734 3493 shimamu1@is.titech.ac.jp ABSTRAT Failure in visual communication between the author and the colorblind reader is caused when color effects that the author expects for the reader to experience are not observed by the reader. The proposed framework allows the author to annotate his/her intended color effects to the colored document. They are used to generate a repainted document that let the colorblind enjoy similar color effects that normal color vision person does for the original document. The annotations are formulated as a set of mathematical constraints that can describe several commonly used color effects. onstraints are defined over the normal vision color space. Then they are projected onto the restricted color space that corresponds to the one that the colorblind perceives. Finally, the projected constraints are resolved for the search of best repainting of the document that most successfully presents to the colorblind person the color effects experienced by the normal vision person on the original document. Effectiveness of the proposal is shown by colorblind simulation. ategories and Subject Descriptors H.1.2 [User/Machine Systems]: Human factors; I.4.8 [Scene Analysis]: olor; K.4.2 [Social Issues]: Assistive technologies for persons with disabilities General Terms Algorithms, Experimentation, Theory Due to the nature of this research, most of the figures are not meaningful when watching at a monochromatic output. In such case, please read the full-color PDF form available from AM digital library. For the reader s convenience, I placed images included in this paper online in their full-color forms at [20]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ASSETS 05, October 9 12, 2005, Baltimore, Maryland, USA. opyright 2005 AM 1-59593-159-7/05/0010...$5.00. Keywords olorblindness, Vision, onstraint system 1. INTRODUTION Most colorblindness is due to a genetic cause that leads to defect or lack of certain type of photo-receptors that serve for color vision. The frequency of colorblindness is fairly high among males: 8% among aucasian, 5% among Asian, and 4% among African. olorblindness is usually not a serious problem in the acts of daily life. In fact, John Dalton s essay in late 18th century is the earliest known description of colorblindness in the literature. In modern days, due to availability of color printers and full color display devices, use of colors in digital documents is being more common. Unlike early stage of the Web-era, it is difficult to find web sites that are monochromatic. Since colors play important role in color documents, colorblind people occasionally experience difficulty due to failure to perceive colors and color effects used in documents. This research attempts to resolve communication difficulty between people with normal vision and colorblind people. The research focuses on the author s intention about the colors he/she uses in the document. The research is based on two hypothesises: (1) in color assisted documentation information is conveyed by the color effects (relations among colors) but not by absolute natures of colors and (2) if color effects found in a document is observed in a similar form by the colorblind information expressed in terms of colors is transmitted to the colorblind without loss. On these hypothesises, we developed a repainting system that we call Smartolor. Smartolor replaces the set of colors used in the original document with another. Among various repainting schemes, Smartolor finds the one that most successfully transmits color effects observed by people with normal vision to the colorblind. This process is achieved by mathematically formulating the color effects found in the document in terms of constraint system and optimizing it in the color space of the colorblind people. Unlike traditional constraint-based interface designs that use geographical constraints, (1) Smartolor uses constraints defined over the chromatic space and (2) translation of constraints from the normal vision color space to the colorblind s color space is performed before constraint resolution phase. The rest of this paper is structured as follows: Section 2 158
briefly describe human perception of colors and explain causes of colorblindness. Section 3 illustrates the basic ideas which are detailed in sections 4 and 5. The latter section also shows interesting applications of Smartolor. Section 6 compares this work with others and discuss about research issues, and the last section concludes this paper. 2. OLOR AND OLORBLINDNESS ones found on the surface of human retina are responsible for daylight color vision, or photopic vision. ones absorb photons and send electrical signal to the brain. Three types of cones have different spectral sensitivities. L-cones absorb long wavelengths effectively, M-cones absorb medium wavelengths, and S-cones short wavelengths. onsequently light is perceived as three numbers: (l, m, s) where l, m, and s represent amount of photons absorbed by L-, M-, and S-cones, respectively. Hence, normal vision humans are trichromat (i.e., three-colored). More formally, color stimulus (S i) for a light can be given by a numerical integration over the wavelengths λ: Z S i = φ(λ)l i(λ)dλ, (i = L, M, S) where φ stands for spectral power distribution of the light, l L, l M, and l S for spectral sensitivity for L-, M-, and S- cones. Spectral sensitivities of cones have been measured over decades [18, 17]. olorblindness is caused from deficiency or lack of certain type of cone. Anomalous trichromats are those who have three types of cones but one of them is defective. An anomalous trichromat can be protanomalous, deuteranomalous, or tritanomalous, depending on deficiency in L-, M-, or S-cones, respectively. If one lacks certain type of cones, he/she is a dichromat (i.e., two-colored), who can be protanope, deuteranope, ortritanope, respectively, depending on lack of L-, M-, or S-cones. A dichromat perceives a color in two dimensional color space. For example, a color (l, m, s) perceived by a trichromat is perceived as (0,m,s), (l, 0,s), and (l, m, 0) by dichromats of different types, respectively. Two colors that differ only on l values (e.g., (l 1,m,s)and(l 2,m,s)) can confuse protanopes they perceive both colors as (0,m,s). Dichromats perception of colors have long been hard to understand for the trichromat until computer simulation of dichromats became available. The first computer simulation was conducted by Brettel et al [19, 4]. As conversion from RGB color space to LMS color space is a linear transformation, which can be represented by a 3 3 matrixt RGB LMS, protanope simulation (π L ) can roughly be formulated by π L (c) = T 1 RGB LMS (P L (T RGB LMS(c)), where P L stands for projection of a trichromat LMS-color onto protanope s limited color space. We can obtain simulations of deuteranopes (π M (c)) and tritanopes (π S (c)) in a similar manner. This algorithm is adopted by colorblindness simulation systems such as Visheck[23] and IBM adesigner s low vision mode[22], among others. There is misconception about how colorblind people perceive colors. It is important that colorblindness is not total loss of color vision. They recognize wide range of colors. It is also true, however, that unfortunate combination of colors can confuse the colorblind. Okabe and Ito (both protanopes) take many examples from scientific presentations where appropriate choice of colors eliminates problems due to color confusion[14]. 3. SMARTOLOR OVERVIEW A document is a presentation of ideas and concepts that an author has in his/her mind. The author prepares the document so that it effectively transfer author s ideas and concepts to the intended readers. A visual entity of a document is a meaningful part of the document, which can be a word, a phrase, a sentence, a graphical object (drawing, graph), an image, etc. The author often attaches to visual entities various visual effects, such as fonts, colors, and line-thicknesses, to present his/her ideas and concepts more effectively. Sometimes use of such visual effects plays crucial role in visual presentations. Here, we would like to investigate author s intention behind his/her use of colors in adocument. Sometimes we highlight a phrase using colors in the document trying to emphasize it, as I did for the previous sentence. In this case, I chose the color so that it visually attracts readers attention. For this purpose, choice of color is arbitrary as far as the colored text stands out to readers eyes; it could be red, for example. Let s take another example. When we paint bars in a bar chart, we often use equally prominent colors so that one bar does not attract reader s attention more strongly than others[6]. It is also important that one color is easily distinguishable from another. This careful color selection scheme, however, is prone to offer a set of colors that confuses the colorblind. The author uses various types of color effects in his/her document. The author expects and hopes that those color effects are experienced by the readers as the author does. If a dichromat reader views a colored document prepared by a trichromat author the author and the reader do not share the same color effect experience. This is the cause of failure in color assisted communication between the trichromat and the dichromat. What if we could repaint the original document so that the dichromat enjoys a similar color effects that the trichromats do? This is the main idea that is proposed in this paper. Smartolor is a repainting technology. It replaces a set of colors used in the document with another set. In Smart- olor framework, color effects that the author intends for the reader to experience are modeled by a set of desirability formulae. The desirability formulae is first modeled in trichromat s three-dimensional color space and then it is projected onto dichromat s two-dimensional color space. We can regard this projection as a process to translate author s expected color effects, which originally was targeted for a trichromat, for the dichromat. Theoretically, the color replacement for the document that most successfully satisfies the projected formulae is expected to carry the original color effects most successfully to the dichromat and thus is the most appropriate repainting. As we mentioned earlier, there are different types of colorblindness (protanope, deuteranope, and tritanope). Persons with different types of colorblindness observe the same document differently. It may be the case that, one repainting works successfully for a class of colorblind people but not for others. Smartolor generates repaint of the original document for each class of colorblindness. urrent prototype implementation supports dichromats (namely, protanope, deuteranope, and tritanope). 159
Smartolor technology can be used in a personal environment. The user specifies his/her visual profile (namely, the type of colorblindness a la [10]) and Smartolor presents a repainted document with respect to the specific visual profile. One promising application is a Web browser. We can setup a set of Web proxy servers, one for each type of colorblindness, which repaints web pages according to the respective colorblind people. 4. FORMULATION OF OLOR EFFETS In this section, we review how authors use colors in their documents. We demonstrate how such color usages can be modeled as a set of mathematical formulae. These formulae express author s desire to transmit the color effects found in the document to the (normal or trichromat) reader. In this sense, we can regard the formulae as a formulation of author s desire. 4.1 olor ontrasts olor contrast between two colors is perceived color difference between them. If a visual entity is surrounded by another and both are painted by similar colors, it is difficult to identify the inner one. Therefore the author usually keeps enough color contrast between neighboring visual entities. Let (c 1,c 2) give the distance between colors c 1 and c 2. 1 Suppose that two visual entities e 1 and e 2 are painted by c 1 and c 2, respectively, and that Smartolor replaces these colors with c 1 and c 2, respectively. When a protanope sees these colors they are perceived as π L (c 1)andπ L (c 2), respectively. The color difference as observed by the protanope is (π L (c 1),π L (c 2)). Here, we assume that color difference formula ( ) holds for both the trichromat and the dichromat. The following equation demands that the Smartolor repainting maintains the same amount of color contrast between e 1 and e 2 for the trichromat and the protanope: (c 1,c 2)= π L (c 1),π L (c 2) From this equation, we obtain a formula that we can use to evaluate how successfully a repainting method presents color contrast observed by the trichromat to the protanope: n o 2 D(e L 1,e 2)= (c 1,c 2) π L (c 1),π L (c 2) This function takes the maximum value of 0. Better repainting gives larger value for the result of this function. We can regard this function properly models how successfully the repainting strategy transmits the color contrast effect observed by a trichromat to a protanope. Hence, we call this function contrast desirability function. We collect contrast desirability functions in a pair-wise manner for all the visual elements in a document, Doc = {e 1,e 2,...}, and obtain the following: D L (Doc) = X i<j D L (e i,e j) (1) 1 In our implementation, we use IE 1976 (IEL a b ) color-space and its associated formula for calculation of color contrast. IEL a b is a color space that approximates human color perception. should be 0. On the other hand, if the author thinks this color contrast plays an important role in the document presentation, where is a non-negative value that represents how strongly the author desires to the transmit contrast effect between e i and e j to the reader. If the author is not interested in color contrast between e i and e j at all, he/she should assign a larger value to. ollective desirability functions for deuteranopes and tritanopes can also be formulated as follows: D M (Doc) = X i<j D S (Doc) = X i<j D M (e i,e j) (2) D S (e i,e j) (3) 4.2 olors as Shared Property It is common practice to use colors consistently. Usually, on web pages, headings of the same level are painted by the same color and hyperlinks have the same color throughout the web site. Style sheet technology helps and promotes consistent coloring over the web site and digital documents. Also readers who find visual entities painted by the same color tend to think that the the use of the color implies certain common feature shared among the concepts that those visual entities represent. For these reasons, it is important that repainting strategies respect color consistency found in the original document. In other words, objects painted with the same color should be repainted by the same color. More formally, for a repainting strategy P = {...,c i c i,...,c j c j,...}, we require that c i = c j c i = c j. This decision is beneficial from performance point of view. As we will see shortly, finding the best repainting is a computation intensive task. The computational complexity is exponential of the problem size. olors as shared property decision effectively reduces the problem size from the number of visual objects down to the number of colors used in a document. Most often a very complex drawing uses surprisingly smaller number of colors than the number of visual entities[10] and thus focusing on colors substantially reduces computation cost. 4.3 Distinguish-ability olors are sometimes used to clearly distinguish a visual entity from its neighbors. For example, in painting geographical map, it is important that regions on the map can easily be visually identified. Therefore neighboring geographic regions should be painted by different colors and each pair of colors should have enough contrast. It is, however, not so important which specific colors are used nor which one is used to paint a specific region. In this situation, it would be reasonable to enforce colors to have maximum contrast with each other. If visual entities e 1 and e 2, colored with c 1 and c 2, respectively, have maximum contrast ( max) the following equation holds: (c 1,c 2)= max This equation can be transformed into the following desirability formula with respect to distinguish-ability: ( (c 1,c 2) max) 2 We consider Smartolor repainting and interpret the de- 160
sirability function in protanope s color space as follows: n o 2 DD(e L 1,e 2)= π L (c 1),π L (c 2) max ollective desirability functions with respect to distinguishability for three types of colorblindness are: Better repainting strategy A B D L D(Doc) = X i<j D DL D(e i,e j) (4) D D M D (Doc) = X i<j D DM D (e i,e j) (5) D S D(Doc) = X i<j 4.4 Natural oloring D DS D(e i,e j) (6) There are cases where repainting is not desired. It is enough to look at a photographic negative image to understand that arbitrary color replacement for a photo-image would produce an image that is difficult to identify even for trichromats. Therefore it is not appropriate to apply Smart- olor technology to photo-images. To apply Smartolor technology to a document that comprises of photo-images, drawings, and texts, it is reasonable to apply Smartolor to drawings and texts but leave photo-images in their original forms. It is the case that a visual entity is painted with some color and the color is referred to with its name somewhere in the document (e.g., a blue arrow points to a red box in the figure ). In this case, the author may oppose to replace the color with others. It might also be the case that the author reluctantly allows a repainting that replaces a color with a similar color. All these types of desire can be described in a uniform fashion by the following desirability functions: DN L (Doc) = X 2 kn i π L (c i),π L (c i) (7) i DN M (Doc) = X 2 kn i π M (c i),π M (c i) (8) i DN S (Doc) = X 2 kn i π S (c i),π S (c i) (9) i where kn i stands for the strength of author s desire to keep the color unchanged. If the author does not care about the color of e i then kn i = 0. If the author strongly desires the color to be unchanged, kn i should have arbitrarily large number. 4.5 Putting Things Together So far, we have reviewed three types of color effects that are found in practice and presented a mathematical foundation that allows us to handle such color effects. We aggregate these desirability functions to obtain a single collective function that represents author s comprehensive intention with respect to color effects, as follows: D L (Doc) =D L (Doc)+D L D(Doc)+D L N (Doc) (10) As we have seen, a repainting that maximizes the value of this function is expected to let a protanope enjoy color effects experienced by a trichromat in the most successful way. Worse repainting strategy Figure 1: Illustration of the simulated annealing process incorporated by the Smartolor system. 5. FINDING OPTIMAL OLORING The desirability formulation (Equation 10) produces a complex expression. It comprises various parameters: c i stands for colors found in the original document and, D, ki N are determined by the author in accordance to his/her intention behind the color usage. The only variable parameters are c i which stand for replacement colors for c i. They collectively represent repainting. To make this fact more explicit, let us rewrite equation 10 as follows: D L (Doc, c )=D L (Doc) (11) where c =(c 1,c 2,...) To find the best repainting ([c 1 c 1,c 2 c 2,...]), it suffices to find c that maximizes D L (Doc, c ). Because this is a multi-variant non-linear function, it is difficult to solve this maximization problem using analytical techniques. There are at least two numerical approaches to solve maximization problem for a non-linear function: Raphson-Newton method and simulated annealing. Raphson-Newton method is an efficient algorithm if the second partial derivatives of the respective function have reasonably simple forms. Unfortunately, it is not the case with the desirability functions because of irregularity of color space conversion function (T RGB LAB) and dichromat simulation algorithm (π i). Therefore we adopt simulated annealing algorithm[12], which is generally not so efficient as Raphson-Newton method but does not require partial derivatives. Simulated annealing is a kind of Monte arlo simulation method but is targeted for finding lowest energy orientation of a system. In our system, undesirability ( D L (Doc, c )) serves as the energy. The simple Monte arlo simulation searches the parameter space at random but simulated annealing takes advantages of locality in the system and accelerates the searching process. Simulated annealing repeatedly chooses a parameter setting in a stochastic manner and checks if the parameter setting gives better result than the best known parameter setting so far. This process stops when stochastically good result is obtained. Simulated annealing finds a parameter setting that locally maximizes the given function but the parameter setting does not necessarily gives the global maximum. Smartolor attempts several trials of simulated annealing process to find a set of parameter settings that give lo- 161
cal maximums. The Smartolor compares function values for those parameter settings and offers a subset of the parameter settings that give fairly good results. For example, suppose that we would like to find nearly maximum values of a function f that is illustrated in figure 1. Function f has four local maximums A, B,, andd but simulated annealing may fail to find B and offers only A,, andd. AsD s value is too worse than A and, SmartolormaydropD and offer A and as recommended parameter settings. The reason why, Smartolor offers not only the best known result (in the above case A, for example) but also others (like ) is that we thought that in building the desirability function it is the case that the author fails to formulate/specify his/her intention properly. Because of this uncertainty, it is reasonable to regard desirability function as an approximation of author s actual desire. For this reason, it is fair to consider that the author may feel more comfortable with a secondary choice than the best. Smartolor system uses IEL a b as the primary color space because it is a uniform color space where Euclid distance between colors roughly approximates human cognitive color distance between them. Due to the stochastic nature of simulated annealing algorithm, it is the case that the algorithm chooses an imaginary color (non-existing color) that is outside the human gamut (the color space that is visible to normal people). 2 In that case, Smartolor projects the imaginary color on the surface of the human gamut to the color center of the color space. This projection is performed in the S-RGB color space. 5.1 Bar hart The first example is application of Smartolor to a colored bar chart (Figure 2). The bar chart presented on figure 2- (a) is colored so that three colors (red, green, and blue) are equally prominent to the trichromat s eyes. Also they have equal color contrast to each other. Healey[6] discusses that this color choice scheme is effective in scientific presentation. Unfortunately, the deuteranope sees this chart as presented on figure 2-(b). From this figure, we recognize that the bars originally colored by red and green are difficult to identify for the deuteranope. We have applied Smartolor technology on the chart targeting for the deuteranope. In this repainting, we have declared three types of color effects: (1) Natural oloring for the white and black areas, so that black lines and white background are not repainted, (2) distinguish-ability between bar colors and background color, and (3) color contrast between distinct bar colors. This arrangement can be stated formally by the following desirability functions: DN M (Doc) = k N `π M (white),π M (white ) 2 + k N `π M (black),π M (black ) 2 (12) X DD M (Doc) = k D DD M (e 1,e 2) (13) D M (Doc) = k e 1 {B,F } e 2 bars X e 1,e 2 bars D M (e 1,e 2) (14) where bars stands for the set of three types of colored bars, B for background white area, and F for foreground black lines. 2 An example of an imaginary color is one whose RGB representation involves negative value. (a) Original bar chart as observed by a normal person (c) Smartolor applied for the deuteranope. (e) Image (c) seen by the tritanope. (b) Image (a) seen by the deuteranope. olor confusion makes identification of bars difficult. (d) Image (c) seen by the deuteranope. Figure 2: A colored bar chart can cause problem when observed by a colorblind person (b). We apply Smartolor (c) that a bar chart (d) that is comfortable for the colorblind. olor images are available from [20]. 162
These equations involve three parameters k N, k, and k D. If the designer of the bar chart thinks that distinguishability of the bars relative to the background is more important than color contrasts among bars, he can assign larger amount to k N than k. In this manner, Smartolor offers a flexible framework that allows the designer to describe his/her intention behind the color usage by means of parameter adjustment. Smartolor repainting takes the original chart and produces figure 2-(c), which a deuteranope sees as figure 2-(d). This time, the deuteranope can easily recognize three types of bars colored differently. olor distinguish-ability between bars and black and white is enough. Also white and black colors are unchanged after Smartolor process. An output of Smartolor works fine for one type of colorblind people but not necessarily for others. For example, the tritanope views figure 2-(c) as figure 2-(e). For the tritanope, the color of the topmost bar is too pale and is difficult to identify on the white background. In this sense, Smartolor technology differs from universal design system. 5.2 Ishihara Test hart The next example is application of Smartolor to Ishihara test chart. Ishihara test chart is a book that consists of cards that identify protanopes and deuteranopes. On each page of the book, a number or a shape is drawn using color dots in a mosaic manner. In making the mosaic, confusing colors for dichromats are chosen. Figure 3-(a) is taken from one of the Ishihara test card. To protanope s eyes this figure appears like Figure 3-(b). This simulation was done using Visheck[23]. We have digitized one of Ishihara test charts using a photo scanner. Then we have applied color clustering and color reduction process to obtain a digitized test chart (see Figure 3- (a)). The color reduction process reduces the number of colors down to 10 most prominent colors. This process converts the original raster image into a vector image that comprises of colored circles which allows us to manipulate the image in a per-circle manner. The number 73 is vaguely observable to trichromat s eyes but is obscured to protanope s eyes (3-(b)). The desirability function that we used requires that contrast desirability should be maintained for each pair of 10 colors and that white background is unchanged. olor distinguishability is ignored because the charts were designed to cause color confusion and it seems obvious that color distinguishability is the last thing Dr. Ishihara had in his mind: D(Doc) L = X n o 2 (c 1,c 2) π L (c 1),π L (c 2) D L D(Doc) =0 c 1,c 2 D L N (Doc) = k N 2 π L (white),π L (white ) (a) One of Ishihara test chart. Normal readers can read 73 in this test chart. (c) Smartolor repainting targeted for the protanope is applied. (e) Simulation of tritanope s view watching at (c). (b) Simulation of protanope s view watching at (a). Due to color confusion, the number is obscured (d) Simulation of protanope s view watching at (c). where stands for a set of colors that are found in figure 3- (a), including the white background and k N is a reasonably big positive number (from theoretical point of view it can be ). In the previous section, collective desirability functions were built by aggregating pair-wise desirability functions over the visual elements but here aggregation is done over colors, in stead. This is possible without modification of the theoretical foundation because we can adjust ap- Figure 3: A normal reader can identify the number 73 on figure (a) but to protanope s view the figure seems random dots. We apply Smartolor technology and replace colors (c). This time the number appear clearly to the protanope. olor images are available from [20]. 163
propriately for this purpose. This color-wise aggregation is important for this experiment because it can handle each color equally in spite of their different occurrence frequency. omputation of the best repainting strategy leads us to figure 3-(c). This figure appears to protanope s eyes as figure 3-(d), where the number is clearly observable. It should be noted that the number on repainted chart is easier to identify for the protanope s eyes (d) than normal people s (c). In general, the output from Smartolor is targeted for a specific class of colorblind people. An output targeted for one class of colorblind people may not be appropriate for another class of colorblind people nor the trichromat. For example, to a tritanope s eyes figure 3-(c) look like figure 3-(e). 6. DISUSSION This section puts our idea into the context of current related work. 6.1 olor Adaptive Graphics It is known that human perception of light is affected by various environmental factors. Among them is simultaneous color contrast that causes one to perceive the same color differently when presented on different background. Ishizaki pointed out naive presentation of colors misguides human in a multimedia environment[8]. olor adaptive graphics is a system that automatically adjusts color differences caused by simultaneous color contrast[9]. The system adjusts objects color so that they have similar color contrast relative to their reference background colors. One of their example shows a new article overlay-ed on a colored satellite map, where letters placed over the ocean (navy blue) are marked by darker color and letters placed over the desert area (light gray) so that those letters maintain equal color contrast relative to their reference background. This research result claims that colors are not interpreted as they are even by the trichromat. Though techniques developed for color adaptive graphics and Smartolor differ considerably, both focuses on color effects perceived by the human. We believe that this is the key issue that we need to offer better UI technologies for colorblind people. 6.2 onstraint-based User Interface Systems Our work is closely related to constraint-based user interface (BUI) systems. Use of a constraint system in a user interface design dates back to 1970s[2, 15]. onstraints are used to describe logical rules for placement of mutually related objects. One example is graph layout algorithm. It is preferable to layout a graph in a smallest possible area minimizing the number of edge crossings. The idea of constraintbased user interface is to express the inter-object relationship in a set of constraints, transform the constraints into two dimensional space (or display), and resolve the constraints in that space to obtain the most successful twodimensional layout of the graphical objects[2, 15]. There are a few work that has applied this technology to Web documents[3]. Smartolor differs from traditional BUI systems in two places. Firstly, constraints used in BUI systems constrain locations and shapes of the graphical objects and therefore they are defined over geographical space. On the other hand, constraints used in Smartolor system constrain choice of colors and are defined over the chromatic space. Secondly, constraints adopted by BUI systems are resolved in the same geographical space as they are defined but in Smart- olor constraints are defined over a three-dimensional trichromatic space and are projected onto two-dimensional dichromat s color space before they are resolved. One benefit of BUI systems is that both the user interface designer and the reader can enforce constraints. Usually a user interface is offered by the user interface designer to the reader and the reader has limited control over customization of the user interface. On the other hand, a BUI system allows both the designer and the reader to add layout constraints. Therefore the output from a BUI system can be regarded as a collaborative work between the designer and the reader. Likewise, a constraint-based accessibility assistance system such as Smartolor can be regarded as a collaborative painting framework which produces a color arrangement by a collaborative effort between the designer and dichromat reader. 6.3 Semantic Approaches Smartolor assists the colorblind by grasping the author s intention behind his/her color usage and attempts to find a color repainting that most effectively pass the intention to the colorblind person. In this sense it is a semantic approach. There are attempts to apply semantic Web technology to improve Web accessibility[7, 16]. Semantic Web technology offers a systematic mechanism to self-describe the document where document s meta-information, or the semantics, is encapsulated in itself. To aid handicapped persons, semantics- Web-based assistive technologies attempt to create an alternative presentations for otherwise unviewable contents of the document using the meta-information. Technically, this translation is usually done by XML transcoding. Some assistive technologies restructure the document by transcoding[7, 16] and others perform inter-mode translation. For example, AUDIOGRAPH attempts to convert a graphical interface into a auditory interface[1]. Smartolor belongs to the latter approach. For a normal human visual input comprises of three channels of chromatic visual information. For a dichromat one of them is missing. Smart- olor attempts to embed missing chromatic information in the other two chromatic information using the semantic information (i.e., author s intention with respect to his/her color usage). 6.4 omputational ost The computational complexity of this algorithm is rather expensive. The optimization algorithm of Smartolor searches for optimal repainting parameters whose search space expands in an exponential order to the number of colors used in a document. The number of colors used in colored documents is small (usually no more than 10) but 10 colors are still too many for the Smartolor to be used in an interactive environment. Further investigation is required to speedup the system. One possibility is simplification of the constraint system. urrently, we have adopted bi-directional kinetic constraint system, which is expressive but requires large amount of computation power. We believe that incorporation of hierarchical constraint system and one-way constraint system will greatly reduces the computation cost[15]. 164
Although current implementation of Smartolor framework is not suitable to interactive user interface, a promising application domain is production of style sheets such as S for HTML and XSL for XML. The Smartolor algorithm takes a web page and produce a coloring method as a style sheet forms one for each class of colorblindness. Generation of a style sheet needs to be done once when a new style sheet is installed and for all. Because the web page can be presented with an already generated style sheet, the computational overhead of Smartolor is not observable by the web-surfers. 7. SUMMARY In this paper, we have presented a new approach that alleviate the barrier that colorblind people faces due to color confusion problem. The key ideas are repainting and our focus on author s intention. The document s colors are replaced so that the color effects that the author expects for the reader to experience are effectively recognized by the colorblind reader. Preliminary evaluation of the system has been done by using simulation. Further assessment of this proposal using user test and practical applications remains to be done. Finally, we would like to express our gratitude for suggestive comments and recommendations from anonymous reviewers. 8. REFERENES [1] J. L. Alty and D. I. Rigas. ommunicating graphical information to blind users using music: the role of context. In HI 98: Proc. SIGHI conf. Human factors in computingsystems, pages 574 581, New York, NY, USA, 1998. [2] A. Borning. ThingLab: an object-oriented system for building simulations using constraints. In Proc. 5th intl. joint conf. Artificial intelligence, pages 73 85, ambridge, MA, August 1977. William Kaufmann. [3] A.Borning,R.Lin,andK.Marriott.onstraintsfor the web. In MULTIMEDIA 97: Proc. 5th AM intl. conf. Multimedia, pages 173 182, New York, NY, USA, November 1997. [4] H. Brettel, F. Viénot, and J. D. Mollon. omputerized simulation of color appearance for dichromats. Optical Society of America (Series A), 14(10):2647 2655, October 1997. [5] K. R. Gegenfurtner and L. T. Sharpe, editors. olor Vision: From Genes to Perception. ambridge University Press, 1999. [6]. G. Healey. hoosing effective colours for data visualization. In Proc. IEEE Visualization 96, pages 263 370, San Francisco, F, 1996. [7] A. W. Huang and N. Sundaresan. A semantic transcoding system to adapt web services for users with disabilities. In Assets 00: Proc. 4th intl. AM conf. Assistive technologies, pages 156 163, New York, NY, USA, 2000. [8] S. Ishizaki. Adjusting simultaneous contrast effect for dynamic information display. In Proc. 2nd IS&T/SID olor imaging conf., pages 137 140, 1994. [9] S. Ishizaki. olor adaptive graphics: what you see in your color palette isn t what you get! In Proc. onf. Human Factors in omputingsystems, pages 300 301, 1995. [10] J. A. Jacko, M. A. Dixon, Jr. R. H. Rosa, I. U. Scott, and. J. Pappas. Visual profiles: a critical component of universal access. In HI 99: Proc. SIGHI conf. Human factors in computingsystems, pages 330 337, New York, NY, USA, 1999. [11] N. Jacobson and W. Bender. olor as a determined communication. IBM System Journal, 35(3 & 4):526 538, 1996. [12] S. Kirkpatrick,. D. Jr. Gerlatt, and M. P. Vecchi. Optimization by simulated annealing. Science, 220:671 680, 1983. [13] D. Malacara. olor Vision and olorimetry: Theory and Applications. SPIE Press, 2002. [14] M. Okabe and K. Ito. How to make figures and presentations that are friendly to color blind people. http://jfly.iam.u-tokyo.ac.jp/html/color blind/, November 2002. [15] M. Sannella, J. Maloney, B. Freeman-Benson, and A. Borning. Multiway versus one-way constraints in user interfaces: experience with the DeltaBlue algorithm. Software - Practice and Experience, 23(5):529 566, 1993. [16] L. Seeman. The semantic web, web accessibility, and device independence. In W4A 04: Proc. 2004 intl. cross-disciplinary workshop on Web accessibility, pages 67 73, New York, NY, USA, 2004. [17] A. Stockman and L. T. Sharpe. Spectral sensitivities of the middle- and long-wavelength sensitive cones derived from measurements in observers of known genotype. Vision Research, 40:1711 1737, 2000. [18] A. Stockman, L. T. Sharpe, and.. Fach. The spectral sensitivity of the human short-wavelength cones. Vision Research, 39:2901 2927, 2000. [19] F. Viénot, H. Brettel, L. Ott, A. B. M Barek, and J. D. Mollon. What do colour-blind people see? Nature, 376(6536):127 128, 1995. [20] K. Wakita and K. Shimamura. Full-color images from ASSETS 05 Smartolor paper. http://www.is.titech.ac.jp/ wakita/public/assets2005/, August 2005. [21] K. Wakita and Y. Ueno. SmartGray: information preserving monochromatic rendering of colored documents. In IITA 04: Proc. 2nd intl. conf. Information technology for applications, pages 470 475, Harbin, hina, January 2004. [22] Accessibility research: adesigner. http://www.research.ibm.com/trl/projects/acc tech/ adesigner.htm. [23] Vischeck. http://www.vischeck.com/. 165