(12) United States Patent (10) Patent No.: US B2

Transcription

1 USOO B2 (12) United States Patent (10) Patent No.: US B2 Jiang et al. (45) Date of Patent: Jul. 30, 2013 (54) CHROMA SUPRESSION FEATURES 6,961,085 B2 * 1 1/2005 Sasaki ,972,793 B1* 12/2005 Kameyama ,317,827 B2* 1/2008 Munsil ,162 (75) Inventors: Xiaoyun Jiang, San Diego, CA (US); A1* 2, 2003 Fukui et al ,679 Shuxue Quan, San Diego, CA (US); 2007/ A1* 4/2007 Jaspers ,687 Chinchuan Andrew Chiu, San Diego, FOREIGN PATENT DOCUMENTS CA (US); Szepo Robert Hung, Carlsbad, CA (US); Ruben M. Velarde, DE , 1988 Chula Vista, vers CA (US);; Ying Xie Noyes yes, EP O1458O1 O , 6, San Diego, CA (US) EP O647O69 4f1995 EP O , 1999 (73) Assignee: QUALCOMM Incorporated, San JP A 3, 1993 Diego, CA (US) JP A 6, 2001 s WO f1991 (*) Notice: Subject to any disclaimer, the term of this OTHER PUBLICATIONS patent is extended or adjusted under 35 International Search Report PCT/US2007/ International Oa. CaC eport, Internationa U.S.C. 154(b) by 1657 days. Search Authority European Patent Office-Mar. 27, Written Opinion PCT/US2007/ International Search (21) Appl. No.: 11/533,916 Authority European Patent Office-Mar. 27, (22) Filed: Sep. 21, 2006 * cited by examiner (65) Prior Publication Data Primary Examiner David Czekaj US 2008/ A1 Mar. 27, 2008 Assistant Examiner Leron Beck (74) Attorney, Agent, or Firm Timothy Loomis; James R. (51) Int. Cl. Gambale, Jr. H04N II/02 ( ) (52) U.S. Cl. (57) ABSTRACT USPC / / Apparatus are provided including an image signal carrier, a (58) Field of Classification Search luminance information evaluator, and a chrominance infor USPC , See application file for complete search history. mation modifier. The image signal carrier is encoded with an image signal including luminance information and chromi (56) References Cited nance information. The luminance information evaluator evaluates the luminance information in the image signal for a U.S. PATENT DOCUMENTS 4,136,957 A * 1/1979 Uno et al ,394 4,270,223. A * 5/1981 Marston ,305 4,975,768 A * 12/1990 Takaraga ,538 5,446,504 A * 8/1995 Wada ,645 5,563,666 A * 10/1996 Suzuki ,645 6, A * 8/2000 Noguchi , 167 6,618,502 B1* 9/2003 Okada et al , 167 given region within the image to identify when the given region is one of Substantially white and Substantially dark. The chrominance information modifier is provided to modify the chrominance information corresponding to the given region when the given region is one of substantially white and substantially dark. 33 Claims, 5 Drawing Sheets TRANSMIT RECEIVE DIGITAL DEVICE WIDEOfIMAGE PROCESSOR 16 OTHER FUNCTION(S)

2

3 U.S. Patent Jul. 30, 2013 Sheet 2 of 5 US B2 RIGHGIOONIGH -VWNORIHO NOISSERHdd [] S ONISSEHOORHdH Z * OIH -VWNONIHO JLN@HWN@HONVH NGH [ONISSEHOORHdH

4

5 U.S. Patent Jul. 30, 2013 Sheet 4 of 5 US B2 A. s

6 U.S. Patent Jul. 30, 2013 Sheet 5 of 5 US B2 3 S

7 1. CHROMA SUPRESSION FEATURES COPYRIGHT NOTICE This patent document contains information Subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent, as it appears in the US Patent and Trademark Office files or records, but otherwise reserves all copyright rights whatsoever. BACKGROUND Aspects of the disclosure may relate to devices for captur ing, storing, playing back, and/or displaying images. Image and video capture, Storage, playback, and display capabilities are being incorporated into a wide range of devices, including, e.g., wireless communications devices (e.g., mobile phones), portable digital assistants (PDAs), lap top computers, and desktop computers. In these devices, images, when viewed, played back, or printed, may include unnatural artifacts such as inconsistencies in dark areas of the image and tinting in areas that are Suppose to be white. SUMMARY OF THE DISCLOSURE Apparatus are provided, including an image signal carrier, a luminance information evaluator, and a chrominance infor mation modifier. The image signal carrier is encoded with an image signal including luminance information. The lumi nance information evaluator evaluates the luminance infor mation in the image signal in a given region within the image to identify when the given region is one of substantially white and Substantially dark. The chrominance information modi fier modifies the chrominance information corresponding to the given region when the given region is one of substantially white and substantially dark. BRIEF DESCRIPTION OF THE DRAWINGS The subject matter disclosed herein is further described in the detailed description, which follows, by reference to the noted drawings by way of non-limiting example embodi ments, in which like reference numerals represent similar parts throughout the several views of the drawings, and wherein: FIG. 1 is a block diagram of a digital device in accordance with one embodiment of the present disclosure; FIG. 2 is a schematic diagram of a portion of an image processing pipeline; FIG. 3 is a more detailed schematic diagram of the chroma Suppression processing mechanism shown in FIG. 2; FIG. 4 is a graph illustrating the Suppression ratio in the illustrated embodiment; and FIG. 5 is a graph illustrating the Suppression weight vari able in the illustrated embodiment. DETAILED DESCRIPTION Referring now to the drawings in greater detail, FIG. 1 is a block diagram of a digital device in accordance with one embodiment of the present disclosure. The illustrated digital device 10 may, for example, be a digital television, a digital direct broadcast system, a camera, a camcorder, a wireless communications device (e.g., mobile phone), a portable digi talassistant (PDA), a laptop computer, a desktop computer, or another digital device that involves or includes functionality US 8,498,332 B for image and/or video capture, storage, playback, and/or display. The illustrated digital device 10, therefore, may or may not include a transmit/receive mechanism 12, an image sensor 18, a display/view finder 20, and another interface or interfaces 24. The illustrated digital device 10 further includes a video or image processing component 14 and a memory 22. In addition, other function elements 16 may be provided. In the illustrated embodiment of digital device 10, Video and/or image processor 14 includes a darkness and white processing portion 30. The illustrated processing por tion 30 may include a portion of a video or image processing pipeline within video/image processor 14. The illustrated processing component 30 includes an image signal carrier 32 encoded with or otherwise carrying image data, a luminance information evaluator 34, and a chrominance modifier 36. The image signal carrier 32 may, for example, be a serial or parallel interface for carrying an image signal, a frame buffer, another memory mechanism, or any type of structure for carrying or holding, or otherwise encoding machine-readable media to represent, the image signal to be processed. The illustrated processing component 30 may be operational as part of an image capture function, for example, when an image is captured by image sensor 18. It may be also operational as part of a processing component of a digital device which takes existing images or video pre viously recorded and processes such images or video in order to improve the quality of the same. In addition, or alterna tively, processing component 30 may be operational when displaying or otherwise playing back images or video stored, for example, in memory 22 of digital device 10. The process ing component 30 processes the image or video frame to improve white and dark portions of the image. Specifically, unnatural artifacts, such as inconsistencies in dark areas of the image and tinting in areas that are Suppose to white, are removed or otherwise mitigated. The image signal, regardless of whether the image is a still image or an image within a video signal, may be in one of several different formats, including Lab, RGB, YUV.YCbCr, YIQ, CMY. CMYK, or another type of data for representing image information. The image signal includes luminance information as well as chrominance information. The luminance information, in the case of Lab data, may be the L parameter. In the case of RGB data, it may be the weighted sum of the linear RGB components, and in the case of R'G'B' data, it may be the weighted sum of the non linear R'G'B' components. In the event the data is CMY data or CMYK data, the luminance information may be K and/or the weighted sum of the CMY components. When YUV data is used, the luminance infor mation may be the Y portion of that data. When YCbCr is used, the luminance information may be the Y component. When YIQ data is used, the luminance information may be the Y component. The chrominance information generally includes or is based on a and b of the Lab, RG, and B when RGB data is used, and CM and Y when CMY data is used. The chrominance information may also include or be based on U and V when YUV data is used, Cb and Cr if YCbCr data, and IQ of YIQ data. The terms luminance and chrominance, as used for one or more embodiments herein, may refer to darkness, i.e., as related to the luminous-efficiency function of human eyes (for luminance) and color (e.g., including both hue and satu ration components) (for chrominance). The illustrated luminance evaluator 34 is provided to evaluate the luminance information in the image signal for a given region within the image (for example, for a given pixel within an image) to identify when the given region is one of

8 3 substantially white and substantially dark. This may be deter mined, depending on the scheme used to represent the lumi nance (e.g., if it is the Y value inaycbcr signal) by evaluating when the luminance value is less thana lower threshold value, meaning that the given region is Substantially dark, or when the luminance value is greater than a higher threshold value, meaning that the given region is Substantially white. The image may be a two dimensional (2-D) image, in the form of a three-dimensional (3-D) model, or in the form of a two-dimensional frame which serves as a two dimensional representation of a three dimensional image, for example, as is the case in animated graphics based upon a 3-D model. The image data may be in the form of image data pixels represent ing respective different areas or regions within the image. Accordingly, a given image may include a pattern (e.g., an array) of pixels. Each pixel may correspond to a defined location within the image (for example, a grid cell) and com prise tone information. Tone information for a given point (e.g., corresponding to a pixel) or region of an image gener ally comprises a color component and an intensity compo nent. The illustrated chrominance modifier 36 is provided to modify the chrominance information corresponding to the given region, when the given region is one of Substantially white and substantially dark. The modifier may multiply a Suppression ratio times individual values making up the chrominance information, for example, using one or more lookup tables. FIG. 2 is a schematic diagram of a portion of an image processing pipeline. The circuitry illustrated in FIG. 2 includes a chroma enhancement processing mechanism 40, a chroma Suppression processing mechanism 42, and an encoder 44. The chroma enhancement processing mechanism 40, may, for example, perform a transformation from a RGB signal to a YCbCr signal, to facilitate color reproduction and color adjustment, and accordingly, output Y data, in this embodiment representing luminance, and Cb and Cr data, in this embodiment representing chrominance. Chroma Suppression processing mechanism 42 receives Y and Cb, Cr data as inputs thereto, and outputs modified chrominance information. Encoder 44 encodes the video data, including the luminance information (Y) and the chrominance information Cb, Cr (as modified by chroma Suppression processing mechanism 42). By way of example, encoder 44 includes a JPEG compression mechanism. FIG. 3 shows a more detailed schematic diagram of one example implementation of the chroma Suppression process ing mechanism 42 shown in FIG. 2. The illustrated circuitry 42 shown in FIG. 3 includes a Suppression ratio mechanism 46, a max chrominance magnitude mechanism 48, a Suppres sion weight mechanism 50, and an overall Suppression ratio calculator 52. Overall suppression ratio calculator 52 deter mines an overall suppression ratio value by which each of Cb and Cr will be multiplied in order to produce modified chrominance values Cb' and Cr'. A multiplier 54 is provided in the circuit to effect the multiplication by this overall sup pression ratio value. An initial Suppression ratio R is calculated by Suppression ratio mechanism 46. Suppression ratio mechanism 4.6 may, for example, include a lookup table (LUT) populated based upon the graph representing the Suppression ratio in relation to luminance Y as shown in FIG. 4. The Suppression weight mechanism 50 may include a lookup table, populated in accordance with the graph repre senting weight values in relation to a max chrominance value max (Cbl, Cr) as shown in FIG. 5. US 8,498,332 B As shown in FIG. 4, the initial Suppression ratio R, as determined by Suppression ratio mechanism 46, will gener ally be equal to one, when the region of the image being processed is neither substantially white nor substantially dark, i.e., when the luminance value Y is between m2 and m3. As the luminance goes below the lower threshold m2 and approaches the value m1, the Suppression ratio gradually goes from one down to Zero, e.g., in a linear fashion as shown in the graph of FIG. 4. As the luminance value goes beyond the higher threshold value m3 and approaches a value mak, the Suppression ratio again goes from one downto Zero. When the luminance value is below m1 or above ma, the Suppression ratio is Zero. In determining the Suppression weight based upon the chrominance values, a calculation is made by mechanism 48 of the maximum of the absolute values of Cb and Cr. Sup pression weight mechanism 50 then compares this maximum value to the graph as shown in FIG. 5, which may be done with the use of a lookup table. When the max value is between Zero and mm1, the Suppression weight is equal to one. As the max value goes from mm.1 to mm2, the Suppression weight is lowered from one to zero in a linear fashion. Beyond mm2, the Suppression weight is Zero. In some images there are bright colors (for example, a yellow flower), that should not be suppressed. Those colors generally have high chroma values. This weighting function helps preserve Such information for certain colors, such as yellow. The formula representing the overall Suppression ratio, which is determined by overall Suppression ratio mechanism 52, is as follows: RotzRAL-Ry'WC+1-WC, Where Roz is the overall Suppression ratio, R is the initial suppression ratio and W, is the Suppression weight. Other variations are also contemplated. For example, the chrominance information may be modified or suppressed, as appropriate, using a different approach than that of the illus trated example embodiment. In the event one or both of the waveforms shown in FIGS. 4 and 5 isfare used, such wave form(s) may have a different shape. For example, one or more of the transitions from 1 to Zero may be curved rather than linear. While the specific embodiment illustrated above, for example, as shown in FIGS. 2-5, is applied to an image signal in the format of YCbCr, modifications are possible to accom modate different representations of luminance and chromi nance information when other schemes are used, for example, RGB, CMY, CMYK, and so on, as noted above. Each element described hereinabove may be implemented with a hardware processor together with computer memory executing Software, or with specialized hardware for carrying out the same functionality. One or more components, for example, the image processing pipeline, or the portion of the image processing pipeline as shown in FIG. 1, may be in the form of an integrated circuit. Any data handled in Such pro cessing or created as a result of such processing can be stored, for example, in any memory available to the artisan. By way of example, such data may be stored in a temporary memory, Such as a random access memory. In addition, or in the alter native, Such data may be stored in longer-term storage devices, for example, magnetic disks, rewritable optical disks, and so on. For purposes of the disclosure herein, machine-readable media may comprise any form of media including computer-readable media, data storage mecha nisms and specific hardware circuit representations of struc tures and of data.

9 5 The claims, as originally presented and as they may be amended, encompass variations, alternatives, modifications, improvements, equivalents, and Substantial equivalents of the embodiments and teachings disclosed herein, including those that are presently unforeseen or unappreciated, and that, for example, may arise from applicants/patentees, and others. What is claimed is: 1. An apparatus for modifying the color of an image region based on a luminance of the region, comprising: an image signal carrier, the image signal carrier being encoded with an image signal including luminance information and chrominance information of an image, the chrominance information including chrominance values; aluminance information evaluator to evaluate aluminance for a region within the image to identify when the lumi nance is less than a lower threshold value and identifies when the luminance is greater than a higher threshold value; and a chrominance information modifier to modify the chromi nance information corresponding to the given region by multiplying the chrominance information by an overall Suppression ratio, the overall Suppression ratio based on a Suppression ratio and a Suppression weight, the Sup pression ratio selected based on luminance information in the image signal, and the Suppression weight selected based only on the maximum of the chrominance values in the region, wherein a Suppression of chrominance information resulting from the modification is lower when the luminance is between the lower threshold and the upper threshold relative to the suppression of chrominance information when the luminance is lower than the lower threshold value or higher than the higher threshold value. 2. The apparatus according to claim 1, wherein the image signal carrier includes one of a parallel and a serial interface. 3. The apparatus according to claim 1, wherein the image signal carrier includes a video pipeline. 4. The apparatus according to claim 1, wherein the image signal carrier includes a memory. 5. The apparatus according to claim 1, wherein the image signal is one of a Lab, RGB, CMY, CMYK, YUV.YCbCr, and YIQ type of image signal. 6. The apparatus according to claim 1, wherein the chromi nance information modifier includes a Suppression ratio mul tiplier to multiply a Suppression ratio by individual values making up the chrominance information. 7. The apparatus according to claim 6, wherein the Sup pression ratio is equal to 1 when the given region is neither substantially white nor substantially dark. 8. The apparatus according to claim 1, further comprising at least one lookup table to evaluate the luminance informa tion and to modify the chrominance information. 9. The apparatus according to claim 8, wherein the lookup table includes mechanism to output a Suppression ratio value based on an input luminance value. 10. The apparatus according to claim 7, further comprising a weighting mechanism to weight the Suppression ratio based on an amount of chrominance. 11. The apparatus according to claim 6, wherein the Sup pression ratio is based upon an evaluation of a level of a luminance value. 12. The apparatus according to claim 11, wherein the Sup pression ratio is further weighted based upon absolute values of the chrominance values. US 8,498,332 B The apparatus according to claim 1, wherein the lumi nance information is Y of a YCbCr signal, and wherein the chrominance information is Cb and Cr of the YCbCr signal. 14. A method of modifying the color of an image region based on a luminance of the region, comprising: carrying an image signal including luminance information and chrominance information of an image, the chromi nance information including chrominance values; evaluating the luminance information in the image signal for a given region within the image to identify when a luminance value of the image region is less than a lower threshold value and to identify when the luminance value is greater than a higher threshold value; and modifying the chrominance information corresponding to the given region based on a Suppression ratio determined from the luminance information and a Suppression weight determined only from the maximum of the chrominance values in the region, whereina Suppression of chrominance information resulting from the modifi cation is lower when the luminance value is between the lower threshold and the upper threshold relative to the Suppression of chrominance information when the lumi nance value is lower than the lower threshold value or higher than the higher threshold value, said modifying performed under control of at least one electronic device. 15. The method according to claim 14, wherein the image signal is carried by one of a parallel and a serial interface. 16. The method according to claim 14 wherein the image signal is carried using a video pipeline. 17. The method according to claim 14, wherein the image signal is carried in a memory. 18. The method according to claim 14, wherein the image signal is one of a Lab, RGB, CMY, CMYK, YUV.YCbCr, and YIQ type of image signal. 19. The method according to claim 14, wherein the modi fication includes multiplying a Suppression ratio by indi vidual values making up the chrominance information. 20. The method according to claim 19, wherein the Sup pression ratio is equal to 1 when the given region is neither substantially white nor substantially dark. 21. The method according to claim 14, further comprising providing at least one lookup table to evaluate the luminance information and to modify the chrominance information. 22. The method according to claim 21, wherein the lookup table includes a mechanism to output a Suppression ratio value based on an input luminance value. 23. The method according to claim 20, further comprising weighting the Suppression ratio based on an amount of chrominance. 24. The method according to claim 19, wherein the Sup pression ratio is based upon an evaluation of a level of a luminance value. 25. The method according to claim 24, wherein the Sup pression ratio is further weighted based upon absolute values of the chrominance values. 26. The method according to claim 14, wherein the lumi nance information is Y of a YCbCr signal, and wherein the chrominance information is Cb and Cr of a YCbCr image signal. 27. An apparatus for modifying the color of an image region based on a luminance of the region, comprising: means for encoding a carrier with an image signal includ ing luminance information and chrominance informa tion of an image, the chrominance information including chrominance values;

10 7 means for evaluating the luminance information in the image signal for a region within the image to identify when a luminance value of the image region is less than a lower threshold value, and to identify when the lumi nance value is greater than a higher threshold value; and means for modifying the chrominance information corre sponding to the given region by multiplying the chromi nance information by an overall suppression ratio, the Overall suppression ratio based on a suppression ratio and a Suppression weight, the suppression ratio selected based on luminance information in the image signal, and the Suppression weight selected based only on the maxi mum of the chrominance values in the region, wherein a Suppression of chrominance information resulting from the modification is lower when the luminance value is between the lower threshold and the upper threshold relative to the suppression of chrominance information when the luminance value is lower than the lower thresh old value or higher than the higher threshold value. 28. The apparatus according to 27, wherein the means for modifying include means for multiplying a suppression ratio by individual values making up the chrominance information. 29. An integrated circuit for modifying the color of an image region based on a luminance of the region comprising: a semiconductor substrate carrying: an image signal carrier, the image signal carrier being encoded with an image signal including luminance information and chrominance information of an image, the chrominance information including chrominance values: a luminance information evaluator to evaluate the lumi nance information in the image signal for a region within the image to identify when a luminance value of the image region is less thana lower threshold value and to identify when the luminance value is greater than a higher threshold value; and a chrominance information modifier to modify the chrominance information corresponding to the given region by multiplying the chrominance information by an overall suppression ratio, the overall suppres Sion ratio based on a suppression ratio and a suppres Sion weight, the suppression ratio selected based on luminance information in the image signal, and the Suppression weight selected based only on the maxi mum of the chrominance values in the region, wherein a suppression of chrominance information US 8, B resulting from the modification is lower when the luminance value is between the lower threshold and the upper threshold relative to the suppression of chrominance information when the luminance value is lower than the lower threshold value or higher than the higher threshold value. 30. The integrated circuit according to claim 29, wherein the chrominance information modifier includes a suppression ratio multiplier to multiply a suppression ratio by individual Values making up the chrominance information. 31. A non-transitory machine-readable media encoded with data, the data being encoded to cause execution of method of modifying the color of an image region based on a luminance of the region, the method comprising: carrying an image signal including luminance information and chrominance information of an image, the chromi nance information including chrominance values; evaluating the luminance information in the image signal for a given region within the image to identify when a luminance value of the image region is less than a lower threshold value and to identify when the luminance value is greater than a higher threshold value; and modifying the chrominance information corresponding to the given region by multiplying the chrominance infor mation by an overall suppression ratio, the overall sup pression ratio based on a suppression ratio and a sup pression weight, the suppression ratio selected based on luminance information in the image signal, and the sup pression weight selected based only on the maximum of the chrominance values in the region, wherein a suppres sion of chrominance information resulting from the modification is lower when the luminance value is between the lower threshold and the upper threshold relative to the suppression of chrominance information when the luminance value is lower than the lower thresh old value or higher than the higher threshold value. 32. The non-transitory machine-readable media according to claim 31, wherein the data is encoded to further cause the modification including a suppression ratio multiplication multiplying a suppression ratio by individual values making up the chrominance information. 33. The method of claim 1, wherein the overall suppression ratio is equal to the suppression ratio suppression weight-- 1-suppression weight.