Interactive Graphical Systems Fall 2002

Historical Overview Interactive Graphical Systems Fall 2002 1960 First random scanned display available 1 1964 A few applications using computer graphics ($100.000) 100 1968 Phosphor storage tubes increasingly available ($5000) 1000 1978 Raster scanned displays become increasingly popular 100.000 Image Generators and Displays Stefan Seipel 1984 First PC with graphical user interface (Macintosh) 1000.000 1994 Graphical user interfaces replaces increasingly text interface 50.000.000 1999 Almost no computer available without GUI 200.000.000 (Source: E.Bengtsson, Uppsala University) Random scan display Random scan display - computer control based on phosphor storage tube cathode ray is controlled arbitrarily in XY direction analog control of ray implemented in electronic circuits sets of vector primitives available (points, lines ) very smooth lines / high resolution line drawing only no filled surfaces no block transfers limited time frame to draw graphics up to 100.000 short vectors per refresh cycle computer keeps a list of graphical primitives computer issues rendering and parameters very little memory required to store display list no management of graphics buffer required display/engine and computer strictly separated

Raster scan display Raster scan display - computer control based on TV technology - short glowing phosphors screen is scanned in a line-by-line order pixels within a line are switched on/off while line is scanned two-dimensional memory matrix (frame buffer) contains pixel value filled areas can be drawn screen must be refreshed in short intervals computer manipulates values in the frame buffer discrete frame buffer matrix -> rasterization arbitrary graphical objects can theoretically been displayed need for rasterization algorithms graphical primitives must be computed host computer calculates and accesses frame-buffer memory must be read out -> video signal must be generated Frame buffer readout - RAMDAC Simple Raster Scan Display Architecture Frame buffer RAM (random access memory) Video Controller CPU other periphery host memory write memory read DAC (digital to analog converter) create video signal System Bus (frame buffer operations) readout of 1280x1024 pixels at 60 Hz -> memory access time 12.7 nanoseconds! normal RAM have an average access time of 60ns System Memory Video Memory Video Controller Monitor special purpose frame buffer required must allow for simultaneous read and write operations More or less typical configuration with common graphics cards

Typical frame buffer operations: 2D graphics subsystem 2D - Graphics operations CPU other periphery BitBlt (bit block transfer) Draw Pixel Draw Line Frame buffer operations are a heavy burden for a single CPU system! System Bus (2D graphical primitives) Draw Text Fill Area System Memory Raster Engine Frame (color) Buffer Video Controller Monitor 2D graphics subsystem Color buffer standards (common resolutions) Spatial resolutions: CGA - 320x200 EGA - 640x350 VGA - 640x480 SVGA - 800x600 XGA - 1024x768 SXGA - 1280x1024 or 1600x1280 HDTV - 1900x1200 ATC - 2000 x 2000 Human eye - 10.000 x 10.000 (not evenly distributed) Color resolutions: palletized - 256 colors lookup-table (8 bit/pixel) grayscale - 256 shades (8 bit /pixel) grayscale - 4096 shades (12 bit /pixel) e.g. in medical apps. Hi color - 5:5:5 RGB, or 5:6:5 RGB (15 or 16 bit/pixel) true color - 8:8:8 RGB (24 bit/pixel) true color - 8:8:8:8 RGBA (32 bit/pixel) Memory requirements and signal bandwidths Spatial Resolution SXGA 1280x1024 Color Resolution True color 24 bit Screen Refresh 72 Hz Video Memory Requirement : 1280*1024*24 bit = 3.75 Mbyte (1.3Mio pixel) Video readout : 94 Mio./pixel per sec. Memory readout : 226 Mio. bits per sec.! -> very special memory readout required (e.g. 240 MHz RAMDAC)

The standard 3D graphics pipeline 3D graphics adds another dimension Display List Traversal Modeling Transformation Lightning Clipping Projection Viewing Transformation Rasterization 2D color buffer only is not sufficient Geometric processing - thousands of flops - vertex transform - normal transforms - lightning calculation Rasterization - operates on pixels (framebuffer) - millions of iops - alpha compares - depth buffer test - stencil test - alpha arithmetic - texture addressing Features of 3D image generators Geometric processing (per vertex operations): one or several floating point geometry engines perform matrix and vector operations Features of 3D image generators Rasterization (per pixel operations): One or several raster engines feature: discrete line drawing and polygon fill z-buffer test (depth buffer test) hardware supported graphics operations: transformation of vertices rotation of vectors normalization of vectors (after scale of object) calculation of lightning projection & clipping z-buffer blending (fog) blending with alpha-channel (transparency) color interpolation (e.g. Gouraud shading) texel addressing tri/bilinear interpolation of textures anti-aliasing of edges stencil test

Advanced 3D graphics subsystem Buffers configuration example (3D system) CPU other periphery double buffer (front and back) System Bus accumulation buffer 8 b depth buffer 24 / 32 bit alpha buffer 8 bit blue-color buffer 8 bit green-color buffer 8 bit red-color buffer 8 bit System Memory Geometry Processor(s) Raster Engine(s) Video Controller Monitor stencil and/or overlay buffer 8 bit Local Memory (caches, textures) Frame Buffer(s) Other Buffers (Stencil,Depth,Alpha ) Memory required per pixel: 2*24+8+8+32+8 = 128 bit = 16 byte Graphics subsystem For a resolution of 1280x1024 -> 20 MB frame buffer Buffers configuration example (3D stereo system) Depth-Buffer Aliasing left right Quadbuffer front back alpha buffer 8 bit blue-color buffer 8 bit green-color buffer 8 bit red-color buffer 8 bit depth buffer 24 / 32 bit accumulation buffer 8 bit Fixed number of bits in the z-buffer limits resolution of the scene depth Example: 16 bit depth buffer allows only 65536 discrete steps in depth Arithmetical rounding operation of floating point depth values causes ambiguous z-values! Memory required per pixel: 4*24+8+8+32+8 = 176 bit = 22 byte For a resolution of 1280x1024 -> 27,5 MB frame buffer stencil and/or overlay buffer 8 bit Visual artifacts in rendering of objects which are very close to each other (see picture)

Performance parameters History (1999) Pixel Fillrate: refers to rasterization performance number of shaded/textured/buffered pixels per second common: 5-40 million/sec. (low and medium cost game accelerators) quite good: 100 million/sec. (graphics workstations) high end: 500-2000 million/sec. (e.g. SGI top of the line) Geometry Performance: refers to throughput of graphical primitives number of 3D shaded triangles/sec. (of certain size e.g. 25 pixel) number of 3D shaded lines/sec. (of certain length e.g. 10 pixels) common: <0.5 million triangles/sec. (low cost game accelerators) quite good: <1-2 million triangles/sec. (graphics workstations) high end: <50 million/sec. (e.g. SGI Octane) Nowadays (2002) performance doubled with ten even on PC hardware The gap between professional systems and consumer products is closing Performance parameters - What do they say? Problem: stated performance values are often only achieved under specific circumstances often these values refer to the most simple rendering modes (no shading, no z-buffer) performance is often achieved with native implementations quality of driver implementation is essential comparability is a tricky issue, since various new technical features Therefore: don t trust vendor supplied specifications! if you are lucky, you reach 10%-20% of the stated performance! graphics sub-systems should be tried before bought! in the target system (depending largely on CPU configuration) using your application (depending on typical graphics operations) Always check : is there driver support? are there drivers at all? Bottlenecks in 3D graphics systems Geometry bound systems - too many polygons in the scene - too complex lightning calculation - too complex model transformations Example bottleneck evaluation User requirements for a flight simulation: terrain model 1000 textured polygons, will fill most of the screen background airplane model 5000 polygons, average size 200 pixels for quality reasons, rendering should appear in SXGA resolution frame-rate >40/sec. Fill bound systems - too big areas to fill (number of pixels per polygon) - too many large polygons - high overdraw ratio - too many textures in scene - too complex alpha arithmetic (blending, fog) Depends on your application and graphics system There is a graphics subsystem available: 60 Mio. textured, lit, shaded pixels/sec. 2.000.000 triangles/sec. (25 pixels) cost: 30.000 SEK Is it advisable to buy this graphics subsystem? (1280x1024+5000x200)x40 = 92.428.800 pixels/sec. (1000+5000)x2x40 = 480.000 triangles/sec Fill bound!

API s (application programmers interface) API s continued Avoid to develop hardware oriented software Minimize turnaround costs and time Use well established and standardized 3D API s Use graphics accelerators which support those API s - full functional support - optimized performance for these drivers The best graphics hardware is worth nothing without appropriate API and driver support! Different standard 3D API s: Many different available today. The most renowned are: PHIGS and GKS (old DEC machines, not common any longer) OpenGL (SGI, Microsoft, many others) QuickDraw 3D (Apple) Direct3D (Microsoft) Glide (3Dfx) Heidi (used by many CAD programs) Quake (sort of standard for games) Displays Physiological Aspects Spatial Retinal Resolution : 1 Additional Reading Visual Field : approx. 200 o, with 120 o binocular overlap Roy S. Kalawsky: The Science of Virtual Reality and Virtual Environments Addison-Wesley Publishing Company, 1993, ISBN 0-201-63171-7 Perception: pages 50-59 Displays techniques: pages 98-107 Limits of depth perception from lateral disparity Temporal Resolution : approx. 50 Hz, increasing with luminance

Visual Displays - Basic Technologies Basic Technologies - Cathode Ray Tubes (CRT) Cathode Ray Tubes Flat Panel Displays Electroluminiscence Displays LCD Displays Active Matrix TFT Light Valves Laser Scanners Micro Mirror Devices Advantages: - high resolution - easy to control - reliable technology -low cost Cathode Acceleration Anode Focussing Electron gun x-/y- Deflection (elstat / magn.) Phosphor Coating Basic Technologies - Cathode Ray Tubes (CRT) Operation modes of CRTs: Basic Technologies - Random Scan CRT How it works: Display primitives are stored in a display list (ellipses, circles, lines...) Random scan mode Raster scan mode Electron beam is controlled continuously between vertice -> analog line drawing + no aliasing, smoth lines + transformation in hardware possible + little memory required - only line drawing possible - complex control hardware for the beam - flickers if scene becomes complex

Basic Technologies - Raster Scan CRT Basic Technologies - Raster Scan CRT How it works: Display is rasterized into pixels which are stored in a two-dimensional memory array (raster memory) Electron beam is traversing the screen line by line in a regular time frame and scheme Content of the raster memory controls beam intensity + uses old TV technology (simple approach) + filled and shaded surfaces are possible + guaranteed screen refresh rate - aliasing problems staircase effect - electronics required for raster memory readout - drawing algorithms more complex Host Frame buffer 00000000011100000000000000100010 00000000101010000001010100011000 00000000000000000100100011111000 00000000000000000010010100001001 00000000000010101000100111111000 00000000000001011001111001011000 00000000000000000000101001010011 Display Controller Converts analog primitive into discrete representation Video Controller Converts discrete frame buffer into analog video signal scan line horizont retrace vertica retrace Basic Technologies - Cathode Ray Tubes (CRT) The role of the phosphor coating : - Fluorescence (glowing when hit by electron beam) - Phosphorescence (after glowing while being activated) - Persistency (time until glowing phosphorescence decreases below 10%) typically 5-60 milliseconds. - Persistency is important. Short persistency requires high update rates otherwise flicker. Long persistency causes stabile but smeary images. - Granularity of the phosphor -> spot size, image resolution - Type of phosphor defines color: p1 : green, average persistency p4 : white, short persistency p12 : orange, average persistency p31 : green, short persistency Basic Technologies - Cathode Ray Tubes (CRT) How is color accomplished in displays? Most usually, colors are mixed by additive composition of base colors 1. Spatially modulated color composition see page 99, in Kalawsky 2. Temporally modulated color composition see next slide

Basic Technologies - Cathode Ray Tubes (CRT) Parameters for display assessment How is color accomplished? Color shutter technology. - Sequential display of color fields on monochromatic CRT - Synchronization of the fields with color filters - Filters can be : a) LCD filters (electronically controlled) b) Optical filters (mechanically coupled) - Advantages : No color convergence errors - Disadvantages : High frequent oscillations in the visual field decompose colors - dot size (mm) - dot pitch (mm) - resolution (lp/mm) -brightness(cd/m 2 ) - contrast ratio -display size - addressability - refresh rate - color range - convergence - weight / power consumption 1mm 1mm 1mm dot size? dot pitch? resolution? 1mm Basic Technologies - Typical parameters for CRTs Basic Technologies - Flat Panel Display - LCD Liquid Crystal Displays Screen Diagonal Size (14-26 ) Shadow masks (Triple holes, Strips) Dot pitch (0.24-0.30 mm) Video bandwidth (50-250 MHz) Horizontal Sync. Frequency (30-170 khz) Vertical Sync. Frequency ( 48-170 Hz) max. Resolution (1280x1024-4800 x 4000) vert. pol.filter vertical electrodes liquid crystal layer horizontal transparent electrodes hor.pol.filter No electric field = light passes through Light source or mirror

Basic Technologies - Flat Panel Display - LCD Basic Technologies - Flat Panel Display Addressing of Pixels: Deposit an electronic charge on intersections between horizontal and vertical electrodes sequentially row by row. Limited speed, since certain minimum time is required to is deposited (depending on the capacity of the intersections) When last row has been addressed, first rows have already lost their electric charge Poor contrast image Thin Film Transistor Matrix (TFT): An array of transparent transistors is deposited on the LCD Pixels can be switched on and off Pixel keep their electrical state and optical properties => Significant contrast and intensity enhancements Basic Technologies - Flat Panel Display Basic Technologies - Flat Panel Display Color reproduction / Optical Efficiency: Electro Luminescence Displays (plasma panel displays): Groups of adjacent pixels are forming one effective color pixel Gas is encapsulated between electrodes Potential Sub-pixels are covered with color filters Common sub-pixels configuration are RGB stripes, triads or quads When a certain amount of voltage is applied (striking voltage) the plasma discharges and glows until the potential drops below the discharge voltage. S Strike Voltage Efficient resolution is reduced Plasma cell keeps luminating for a while without being refreshed. D Discharge Voltage Light intensity is diminished significantly when passing through polarizes, liquid crystals, and color filters (poor optical efficiency) Active luminance, high intensity display Plasma generates light t (see also page 96)

Basic Technologies - Flat Panel Display Electro Luminescence Displays (plasma panel displays): Basic Technologies - Light Valves Application : Projection Displays glas Sandwich-Technique vertical transparent electrodes glass substrate with plasma cells (Neon, Argon) horizontal transparent electrodes glass High resolution light modulation (>1600 x 1200) High refresh rates possible (>130 Hz) Usable for high light output projection displays The first choice for stereo projection systems Tricky problems : Ghost images with slow phosphor see page 103 Basic Technologies - Laser Scanner Basic Technologies - Micromirror Devices (MMD) Used for: large screen projection displays direct retinal displays (very high resolution) Screen Matrix of micro mirrors Addressable and electronically controllable Used for Light Reflection and Projection Display Systems for color displays, several lasers required (convergence?) vertical deflection mirror horizontal deflection mirror Extremely high optical efficiency laser source

Basic Technologies - Micro Mirror Devices (MMD) Visual Displays - 3D Displays and Optical Systems MMD System Working Principle 3D Projection Systems Front Projection Rear / Retro Projection Autostereoscopic Displays Slot Mask Lenticular / Double Lenticular Arrays Volume Displays Other Optical Coupled Displays Fiber coupled displays (HMD see chapter 4.2.1) Lens/Mirror coupled displays (HMD see chapter 4.2.1) 3D Displays and Optical Systems - Projection Systems 3D Displays and Optical Systems - Projection Systems 3D Projection Display Systems require: Very high intensity image source Transmission LCD/TFT Panel + Light Source Reflection LCD/TFT Panel + Light Source Projection CRT (specialized high intensity CRT tube ) MMD Light Valve (Laser) Focusing optics/color splitter Wide / narrow angle optics, fixed or variable Projection screen Transparency / Diffusion / Specular Properties Means of splitting left/right channel Time Multiplexing / Color Separation / Polarization 3D Front Projection Systems: Image source and observer are located on the same side of the projection surface ( - user may interfere with projection beam) Stereo 3D with active shutter glasses requires very fast image source (>120 Hz) light valve / projection CRT extremely expensive single graphics pipeline screen with good diffuser properties Stereo 3D with passive polarizing glasses requires two image sources image alignment problems dual graphics pipeline required requires special silver screen which preserves polarization Stereo 3D with color field separation (red/green) one color capable projector (cheap) poor image result screen with good diffuser properties Image Source + Optics Observer Screen

3D Displays and Optical Systems - Projection Systems 3D Displays and Optical Systems - Projection Systems 3D Rear / Retro Projection Systems: Image source is positioned behind the projection screen No interference between user and image source Image Source + Optics 3D Rear / Retro Projection Systems: Examples Requires transparent screen material No polarized 3D stereo possible since polarization is disturbed in transmission Screen Stereo 3D with active shutter glasses requires very fast image source light valve / projection crt expensive single graphics pipeline Attention has to paid to mirror effects Observer Caves Virtual Planes Viewing wands 3D Displays / Optical Systems - Autostereoscopic Displays 3D Displays / Optical Systems - Autostereoscopic Displays Autostereoscopy - stereoscopic perception with the naked eye Image Splitter (Sanyo) (Double) Lenticular Lens Arrays LCD-Projektor LCD-Projektor Display divided in vertical stripes Alternate stripes display left and right image Slit-mask is blocking out the view of the left eye onto the right picture and vice versa Only a single user Dedicated observer position Horizontal resolution decreased Display surface Slit mask Pixel Column R Pixel Column L Pixel Column R Pixel Column L Pixel Column R Pixel Column L Display divided in vertical stripes Alternate stripes display left and right image Half-Cylinder shaped lenses project the stripes to the corresponding eye Several viewing zones Dedicated observer distance Horizontal resolution decreased Double-lenticular retro-projection system LCD/TFT Panel Lenticular flat panel display system

3D Displays / Optical Systems - Autostereoscopic Displays Examples (Heinrich-Hertz Institute, Berlin) 3D Displays / Optical Systems - Volumetric Displays The display creates a real volumetric representation which is perceived as a 3D structure without the need for glasses or other aids. Allows for user movements Uses head-tracking Screen is automatically positioned correctly with a robot arm Allows for user movements Uses head-tracking Lenticular array is shifted The idea is to project dynamic images onto oscillating or rotating surfaces in order to create the sensation of a volumetric object. Prototypes have been build using: - rotating LED matrices - rotating helical projection surfaces with laser projection laser - lasers projecting into fog - experiments are underway to bring a solid crystal to illumination on addressable positions All these systems can only show transparent/monochrome objects Mechanical problems and limits, dead viewing areas Commercial systems are far ahead 3D Displays / Optical Systems - Volumetric Displays Choice of VR Displays - Evaluation of Requirements How many observers are watching at the same time? What resolution and color fidelity requirements are there? -> basic display technology Is wide field of view desirable? Is immersion an important issue for the application? Is stereoscopic 3D rendering required? If yes, decide which type one screen polarized -> take care for optical properties of the system one screen time multiplexed -> display must tolerate high refresh rates dual screen (HMD) -> check for resolution autostereoscopic? Does the application require interaction with haptic stimuli?

Displays Technologies - Features 3D Displays and Optical Systems - Projection Systems CRT LCD TFT M M D Lightw alv Addressability 4 kp ixel 2 kp ixel 2 kp ixel 1.3 kp ixel 4 kp ixel Contrast h ig h lo w h ig h h ig h h ig h Colors very good m edium good very good very good Dim ensions huge sm all/m edium sm all/m ediumsm all m edium Refresh <180 Hz 60 Hz 60 Hz 60 (180Hz) 140 Hz Costs low average high high very high Considerations with regard to stereo image projection Time-multiplexing with active shutters: both front and retro projection possible active glasses are quite expensive (if many are required) very high speed projector is required (light valve technology, expensive) Polarized filtered images: projection screen must preserve polarization (aluminized silver screen) retro projection not yet possible (no suitable screen material available) glasses are very cheap two projectors are required (can cause image alignment problems) FULLY IMMERSIVE SPHERICAL PROJECTION SYSTEM (THE CYBERSPHERE) http://www.ndirect.co.uk/~vr-systems/sphere1.htm RealityVision s autostereoscopic display using holographic optical elements Contact: David Trayner or Edwina Orr.RealityVision Ltd. 6 Yorkton St. London E2 8NH, UK: +44 (0)171 7399700 F: +44 (0)171 739 9707 E: reality@augustin.demon.co.uk