Low-Level PlumbingforMedia Integration Turner Whitted Microsoft Research
Outline • Part I: Implementation chronology • Then, Now, Then again … • aka Wheel Of Reincarnation • Part II: Architectural musing • Perceptual/content requirements • Data paths, data types • Not about programming models
Starting points – E&S Frame Buffer • NO fixed function units • mCode for basic logic • mController programmable by designer only • Treated as peripheral Ref: Kajiya, J.T., Sutherland, I.E., and Cheadle, E.C., "A Random-Access Video Frame Buffer," Proceedings of the Conference on Computer Graphics, Pattern Recognition, and Data Structure, UCLA Extension, Los Angeles, California, May 14-16, 1975
Starting points – Ikonas RDS3000 • Single, 32-bit wide datapath based on 2901 bit-sliced DSP • Programmed in C (Gary’s Ikonas Assemler) • NO fixed function units • Bound [later memory mapped] to single application Ref: N. England, A graphics system architecture for interactive application-specific display functions, IEEE CGA, pp. 60-70, Jan 1986.
Starting points – Pixar CHAP • SIMD processors • Loops, conditional execution • Focus on parallel programming issues Ref: Adam Levinthal and Thomas Porter, “Chap – A SIMD Graphics Processor,” Proceedings of SIGGRAPH 84, (18) 3, July 1984, pp. 77 – 82.
Host Bus Example: AT&T Pixel Machine One or, optionally two, 9-element 16, 20, 32, 40, or 64 geometry pipelines Pixel Nodes “The” pipeline • [Mostly] Fixed structure, programmable nodes • Vector graphics legacy
Structural evolution • Texture engine with remnants of line drawing DNA
Performance • Ikonas RDS3000 (1980) • 20 MB/s processor to memory • 20 MIPS equiv. • Pixar CHAP (1984) • 240 MB/s processor to memory (P-bus) • 64 MIPS peak • ATI 9700 (2002) • 20800 MB/s chip to memory • 8400 MIPS
API abstractions (OpenGL, DX) • Fixed structure pipeline accessed through API • API tracks hardware through several generations while maintaining consistency • Integrated with mainstream computing • Easy to program • Single largest key to commercial success of graphics systems and applications
Complete the circle • FPGA+memory • NO fixed function • Programmable in C (Verilog actually) • Good for simple prototyping • No API • No device drivers • No SDK
Part II: Going forward … • Motivation • We’ve tacked every imaginable feature onto what was initially a line drawing pipeline • It’s time to start over • The window of opportunity is wide open
Alternatives Feed eyeballs with Integrated media Real-time photorealism Parallel execution for scientific applications Parallel execution for interactive applications Going forward …with graphics processors 3D Raster Graphics
Integrated media(partial illustration) Concentric mosaics Image centric Geometry centric Sprites with depth View-dependent texture View-dependent geometry Light field Lumigraph Fixed geometry Polygon rendering + texture mapping Interpolation Warping
Geometry Image completely regular sampling 3D geometry geometry image257 x 257; 12 bits/channel Ref: X. Gu, S. Gortler, H. Hoppe, “Geometry images,” ACM Transactions on Graphics 21(3): 355-361 (2002)
From first principles: function U-VWL* Content Line drawing Display Processor Display Device Video Still photos Animated 3D shapes Text *Ultra-Vast Wasteland
3D text experiment • Extend IBR/volume rendering to text • Superior image reconstruction • Higher image quality than mip-mapped texture Olynyk, Mitchell, Snyder, MSR
General purpose front end Fixed function back end Generic physical blocks Read Mapper Cache Common Memory Recon- Write Struction/ Filtering Cache Ref: T. Whitted, “Overview of IBR:Hardware and Software Issues,” ICIP 2000.
Don’t count on quantum GPUs soon – stick to CMOS digital logic Count on CAD more than feature size Heat is the enemy The economy of commodity DRAM is hard to beat But there is huge performance pressure on DRAM Designers are restricted only by a lack of experience CMOS digital circuitry Commodity DRAM From first principles: implementation Content
Design challenges • Essence of the problem • We don’t have a function to implement • We must design specifically for unknown methods • Brute force is prohibited • Feeds and speeds …What do we know? • Regulated by content • But we rarely turn content conventions into quantitative measures • Limited by perception • Which we don’t fully understand • Function of representation • If we don’t know the representation, we don’t know the flows • In my group’s research we limit flow to HW “sanity” and then work backwards
Summary • Graphics hardware has nearly completed the circuit back to its starting point • Flexible, powerful, programmable • Media processing requirements extend beyond the classic 3D pipeline • Unusual window of opportunity • to match architecture with a broader range of applications and content