Seeing Between the Pixels

Pictures are at the heart of how we communicate with computers, emblematic of our cur- rent fascination with multimedia and web-based computing. Nevertheless, most of us know far less about pictures and the way in which they work than we know about the text that often accompanies them. In an attempt to understand pictures, perhaps the most fundamental question we can ask is, "What is a picture?" What is it that objects as di- verse as icons, bar charts, paintings, and photographs have in common that makes us refer to all of them as pictures? And what is it about pictures that convinces us to use them instead of, or in addition to, text? We often talk about how pictures "depict" things. But, even the process of depiction seems to differ from one picture to another. On a computer, we may use a paint system to guide a virtual brush over the screen, a video camera to capture a live image, a spread- sheet to automatically generate a corresponding bar chart, or a rendering system that models the interactions of synthetic lights, objects, and cameras. Is there some un- derlying property that these processes all share? Computer scientists are used to thinking of pictures in terms of their representation: an array of pixels, a list or hierarchy of graphics primitives, or even a program written in a language such as PostScript.