The Ultimate Display

In 1965, Ivan Sutherland published a short paper called The Ultimate Display. This article describes much of the computing we use today: area-filling color graphics, drag-and-drop, force feedback, eye tracking, and something very close to virtual reality.

At the time, Sutherland had just built Sketchpad, the first program that let you draw directly on a computer screen with a light pen. Most displays could only plot dots and straight lines, so the paper reads more like a wish list than a technical report. What makes it remarkable is how much of that list has come true.

In this post, I go through his predictions: which ones are already part of everyday life, and which ones might still become reality.

A Looking Glass Into a Mathematical Wonderland

The paper opens with an observation about how we learn. As humans, we develop an intuitive feel for how the world works: we can predict the trajectory of a ball or how hard it is to push a box. However, our intuition does not help us with things like forces on charged particles or motion without friction.

Sutherland envisions the computer display as a looking glass into this mathematical wonderland, letting us see and explore what we can’t experience directly. He was right: today, physics and fluid simulations routinely make such phenomena visible.

The Prophecies

Curves and Color

An ability to draw simple curves would be useful. … It is equally possible for a computer to construct a picture made up of colored areas.

Both are now taken for granted. Vector graphics describe shapes such as fonts as mathematical curves, so they stay sharp at any size, and modern GPUs fill billions of colored pixels every second.

No commercial display could do this in 1965. His remark that “we have much to learn about how to make good use of this new ability” turned out to describe an entire field: computer graphics.

The Typewriter

His most confident prediction is also the least exciting:

Tomorrow’s computer user will interact with a computer through a typewriter. He ought to know how to touch type.

Sixty years later, he’s still right. Despite every new interface since, the keyboard remains the primary main way to enter text. Even touchscreens use a virtual version of the keyboard.

The Stylus

Sutherland also stressed the importance of the stylus as a tool for pointing and drawing. The prediction holds true: the stylus remains the preferred input for artists and graphic designers. His comment about changing a digit on a screen by writing over it is basically handwriting recognition.

He also described pointing at objects on the screen and moving them. He was referring to drag-and-drop, almost 20 years before the Macintosh made it mainstream.

The Joystick

… adjustment of the viewing angle of a perspective view is conveniently handled through a three-rotation joystick.

This is exactly how we look around in 3D games today, where a thumbstick on the controller rotates the camera. In CAD, 3D mice like the SpaceMouse do the same job.

Voice Input

Syllable voice input should not be ignored.

It took decades to become reliable, but with Siri, Alexa and modern speech-to-text, we now talk to computers every day. With the rise of Large Language Models (LLMs), voice input has become even more important and natural.

Object Selection

In a surprisingly technical section, Sutherland complains that when a user points at a picture, the computer struggles to work out which object, and which part of it, is being pointed at. Searching screen coordinates was slow, and no display could handle nested structures.

Today, this hit testing happens invisibly. Browsers find the topmost element under the pointer (using z-index) in a tree that already knows what contains it. 3D engines cast a ray (raycasting) through the cursor to the first object hit, with a scene graph providing the nesting Sutherland lacked.

The Kinesthetic Display

Turning to other senses, Sutherland describes a kinesthetic display: a joystick whose resistance is computer-controlled, like flight simulator controls that recreate the feel of a real airplane. He mentions General Electric’s “Handyman”, a robotic arm whose controls are “nothing but joysticks with nearly as many degrees of freedom as the human arm.” With such devices, he says:

we can add a force display to our sight and sound capability.

The computer would not only show and play things for us, but also push back.

We now call this haptics. Racing wheels resist your turns, the PlayStation 5’s adaptive triggers stiffen when you draw a bow, and phones vibrate to make virtual buttons feel like clicks.

Eye Tracking

Sutherland points out that only our hands and arms control computers, even though

the computer can easily sense the positions of almost any of our body muscles.

He singles out the eyes, predicting that

machines to sense and interpret eye motion data can and will be built.

They have been. The Apple Vision Pro uses eye tracking as its main way of pointing: you look at a button and pinch to click. That’s quite literally the “language of glances” he wondered about.

The Rounded Triangle

He then proposes an experiment: a display that changes depending on where you look, such as a triangle where “whichever corner of it you look at becomes rounded.” What would such a triangle look like?

This is essentially foveated rendering. Our eyes only see sharply at the center of our gaze, so headsets track where you’re looking and render only that area in full detail. Done well, you never notice, just like you’d never catch Sutherland’s triangle with a sharp corner. It’s both a new way to control machines and, as he predicted, a product of understanding “the mechanisms of vision.”

Looking Forward

Smell and Taste

Sutherland writes that “no one seriously proposes computer displays of smell, or taste,” and this is still largely true. Attempts like Smell-O-Vision in cinemas and scent cartridges for PCs never caught on, and experiments that electrically stimulate the tongue to change how salty food tastes remain niche gadgets. Unlike light or sound, these chemical senses can’t easily be synthesized from a few basic signals.

Beyond Hands and Eyes

He saw no reason why only our hands should control computers, yet besides eye tracking, we’ve barely explored this. One promising direction is electromyography (EMG), which reads the electrical signals in your muscles. Meta’s Neural Band, a wristband released with its display glasses in 2025, uses it to detect finger gestures, even tiny ones made with your hand resting in your lap. Brain-computer interfaces go further, but are still mostly limited to medical research.

Breaking the Laws of Physics

Sutherland argues that

there is no reason why the objects displayed by a computer have to follow the ordinary rules of physical reality.

A kinesthetic display could simulate negative mass, and a visual display could make solid objects transparent so we can “see through matter.”

Parts of this exist: CAD and medical imaging let us see through objects, and simulations and games let us explore altered physics and non-Euclidean worlds. But his deeper hope, that “we can learn to know them as well as we know our own natural world,” is still far off.

Controlling Matter

The paper ends with its most famous paragraph:

The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. … a bullet displayed in such a room would be fatal.

What Sutherland describes here is a completely immersive reality that would be indistinguishable from our real lives. A computer that can “control the existence of matter” is still far-fetched, and today’s VR is nowhere near that convincing. Interestingly, Sutherland didn’t shy away from the darker side. He understood that a display convincing enough to be useful must also be dangerous. As immersive technology improves, questions about safety, manipulation, and the nature of our own reality will only grow.

Conclusion

What stands out to me is Sutherland’s foresight. Many of the principles and interfaces he described remain at the core of technology today. That’s why his paper is still a useful guide for HCI research sixty years later.

However, it remains to be seen whether, and when, we will build The Ultimate Display that truly fits Sutherland’s vision.

References

1. Ivan E. Sutherland. 1965. The Ultimate Display. Proceedings of the IFIP Congress, pp. 506-508. PDF
2. Ivan E. Sutherland. 1963. Sketchpad: A Man-Machine Graphical Communication System. AFIPS Spring Joint Computer Conference, pp. 329-346. doi:10.1145/1461551.1461591