Designing an AR App: Prototyping

In the previous post, I pitched Suber, an AR app that shows live captions next to the people around you. On paper, the idea sounds useful. But there is no way to know whether it actually works until someone tries it, and building real AR glasses would take months. So in this post, I turn the idea into something a person can try, as quickly and cheaply as possible.

Prototype in a Day

Months of building isn’t the only option, and few stories show this better than Google Glass. At Google X, Tom Chi’s team wanted to know what it would feel like to wear a computer on your face. Instead of waiting for custom hardware, they built a working prototype in a single day, using a coat hanger, a sheet protector, a pico projector and a netbook.

Tom Chi draws two rules from this. The first is to find the quickest path to experience. The second is that doing is the best kind of thinking: one of his team’s quick prototypes showed that controlling Glass with hand gestures felt socially awkward, something no amount of discussion would have revealed.

Why Prototype?

A prototype is a question you can hold in your hands. Instead of debating whether an idea works, you build just enough of it to find out. In the lectures, prototypes had three goals:

1. Learning: finding out whether the idea works at all.
2. Searching: exploring different versions of a design and comparing them.
3. Communicating: showing others what the idea is, which is much easier than describing it.

Prototypes range from low fidelity, like storyboards, paper prototypes and roleplay, to high fidelity, like a Figma mockup, a Unity scene or an Arduino circuit. They can also be horizontal, covering many features at a shallow level, or vertical, covering a single feature in depth. Early on, low fidelity wins: the goal is to move fast and fail fast, before any real effort is wasted.

Prototyping for AR

AR is especially hard to prototype because the interface lives in the real world. A screen mockup can’t show how a caption feels when it floats next to a real person’s face. The lectures suggested a few ways around this. Physical props and acting things out are often faster than animating them, and tools like ProtoAR let designers build AR scenes from paper sketches and clay models captured with a phone.

For Suber, the most natural choice was a Wizard of Oz prototype. In this method, a hidden person plays the part of the computer. The user believes the system works on its own, while the “wizard” behind the curtain produces its responses by hand. J. F. Kelley named it after the film in the 1980s, when he used it to test a natural language interface that didn’t exist yet. For Suber, a wizard could type out what the speaker says, and the captions would appear on a phone in front of the listener.

The catch is that this usually needs three people: a speaker, a wizard and a listener. I had only myself, so I took on two roles. I am the speaker and the wizard, typing out my own words as captions, and a camera takes the place of the listener.

What I Wanted to Learn

A prototype should answer specific questions. For Suber, these were:

1. Placement: are captions easier to follow just below the speaker’s chin, or at the bottom of your view like TV subtitles?
2. Usefulness: do the captions actually help you follow a conversation?
3. Delay and errors: how much can the captions lag behind, or get wrong, before the conversation breaks down?
4. Social comfort: how does it feel to talk to someone while they are reading your words off your chest?

Answering them only takes one feature, so this is a vertical prototype. It focuses on live captions for speech and leaves out sign language, since faking sign language recognition would require a fluent signer.

The Storyboard

Before building anything, I drew a storyboard of the dinner table scenario from the ideation post. It is the cheapest prototype of all, and it covers the communicating goal: anyone can understand what Suber does in a few seconds.

Drawing it also exposed a gap in the idea. Suber only helps you understand the people around you. It doesn’t help them understand you. So in the storyboard, you reply in simple French. That is exactly the situation Suber is for: you can manage a few words, but you can’t keep up with fast speech. Two-way translation would need the other person to wear Suber too.

Building the Wizard of Oz Prototype

The roles. I play the French dinner guest from the storyboard. I speak French, and Suber shows my words in English for the newcomer at the table. The newcomer is a camera, filming me from where a listener would sit, so the recording shows exactly what a Suber user would see.

The wizard’s tools. I wrote a small web page for the wizard. Whatever I type on my laptop appears on my phone a moment later. While I type, the text streams in light italics, like speech recognition still working out a sentence. When I press Enter, it turns into a bold caption and fades out after a few seconds. Lines can also be prepared in advance and sent with a single key, though for the pilot I typed everything live.

The phone is the glasses. Instead of floating in the air, the caption sits on a phone just below my chin, hanging from the lanyard of a card holder. It’s the placement from the ideation post, faked with a ribbon, in the spirit of Tom Chi’s coat hanger.

Pilot Run

My first attempt didn’t go as planned. I held the phone in one hand and typed with the other, which left no way to also hold a conversation. Even a one-person prototype needs free hands, so the phone had to hang on its own.

For the second attempt, I hung the phone on the lanyard. It still swung around and sat too low, so I tucked it into the collar of my hoodie, which held it flat just below my chin. Then I introduced myself in French while typing the English captions: “Bonjour. Je suis étudiant. Je viens d’arriver en France. Et toi ? Tu viens d’où ?”

A few things stood out when I watched it back:

1. Two languages at once is hard. Speaking French while typing English at the same time was difficult, and it introduced some delay. Even so, each caption was finished within about a second of the end of its sentence. That is optimistic, though: I knew exactly what I was going to say.
2. Face and caption fit in one view. Below the chin, the caption sits close enough to the face that the camera, like a listener, can take in both at once.
3. The wizard looks away. Every time I typed, my eyes dropped to the keyboard. For a real listener, that would look like I was avoiding eye contact, which has nothing to do with Suber itself.
4. The text is small. On camera, the caption takes up only a small part of the frame. At dinner table distance, it would need a larger font or fewer words at a time.

Conclusion

The prototype took very little to build, and it already shows the core of Suber: a person’s words, in your language, right below their face. The pilot also showed what the setup needs: free hands for the wizard and a phone that stays put.

It is still far from the real thing. The wizard is also the speaker, so the captions are only as fast as my typing, and the only errors are the ones I make myself. I also tested just one placement, so captions at the bottom of the view, like TV subtitles, are still untested.

The next step is to put it in front of a real person. In the next post, a participant takes the place of the camera, while I stay the speaker and the wizard.

References

1. Tom Chi. 2012. Rapid Prototyping Google Glass. TED-Ed. https://www.youtube.com/watch?v=d5_h1VuwD6g
2. J. F. Kelley. 1984. An Iterative Design Methodology for User-Friendly Natural Language Office Information Applications. ACM Transactions on Office Information Systems 2(1), 26–41. https://doi.org/10.1145/357417.357420
3. Michael Nebeling, Janet Nebeling, Ao Yu and Rob Rumble. 2018. ProtoAR: Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications. In Proceedings of CHI 2018, 1–12. https://doi.org/10.1145/3173574.3173927