Designing an AR App: Evaluation

In the previous post, I built a Wizard of Oz prototype of Suber, an AR app that shows live captions just below the speaker’s chin. In this last part of the series, I look at what holds up and what doesn’t.

Watching People Use Things

In the early 1980s, Lucy Suchman, an anthropologist at Xerox PARC, filmed first-time users trying to use a new photocopier. Its “expert help system” seemed perfectly clear to its designers, yet users kept getting lost and misreading what the machine expected of them.

Suchman’s lesson still shapes HCI. People’s actions are situated: they depend on what is in front of them and what just happened. A designer can predict how a system should be used. Only someone else using it shows how it is used.

Method

The lectures presented three main ways to evaluate a design: usability testing, where you watch a few people perform tasks; A/B testing, where you compare numbers across many users; and heuristic evaluation, where experts review an interface without real users. I combine the notes of a friend who reviewed the demo with my own observations.

Findings

1. Placement. The reviewer noted that the text and the mouth were close enough that he could see both at the same time. In a real conversation, this would let someone not only read the captions but also follow the speaker’s facial movements.

2. Text size. The reviewer found the text a bit small and narrow, a limitation of my prototyping method that can be fixed in AR. He also noticed that the phone was sometimes tilted, which made it harder to read.

3. Text color. He also found the white box behind the text a bit too bright and distracting. AR text does not need a background, so this problem can be avoided. As for the text color itself, AR allows environment-aware coloring.

4. Delay. The reviewer does not speak French, but he was able to follow the subtitles in English. Even though there is a lag between speaking and text, the system works. Natural conversation tolerates some delay, as long as it is not too noticeable. Then again, my demo used short, slowly spoken sentences. We should test faster speech and captioned examples to see at what point the system breaks.

5. Eye contact. The reviewer remarked that he found it hard to read the captions and keep eye contact at the same time. This is worth looking into, since eye contact is essential in conversation. Perhaps we can test alternative text placements, such as on the forehead.

Limits

Even though the reviewer raised some strong points that I had also noticed, one reviewer is not enough. This is why testing usually involves multiple participants from diverse backgrounds.

The demo can be fleshed out a bit more to include the sign language feature (horizontal slicing). Inclusion of Deaf and hard-of-hearing people would also be necessary in the testing phase.

In my pilot, I also had no wrong captions, but a real system will produce them. Further testing is required to gauge which kinds of errors are correctable and which lead to misunderstandings in the conversation.

This was a formative evaluation: it was meant to find problems and sharpen the next questions, not to prove that Suber works.

Conclusion

Looking back at the series, each phase answered a different question. Ideation asked whether Suber was the right design. Prototyping asked how to make it real enough to try in an afternoon. Evaluation asked whether it works for someone other than me. The design is still far from perfect, but by following the app development lifecycle, we were able to brainstorm different ideas, identify real needs, create simple demos, and critically evaluate our own design.