Designing an AR App: Ideation

Augmented Reality (AR) glasses are slowly moving from research labs to our faces, and with them comes the chance to design apps that expand what we can do and take care of everyday chores.

This post is part of a three-part series where I explore the three phases of ideation, prototyping and evaluation. First, I look at how designers generate and narrow down ideas, and then present three ideas around the theme Enhancing human abilities in the age of augmented reality.

Getting the Right Design

In his highly influential book Sketching User Experiences: Getting the Design Right and the Right Design, Bill Buxton stresses the importance of early-stage low-fidelity sketching. He distinguishes between getting the design right and getting the right design, and argues that the latter must come first. No amount of polishing can save an idea that solves the wrong problem. Ideation is about exploring several directions before committing to one.

Paul Laseau, a pioneer of graphic thinking, describes this process as a funnel made of two alternating steps:

1. Elaboration: opening up the design space by generating as many ideas as possible, without judging them too early.
2. Reduction: narrowing the options down by comparing them against criteria, such as user needs, feasibility and risks.

Through this iterative process, we end up with fewer but more refined ideas. Sketching drives the elaboration step, and it is not about drawing well: a sketch is quick and disposable, and its job is to express, develop and communicate an idea. My sketches below are rough on purpose.

1. Suber: Live Captions for Speech & Sign Language

Imagine sitting at a dinner table where everyone is speaking a language you can barely speak. You struggle to keep up, and even when you understand something, the conversation has already moved on. If only you could have subtitles in real life…

The idea. Suber shows live subtitles for the people around you. As someone speaks, their words appear as text next to them, translated into your language if needed. It also supports sign language, turning signs into text for people who don’t know sign language.

Who it helps. Deaf and hard-of-hearing people, but also anyone in a language they don’t speak well, like an international student in Paris at that dinner table who wants to join the conversation instead of nodding along.

Where should the text go? Subtitles anchored at the bottom of your view are easy to read but disconnected from the speaker. Speech bubbles attached to each person use the Gestalt law of proximity, so you can instantly tell who said what. Many Deaf users also rely on lip reading, so the text must not cover the face. If the text is placed far from the face, it forces the eyes to jump back and forth. Overlaying the text below the speaker’s chin seems like a good compromise.

Who is speaking? A microphone array could locate the speaker using the same cues our ears use: tiny differences in timing and loudness between the two ears. Captioning everyone in a busy café would be the AR version of the Midas touch problem, so Suber should only caption the people you are looking at.

Risks. Reading while watching someone’s face splits attention, and any delay in the speech recognition breaks the natural rhythm of a conversation. There is also a privacy issue: Suber transcribes bystanders who never agreed to it. Finally, sign languages are not universal (French Sign Language is different from ASL) and have their own grammar based on space and facial expression, so Suber should take these differences into account.

2. MeetMe: A Real-Life Social Network

You’re new in town and know no one. You don’t want to go online — who knows the kind of people there are. Then you see someone sitting across from you in the café. You want to talk to them but you fear it might bother them. If only there were a way to know what their vibe is like.

The idea. MeetMe is a real-life social network. Users can share public information about themselves, such as looking for work, new in town or down for a beer!, and people nearby wearing AR glasses can see it floating above them. Privacy settings control whether a tag is visible to friends only or to everyone.

Who it helps. Approaching strangers is hard, especially for newcomers and introverts. MeetMe enhances social perception by showing who is open to talking.

Risks. While sketching this idea, the risks quickly started to outweigh the benefits. Ironically, MeetMe brings the very danger that keeps people away from meeting strangers online into the physical world. Broadcasting a status like single to strangers nearby is an easy tool for stalking and harassment, and the risk isn’t the same for everyone. Context-aware systems also suffer from an awareness mismatch: users tend to forget what they are sharing, so Norman’s principle of visibility becomes a safety requirement here. You should always be able to see exactly what others see about you.

It also raises a question: what are the mixed reality equivalents of dark patterns? MeetMe could push users into always sharing a status, create a fear of missing out (“3 people nearby are down for a beer”), or disguise ads as people’s tags.

Making it safer. A more careful version would be opt-in and mutual (you only see someone’s tags if they can see yours), with statuses that expire after a few hours and only work inside a specific event, like a conference or a meetup. This is already a good example of reduction: a narrower idea that solves a clearer problem.

3. Astor: Find Your Way Around the Supermarket

Have you ever gotten lost in a supermarket trying to find that one last item on your grocery list? Or perhaps you just can’t seem to find the price tag on that one thing?

The idea. Astor helps customers find their way around supermarkets. It reads your shopping list and guides you to each item using markers in AR. It also overlays useful information on top of products, such as prices, nutritional value and whether it fits your dietary preferences.

Who it helps. A busy parent with a long shopping list, someone with a food allergy, or a tourist who can’t read the labels in a foreign supermarket.

Design questions. Figure-ground contrast and color could make the target product “pop out” of the shelf. Showing information on every product, however, would turn the shelf into noise. Thus, details should only appear for products on or near your line of sight, after a short dwell or a pinch to avoid the Midas touch problem. Personal filters help too, like outlining products that contain your allergen in red. Items could also tick themselves off the list when they go into the cart, a nice use of implicit interaction.

Risks. Supermarkets regularly rearrange their shelves, so Astor’s map would quickly go out of date unless the store keeps it updated. Some stores may not even want to: a customer who walks straight to each item sees fewer products along the way. Shoppers also rarely follow a plan. They browse, change their minds and pick things up on impulse, so a rigid route would fight against how people actually shop. There is also the question of who owns the app. If the supermarket runs Astor, the “shortest path” could quietly lead you past the promotions, which is another mixed reality dark pattern.

Choosing an Idea

I compared these ideas against four criteria:

1. Enhancing an ability: does it give people an ability they don’t have?
2. Real need: does it solve a real problem?
3. Risks: can the privacy and safety risks be managed?
4. Testability: can I prototype it quickly and test it with one person?

MeetMe was the first to go. It needs a crowd to test properly, and its privacy risks are built into the core of the idea rather than being a side effect.

Astor is easy to prototype and test, but supermarkets already have signs, staff and phones. It makes shopping more convenient rather than giving people a new ability.

Suber fits the theme best. Following a conversation you would otherwise miss is a real enhancement of a human ability, and the need is very real for Deaf and hard-of-hearing people. It is also easy to prototype, since a person can simply type out the subtitles by hand. Suber is the idea I will prototype in the next post.

Conclusion

Ideation is about finding the right design before getting the design right. Sketching three ideas forced me to look at each one from the user’s point of view, and the problems showed up long before any development. MeetMe sounded fun at first, but its biggest risks only became clear once I sketched it out.

The same questions kept coming up across all three ideas: where to place information so it helps rather than distracts, how to avoid overload, and who else is affected. In AR, design decisions become more important because the interface sits in the real world and affects people in a very real way.

References

1. Bill Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann.
2. Paul Laseau. 2000. Graphic Thinking for Architects and Designers (3rd ed.). Wiley.