Exploring surroundings using 3D audio
Client
Microsoft Research
Service
Interaction design
Project timeline
3 year, 4 month
(Jan 2017 – May 2020)
Wayfinding challenges for visual impaired people (VIP’s)
Creating a mental representation of one’s surroundings can be challenging, particularly because spatial awareness heavily relies on visual memory.
The lack of visual input complicates understanding the spatial relationships between different elements and places in the environment.
Furthermore, individuals with low vision or blindness cannot depend on visual cues like signs and landmarks to navigate.
Despite there are tools such as Google Maps that offer point-to-point navigation, they do not facilitate the development of a comprehensive mental map of the environment.
Mapping the user experience
The Soundscape team consisted of individuals with diverse backgrounds. The first step was to listen to the different perspectives about the project. Ideas, concerns, limitations, etc.
Under the guidance of Arturo Toledo from Pixelspace, these conversations took shape. I analyzed, and documented key insights.
We iterated multiple times to find the right balance between contextual information, the app’s sound elements, and the physical environment. We considered scenarios such as walking along a street, utilizing public transport, or being indoors during the experience.
To ensure this knowledge was preserved, I created diagrams, slides, maps, and other materials. Some of these were later printed and showcased at Microsoft’s offices.
ONE concept and audio-based augmented reality
One of the first challenges in the project was to have a good understanding the elements of the experience. The Microsoft research team came up with the ONE concept: Orientation Navigation and Exploration.
Key components of the ONE
- User’s position
- Direction
- Landmarks
- Points of reference
- Destination
- Waypoints
- Points of interest
- Surroundings
Orientation sources
- OpenStreetMaps
- Soundscape itself
- Mobility instructors
- Family and caregivers
Designing audio-based AR concepts
The design process followed a Lean UX design process. Different techniques and tools were used to transform ideas or hypothesis into concepts and features. Here are some examples.
Prototyping tools are unable to work correctly with iOS VoiceOver, even nowadays. However, I knew HTML and ARIA properties could be an option since I created websites from scratch in the past. To symplify the process, I used tumult Hype 4.
Testing Audio-based AR experience concepts
I created binaural audio to test the 3D audio experience and connected the phone to a sampler. So, when users interacted with a UI element, such as a button, with the sampler, the team manually reproduced the specific audio directly to the headphones. Additionally, a speaker was used to hear what the user was listening to through the headphones.
By doing this, users could feel both the audio experience and perceive the app using VoiceOver at the same time.
Usability testing
Additionally, usability testing were ran by the Microsoft Research team with the help of different organizations. The team also used events to showcase the project and get more insights
Launch
Microsoft Soundscape officially launched in 2019 to positive reception, receiving praise from users and reviewers. New feedback were received as well.
However, Microsoft discontinued support in 2022, transitioning the technology to an open-source project. Fortunately, the Soundscape Community, including former team members, continues to develop and improve the platform.
Takeaways
I found this project very enjoyable. I appreciate working with such dedicated people and being part of a project that had many challenges. Even though the original app is no longer available, Microsoft Soundscape paved the way for new innovations in accessible technology. Its impact on mobility and independence for visually impaired users remains a testament to the power of 3D audio in making the world more inclusive.
Check another project
MyRX: Vaccine Appointment Hub
During the COVID lockdown, myRX enabled remote scheduling of COVID tests and vaccines to reduce the spread of the virus.