Annabel Vanessa Ruslim, Benedikta Eufrasia Kinanti Sasongko, Madeline Valerie Kwan, and Natalie Elena Widjaja
Abstract :
RE:SONA is an integrated concert experience system designed to create a more inclusive and engaging experience for deaf audiences. The project addresses the limitations faced by deaf people in experiencing the emotions, rhythms, and atmosphere of live concerts, as conventional accessibility features such as subtitles and sign language interpretation do not fully convey the sensory experience of live music. Research on hearing loss and deafness highlights communication and accessibility barriers that can affect participation in social and cultural environments, while studies on music demonstrate the potential of vibration and sensory substitution to support the perception of musical elements. The design process applies the Design Thinking approach introduced by Tim Brown (2008), consisting of five stages: Empathize, Define, Ideate, Prototype, and Test, to understand user needs and develop an experience system based on those needs. Inspired by the idea of “resonance,” RE:SONA is built around the concept of “Feel the Connection,” translating music into visual, physical, and sensory experiences through multiple interconnected touchpoints. The system consists of a mobile application, venue signage, a lightstick controller, a real-time soundwave visualizer, and a mascot. The application provides accessible information such as venue details, schedules, wayfinding, and guidance for using the lightstick controller, while the signage supports navigation throughout the concert venue and helps users independently identify important facilities and locations. The lightstick controller translates rhythm and musical genres into changing colors and vibrations, allowing users to physically and visually experience the energy and characteristics of the performance. Meanwhile, the soundwave visualizer translates music into real-time visual forms that connect with the lightstick controller, creating a synchronized visual representation of the live performance. Together, these touchpoints form an integrated design system that enables deaf audiences to experience and participate in live concerts through multiple senses. By connecting information, navigation, visual communication, and haptic interaction within one cohesive system, RE:SONA aims to provide a more accessible, informative, engaging, and inclusive concert experience for deaf audiences
Objectives :
The objective of this project is to design an integrated concert experience system that enables deaf audiences to experience, engage with, and participate in live music through visual, physical, and sensory interactions. The project aims to address the limitations of conventional concert accessibility by translating elements of music that are primarily perceived through hearing, such as rhythm, musical energy, and changes in sound, into alternative sensory experiences that can be accessed visually and physically.
The system is designed to provide deaf audiences with greater independence in navigating and understanding the concert environment, while also supporting their participation in the atmosphere and collective experience of a live performance. To achieve this, the project aims to develop a series of interconnected touchpoints, including a mobile application, venue signage, a lightstick controller, a real-time soundwave visualizer, and a supporting mascot. Each touchpoint is designed to fulfill a specific role within the overall concert experience while maintaining a consistent visual language and user experience.
The mobile application aims to provide accessible and centralized information regarding the concert, including venue facilities, event schedules, wayfinding, and instructions for using the lightstick controller. The venue signage aims to improve accessibility and navigation by helping users independently identify important locations and facilities within the concert environment. The lightstick controller aims to translate musical elements such as rhythm and genre into visual and physical feedback through changing colors and vibrations, allowing users to perceive and respond to the energy of the performance. The real-time soundwave visualizer aims to provide an additional visual representation of live music while establishing a visual connection with the lightstick controller.
Ultimately, the project aims to create a cohesive and inclusive concert experience in which deaf audiences are not only able to access essential event information, but can also actively engage with the music, performers, and surrounding audience through multiple sensory channels.
Conceptual Framework :
To develop the RE:SONA concert experience, the designer applied the Design Thinking approach (2008), consisting of Empathize, Define, Ideate, Prototype, and Test stages. The process begins by understanding the experiences and challenges faced by deaf and hard-of-hearing audiences in conventional concert environments. Information is gathered through relevant literature, user research, and observations to identify the barriers and needs associated with experiencing music in a live concert setting.
Based on the findings, the Define stage is conducted to synthesize the collected information and establish the main design problem and requirements. The design focuses on creating an inclusive concert experience that enables deaf and hard-of-hearing audiences to engage with music through visual and tactile communication.
In the Ideate stage, various solutions are explored based on the identified user needs. These include an interactive ring lightstick that translates musical elements such as beats and genres into visual
and tactile stimuli, as well as supporting elements such as an application, venue wayfinding, signage, event rundown, and mascot. These components are developed as an integrated experience rather than as individual visual outputs.
The selected concepts are then developed into prototypes during the Prototype stage. The prototypes include the RE:SONA visual identity, interactive ring lightstick, digital interface, venue communication system, and supporting event materials. The prototypes are designed to demonstrate how the proposed visual and tactile system can function throughout the concert experience.
Finally, the Test stage involves presenting the prototypes to relevant users and experts to obtain feedback regarding accessibility, clarity, usability, and contextual appropriateness. The feedback is then used to refine the design before the final RE:SONA concert experience is produced.
Process / Methodology :
The methodology and design process of RE:SONA applies the Design Thinking approach introduced by Tim Brown (2008), consisting of five stages: Empathize, Define, Ideate, Prototype, and Test.
The process begins with the Empathize stage, focusing on understanding the experiences and accessibility challenges faced by deaf audiences when attending live concerts, particularly their limitations in experiencing rhythm, emotions, and the atmosphere of performances. Based on these findings, the Define stage identifies the need for an alternative way for deaf audiences to experience music beyond auditory perception. Three keywords: Connection, Vibrant, and Global were established to define the desired character and direction of the RE:SONA experience. “Connection” represents the connection between deaf audiences across different backgrounds and ways of experiencing music. The Ideate stage explores solutions that translate musical elements into visual, physical, and sensory experiences, leading to the development of the RE:SONA concept, “Feel the Connection.” The Prototype stage develops the proposed experience system through multiple interconnected touchpoints, including a mobile application, venue signage, lightstick controller, soundwave visualizer, and mascot. The Test stage is used to evaluate and refine the system and its ability to support a more accessible and engaging concert experience.
The five touchpoints support different aspects of the concert experience while remaining connected as one system. The mobile application functions as the main information system, providing venue information, performance schedules, wayfinding, and guidance for using the lightstick controller. Venue signage supports navigation throughout the concert area, helping users independently locate important facilities and areas. The lightstick controller translates rhythm and musical genres into changing colors and physical vibrations, allowing deaf audiences to experience the energy and mood of live performances through visuals. The soundwave visualizer translates soundwaves, rhythm, and musical mood into real-time visual forms that are connected to the lightstick controller. Meanwhile, the mascot strengthens the visual identity of RE:SONA and represents the connection and sensory experience established throughout the system. By connecting these touchpoints, RE:SONA creates an integrated experience that allows deaf
audiences to participate in live concerts through multiple forms of sensory interaction.
Techniques and Materials :
AI-generated prompts must be indicated below.* (Required only if you used AI.)
-What are the keywords suitable for our concept? Something along colorful, vibrant and people’s connection for the design solution.
Can you shorten this text: The mascot design is inspired by vibration and the connection RE:SONA aims to build with people who are deaf or hard of hearing. The mascot is created and applied to serve as a strong visual identity that represents the emotions experienced by RE:SONA’s audience while remaining adaptable across various media.
The design solution for deaf people wanting to attend concerts there were a few suitable key-words, right? Well, the ones chosen are: connection, vibrant, and global. Could you put together some mind-map points to help us draw a conclusion regarding the keywords?
Hello, can you help me to translate this to English? – Hello, I want to make the mascot design so it can be one of the visual identities. The keywords that we’re setting up are connection, vibrant, and global. Can you help me to identify things or objects that have value like that?
Hello so i want to make an ui/ux app design for a people with disability of hearing to a concert and the app itself is a guide for them and what can i add to the app to make it useful for them ? I was thinking maybe there’s a map for them and what else can I add?
Result / Conclusion :
RE:SONA is an integrated experience system designed to help deaf audiences to experience the rhythm of a concert. The system translates elements of music, such as beats & rhythms that resonate into a sensoric experience through a visual and vibration. It consists of mobile application, signage, a lightstick controller, and a soundwave visualizer, with each touchpoint supporting a different part of the concert experience. The application functions as the main information system, while the lightstick control-ler and soundwave visualizer translates the vibration into a visual & can help vibrate. While the mascot, and signage are able to help through creating a recognizable visual identity and improving wayfinding through the concert venue. By implementing these features into a one whole experience, it allows deaf people to actually participate in concerts without relying on the ability to hear music.
RE:SONA demonstrates how to integrate a design system in order to help deaf audiences engage with live concerts. The system is to help translate rhythm and musical elements into a part of experiences that can be enjoyed through multiple touchpoints. By connecting those touchpoints, we are able to create a system that can help concert to a more accessible, engaging, informative, and also inclusive experience for deaf audiences.
References :
Brown, T. (2008). Change by design: How design thinking transforms organizations and inspires innovation
IDEO. (2026). Design thinking. Design Thinking by IDEO. https://designthinking.ideo.com/
Mansutti, I., Achil, I., Gastaldo, C. R., Tomé Pires, C., & Palese, A. (2023). Individuals with hearing impairment/deafness during the COVID-19 pandemic: A rapid review on communication challenges and strategies. Journal of Clinical Nursing, 32(15-16), 4454-4472. 10.1111/jocn.16572
Pryce, H., Burns-O’Connell, G., Smith, S., & Shaw, R. (2025). The Lived Experience of Hearing Loss: A Systematic Review with Narrative Synthesis. 10.1080/14992027.2025.2523902
The Ongoing Challenges of Hearing Loss: Stigma, Socio-Cultural Differences, and Accessibility Barriers (2025). https://www.researchgate.net/publication/391147047_The_Ongoing_Challenges_of_Hearing_Loss_Stigma_Socio-Cultural_Differences_and_Accessibility_Barriers
Phillips-Silver et al. (2017), Feeling the Beat: Bouncing Synchronization to Vibrotactile Music in Hearing and Early Deaf People https://pmc.ncbi.nlm.nih.gov/articles/PMC5601036/
Eagleman et al. (2021), Deciphering Sounds Through Patterns of Vibration on the Skin https://pubmed.ncbi.nlm.nih.gov/33465416/
Eagleman & Goodale (2023), The future of sensory substitution, addition, and expansion via haptic devices https://pubmed.ncbi.nlm.nih.gov/36712151/
Hopkins et al. (2016), Vibrotactile Presentation of Musical Notes to the Glabrous Skin for Adults with Normal Hearing or a Hearing Impairment. https://pubmed.ncbi.nlm.nih.gov/27191400/