Dedicated to innovation and authentic experiences for visitors, the Whitney’s current Audio Guide features audio content sourced directly from artists and diverse voices described by one participant as, “more like a conversation, than the dryness of a normal lecture [that] you hear in Audio Guides.” The Museum created a new prototype of this popular Audio Guide, and the usability study contained here observed participants’ behaviors in using it to complete relevant tasks. By exploring the prototype’s understandability, the study resulted in findings and recommendations to further enrich visitor experiences with this digital product that helps to set the Whitney apart from other museums.
A major intersection in the Whitney experience is between Audio Guide content and the galleries as physically experienced. The Whitney’s Audio Guide is a stand out product, providing context for artworks through recorded primary sources (most often the artists themselves). Listeners are given a chance to gain deeper understanding and connection to the many artists’ visions and personalities. A prototype for an updated Audio Guide is in development with the goal of connecting visitors to this rich content as best as possible.
To support the development of this prototype, the user test described herein was designed and conducted. Participating Whitney visitors were the first to test and use the prototype, and their approaches to resolving specific tasks with it were observed, recorded, and analyzed. This report presents recommendations to enhance the visitor experience of the Audio Guide based on direct feedback from this group of participants.
The purpose of this user test is to support the development of the Audio Guide prototype. The main goal to examine the performance and several features of the Audio Guide was made possible by observing, recording, and analyzing the approaches that participants took in performing the Task Scenarios.
Once the participants completed their tasks and provided feedback, the information was analyzed to rate instances when participants relied on prior knowledge or more time to perform actions. These instances were accumulated by reviewing the observations, recorded conversations, surveys and participant feedback. This user test involved several methods and analysis techniques; they include: pilot testing, random user sampling, task analysis, think-aloud protocol, probing questions, rating scale.
A pilot test was performed to determine how effectively the tasks and questions in the study revealed visitor behaviors and preferences regarding the understandability of the Audio Guide prototype. The results of the pilot test informed the redrafting and refinement of the study into a final version for implementation.
The study was implemented on site at the Whitney by two teams of two researchers each. Each pair consisted of a Tester and Moderator. A random sample of eight visitors were selected to participate. Participants had varying levels of familiarity with the Whitney’s current Audio Guide system — five of the eight were observed with the current Audio Guide at the time of intercept. Upon granting written consent to the study conditions, participants were given the Audio Guide prototype on an iPhone device.
Eight visitors were chosen to participate in the user test of the Audio Guide prototype. The test took place on the seventh floor of the Whitney Museum. One team consisting of a Tester and Moderator performed the user test on 4/19/2018 with four participants. The following day, the second team, also consisting of a Tester and Moderator duo, performed the user test with four participants. After participants signed a consent form (see Appendix A), Testers followed the script (see Appendix B) to properly guide participants as they performed the tasks on an iPhone mobile device with iOS 11.3.1 from Apple Inc. Moderators used a mobile device to record audio feedback, noted important points during scenarios, and observed the interactions that the participants followed to complete tasks. Participants completed a post-test questionnaire created with Google Forms.
They were then presented with the following Task Scenarios:
Using the Audio Guide, find any information you can about Gorky’s
The Artist and His Mother. Go to the painting and listen to the audio provided (see Figure 3).
Using the Audio Guide, find an exhibition that you can view today
(see Figure 4).
Task Scenario 1: Using the Audio Guide or your mobile device: Find any information you can about Gorky’s The Artist and His Mother. Go to the painting and listen to the audio provided.
Task Scenario 2: Find an exhibition that you can view today.
Participants were encouraged by Moderators to explain aloud their decisions and thoughts throughout task resolution. Specifically, the two common moderating techniques used are referred to as Concurrent Think Aloud (CTA) and Concurrent Probing (CP) (Nielsen, 1993). Once it was determined through visual observation that participants had completed a Task Scenario, they were prompted to both judge whether they had accomplished the task and to rate the difficulty experienced in completing it (see Figure 4).
Participants were assured that their feedback is crucial, regardless of whether the tasks were successfully completed. Following the Task Scenarios, participants were asked a series of reflective questions to gather more critical feedback (see Appendix D and Appendix E). Demographic questions (see Appendix F) were an optional follow-up to the reflective questions, and concluded the study’s data collection. Figures 2 and 3 are detailed, step-by-step descriptions of the Task Scenarios (see Appendix C).
The post-test questionnaire provided the information to survey the participants’ profiles. Participants were asked about their art and museum knowledge, how they gather information about the museum events, their nationalities, device preferences and their past visits to the Whitney and other museums. The analysis of this survey is detailed in the Figures 5–11.
Participants Profile: the number of visits participants have made to museums in the past year.
Participants profile: how participants find information about the current exhibitions on display at the Whitney Museum.
Participants profile: participants’ self-assessed knowledge of the Whitney Museum.
Participants profile: participants’ self-assessed knowledge of the Whitney Museum.
While it may seem that user tests are limitless in their benefits, researchers should be aware of certain tendencies that may cause errors in collecting critical information and making the right observations. To avoid error on the part of the researchers, drafts of the Task Scenarios were tested prior to the actual test. These test runs and edits also alleviated the possibility of incorrectly screening or inadvertently assisting participants.
Because of existing, inevitable social pressure, some responses to our questions may not be true. Mostly this is due to participants aligning their answers with what they perceive to be more popularly accepted by society, regardless of the truth. There are times when participants may not feel at ease with or confide in the reliability of the Testers or Moderators. To reduce the significance of these limitations, participants received reassurance that all materials are confidential and that participants were not tested on their skills. Participants were also allowed to skip questions they felt uncomfortable answering and even end the test at their discretion.
Participants revealed a high level of interest in the Audio Guide’s content. Some navigation links were expected to perform differently than their actual actions. Participants mainly relied on matching artwork thumbnails with art in the gallery to relate the correct artwork stop number. They also relied on their memory to determine the current exhibitions.
Those findings and information led to this proposal of recommendations in support the participants and their interaction with the Audio Guide:
Recommendation 1: Increase legibility of artists’ names and the titles of artworks [visual design and information organization] (see page 19)
Recommendation 2: Refine shapes used in navigation and increase contrast to distinguish content [visual design and organization of navigation links] (see page 21)
Recommendation 3: Incorporate filter by exhibition, filter by floor, and search by stop number functions [content structure and information organization] (see page 24)
The Whitney’s Audio Guide is another symbol of their excellent efforts to promote their culture, art content and critical thinking. Innovation has always been a hallmark of The Whitney Museum of American Art always, since its beginnings. It is dedicated to showcasing American art and provides a deeper understanding of artists’ visions. This is accomplished by offering an Audio Guide that offers a deeper connection between the visitors and the artworks.
Through the usability test process, we realized that the rigor and scope of the set test tasks may have created a lead to the users’ completion rates and ways of thinking. It is imperative that some of the findings and recommendations presented in this report be considered along with additional research methods. Further analysis is required by conducting different user-testing evaluations with the goal to deliver a seamless, digital experience. Due to the limitations of time and testing scope, and some of the general, technological and information bias that is sometimes encountered in user experience testing, further analysis is required by conducting different user-testing evaluations with the goal to deliver a seamless, digital experience.
Designing a reasonable task by scientific standards is the basis of the user experience test. Usability testing provides methodologies that made it suitable to recruit and test participants on-site. The entirety of the research process involved the team developing a comprehensive usability test for the Audio Guide, moderating the test, gathering feedback, finding and making comparisons in the test results, and complementing it with recommendations for the Whitney Museum. The solutions are only a few based on the visual design choices and the introduction of special functions to the Audio Guide. These recommendations may help to further support the participants’ desire to find specific content and search for particular artworks.