Project Date: February 2019 - July 2022
My Role: interviews, participant observation, heuristic evaluation, survey analysis, data analysis, content audit, poster preparation/write up, manuscript preparation/write up
Project Team:
Dr. Jamie Adams, GEORGE® Study Principal Investigator
Associate Professor, University of Rochester Department of Neurology
Associate Director, University of Rochester Center for Health + Technology (CHeT)
Director, Movement Disorders Fellowship, University of Rochester Department of NeurologyEllie Wagner, UX Researcher
University of Rochester Center for Health + Technology (CHeT)Emma Waddell, GEORGE® Study Research Coordinator
MD Candidate at the Warren Alpert Medical School of Brown University
Funding: New York State Department of Health Empire Clinical Research Investigator Program (ECRIP)
Overview
Background
The GEORGE® smartphone application is designed to measure objectively and continuously the symptoms of Huntington’s disease (HD), measure daily variability of symptoms, and assess how changes in standard clinical assessments correlate with daily changes in symptoms. The application is named in tribute to Dr. George Huntington.
As part of a one-month pilot study, participants completed:
During and following the one-month pilot study, I completed a mix-method usability assessment. This approach, involving generative and evaluative research, afforded a detailed, nuanced, and context-rich understanding of the issues associated with the use of and engagement with the GEORGE® application. Each methodology elicited distinct yet complementary findings, which informed the final recommendations. Pilot study participants voluntarily took part in the assessment.
Goal
The goal of this assessment was to identify issues in app usability, functionality, and interface design to inform refinements for the next GEORGE® iteration for use in clinical research. Looking ahead to the development of an improved GEORGE® app, the assessment provided valuable insights, findings, and recommendations aimed at enhancing the overall user experience, especially for those with manifest HD.
Process
Heuristic Evaluation
As the first step, I independently conducted a heuristic evaluation to examine GEORGE® against 12 recognized usability principles to identify issues and pain points. Encompassing key criteria relating to design, navigation, and user feedback, these heuristics were based on the original 10 principles as proposed by Jakob Nielsen and modified for this specific mobile health app interface (Joyce at al, 2016; Bertini et al, 2006; Arellano et al, 2012; Yáñez Gómez et al, 2014; Bertini et al, 2009).
Key Findings & Recommendations
Observations
I then conducted observations during in-person study visits at the University of Rochester Movement Disorders Clinic. The aim was to gain insights into how users interacted/engaged with the app - what worked and what did not. Twenty-one GEORGE® users - 11 female and 10 male - were observed. Included were those with manifest HD, those with prodromal HD, and those who do not have HD. Over the course of 6 months between February and July 2019, I observed 4 users in person and 17 users via video recordings of both clinic visits, which took place in 2018. A total of 87 separate videos were viewed.
Below are select images of my planning and observation notes.
Key Findings and Recommendations
Interviews
Next, I conducted 11 in-person interviews at the URMC Westfall Neurology Clinic in Rochester, NY. The interviews were conducted at appropriate break times during study visits. Interviews lasted no more than ten minutes. Immediately following each interview, I organized the interview notes in a Word document and synthesized data into a spreadsheet, which allowed for the categorization of common themes. The purpose of the interviews was to:
Gather feedback from GEORGE® users on their experiences engaging with the app as well as on their experiences as participants in the study;
Gain a more enhanced understanding of technology use, capacity for use, behaviors, expectations, and needs of people with HD.
Key Findings and Recommendations
Survey
A post-pilot survey was conducted by the GEORGE® study team at the end of each participant’s final in-clinic study visit. Questions focused on key features of the app’s usability and functionality, technology access and use, and health information-seeking behaviors. I analyzed survey responses from a user-centered perspective with a focus on accessibility and inclusion.
The survey was conducted individually with each participant. The study coordinator read the survey out loud, and if necessary, the response options as well. Participants responded orally to the questions, and the coordinator recorded the answers in REDCap. I was able to sit in on a number of these sessions. This approach served to:
ease any potential participant burden especially for those with severe HD symptoms;
simplify the input of data for the coordinator.
All 23 GEORGE® pilot study participants completed the survey – but not all participants answered all questions. Survey participants included (1) those clinically diagnosed with HD, (2) those with prodromal HD, and (3) those without HD. Feedback from all three groups will help to ensure that the next iteration of the GEORGE® app meets the needs, expectations, and preferences of all potential users for full accessibility.
Key Findings & Recommendations by Question
Limitations
There are limitations to each of the methods used, though when taken together, they provide a full picture of the usability of GEORGE®.
PublicationS & Presentations
Wagner, E., Waddell, E., Dorsey, R., and Adams, J. (2022). GEORGE® Huntington’s Disease App: Usability Assessment and Design Implications. American Neurological Association (ANA) Annual Meeting. Chicago, Illinois. Annals of Neurology, 92(Suppl. 29), S115.
*Selected as Abstract of Distinction
Wagner, E., Waddell, E., Dorsey, R., Adams, J. (2022). A User Experience Evaluation of a Huntington’s Disease App: Implications for Mobile-First Design. International Congress of Parkinson’s Disease and Movement Disorders. Virtual. Movement Disorders, 37(Suppl. 2), S514.
Waddell, E., Dinesh, K., Spear, K., Elson, M., Wagner, E., Curtis, M., Mitten, D., Tarolli, C., Sharma, G., Dorsey, R. and Adams, J. (2021). GEORGE ®: A Pilot Study of a Smartphone Application for Huntington’s Disease. Journal of Huntington Disease, 10(2), 293-301.
Wagner, E., Waddell, E., Dorsey, R., Adams, J. (2020). Usability Evaluation of a Huntington Disease App: A User-Centered Approach. Poster presented at the Brain Tech Project Summit (Virtual).