ROC COVID

Project Date: May 2020 - January 2021
*Note that the ROC COVID health screenings were launched in April 2020 and ended in November 2020.

My Role: UX Working Group Lead, user research, UX design, content strategy, copywriting, survey design and analysis, A/B testing, email design and marketing

Working Group: 

• Members represented the University of Rochester, Common Ground Health, Rochester Regional Health and included public health experts, physicians, digital health experts, project managers, data scientists, web developers, and visual designers

Overview

Background

In March 2020 a COVID-19 screening tool, ROC COVID, was launched as a collaborative effort with the city of Rochester, University of Rochester, Rochester Regional Health, and Common Ground Health, a local non-profit dedicated to improving community health.

Individuals across the 14 counties of the Finger Lakes region volunteered to complete daily online health screenings in chatbot format. Each morning participants received an email reminder with a personal link to the day’s screening. As envisioned, the aim was to obtain data for use in identifying patterns at the zip code level to help health care and community leaders to: (1) track potential coronavirus hot spots and (2) identify potential at-risk populations. By the fall of 2020, some 8 months after the launch, 55,000 individuals had enrolled in ROC COVID, and well over 1.3 million screenings had been completed. 

After the launch, I was tasked with facilitating a new UX Working Group, adding to the eight original groups which supported the ROC COVID initiative. Members of the UX Working Group were experts in public health, information science, visual design, user experience, data science, computer science, and communications. The other Working Groups included: (1) Oversight, (2) Communications, (3) Community Engagement, (4) Participation & Recruitment, (5) Dashboard, (6) Symptom Surveillance, (7) Clinical, and (8) Validation.

Note: As vaccines became available, ROC COVID priorities shifted and resources were reallocated to focus on vaccine outreach and deployment. As a result, some Working Groups, including UX, were disbanded.

Goal

The goal of the UX Working Group was to sustain and increase the numbers of daily screenings while expanding the reach of the coronavirus chatbot by managing and improving the design, content, and technical components. As noted above, participation in daily screenings was relatively high after the launch but began to decline over time, down to an average of 11-12,000 completions per day. The UX Working Group recognized that we were confronting both COVID-19 and chatbot fatigue. Many participants also may have felt that continued screenings were unnecessary given declining infection rates in the region. Moreover, participation rates were highest in the suburbs but much less so in lower-income Rochester zip codes, rural counties, and among young adults – the populations most impacted by COVID-19 and targeted for our outreach efforts.

We measured the success of our interventions aimed at increasing chatbot engagement by assessing:

User experience initiativeS

Over the course of 8 months, we modified, developed, tested, and deployed a number of UX interventions to improve the chatbot experience with the aim of sustaining ongoing engagement with the daily screenings. Key initiatives are highlighted below.

Daily Reminder Emails

At the time of registration, ROC COVID participants chose to receive a daily reminder either via text or email, which included a unique link to the screening. Our first intervention involved enhancements to the email reminders, a decision based on data which showed that 83.7% of participants utilized the daily email notification.

Timing
Our data also showed that many participants completed screenings around 6 am. Based on this information, we changed the timing of the email to 5:40 am instead of the original 1 am.  We also wanted to avoid the possibility of reminders being buried among emails that arrived during the night and early morning. Our rationale to optimize the timing of screenings paid off with an increase in the number of completed screenings.

Email Content & Subject Lines
The UX Working Group also decided to reframe the email message to leverage social influence – guiding behavior by reference to the behaviors of others. I created new email content based on feedback and insights from other Working Group members who had public health communication expertise. In the email versions, I emphasized the importance of individual daily participation in contributing to the well-being and safety of the community as a whole - even if participants did not have COVID-19 symptoms to report. 

To increase email open rates, more engaging subject lines, emphasizing why individual participation mattered, also were created to prompt action.  Within the UX Working Group, we discussed using “ROC COVID” as the first words in the subject line and the trade-offs between a short subject line without an action directive vs. a long subject line with an action directive - which had the potential to be cut off in the inbox.  In addition, I conducted research on best practices, consulted with the Communications Working Group, and leveraged the language used in ROC COVID social media ads, which were performing well. The aim was to re-engage those who no longer opened the emails to complete daily screenings. 

Visual Design & Layout
We also considered ways to enhance the visual appeal of the email design.  Our aim was to contribute to a consistent and cohesive visual identity for the ROC COVID experience by coordinating the email design with other ROC COVID assets, including the website and print marketing, for example.  I worked through multiple iterations with a former visual/graphic design colleague to create the two versions of the email layout presented below. Feedback from the UX Working Group and ROC COVID Leadership team also informed the designs. 

A/B Testing
To optimize the potential impact of the daily email on increased engagement and in support of data-driven decisions, we separately A/B tested each version of the email and subject line. The Lead Computer Scientist on the UX Working Group handled the deployment/running of all A/B tests. Note: the Lead Computer Scientist for the project handled the deployment of all the A/B tests.

Email Design: The aim of the A/B test was to compare the performance of each redesigned reminder email (Version A and Version B) against the original. All enrolled participants received either email Version A, email Version B, or the original email. Every third participant was grouped together (Example: ID 1,4,7,10 received Version A; ID 2,5,8,11 received Version B; ID 3,6,9,12 received the original email). All emails were sent out at 5:40 am. The test ran for one week. The UX Working Group then reconvened after the week and decided to run the test for another week. A Data Scientist on the UX Working Group designed this approach to the A/B testing.

For reference, prior to the A/B test ~13,000 out of ~18,000 participants (~75%) opened their daily reminder emails and of those ~80% completed the daily health screening.

We used the three aforementioned metrics to compare the performance of Version A and Version B against the original daily reminder email.

Subject Lines: We then conducted a week-long A/B test of two new subject lines against the original subject line.  Given the fact that the majority of individuals, who opened the emails, completed the screenings, this approach was viewed as an area of opportunity. Focusing on increasing the number of participants who open the email had the potential to be more impactful in increasing the number of completed screenings. We randomized the participants as we did for the email A/B test and followed similar procedures. 

Survey

In August 2020, the UX Working Group conducted a survey to gain a more enhanced understanding of why ROC COVID participants were/were not completing the daily screening regularly. We sought feedback on individual chatbot experiences to identify aspects of the experience which worked while identifying issues and pain points, which affected daily completion. This information also served to help us gauge interest in initiatives and activities, such as a newsletter or give-a-ways, designed to encourage participant engagement and completion. Of note, this survey was the first effort to collect feedback directly from participants on their experiences with the ROC COVID chatbot – over the entire project up to this point.

I drafted the survey questions and completed multiple iterations based on feedback from project leadership and engagement with other working groups. Customizing the look and feel, I also designed and built out the instrument in SurveyMonkey. Three versions were developed based on participation rates: one for highly engaged, one for partially engaged, and one for disengaged. These groupings were defined by our data scientists based on the number of screenings completed in the previous 10 days. Survey questions were tailored to each participant group.

The survey was launched in October 2020.  I managed the deployment of the survey via SurveyMonkey to 21,231 participants in the Finger Lakes Region, all of whom originally had signed up for ROC COVID via email. We then sent follow-up emails via SurveyMonkey to participants in each of the three groups who did not complete the survey after 5 days.  With input from the Communications Working group, I tailored the messaging in these invites/emails to specific group motivations. After analyzing the data, I developed a series of data displays and visualizations, which I presented in slide deck format to various groups, including project leadership and stakeholders. An intern helped with the analysis of the open-ended questions. The tracking metrics are presented in the graphic above. As expected, the highly engaged group had the highest response rate.

Key Results
The survey data provided evidence to inform priority next steps for improvements to the chatbot. In addition, areas in need of further evaluation, were highlighted.

Health Communication INITIATIVES

I also contributed as a member of the ROC COVID Communications Working Group, which brought together public health communications experts and communications professionals from across collaborating organizations. We met biweekly to prioritize health-promoting outreach efforts.

The initial aims of this Working Group were to:

Over time the Communications Working Group transitioned to focus on additional avenues of engagement, including digital marketing. These efforts involved:

While this shift realigned the Working Group’s methods, we continued to frame communications strategies around cultural competency and health literacy. My contributions involved changes to copy for the chatbot and emails and, as the focus shifted, drafting and editing Facebook and Twitter ad copy. See below for examples.

INITIATIVES STARTED BUT NOT IMPLEMENTED

In late 2020, priorities shifted as ROC COVID project leadership pivoted resources toward COVID-19 vaccination deployment and outreach – away from daily health screenings. Up until that time, the UX Working Group had focused on ways to enhance the overall ROC COVID patient experience.  The visuals below provide overviews of selected deliverables, which were developed but not implemented, due to the change in emphasis. Each had been reviewed and approved by the Oversight Committee, whose members included hospital leadership and local as well as regional health authorities.

New Validation Process

I redesigned a more streamlined and efficient validation process for individuals enrolling in the ROC COVID program for the first time. As originally conceived, participants completed their first health screenings - and immediately after received a text or email with a link to confirm contact information provided.  Some participants did not open this confirmation text or email and thus did not validate contact information. The proposed validation process combined the two steps into one for automatic information validation upon completion of the screening. This streamlined workflow created a simplified user experience. 

All possible workflow scenarios and participant groups, who might have been impacted by the redesigned workflow, were considered. My redesign effort included multiple iterations of the user workflow and content rewriting for the chatbot and email/text. New content was reviewed multiple times with the Communications Work Group. Below are examples of several deliverables.

Unsubscribe Process

While I worked to simplify the unsubscribe process for the daily email reminder to reduce user frustration, my intent also was to provide a “safety net” to ensure that participants did not unsubscribe – unintentionally.  The original process involved “one click” on a link to unsubscribe automatically, no questions asked. Instead, I designed an additional confirmation step with clear, concise language, which was consistent with text used across all digital touchpoints in the process. The intent was to encourage participants to rethink their decision and sustain engagement.

The following issues were identified with the original unsubscribe process:

Also added to the confirmation page was an optional question to obtain feedback on why participants chose to unsubscribe. This information would be used to inform future messaging and outreach efforts.

Newsletter

Early on in the Roc Covid project, the UX Working Group proposed an email newsletter as a means of promoting ongoing daily engagement. The aim was to encourage participation by emphasizing the bigger picture – the importance of individual participation to the overall well-being of the community.

Feedback from the survey (discussed above) indicated interest in such a newsletter, particularly among “partially engaged” participants.  We learned that many in this sub-group were open to receiving information in two areas: (1) symptom-related statistics and (2) ROC COVID participation rates in the region. Based on this input, we decided to explore additional options to tailor content to ensure a newsletter, which was timely, current, and provided the most value to participants. In addition, we sought to tailor the newsletter to level of engagement by sub-group.  With these ideas in mind, I designed a newsletter layout with a visual design and content, which could be adapted appropriately.

During a brainstorming process, I held discussions with the Legal Working Group to ensure that using email for purposes of such a newsletter aligned with the privacy policies, terms and conditions relating to ROC COVID.

Focus Groups

In 2020 the Community Outreach Working Group had plans to engage with groups of participants representing populations with limited ROC COVID engagement such as youth and those living in rural areas. At the time, the UX Working Group was in discussions to leverage engagement with these proposed focus groups as well.

Within this context, I developed a series of UX-related focus group questions to:

Re-engagement Email

Based on the work described above, including feedback from the original survey, I drafted an email to the “disengaged” sub-group - participants who had not completed a screening in the past 10 days. The aim was to re-engage them by emphasizing the ease of completing the less-than-one-minute daily screening and the importance of participation to community health, especially given the upcoming flu season and re-opening of schools at that time. The email content was edited in collaboration with the Communications Working Group.

Maintaining consistency with the overall ROC COVID brand, I worked with the colors, typography, and layout to create a visually appealing and engaging visual design. Though not implemented, the email draft was well-received by ROC COVID Leadership team.

Lessons Learned

The value of ROC COVID to healthcare and community leaders depended on widespread, ongoing participation - and some 60,000 individuals completed over 7 million screenings over the active period. While the UX Working Group simplified and streamlined the symptom-reporting experience and contributed to targeted, repeated outreach efforts, regional communities hardest hit by COVID-19 - where those who likely needed the most assistance lived - continued to have the lowest participation rates. In the end, the technology was usable and useful in intent, but the data generated ultimately was not actionable in identifying and addressing disparities in communities disproportionately affected by COVID-19.

To ensure access to those most at risk of being left out, we cannot overlook the key role that community context plays in the design, development, and successful deployment of technology-enabled solutions intended to yield wider public health benefits - built from the ground up with and within the communities which they aim to serve. Thus, the ROC COVID experience reinforced the importance of prioritizing community preferences and motivations, as well as incorporating UX considerations for the individual user experience - from the beginning and throughout the design, development, and deployment processes.

Selected Press

• Local officials push survey tool to catch early signs of COVID-19 [Democrat & Chronicle]

• ROC COVID, a new online survey will help track virus in the region [Rochester First]

• Monroe Co. leaders announce new ROC COVID Symptom Tracker [WHAM ABC Rochester]

• Volunteers sought in Allegany County for ROC COVID initiative [Olean Times Herald]

• UR team develops COVID-19 screening software for URMC workers [Campus Times]