The Approach

Phase 1

To meet Phase 1 goals, we developed a hybrid research design to provide actionable insights, creating a deep understanding of product users and delivering usability insights that helped drive product development. The hybrid methodology included generative methods to uncover motivations and find broad insights about users, and usability testing to evaluate the current product. Specific tools included: 

• Amplitude for behavioral analytics captured in UX Dashboards
• SurveyMonkey for surveys and unmoderated usability tests
• Lookback.io for moderated usability tests
• Userlytics for Phase 2 feature testing (unmoderated usability tests)
• DScout for a diary study evaluating the product in context of use

For data analysis, we used Miro to collate insights from moderated and unmoderated tests. Using the newly released table feature, we developed a custom board that presented insights as cards within each stage of the user journey. The cards were linked directly to source data and to related observations, making it easy for project stakeholders to connect the dots and dive deeper into what users had to say. 

We complimented the qualitative data with data visualizations from rated scales that measured task ease, confidence, satisfaction, and other salient metrics. The mix of qualitative and quantitative data enabled the team to easily see and prioritize usability issues in a facilitated insights workshop.  From this, we developed a second Miro board that translated findings into prioritized tasks, which the team then used to roadmap UX improvements.

Phase 2

In Phase 2 we worked together in Miro and Adobe XD to define and design new UX flows and UI elements. To guide the process, we synthesized insights into a user journey/ problem statements matrix that gave us specific UX targets.

In a series of moderated usability tests in Lookback, we tested initial solution ideas at mid-fidelity to inform design decisions. The fidelity was then raised for final designs, which we tested in Userlytics- a platform that enabled us to rapidly test design iterations and collect robust data to drive development decisions. The slider below shows our process for one feature, from a wireframe in Miro that started the dialogue, to its development in Adobe Xd and the prototype screens of one flow.

Discoveries

Phase I research revealed a host of usability issues. We knew we had to fix the UX of certain features, but more importantly, we discovered something about users that would impact everything else we worked on. Users wanted a personalized interaction with the AI…they wanted Lucy (our avatar) to get to know them well enough to trust her recommendations and manage their trips. While her presence was strong in the onboarding stage of the user journey, she had little role thereafter. We had to fix that, developing customized, authentic feeling interactions that would build and maintain trust throughout the user experience. If we could create that bond, then Travel Fairy would be a truly unique assistant.

The desire for more interaction enabled a new perspective on the help feature, which user testing revealed to be a central need. While the AI recommendation engine is robust, users did not understand the logic of what they were being presented with. The lack of an intuitive interface when editing a trip was causing critical errors, which helped explain the low retention we were seeing in the analytics dashboard. While we did redesign the UI to be more intuitive, we used the desire for interaction with Lucy as an excuse for her to pop up in a chatbot and explain the interface to first time users. Once they got past that hurdle, critical errors sharply dropped. At the same time, we created a new bonding experience with Lucy shepherding the user through the interface. Win win.

Developing a new UX to resolve the edit feature led to our greatest problem, an “existential crisis” centered on choice. We found ourselves faced with the question, “What is the right balance between freedom and constraint?” The app was initially designed to limit choice, giving users a small set of options to choose from when replacing an event. But users overwhelmingly asked for more control. “How much? And how do we deliver it?” became central design questions.  The hypotheses to test were:

• Users want a large degree of control over their itinerary, and prefer to choose from a list of options
• Users don’t want to work and would prefer to choose from recommendations.

To solve this crisis, we developed  a UX  prototype test design specifically to address this question, with two different flows that represent each user orientation.

The results, overwhelmingly, came back as “both and.” Users felt that, if the AI engine was good enough, they would first like to see its recommendations to make their choice as simple as possible. Yet they rejected the idea of being forced to choose from only those options. Users wanted control over choice, and the ability to see beyond the AI engine’s recommendations. These findings contradicted the original design logic of the app, which assumed that people were lazy and did not want to spent the time making their own decisions. With travel, it seems, most people are willing and want to spend the extra effort to carefully pick among alternatives, but that lazy side of us (do it for me) is the starting point of decision making.

The end result was an experience designed to give choice, allowing the user to decide between Lucy’s recommendations and a list of nearby options they can choose from. Problem solved.