All you need to know about effective UX research part 3: Analysis and reports

A problem well stated is a problem half solved.

– Charles Kettering

Meeting the UI expectations of modern users requires design teams to spend considerable time and effort carrying out thorough UX (user experience) research. Such research helps ensure that every design element is planned and developed in a way that best fits the needs of all the users. UX research is crucial to deliver an intuitive and gratifying user experience while also fostering longer and stronger relationships. 

In Part 1 – Preparing for UX Research and Part 2 – Conducting UX Research of this three-part series, we talked about the pressing need for UX research, how organizations can prepare for UX research with training, what steps they need to follow to conduct the research, and how they can deal with any challenges that come their way. 

In Part 3, we will be talking about the importance of analysis and reports to improve the outcomes and return on investment of the research effort. 

The need for analysis

Many design teams that carry out UX research consider the activity to be complete once user inputs have been received, but in reality, the research process is far from over. Earlier phases were focused on the right questions, but this phase is all about digesting the answers!

After answers to the research questions have been received, it is time to review the qualitative insights. They will build a strong picture of the required design elements to be incorporated into the product. The analysis helps design teams clearly understand the qualitative aspects of user needs (their first thoughts, feelings, and expectations) instead of quantitative aspects (the number of users who liked or disliked a particular feature). 

The analysis also allows teams to revisit their own notes, as well as those of observers to ensure the 1-to-1 UX research is comprehensive and provides specific insights. Specificity is important and this is consciously brought in at every stage. For example, by avoiding references to terms like ‘a few’, ‘some’, or ‘most’ users and talking about actual numbers. At this point, researchers try to identify things that may have been missed during the live sessions and to identify themes that let the report tell a story, instead of just a linear list of issues.

Structuring the responses and conducting analysis

One of the best and easiest ways of structuring the responses to conduct the analysis is by using Excel, although teams can also resort to using professional tools to carry out the analysis process – especially as research activities expand. These tools can easily capture verbatim responses and automatically carry out analysis afterward too, for example, tag by issue/theme.

• By writing test scripts and responses in Excel, research teams can (more) easily compare users’ answers across a single question.
• Teams can identify common themes that users mention at any point. These typically include look and feel, navigation, views on desktop vs. mobile, etc. It can also help unearth insights into how they feel about crucial steps in the workflow, such as creating an account, accessing help, and more. 
• Any response that stands out can easily be color-coded or highlighted for easy reference.
• When common themes aren’t immediately apparent, researchers can rely on powers of memory and concentration, repeated readings of the user responses, and thinking outside the box. 
• Interesting user views and verbatim comments can also be put into tools like Trello, Mural, on the wall, etc. Repeatedly looking at them can help in grouping those into fundamental underlying themes.

Things to keep in mind

Although analysis is a critical step in the UX research process and helps make report writing a lot easier, it is important to guard against overuse. 

• Try not to include every user comment in the statistics; if a considerable number of people have expressed a similar opinion, make a note of them and use it as a guide. 
• But make it a habit of inserting lots of user comments into reports. This will give a better feel for those who weren’t part of the research sessions. This also helps the research rely more directly on what users said, rather than individual interpretations. 
• Also, try not to give in to the pressure of providing clear yes or no answers; if the user response is “yes, but only just” or “no, but only just”, make a note of that. 

UX research is an essential part of Windmill’s Digital Product Strategy Services. Validate your product vision early in the design process.

The need for reporting

The time spent with users is extremely valuable. Therefore, how they feel about the product, what features they like or don’t like, or what improvements they expect must be documented properly. This brings us to the need for robust reporting. Reports help in: 

• Getting a birds-eye view of all issues identified so they can be transformed into an issues log or working document that details fixes and priorities;
• Revisiting the reasoning behind a design decision. If there’s a later challenge it’s possible to refer to the exact finding and not just someone’s interpretation of it;
• Providing an excellent and fairly user-friendly way for many new team members to understand what’s going on in a project and what fix is being attempted; 
• Reviewing other reports and improving their own techniques and knowledge sharing.

Benefits of reporting 

Reporting plays a huge role in assimilating the views of all users who’ve been a part of the UX research project. 

• Bringing all customer responses into a single document not only helps list out milestones that make the UX research process more professional; it also provides a great opportunity to show milestones to demonstrate progress. 
• User-friendly reports are also easy to share, and such write-ups help persuade any skeptics who think UX research doesn’t deliver value. 
• Moreover, reports are also a great way of demonstrating that the effort put into the research has actually led to actionable discoveries – both short term and long term. 
• In addition, user research helps teams decide which issues need immediate fixes and which can be saved for later. 
• Design teams can constantly refer to reports, even as further studies are conducted, to see how the problem and their understanding of it have evolved. 
• Research reports can also be used as a guideline for future research projects; over time, teams can build up a UX research library where the accumulated knowledge is constantly shared and distilled.

Choosing the right report style

To get the maximum impact from the analysis and reporting process, it is important for teams to choose the right report style. For instance, it may be better to choose a formal style to list identified issues, rate severity, and proposed recommendations – especially when more than one researcher is working on the same project. This helps to keep styles consistent. In other instances, it may be possible to pick an informal style of reporting such as storytelling with more variation in presentation and discussion. No matter the style chosen, it’s crucial to ensure that all the identified issues are graded for severity.

While planning for content, make sure to include: 

• Project background/context
• Research objectives
• General methodology
• General session structure (script separately or in appendix)
• Key for grading issue severity
• Statistics on the participants recruited
• Executive summary
• Summary of issues (becomes issue log)
• Findings sections structured sequentially or by theme
• Next steps
• Appendix

At Windmill

At Windmill, user researchers spend a lot of time perfecting their individual methods of reporting. It is important to update them after every session, and send them to the client if they have any additional comments. Recordings of the session also aids in the interviewer being 100% engaged, since they will not be scrambling to write down a specific quote or comment from the prospective user. It is great to go back and watch the session while reporting to observe body language, which can say a lot about what a person is experiencing. 

UX research is an indispensable part of the product development process. When done correctly, it can greatly help designers and product developers understand the specific issues of modern customers. This allows them to design product elements in a way that helps their users overcome the challenges in the best possible manner.

Let’s give the last word to Julie Zhou, Designer and Author. She says, “To find ideas, find problems. To find problems, talk to people.” It’s time to talk.