Transcription Tools & Qualitative Data in UXR
A Blog Post for Bentley User Experience Center
A Review of Otter.AI for Interviews
In the world of user experience research (UXR), we allocate and expend significant resources to conduct qualitative data analysis. However, sometimes a project requires greater efforts than initially scoped — perhaps a client introduces a new hurdle, or maybe your team of many is reduced to a team of one — and your resources are spread thin. How do we compensate? By introducing efficiency through transcription tools to make the most of existing resources.
In qualitative UXR, transcription services can significantly reduce the workload on a team. Instead of spending hours transcribing qualitative data, the team can outsource transcription to a service. This leaves a team with more time to evaluate the content and provide insights.
Otter is one online transcription service that converts audio files — simply upload digital files to Otter’s online portal — into relatively legible and searchable text files. In a recent study, we evaluated various transcription services as part of our UXR process including Otter.ai.
Here is a breakdown of our process:
For each session, our team took notes on significant terms, phrases, and concepts and included timestamps to refer back to minutes we missed in the moment. In this case, we conducted note-taking as we regularly would and relied on Otter to complement our existing process. While we could download the transcripts to Word documents, for our research, we primarily used the web-based platform to access Otter’s features. Reviewing the transcripts in Otter’s platform, we filled in any gaps in our notes by referring to timestamps of the text making it easy to verify outstanding questions.
For your own experience with the interface, check out our sample conversation on Otter’s platform.
WHAT WORKED?
We review some of our observations of this tool below:
Transcription Content. According to TechRadar and other well-established internet reviewers, Otter offers between 85% and 95% accuracy in transcriptions. (Blechynden, 2020) (Whittaker, 2018) We experienced similar success, achieving reliable transcripts. While these were not accurate enough to share sight unseen with the client, they allowed us to quickly verify data saving us hours of manual transcription.
Legibility Features. As far as readability and text features, Otter optimizes text display for natural legibility as you can see in the following image — ragged right text, reasonable font type and size, consistency in word spacing, and no hyphenation. And Otter clearly indicates breaks in speakers, allowing researchers to clearly see who said what. Though seemingly simple, this focused our attention on the content instead of the text itself.
Search & Query. To help readers search and query text, Otter provides searchable keywords at the top of the transcription page. We found that we could quickly narrow down our area of search based on proximity to keywords that were identified by Otter’s AI — getting to rich and relevant content faster.
Modality Flexibility. Otter offers researchers two modalities of data to review—the initial audio recording and the text transcription. And for user ease, the audio file and text synced to each other so that a researcher can seamlessly navigate between either modality. Especially after hours of reading text on a screen, when we found ourselves visually fatigued, we switched to listening to the content in an audio format. This flexible and compatible data experience helped us maintain productivity.
AREAS OF IMPROVEMENT
While we are likely to continue to use Otter as a research tool, there are a few opportunities for this product:
Platform Integration. At this point in time, Otter only works with Zoom. Moving forward, they could integrate transcription technology with numerous platforms in our new digital world. While we often use zoom recordings for our interviews, sometimes we switch between platforms based on the interviewee or recruiter.
Transcript Interface. Otter teased us with comprehensive searchable terms fixed to the header of the document as you can see in the image below. However, as we scrolled we lost access to these keywords. We would prefer to have these keywords float with us as we navigated the transcript.
Transcript Collection Analysis. Otter only provides significant opportunities for analysis in individual transcripts but does not offer any features that reflect patterns across a collection of transcripts.
Voice Identification. Otter does not identify speakers across multiple recordings. Especially in the context of our UXR interviews, it would be incredibly helpful if Otter identified recurring interviewer voices across several transcripts — in our small team, the voices of our UXR team were fairly consistent. Otter may consider training the AI to recognize voices across a collection of recordings.
Keyword Metrics. Otter offers keywords unique to each transcript. However, we often found ourselves asking “I wonder if these keywords are the same in our other interviews?” Otter could indicate a count for each keyword occurrence, giving researchers quantitative metrics around qualitative data to see patterns in common words. Further, while Otter’s AI offers fairly comprehensive lists of keywords for a transcript, researchers may like an option to add their own keywords as they have invaluable human insight into key content, words, and phrases.
Text Comparisons. Otter’s interface does not provide an easy way for researchers to review and cross-reference multiple texts at once. They could incorporate an option to compare these transcripts simultaneously. Further, since many of our interviews included the same questions, Otter could identify and indicate these similar questions across a set of transcriptions.
CONCLUSION
So in the case of our research study, did our use of this service really justify our investment in Otter as a qualitative tool? We would say yes, the efficiencies that Otter introduced to our project far outweighed the costs. Otter offers a $20 per month business subscription, which is less expensive than other services that typically charge per minute of audio. (Chen, 2020) And for those still skeptical, it offers a free trial period for both individuals and businesses.
Incorporating transcriptions, generated by AI, offers the world of UXR new ways to record, read, and analyze verbal qualitative data. As a researcher, Otter offered our team new ways to capture verbal data with relatively little extra effort. Overall, it complimented our own notes and we would recommend it to other researchers.
Sources
Blechynden, D. (2020, March 31). Otter review. TechRadar. https://www.techradar.com/reviews/otter-review.
Chen, C. (2020, March 17). Transform your UX workflow with Otter.ai. Otter.ai. https://blog.otter.ai/featured-ux-designer-heather-applegate/.
Whittaker, Z. (2018, March 6). Otter, a popular new transcription app, raises privacy red flags. ZDNet. https://www.zdnet.com/article/otter-popular-new-transcription-app-raises-privacy-red-flags/.
