We spoke with the Lead UX Researcher at a UK-based AI-powered finance app about the challenges the business faced before partnering with Chattermill and the significant improvements in product and user experience after uncovering customer insights.
The company serves an average of 800,000 customers monthly, helping users manage their personal finances. As the business scaled, the Product and User Research Teams partnered with Chattermill to better understand the user journey, identify potential gaps, and ultimately develop a product that meets customer needs.
Today, they’re seeing huge benefits across their Product & UX:
- 50% reduction in customer complaints related to the key product
- Enhanced product offering, benefiting over 4 million users
- Earned a 4.6 App Store rating, showcasing an exceptional user experience
- 70% of users reported that the app improved their financial lives
Let’s dive right in.
Challenge
Q: What are the primary challenges and objectives for UX teams right now?
Our business is very data-driven, so we closely monitor metrics such as LTV, conversions, Google Play and App Store reviews, CSAT, and NPS at various touchpoints. When any of these metrics decline, it affects our strategic business goals. Therefore, we must truly understand the entire user journey to help Product and Engineering teams identify and address the biggest pain points.
To achieve this, we need to understand our user's needs and plan the next major milestones in our roadmap. Our task is to identify any potential issues and provide actionable insights to help teams thoroughly understand the user experience and resolve issues that impact those key metrics.
Q: What tools have you been using to understand the user journey?
We've always used product analytics platforms, but they only provided quantitative data. To fully understand the user journey, we also needed qualitative feedback.
Our biggest challenge with qualitative feedback was handling large amounts of unstructured data. We're now processing over 3,000 customer comments monthly, so manual analysis would be simply time-consuming and expensive. We needed a platform to quickly and objectively process the feedback data for a comprehensive understanding of user behavior.
Q: What about user interviews?
The main issue is scalability. We conduct many interviews to identify potential issues but not to quantify them. Even with 15-minute interviews, building a large enough sample size to determine if an issue is significant would take too much time. This approach has a high opportunity cost, consuming a lot of time that could be spent more effectively.
Solution
Q: What are the key touchpoints in the user journey where you collect feedback?
When a user joins our app and connects a bank account, we track their journey closely. We ask for an NPS score between three and ten days after signing up. We repeat this after three weeks. We're also working on collecting feedback based on specific events, like after taking out a cash advance. This event-based approach will help us tie feedback to actual user actions rather than just the time they've been using the app.
Additionally, during the cancellation process, we ask users about their main reason for leaving and follow up with emails to understand why they decided to cancel their subscription. We pull all that data into Chattermill to better understand the user journey.
Q: How do you use Chattermill to analyze user experience at different touchpoints?
We use Chattermill to analyze data from various sources, creating reports and dashboards to monitor the user experience.
First of all, Chattermill helps us identify key issues across different product areas, allowing us to prioritize and rank the most significant problems. We can track changes in feedback volume and sentiment around specific themes over time. This enables us to monitor the user experience, understand the main drivers of negative sentiment, and proactively investigate potential issues before they escalate.
Additionally, quantifying the issues helps us prioritize initiatives for our roadmap and persuade the business that these projects are worth pursuing.
We also use Chattermill to validate our hypotheses when we release new features or run A/B tests. For example, we might roll out a feature to 40 users and then analyze their responses in Chattermill compared to a larger segment. We track whether it results in fewer negative comments or more positive feedback. We also monitor whether it leads to increased LTV or improved retention. Chattermill is a key tool in that type of analysis.
Finally, we use Chattermill for user experience reviews. We identify key user journeys and if we see significant conversion drops on certain pages, we highlight user complaints related to those areas. This helps us prioritize fixes and enhance the user experience, which is just as important as launching new features. It's a valuable tool for pinpointing where we need to focus our efforts.
Q: How do you use Chattermill to inform the product roadmap?
Typically, team members reach out to me during our OKR planning sessions to find insights. We focus on various metrics related to financial performance or app usage. When people brainstorm initiatives, they often ask, 'What are users saying about this feature?' I then filter through the data in Chattermill to extract specific comments on relevant themes to share with them, which helps shape the scope of the roadmap.
Q: Is the platform easy to use?
Yes, Chattermill is user-friendly and very intuitive. If someone isn't familiar with the data, they can look at a chart that shows the biggest issues. By clicking on a specific item, they can access an AI-generated summary of user comments related to that topic. This allows anyone, regardless of their technical proficiency, to quickly get up to speed on the challenges in that product area.
Q: It sounds like you rely heavily on Chattermill's Insight Assistant.
Absolutely! Insight Assistant generates AI summaries that save me time by delivering quick insights, so I don't have to write detailed updates. Instead of sifting through 50 or 100 comments, the AI summaries pinpoint key issues, helping us quickly understand the main themes. They provide a great starting point for diving deeper into specific problems.
Results
Q: Can you share examples of surprising insights that had a big impact?
Our biggest product is cash advances, which allow users to access their money for emergency expenses until their next payday, without the need for credit checks.
Through Chattermill, we discovered that our biggest complaint was that we were offering less money in cash advances compared to competitors. By analyzing app reviews and NPS data, we identified that negative sentiment consistently increased on the topic of advance amounts. This was a significant wake-up call for our team, highlighting the need to reassess our product offerings.
Q: What product changes did you make on the back of these insights?
After analyzing user feedback in Chattermill, we realized that to provide larger cash advances, we had to adjust our fees - the charge for disbursing money immediately. By increasing this fee, we could offer bigger advances while managing the risk of non-repayment.
However, this change came with concerns. We worried that users might react negatively to the increased fees. To address this, we turned to Chattermill to track user feedback regarding both the fees and the cash advance amounts.
Ultimately, we increased the amount of cash advances while also increasing the fee. We also increased the minimum cash advance amount by 100%.
As we implemented the changes, we closely monitored user feedback. To our relief, complaints about cash advance amounts dropped by 50%, while feedback on fees remained stable. This insight revealed that users cared more about advance limits than fees. With this reassurance, we felt confident moving forward, knowing we could enhance our offerings without worrying about negative reactions to the fees.
Q: What would be the risk of not having a feedback analytics platform in this situation?
It could really hold us back. It helped us identify the key opportunities we needed to focus on. With our cash advances, we learned that users were unhappy with the amounts we offered - they were much lower than what people expected. This insight guided us to make meaningful changes.
You can operate without feedback analytics for a long time, but then you can easily miss these important issues. You might think you’re doing fine, but without understanding your customers’ needs, you risk falling behind. Over time, this can lead to stagnation and a lack of innovation, making it difficult to know what to improve. In the long run, that can put your business in a tough spot.
Q: How did these changes affect the user retention?
Retention is often a lagging metric, so it can take months to see the impact. We’ve used Chattermill as a leading indicator to gauge future retention. Once we increased advance amounts and noticed fewer negative comments about them, we inferred that this would eventually lead to better retention - and it did. Using Chattermill, we can act more quickly rather than wait three months to see if our retention improves. It’s a useful way to accelerate our decision-making and respond to user feedback more effectively.
Q: What would you say are the biggest benefits of using Chattermill?
The main benefit is the time saved by not having to read through individual comments. But Chattermill also significantly improves our efficiency in diagnosing and understanding issues. Identifying the biggest drivers of user feedback helps us prioritize effectively. We can also track trends over time, quantifying the impact of our changes.
Without it, there are things we wouldn’t even realize we were missing. It helps everyone grasp the problems we need to solve more quickly, allowing us to start addressing them sooner rather than spending a lot of time trying to understand them.
Q: If someone is considering using Chattermill, what would you tell them?
I’d highlight the strong collaboration with the Chattermill team. They recently helped us redesign our theme structure and worked closely with us to customize the system. Their hands-on approach, including experimenting with new models and retraining the data, has greatly improved our confidence in the results. This personalized approach makes Chattermill stand out compared to off-the-shelf solutions, as they remain responsive and open to feedback.
Insights from Chattermill have significantly enhanced our user experience, boosting both conversions and retention rates. This has empowered the Product and Engineering teams to make informed decisions, improving the experience for over 800,000 users.
Discover how Chattermill can help you leverage customer feedback to optimize the user journey and improve conversions – book a demo today.
