Rainy afternoon and a mundane workday. I was sitting at my desk when a mail pop up in my inbox with subject — “Request Completed — XXX Performance data from Nov 2016 to Jan 2017”
It turned out to be a custom for me to first look at data points(wherever applicable) on the problem I am trying to solve. This usually helps me understand how users are currently reacting to the particular feature or product that I am all set to explore. Other research methods to validate or gain more insights on the topic comes fairly in later stages of my design process. But no matter what despite all attempts on relating, interpreting and summarizing those huge piles of numbers to comprehend users intent and activities, I always realized a gap in my process. What was I doing wrong?
I tried looking for explanations to my doubts in the existing data analysis metrics that are used to determine how well the product or feature is doing. There were PULSE metrics — Page View, Uptime, Latency, Seven days active users, Earnings and then there were key performance indicators(KPIs) like conversions, bounce rates, active users(daily or monthly).
Amidst all these excesses of metrics, it was difficult for me to look for things that matter to me as a designer so I started asking myself a couple of questions —
How does a low conversion on a feature implies that customers are not approving it?
Is it good to say that this design is successful only if conversion rates are high?
Is conversion a good metrics to measure the overall effectiveness of this design?
What are the primary user goals of the feature — Was it how many users are converting or to help the users to move forward in their journey which could be a next page or screen of the application? … and here comes the answer.
There was a serious flaw in my mapping of these metrics to user experience goals.
What I was trying before was to measure the design based on business goals that talks about the product as a whole and could be affected by several factors like — how innovative the business idea is, what is the potential customer base, what is the sales and marketing strategy etc. On the other hand UX can be built or enhance in generic ways through design that may not directly affect these business-centric metrics. Therefore, it is not fair to directly measure things like ‘user delight or happiness’ using these metrics. An example could be a component showing best deals on an ecommerce site may be the one most clicked on the page but that doesn’t necessary qualify it to be meeting design goals.
I started backtracking my ways of applying qualitative methods where the first thing I do is to define goals of the user research or usability testing. Defining objectives help me pick right data to unravel the user behavior and to evaluate the design.
Can I use a similar approach like qualitative methods to measure user experience towards a feature using large scale analytics? May be the answer is hidden in picking the metric that resonates the most with the goal you are trying to solving for your users.
I looked over the internet and came across “HEART” Framework and “Goal-Signal-Metrics” process proposed by quantitative UX researchers at Google and it occurred to me that this was exactly something I was looking for so long. I recommend the same to every designer, who like me sometimes struggle to decipher the pile of analytical data. A link to “HEART” Framework is provided at the end of this article.
As a designer, I believe in the fact that design is measurable and so is user experience. For me, obtaining relevant data through both qualitative and quantitative approaches require certain skills, skills every designer should practice. Where user-centric data gives me the ability to support my assumptions & hypothesis on a problem or concept, it also helps me getting buy-in from other team members on recommended solutions especially while working with a cross functional team. But the trick lies in choosing the right process and right sources to gather these data.
To summarize, play by rules and these huge tons of data will automatically start falling in right places for you to work towards creating awesome experiences.
HEART framework link -There are ample of content over net on this topic but the one summarizes the best goes here — http://www.dtelepathy.com/ux-metrics/