Vanity metrics can be fickle, misleading, and appealing for all the wrong reasons. You’ve probably heard about them and want to know why so many data analysts curse the term under hushed breath. It’s important to know not just what vanity metrics are, but to also understand why this data can’t be used effectively. In this article, we’re going to define vanity metrics as a concept, help you identify them in your business, and provide a few examples and corresponding alternatives.
What are vanity metrics?
Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies. These metrics are exciting to point to if you want to appear to be improving, but they often aren’t actionable and aren’t related to anything you can control or repeat in a meaningful way. Vanity metrics are most often contrasted against actionable metrics, which is data that helps you make decisions and helps your business reach its goals or grow. It is important to mention: Any metric can be a vanity metric. There just happen to be some telltale signs. They are hollow metrics that look nice on the surface but hold little substance. For example, 10,000 total registered accounts might seem impressive, but that number loses face quickly if there are only 100 active monthly users. Even then, active monthly users could be another vanity metric if those users aren’t analyzed appropriately. There’s nothing inherently wrong with taking pride in your earned numbers and nothing wrong with thinking about how your business or department appears. But gaudy numbers touted as evidence of overwhelming growth only make for mildly interesting headlines and press releases—they rarely have a place within a business or project. The reason vanity metrics are so decried is that they’re overly simplistic to measure, they skip nuance and context, they are often misleading, and they don’t really help you improve in any meaningful way. It can be a great ego boost if millions of people follow your brand’s new Twitter account and seem interested in your product, but follows don’t mean much if those follows don’t translate into product sales. Basically, they are metrics that look impressive without doing any real work. They’re devoid of substance.
How to identify vanity metrics: 3 considerations
Before we discuss some common considerations and questions, we’d recommend reading up on SMART KPIs: those being metrics that are specific, measurable, assignable, realistic, and time-related. Adhering to the SMART criteria is often a tried-and-true way to avoid the most common metric pitfalls.
1. What business decision can we make with the metric?
It can be very hard to sift through data and figure out what actually helps versus what just looks good. If you’re unsure if a KPI is a vanity metric, a quick shortcut is to ask: “Can this metric lead to a course of action or inform a decision?” If the answer is “no” or “I don’t know,” then you should probably re-evaluate it. Smart, actionable metrics help you make a decision. They provide feedback and context for what your business is doing and whether or not it’s working. They help you adjust your marketing strategies to attract customers or tweak a pitch when landing sales within a specific industry. Any data you track and use should help you make your business better. Example: Think about a marketing landing page for an ebook download. Measuring pageviews doesn’t allow you to make a business decision, but measuring download rate might inspire you to test out different on-page wording, call to action buttons, or styles of form submission.
2. What can we do to intentionally reproduce the result?
Another clue is whether or not you can manage cause and effect within your data. Observing random occurrences isn’t helpful and lightning rarely ever strikes the same spot twice. Page views earned from content going viral is great, but not useful if you can’t repeat it to expand on that success. It also doesn’t help if you didn’t plan for that virality in the first place (see SMART metrics for more). Can you consistently reproduce the same result? If you can’t control the variables and repeat the process to produce a statistically similar metric, then you can’t improve that process. If you can’t improve the process, then you can’t improve that metric and it isn’t doing anything to help you. Example: Think about a major subscription magazine, like the New York Times. From time to time, the President of the United States will tweet about the magazine and the running total number of subscriptions spikes upwards. However, because the result is so tied to an erratic external factor, the magazine can’t reliably reproduce the result.
3. Is the data a real reflection of the truth?
Often, especially in some online fields, data can be manipulated or punched up with extra spend. Social media metrics, for example, are fun to look at, but the fact that you can literally pay to increase their numbers make them unreliable at best. Seriously, you can pay $50 for 90,000 followers, and create an internet-famous identity out of nothing in about two hours. This means the number of followers your brand has is a vanity metric—but you would have a great metric if you wanted to optimize for how many followers you can buy with $50. Additionally, consider whether or not your data source is consistent and reliable or if there is a 3rd party algorithm, like Google’s algorithms, or a seasonality in play that you can’t control. A month over month view can look really impressive during the winter holidays or at the end of a quarter. These factors shouldn’t immediately disqualify any data but should be closely considered and accounted for to inform your metrics. Example: Think about social media. In January of 2018, Facebook fundamentally changed its news feed algorithms. News feeds began to prioritize local news. This means that local news publishers began to appear more valuable than international news publishers, which offers a sudden conflicting message compared to different social media and distribution outlets. Vanity metrics may be hard to identify and can vary depending on the industry and individual business needs. The central question to ask yourself when considering a metric is whether or not it will help your business achieve its goals.
Create beautiful visualizations with your data.
Examples of vanity metrics
Again, remember that any metric can be a vanity metric, it all depends on the analysis. However, below are some of the most common examples of metrics prone to being vanity metrics.
Pageviews
This is one of the most popular ones. On the surface, pageviews look great. “People are visiting our site! They love us!” Pageviews can be a useful metric if you can turn it into something actionable or show how it relates to your business goals. But by itself, with no context, it just makes you feel popular. It’s more important to know who is visiting and whether they convert into sales. Alternatives: Instead of citing pageviews, focus on the quality and behavior of those views. Think about factors like bounce rate, time spent on page, sessions and pages per session, and click-through rate (CTR) on calls to action. If you are driving a lot of paid traffic to a brand website, then you might want to think about how much traffic you gain per ad dollar you spend or you might want to pay attention to direct and branded organic traffic to see if your paid efforts result in more people intentionally seeking out the brand by name. Consider the following:
- Bounce rate
- Time on page
- Sessions
- Unique users
- Pages per session
- Pages per user per month
- Organic branded search lift
- Direct traffic lift
- Cost per Lead vs. Marketing Qualified Lead vs. Sales Qualified Lead
Running total of customers
This one also offers feel-good numbers with no context. This number literally cannot go down and it won’t tell you much about how well your business is doing. At the bottom of this page, there’s a great real-world example of this metric going wrong. Alternatives: A better use of customer data is in analyzing how much customers spend per order and how often you retain customers for repeat orders versus a one-time purchase. In the world of SaaS, for example: customer subscriptions, renewals, upgrades, additional spend on training, and account expansions can all be valuable and actionable metrics. Consider the following:
- Spend per order
- Items per order
- Percentage new/returning customers
- Active subscriptions
- Renewal rate
- Upgrade rate
- Spend on training per account
- Opportunity generated by existing accounts
Running totals of purchases or downloads
Software, games, service trials, and apps can fall particularly prey to this metric. Total sales or downloads are like total customers, the number will only ever go up. But they don’t tell the whole story. It’s not helpful to just look at 1 million downloads and think you’re doing great when there are 900,000 uninstalls. Alternatives: Better than a complete running total, consider year-over-year (YoY) or month-over-month (MoM) totals, coupled with percent differences. Or, instead of counting purchases or downloads, you might want to focus on usage statistics like how much time consumers spend in an app compared to how often they open it, how many users keep the app versus uninstalling, the conversion rate from trial to purchase, subscription renewals, or growth trends like changes in organic download counts year over year. Consider the following:
- YoY total purchases and YoY percent change
- Time in software by open/by user
- Uninstall rate
- Trial conversion rate
- Renewal rate
- Upgrade rate
- Year over year organic growth
- Cost of paid growth
- Referral program growth
- Seasonal adjustments
- Telemetry and behavioral data: active use, specific function use frequency, key action frequency, etc
Social media followers
This is also a deceiving statistic and does not reflect the quality or impact of a social presence. It doesn’t help that it’s really easy to inflate these numbers just by buying followers or likes. Sure, you may get 10,000 Instagram likes or 600 new Twitter followers with some spend, but that rarely translates to sales, or even engagement. In rare cases, new brands may attract more real followers by appearing to be popular with loads of followers, but that would necessitate some sort of “follower lift as a result of total followers” metric. People can “like” anything for free; all it takes is a one-time click of a button or tap of the finger or a scripted bot. Alternatives: Having a lot of followers can make you feel like a lot of people are interested in what you have, but better metrics might focus on traffic, engagement, and share of voice among competitors. What percentage of your followers are interacting with your brand? How many are following through to your site and becoming customers? You might want to break down followers by title or even focus on having a small number of highly loyal followers who each have a huge influence. Consider the following:
- Sessions from social
- Click-through rate
- Engagement per post/per follower
- Share of voice compared to the competition
- Sentiment analysis
- Mentions and shares from VIP followers
A real-world living room example of vanity metrics
Entertainment media and hardware marketing is rife with vanity metric-fueled hype. In a great 2016 example, Microsoft publicly abandoned a common vanity metric of console hardware sales. Instead of reporting on the running total of Xbox sales, Microsoft replaced that public metric with reports on monthly active users of their Xbox Live service. Xbox’s Phil Spencer said about the change, "The nice thing about us selling consoles is your console install base will always go up. But that's not really a reflection of how healthy your ecosystem is. We focus on the monthly active user base because we know those are [people] making a conscious choice to pick our content, our games, our platform, our service. We want to gauge our success on how happy and engaged those customers are.” In this case, the vanity metric was the running total of console sales. The number would never go down, was a poor indicator of success, and was tied to one-time hardware sales rather than the more lucrative recurring service subscription sales. Pre-change, the vanity metric dictated that a sale resulting in an avid new user counted the same a sale that resulted in an Xbox collecting dust in a living room. While reporting on something as volatile as active subscriptions may seem risky, it became a much more substantial metric and more honest reflection of the health and longevity of service.