You Manage What you Measure
One of the things I discuss the most with the portfolio companies I’m involved with is that “you manage what you measure.”
It’s a very important concept for me because in a startup you are constantly under pressure and have way too many distractions. Having a set of metrics that you watch & that you feel are the key drivers of your success helps keep clarity.
And the more public you can make your goals for these key metrics the better. Make them widely available inside the company and share your most important goals with your board. Transparency of goals drives performance because it creates both a commitment and a sense of urgency.
Commitment & urgency are key drivers of success in startup businesses.
You already know it from your personal lives. The surest way to run a marathon is to tell everybody you’re going to do it (transparency). Even better is to tell them which race you’re going to run in the near future (urgency). The best yet is to raise money from them for a good cause – then you’re SURE to run it (commitment). Nobody likes to raise money then look like a loser.
I ran my first marathon in London this way in 2003 raising $3,000 for Parkinson’s disease (and finishing in under 4 hours – my publicly stated goal). FWIW my private goal was 3:45 but I missed that.
I know with the recent emphasis on measurement form Dave McClure & Eric Reis you’d think everybody is measuring. My experience has proven that even some well known companies are under-whelming in this department. Or they go in the opposite direction and spend too much time capturing & measuring data that isn’t used to make management decisions. This leads companies to stare at data but not be able to see “the forrest from the trees.”
I was recently talking with a startup company who wanted me to try their product. They have a mobile app and I felt like it crashed too much for my liking. In our next meeting I asked them how often it crashed. Only one guy in the room knew – their tech lead.
He told me in some combinations of device / OS / network they are crashing 4 times per 100. I’m a big believer in product stability & performance before adding too many features. Once you churn a user due to stability or performance problems it can be hard to get them back.
4 times / 100 means if a customer uses your app frequently (say 10-20 times / day) then they are crashing nearly every day. That’s not acceptable.
But what is industry standard? Is it 4/1,000? 1/1,000? And given your stage of development you sure better at least know what your goal is. All applications crash and this is especially true in the nascent mobile world where dealing with device types, OS’s & networks adds one hell of a configuration management problem.
What I know for sure is that if you don’t have a stability goal stated for the company and if you don’t regularly measure how you’re doing against this goal you won’t have your resources focused on the right priorities in the company.
Most companies have some measurements, but I would argue that people often measure the wrong stuff, measure with the wrong precision (either too high-level or sometimes too detailed to draw conclusions). I see this more often than I see good practices.
The best way is to start by asking yourself at management team level: what are our company objectives and how do we best measure them? Because it can be hard to define or agree company objectives at an early stage I believe most people avoid them.
Don’t. If you change your company objectives or measurements later that’s fine. In fact, I would argue that if you’re producing charts that nobody is reading or acting on you’re probably measuring the wrong stuff.
And if you’re not meeting as a team to discuss these metrics and have a regular debate about how you’re doing and what needs to change then I can assure you that you’ll never reach your destination. You’ll have no idea when you’re off course.
You will likely have multiple sets of metrics you keep depending on the company’s stage, one’s function in the company and level. For example, I highly recommend a set of board metrics that the CEO communicates to board members at every meeting. With a set of metrics the board can keep know whether the company is tracking to its objectives.
Here are some measurements I think about. How you implement them will obviously depend on the type of company you are – there is no “one size fits all” approach but there are pretty universal measures.
1. Customer Acquisition
At the highest level you’ll obviously want to track how many customers your adding every month (and for some businesses that have hit scale this is measured on a daily basis). If you can break this down by channel that you’ve acquired them from this is obviously better.
How many adds came through organic SEO? How many through affiliate deals? How many through SEM? Do you have a customer referral program? If so, make sure you can track which leads come from this. Measuring viral adoption is obviously important.
Usually you have a catch-all bucket for “direct” or similar that often came through PR or word-of-mouth.
If you have multiple versions of your product, how many are web vs. mobile? How do the mobile customers break down by device type?
The next step after measuring the customers you’re adding is to add the “cost to acquire” by channel. This is important because it will later tell you whether you have a scalable business or not. In the early phases if you can’t acquire customers cost effectively enough you’ll need to diagnose why and how to fix it.
Make sure that you count the “true” cost to acquire customers. For example, if you have developers, content people or SEO folks working on SEO programs you’ll need to allocate their time / costs to this effort. SEO is seldom “free.”
It mind sound obvious but if you’re paying $1.50 per click on an SEM basis this is NOT your cost to acquire a customer – you need to add conversion rate. I see this mistake all the time, actually. So if you convert 12.5% of the people who click on Google paid links then your true cost to acquire is actually = $12 ($1.50 / 0.125).
Now you can two levels to get your cost-to-acquire down. You can find out how to more cost effectively buy search terms (i.e. lowering $1.50 to $1.10) and you can focus on improving conversion (i.e. increasing conversion from 12.5% to 18%). Those two things together would lower your acquisition costs nearly in half to $6.11.
Stating the obvious, but if you don’t have very clear metrics on how much you can make from a person who converts into a customer you sure better not be spending $6.11 per customer! That’s for people with very clear monetization results from customers.
Ironically, there are times where it may actually pay to INCREASE your customer acquisition costs. In a fast growing market where you have clear monetization that greatly exceeds your cost of acquisition then increasing your average acquisition costs can have two clear advantages: 1) you pick up a lot of additional customers that were falling off due to not buying enough ad inventory and 2) you make it harder for less optimized companies in the market to compete.
I suspect some of this is going on at GroupOn & LivingSocial right now. Their monetization is so sick (LA speak for good ;-)) right now that it’s hard to compete with them customers – you have to have more clever sources of customer acquisition.
I’m guessing this was also the case over the first few year’s of Zynga’s growth on Facebook. Once they knew how much money they could make with virtual goods / customer then they seemed to buy up much of the Facebook ad inventory.
2. Retention / Churn
Measuring customer acquisition is clearly not enough because not all customers stick around. This is especially true in the mobile space where apps are either free or cheap. At 99 cents they’re disposable.
Most people under estimate the challenge of winning “share of mind” the least understood concept with tech entrepreneurs. Everybody thinks if I build this cool app people will come and use it. Sure, but will they still be using it in a year? In 6 months? In 3 months?
The biggest limitation we tech consumers have is our time. How many social networks, picture sharing sites, new aggregators or blogs can we really spend time on? It has to come from somewhere. You need to win share of mind.
But there are other reasons people churn – low product quality, inability to understand the value of the product, costs, competitive products, etc.
You need to start by measuring your “churn” or attrition. I like to break this down into to buckets – immediate (think almost like a bounce rate on a website) and other churn. In the mobile world many apps are downloaded but never used or perhaps only used for one day.
This type of churn is likely different from garden-variety churn and therefore ought to be measured separately because the remedies are likely to be different. Fixing a problem with somebody who downloads your app uses it once and churns versus somebody who quits after 30 days are clearly very different resolutions.
Make sure to poll your users to find out why they’re churning. The majority of churn isn’t that your app gets deleted, just not used. If you could message to a subset of these users and ask them why they didn’t use your product you will probably learn a lot. One suggestion I give is to message them with a $5 Starbucks gift card. Many people will give you a small bit of time in exchange for a small gift
The other obvious measurement is the “lifetime value of a customer” or LTV. Clearly in the early stages of your company you’ll have to estimate this because you don’t know how long each customer will stick around for or how your monetization will change over time.
Many times of businesses can get away without measuring this in the earliest phases but nonetheless it’s good to have a goal. If you plan to spend any serious amount of money on customer acquisition you sure better have a handle on LTV (or estimated LTV).
4. Revenue Metrics
Revenue metrics are one of the first things I ask for from the startups in which I invest. I like to think of revenue drivers. If you’re an ad business, for example, you’ll want to measure things such as: impressions served, fill rate and eCPM (effective costs per 1,000 views).
Once you have a baseline then we can have a discussion every month about those three drivers: how are you doing at getting your impressions up, how are we doing on fill rate, and what is our eCPM? They are each independent components with different actions to improve performance.
And they are revenue drivers in that simplistically impressions x fill rate x eCPM equals revenue. At the highest level (and with a board) these are great metrics to keep focused on.
As you get more granular you’ll start to break down premium inventory vs. remnant and you’ll measure “custom buys” (sponsorships) versus standard. Once you “bucket” your revenue into different types you can have more intelligent conversations.
An example might be, for a mobile app company:
- 35% of our revenue is coming from home page take-overs, we allow 2 / day
- 40% of our revenue is coming from remnant banner ads served by ad networks
- 10% of our revenue is coming from direct sales of our banner inventory
- 15% is coming from in-app product sales (25% of these with cash, 75% with “incentivized offers.”)
Now we can have an intelligent discussion about the size & shape of your business.
- Should we increase home page take-overs to 4x / day? Or will that ruin the user experience? Or should we be lowering it to 1x?
- If we increase home page take-overs, can we reduce our total banner ad inventory to improve the user experience?
- If we’re getting $1 eCPMs on banners sold through ad networks, could we focus on getting our direct fill rate up in stead where we get $15 eCPMs?
- What would that take? How many people would we need to hire? How long would it take for us to recover their costs?
Metrics drive more intelligent conversations about your business amongst your management team, with investors and with knowledgeable advisors. No metrics = high level, more generalized advice.
Already stated above but know what you’re shooting for in terms of load times, crashes, known bugs, etc.
6. Salesman Metrics
I don’t want to go in depth here because it could take a whole blog post, but if you’ve got direct sales teams make sure to have performance metrics in place.
It’s obvious stuff you’ll want to measure: revenue / sales person, leads, win/loss ratios, etc.
Just be careful because nowhere is it more true that “you manage what you measure” than in sales. If you start measuring calls / day, call length, meetings / week, etc. and especially if you make the results public then you’ll notice a change in sales person behavior.
If you measure the above metrics and believe they are the right ones for your business – great. But in some businesses call volumes might incentivize your reps to get off the phone quickly, which in some businesses is the wrong strategy.
So start having the discussion with your teams and your boards what the right objectives of the company are and what are the best data to measure them. Don’t wait for others to give you the recipe – you’ll be waiting for a long time.
Happy measuring. If you have any good tips for others feel free to leave them in the comments section.
[one of the best in industry that I've seen is how Scott Painter measures his business at TrueCar. I got him talking about this on This Week in VC including a quick demo of his dashboard - I think I'll shoot a separate sequence with just a walk-through of their analytics. It's brilliant.]