How Startups Can Use Metrics to Drive Success

Posted on Apr 4, 2011 | 69 comments


You Manage What you Measure

One of the things I discuss the most with the portfolio companies I’m involved with is that “you manage what you measure.”

It’s a very important concept for me because in a startup you are constantly under pressure and have way too many distractions. Having a set of metrics that you watch & that you feel are the key drivers of your success helps keep clarity.

And the more public you can make your goals for these key metrics the better. Make them widely available inside the company and share your most important goals with your board. Transparency of goals drives performance because it creates both a commitment and a sense of urgency.

Commitment & urgency are key drivers of success in startup businesses.

You already know it from your personal lives. The surest way to run a marathon is to tell everybody you’re going to do it (transparency). Even better is to tell them which race you’re going to run in the near future (urgency). The best yet is to raise money from them for a good cause – then you’re SURE to run it (commitment). Nobody likes to raise money then look like a loser.

I ran my first marathon in London this way in 2003 raising $3,000 for Parkinson’s disease (and finishing in under 4 hours – my publicly stated goal). FWIW my private goal was 3:45 but I missed that.

I know with the recent emphasis on measurement form Dave McClure & Eric Reis you’d think everybody is measuring. My experience has proven that even some well known companies are under-whelming in this department. Or they go in the opposite direction and spend too much time capturing & measuring data that isn’t used to make management decisions. This leads companies to stare at data but not be able to see “the forrest from the trees.”

On measurement

I was recently talking with a startup company who wanted me to try their product. They have a mobile app and I felt like it crashed too much for my liking. In our next meeting I asked them how often it crashed. Only one guy in the room knew – their tech lead.

He told me in some combinations of device / OS / network they are crashing 4 times per 100. I’m a big believer in product stability & performance before adding too many features. Once you churn a user due to stability or performance problems it can be hard to get them back.

4 times / 100 means if a customer uses your app frequently (say 10-20 times / day) then they are crashing nearly every day. That’s not acceptable.

But what is industry standard? Is it 4/1,000? 1/1,000? And given your stage of development you sure better at least know what your goal is. All applications crash and this is especially true in the nascent mobile world where dealing with device types, OS’s & networks adds one hell of a configuration management problem.

What I know for sure is that if you don’t have a stability goal stated for the company and if you don’t regularly measure how you’re doing against this goal you won’t have your resources focused on the right priorities in the company.

Most companies have some measurements, but I would argue that people often measure the wrong stuff, measure with the wrong precision (either too high-level or sometimes too detailed to draw conclusions). I see this more often than I see good practices.

The best way is to start by asking yourself at management team level: what are our company objectives and how do we best measure them? Because it can be hard to define or agree company objectives at an early stage I believe most people avoid them.

Don’t. If you change your company objectives or measurements later that’s fine. In fact, I would argue that if you’re producing charts that nobody is reading or acting on you’re probably measuring the wrong stuff.

And if you’re not meeting as a team to discuss these metrics and have a regular debate about how you’re doing and what needs to change then I can assure you that you’ll never reach your destination. You’ll have no idea when you’re off course.

You will likely have multiple sets of metrics you keep depending on the company’s stage, one’s function in the company and level. For example, I highly recommend a set of board metrics that the CEO communicates to board members at every meeting. With a set of metrics the board can keep know whether the company is tracking to its objectives.

Here are some measurements I think about. How you implement them will obviously depend on the type of company you are – there is no “one size fits all” approach but there are pretty universal measures.

1. Customer Acquisition
At the highest level you’ll obviously want to track how many customers your adding every month (and for some businesses that have hit scale this is measured on a daily basis). If you can break this down by channel that you’ve acquired them from this is obviously better.

How many adds came through organic SEO? How many through affiliate deals? How many through SEM? Do you have a customer referral program? If so, make sure you can track which leads come from this. Measuring viral adoption is obviously important.

Usually you have a catch-all bucket for “direct” or similar that often came through PR or word-of-mouth.

If you have multiple versions of your product, how many are web vs. mobile? How do the mobile customers break down by device type?

The next step after measuring the customers you’re adding is to add the “cost to acquire” by channel. This is important because it will later tell you whether you have a scalable business or not. In the early phases if you can’t acquire customers cost effectively enough you’ll need to diagnose why and how to fix it.

Make sure that you count the “true” cost to acquire customers. For example, if you have developers, content people or SEO folks working on SEO programs you’ll need to allocate their time / costs to this effort. SEO is seldom “free.”

It mind sound obvious but if you’re paying $1.50 per click on an SEM basis this is NOT your cost to acquire a customer – you need to add conversion rate. I see this mistake all the time, actually. So if you convert 12.5% of the people who click on Google paid links then your true cost to acquire is actually = $12 ($1.50 / 0.125).

Now you can two levels to get your cost-to-acquire down. You can find out how to more cost effectively buy search terms (i.e. lowering $1.50 to $1.10) and you can focus on improving conversion (i.e. increasing conversion from 12.5% to 18%). Those two things together would lower your acquisition costs nearly in half to $6.11.
Stating the obvious, but if you don’t have very clear metrics on how much you can make from a person who converts into a customer you sure better not be spending $6.11 per customer! That’s for people with very clear monetization results from customers.

Ironically, there are times where it may actually pay to INCREASE your customer acquisition costs. In a fast growing market where you have clear monetization that greatly exceeds your cost of acquisition then increasing your average acquisition costs can have two clear advantages: 1) you pick up a lot of additional customers that were falling off due to not buying enough ad inventory and 2) you make it harder for less optimized companies in the market to compete.

I suspect some of this is going on at GroupOn & LivingSocial right now. Their monetization is so sick (LA speak for good ;-)) right now that it’s hard to compete with them customers – you have to have more clever sources of customer acquisition.

I’m guessing this was also the case over the first few year’s of Zynga’s growth on Facebook. Once they knew how much money they could make with virtual goods / customer then they seemed to buy up much of the Facebook ad inventory.

2. Retention / Churn
Measuring customer acquisition is clearly not enough because not all customers stick around. This is especially true in the mobile space where apps are either free or cheap. At 99 cents they’re disposable.

Most people under estimate the challenge of winning “share of mind” the least understood concept with tech entrepreneurs. Everybody thinks if I build this cool app people will come and use it. Sure, but will they still be using it in a year? In 6 months? In 3 months?
The biggest limitation we tech consumers have is our time. How many social networks, picture sharing sites, new aggregators or blogs can we really spend time on? It has to come from somewhere. You need to win share of mind.

But there are other reasons people churn – low product quality, inability to understand the value of the product, costs, competitive products, etc.

You need to start by measuring your “churn” or attrition. I like to break this down into to buckets – immediate (think almost like a bounce rate on a website) and other churn. In the mobile world many apps are downloaded but never used or perhaps only used for one day.

This type of churn is likely different from garden-variety churn and therefore ought to be measured separately because the remedies are likely to be different. Fixing a problem with somebody who downloads your app uses it once and churns versus somebody who quits after 30 days are clearly very different resolutions.

Make sure to poll your users to find out why they’re churning. The majority of churn isn’t that your app gets deleted, just not used. If you could message to a subset of these users and ask them why they didn’t use your product you will probably learn a lot. One suggestion I give is to message them with a $5 Starbucks gift card. Many people will give you a small bit of time in exchange for a small gift

3. LTV
The other obvious measurement is the “lifetime value of a customer” or LTV. Clearly in the early stages of your company you’ll have to estimate this because you don’t know how long each customer will stick around for or how your monetization will change over time.

Many times of businesses can get away without measuring this in the earliest phases but nonetheless it’s good to have a goal. If you plan to spend any serious amount of money on customer acquisition you sure better have a handle on LTV (or estimated LTV).

4. Revenue Metrics
Revenue metrics are one of the first things I ask for from the startups in which I invest. I like to think of revenue drivers. If you’re an ad business, for example, you’ll want to measure things such as: impressions served, fill rate and eCPM (effective costs per 1,000 views).

Once you have a baseline then we can have a discussion every month about those three drivers: how are you doing at getting your impressions up, how are we doing on fill rate, and what is our eCPM? They are each independent components with different actions to improve performance.

And they are revenue drivers in that simplistically impressions x fill rate x eCPM equals revenue. At the highest level (and with a board) these are great metrics to keep focused on.
As you get more granular you’ll start to break down premium inventory vs. remnant and you’ll measure “custom buys” (sponsorships) versus standard. Once you “bucket” your revenue into different types you can have more intelligent conversations.

An example might be, for a mobile app company:

  • 35% of our revenue is coming from home page take-overs, we allow 2 / day
  • 40% of our revenue is coming from remnant banner ads served by ad networks
  • 10% of our revenue is coming from direct sales of our banner inventory
  • 15% is coming from in-app product sales (25% of these with cash, 75% with “incentivized offers.”)

Now we can have an intelligent discussion about the size & shape of your business.

  • Should we increase home page take-overs to 4x / day? Or will that ruin the user experience? Or should we be lowering it to 1x?
  • If we increase home page take-overs, can we reduce our total banner ad inventory to improve the user experience?
  • If we’re getting $1 eCPMs on banners sold through ad networks, could we focus on getting our direct fill rate up in stead where we get $15 eCPMs?
  • What would that take? How many people would we need to hire? How long would it take for us to recover their costs?

Metrics drive more intelligent conversations about your business amongst your management team, with investors and with knowledgeable advisors. No metrics = high level, more generalized advice.

5. Quality
Already stated above but know what you’re shooting for in terms of load times, crashes, known bugs, etc.

6. Salesman Metrics
I don’t want to go in depth here because it could take a whole blog post, but if you’ve got direct sales teams make sure to have performance metrics in place.

It’s obvious stuff you’ll want to measure: revenue / sales person, leads, win/loss ratios, etc.

Just be careful because nowhere is it more true that “you manage what you measure” than in sales. If you start measuring calls / day, call length, meetings / week, etc. and especially if you make the results public then you’ll notice a change in sales person behavior.

If you measure the above metrics and believe they are the right ones for your business – great. But in some businesses call volumes might incentivize your reps to get off the phone quickly, which in some businesses is the wrong strategy.
So start having the discussion with your teams and your boards what the right objectives of the company are and what are the best data to measure them. Don’t wait for others to give you the recipe – you’ll be waiting for a long time.

Happy measuring. If you have any good tips for others feel free to leave them in the comments section.

[one of the best in industry that I've seen is how Scott Painter measures his business at TrueCar. I got him talking about this on This Week in VC including a quick demo of his dashboard - I think I'll shoot a separate sequence with just a walk-through of their analytics. It's brilliant.]

  • Emily Merkle

    Forgive – this is slightly off-topic but relevant to this audience:

    President Obama’s programs to promote innovation and promotion/funding for start-ups – both public and private sector initiatives:

    Programs for Innovation:

    http://www.whitehouse.gov/innovation/strategy

    Programs for start-ups: specifically private sector:

    http://www.whitehouse.gov/issues/startup-america

  • http://www.twitter.com/rashaunps Rashaun P. Sourles

    Emily,

    I appreciate your precise questioning of my verbiage and logic.

    My short response is to reiterate that metrics (KPIs) are very important–they are the mirror to help us rationalize our choices. I only contend that they must be contextualized. I’m a masters student of strategic management and organizational behavior, so I argue that the focus should always be on context (why does this metric matter?) more than simply unpacking a metric as a yardstick and applying it to our business question.

    My response to you in more detail is below:

    1) I meant KPIs, (sorry for the lax verbiage and thanks for the heads up)

    2) Outmoded metrics refers to metrics that are not relevant because our reality (or the business environment) has changed.

    Please allow me to share an anecdote:

    My buddy works in PE in SF. I was recently visiting CA, where I grew up, to visit my father in Sacramento. Sacramento is likely losing their basketball team to Anaheim, of all places. So the Kings may ditch Sac–simple enough.

    I was conversing with my buddy about why a basketball team would move into a market like LA, that already has two basketball teams. His response: a series of metrics that I can barely recall. What struck me were his assertions about the total addressable market (TAM) in LA metro.

    My responses to him:

    Primary

    A) Why does the TAM matter in a market that’s already saturated?
    B) Is the TAM really as large as current MSA Census data would suggest? Qualifier: In a post-car future, fast approaching with each rise in gas prices, how will the fans get to Anaheim games if they live outside of the OC? There’s not much public transport in the OC or LA, either.
    C) Cultural demographics are changing in LA. LA Galaxy is hugely popular due to the emergence of a loyal soccer fan base. Does the TAM reflect current and more importantly, emerging trends?

    Weaker

    D) The Maloof brothers (owners of the Kings) argue that they need a new arena and that Sacramentans are not loyal fans. Having lived in Sacramento, I know that Sacramentans take so much pride in the Kings, it’s insane. Their mayor, Kevin Johnson was a pro basketball player! It’s the only pro team in town; It puts them on the map. In Anaheim, the Kings would be the third team, after even the Clippers (not a good place to be).
    E) Would top basketball talent join the Anaheim team with the Lakers down the road?
    F) If game attendance is low, will the business model for the team survive? Will Anaheim Kings games be popular on the web or digital cable?

    [I only use this as my example because tax dollars often subsidize our sports teams--politicians vie for the team and owners get tax breaks and special financing on new stadiums. Therefore, it's more than just a business decision, we all need to question the ROI.]

    So my point is–contextualize metrics by linking them to real human behaviors that matter. Without context, without asking the “why” or “how,” I fear that we will overvalue metrics in some cases. My assertion that we often cling to outmoded metrics is because inherent in our psyche is a deep attachment to the heuristics we’ve used in the past. Metrics often become heuristics. And we cling to them even when they aren’t relevant (which of course, doesn’t imply that the metric isn’t true).

    In a world where the pace of change is accelerating (or at least, or where our awareness of the change is amplified each day), we need to ask deeper questions about the metric itself every time we use a metric to inform our decision making.

    @Rashaunps

  • http://www.twitter.com/rashaunps Rashaun P. Sourles

    You wouldn’t believe it, Emily, but I attended the Obama 7-city roadshow on entrepreneurship this week, when they stopped in Minneapolis, where I reside. I’m available with pointed feedback on the event, if you are interested.

    http://www.minnpost.com/politicalagenda/2011/04/01/27109/obama_administration_officials_coming_to_twin_cities_to_promote_business_innovation

    Regards,

    @Rashaunps

  • http://notesfromtheninjabunny.tumblr.com/ Emily Merkle

    Interested indeed!

  • http://www.twitter.com/rashaunps Rashaun P. Sourles

    Emily,

    I’ll be brief and spare any other details for an email or LinkedIn response (link me @ Rashaun P. Sourles).

    In short, Obama has the right perspective: dialogue and engagement is always better than arrogance and antagonism.

    But this administration is about the least entrepreneurial that I’ve ever imagined. Obama evidently believes that entrepreneurs have something called innovation and that he needs to tap into that to get re-elected. Because the signs were often contradictory (not the least by having the event at Medtronic when, ostensibly, the event should have been in the loft offices of some successful local startup [we don’t have enough space, I would have volunteered my condo common room, though!)

    The problem is that one cannot be in bed with the big, old guard businesses (whose business models are to blame in large part for the economic dislocation we continue to face–can’t blame government for everything!) while simultaneously engaging entrepreneurs wholeheartedly.

    Entrepreneurs seek the path of least resistance, to transform broken business models with as few resources as possible. Tech entrepreneurs, in particular, seek to do this by illuminating, combining and curating information so that bad decisions (and decision making processes) are transparent and lead to better use of limited resources.

    So it was an interesting dynamic, to say the least. LOTS of “grumbling-under-your-breath” to your neighbor. In fact, it was that tongue-in-cheek grumbling that was actually the silver lining for spending a 1/2 day with the administration telling us entrepreneurs their limitations to spur innovation.

    @Rashaunps

  • http://notesfromtheninjabunny.tumblr.com/ Emily Merkle

    Great illustration – thanks.
    Of course – in a competently run business – KPI are by no means static.

    Your assertion “we often cling to outmoded metrics is because inherent in our psyche is a deep attachment to the heuristics we’ve used in the past. Metrics often become heuristics. And we cling to them even when they aren’t relevant (which of course, doesn’t imply that the metric isn’t true).
    - is a more relevant trait of the human psychological decision-making process in general. But we’re talking business here.

    An example:

    The Freemium Model http:

    http://bit.ly/9Sl7yU
    (borrowing from a Fred Wilson post on http://www.avc.com circa 2006 – not to mean it is outmoded at all.)

    The “freemium” model that many start-ups adopt. A KPI of this model is to acquire new users without gating content/charging for access or use – in other words – to “scale”.

    At this point an additional, supporting KPI may be: acquire x-$ in advertiser revenue to monetize/fund venture at this stage while scaling.

    Obvious examples:
    Facebook
    Twitter

    Once a company has sufficient scale (users) – as determined by the company’s business model …

    Note: the business model should be embraced as a fluid model, not just a static document, framed, notarized, and enshrined as THE Business Model”.

    ….at this point the model loosely dictates that a “premium” extension of access/content/service may be had for a fee. An enticement for formerly free access may be – added content/services, and an advertising-free environment.

    These are KPIs a smart business is constantly testing on its audience and the greater public in general, to see what “sticks”.

    KPIs are always subject to change as the business grows, external factors change, initial assumptions are decimated.

    To wit:
    “You can never have too much data.” – but its imperative that a successful business:

    1) constantly experiment in varied ways to collect data with which to make decisions on KPIs, UI tweaks, service/product offerings,etc.

    and

    2) Have the proper analytics – and the personnel to understand and make deductions from said data output.

  • http://notesfromtheninjabunny.tumblr.com/ Emily Merkle

    Interesting. You know that 1) Minnesota is not exactly a hotbed of innovation – ergo, no “Foursquare0like” lofts, replete with ping pong tables and bean bags, available for a Presidential town hall; and 2) by nature the government cannot be innovative as a matter of governing.

    I disagree; an innovator/entrepreneur can indeed be “in bed” ( I hate that colloquialism) with “old-guard” business” and innovators/start-ups/entrepreneurs – who do you think funds these ventures??

    Read the links I posted. They fully flesh out the Admin’s aspirations for promoting innovation and start-ups/entrepreneurial ventures in this country – and they are governmental, private sector, and collaborative initiatives.

    They are an effort to help this country transfer from a manufacturing-based economy (that ship has sailed) to one capitalizing on our eternal best export – our ideas.

    I also disagree with your assumption that Obama is promoting innovation out of electoral interests. That is fairly cynical, and I fail to see any connection.

  • http://www.twitter.com/rashaunps Rashaun P. Sourles

    Emily,

    I really appreciate your thoughtful response. I always try to signpost “I’m a designer just playing-CEO”, so buyer beware my approach to business :)

    You come from a much more innovative industry than I just emerged from (Pharma), so I’ll be the first to admit how excited I am to be joining a world where, as you said, business models are perceived as fluid even from the people running them.

    PS: Thanks for helping me justify not going to church. Enlightenment of any kind on a Sunday morning is good enough for me!

    @Rashaunps

  • http://notesfromtheninjabunny.tumblr.com/ Emily Merkle

    Welcome. I think you’ll enjoy it here.

    re: church. I do what I can…

  • http://www.twitter.com/rashaunps Rashaun P. Sourles

    Emily,

    My response was influenced by a dinner-party discussion recently about how even the words “innovation” and “entrepreneur” are being co-opted by the establishment right now. In my opinion, innovation is not something you have–it’s something you live. It’s a willingness to change when new evidence emerges that discredits a former viewpoint or strategy. Innovation is an aversion to becoming doctrinaire and dogmatic in your thinking and decision-making.

    I worry that “innovation” and “entrepreneurship” are in vogue for the wrong reasons. I see public-private institutions emerging that are run by the least innovative people you could ever imagine.

    It’s too early to tell, though. I will indeed have a look at your links.

    PS: It was only my opinion that Obama is latching on to these concept for votes. Probably because I worked for Senator McCain in the last presidential campaign, I was surprised and flattered to get an invitation to a White House event and–cynically–assumed that it most be vote-mongering. You are right that I shouldn’t be so cynical!

    @Rashaunps

  • http://notesfromtheninjabunny.tumblr.com/ Emily Merkle

    haha – we are by nature a cynical lot. I am not. My motto is “Trust but verify” – fine words by our “It’s morning in America” Ronald Reagan.

    Book recomendation: “The Black Swan” – Nassim Taleb. Finest intellectual I (know) along with Fareed Zakaria, who I saw a couple weeks ago in person being interviews by no other than Eliot Spitzer. I met the Spitzers post-event – I’ll share my major foot-in-mouth moment in another forum.

    The steps being taken for “promoting innovation” is more, opening young minds to the concept. Innovation is not to be confused with entrepreneurship. Innovation is something you can learn about, think about, make a part of your DNA.

    What you describe as “innovation” more aptly describes the entrepreneur. I firmly believe that the entrepreneurial spirit cannot be taught – it’ s a function of personality traits and worldview.

    I am an entrepreneur; no choice there. I am high-energy, anathema to the status quo, embrace the concept of “high risk/high reward”, cannot be “managed” (actually I rail against it), abhor bureaucracy – you get the picture.

    I’ve been at this 12 years, and wouldn’t trade it fof the world.

    PS – sorry about McCain. I used to respect him – before he chose Palin and became a professional policy flip-flopper. Oh well.

  • http://twitter.com/Joshen5252 JoshuaAnsellMcKinnon

    I have been working on few start ups of my own, but pay the bills as a SEO consultant. Most of the businesses that I work with are not good a tracking metrics.

    One of the first things that I do is establish baselines for KPI’s. This can be tough for companies doing business online and offline.

    Digging deep into analytics and meeting with sales staff, accounting, and executives usually turns up some very interesting numbers.

    -Digital Director at http://Polidigital.org

  • http://twitter.com/DillonVought Dillon Vought

    Good list, great blog content.

  • http://twitter.com/DillonVought Dillon Vought

    Good list, great blog content.

  • http://essayserve.com/ essay writing service

    Useful post. Thanks.

  • Anonymous

    Mark I really enjoyed reading this post. I agree totally that “if you cant measure it you cant manage it”. The key is not to paralyze your organization with too many metrics. I also like that you feel that these metrics should be established at various levels with many people giving input. Each part of a companies operations should understand and value the others metrics. I tend to recomend that “leading indicator” or performance metrics by the people resoponsible for executing them. Your suggestion around sales metrics confirms this. If a salesperson doesnt know why they are being measured or agrees to the measurement, whats the point?

    Thanks again for the read!!

  • Anonymous
  • Demian Farnworth

    Emily and Rashaun, hope you are still listening: I really enjoyed your discussion in the comments. I think one of the better parts of this post. ;)

  • Emilymerkle

    Hi Demian,
    Thanks for the shout out! Still listening, if not jumping into the fray quite as much….busy with my 2nd startup; went live in March, we are very lean and have scaled far beyond expectations – double-edged sword as less time to interact here. Apropos – Steve Jobs passed today – the quintessential entrepreneur. I named my blog after one of his quotes: he said “I want to put a ding in the universe”. blog: http://adingintheuniverse.com. To revisit this string: specifically, Rashaun (you out there?!) ascertaining that the Administration is not entrepreneurial…the gov’t is and always has been the antithesis of efficiency, innovation, agility. However – what the gov’t IS good at is funding R&D – essentially acting as a VC firm.
    Did some research – various depts. such as DOE have funded solar/wind turbine dev/alternative fuel dev – through private enterprise. If there is anything the gov’t does well is toss money around. The public/private initiatives are alive and well.
    emm