How Startups Can Use Metrics to Drive Success

Posted on Apr 4, 2011 | 69 comments


You Manage What you Measure

One of the things I discuss the most with the portfolio companies I’m involved with is that “you manage what you measure.”

It’s a very important concept for me because in a startup you are constantly under pressure and have way too many distractions. Having a set of metrics that you watch & that you feel are the key drivers of your success helps keep clarity.

And the more public you can make your goals for these key metrics the better. Make them widely available inside the company and share your most important goals with your board. Transparency of goals drives performance because it creates both a commitment and a sense of urgency.

Commitment & urgency are key drivers of success in startup businesses.

You already know it from your personal lives. The surest way to run a marathon is to tell everybody you’re going to do it (transparency). Even better is to tell them which race you’re going to run in the near future (urgency). The best yet is to raise money from them for a good cause – then you’re SURE to run it (commitment). Nobody likes to raise money then look like a loser.

I ran my first marathon in London this way in 2003 raising $3,000 for Parkinson’s disease (and finishing in under 4 hours – my publicly stated goal). FWIW my private goal was 3:45 but I missed that.

I know with the recent emphasis on measurement form Dave McClure & Eric Reis you’d think everybody is measuring. My experience has proven that even some well known companies are under-whelming in this department. Or they go in the opposite direction and spend too much time capturing & measuring data that isn’t used to make management decisions. This leads companies to stare at data but not be able to see “the forrest from the trees.”

On measurement

I was recently talking with a startup company who wanted me to try their product. They have a mobile app and I felt like it crashed too much for my liking. In our next meeting I asked them how often it crashed. Only one guy in the room knew – their tech lead.

He told me in some combinations of device / OS / network they are crashing 4 times per 100. I’m a big believer in product stability & performance before adding too many features. Once you churn a user due to stability or performance problems it can be hard to get them back.

4 times / 100 means if a customer uses your app frequently (say 10-20 times / day) then they are crashing nearly every day. That’s not acceptable.

But what is industry standard? Is it 4/1,000? 1/1,000? And given your stage of development you sure better at least know what your goal is. All applications crash and this is especially true in the nascent mobile world where dealing with device types, OS’s & networks adds one hell of a configuration management problem.

What I know for sure is that if you don’t have a stability goal stated for the company and if you don’t regularly measure how you’re doing against this goal you won’t have your resources focused on the right priorities in the company.

Most companies have some measurements, but I would argue that people often measure the wrong stuff, measure with the wrong precision (either too high-level or sometimes too detailed to draw conclusions). I see this more often than I see good practices.

The best way is to start by asking yourself at management team level: what are our company objectives and how do we best measure them? Because it can be hard to define or agree company objectives at an early stage I believe most people avoid them.

Don’t. If you change your company objectives or measurements later that’s fine. In fact, I would argue that if you’re producing charts that nobody is reading or acting on you’re probably measuring the wrong stuff.

And if you’re not meeting as a team to discuss these metrics and have a regular debate about how you’re doing and what needs to change then I can assure you that you’ll never reach your destination. You’ll have no idea when you’re off course.

You will likely have multiple sets of metrics you keep depending on the company’s stage, one’s function in the company and level. For example, I highly recommend a set of board metrics that the CEO communicates to board members at every meeting. With a set of metrics the board can keep know whether the company is tracking to its objectives.

Here are some measurements I think about. How you implement them will obviously depend on the type of company you are – there is no “one size fits all” approach but there are pretty universal measures.

1. Customer Acquisition
At the highest level you’ll obviously want to track how many customers your adding every month (and for some businesses that have hit scale this is measured on a daily basis). If you can break this down by channel that you’ve acquired them from this is obviously better.

How many adds came through organic SEO? How many through affiliate deals? How many through SEM? Do you have a customer referral program? If so, make sure you can track which leads come from this. Measuring viral adoption is obviously important.

Usually you have a catch-all bucket for “direct” or similar that often came through PR or word-of-mouth.

If you have multiple versions of your product, how many are web vs. mobile? How do the mobile customers break down by device type?

The next step after measuring the customers you’re adding is to add the “cost to acquire” by channel. This is important because it will later tell you whether you have a scalable business or not. In the early phases if you can’t acquire customers cost effectively enough you’ll need to diagnose why and how to fix it.

Make sure that you count the “true” cost to acquire customers. For example, if you have developers, content people or SEO folks working on SEO programs you’ll need to allocate their time / costs to this effort. SEO is seldom “free.”

It mind sound obvious but if you’re paying $1.50 per click on an SEM basis this is NOT your cost to acquire a customer – you need to add conversion rate. I see this mistake all the time, actually. So if you convert 12.5% of the people who click on Google paid links then your true cost to acquire is actually = $12 ($1.50 / 0.125).

Now you can two levels to get your cost-to-acquire down. You can find out how to more cost effectively buy search terms (i.e. lowering $1.50 to $1.10) and you can focus on improving conversion (i.e. increasing conversion from 12.5% to 18%). Those two things together would lower your acquisition costs nearly in half to $6.11.
Stating the obvious, but if you don’t have very clear metrics on how much you can make from a person who converts into a customer you sure better not be spending $6.11 per customer! That’s for people with very clear monetization results from customers.

Ironically, there are times where it may actually pay to INCREASE your customer acquisition costs. In a fast growing market where you have clear monetization that greatly exceeds your cost of acquisition then increasing your average acquisition costs can have two clear advantages: 1) you pick up a lot of additional customers that were falling off due to not buying enough ad inventory and 2) you make it harder for less optimized companies in the market to compete.

I suspect some of this is going on at GroupOn & LivingSocial right now. Their monetization is so sick (LA speak for good ;-)) right now that it’s hard to compete with them customers – you have to have more clever sources of customer acquisition.

I’m guessing this was also the case over the first few year’s of Zynga’s growth on Facebook. Once they knew how much money they could make with virtual goods / customer then they seemed to buy up much of the Facebook ad inventory.

2. Retention / Churn
Measuring customer acquisition is clearly not enough because not all customers stick around. This is especially true in the mobile space where apps are either free or cheap. At 99 cents they’re disposable.

Most people under estimate the challenge of winning “share of mind” the least understood concept with tech entrepreneurs. Everybody thinks if I build this cool app people will come and use it. Sure, but will they still be using it in a year? In 6 months? In 3 months?
The biggest limitation we tech consumers have is our time. How many social networks, picture sharing sites, new aggregators or blogs can we really spend time on? It has to come from somewhere. You need to win share of mind.

But there are other reasons people churn – low product quality, inability to understand the value of the product, costs, competitive products, etc.

You need to start by measuring your “churn” or attrition. I like to break this down into to buckets – immediate (think almost like a bounce rate on a website) and other churn. In the mobile world many apps are downloaded but never used or perhaps only used for one day.

This type of churn is likely different from garden-variety churn and therefore ought to be measured separately because the remedies are likely to be different. Fixing a problem with somebody who downloads your app uses it once and churns versus somebody who quits after 30 days are clearly very different resolutions.

Make sure to poll your users to find out why they’re churning. The majority of churn isn’t that your app gets deleted, just not used. If you could message to a subset of these users and ask them why they didn’t use your product you will probably learn a lot. One suggestion I give is to message them with a $5 Starbucks gift card. Many people will give you a small bit of time in exchange for a small gift

3. LTV
The other obvious measurement is the “lifetime value of a customer” or LTV. Clearly in the early stages of your company you’ll have to estimate this because you don’t know how long each customer will stick around for or how your monetization will change over time.

Many times of businesses can get away without measuring this in the earliest phases but nonetheless it’s good to have a goal. If you plan to spend any serious amount of money on customer acquisition you sure better have a handle on LTV (or estimated LTV).

4. Revenue Metrics
Revenue metrics are one of the first things I ask for from the startups in which I invest. I like to think of revenue drivers. If you’re an ad business, for example, you’ll want to measure things such as: impressions served, fill rate and eCPM (effective costs per 1,000 views).

Once you have a baseline then we can have a discussion every month about those three drivers: how are you doing at getting your impressions up, how are we doing on fill rate, and what is our eCPM? They are each independent components with different actions to improve performance.

And they are revenue drivers in that simplistically impressions x fill rate x eCPM equals revenue. At the highest level (and with a board) these are great metrics to keep focused on.
As you get more granular you’ll start to break down premium inventory vs. remnant and you’ll measure “custom buys” (sponsorships) versus standard. Once you “bucket” your revenue into different types you can have more intelligent conversations.

An example might be, for a mobile app company:

  • 35% of our revenue is coming from home page take-overs, we allow 2 / day
  • 40% of our revenue is coming from remnant banner ads served by ad networks
  • 10% of our revenue is coming from direct sales of our banner inventory
  • 15% is coming from in-app product sales (25% of these with cash, 75% with “incentivized offers.”)

Now we can have an intelligent discussion about the size & shape of your business.

  • Should we increase home page take-overs to 4x / day? Or will that ruin the user experience? Or should we be lowering it to 1x?
  • If we increase home page take-overs, can we reduce our total banner ad inventory to improve the user experience?
  • If we’re getting $1 eCPMs on banners sold through ad networks, could we focus on getting our direct fill rate up in stead where we get $15 eCPMs?
  • What would that take? How many people would we need to hire? How long would it take for us to recover their costs?

Metrics drive more intelligent conversations about your business amongst your management team, with investors and with knowledgeable advisors. No metrics = high level, more generalized advice.

5. Quality
Already stated above but know what you’re shooting for in terms of load times, crashes, known bugs, etc.

6. Salesman Metrics
I don’t want to go in depth here because it could take a whole blog post, but if you’ve got direct sales teams make sure to have performance metrics in place.

It’s obvious stuff you’ll want to measure: revenue / sales person, leads, win/loss ratios, etc.

Just be careful because nowhere is it more true that “you manage what you measure” than in sales. If you start measuring calls / day, call length, meetings / week, etc. and especially if you make the results public then you’ll notice a change in sales person behavior.

If you measure the above metrics and believe they are the right ones for your business – great. But in some businesses call volumes might incentivize your reps to get off the phone quickly, which in some businesses is the wrong strategy.
So start having the discussion with your teams and your boards what the right objectives of the company are and what are the best data to measure them. Don’t wait for others to give you the recipe – you’ll be waiting for a long time.

Happy measuring. If you have any good tips for others feel free to leave them in the comments section.

[one of the best in industry that I've seen is how Scott Painter measures his business at TrueCar. I got him talking about this on This Week in VC including a quick demo of his dashboard - I think I'll shoot a separate sequence with just a walk-through of their analytics. It's brilliant.]

  • http://bradleyjoyce.com bradleyjoyce

    Everyone has gotten the message loud and clear that startups need to measure stuff… they’ve even been told what to measure…. what nobody seems to be talking about is *how to actually go about it*.

    Does Google Analytics do it? Does KISS Metrics do it? Does a startup have to roll their own measurement tools? If so, how?

  • http://twitter.com/diegomarino Diego Mariño

    Clap, clap, clap… But I’m a little biased because I’ve founded a SaaS dashboard aimed to startups and SMBs :)

    It’s nice to read this because in our customer discovery phase we found a huge number of managers who have intuition as a main driver for taking decisions instead of data :-/

    Also, companies need coaching about which metrics they should consider, apart from really recognize which are “vanity metrics” and which are “actionable metrics”.

    Finally, showing real-time data to your employees is a motivational boost: they realize you really trust them for sharing it, and also they can see that they work has a real effect outside the walls of the company.

    (Our app is http://ducksboard.com . Feel free to redact it if you consider posting it as spam)

  • http://bothsidesofthetable.com msuster

    People may have the message but I’m not sure enough of them actually do it well.

    On the site traffic – obviously there are all the usual tools like GA, Kiss Metrics, Omniture, etc. You can get a reasonable enough granular level of detail from these. Most sophisticated companies I know prefer their server logs, which they claim to be more accurate. So I suspect it’s a combo of RYO & off the shelf tools.

    On A/B testing, the sharpest companies I know seem to roll their own. Not sure why but this is my experience.

    But anyway, I’m talking more broadly than that. I’m talking about business metrics. And this starts top down by defining the success criteria for your business.

  • http://twitter.com/glehel Glen Hellman

    Excellent! I preach this to clients and have struggled to put it in a “how to” form. Thanks!

  • http://bothsidesofthetable.com msuster

    Not spam. I encourage relevant links – thank you. I think intuition is also important but has to be underpinned by data. Good luck with DucksBoard

  • Anonymous

    I think Mark hit the nail on the head when he said “most sophisticated companies I know prefer their server logs”.

    It’s pretty easy to build a simple section in your admin panel for statistics. We use Google analytics, but we also have a ton of custom data that we pull from our own server logs.

    I think the more important thing is to start small. It can be incredibly overwhelming to start trying to track every aspect of CPU to LTV, etc. We started with one channel (Google Adowords) and tracked and optimized that to a point where we’re comfortable with it (acquisition cost went from $1.50 to $0.60 over 3 months). Once we were at that point we added another channel and began tracking and comparing that.

    We’re still pre-revenue so tracking LTV isn’t possible. We do, however, calculate immediate and monthly churn numbers. We do this because we want to have the data now. This is incredibly important, even for (if not more so) pre-revenue companies. I want to be able to show our investors that we are already thinking about this data and mapping it out. We don’t have all the pieces yet, but as we progress and are able to fill assumptions with actual data, we’ll soon have the complete picture. The more we can fill in until then, the less of a crapshoot our “estimates” are.

    Again, start small. It can be incredibly overwhelming and stressful. Map out a plan and an end goal and obtain each piece of data as you progress.

  • http://about.me/bradleyjoyce bradleyjoyce

    I guess that’s sort of my point.

    No one is doing it well because no one knows how to do it… the high level stuff is the easy part. Actually figuring out how to take the idea and goal of tracking stuff and turning it into code that does it is where I see a lot of people struggling.

    It’s also many many times harder to integrate metrics tracking code after the fact than it is to develop the original product with metrics in mind.

    Startups need more help in the actual implementation of tracking metrics.

  • http://twitter.com/timbarnes10 Tim Barnes

    Mark – quick question on Salesman Metrics. Many start-ups launch in this space without a direct sales force. Maybe to start it is simply the founder or key members of the founding team doing the heavy selling. As the start-up expands and starts proving out their model, the question becomes how quick to hire new sales staff to really drive growth. You can hire seasoned sales staff (i.e. someone that can land that elephant deal), or you can hire less experienced sales staff (that may take time to grow into role and deliver solid ROI). You would obviously pay more for the heavy hitter. How do you decide what is best? What metrics (ROI) do you set to see if you are tracking appropriately once you have made the decision? In the content/online advertising, what is a good tenure for your sales staff?

  • http://twitter.com/ManAtWorkBlog Man At Work

    Great post Mark. I would also say that keeping a comprehensive list of concrete measurement is critical for staff management. All of these metrics allow you to show improvement over time - nothing drives the urgency and commitment of your team like a little confidence in the business.

    http://www.manatworkblog.com

  • http://twitter.com/ManAtWorkBlog Man At Work

    *Duplicate Post Deleted*

  • Anonymous

    Hmmm… I think I agree with some of what you say.

    Firstly, I don’t think it’s that hard to do. If you have logged the appropriate data. it’s generally very easy.

    Your last point regarding building a product from the start with metrics in mind has some merit. We started out logging EVERYTHING from time of day the users sign up to their actions when online. Everything is time stamped, logged, etc. and stats are only hindered by our own creativity in what we want to track and calculate.

    However, it’s pretty easy to start tracking these things at any time. It’s simply a matter of collecting user data and then organizing it. If you don’t track something… start! Sure, you won’t have the historical data, but the sooner you start, the sooner you’ll be able to begin building that knowledge database.

  • http://www.tmarkiewicz.com tmarkiewicz

    There are actually a few startups working to solve this problem (disclosure: my company is one of them: http://www.statsmix.com), but the biggest problem we’ve seen is the “pull in all your data and make sense of it later approach.” When you dump everything into logs or rely on any analytics service, you still have to sift through all the data to discern the key metrics.

    As Mark mentions, it’s important to figure out what the handful of really important metrics are to the business and decide how to track them. So for some metrics, it may be enough to use web analytics, social media, email marketing, etc and then have them pulled into a custom dashboard. But other metrics aren’t so easy to track and typically just get dumped into logs (if at all) to sort out later (which of course never happens).

    I definitely agree with tbiz’s comments to start small – we’ve found our customers usually start tracking just a few metrics and then eventually add more over time.

    This is why we built our API at StatsMix, so companies could easily send us any metric real-time and then roll them up into dashboards or simply share them with others.

    Different companies competing in our space are taking different approaches to getting data into their systems and displaying via dashboards.

    So if you look to outsource some of this it really depends on the approach you want to take and how much you’re willing to integrate.

  • Dave

    Your assertion that “all applications crash” is absolutely false. It is, however, a classic self-fulfilling prophecy.

  • http://twitter.com/ankur10 Ankur Jain

    Mark – Awesome blog post! You have raised very good points especially people don’t focus on the metrics at right granular level – sometimes too deep that they don’t make sense and sometimes too abstract!!

    I have seen startups spending too much time in ‘customizing’ Google analytics etc. In my opinion, the easier and faster way is to build your own metrics dashboard. This will give them flexibility of measuring whatever they want. They can build a simple dashboard in the starting (e.g. number of crashes/user) and then add new metrics as per the objectives.

    This is what we do here at Kosmix and our metrics system is one of the most sophisticated I have ever seen!

  • http://twitter.com/#!/Mr_RamV RamVaz

    Thanks for this. I am currently trying to get a better grip on my estimates of life time value and customer acquisition costs. I found this link to be a real gem for anyone else dealing with this “How to create a profitable Freemium startup (spreadsheet model included!)” by Andrew Chen. Estimating virality and horizon seem pretty tricky.

    http://andrewchenblog.com/2009/01/19/how-to-create-a-profitable-freemium-startup-spreadsheet-model-included/

  • http://twitter.com/rogercorn Roger Corn

    Great post, Mark. Even imperfect measurement is better than measuring nothing at all. Another great phrase is “what gets measured, improves.”

    To your point #1, measurement is mandatory when paying for any kind user acquisition. And sources of new users will have greatly different degrees of quality (#2 and #3).

    For example, one source might send you a user that generates $2.00/month for 1 year. And, another might generate $0.50 for 1 month.

    Most start-ups probably won’t have these kind of metrics handy.

    But, they can include “blow up” clauses in contracts. This can help them get out of bad deals where the quality of users turns out to be way below expectations.

    In my experience, this kind of clause gets triggered about 5% of the time.

  • http://twitter.com/TMattCameron Matt Cameron

    Tim, if you are starting out then I my experience has been that it is imprudent (and usually wasteful) to attempt to go elephant hunting in the early lifecyle of your business. I am a former Sales Director for Salesforce.com and was part of the early team that launched in Australasia – The idea is to get your messaging and delivery honed with easily digestible new customers and then leverage that experience to get further up the food chain.

    I can share a lot more on the subject if you want to flick me an email

  • http://twitter.com/IanGertler Ian Gertler

    Those of you who have followed Mark as long as I have realize that his gut instinct is very good, but as he says it’s important to validate that with data — especially today, when there is so much and it can really make the difference between success and failure.

    While I used to believe in the position that not everything measurable is worthwhile and not everything worthwhile is measurable, this seems to be changing before our eyes. As tbiz highlights, it’s good to embed a metrics strategy from the start in the foundation of your product or service.

    I guess we’re basically following the mantra of cut once and measure twice (or much more, as long as it doesn’t impede you from actually DOING business). This is why “Big Data” is such an incredible opportunity for businesses, governments, consumers, healthcare organizations and more. We have the capabilities to use information to improve decisions and results. This can change lives.

    Data really is becoming the new oil. We just need to be able to extract, process and leverage it to the full value to be rewarded. And with this comes countless opportunities for new start-ups to emerge and help us do it better … as we see highlighted throughout some of these comments. Good luck to all of you, since your achievements will help me do better too.

  • http://www.aaronklein.com/ Aaron Klein

    My personal opinion is that each startup is going to focus on a different mix of metrics. So others may find off-the-shelf solutions workable, but this is the approach I decided to take.

    1. Build a “log everything” mentality into your app. You can overdo this, but it’s much easier to underdo it.

    2. Budget some of your engineering time right after you launch your beta for analytics.

    3. Sit down on day two of beta (assuming you’ve had some user activity) and look through the logs. Do your expected metrics look useful?

    4. Hack together a simple analytics dashboard with your key metrics. Doesn’t have to be overly pretty, simple is good. Put “start date” and “end date” filters on the page, defaulted to last 30 days, and you can now pull 90% of what you need from your dashboard.

    5. The rest, have your team pull out with a manual query until your needs grow and you need it automated.

    6. Above all else, budget time for analytics engineering. Four hours a week goes a long way for most apps.

  • http://twitter.com/Sirachm Sirach Mendes

    Interesting article Mark
    Metrics is great tool for creating urgency and creating goals for the company to take action

    I have one confusion with LTV – there are certain ways to calculate but how does a early stage company pre-revenue with limited or no customers calculate this.

    It would be great if you could give an example of this for gaming social and mobile development companies (doing IPs and working with publisher model)

    Thanks,
    Sirach Mendes

  • http://twitter.com/timbarnes10 Tim Barnes

    Matt – thanks for the response. The feedback is helpful. Managing time and effort is critical for high growth firms, so I agree that it is very imprtant to figure out the best customer profile to sell to and what it takes to close these deals. An interesting subject.

  • http://www.gsharma.com Gaurav

    I think that is a great question. A startup in a closed alpha, public beta or growth stages are different and need different metrics and tools. When a startup is trying hard to launch that alpha, it is hard to focus on building custom A/B testing tools for those 20 alpha users. I think it is waste of time.

    Here is what I do – throwing in that GA code on the pages barely takes any time. I put it the very first day and start collecting some data. It usually covers most of the use cases that an early stage product needs to measure. As I look at the aggregated data (say once a week) it starts showing patterns. Where the traffic is coming from, what people are doing, where do they stay longer and where they don’t. My next step is to create goals/funnel setup in GA based on the patterns and other actions I’d like the users to take. If you are an e-commerce play, it is pretty simple to track all the transactions in GA.

    The reason there are not too many people who answer “how to actually go about it” is because the answer is “it depends”. Once you have outgrown Google Analytics, it is all about what you are doing and where you need to go. There are several ways of acquiring users for a startup, but you won’t know which one works best for you until you try at least most of them. Same goes with metrics and measuring tools when you have tons of data to go through, till then GA is just fine.

  • http://profiles.google.com/mvg210 Mike Gnanakone

    Does customer feedback provide a good metric? And does customer support pre-launch

    (example : 5,000 signatures of SoCal college students via Facebook in support of your product)

    influence your investment strategy in any way? Is it a dealmaker to have a lot of customer support?

  • http://twitter.com/karelvanderpoel Karel van der Poel

    Hi Mark,
    Thanks for your post. First of all a disclosure: My company Mirror42 is the SaaS performance management company. We operate http://www.kpilibrary.com with over 6.000 KPI templates and 280.000 members the largest community on the globe related to this topic. Our premium SaaS products are KPI Dashboard (www.kpidashboard.com) and KPI Benchmark. KPI Dashboard enables businesses to manage and measure the KPI’s for their business.

    Having said all this, I totally agree with you that transparency is the way to go. Once you make your performance transparent to shareholders you are committed to improve them. I know it is a scary thing to do, but great CEO’s never fear the truth. I would like to add the following tips to your article for startups:

    1. Forget Real time. Your business is not changing in real time. You should measure daily over the last 7 and 30 days and focus on identifying trends.
    2. Forget Full BI and Ad-Hoc reporting: You do not want to drown in information overload and create tons of reports. You want to set strategy, translate in KPIs and measure. Then get your head down and improve your business.
    3. Don’t jump to conclusions. Statistical proof of trends require lots of data points. Test, Test, Test. Test all area’s of your business model over and over again.
    4. Automate KPI Management. Most likely you will find your data in: your Web Analytics app, your CRM system and your billing system.

  • http://arnoldwaldstein.com awaldstein

    Great stuff Mark.

    My only two adds:

    -don’t mistake the trees for the forest and make measurements the thought rather than the data to spur thinking.

    -with social data, measurements get more interesting and more telling and less clear. Handle social measurements with care.

    Post I did this morning on the changes to SEO in a social world is connected to this thinking @ http://bt.io/GtKN

  • Anonymous

    Another great post. I have seen first hand how important it is to 1) pick your measuring tools based on KPIs rather than vice-versa and 2) focus on quality over quantity. I also wholeheartedly agree with the transparency -> commitment argument. Whenever I need extra impetus to get in action nowadays, I will tell key people what I am going to do – the ones I really don’t want to disappoint, the ones I know will hold me to account

  • Dave W Baldwin

    Thanks for A Chen link.

  • Dave W Baldwin

    Great post. Recommend everyone read the Chen link via RamVaz also.

    You know I like the ‘transparency’ and ‘Forrest Thru Trees’.

    Regarding measurement. Very important. Unfortunately, it can be set somewhat deceiving. My wife watches the ‘Housewives/Wherever’ stuff and one had the wife and nanny in one car, husband in other. He was trying to use the voice command to make a call, saying, “Dan.” The device was coming back with anything but, usually 3 syllables, begin with ‘R’ and so on. He was getting mad. Would have been as worthwhile to program ‘start at top of contact list, go down, good luck’.

    Point? Even the 98% claims are probably not as 98% as claimed…

    Putting together the ‘real’ product, delivering stickiness due to multiple features that appeal to wider customer base demands the metrics. I’d say, keep simple but have the proposed metric point to useful patterns reflecting different customer brackets down the road. Each bracket you gain data/pattern for will turn into two forward looking data/pattern groups.

  • http://twitter.com/RepTivity RepTivity

    I like #6.

  • Anonymous

    Great post Mark, as always. One key metric that you didn’t really mention is some sort of a “Virality Coefficient.” For many products, the key means of proliferation involve peer-to-peer activities. You don’t see Quora and Twitter running too many Google AdWords.

    There are many ways in which you can get your users to do the marketing dirty work for you. (1) Sharing (an article, a game score, etc.) on Facebook & Twitter. (2) “Following” or becoming “Friends” with other users (which triggers emails to those people drawing them back to the site). (3) Uploading user-generated content. (4) Commenting on company blog articles or other site components. (5) Writing 5-star app reviews on iTunes (which will improve search results). Etc.

    Optimizing those metrics is often the most important thing BEFORE focusing on CPC marketing, etc. (Why would I spend money on ads if I know that my virality coefficient is 0?) I’d love to see a future article about how to best measure and iterate on your product’s inherent spread-ability.

  • http://twitter.com/ToddZipper Todd Zipper

    Yet another amazing post by Mark Suster. I just came off of starting and selling an education lead generation business where we lived and died by the numbers. In fact, I produced a TV commercial with a girl going to school in her Pajamas (online of course) and absolutely no one said it would work. The numbers painted a very different picture and it ended up defining our business. I actually found out that my gut was frequently wrong when it came to online advertising.

    Additionally, it is imperative that marketers use cost per acquisition metrics only as it relates to lifetime value of a customer. This is a major problem in the education lead gen industry where not all customers are created equal, but the marketing and admissions departments only care about hitting certain goals and don’t really factor in retention (churn) and lifetime value.

    Finally, I want to make a plug here for hiring MBAs to help in the process of making your startup analytically focused. I know Mark has made jabs at MBAs in the past (he being one as well as myself); however, I think hiring a recent MBA graduate could be a great way to get relatively cheep talent ($80K – $120K per year) that will work their ass off and help define and measure the analytics of the company. Make sure they have a finance focus, so you know they are strong in financial modeling and statistics.

  • http://euonymous.wordpress.com euonymous

    Mark, a spell and grammar checker would go a long way to improving this piece. Having said that, this is good, solid content. I’d like to see it recast with a slant toward more B2B businesses. Focus on sales related metrics is important; so is knowing your overall fixed and variable costs, and profitability (not just revenue) drivers.

  • http://twitter.com/NickyChips Nik Souris

    Mark, Another excellent post and great way to communicate with a company’s outsider interests. Access to data today makes getting metrics calculated a lot better today but still, focusing on the results can be too late – it’s operational “gauges” that give you a feel or tell you when you’re overheating or losing power/control.

    So as everyone gets hyped on defining their respective key performance metrics, I would add or rather insert this little reminder for the inside of the house folks “Don’t expect what you don’t inspect.” Equal, if not more vital, to operational excellence are the gauges on the operations that feed those metrics — queue counts, customer hold times, abandoned calls in call center; prospect touches / lengths for a sales resource; time spent on site/app; absenteeism. While it can be costly to monitor, especially at early stages, start-ups poised to change the world figure out how to do it and make it part of their culture/MO… ones that don’t will often make their push, reach some achievement and then end up having the world change them.

  • Jvaio2003

    What part of the business does metrics fall under? Is it a sales department category or does it fall under business development?

  • Anonymous

    Hi, Mark,
    Great valuable content here and wish I had this at my previous startu up! I have a question. Your ROI calc. on Google ads…”…if you convert 12.5% of the people who click on Google paid links then your true cost to acquire is actually = $12 ($1.50 / 0.125).” when you say “convert” do you mean the people who click on the ad or people who actually bought something? I’m thinking there’s a step in between here…where you would have a click thru and then a buyer. Right? The click thru rate would be low… but the conversion rate might be higher.

  • Emily Merkle

    You can never have too much data.

    Conversion = acquisition of new user.

  • Anonymous

    Excellent post, as always. I’d like to add two items. First, it’s much easier to establish this culture of transparent analytics and measurement early in a company’s lifetime than later. Trying to add it later will take much longer and can be quite difficult, especially as the company gets larger.

    Second (and this is implied in the comments), testing and learning is as important as measuring itself. Measuring can tell management whether or not it is successful, but may not explain why. Developing a hypothesis, testing it, and measuring the results brings analytics to a higher level.

  • Rebecca

    I like that you mention that it isn’t just start ups that fail to measure. We stress that with our clients. For one thing it is a great motivator and as you mentioned, gains commitment. In addition to the areas that you mention, we suggest that organizations measure the ‘human elements’ of communication, trust, and alignment as well. As the culture improves there is a greater commitment to addressing and improving upon the other areas being measured. It is a much easier task when everyone shows up for the same game.

  • http://notesfromtheninjabunny.tumblr.com/ Emily Merkle

    I could not agree more. Start by emphasizing the absolute essentials – transparency, respect, honesty, humility, “in or out – no straddling”. That sums it up for my start-ups.
    As a scientist by schooling – hypothesize based on observation/small sample, test – controlling all variables possible, collect data at all points, understand what you have collected – and you may not get a “bingo” moment (most times you will not) – but keep at it and 1) you’ll get better at analyzing your data and running critical experiments, and 2) you will learn something.

    great post, Phil.

  • http://www.safehostel.com/ Tim Zenderman

    Hey, you should check out CrowdSavvy.com as they are trying to enable a lot of what you are talking about here, particularly in Retention/Churn, for mobile app developers. I don’t work for them, but I know the team very well.

  • http://twitter.com/justinstoddart Justin Stoddart

    This blog is the best stuff I’ve found on the Internet in this space. Mark–thank you. To any who read this, I would recommend more than a quick skim. The best way I’ve found to learn really good material like this is to read, share, discuss and teach others (knowing well enough to teach it is a great test of how well you know it), and most important–implement whatever’s appropriate right away. Knowledge exists through accomplishment, everything else is familiarity or understanding.

  • http://technbiz.blogspot.com paramendra

    @Jack Dorsey says something similar. He calls it instrumentation.

  • http://pulse.yahoo.com/_ICU4N7W4V6QXLFOYWIL5X7Y4LE Rose Slagan

    I couldn’t agree more. Too many people just think doing something will be a good thing. If you want to get a decent roi you need to break things down to specific results and investment required. Good post.

  • Anonymous

    Absolutely, practice makes perfect. I think it’s wonderful that your scientific background helps you understand the test and learn philosophy. It’s also important to make sure those without scientific backgrounds also buy in to this way of thinking at a company.

  • Anonymous

    Absolutely, practice makes perfect. I think it’s wonderful that your scientific background helps you understand the test and learn philosophy. It’s also important to make sure those without scientific backgrounds also buy in to this way of thinking at a company.

  • Anonymous

    Absolutely, practice makes perfect. I think it’s wonderful that your scientific background helps you understand the test and learn philosophy. It’s also important to make sure those without scientific backgrounds also buy in to this way of thinking at a company.

  • Anonymous

    It depends on the dynamics of the organization. In some companies, there is a specific department (whether it be finance, marketing analytics, or even IT) that has sole responsibility for metrics. This is more likely to happen when building a metrics-based culture starts from the bottom up.

    In other organizations, the idea of measuring is part of the DNA. As such, every department is responsible for developing and measuring its own metrics. In many cases, this culture starts from the top (e.g. CEO or President).

  • http://www.qualtrx.com Rashaun P. Sourles

    Mark and the “Both Sides” community -

    Coming from where I do in the land of pharma where “data” isn’t really data at all–here’s I ask when it comes to metrics:

    1) Is the metric related to critical success factors today or from some previous moment in time?
    2) How does the metric relate to human behavior?
    3) How reliable is the “data” that is being captured and tracked?

    Am I possibly getting too esoteric here? While I’m confident that metrics are valuable, I’ve also seen that in the fast changing world in which we all live, we often cling to outmoded metrics; even worse, we cling to these metrics well-beyond the point where we have reasonably understood them to be inaccurate or misleading.

    Metrics yes! Contextual metrics!

    ~@Rashaunps

  • Emilymerkle

    Rashaun:

    My sympathies that you come from ‘PharmaLand” – I despise BigPharma, and it’s not a secret that many times their clinical trial “data” is purposefully tweaked or flat-out fabricated to justify the means to their end.

    That said:

    1) Can you define your understanding of “metrics”? Do you mean Key Performance Indicators (KPIs) – or are you referring to analytics, a major vehicle for capturing analyzing data?

    2) I don’t know what you mean by “we often cling to outmoded metrics”. Please clarify.

    3) In this industry, and I feel comfortable making a fairly blanket statement for the professionals in this audience at least, we not only do not cling to “outmoded metrics” – whatever that means – but the moment data is analyzed and deemed to be inaccurate/misleading by way of the methodology the data is collected and/or analyzed.

    Interested in your perspective on this. You’re not being the least esoteric – just would like some clarification. Thanks!

  • Emilymerkle

    True dat – though easier said than done ;) In my experience, with respect to rigorous experimentation, you have to take on the role of “professor” and demonstrate by way of examples…