You are previewing Lean Analytics.

Lean Analytics

Cover of Lean Analytics by Alistair Croll... Published by O'Reilly Media, Inc.
  1. Praise for Lean Analytics
  2. Dedication
  3. Foreword
  4. Preface
    1. Who This Book Is For
    2. How This Book Works
    3. The Building Blocks
      1. Customer Development
      2. Lean Startup
    4. We’d Like to Hear from You
    5. O’Reilly Safari
    6. Thanks and Acknowledgments
  5. I. Stop Lying to Yourself
    1. 1. We’re All Liars
      1. The Lean Startup Movement
      2. Poking a Hole in Your Reality Distortion Field
      3. Airbnb Photography—Growth Within Growth
    2. 2. How to Keep Score
      1. What Makes a Good Metric?
      2. Qualitative Versus Quantitative Metrics
      3. Vanity Versus Real Metrics
      4. Eight Vanity Metrics to Watch Out For
      5. Exploratory Versus Reporting Metrics
      6. Circle of Moms Explores Its Way to Success
      7. Leading Versus Lagging Metrics
      8. Correlated Versus Causal Metrics
      9. Moving Targets
      10. HighScore House Defines an “Active User”
      11. Segments, Cohorts, A/B Testing, and Multivariate Analysis
      12. The Lean Analytics Cycle
      13. Evaluating the Metrics You Track
    3. 3. Deciding What to Do with Your Life
      1. The Lean Canvas
      2. What Should You Work On?
      3. Create a Lean Canvas
    4. 4. Data-Driven Versus Data-Informed
      1. How to Think Like a Data Scientist
      2. Lean Startup and Big Vision
  6. II. Finding the Right Metric for Right Now
    1. 5. Analytics Frameworks
      1. Dave McClure’s Pirate Metrics
      2. Eric Ries’s Engines of Growth
      3. Ash Maurya’s Lean Canvas
      4. Sean Ellis’s Startup Growth Pyramid
      5. The Long Funnel
      6. The Lean Analytics Stages and Gates
    2. 6. The Discipline of One Metric That Matters
      1. Moz Tracks Fewer KPIs to Increase Focus
      2. Four Reasons to Use the One Metric That Matters
      3. Solare Focuses on a Few Key Metrics
      4. Drawing Lines in the Sand
      5. The Squeeze Toy
      6. Define Your OMTM
    3. 7. What Business Are You In?
      1. About Those People
      2. The Business Model Flipbook
      3. Six Business Models
      4. Pick Your Business Model
    4. 8. Model One: E-commerce
      1. What Mode of E-commerce Are You?
      2. A Practical Example
      3. WineExpress Increases Revenue by 41% Per Visitor
      4. Offline and Online Combinations
      5. Visualizing the E-commerce Business
      6. Wrinkles: Traditional E-commerce Versus Subscription E-commerce
      7. Key Takeaways
    5. 9. Model Two: Software as a Service (SaaS)
      1. Backupify’s Customer Lifecycle Learning
      2. Measuring Engagement
      3. Churn
      4. Visualizing the SaaS Business
      5. ClearFit Abandons Monthly Subscriptions for 10x Growth
      6. Wrinkles: Freemium, Tiers, and Other Pricing Models
      7. Key Takeaways
    6. 10. Model Three: Free Mobile App
      1. Installation Volume
      2. Average Revenue Per User
      3. Percentage of Users Who Pay
      4. Churn
      5. Visualizing the Mobile App Business
      6. Wrinkles: In-App Monetization Versus Advertising
      7. Key Takeaways
    7. 11. Model Four: Media Site
      1. Audience and Churn
      2. Inventory
      3. Performance and the Sessions-to-Clicks Ratio
      4. Ad Rates
      5. Content/Advertising Trade-off
      6. Visualizing the Media Business
      7. Wrinkles: Hidden Affiliates, Background Noise, Ad Blockers, and Paywalls
      8. Key Takeaways
    8. 12. Model Five: User-Generated Content
      1. Visitor Engagement
      2. Content Creation and Interaction
      3. Engagement Funnel Changes
      4. Value of Created Content
      5. Content Sharing and Virality
      6. Notification Effectiveness
      7. Visualizing a UCG Business
      8. Wrinkles: Passive Content Creation
      9. Key Takeaways
    9. 13. Model Six: Two-Sided Marketplaces
      1. What DuProprio Watches
      2. Rate at Which You’re Adding Buyers and Sellers
      3. Rate of Inventory Growth
      4. Buyer Searches
      5. Conversion Rates and Segmentation
      6. Buyer and Seller Ratings
      7. Visualizing a Two-Sided Marketplace
      8. Wrinkles: Chicken and Egg, Fraud, Keeping the Transaction, and Auctions
      9. Key Takeaways
    10. 14. What Stage Are You At?
      1. Pick the Stage That You’re At
    11. 15. Stage One: Empathy
      1. Metrics for the Empathy Stage
      2. This Is the Best Idea I’ve Ever Had! (or, How to Discover Problems Worth Solving)
      3. Finding a Problem to Fix (or, How to Validate a Problem)
      4. Signs You’ve Found a Problem Worth Tackling
      5. Running Lean and How to Conduct a Good Interview
      6. How to Avoid Leading the Witness
      7. Convergent and Divergent Problem Interviews
      8. How Do I Know If the Problem Is Really Painful Enough?
      9. Cloud9 IDE Interviews Existing Customers
      10. How Are People Solving the Problem Now?
      11. Are There Enough People Who Care About This Problem? (or, Understanding the Market)
      12. What Will It Take to Make Them Aware of the Problem?
      13. A “Day in the Life” of Your Customer
      14. Finding People to Talk To
      15. Getting Answers at Scale
      16. LikeBright “Mechanical Turks” Its Way into TechStars
      17. Creating an Answers-at-Scale Campaign
      18. Build It Before You Build It (or, How to Validate the Solution)
      19. Localmind Hacks Twitter
      20. Before You Launch the MVP
      21. Deciding What Goes into the MVP
      22. Measuring the MVP
      23. Static Pixels Eliminates a Step in Its Order Process
      24. A Summary of the Empathy Stage
      25. Should You Move to the Next Stage?
    12. 16. Stage Two: Stickiness
      1. MVP Stickiness
      2. Iterating the MVP
      3. qidiq Changes How It Adds Users
      4. Premature Virality
      5. The Goal Is Retention
      6. Seven Questions to Ask Yourself Before Building a Feature
      7. How Rally Builds New Features with a Lean Approach
      8. How to Handle User Feedback
      9. The Minimum Viable Vision
      10. The Problem-Solution Canvas
      11. VNN Uses the Problem-Solution Canvas to Solve Business Problems
      12. A Summary of the Stickiness Stage
      13. Exercise #1: Should You Move to the Next Stage?
      14. Exercise #2: Have You Identified Your Biggest Problems?
    13. 17. Stage Three: Virality
      1. The Three Ways Things Spread
      2. Metrics for the Viral Phase
      3. Beyond the Viral Coefficient
      4. Timehop Experiments with Content Sharing to Achieve Virality
      5. Instrumenting the Viral Pattern
      6. Growth Hacking
      7. A Summary of the Virality Stage
      8. Should You Move On to the Revenue Stage?
    14. 18. Stage Four: Revenue
      1. Metrics for the Revenue Stage
      2. The Penny Machine
      3. Finding Your Revenue Groove
      4. Customer Lifetime Value > Customer Acquisition Cost
      5. and the Pivot to Revenue
      6. Market/Product Fit
      7. The Breakeven Lines in the Sand
      8. Revenue Stage Summary
    15. 19. Stage Five: Scale
      1. The Hole in the Middle
      2. Metrics for the Scale Stage
      3. Is My Business Model Right?
      4. Buffer Goes from Stickiness to Scale (Through Revenue)
      5. The Three-Threes Model
      6. Finding Discipline as You Scale
      7. A Summary of the Scale Stage
    16. 20. Model + Stage Drives the Metric You Track
  7. III. Lines in the Sand
    1. 21. Am I Good Enough?
      1. WP Engine Discovers the 2% Cancellation Rate
      2. Average Isn’t Good Enough
      3. What Is Good Enough?
      4. Growth Rate
      5. Number of Engaged Visitors
      6. Pricing Metrics
      7. Socialight Discovers the Underlying Metrics of Pricing
      8. Cost of Customer Acquisition
      9. Virality
      10. Mailing List Effectiveness
      11. Uptime and Reliability
      12. Site Engagement
      13. Web Performance
      14. Make Your Own Lines in the Sand
    2. 22. E-commerce: Lines in the Sand
      1. Conversion Rate
      2. Shopping Cart Abandonment
      3. Search Effectiveness
    3. 23. SaaS: Lines in the Sand
      1. Paid Enrollment
      2. Freemium Versus Paid
      3. Upselling and Growing Revenue
      4. Churn
      5. OfficeDrop’s Key Metric: Paid Churn
    4. 24. Free Mobile App: Lines in the Sand
      1. Mobile Downloads
      2. Mobile Download Size
      3. Mobile Customer Acquisition Cost
      4. Sincerely Learns the Challenges of Mobile Customer Acquisition
      5. Application Launch Rate
      6. Percent Active Mobile Users/Players
      7. Percentage of Mobile Users Who Pay
      8. Average Revenue Per Daily Active User
      9. Monthly Average Revenue Per Mobile User
      10. Average Revenue Per Paying User
      11. Mobile App Ratings Click-Through
      12. Mobile Customer Lifetime Value
    5. 25. Media Site: Lines in the Sand
      1. Click-Through Rates
      2. Sessions-to-Clicks Ratio
      3. Referrers
      4. Engaged Time
      5. What Onsite Engagement Can Tell You About Goals and Behaviors
      6. Sharing with Others
      7. JFL Gags Cracks Up YouTube
    6. 26. User-Generated Content: Lines in the Sand
      1. Content Upload Success
      2. Time on Site Per Day
      3. Reddit Part 1—From Links to a Community
      4. Engagement Funnel Changes
      5. Reddit Part 2—There’s Gold in Those Users
      6. Spam and Bad Content
    7. 27. Two-Sided Marketplaces: Lines in the Sand
      1. Transaction Size
      2. What Etsy Watches
      3. Top 10 Lists
    8. 28. What to Do When You Don’t Have a Baseline
  8. IV. Putting Lean Analytics to Work
    1. 29. Selling into Enterprise Markets
      1. Why Are Enterprise Customers Different?
      2. Legacy Products
      3. The Enterprise Startup Lifecycle
      4. How Coradiant Found a Market
      5. So What Metrics Matter?
      6. The Bottom Line: Startups Are Startups
    2. 30. Lean from Within: Intrapreneurs
      1. Span of Control and the Railroads
      2. Skunk Works for Intrapreneurs
      3. Changing—or Innovating to Resist Change?
      4. Stars, Dogs, Cows, and Question Marks
      5. Swiffer Gives Up on Chemistry
      6. Doritos Chooses a Flavor
      7. Working with an Executive Sponsor
      8. EMI Embraces Data to Understand Its Customers
      9. The Stages of Intrapreneur Lean Analytics
    3. 31. Conclusion: Beyond Startups
      1. How to Instill a Culture of Data in Your Company
  9. A. References and Further Reading
  10. B. About the Authors
  11. C. The Lean Series
  12. Index
  13. About the Authors
  14. Copyright
O'Reilly logo

Chapter 4. Data-Driven Versus Data-Informed

Data is a powerful thing. It can be addictive, making you overanalyze everything. But much of what we actually do is unconscious, based on past experience and pragmatism. And with good reason: relying on wisdom and experience, rather than rigid analysis, helps us get through our day. After all, you don’t run A/B testing before deciding what pants to put on in the morning; if you did, you’d never get out the door.

One of the criticisms of Lean Startup is that it’s too data-driven. Rather than be a slave to the data, these critics say, we should use it as a tool. We should be data-informed, not data-driven. Mostly, they’re just being lazy, and looking for reasons not to do the hard work. But sometimes, they have a point: using data to optimize one part of your business, without stepping back and looking at the big picture, can be dangerous—even fatal.

Consider travel agency Orbitz and its discovery that Mac users were willing to reserve a more expensive hotel room. CTO Roger Liew told the Wall Street Journal, “We had the intuition [that Mac users are 40% more likely to book a four- or five-star hotel than PC users and to stay in more expensive rooms], and we were able to confirm it based on the data.”[14]

On the one hand, an algorithm that ignores seemingly unrelated customer data (in this case, whether visitors were using a Mac) wouldn’t have found this opportunity to increase revenues. On the other hand, an algorithm that blindly optimizes based on customer data, regardless of its relationship to the sale, may have unintended consequences—like bad PR. Data-driven machine optimization, when not moderated by human judgment, can cause problems.

Years ago, Gail Ennis, then CMO of analytics giant Omniture, told one of us that users of the company’s content optimization tools had to temper machine optimization with human judgment. Left to its own devices, the software quickly learned that scantily clad women generated a far higher click-through rate on web pages than other forms of content. But that click-through rate was a short-term gain, offset by damage to the brand of the company that relied on it. So Omniture’s software works alongside curators who understand the bigger picture and provide suitable imagery for the machine to test. Humans do inspiration; machines do validation.

In mathematics, a local maximum is the largest value of a function within a given neighborhood.[15] That doesn’t mean it’s the largest possible value, just the largest one in a particular range. As an analogy, consider a lake on a mountainside. The water isn’t at its lowest possible level—that would be sea level—but it’s at the lowest possible level in the area surrounding the lake.

Optimization is all about finding the lowest or highest values of a particular function. A machine can find the optimal settings for something, but only within the constraints and problem space of which it’s aware, in much the same way that the water in a mountainside lake can’t find the lowest possible value, just the lowest value within the constraints provided.

To understand the problem with constrained optimization, imagine that you’re given three wheels and asked to evolve the best, most stable vehicle. After many iterations of pitting different wheel layouts against one another, you come up with a tricycle-like configuration. It’s the optimal three-wheeled configuration.

Data-driven optimization can perform this kind of iterative improvement. What it can’t do, however, is say, “You know what? Four wheels would be way better!” Math is good at optimizing a known system; humans are good at finding a new one. Put another way, change favors local maxima; innovation favors global disruption.

In his book River Out Of Eden (Basic Books), Richard Dawkins uses the analogy of a flowing river to describe evolution. Evolution, he explains, can create the eye. In fact, it can create dozens of versions of it, for wasps, octopods, humans, eagles, and whales. What it can’t do well is go backward: once you have an eye that’s useful, slight mutations don’t usually yield improvements. A human won’t evolve an eagle’s eye, because the intermediate steps all result in bad eyesight.

Machine-only optimization suffers from similar limitations as evolution. If you’re optimizing for local maxima, you might be missing a bigger, more important opportunity. It’s your job to be the intelligent designer to data’s evolution.

Many of the startup founders with whom we’ve spoken have a fundamental mistrust of leaving their businesses to numbers alone. They want to trust their guts. They’re uneasy with their companies being optimized without a soul, and see the need to look at the bigger picture of the market, the problem they’re solving, and their fundamental business models.

Ultimately, quantitative data is great for testing hypotheses, but it’s lousy for generating new ones unless combined with human introspection.

How to Think Like a Data Scientist

Monica Rogati, a data scientist at LinkedIn, gave us the following 10 common pitfalls that entrepreneurs should avoid as they dig into the data their startups capture.

  1. Assuming the data is clean. Cleaning the data you capture is often most of the work, and the simple act of cleaning it up can often reveal important patterns. “Is an instrumentation bug causing 30% of your numbers to be null?” asks Monica. “Do you really have that many users in the 90210 zip code?” Check your data at the door to be sure it’s valid and useful.

  2. Not normalizing. Let’s say you’re making a list of popular wedding destinations. You could count the number of people flying in for a wedding, but unless you consider the total number of air travellers coming to that city as well, you’ll just get a list of cities with busy airports.

  3. Excluding outliers. Those 21 people using your product more than a thousand times a day are either your biggest fans, or bots crawling your site for content. Whichever they are, ignoring them would be a mistake.

  4. Including outliers. While those 21 people using your product a thousand times a day are interesting from a qualitative perspective, because they can show you things you didn’t expect, they’re not good for building a general model. “You probably want to exclude them when building data products,” cautions Monica. “Otherwise, the ‘you may also like’ feature on your site will have the same items everywhere—the ones your hardcore fans wanted.”

  5. Ignoring seasonality. “Whoa, is ‘intern’ the fastest-growing job of the year? Oh, wait, it’s June.” Failure to consider time of day, day of week, and monthly changes when looking at patterns leads to bad decision making.

  6. Ignoring size when reporting growth. Context is critical. Or, as Monica puts it, “When you’ve just started, technically, your dad signing up does count as doubling your user base.”

  7. Data vomit. A dashboard isn’t much use if you don’t know where to look.

  8. Metrics that cry wolf. You want to be responsive, so you set up alerts to let you know when something is awry in order to fix it quickly. But if your thresholds are too sensitive, they get “whiny”—and you’ll start to ignore them.

  9. The “Not Collected Here” syndrome. “Mashing up your data with data from other sources can lead to valuable insights,” says Monica. “Do your best customers come from zip codes with a high concentration of sushi restaurants?” This might give you a few great ideas about what experiments to run next—or even influence your growth strategy.

  10. Focusing on noise. “We’re hardwired (and then programmed) to see patterns where there are none,” Monica warns. “It helps to set aside the vanity metrics, step back, and look at the bigger picture.“

Lean Startup and Big Vision

Some entrepreneurs are maniacally, almost compulsively, data-obsessed, but tend to get mired in analysis paralysis. Others are casual, shoot-from-the-hip intuitionists who ignore data unless it suits them, and pivot lazily from idea to idea without discipline. At the root of this divide is the fundamental challenge that Lean Startup advocates face: how do you have a minimum viable product and a hugely compelling vision at the same time?

Plenty of founders use Lean Startup as an excuse to start a company without a vision. “It’s so easy to start a company these days.” They reason, “the barriers are so low that everyone can do it, right?” Yet having a big vision is important: starting a company without one makes you susceptible to outside influences, be they from customers, investors, competition, press, or anything else. Without a big vision, you’ll lack purpose, and over time you’ll find yourself wandering aimlessly.

So if a big, hairy, audacious vision is important—one with a changing-the-world type goal—how does that reconcile with the step-by-step, always-questioning approach of Lean Startup?

The answer is actually pretty simple. You need to think of Lean Startup as the process you use to move toward and achieve your vision.

We sometimes remind early-stage founders that, in many ways, they aren’t building a product. They’re building a tool to learn what product to build. This helps separate the task at hand—finding a sustainable business model—from the screens, lines of code, and mailing lists they’ve carefully built along the way.

Lean Startup is focused on learning above everything else, and encourages broad thinking, exploration, and experimentation. It’s not about mindlessly going through the motions of build>measure>learn—it’s about really understanding what’s going on and being open to new possibilities.

Be Lean. Don’t be small. We’ve talked to founders who want to be the leading provider in their state or province. Why not the world? Even the Allies had to pick a beachhead, but landing in Normandy didn’t mean they lacked a big vision. They just found a good place to start.

Some people believe Lean Startup encourages that smallness, but in fact, used properly, Lean Startup helps expand your vision, because you’re encouraged to question everything. As you dig deeper and peel away more layers of what you’re doing—whether you’re looking at problems, solutions, customers, revenue, or anything else—you’re likely to find a lot more than you expected. If you’re opportunistic about it, you can expand your vision and understand how to get there faster, all at the same time.

The best content for your career. Discover unlimited learning on demand for around $1/day.