You are previewing Lean Analytics.

Lean Analytics

Cover of Lean Analytics by Alistair Croll... Published by O'Reilly Media, Inc.
  1. Praise for Lean Analytics
  2. Dedication
  3. Foreword
  4. Preface
    1. Who This Book Is For
    2. How This Book Works
    3. The Building Blocks
      1. Customer Development
      2. Lean Startup
    4. We’d Like to Hear from You
    5. O’Reilly Safari
    6. Thanks and Acknowledgments
  5. I. Stop Lying to Yourself
    1. 1. We’re All Liars
      1. The Lean Startup Movement
      2. Poking a Hole in Your Reality Distortion Field
      3. Airbnb Photography—Growth Within Growth
    2. 2. How to Keep Score
      1. What Makes a Good Metric?
      2. Qualitative Versus Quantitative Metrics
      3. Vanity Versus Real Metrics
      4. Eight Vanity Metrics to Watch Out For
      5. Exploratory Versus Reporting Metrics
      6. Circle of Moms Explores Its Way to Success
      7. Leading Versus Lagging Metrics
      8. Correlated Versus Causal Metrics
      9. Moving Targets
      10. HighScore House Defines an “Active User”
      11. Segments, Cohorts, A/B Testing, and Multivariate Analysis
      12. The Lean Analytics Cycle
      13. Evaluating the Metrics You Track
    3. 3. Deciding What to Do with Your Life
      1. The Lean Canvas
      2. What Should You Work On?
      3. Create a Lean Canvas
    4. 4. Data-Driven Versus Data-Informed
      1. How to Think Like a Data Scientist
      2. Lean Startup and Big Vision
  6. II. Finding the Right Metric for Right Now
    1. 5. Analytics Frameworks
      1. Dave McClure’s Pirate Metrics
      2. Eric Ries’s Engines of Growth
      3. Ash Maurya’s Lean Canvas
      4. Sean Ellis’s Startup Growth Pyramid
      5. The Long Funnel
      6. The Lean Analytics Stages and Gates
    2. 6. The Discipline of One Metric That Matters
      1. Moz Tracks Fewer KPIs to Increase Focus
      2. Four Reasons to Use the One Metric That Matters
      3. Solare Focuses on a Few Key Metrics
      4. Drawing Lines in the Sand
      5. The Squeeze Toy
      6. Define Your OMTM
    3. 7. What Business Are You In?
      1. About Those People
      2. The Business Model Flipbook
      3. Six Business Models
      4. Pick Your Business Model
    4. 8. Model One: E-commerce
      1. What Mode of E-commerce Are You?
      2. A Practical Example
      3. WineExpress Increases Revenue by 41% Per Visitor
      4. Offline and Online Combinations
      5. Visualizing the E-commerce Business
      6. Wrinkles: Traditional E-commerce Versus Subscription E-commerce
      7. Key Takeaways
    5. 9. Model Two: Software as a Service (SaaS)
      1. Backupify’s Customer Lifecycle Learning
      2. Measuring Engagement
      3. Churn
      4. Visualizing the SaaS Business
      5. ClearFit Abandons Monthly Subscriptions for 10x Growth
      6. Wrinkles: Freemium, Tiers, and Other Pricing Models
      7. Key Takeaways
    6. 10. Model Three: Free Mobile App
      1. Installation Volume
      2. Average Revenue Per User
      3. Percentage of Users Who Pay
      4. Churn
      5. Visualizing the Mobile App Business
      6. Wrinkles: In-App Monetization Versus Advertising
      7. Key Takeaways
    7. 11. Model Four: Media Site
      1. Audience and Churn
      2. Inventory
      3. Performance and the Sessions-to-Clicks Ratio
      4. Ad Rates
      5. Content/Advertising Trade-off
      6. Visualizing the Media Business
      7. Wrinkles: Hidden Affiliates, Background Noise, Ad Blockers, and Paywalls
      8. Key Takeaways
    8. 12. Model Five: User-Generated Content
      1. Visitor Engagement
      2. Content Creation and Interaction
      3. Engagement Funnel Changes
      4. Value of Created Content
      5. Content Sharing and Virality
      6. Notification Effectiveness
      7. Visualizing a UCG Business
      8. Wrinkles: Passive Content Creation
      9. Key Takeaways
    9. 13. Model Six: Two-Sided Marketplaces
      1. What DuProprio Watches
      2. Rate at Which You’re Adding Buyers and Sellers
      3. Rate of Inventory Growth
      4. Buyer Searches
      5. Conversion Rates and Segmentation
      6. Buyer and Seller Ratings
      7. Visualizing a Two-Sided Marketplace
      8. Wrinkles: Chicken and Egg, Fraud, Keeping the Transaction, and Auctions
      9. Key Takeaways
    10. 14. What Stage Are You At?
      1. Pick the Stage That You’re At
    11. 15. Stage One: Empathy
      1. Metrics for the Empathy Stage
      2. This Is the Best Idea I’ve Ever Had! (or, How to Discover Problems Worth Solving)
      3. Finding a Problem to Fix (or, How to Validate a Problem)
      4. Signs You’ve Found a Problem Worth Tackling
      5. Running Lean and How to Conduct a Good Interview
      6. How to Avoid Leading the Witness
      7. Convergent and Divergent Problem Interviews
      8. How Do I Know If the Problem Is Really Painful Enough?
      9. Cloud9 IDE Interviews Existing Customers
      10. How Are People Solving the Problem Now?
      11. Are There Enough People Who Care About This Problem? (or, Understanding the Market)
      12. What Will It Take to Make Them Aware of the Problem?
      13. A “Day in the Life” of Your Customer
      14. Finding People to Talk To
      15. Getting Answers at Scale
      16. LikeBright “Mechanical Turks” Its Way into TechStars
      17. Creating an Answers-at-Scale Campaign
      18. Build It Before You Build It (or, How to Validate the Solution)
      19. Localmind Hacks Twitter
      20. Before You Launch the MVP
      21. Deciding What Goes into the MVP
      22. Measuring the MVP
      23. Static Pixels Eliminates a Step in Its Order Process
      24. A Summary of the Empathy Stage
      25. Should You Move to the Next Stage?
    12. 16. Stage Two: Stickiness
      1. MVP Stickiness
      2. Iterating the MVP
      3. qidiq Changes How It Adds Users
      4. Premature Virality
      5. The Goal Is Retention
      6. Seven Questions to Ask Yourself Before Building a Feature
      7. How Rally Builds New Features with a Lean Approach
      8. How to Handle User Feedback
      9. The Minimum Viable Vision
      10. The Problem-Solution Canvas
      11. VNN Uses the Problem-Solution Canvas to Solve Business Problems
      12. A Summary of the Stickiness Stage
      13. Exercise #1: Should You Move to the Next Stage?
      14. Exercise #2: Have You Identified Your Biggest Problems?
    13. 17. Stage Three: Virality
      1. The Three Ways Things Spread
      2. Metrics for the Viral Phase
      3. Beyond the Viral Coefficient
      4. Timehop Experiments with Content Sharing to Achieve Virality
      5. Instrumenting the Viral Pattern
      6. Growth Hacking
      7. A Summary of the Virality Stage
      8. Should You Move On to the Revenue Stage?
    14. 18. Stage Four: Revenue
      1. Metrics for the Revenue Stage
      2. The Penny Machine
      3. Finding Your Revenue Groove
      4. Customer Lifetime Value > Customer Acquisition Cost
      5. and the Pivot to Revenue
      6. Market/Product Fit
      7. The Breakeven Lines in the Sand
      8. Revenue Stage Summary
    15. 19. Stage Five: Scale
      1. The Hole in the Middle
      2. Metrics for the Scale Stage
      3. Is My Business Model Right?
      4. Buffer Goes from Stickiness to Scale (Through Revenue)
      5. The Three-Threes Model
      6. Finding Discipline as You Scale
      7. A Summary of the Scale Stage
    16. 20. Model + Stage Drives the Metric You Track
  7. III. Lines in the Sand
    1. 21. Am I Good Enough?
      1. WP Engine Discovers the 2% Cancellation Rate
      2. Average Isn’t Good Enough
      3. What Is Good Enough?
      4. Growth Rate
      5. Number of Engaged Visitors
      6. Pricing Metrics
      7. Socialight Discovers the Underlying Metrics of Pricing
      8. Cost of Customer Acquisition
      9. Virality
      10. Mailing List Effectiveness
      11. Uptime and Reliability
      12. Site Engagement
      13. Web Performance
      14. Make Your Own Lines in the Sand
    2. 22. E-commerce: Lines in the Sand
      1. Conversion Rate
      2. Shopping Cart Abandonment
      3. Search Effectiveness
    3. 23. SaaS: Lines in the Sand
      1. Paid Enrollment
      2. Freemium Versus Paid
      3. Upselling and Growing Revenue
      4. Churn
      5. OfficeDrop’s Key Metric: Paid Churn
    4. 24. Free Mobile App: Lines in the Sand
      1. Mobile Downloads
      2. Mobile Download Size
      3. Mobile Customer Acquisition Cost
      4. Sincerely Learns the Challenges of Mobile Customer Acquisition
      5. Application Launch Rate
      6. Percent Active Mobile Users/Players
      7. Percentage of Mobile Users Who Pay
      8. Average Revenue Per Daily Active User
      9. Monthly Average Revenue Per Mobile User
      10. Average Revenue Per Paying User
      11. Mobile App Ratings Click-Through
      12. Mobile Customer Lifetime Value
    5. 25. Media Site: Lines in the Sand
      1. Click-Through Rates
      2. Sessions-to-Clicks Ratio
      3. Referrers
      4. Engaged Time
      5. What Onsite Engagement Can Tell You About Goals and Behaviors
      6. Sharing with Others
      7. JFL Gags Cracks Up YouTube
    6. 26. User-Generated Content: Lines in the Sand
      1. Content Upload Success
      2. Time on Site Per Day
      3. Reddit Part 1—From Links to a Community
      4. Engagement Funnel Changes
      5. Reddit Part 2—There’s Gold in Those Users
      6. Spam and Bad Content
    7. 27. Two-Sided Marketplaces: Lines in the Sand
      1. Transaction Size
      2. What Etsy Watches
      3. Top 10 Lists
    8. 28. What to Do When You Don’t Have a Baseline
  8. IV. Putting Lean Analytics to Work
    1. 29. Selling into Enterprise Markets
      1. Why Are Enterprise Customers Different?
      2. Legacy Products
      3. The Enterprise Startup Lifecycle
      4. How Coradiant Found a Market
      5. So What Metrics Matter?
      6. The Bottom Line: Startups Are Startups
    2. 30. Lean from Within: Intrapreneurs
      1. Span of Control and the Railroads
      2. Skunk Works for Intrapreneurs
      3. Changing—or Innovating to Resist Change?
      4. Stars, Dogs, Cows, and Question Marks
      5. Swiffer Gives Up on Chemistry
      6. Doritos Chooses a Flavor
      7. Working with an Executive Sponsor
      8. EMI Embraces Data to Understand Its Customers
      9. The Stages of Intrapreneur Lean Analytics
    3. 31. Conclusion: Beyond Startups
      1. How to Instill a Culture of Data in Your Company
  9. A. References and Further Reading
  10. B. About the Authors
  11. C. The Lean Series
  12. Index
  13. About the Authors
  14. Copyright
O'Reilly logo

Chapter 1. We’re All Liars

Let’s face it: you’re delusional.

We’re all delusional—some more than others. Entrepreneurs are the most delusional of all.

Entrepreneurs are particularly good at lying to themselves. Lying may even be a prerequisite for succeeding as an entrepreneur—after all, you need to convince others that something is true in the absence of good, hard evidence. You need believers to take a leap of faith with you. As an entrepreneur, you need to live in a semi-delusional state just to survive the inevitable rollercoaster ride of running your startup.

Small lies are essential. They create your reality distortion field. They are a necessary part of being an entrepreneur. But if you start believing your own hype, you won’t survive. You’ll go too far into the bubble you’ve created, and you won’t come out until you hit the wall—hard—and that bubble bursts.

You need to lie to yourself, but not to the point where you’re jeopardizing your business.

That’s where data comes in.

Your delusions, no matter how convincing, will wither under the harsh light of data. Analytics is the necessary counterweight to lying, the yin to the yang of hyperbole. Moreover, data-driven learning is the cornerstone of success in startups. It’s how you learn what’s working and iterate toward the right product and market before the money runs out.

We’re not suggesting that gut instinct is a bad thing. Instincts are inspiration, and you’ll need to listen to your gut and rely on it throughout the startup journey. But don’t disembowel yourself. Guts matter; you’ve just got to test them. Instincts are experiments. Data is proof.

The Lean Startup Movement

Innovation is hard work—harder than most people realize. This is true whether you’re a lone startup trying to disrupt an industry or a rogue employee challenging the status quo, tilting at corporate windmills and steering around bureaucratic roadblocks. We get it. Entrepreneurship is crazy, bordering on absurd.

Lean Startup provides a framework by which you can more rigorously go about the business of creating something new. Lean Startup delivers a heavy dose of intellectual honesty. Follow the Lean model, and it becomes increasingly hard to lie, especially to yourself.

There’s a reason the Lean Startup movement has taken off now. We’re in the midst of a fundamental shift in how companies are built. It’s vanishingly cheap to create the first version of something. Clouds are free. Social media is free. Competitive research is free. Even billing and transactions are free.[3] We live in a digital world, and the bits don’t cost anything.

That means you can build something, measure its effect, and learn from it to build something better the next time. You can iterate quickly, deciding early on if you should double down on your idea or fold and move on to the next one. And that’s where analytics comes in. Learning doesn’t happen accidentally. It’s an integral part of the Lean process.

Management guru and author Peter Drucker famously observed, “If you can’t measure it, you can’t manage it.”[4] Nowhere is this truer than in the Lean model, where successful entrepreneurs build the product, the go-to-market strategy, and the systems by which to learn what customers want—simultaneously.

Poking a Hole in Your Reality Distortion Field

Most entrepreneurs have been crushed, usually more than once. If you haven’t been solidly trounced on a regular basis, you’re probably doing it wrong, and aren’t taking the risks you need to succeed in a big way.

But there’s a moment on the startup rollercoaster where the whole thing comes right off the rails. It’s truly failed. There’s little more to do than turn off the website and close down the bank account. You’re overwhelmed, the challenges are too great, and it’s over. You’ve failed.

Long before the actual derailment, you knew this was going to happen. It wasn’t working. But at the time, your reality distortion field was strong enough to keep you going on faith and fumes alone. As a result, you hit the wall at a million miles an hour, lying to yourself the whole time.

We’re not arguing against the importance of the reality distortion field—but we do want to poke a few holes in it. Hopefully, as a result, you’ll see the derailment in time to avoid it. We want you to rely less on your reality distortion field, and rely more on Lean Analytics.

Airbnb Photography—Growth Within Growth

Airbnb is an incredible success story. In just a few years, the company has become a powerhouse in the travel industry, providing travelers with an alternative to hotels, and providing individuals who have rooms, apartments, or homes to rent with a new source of income. In 2012, travelers booked over 5 million nights with Airbnb’s service. But it started small, and its founders—adherents to the Lean Startup mindset—took a very methodical approach to their success.

At SXSW 2012, Joe Zadeh, Product Lead at Airbnb, shared part of the company’s amazing story. He focused on one aspect of its business: professional photography.

It started with a hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.” This is where the founders’ gut instincts came in: they had a sense that professional photography would help their business. But rather than implementing it outright, they built a Concierge Minimum Viable Product (MVP) to quickly test their hypothesis.

What Is a Concierge MVP?

The Minimum Viable Product is the smallest thing you can build that will create the value you’ve promised to your market. But nowhere in that definition does it say how much of that offering has to be real. If you’re considering building a ride-sharing service, for example, you can try to connect drivers and passengers the old-fashioned way: by hand.

This is a concierge approach. It recognizes that sometimes, building a product—even a minimal one—isn’t worth the investment. The risk you’re investigating is, “Will people accept rides from others?” It’s emphatically not, “Can I build software to match drivers and passengers?” A Concierge MVP won’t scale, but it’s fast and easy in the short term.

Now that it’s cheap, even free, to launch a startup, the really scarce resource is attention. A concierge approach in which you run things behind the scenes for the first few customers lets you check whether the need is real; it also helps you understand which things people really use and refine your process before writing a line of code or hiring a single employee.

Initial tests of Airbnb’s MVP showed that professionally photographed listings got two to three times more bookings than the market average. This validated the founders’ first hypothesis. And it turned out that hosts were wildly enthusiastic about receiving an offer from Airbnb to take those photographs for them.

In mid-to-late 2011, Airbnb had 20 photographers in the field taking pictures for hosts—roughly the same time period where we see the proverbial “hockey stick” of growth in terms of nights booked, shown in Figure 1-1.

It’s amazing what you can do with 20 photographers and people’s apartments
Figure 1-1. It’s amazing what you can do with 20 photographers and people’s apartments

Airbnb experimented further. It watermarked photos to add authenticity. It got customer service to offer professional photography as a service when renters or potential renters called in. It increased the requirements on photo quality. Each step of the way, the company measured the results and adjusted as necessary. The key metric Airbnb tracked was shoots per month, because it had already proven with its Concierge MVP that more professional photographs meant more bookings.

By February 2012, Airbnb was doing nearly 5,000 shoots per month and continuing to accelerate the growth of the professional photography program.


  • Airbnb’s team had a hunch that better photos would increase rentals.

  • They tested the idea with a Concierge MVP, putting the least effort possible into a test that would give them valid results.

  • When the experiment showed good results, they built the necessary components and rolled it out to all customers.

Analytics Lessons Learned

Sometimes, growth comes from an aspect of your business you don’t expect. When you think you’ve found a worthwhile idea, decide how to test it quickly, with minimal investment. Define what success looks like beforehand, and know what you’re going to do if your hunch is right.

Lean is a great way to build businesses. And analytics ensures that you’ll collect and analyze data. Both fundamentally transform how you think about starting and growing a company. Both are more than processes—they’re mindsets. Lean, analytical thinking is about asking the right questions, and focusing on the one key metric that will produce the change you’re after.

With this book, we hope to provide you with the guidance, tools, and evidence to embrace data as a core component of your startup’s success. Ultimately, we want to show you how to use data to build a better startup faster.

[3] When we say “free,” we mean “free from significant upfront investment.” Plenty of cloud and billing services cost money—sometimes more money than you’d spend doing it yourself—once your business is under way. But free, here, means free from outlay in advance of finding your product/market fit. You can use PayPal, or Google Wallet, or Eventbrite, or dozens of other payment and ticketing systems, and pass on the cost of the transaction to your consumers.

[4] In Management: Tasks, Responsibilities, Practices (HarperBusiness), Drucker wrote, “Without productivity objectives, a business does not have direction. Without productivity measurements, it does not have control.”

The best content for your career. Discover unlimited learning on demand for around $1/day.