Managing Startups: Best Blog Posts

Chapter 36. Live-Data Prototypes Versus Production

Marty Cagan

I have written about the differences between user prototypes (simulations intended to test the user experience) and live-data prototypes (actual code intended to send live traffic to in order to test real behavior).

We use a live-data prototype (usually in the context of an A/B test) to prove something works, and we perform user testing to understand why it doesn’t work, and what we can do to correct.

I’m happy to see so many teams now doing live-data prototypes. However, I often find that teams confuse live-data prototypes with production software. The purpose of this article is to clarify the difference.

First, let’s be clear for those that do not necessarily know what is meant by the term “production software.”

Generally, this means that this is software we can run our business on. But specifically, this means:

• All of the critical use cases have been implemented and tested.

• The necessary level of test automation is in place.

• The software has been architected and engineered so that it can work at the necessary scale and performance.

• The associated SEO work that may be necessary based on the changes is complete.

• If the product is a global product, the software has been internationalized and localized.

• Activities like code reviews have been done to ensure that the software is something the team can maintain and stand behind.

• The sales, customer support, and marketing organizations have been briefed on the differences ...