The Discipline of Business Experimentation

The data you already have can't tell you how customers will react to innovations. To discover if a truly novel concept will succeed, you must subject it to a rigorous experiment. In most companies, tests do not adhere to scientific and statistical principles. As a result, managers often end up interpreting statistical noise as causation—and making bad decisions. To conduct experiments that are worth the expense and effort, companies need to ask themselves several questions: Does the experiment have a clear purpose? Managers must figure out exactly what they want to learn in order to determine if testing is the best approach. Have stakeholders made a commitment to abide by the results? Are they willing to walk away from a project if the findings suggest they should? Is the experiment doable? The complexity of the variables in a business experiment and their interactions can make it difficult to determine cause-and-effect relationships. Choosing the right sample size is important. How can we ensure reliable results? Randomized field trials, "blind" tests, and big data can help. Have we gotten the most value out of the experiment? Conducting the experiment is just the beginning. Use the data to assess which components of a new initiative might have the highest ROI or the markets where it is most likely to be successful.