Control Groups and Meaningful Improvements

I just finished reading Superforecasters by Philip E. Tetlock and Dan Gardner (Crown Publishers, 2015). In the book the authors talk about how the accuracy of most forecasts is never measured. Forecasters make predictions (often too vague to be verified or refuted), and no one later gathers data to determine who's right and what forecasting methods work best. So there's no feedback loop to improve forecasting accuracy. Id.

Tetlock and Gardner make an analogy to medicine. For many years doctors relied mostly on personal experience and intuitive judgment to decide what treatments to use. Over this time treatment outcomes either didn't improve or improved slowly. Then control groups were introduced, with one set of patients receiving a placebo and one set of patients receiving the new treatment. This allowed researchers to carefully measure treatment outcomes to determine what worked, what didn't, and how treatments could be improved. This feedback loop led to rapid improvement in medical treatments and outcomes. Id.

A company making smartphones could apply these same concepts to make more meaningful product improvements. So Apple could look at iPhone usage, and the value of its latest improvements, by providing 1000 of its employees with an iPhone 6s (the control group) and 1000 of its employees with a prototype of Apple's upcoming smartphone release (the test group). It could then measure usage of both phones, including various phone features, to determine whether the latest improvements are meaningful/used or whether they're overserving/unused. Feedback from these in-house controlled studies could help keep Apple from releasing overserving products.

Because Apple is so vertically integrated, it already is well-positioned to collect usage feedback from customers. This feedback makes it easier for Apple to identify meaningful product improvements while paring back features that aren't used or are overserving. This may be one of Apple's biggest competitive advantages in designing/creating new products and improving existing products: the customer feedback loop that comes from making the "whole widget."

OEM's that rely on the Android OS, or Android software engineers who must rely on OEM's, don't make the whole widget and therefore don't get end user feedback about the entire product. And when Google and Android OEM's do get meaningful end user feedback, their ability to act on it is limited by the fact that Google must design/improve Android not just for end users, but for advertisers that pay Google to collect user data and target these users with ads. Android is designed/improved based on end user feedback and advertiser feedback.

The author owns stock shares of Apple.