Month: February 2012

Rethinking data and decisions – Big Data’s Impact in the World – NYTimes.com

The New York Times recently ran an excellent overview of the evolving state of data analytics.

Big Data’s Impact in the World – NYTimes.com: “The story is similar in fields as varied as science and sports, advertising and public health – a drift toward data-driven discovery and decision-making. “It’s a revolution,” says Gary King, director of Harvard’s Institute for Quantitative Social Science. “We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.”

Welcome to the Age of Big Data. The new megarich of Silicon Valley, first at Google and now Facebook, are masters at harnessing the data of the Web – online searches, posts and messages – with Internet advertising. At the World Economic Forum last month in Davos, Switzerland, Big Data was a marquee topic. A report by the forum, “Big Data, Big Impact,” declared data a new class of economic asset, like currency or gold.

This is the latest iteration in the ongoing interplay between judgment and evidence in decision making. Which means it’s worth considering how this argument has been evolving over time and how new discoveries, technologies, and techniques could change the issues or cause lasting change in how we go about making decisions.

Probability theory traces its roots to a conversation between Blaise Pascal and Pierre de Fermat over how to divide the pot in a card game if it weren’t possible to finish the game. Could you estimate the relative chances of each player winning the game based on the current state of the game and use those estimates to fairly distribute the pot? In other words, how can you use the evidence at hand to make a better decision?

When I was getting an undergraduate degree in probability and statistics (a long time ago), the core issues centered on what inferences you could draw about the real world from limited samples. What kinds of errors and mistakes did you need to protect yourself from? What precautions were appropriate to keep you from going beyond the data? The tools would always find some pattern in the data and we were repeatedly cautioned to take care not to see things that weren’t there.

A few years later, in graduate school, I revisited the topic in various required methods and analytical tools courses. The software tools were more powerful and were still capable of finding the slightest hint of a pattern in the noise. The faculty offered their obligatory cautions, but I watched plenty of students wreaking intellectual havoc with their new power tools, spinning conclusions from the thinnest threads of pattern in the data. For every hundred MBAs who learned to run a multivariable regression, one might read Darrel Huff’s How to Lie with Statistics.

Today, the tools continue to become more and more powerful at teasing out patterns from the data. At the same time, the exponential growth in available data means that we aren’t sampling so much as we are searching for patterns in the population as a whole. What is the lag between the power of our analytical tools and our capacity to apply sound judgment to the results?

Here are some of the questions I am beginning to explore:

  1. How does statistical inference change as we move from small, representative, samples to all, or most, of the population of interest?
  2. How do we distinguish between patterns in the data that are spurious and patterns that reveal important underlying drivers?
  3. When is an arbitrary or spurious correlation good enough to support a business course of action (Amazon doesn’t, and probably shouldn’t, care why “other people who bought title X, also bought title Y.” Calling my attention to title Y leads the incremental sales; who needs a causal model?)
  4. How does our deepening understanding of the limits and biases of human decision making connect to the opportunities presented in “Big Data”? Here, I’m thinking of the work Dan Ariely on behavioral economics and Daniel Kahneman on decision making.

I would value pointers and suggestions on where to look next for answers or insight.

Defining Characteristics of Wicked Problems

I’m just wrapping up a course I’ve been teaching at DePaul’s School for New Learning on Understanding Organizational Change. I’ve grounded the course in a view of organizations as dynamic systems from the perspective of Jay Forrester, Donella Meadows, and Peter Senge. In the last few sessions, we’ve also been discussing the notion of Wicked Problems and the challenges they present in today’s organizational environment.

I introduced the following list of “defining characteristics of wicked problems” drawn from The Heretic’s Guide to Best Practices: The Reality of Managing Complex Problems in Organisations. I’m not yet finished with that book, although it is excellent so far. I’ll post a review when I’ve finished it. Here is their list:

  • There is no definitive formulation of a wicked problem. In other words, the problem can be framed in many different ways, depending on which aspects of it one wants to emphasise. These different views of the problem can often be contradictory. Take, for example, the problem of traffic congestion. One solution may involve building more roads, whereas another may involve improving public transport. The first accommodates an increase in the number of vehicles on the road, whereas the second attempts to reduce it.
  • Wicked problems have no stopping rule. The first characteristic states that one s understanding of the problem depends on how one approaches it. Consequently, the problem is never truly solved. Each new insight or solution improves one s understanding of the problem yet one never completely understands it. This often leads to a situation in which people are loath to take action because additional analysis might increase the chances of finding a better solution. Analysis paralysis, anyone?
  • Solutions to wicked problems are not true or false but better or worse. Solutions to wicked problems are not right or wrong but are subjectively better or worse. Consequently, judgements on the effectiveness of solutions are likely to differ widely based on the personal interests, values, and ideology of the participants.
  • There is no immediate and no ultimate test of a solution to a wicked problem. Solutions to wicked problems cannot be validated as is the case in tame problems. Any solution, after being implemented, will generate waves of consequences that may yield undesirable repercussions which outweigh the intended advantages. (Offering Britney Spears a recording contract is a classic example).
  • Every solution to a wicked problem is a one-shot operation because there is no opportunity to learn by trial-and-error, every attempt counts significantly. Rittel explained this characteristic succinctly, with the example One cannot build a freeway to see how it works.
  • Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions. There are no criteria that allow one to test whether or not all possible solutions to a wicked problem have been identified and considered.
  • Every wicked problem is essentially unique. Using what worked elsewhere will generally not work for wicked problems. There are always features that are unique to a particular wicked situation. Accordingly, one can never be certain that the specifics of a problem are consistent with previous problems that one has dealt with. This characteristic directly calls into question the common organisational practice of implementing best practices that have worked elsewhere.
  • Every wicked problem can be considered to be a symptom of another problem. This refers to the fact that a wicked problem can usually be traced back to a deeper underlying problem. For example, a high crime rate might be due to the lack of economic opportunities. In this case the obvious solution of cracking down on crime is unlikely to work because it treats the symptom, not the cause. The point is that it is difficult, if not impossible, to be sure that one has reached the fundamental underlying problem. The level at which a problem settles cannot be decided on logical grounds alone.
  • The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem s resolution. In other words, a wicked problem can be explained in many ways with each explanation serving the interests of a particular group of stakeholders.
  • The planner has no right to be wrong (planners are liable for the consequences of the actions they generate). Those who work with wicked problems (town planners, for example) are paid to design and implement solutions. However, as we have seen, solutions to wicked problems cause other unforeseen issues. Planners and problem solvers are invariably held responsible for the unanticipated consequences of their solutions.

Culmsee, Paul; Kailash Awati (2011-12-02). The Heretic’s Guide to Best Practices: The Reality of Managing Complex Problems in Organisations (Kindle Locations 2759-2839). iUniverse. Kindle Edition.