Repeatable Processes and Magic Boxes

There is a trap hidden in most efforts to create repeatable processes and systems that try to guarantee predictable results in doing knowledge work. To avoid that trap, you must learn to recognize and manage the magic box.

The promise of repeatable processes is to identify, design, and sequence all of the activities that go into producing a specified output. It is the core of industrial logic. For industrial logic to turn out uniform results with predictable quality, craft must be transformed into proven systems and repeatable processes. Variation must be rooted out and eliminated. To do this, all of the design thinking necessary to produce the product must be extracted from the process and completed before any further work can be done. You get the prototype product done right and then you churn out a million replicas. For this strategy to work, all of the magic must take place before the first process step ever occurs.

Knowledge work cannot be forced into an industrial straightjacket. The essence of knowledge work is to produce a bespoke result. I have no interest in the strategy McKinsey developed for its last client; I want the strategy that applies to my unique situation. I do not need the accounting system tailored to GM’s business; I need an accounting system matched to my organization. And therein lies the rub. I may want a bespoke accounting system, but I’m not willing to pay for it. Forty years ago, in fact, I had to pay for one anyway because off-the-shelf accounting software didn’t exist. All software development was custom; as was all consulting work.

Consultants and software developers are not stupid people. They could see that, despite the necessity of producing a unique product at the end, much of their work had elements of the routine and predictable. To increase the quality of their work, to train new staff, to improve their economics, and to better market their cumulative experience, knowledge work organizations worked to transform their practices into repeatable processes and methods. New industries were created as software developers redesigned their code to segregate what needed to be customized from what constituted a common core.

In this effort to apply industrial logic to what was fundamentally creative work, most organizations were sloppy or short-sighted about managing what was a fundamental tension between industry and craft. What was big, and shiny, and marketable was the packaging of cumulative experience into a consulting methodology, or a software development process, or a customizable software product.

What could not be eliminated, however, was the essential craft work necessary to employ the methodology or to customize the product. What happened was that this craft work was pushed into a box somewhere on the process map or into a line item in the standard workplan.

This is the “magic box.” Whatever its name or label, it is the step where the necessary creative work takes place. This is work that cannot be done until the moment arrives.

Why does it matter to identify which boxes in the process require magic? Because they determine the quality of the final result. The other boxes only matter to the extent that they set you up to succeed in the magic box.

How do you recognize the your are dealing with a magic box? What are the clues that differentiate it from among all the other boxes? Sometimes you must approach this by process of elimination; many boxes—“conduct field interviews”, for example—are more easily identified as not possibly containing magic. As for positive identifying features, look for language that suggests design thinking steps or analysis that isn’t tightly specified. “Develop market segmentation approach”, or “design chart of accounts” are examples of possible magic boxes.

Understanding which boxes are which in a process is essential to managing the process effectively. Regular boxes can be estimated and managed more tightly than magic boxes. Data collection, for example, is usually straightforward; analyzing it, requires more flexibility and adaptability. Collapsing these related, but distinct, activities into a single step would be a poor project management decision. Just because creative work cannot be controlled in the same predictable way as industrial work does not relieve managers of their responsibility to make effective use of limited resources. Being clear and isolating the magic boxes from the ordinary ones is essential to making those resource deployment questions.

Repeatable processes are often marketed and sold as “proven approaches” that eliminate the trial and error that the uninitiated risk if they strike out on their own. This has enough truth to be dangerous. Traveling in new terrain is safer with an experienced guide. The guide may help you get to where the underlying geology is promising, but cannot guarantee that you will strike gold. Honest guides will emphasize the distinction. But it is a distinction that is only meaningful to those who can hear it.

Trial and error is an unavoidable feature of creation. A serendipitous error is often the seed of a creative solution. Understanding where the magic needs to occur helps you distinguish between unproductive and potentially productive error. Unproductive error is an opportunity for learning and process improvement; it should be carefully reined in. Potentially productive error must be permitted and encouraged. That can only be done effectively by understanding where the magic needs to occur.

Some related links:

Culture, Process, and Practice – Effective leverage for Enterprise 2.0

The discussion about organizational culture in knowledge management and Enterprise 2.0 efforts is evolving in useful and pragmatic ways. The earliest discussions ignored culture entirely and implicitly assumed that technology would magically shape the organization as needed. The next round of discussion identified sharing as a desirable global cultural characteristic. If you were in a sharing culture, all was good. Time would ultimately reward your virtue. Those with equal need but less virtue whispered of incentives and WIIFM (what’s in it for me?). Crass and vulgar, but perhaps sufficient for most organizations.

In general these open ended discussions of culture are unsatisfying. They make gross assumptions on the basis of little data. Dave Snowden’s principles of knowledge management (see Rendering Knowledge) provide important pointers to a better answer with his emphasis on the behavior of individual knowledge workers.

Enterprise 2.0 as the extension of ERP

Michael Idinopulos of SocialText noted a shift in the debate at the recent Enterprise 2.0 conference in Boston. In a post titled The End of the Culture 2.0 Crusade?, he observed that:

Last week, the Enterprise 2.0 world turned a corner. Nobody pounded the table for cultural change. Nobody talked about incentives or change management. Nobody talked about transparency or modeling collaborative behavior.

Instead, people talked about process.

This is the most pragmatic shift in focus since the inception of Enterprise 2.0. It will have huge effects on the pervasiveness of social software in the enterprise, because it shows a clear path to the business value companies can realize from their implementations.

I ve been arguing for some time that social software achieves widespread adoption only when workers use it in the flow of work. Asking your colleagues to step outside their daily processes and tools to share what they know or network with others won t get you very far. (Trust me, I ve tried.) Bringing your colleagues collaborative tools and practices that make their daily processes better, faster, cheaper, and more interesting does work. It s all about process. Improve the process, you win. Don t improve the process, you lose.

This point of view squares with Ross Mayfield’s that broken business processes contribute to email overload and that

(Employees) spend most of their time handling exceptions to business processes. That s what they are doing in their inbox for four hours a day. E-mail has become the great exception handler.

While this is a very attractive position, it is incomplete in an important, and potentially dangerous, way. If you focus Enterprise 2.0 efforts solely on business process you may get scale, you will likely avoid the issue of culture, BUT you risk missing an opportunity for great leverage.

Leaving Process for Practice: Leveraging Knowledge Work as Craft

Focusing solely on Enterprise 2.0 efforts as an extension of existing business processes treats Enterprise 2.0 as a residual, clean-up, effort. It presumes that the goal worth pursuing is the efficient execution of well-defined processes. It reduces Enterprise 2.0 to an afterthought to ERP.

There is an assumption in this process-centric view that all relevant behavior can be reduced to business rules in the automated system. Exceptions are viewed as system failures or design failures. Moreover, failures in the system are attributed to resistance on the part of system users and operators. Instead of interpreting failures as resistance, what if we start by treating them simply as data?

If you start with process, your goal is uniformity at scale. Ideally, every situation should be treated in exactly the same way. This is eminently logical if you are calculating payroll withholding taxes. It is clever when you extend that logic to treating airline seats as a perishable commodity whose price might vary depending on the day of the week. Can you push in this direction forever?

Are there situations where uniformity and scale are not the appropriate criteria? Of course. The realm of art and craft is all about uniqueness and the value of limited scale. What happens if we opt to start from the art and craft end of the spectrum?

One choice is to accept the dichotomy. There is a class of problems suited to process thinking and a class of problems suited to art and craft. Another choice is to continue to force fit problems into a process view of the world. This has been a successful approach. Consider the number of problems that have been transformed into algorithmic problems well-suited to automation — inventory control and demand forecasting to name two.

A third choice is to ask how to improve art and craft without presupposing that uniform process is the goal. This was the approach started by Vannevar Bush, Doug Engelbart, and their intellectual heirs. This approach spawned much of the technology environment that we operate within today. Oddly, we’ve largely ignored their motives in creating this technology; to support more effective ways of wrestling with intellectual problems.

Engelbart, and those working on his agenda, started building new technology tools because our previous tools and methods were inadequate for the problems we had to solve. They built tools to support new kinds of problem-framing and problem-solving practices. We’ve adopted the tools without considering the practices and behaviors they were designed to enable.

Culture as the sum of shared behaviors

Those who lay the blame for failed efforts to introduce knowledge management or Enterprise 2.0 tools on organizational culture are picking up on this behavioral issue. They are doing so, however, at the wrong level of detail. Organizational culture is a convenient shorthand for the practices and behaviors that constitute "they way we work around here." Changing culture is hard because it’s an abstraction; there’s nothing to push against to get it to move.

While changing behaviors can also be hard, as anyone who’s tried to adopt a new habit can attest, it is possible. Change the right behaviors and eventually you have a new culture. The opportunity in Enterprise 2.0 technologies lies in the new behaviors that become possible. The challenge is that these technologies do not dictate a single set of obvious behaviors. In fact, it is possible to adopt the technologies and make no behavioral changes at all. 

Focusing on behavior is still a challenge. Technology opens up possibilities while setting few constraints. The activities we are interested in here are equally unconstrained by business process; we’ve defined them as the behaviors that don’t fit within the mantle of process. We need to go back to observing how work is currently being done, ask what flows smoothly, see where things get stuck, and design alternate ways to make use of the new range of available tools. We’ll visit those questions tomorrow.

LATE BREAKING LINK: As I was about to publish this post, John Tropea of Library Clips posted a lengthy piece on Have we been doing Enterprise 2.0 in reverse : Socialising processes and Adaptive Case Management. I’ve only just skimmed it, but it’s squarely part of this evolving conversation.