Scoping the knowledge work data layer

I’ve been nibbling around the notion of the data layer of knowledge work for some time. We’re accustomed to talking about deliverables in knowledge work. I’ve argued about the problems of visibility in knowledge work and the benefits of making knowledge work observable. Lately, I’ve started to explore the realm of working papers and intermediate knowledge work artifacts.

Three broad layers are emerging in my model of the data layer:

  • Deliverables: Presentations, reports, infographics, e-books, or any other self-contained package designed to be left in the hands of a client or the public. The notion of a deliverable is a powerful organizing idea, although it sometimes implies that anything not a deliverable isn’t relevant.
  • Working papers: Spreadsheets, process flows, trip reports, design sketches, use cases, statistical models, simulations or any other intermediate work product useful to an individual knowledge worker or to a project team. Working papers may or may not be shared with others.
  • Notes: Everything else. Interview notes, reading notes, journal entries, outlines, mindmaps, whiteboards, marked up placemats from breakfast meetings, or any other external representation that helps a knowledge worker capture or elaborate an idea.


Since we’re thinking about data, we also have to deal with what metadata is necessary or useful to maintain. If you work down from the deliverables layer, the default choice is to group deliverables by project. If you do knowledge work for any length of time, project information grows complex. Any given project might be part of ongoing work for a particular client.

On the other hand if you work up from notes, other metadata questions surface. While you might be able to identify a project, notes and note taking often happen before you have a specific project in mind. Do you collect or preserve metadata about the source of a note—is a note about an interview with a client, your thoughts about something you’re currently reading, or connected to some ongoing topic of interest? ‘

Working papers can be carved out of bigger deliverables or bubble up from the notes layer as your thinking develops. Which suggests that these metadata requirements will be a blend of the layers above and below.

The point of collecting this metadata is to make the proliferation of materials in the data layer manageable. Extracting and reporting on metadata ought to make it easy to monitor the developing status of the collection of notes, working papers, and deliverables that comprise an active project. How many interviews have we completed? Have we tracked down the data for the regression analysis we are about to perform? How many uses cases have we identified? How many have been written? Reviewed?

Formality and Explicitness

For one person working on a single project, this seems to be an inordinate amount of fuss and worry about something you can keep track of in your head or informally by reviewing an inbox or browsing a file directory. For large scale efforts by large teams, it might pay to invest in full time staff and formal systems.

For the middle arena where we spend the bulk of our time, the temptation is to rely on informal methods and practices. That’s a mistake. A better choice is to invest some thought and effort into making these distinctions and introducing a modicum of formality.


I’ve worked on these notions in a number of place. There’s a lot of other, excellent, work in this realm; I’ll save that for another day and another blog post. This list is chronological:

Thinking about the data layer of knowledge work

Early in my education as a computer programmer I encountered Niklaus Wirth’s seminal Algorithms + Data Structures = Programs. The fundamental insight was that algorithms and data structures have to be fashioned in concert; a good choice of data structure can simplify an algorithm, a clever algorithm might allow a simple data structure.

An example from the pandemic environment we are all living through is working with exponential functions (an algorithm). You quickly learn that expressing the data as logarithms (a structural choice) greatly simplifies much of the analysis. Complex curves turn into simple straight lines.

If you’re a good engineer, computer programmer, or data scientist, you’re trained to think about these tradeoffs in a systematic way. In the realm of knowledge work, we have lost sight of this useful distinction. We spend the bulk of our time and attention talking about the equivalent of algorithms.

What is the opportunity to simplify or improve our effectiveness at knowledge work if we devote more attention to the data layer of knowledge work? What kinds of tradeoffs should we be looking for when doing knowledge work? What choices about how we organize and manage data might improve the quality or effectiveness of doing knowledge work?

Managing yourself as a knowledge worker – building guardrails

Working from home is revealing how much of our daily work is kept in check by guardrails we don’t see or think about. We’re struggling to explicitly handle and deal with stuff that the environment handled for us invisibly. I’ve written elsewhere about the value of making knowledge work visible to make it more manageable. This is further elaboration of that line of thought.

What’s guardrail? It would be anything in your environment that provides a constraint on how you work without interfering with the work. Examples include;

  • dedicated office space
  • dedicated personal computer in the office
  • work email address
  • work calendar with standing meetings scheduled
  • formal title and position in the organization
  • reporting relationship to a boss
  • specified working hours
  • work phone number and voice mail
  • team assignments

None of these things are exotic. We take them for granted and that’s the point. They help everyone “stay in their lane” and maintain focus on getting work done.

Being able to say “not my job” has been a time-honored prerogative of many an office drone and organizations functioned quite well. As you rise within an organization, there are fewer opportunities to claim “not my job.” As an executive much of your job is to define the jobs and the lanes. In well run organizations, executives also acquire support systems they can call on to handle that scope and responsibility.

Executives set agendas, which means they also set the boundaries in the environment that matter. If you accept that knowledge workers also function as executives, then one consequence is that knowledge workers are responsible for setting and managing their boundaries.

Carving out a lane is a more demanding task than staying in one.

If you’re a knowledge worker, you are only person aware of all the competing demands for your  attention. Matrix management is one technique to acknowledge and negotiate conflicting priorities. It also assumes that those managing a row or a column in the matrix know more than the individual knowledge worker occupying a matrix cell. I’m tempted to leave off the qualifier and visualize knowledge workers simply occupying cells.

To return to the perspective of an executive for a moment, executives contend with many distinct systems—leading their organizations, serving on boards of other organizations, collaborating with peer professionals, contributing to their communities, and the like.

Each of those systems operates on an implicit assumption that their priorities are preeminent. Which leaves you as an executive or knowledge worker as the only person in a position to reconcile competing priorities.

One element in that reconciliation is working out what to do about guardrails. We don’t tend to think about them if they are well designed. As a knowledge worker you get to lay down guardrails or choose to ignore them. Either way, the responsibility has to be yours.

This feels like a separate task from designing and managing your substantive knowledge work. Let me offer a simple example from my own work. In addition to my consulting work, I am teaching, I serve on several not-for-profit boards, and I manage several ongoing activities at our church. One of the things I’ve done to create guardrails between these responsibilities is  to use separate email addresses and discrete inboxes for each activity. I’m letting a simple feature of my technology environment help me define lanes.

This process of consciously seeking opportunities to define lanes or guardrails in the work environment is an example of what my friend Benn Konsynski describes as “cognitive reapportionment.” It is a component of being a knowledge worker in today’s environment.

We are all executives in a knowledge economy

Photo by Roberto Lopez on Unsplash

Peter Drucker is one of those intellectual heroes you end up with if you’re of a certain business/nerdish bent. What can you say about the guy generally thought of as one of the first management gurus who also observed that:

I have been saying for many years that we are using the word ‘guru’ only because ‘charlatan’ is too long to fit into a headline.

One of my favorite Drucker books is *The Effective Executive.* I did a fairly lengthy review a couple of years ago (Effective Executives Are Design Thinkers). His  argument is that executives must focus on being effective—on doing the right things—and not worry terribly much about whether they are efficient.

He would be bemused by today’s obsession with productivity and “getting things done.” He had no objection to doing things right; he simply thought it was a very distant second to doing the right thing.

Drucker is also credited with coining the term knowledge worker. Lately, I’ve been thinking that the term “worker” is misleading. A worker is someone who operates within the structure and guardrails laid down by those executives focusing on doing the right think. If we’re not careful, this absolves the worker from responsibility for choosing what constitutes “the right thing.”

How often have you heard someone claim that they are only a “worker bee?” Perhaps you’ve said it yourself at some point. This is an attempt to deny responsibility for effectiveness; to ask someone else to make the hard decisions about the right things to do.

What the knowledge economy does is to remove the distinctions between executive and worker. We are all both and own the problem of choosing the right things to do. Each of us needs to work out and continually update our list of right things to do; we are each responsible for becoming effective.

That’s much more demanding work than being efficient. Being efficient means optimizing within the context of a stable environment. Who wouldn’t like that?

The last few weeks have been a powerful reminder that we don’t operate in a stable environment. That has been true for some time. Now, it’s harder to pretend otherwise.

If we want stability, then we must create it from the choices we make.

This was Drucker’s insight about executives. Their first responsibility was to make conscious choices about what were the right things to do. What executives do is to create stability. And this responsibility now belongs to each of us.

Picking an organizational stack

I suspect my early experiences with organizations were similar to most. Most of what we encountered was pretty simple to see and understand. We were students in a classroom, there was a teacher in the front of the room and a principal down the hall. Add a librarian, a school nurse, and the cafeteria ladies and you pretty much had it all.

At the other extreme, we tried to get the phone company or the Department of Motor Vehicles to help with a simple problem and encountered bewildering complexity. Why organizations looked the way they do wasn’t a question that occupied much of my attention.

My path to organization change and design was by way of programming and information systems design. What I learned in systems design was that complexity was the enemy. You had to understand the complexity of the problem you wanted to solve for there to be any hope of crafting a solution. The biggest mistake you could make was to miss some essential element of the complexity. The second biggest mistake was to add complexity by accident with your solution.

My technology training and experience tackled complexity with notions of designed modularity and looking for the natural places to carve the overarching system into discrete pieces.  There’s a rich assortment of models for thinking about problems of organizational design. I’ve taught them and I’ve used them. What I’m working through now is whether I can articulate a stack of organizational ideas that might help keep the complexity under control when looking at a new problem.

One organizational stack that intrigues me is

  • Purpose
  • Power
  • Process
  • Practice
  • People

I like the alliteration and I think the layers are largely self-defining/self-explanatory. I can see how I might map what I observe onto layers and how I might navigate up or down the stack while exploring a design question.

I’m now curious whether this cursory perspective is enough to generate some reaction.

Tackling technology complexity with stacks

Last time out we talked about the idea of a stack as a simple metaphor for organizing and thinking about underlying complexity in technology or organizations. I thought it would be worth taking a look at some of the origins of fighting complexity in the technology realm that brought us here.

Computer software is among the most complex constructs of human creativity. Wrestling that complexity under control has occupied the attention of many smart people. Talking about technology stacks is a shorthand way of thinking about this complexity. We’ve touched on the problem of complexity. The other problem we have to address is change.

There are three core concepts from the systems design world that are worth understanding and adapting to the organizational realm. They all relate to the design question of how best to carve things up into reasonably discrete pieces. The world of software is all thought stuff. There are few external constraints to shape your designs.

Systems designers look at three concepts when they are evaluating design choices about ways to carve a big system into more manageable pieces:

  • information hiding
  • coupling
  • cohesion

Information hiding is the systems design equivalent of “need to know.” How do you keep what you reveal about a module to a minimum? Put another way, what secrets are useful to keep.

Coupling and cohesion are complementary concepts. Cohesion is a measure of how closely the internal details of a module fit together. Do we have a team where everyone knows their role and responsibilities or do we have a random collection of people moving in the same general direction.

Coupling measures the degree of connectivity between modules. Cars traveling the same highway are more loosely coupled than the cars making up a commuter train.

If you’re interested in digging deeper, I’ve added pointers to some of the underlying literature where these notions were worked out. Think of it as a bit of information hiding on my part. If you’re comfortable with this level of explanation, you’re done. If not, you have the path to where to go next.

Once you have a clean mental model of a stack of modules formed by applying notions of information hiding, coupling, and cohesion you have a strategy to cope with complexity with less risk of finding yourself overwhelmed. If you can get a reasonable answer to your question at the layer you see, then you’re done. If not, you work your way down to the next layer. There are few questions that will require you to dig through multiple layers to find an answer. There are fewer still that require you to keep every layer in the stack in mind to understand.

Next time we’ll see how we might translate this strategy from technology to organizations.

Pointers to background work.

Nygren, Natalie. n.d. “Missing in Action: Information Hiding.” Steve McConnell. Accessed March 7, 2020. .

Parnas, D L. 1972. “On the Criteria To Be Used in Decomposing Systems into Modules.” Communications of the ACM 15 (12): 6.

Stevens, W P, G J Myers, and L L Constantine. 1974. “Structured Design.” IBM Systems Journal 13 (2).

Turtles all the way done: hijacking stories to your own ends

Turtles all the way down
Terrapin stack

There’s an old parable that I first remember encountering in Stephen Hawking’s A Brief History of Time. It often surfaces in explorations of the collision between science and myth. In Hawking’s retelling, the story goes like this:

A well-known scientist (some say it was Bertrand Russell) once gave a public lecture on astronomy. He described how the earth orbits around the sun and how the sun, in turn, orbits around the centre of a vast collection of stars called our galaxy. At the end of the lecture, a little old lady at the back of the room got up and said: “What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise.” The scientist gave a superior smile before replying, “What is the tortoise standing on?” “You’re very clever, young man, very clever,” said the old lady. “But it’s turtles all the way down!”

With the recent resurgence in adherents to belief in a Flat Earth, this tale is offered as a cautionary reminder of the power of compelling myth in the face of plain old evidence. That’s an important reminder in a world buffeted by change. Whatever your commitment to reason and evidence, you only replace a story with a better story.

I want to see if I can hijack the turtles metaphor to my rational ends.

My interests are in connecting technology and organizational realities. I’ve used the obvious image of building bridges but that may be too simplistic to be helpful after everyone nods that a bridge would be nice to have. Perhaps a stack of turtles can help us navigate the space between.

The realm of computing and communications technology is still a relative newcomer in organizational settings. Learning to wrestle the complexity to the ground is a key task in developing comfort and proficiency with technology. One of the principal methods of getting technology under conceptual control is the notion of a stack; of discrete layers of technology the interact with one another in sharply constrained ways.

There is a well known observation about dealing with technology first offered by computer scientist David Wheeler that:

All problems in computer science can be solved by another level of indirection

When things become too complex, we hide the complexity underneath a new layer of abstraction. Tim Berners-Lee invents a scheme, the url, that hides the details of reaching out to another computer and grabbing a file or document. Pretty soon, we’re watching reruns of I Love Lucy in our web browsers.

The number of layers you might traverse in  any given technology environment can become quite deep. The design challenge in new technology often revolves around how cleanly you can define and navigate the stack of layers.

Which brings me back to my turtles. What if we extend the stack upwards from the technology into the organization? Don’t think of the process as a single step across one bridge. Think in terms of what new layers (or turtles) make sense to cross the span. Now your problem is one of working out how to move up or down a layer at a time.

You can’t win. Play anyway

Holloman AFB F-4 Phantom II

One of the first office jobs I held was the summer after my freshman year in college. I worked for McDonnell Aircraft Company. as a material accounting cost clerk. MDAC was a defense contractor that made F-4 Phantom Jets and the Gemini space capsule among other things. Today it is part of Boeing.

I was a very lowly cog in a large complex system. My job was to write up the accounting entries to make sure that the inventory systems were in sync with the actual inventory. Some auditor would go out on the factory floor and count how many left-handed wing tips were sitting somewhere along the assembly line. My entire job was to compare the auditors count of actual inventory with the number recorded in the accounting system and then write out the journal entry to adjust the number in the accounting system to match the number that the auditor had found on the factory floor.

It was as mind numbing as you might imagine. I wrote the entries by hand onto paper forms that had to be reviewed and approved by my supervisor and then sent off elsewhere to be punched onto punch cards and fed into the computer located three buildings away.

There is one entry that sticks in my head. The auditors reported that they had counted three Pratt and Whitney jet engines on the floor. The accounting system thought there were four. How do you lose a 16-foot jet engine? Not my problem and no one else seemed terribly concerned. I wrote up the journal entry and it happily flowed off into the system.

You don’t design and build fighter jets with a handful of smart people. You need a complex system of people and processes and technology working together to pull it off. People, process, and technology is one of those cliches that gets thrown around a lot in organizations. I didn’t know the cliche at the time, but I was living in the middle of its reality.

This was when I began to grasp the complexity baked into large organizations. There are two choices for how to play in this environment. One is to be content with occupying a particular niche in the greater whole. The other is to wonder how it all fits together. Clearly, I opted for door number two.

I chose the word “play” intentionally. There is another lesson that I found in my time as a clerk. You have to accept that systems don’t always match reality. You have to build in mechanisms for correcting when systems and reality fall out of sync. And those mechanisms have their own weaknesses. Complexity and perfection are not reconcilable. Order is always battling chaos.

When I was learning physics, we learned about the second law of thermodynamics and the notion of entropy. There was some nasty math involved but our teacher summed things up this way:

Life is a game. You can’t win. You’re going to lose. You can’t get out of the game.

That leaves you with the option of playing simply for the sake of playing. A good lesson for all seasons.

It’s all a product of design

During my senior year in high school, several of my classmates and I decided that we wanted to put on a play. That wasn’t something baked into the flow of the year at that point. Some years, there was a production; some years, not. We made the case to the powers that be and got permission to proceed.

We were an all boy’s school so the first order of business was to partner with one of the nearby girls’s schools. The ecosystem of private Catholic high schools in St.Louis was no different that any other universe of institutions; there were clear pecking orders of who was worthy to associate with whom. At that time, our school was at the top of the natural order of things.

As one of the producers, I successfully pushed the decision in the direction of a school that was seen by many as beneath several other options. I used to joke that this was a clever ruse to gain access to an environment where we would face less competition. Regardless, it was an early example of a decision that called for incorporating a collection of factors beyond simple technical considerations. Not that I recognized that at the time.

The other curious aspect of this production was the play we chose to stage: The Absence of a Cello by Ira Wallach. I had forgotten most of the details of this comedy until I started thinking about this piece. It is yet another example of seeds that grow in unexpected ways.

Wallach’s play is a comedy about a chemist who is seeking to leave his academic life for a corporate job. He and his family conclude that success depends on concealing all of the interests and eccentricities that define them in order to conform to their expectation of what corporate uniformity demands.

It was a cautionary tale that brains and creativity had no place in the business world. That seemed like a perfectly plausible hypothesis to an eighteen year old nerd. But this particular nerd has also never had a good record of accepting conventional wisdom at face value. More a record of running experiments on things–including organizational systems–to discover what did or didn’t work.

Thinking about The Absence of a Cello reminds me of some of my other reading preferences. I’ve mentioned my love of science fiction, for example, As I think of the kinds of stories that draw and hold my interest, I can discern a long history of stories about subverting systems rather than tearing them down.

One interesting side effect of reading without close oversight or supervision is that you encounter all sorts of ideas before you are theoretically mature enough to understand them. One central idea that I encountered without someone to argue that it was naive was that all systems–including organizations, economies, and cultures–grow out of design choices made by someone.

Design choices then spawn a history of subsequent events. While it can be tempting to view the stream of events as inevitable or a product of natural law or divine intervention, there was always a design decision at the outset.

This is a liberating perspective. Knowledge and experience teach you that change can be slow and frustrating. But from the stance of design, any change is also possible. If there was human agency in the original design decisions, then human agency can always make a new design decision and trigger a new stream of events.

Ritual by design

My first experience with ritual was as an altar boy when being a boy was a prerequisite and you had to memorize the responses in Latin. Others may have thought about the ritual aspects; I was mostly concerned with not tripping on my alb.

Ritual, and superstition, was a major component of life in the theater. Learning to say “break a leg” instead of “good luck” as an actor took the stage. Discovering what the “Scottish Play” was. Remembering to leave the ghost light on when you were the last one out at the end of the night.

As a techie, I did these things because that was what you did even if they seemed silly to my fiercely rational side. Fitting in and being part of the company was far more important than pointing out the essential illogic of these rituals.

Despite my fundamental skepticism I found myself drawn to organizations and settings that valued rituals. I started my professional career at Arthur Andersen & Co, part of what was once the Big 8 accounting firms. Gray suits and white shirts were the uniform of most days. Andersen was one of the professional service firms that invested heavily in training. I spent many days at their training center outside of Chicago as both student and faculty.

The activities that happened after classes wrapped up each day gradually chipped away at my skepticism. Andersen’s training facility had previously been a small Catholic college campus. It was far outside of Chicago proper. We were pretty much prisoners during the week but Andersen was shrewd enough to invest in its own liquor license to keep the natives from rioting. Classes were filled with practical lessons. Evenings were devoted to knitting people into the culture.

Smack me over the head enough times and I eventually catch on. Perhaps the “soft side” of organization is worth paying attention to. An MBA and a Ph.D. later and I finally grasp that it’s more effective to look at organizations as socio-technical systems. Which is an ornate way to say that both the people and the machines matter.

At the outset I viewed this through the lens of an anthropologist. Organizational culture and the methods and rituals that bound people together were objects of study. They were aspects of the organizational environment and had come into existence through the passage of time. You could nudge things at the margins if you worked hard enough at change management.

I had an odd lesson on organizational stability and flexibility early in my career. The theater group that occupied all of my free time and a good bit of my class time in college was about 90 years old at the time. Shortly after I graduated I was asked to serve as a trustee of the group. Instead of the four years that most people spent connected, I spent an additional ten. I saw a curious thing about history vs. tradition. For students, whatever had happened during their few years in the club was all of history. Traditions just sort of existed; they were only loosely connected to the longer history in students minds.

That came back into play when I was part of the founding group of a consulting firm. We had no history but came from places where history and tradition was a key element of success. We understood that we were not simply growing a new business, we were laying down the experiences and stories that would become our traditions and our rituals. We could be intentional about what stories we chose to celebrate or to ignore.

One example comes to mind. We spent most of our time on the road at client sites. We had virtually no infrastructure of offices to hang out in between assignments. Fortunately for our economics, we also had little down time between assignments. You worked with your team during the week and you went out with your team after work in whatever city you happened to be in. But you might not meet your colleagues on other teams for months. They would only be an email address or a disembodied voice on a conference call.

We were smart enough to invest in monthly All Hands Meetings where we flew everyone into Chicago for a meeting to talk about the business and our work. While there was always a formal agenda, we were more interested in providing space for conversations between and after the official meeting. And we were always on the look out for stories to share.

But the clever thing that happened was an end of the day ritual that took place after work at client sites. There was an online trivia game available in many sports bars at the time. It had a leader board that showed who doing well and it showed the leaders across all the bars and restaurants playing the game that night. The game allowed six characters for a name. Our project teams would agree on a time to play (adjusting for multiple timezones) and they would choose a team name that was “GEM” (short for Diamond, which was the firm’s name) plus the airport code for the client city. So the team working in Wichita would be GEMICT and the team on Wall Street would be GEMLGA. We had pretty smart people on our teams and the goal was to see how many slots on the leader board would belong to GEMXXX teams. If your team grabbed the top spot for the night, we picked up the bar tab. Small outlay, but important bragging rights for the next All Hands Meeting.

The final observation I would make is that we learned not to force these things. Instead, we looked for ideas from the field that we could amplify. Learning to discern what could be amplified versus what would be rejected took time.