Designing with “harmless failures” in mind

Ed Felten at Freedom to Tinker has some interesting points to add to Bruce Schneier’s piece on “Security Mindset” that I posted about yesterday. Felten focuses on the notion of “harmless failures.” It provides still more reason to approach all systems design problems with an eye firmly fixed on the social context in which your technology will operate.

The Security Mindset and “Harmless Failures”

…Not all “harmless failures” lead to big trouble, but it’s surprising how often a clever adversary can pile up a stack of seemingly harmless failures into a dangerous tower of trouble. Harmless failures are bad hygiene. We try to stamp them out when we can.

To see why, consider the donotreply.com email story that hit the press recently. When companies send out commercial email (e.g., an airline notifying a passenger of a flight delay) and they don’t want the recipient to reply to the email, they often put in a bogus From address like donotreply@donotreply.com. A clever guy registered the domain donotreply.com, thereby receiving all email addressed to donotreply.com. This included “bounce” replies to misaddressed emails, some of which contained copies of the original email, with information such as bank account statements, site information about military bases in Iraq, and so on. Misdirected ants might not be too dangerous, but misdirected email can cause no end of trouble.

The people who put donotreply.com email addresses into their outgoing email must have known that they didn’t control the donotreply.com domain, so they must have thought of any reply messages directed there as harmless failures. Having gotten that far, there are two ways to avoid trouble. The first way is to think carefully about the traffic that might go to donotreply.com, and realize that some of it is actually dangerous. The second way is to think, “This looks like a harmless failure, but we should avoid it anyway. No good can come of this.” The first way protects you if you’re clever; the second way always protects you.

Which illustrates yet another part of the security mindset: Don’t rely too much on your own cleverness, because somebody out there is surely more clever and more motivated than you are.

Rockwells Retro Encabulator

Geek/nerd humor. I’ve seen this before somewhere and I’m glad to see it again. I’ve certainly sat through my share of presentations like this. You wonder how many takes it took to get it right.

Funny: Rockwells Retro Encabulator

The yin to Common Craft’s yang:

I wish I knew more about this, the YouTube pages offer little info.
Thanks to Paul Ingram and Ryan Turner for the pointers.

Updated:  Here it says “This is a hoax video produced by Rockwell
for a sales meeting. See also:

Turboencabulator
” Thanks Bill

Tags:

Designing with failure in mind

Bruce Schneier is high on my list of smart people to pay attention to. His blog, Schneier on Security, always provides useful insights into the interplay between technology and people. Yesterday, he offered an interesting observation about what he labels “the security mindset.”

Schneier on Security: The Security Mindset.

….

Security requires a particular mindset. Security professionals — at least the good ones — see the world differently. They can’t walk into a store without noticing how they might shoplift. They can’t use a computer without wondering about the security vulnerabilities. They can’t vote without trying to figure out how to vote twice. They just can’t help it.

SmartWater is a liquid with a unique identifier linked to a particular owner. “The idea is for me to paint this stuff on my valuables as proof of ownership,” I wrote when I first learned about the idea. “I think a better idea would be for me to paint it on your valuables, and then call the police.”

Really, we can’t help it.

This kind of thinking is not natural for most people. It’s not natural for engineers. Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail. It involves thinking like an attacker, an adversary or a criminal. You don’t have to exploit the vulnerabilities you find, but if you don’t see the world that way, you’ll never notice most security problems.

I’d push his observations a bit farther. When you are designing and building systems that incorporate people and technology, you had better think about both how to make things work and about how things might fail.

Human systems are interesting and effective because they are resilient. Good designers allow for the reality of human strengths and weaknesses and factor both into their designs. Too many poor or lazy designers ignore or gloss over failure modes. How many project plans have you seen, for example, that assume no one on the project team will ever be out sick? And then management complains when the project fails to meet its deadlines.

There’s actually quite a lot of good material on failure in human/technology systems and how to compensate for reality. I’d recommend the following as good starting points:

Tags:

Ray Sims is collecting definitions of knowledge management

Combine two slippery but important words and it’s little wonder that you can find such a proliferation of definitions. For some reason, this reminds me of Danny Kaye’s Choreography number from White Christmas.

43 knowledge management definitions – and counting

Before I really get going on my day, here is an entertaining (or sobering) list of 43 knowledge management definitions – and counting from Ray Sims, who is heading back into the world of being an official KM’er as I head out to do product management.  It might have been funnier had he stopped at 37.

For many years I’ve been saying that I didn’t like the term “knowledge management” as (a) it was fundamentally an oxymoron, (b) there was no consensus within the industry as to what the term meant, and (c) in many companies the term carries negative connotations due to a perceived lack of value from earlier so-called knowledge management efforts and/or belief that knowledge management was a fad that we have moved on past or has been absorbed into other disciplines.

I like a number of these and have used variations of them in the past.  As someone on the Act-KM mailing list noted, there are easily as many definitions of knowledge.  Ray or another enterprising individual might want to stack these definitions into buckets about how “knowledge” is perceived by the people using the definition.  Process-centric definitions would look at knowledge-as-verb.  Storage-centric definitions might think of knowledge as a thing to be controlled.  People-connection definitions might think of knowledge as appearing via interaction. etc.  Ray has already created a couple tag clouds of the definitions.

Another great TED talk to watch – Jill Bolte Taylor’s Stroke of Insight

What a great way to start off a St. Patrick’s Day. This is certainly worth 20 minutes of your life. As someone inclined to spend entirely too much of my time inside the left-hemisphere of my brain, I found this especially affecting.  

Stroke of insight: Jill Bolte Taylor on TED.com

Neuroanatomist Jill Bolte Taylor had an opportunity few brain scientists would wish for: One morning, she realized she was having a massive stroke. As it happened — as she felt her brain functions slip away one by one, speech, movement, understanding — she studied and remembered every moment. This is a powerful story of recovery and awareness — of how our brains define us and connect us to the world and to one another. (Recorded February 2008 in Monterey, California. Duration: 18:44.)

Watch Jill Bolte Taylor’s talk on TED.com, where you can download it, rate it, comment on it and find other talks and performances.

Read more about Jill Bolte Taylor on TED.com.

NEW: Read the transcript >>

Video: Twitter in Plain English

Once again Lee and Sachi LeFever have produced one of their excellent video introductions to new web technology. This time their topic is twitter. By the way, if you want to follow my updates on twitter, I’m @jmcgee.


This 2.5 minute video is a result of feedback from our fans. We’ve received a number of requests from people who want their
friends to use the micro-blogging service Twitter, but can’t seem to explain it well. We hope this video helps.

If you’d like to follow us on Twitter, we’re @leelefever and
@sachilefever.