Espen reminded me of the following passage from Zen and the Art of Motorcycle Maintenance. It is a wonderfully succinct description of the scientific method and its power to protect us from the risks of wishful thinking when problems call for discipline. We used to use this passage as a piece of our basic training for all incoming consultants at Diamond.
…this morning I talked about hierarchies of thought–the system. Now I want to talk about methods of finding one’s way through these hierarchies–logic.
Two kinds of logic are used, inductive and deductive. Inductive inferences start with observations of the machine and arrive at general conclusions. For example, if the cycle goes over a bump and the engine misfires, and then it goes over another bump and the engine misfires, and then it goes over another bump and the engine misfires, and then it goes over a long smooth stretch of road and there is no misfiring, then it goes over a fourth bump and the engine misfires again, you can logically conclude that the misfiring is caused by the bumps. That is induction: reasoning from particular experiences to general truths.
Deductive inferences do the reverse. They start with general knowledge and predict a specific observation. For example, if from reading the hierarchy of facts about the machine, the mechanic knows the horn of the cycle is powered exclusively from the battery, then he can logically infer that if the battery is dead, the horn will not work. That is deduction. Solution of problems too complicated for common sense to solve is achieved by long strings of mixed inductive and deductive inferences that weave back and forth between the observed machine and the mental hierarchy of the machine found in the manuals. The correct program for this interweaving is formalized as scientific method.
Actually I have never seen a cycle-type maintenance problem complex enough to really require full-scale formal scientific method. Repair problems are not that hard. When I think of formal scientific method, an image sometimes comes to mind of an enormous juggernaut. A huge bulldozer–slow, tedious, lumbering, laborious, but invincible. It takes twice as long, five times as long, maybe a dozen times as long as informal mechanic’s techniques, but you know in the end you are going to get it. There is no fault isolation problem in motorcycle maintenance that can stand up to it. When you’ve hit a really tough one, tried everything, racked your brains and nothing works, and you know that Nature this time has really decided to be difficult, you say, “Okay, Nature, that’s the end of the nice guy,” and you crank up the formal scientific method.
For this you keep a lab notebook. Everything gets written down, formally, so you know at all times where you are, where you’ve been, where you’re going and where you want to get. In scientific work and electronics technology this is necessary because otherwise the problems get so complicated you get lost in them and confused and forget what you know and what you don’t know and have to give up. In cycle maintenance, things are not that involved, but when confusion starts, it’s a good idea to hold it down by making everything formal and exact. Sometimes just the act of writing down the problem straightens out your head as to what they really are.
The logical statements written down into the notebook are broken down into six categories: (1) statement of the problem, (2) hypothesis as to the cause of the problem, (3) experiments designed to test each hypothesis, (4) predicted results of the experiments, (5) observed results of the experiments and (6) conclusions from the results of the experiments. This is no different from the formal arrangement of many college and high school lab notebooks. But the purpose here is no longer just busywork. The purpose is precise guidance of thought that will fail if they are not accurate.
The real purpose of the scientific method is to make sure Nature hasn’t misled you into thinking you know something you don’t actually know. There is not a mechanic or scientist or technician alive who hasn’t suffered from that one so much that he’s not instinctively on guard. That’s the main reason why so much scientific and mechanical information sounds so dull and so cautious. If you get careless or go romanticizing scientific information, giving it a flourish here and there, Nature will soon make a complete fool out of you. It does it often enough anyway even when you don’t give it opportunities. One must be extremely careful and rigidly logical when dealing with Nature: one logical slip and an entire scientific edifice comes tumbling down. One false deduction about the machine and you can get hung up indefinitely.
In Part One of formal scientific method , which is the statement of the problem, the main skill is in stating no more than you positively know. It is much better to enter a statement “Solve Problem: Why doesn’t cycle work?” which sounds dumb, but is correct, than it is to enter a statement “Solve Problem: what is wrong with the electric system?” when you don’t absolutely know the trouble is in the electric system. What you should state is “Solve Problem: What is wrong with the cycle?” and then state as the first entry in Part Two: “Hypothesis Number One: The trouble is in the electrical system.” You think of as many hypothesis as you can, then you design experiments to test them to see which are true and which are false.
This careful approach to the beginning questions keeps you from taking a wrong turn which might cause you weeks of extra work or can even hang you up completely. Scientific questions often have a surface appearance of dumbness for this reason. They are asked in order to prevent dumb mistakes later on.
Part Three, that part of formal scientific method called experimentation, is sometimes thought of by romantics as all of science itself because that’s the only part with much visual surface. They see lots of tubes and bizarre equipment and people running around making discoveries. They do not see the experiment as part of a larger intellectual process and so they often confuse experiments with demonstrations, which look the same. A man conducting a gee-whiz science show with fifty thousand dollars’ worth of Frankenstein equipment is not doing anything scientific if he knows beforehand what the results of his effort are going to be. A motorcycle mechanic, on the other hand, who honks the horn to see if the battery works is informally conducting a true scientific experiment. He is testing a hypothesis by putting the question to Nature. The T.V. scientist who mutters sadly “the experiment is a failure; we have failed to achieve what we had hoped for,” is suffering mainly from a bad script writer. An experiment is never a failure solely because it fails to achieve predicted results. An experiment is a failure only when it also fails adequately to test the hypothesis in question, when the data it produces don’t prove anything one way or another.
Skill at this point consists of using experiments that test only the hypothesis in question, nothing less, nothing more. If the horn honks, and the mechanic concludes the whole electrical system is working, he is in deep trouble. He has reached an illogical conclusion. The honking horn only tells him that the battery and horn are working. To design an experiment properly he has to think very rigidly in terms of what directly causes what. This you know from the hierarchy. The horn doesn’t make the cycle go. Neither does the battery, except in a very indirect way. The point at which the electrical system directly causes the engine to fire is at the spark plugs, and if you don’t test here, at the output of the electrical system, you will never really know whether the failure is electrical or not.
To test properly the mechanic removes the plug and lays it against the engine so the base around the plug is electrically grounded, kicks the starter lever and watches the spark-plug gap for a blue spark. If there isn’t any he can conclude one of two things: (A) there is an electrical failure or, (B) his experiment is sloppy. If he is experienced, he will try it a few more times, checking connections, trying every way he can think of to get that plug to fire. Then, if he can’t get it to fire, he finally concludes that A is correct, there is an electrical failure, and the experiment is over. He has proved that his hypothesis is correct.
In the final category, Conclusions, skill comes in stating no more than the experiment has proved. It hasn’t proved that when he fixes the electrical system the motorcycle will start. there may be other things wrong. But he does know that the motorcycle isn’t going to run until the electrical system is working and he sets up the next formal question: “Solve Problem: what is wrong with the electrical system?”
He then sets up hypotheses for these and tests them. By asking the right questions and choosing the right tests and drawing the right conclusions the mechanic works his way down the echelons of the motorcycle hierarchy until he has found the exact specific cause or causes of the engine failure, and then he changes them so that they no longer cause the failure.
An untrained observer will see only physical labor and often get the idea that physical labor is mainly what the mechanic does. Actually the physical labor is the smallest and easiest part of what the mechanic does. By far the greatest part of his work is careful observation and precise thinking. That is why mechanics sometimes seem taciturn and withdrawn when performing tests. They don’t like it when you talk to them because they are concentrating on mental images, hierarchies, and not really looking at you or the physical motorcycle at all. They are using the experiment as part of a program to expand their hierarchy of knowledge of the faulty motorcycle and compare it to the correct hierarchy in their mind. They are looking at underlying form.
(Chapter 9. pages 107-111 from Pirsig, R. M. (1999). Zen and the art of motorcycle maintenance. 25th Anniversary Edition. Morrow.)