The Malfunction of Evidence-Based Medicine
by Daniel L. Scholten
(OMNS, Jan 3, 2012) What follows is
an innovative new paper that we feel deserves publication. We concede
that this article is a stretch for OMNS in both size and content.
However, the nonstandard but thought-provoking aspects of this work need
presented and we choose to let our readers read or delete. - Andrew W.
Saul, OMNS Editor.
As part of their recent OMNS critique of the practice of "evidence-based" medicine (EBM) http://orthomolecular.org/resources/omns/v07n15.shtml
(1), researchers Steve Hickey and Hilary Roberts argue that the
legalistic requirements of EBM, such as its insistence on treatments
that have met the "gold standard" of "well-designed, large-scale,
double-blind, randomized, placebo-controlled, clinical trials", actually
prevent doctors from effectively diagnosing and treating their
patients. In this article, I would like to elaborate on this part of
their argument, which they warrant with a piece of cybernetic
common-sense (2) known variously as the "Good-Regulator"
theorem (GRT), or "Conant and Ashby" theorem, after the researchers who
original proof. (3)
No need to worry about the technical
jargon. If you can read these words then you have already understood
something important about this result from the system sciences, even if
you don't call it that. (4) Likewise, if you have ever used a street map
to navigate a new city, a book index to browse the contents of a book,
or perhaps an x-ray image or lab report to diagnose a patient's ailment,
then you are already quite comfortable handling at least the gist of
this conceptual power-tool, which can be paraphrased as follows: every
good solution to a problem must be a representation of that problem. (5)
What's It All about?
Here are several other ways to paraphrase the theorem:
- Every good regulator of a system must be a model of that system.
- Every good key must be a model of the lock it opens. (6)
- Control implies resemblance.
- Identical situations imply identical responses.
basic idea of the theorem can be illustrated with simple thought
experiments. (7) Just imagine trying to order a meal in a new restaurant
without using a menu, or assemble a piece of furniture without an
instruction pamphlet, or diagnose diabetes without a blood-sugar lab
report. Of course, you could probably muddle your way through any number
of situations with roughly the same basic set of skills that was
available to our preliterate ancestors, but the unassailable fact of the
matter is that maps, menus, x-ray images, and medical lab reports are
potent performance enhancers
and without them we risk getting lost, going hungry, or medically
Why is There a Problem?
truth of this can be easily obscured. One problem is that some
representations are clearly better than others. At the extreme we have
outdated maps, poorly written instruction pamphlets and menus with
mouthwatering images that turn out to represent bland, salty, or greasy
food. Another problem is that representations - from street-maps to MRI
scans - can be costly to prepare. Furthermore, the expertise required to
prepare or use them is costly to acquire, as measured by the years,
dollars, and brain-sweat it takes to complete one's formal education.
The upshot here is that those paying the costs of such representations
might reasonably wonder whether those costs outweigh the benefits.
Perhaps there is a cheaper way to enhance the performance of our system
regulators, to find "good solutions" to our problems, and "good keys" to
fit the locks we wish to open.
One common work-around is to rely on a
model." Although this approach works fine for simple tasks, such as a
quick stop at the grocery store to pick up extra milk, as soon as a task
becomes even moderately complex, the limitations of working-memory (10)
quickly render this approach useless, little better than using no
representation at all. Another approach is to simply avoid the sorts of
complex behavior that require us to use external representations. In the
end, we must all rely heavily on this approach, if for no other reason
than because the cost, time and effort required to learn how to use,
say, ultrasound imaging equipment, necessarily blocks one from
simultaneously learning to use, say, actuarial modeling techniques, or
perhaps the Hubble Space telescope. To choose is to renounce. But this
approach also has its limits and the total avoidance of such complex
behaviors - perhaps due to illiteracy, innumeracy
or maybe a deliberate decision to return to a preliterate
hunter-gatherer way of life - is just a different
sort of burden.
Yet a third way to dodge expensive
models or modeling expertise is to look for "multipurpose"
representations; for instance, generalized maps, menus, and user-guides,
that can be reused for many different cities, restaurants, and types of
equipment. (11) According to Hickey and Roberts, this third approach is
actually the one that EBM advocates.
One Key Cannot Fit All Locks
illustrate their argument with the above-mentioned lock-and-key
paraphrase of the Good-Regulator theorem. To follow it, we start by
making the analogy that a given patient's symptoms are a a "lock" the
doctor hopes to "open." It follows then, by the Good-Regulator theorem,
that the doctor's diagnostic and therapeutic behaviors must "model"
(represent) these symptoms. A critical qualification to be added,
however, is that the doctor must model these symptoms as they occur
within the specific context of the
patient's genotypically and phenotypically "characteristic anatomy,
physiology, and biochemistry." (12)
Of course, this does not mean that the
doctor must perform some outlandish Jim Carey-esque caricature of the
patient, perhaps donning the patient's same clothing, hairstyle, speech
patterns, behavioral mannerisms, etc. Rather, it means that the
associations that arise between the doctor's diagnostic and therapeutic
responses and the patient's symptoms must be characterized by the same
sort of conventional reliability that holds between the splashes of
color on, for example, a map of Manhattan and the real streets, parks,
and buildings in the actual city of Manhattan.
If that splash only occasionally
represented Lincoln Center - or if it sometimes represented Central
Park, and sometimes, say, the South Street Seaport - you would surely be
confused. Even though one could use the same given splash on a map to
represent two or more real-world landmarks, common-sense and strong
cultural conventions require each given color splash to reliably
represent just one particular real-world location. As established by
Conant and Ashby's Good-Regulator Theorem, a doctor's responses must
have the same sort of reliable association to a given patient's
symptoms. This reliability allows us to construe the doctor's responses
as a representation or model of the patient's symptoms. (13)
"Evidence-based" medicine (EBM), with its insistence on treatments that
have been confirmed by "well-designed, large-scale, double-blind,
randomized, placebo-controlled, clinical trials" (14) will almost always
cripple a doctor's
ability to model symptoms as they actually occur within the
anatomically, physiologically, and biochemically specific context of a
given patient. By way of analogy, we might consider a whimsically
allegorical "evidence-based locksmith" (EBL) attempting to open a
particular lock with the latest and
greatest "Whiz-Bang EBL Master Key," recently developed in accord with
results determined by a meta-analysis of hundreds of "well-designed,
large-scale, double-blind, randomized, placebo-controlled clinical
trials." Those trials have determined the absolute critical attributes
of the perfectly average key, and the patently absurd claim is that the
Whiz-Bang Master Key, by virtue of its perfectly average attributes, can
now be used to open any particular lock.
Pretty silly, isn't it.
Clearly such a perfectly average key
would open very few locks, if any. To reason otherwise is to commit the
"ecological fallacy," which Hickey and Roberts summarize as "the
assumption that a population value...can be applied to a specific
individual." (15) If one tries to shove such a key into some particular
lock, twisting and pulling in an effort to force it, then that violates
the Good-Regulator Theorem, which reminds us that a
good key must actually fit the lock it's supposed to open, not some
other lock, and especially not some hypothetical perfectly average lock.
The same goes for actual medical practice.
EBM Stops Doctors from Effective Practice
still need scientific research and the data it presents.
Representations are potent performance enhancers. Just imagine what our
lives would be like without grocery lists, the periodic table of the
elements, and ultrasound imaging techniques. But however obvious and
abundant the evidence might be, medical judgment is impaired by an
apparent lapse of common sense. The practice of EBM may well be a
consequence of the legal system and pharmaceutical corporate bottom
line. In other words, money.
But whatever the cause of such
impairment, the limitations of real people, real illnesses and real
doctors point to the reality that EBM is DOA. The patient is not a
statistic. The treatment
should not be a statistic. Every good doctor must represent the patient.
(Daniel L. Scholten has a degree in
mathematical sciences and over 12 years of information technology
experience as programmer, analyst and consultant. He founded the The
Good-Regulator Project [http://www.goodregulatorproject.org],
an independent, volunteer research effort dedicated to increasing
public awareness and understanding of the crucial role played by models
and representations in the regulation of complex systems.)
Notes & Reference:
1. Hickey, Steve and Roberts, Hilary, Tarnished Gold: The Sickness of Evidence-Based Medicine, 2011, CreateSpace.
2. A more complete list of "mostly
self-evident" cybernetic principles, including the Good-Regulator
theorem, have been compiled by Francis Heylighen. See "Principles of
Systems and Cybernetics: An Evolutionary Perspective", available on-line
In his paper, Heylighen distinguishes
between Conant and Ashby's "Good-Regulator Theorem" and a "Law of
Requisite Knowledge", which states that "In order to adequately
compensate perturbations, a control system must 'know' which action to
select from the variety of available actions." Note that although
Heylighen distinguishes between them, he also states that these are
3. Conant, Roger C. and Ashby, W. Ross, 1970, "Every Good Regulator Of A
System Must Be A Model Of That System", International Journal of Systems Science, vol. 1, No. 2, 89-97.
4. Those of us who can read sometimes
take it for granted. Many don't have this luxury. According to a recent
UNESCO fact sheet, in 2009 more than 16% of the world's adults (793
million people) were illiterate, with more than 64% of these being
women. "Adult and Youth Literacy", UIS Fact Sheet, September 2011, no.
16, The Unesco Institute for Statistics. Available online at http://www.uis.unesco.org/FactSheets/Documents/FS16-2011-Literacy-EN.pdf
5. I have argued for the plausibility of
this paraphrase in Scholten, Daniel L., 2010, "Every Good Key Must Be A
Model Of The Lock It Opens: The Conant And
Ashby Theorem Revisited", available on-line at http://www.goodregulatorproject.org.
It is also congruent with an observation made by Herbert A. Simon:
"Solving a problem means representing it so as to make the solution
transparent"; Simon, Herbert A., 1981, The Sciences of the Artificial,
2nd edition, MIT Press, Cambridge, MA; as cited in Norman, Donald A.,
Things That Make Us Smart: Defending Human Attributes in the Age of the
Machine, pg. 53, 1993, Basic Books, New York, NY.
6. Scholten, ibid.
7. Although I believe that such thought
experiments are justified in the context of the present argument, their
use in general should not be taken lightly. After all, as notes James
Robert Brown, they have been used to refute the Copernican world
view. See, Brown, James Robert, 1991, The Laboratory of the Mind:
Thought Experiments in the Natural Sciences, Routledge, New York, NY;
page 35. See also, Brown, James Robert and Fehige, Yiftach, "Thought
Experiments", The Stanford Encyclopedia of Philosophy (Fall 2011
Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/fall2011/entries/thought-experiment/.
8. A critical distinction that can be
made between an idealized good-regulator model, which is really a
dynamic entity, and its "technical specification", or what we might call
its control-model. (Scholten, Daniel, L., "A Primer For The Conant And
Ashby Theorem", http://www.goodregulatorproject.org).
Another distinction to be recognized is
that whereas the good-regulator model is dynamic, the control-model may
be either static or dynamic.
As an example of a static control-model,
consider a written recipe for roast duck, being used by an
inexperienced cook to prepare an evening meal for guests. In this case,
the system to be regulated consists of the various ingredients and
kitchen tools to be used to create the meal, the dynamic good-regulator
model is the human being doing the cooking, and the recipe is what we
are calling the static control-model. The recipe is a control-model
because the human being uses it, like a technical specification, to
guide (control) his behavior and thus to "turn himself
into" a good-regulator model.
As an example of a dynamic
control-model, consider the case in which a child learns to use an
idiomatic expression such as "two wrongs don't make a right" by
overhearing an adult use that expression in a conversation. In this case
the system to be regulated is a particular portion of some conversation
in which the child is participating, the dynamic good-regulator model
is the child, and the dynamic control-model is the adult role-model. The
idea here is that the adult's behavior serves as a type of dynamic
technical specification that the child then uses to control his or her
own behavior in the context of the given conversation.
It is important to make these
distinctions between a dynamic good-regulator model and its static or
dynamic technical specification because otherwise the GRT appears to
prove that the technical specification (control-model) is necessary,
which is, I
believe, a misreading of the theorem. The GRT only proves that the
good-regulator model is necessary. On the other hand, it does appear to
be an empirical fact that such technical specifications are also
necessary. The thought-experiments illustrate this explicitly, although
they also help us to see what our behavior looks like when we aren't
acting as good-regulator models.
(For an in-depth, authoritative analysis
of behavioral modeling, see Bandura, A., Social Foundations Of Thought
& Action: A Social-Cognitive Theory, Prentice-Hall, Inc., Englewood
Cliffs, New Jersey)
9. Let's recognize that one uniquely
human characteristic is our astonishing capacity to simulate (in the
manner of a Turing machine) the behavior of an enormous variety of much
simpler and more specific machines. I have written more extensively
about this in the "Three-Amibos Good-Regulator Tutorial," available
on-line at http://www.goodregulatorproject.org .
10 For a recent accessible discussion,
see Klingberg, Torkel, 2009, The Overflowing Brain: Information Overload
And The Limits Of Working Memory, Oxford University Press, New York,
11. I am making the assumption here that
the multipurpose model is meant to apply to cities, restaurants,
equipment, etc. that are not replicas of each other. Clearly there is no
problem if all owners of the same brand of laptop computer use the same
12. Hickey and Roberts, Tarnished Gold,
page 43. Hickey and Roberts emphasize that it is not simply the symptoms
that matter. Also important is the particular person in which those
symptoms occur, where the particularities of
that person have been determined by the complex interactions between
that person's genes and the environments in which those genes have been
expressed over the person's lifetime. In their discussion of this notion
of "biochemical individuality", Hickey and Roberts cite Williams, R.,
1998 (1956), Biochemical Individuality: Basis for the Genetotrophic
Concept, McGraw-Hill, New York.
13. In the words of Conant and Ashby
"...the theorem says that the best regulator of a system is one which is
a model of that system in the sense that the regulator's actions are
merely the system's actions as seen through a mapping...." Conant and
Ashby, 1970, pg. 96.
14. Hickey and Roberts refer to this ponderous, adjectival freight-train as the "EBM-mantra"; ibid, page 164.
15. Ibid, page 24. Hickey and Roberts attribute the term to Robinson, W.S., 1935, "Ecological correlations and the behavior
of individuals," Journal of the American Statistical Association, 30, 517-536.