  |
Chapter 3Chapter 3.
Interpreting the Data
Having
collected a great deal of data—in many cases, much more than the researcher
will need for the task at hand—the next question is what to do with it. Raw
data do not come with interpretations attached. Indeed, the data themselves may
be quite ambiguous, that is, capable of being understood in more than one way.
The researcher then has the obligation of making coherent sense out of this
great welter of facts and ideas. Many
uninformed researchers, however, appear to assume that having collected the
desired information, all they need now to do is to arrange it in some kind of
order. And so, for example, we have a plethora of writings purporting to be
histories which, in fact, are merely chronologies. What the writers present is
merely raw data in chronological order. They fail to enter into critical engagement
with it, to analyze, to question, to suggest what it all means and why it is
important for us to pay attention to it. In one
way, the data are like the pieces of a jigsaw puzzle: they have to be put
together to form a picture. But in another way, they are unlike a puzzle. A
jigsaw puzzle can go together in only one way. In many, if not most, cases,
this is not true of the masses of data we collect. It is precisely this fact
that creates numerous problems for the researcher. James W.
Davidson and Mark H. Lytle, in discussing the writing of history, assert that
history is "something that is done, that is constructed, rather than an
inert body of data that lies scattered through the archives" (1982:xvii). In
other words, history is interpretation of events, rather than the events
themselves. And historians are the interpreters. Davidson and Lytle conclude: For better or worse,
historians inescapably leave an imprint as they go about their business: asking
interesting questions about apparently dull facts, seeing connections between
subjects that had not seemed related before, shifting and rearranging evidence
until it assumes a coherent pattern. This past is not history; only the raw material
of it (1982:xxix).
Interpretation is a complex and difficult task. It is that because the
interpreter's world inevitably intrudes into the interpretive task. Each of us
comes to that task with a different set of perspectives, presuppositions, and
life experiences that predispose us to understand things in certain ways. For
that reason, a common set of data is variously handled by various people. This is
not to say, however, that all readings of that data are equally valid. Nor is
it to say that the data themselves do not impose some categories and restraints
upon us. What it is saying is that those who handle research data have an
obligation to be aware of their own biases, to the extent that anyone can do
that. And, further, that they attempt to follow the data where they themselves
seem to go. And so,
to the task of interpretation. Following is a discussion of several areas of
concern that will, we hope, offer some guidance in this critical undertaking.
The categories of discussion are not exhaustive, by any means. But what we have
included are those which, in our experience, continue to trouble research
students. Selection
Generally our problem is not a scarcity of data, but too much of it. The
success of our data gathering itself usually dictates that we must select from
the mass of data that which will be presented and analyzed. The very act of
selection is the beginning of interpretation. Davidson and Lytle comment, ...the historian's simple
act of selection irrevocably separates "history" from "the
past." The reconstruction of an
event is quite clearly different from the event it-self. Yet selection is only
one in a series of interpretive acts that historians perform as they proceed
about their business (1982:3). If
selection is an interpretive act, the question then is, how do we go about it
in ways that do not force the data to speak too narrowly or in a voice not
their own? Two basic criteria should guide the research's selection of data to
be presented: (1) representativeness; and (2) pertinence. The
question is, is this part of the data representative of the whole or is it, in
some way, an aberration? It is possible to select only what fits one's theories
and ignore what does not. This is dishonest. The data selected should represent
the whole picture, not just that part of it which the researcher may wish to
highlight for apologetic or propagandistic purposes.
Secondly, what is selected must be pertinent to the objectives of the
research project itself. The researcher may uncover a great deal of fascinating
information in the course of research. But a good deal of it may lead the
researcher far away from the stated intentions of the research project. So,
select what enables you to achieve your stated objectives—and be sure that what you select is representative. Jacques
Barzun and Henry F. Graff state: "To be successful and right, a selection
must face two ways: it must fairly correspond to the mass of evidence, and it
must offer a graspable design to the beholder" (1985:198). Barzun and
Graff compare the research-er to a traveler who explores new country. In
selecting from the data, the researcher has no "synoptic view," with
all the facts clearly laid out in plain sight. He or she is rather an "explorer,"
who "forms his opinions as he progresses, and they change with increasing
knowledge." The selective conclusions of the researcher, however, are
always "conditioned" by two things, Barzun and Graff insist. First,
the researchers "temperament," which includes "preconceptions." And, second, "the motive or
purpose" of the research (1985:198).
Undoubtedly Barzun and Graff are correct in asserting that selection is
greatly affected by the temperament
of the researcher. But what exactly do they mean by that? The temperament of
the researcher has to do with his or her guiding ideas, intentions, and
hypotheses. In other words, Barzun and Graff conclude, the researcher's
"total interest." This interest
will determine discoveries, selection, pattern-making, and exposition
(1985:199). While
this apparently is the case—and we shall discuss it more fully below under the
heading of bias—the fact remains that selection need not be fully subjective
and arbitrary. If the criterion of representativeness is maintained, the researcher's
"interest" cannot fully control selection. If it were to do so, then
the data could say only what the researcher has decided they should say. And,
again, that is fundamentally dishonest. Bias The
extent, then, to which the research project and its results are determined by
the researcher's bias is a question warmly debated by scholars. A significant
part of the debate centers on what is meant by the term "bias"
itself. Barzun and Graff make a distinction between "good" and
"bad" interest. Bad interest is that which is uncontrolled, heavily
intrusive, and which leads to unfair or dishonest selection. It is this
"bad" interest which Barzun and Graff designate as "bias"
(1985:199). However,
this gives the term "bias" a bad name—and a name which it may not
deserve. According to Barzun and Graff, the historian, Edward Gibbon, was
"biased in favor of pagan Rome and against Christianity." We cannot, then, trust Gibbon to give us an
accurate account of early Christianity (1985:199). While this is probably true
of Gibbon, the pejorative use of "bias" constitutes a semantic
problem. In this context it appears to closed-minded and intolerant and that
mindset made it impossible for him to be fair in his judgments of it. Bias,
however, is not prejudice. A bias is simply a slant, an angle of vision—in this
case. Bias is thus an inescapable human characteristic, inasmuch as all of us
see things from a perspective, i.e., a particular angle. We can no more be
unbiased than we can be non-human. What is at stake here, of course, is the
tired and worn Positivist view of pure objectivity.
In the Positivist paradigm, "objective" and "subjective"
are synonyms for "true" and "false." The notion was that one could stand outside
one's humanness, personally disinterested, and totally objective. Such a human
being is a fiction. We are
all inescapably bound by our subjectivity, yet able to some extent to transcend
it intellectually. We cannot be unbiased, but we can fight against prejudice.
That is, we can endeavor to collect all of the relevant evidence and to
consider it fairly—even if the subject is personally distasteful. We cannot be
impartial, but we can be intellectually honest, as Barzun and Graff admit
(1985:200). This means that we will constantly put our subjectivities to the
test. It may,
indeed, be better to qualify the term "bias" and thus to redefine it,
rather than limiting it to a pejorative use. All of us are inescapably biased,
but we are not all biased in the same way. That is to say, biases may be
positive, or they may be negative. Better yet, they may be critical or uncritical. Critical bias is the recognition that
one cannot be disinterested, or neutral, or impartial. One has
"interest," to use Barzun and Graff's term, and that
"interest" colors research. But it does not control it to the point
that honesty and fairness are impossible. The researcher of critical bias will
endeavor to get at all of the pertinent data possible and be rigorously fair in
the handling of it. If that means changing one's mind or re-thinking one's
hypothesis, so be it.
Uncritical bias is the failure to recognize or admit the distortion of
perspective, the lack of self-awareness of one's personal perspective and how
that tends to force data into pre-formed boxes. The uncritically biased
researcher is unfairly selective, projects personal views on the data, and
ignores what could force modification of hypotheses. It is this kind of bias,
it seems, which is so often in popular "thought" equated with
prejudice. We must,
then, deal with bias on at least two levels: the bias of the researcher; and
the bias of the writers of books, articles, and documents. Particularly is bias
a critical problem in personal and public documents. The bias of any document
is determined by its character and function. What kind of document is it?
Public or private? Personal or official? What is the purpose of the document?
Descriptive or promotional? Polemical or conciliatory? As we have pointed out,
documents must not be taken at face value. The researcher is obligated to
determine their biases and take those into account in his or her interpretation
of them. Further,
it is important to consider who
wrote the document in question. Observer bias, as David Pitt calls it, is
always at work as well. That is, the observer is influenced by a great many
factors to write in certain ways, and not in others (1972:51). Who is it who
has done the writing? A supporter or a dissenter? A man or a woman? An elderly
person or a young person? What kinds of constraints were they under? What kinds
of emotional, physical, and psychological stresses were they experiencing? Bias is always
with us: in the books and articles we read; in the documents we study; and in
us ourselves. We cannot escape it. But
neither can we ignore it. If we refuse to recognize and own it, its power
over us is all the greater. It then functions in our research efforts as
uncritical bias and all that we do is skewed and distorted by uncontrolled
subjectivity. Constant self-awareness is therefore mandatory. Verification Data
must be selected and biases recognized and dealt with in preliminary stages of the
interpretation of the information we have gathered. A further preliminary step
should then be taken, namely, to verify the accuracy of the data. Barzun and
Graff have stated it well when they say, No [researcher] can hope to
unravel every mystery and contradiction or uncover every untruth, half-truth,
or downright deception that lurks in the raw material with which he must deal.
But his unceasing demand for accuracy must make him put to the test all the
materials he uses. There is no substitute for well-placed skepticism
(1985:144). The Confusion of Facts and Ideas A
library, Barzun and Graff go on to say, is "a sort of ammunition dump of
unexploded arguments." Every book,
every article, every document comes to us "dripping with ideas." They also, of course, contain a great many facts, but facts are seldom free from
interpretation and interpretations are ideas.
Interpretation will be quite inadequate until we recognize that facts and ideas
are two different things (1985:145). The
confusion of facts and ideas is very widespread. A newspaper in our possession
speaks of a Church of God preacher well-known to us as "one of the great
preachers of modern America." That
the person in question is a preacher is a fact;
that he is one of the great preachers of modern America is an idea. It is likely that very many, if
not most, church-going Americans would not agree with the writer of the
article. The idea, in other words, is
disputable. The fact is not. The
problem here, of course, is that the idea is presented as a fact. It is not
qualified in any way. It is as if the writer is saying, "I believe it,
therefore it is true. Just trust me, folks." The writer of the article, like many writers of other things,
obviously has not learned to differentiate between facts and their
interpretation. This
confusion of facts and ideas, or opinions, can, however, be so subtle that it
is difficult for the researcher to tell the two apart. An example taken from
Barzun and Graff well illustrates this. Charles Darwin's book, The Origin of Species, so Barzun and
Graff say, "did not immediately persuade mankind, but set off a violent
controversy that lasted twenty years" (1985:149). It is a fact that the
book occasioned a great deal of controversy. It is not a fact, however, but a
disputable idea that it was "violent" controversy. Darwin himself was
surprised that his ideas resulted in so little furor, particularly from the
Church. Such an idea is easy to overlook, but it is an idea nonetheless. Ali A.
Mazrui, in The African Condition, in
at least one instance makes this same error. He says, "With regard to the
size of the continents, it is quite amazing how far European ethnocentrism has
influenced cartographic projections over the centuries" (1980:3). His complaint
is that Africa is the second largest continent in the world, yet on the map, or
cartographic projection, most commonly used, Africa appears much smaller than
it really is. Further, Europe and North America appear much larger than they
actually are. Mazrui
would have us believe that the reason for these massive cartographic
distortions is European ethnocentrism. The whole Southern Hemisphere, he
appears to believe, is made to appear relatively small and unimportant because
those who made and standardized the maps were ethnocentric. Undoubtedly, to
some extent, they were. But is that the underlying reason why maps were drawn
as they were? Here the
alert researcher will "smell" an idea masquerading as a fact. Can
Mazrui's "fact" be verified? It is no difficult matter to check it
out. An hour or two in the library, browsing through materials on cartography
will soon substantiate the intuition that Mazrui has overreached himself. The
Mercator Projection, to which Mazrui is referring, was developed in the 16th
century by a Dutch geographer and cartographer, Gerhardus Mercator. It was
intended as an aid to navigation at high latitudes, not a "picture"
of the world. The resulting map distorts the size and shape of land areas
closest to the poles. Thus Greenland will appear at least as large as Africa,
even though Africa is really several times larger than Greenland. This
distortion has much less to do with European ethnocentrism than with the
"primitive" state of cartography in the 16th century. The
purpose of this lengthy illustration is to alert the researcher to the constant
need for verification of the materials he or she uses. One of the "red
flags" is this confusion of facts and ideas. This confusion is often difficult
to detect, but it is so frequently present that the researcher must be
constantly alert. Logical Fallacies Another
"red flag" is the occurrence of logical fallacies in our sources of
data. On occasion, what are presented as facts are logically flawed and cannot
therefore be accepted as truthful statements. High on this list is over-generalization.
A student was overheard saying, "Missionaries are boring
speakers." This is
over-generalization. Most would agree that some missionaries are indeed boring
speakers. Some, however, are not. And thus the statement as it stands is a
partial truth. Barzun
and Graff point out that this "overextended generalization," as they
call it, comes from two sources: (1) the inappropriate use of universals; and
(2) failure to think of negative instances (1985:156). The case of missionary
speakers is an example of the inappropriate use of universals. That is, in
generalizing from a single instance, or a few instances, to a whole
"population." In a subsequent conversation, the student who believed
missionary speakers to be boring admitted that he had heard only one—and that
when he was sixteen years of age. The
second cause of over-generalization is failure to think of negative instances.
Following is a statement taken from a church bulletin: "History records
that wherever there has been worship there has been music. Three thousand years
ago the Psalmist wrote, 'O sing unto the Lord a new song.'" Here we must ask, what history and where?
And what of all those who through the ages have worshipped without music? Music
is very often a part of worship, but not always. Therefore, the bulletin
statement as it stands is untrue. A
related logical fallacy is reductionism: "this is nothing
more than that." Frequently we
hear statements such as, "Sexual immorality caused the fall of Rome."
But the historical reality is much more complex than that. No single cause can
account for the fall of Rome, as credible historians well know. Such complex
reality cannot be reduced to a single level of analysis. Marxists, for example,
do this when they attempt to explain all social, political, and psychological
reality economically. Such realities are reduced to an economic base. A third
logical fallacy is known as begging the question. To beg the
question is to use an argument that assumes the truthfulness of what one is
attempting to prove. A common example of this is the use of biblical texts to
"prove" that the Bible is divinely inspired. One begins with the
assumption that the Bible is divinely inspired. Therefore when the Bible says
it is inspired—which is an over-generalization at best—that is sure proof that
it is. Such
arguments are convincing only to those who are already convinced on other than
evidential grounds. This is a very common fallacy, one for which the researcher
must be constantly alert. A fourth
fallacy is illusory correlation. David G. Myers points out that all of us
are, to one degree or another, susceptible to perceiving correlation between
events "where none exists" (1980:74). Martin Marty reports an
interesting example of this. California evangelist Bill
Bright blamed the Supreme Court's ban on school prayers for "crime, racial
conflict, drug abuse, the Vietnam war, sexual promiscuity, and the demise of
American family life." Bright, who says that the court took God out of the
schools, con-tends the question now is, "are we going to bring God back to
our schools" (1980:863)? Marty
goes on to point out that at the time of the Supreme Court decision only a
small percentage of California schools conducted "home-room devotional
services." At only 2.41 percent,
God—in Bill Bright's terms—really did not have much of a foothold in California
public schools in the first place. So just how the Supreme Court's decision
took God out of the schools and resulted in the moral demise of America is
something of a mystery. So
convinced are we that two events which occur at relatively the same time or in
close sequence must be related that we accept as fact that they are. We are, in
Myers' words, "disinclined to recognize chance occurrences for what they
are." Myers concludes: "The
difficulty we have recognizing coincidental, random events for what they are
predisposes us to perceive order even when shown a purely random series of
events" (1980:75). Given this possibility, researchers must then be wary
of authors and their correlations. A fifth
logical fallacy is false analogy. An American President asserts, as an economic
dictum, "The rising tide lifts all ships." Supposedly, if the wealthy become wealthier, the poor will less
poor. Apart from being pragmatic nonsense, the analogy used simply does not
exist. The effect of the rising tide on ships is simply not analogous to the
effect of economic growth on all personal incomes. Tides and economies are
incommensurable phenomena. A second
"gem" comes from the same source: "Giving up Star Wars to the
Russians would be like the British giving up radar to the Germans." Here again, the two situations cannot be
legitimately compared. Britain and Germany were at war; America and Russia were
not. Radar was an accomplished fact; Star Wars was not. And radar did not deter
attack; it only enabled the British to brace themselves for it and inflict more
damage on the invading German bombers squadrons. The analogy is false. Many
such examples of false analogy could be adduced. The Use of Statistics It is
particularly important to seek to verify statistical information. A great many pitfalls
exist in this area. Even noted scholars and writers occasionally blunder in
their acceptance and use of statistics. A noted journalist, Stanley Karnow, in
his syndicated column, sets out to argue statistically that Russia's communist
experiment is a dismal failure. His general argument is probably reasonably
correct. But his statistical method of getting there is shoddy, to say the
least. For
example, he states that in the decade 1972-1982, infant mortality in Russia
increased from 23% to 36%. In checking usually reliable demographic sources, we
discovered that in 1972, the infant mortality rate was 2.3% and in 1982 it was
3.6%. Were the decimal points inadvertently omitted? Or did Karnow conclude
that if one wants to talk about a decadal rate, the way to do it is simply to
multiply the annual rate by ten? But
statistical rates do not work that way. One cannot simply multiply an annual
rate by ten to get a decadal rate. The mathematical procedure is much more
complex than that. An annual rate of 2.3% actually translates to a decadal rate
of 25.53%. Statistically, the difference between 23% and 25.53% is significant.
Writers
who are careless or uninformed in their use of statistical information may well
be careless or uninformed in other areas as well. For this reason, the
researcher should take the time to check out the basic information on which
such mighty ideological castles are built. To be
sure, a researcher cannot be constantly "rediscovering America." Sometimes we have to rely on our sources. We
simply have no means of verifying the accuracy of their information. But too
frequently, researchers re-convey information that has little basis in fact.
This can prove embarrassing. Causation Another
problem area for the researcher is the whole question of causation. Assigning
causes to events is commonly done, not only in the sources we use, but in our
own thinking and writing as well. We are accustomed to saying—and
believing—that a caused b. For example, a local newscaster announces:
"There have been about 100 accidents since midnight. A thin layer of snow
on the roads is the cause of the problem." But is
it? If a thin layer of snow causes accidents, then theoretically anyone who
drives on it should have an accident. But that is not the case. Most drivers
take extra care, reduce speed, and try to avoid abrupt turns or stops. One
could then say that the snow is the necessary condition for the
accidents, to use Barzun and Graff's term (1985:185ff). But it is not, in itself,
the cause
of the accidents. When
events occur, a multiplicity of factors may be at work, some of them
discernable, some of them indiscernible. Causes are more likely to be chains of
events than any single event. So reports which inform us that "the
accident was caused by speeding," are really not to be believed. Speed may
have been a contributing factor, but many other factors, such as poor tires,
lack of driving skill, or poor visibility may also have played a major role. So
rather than hastening to assign causes, perhaps we should talk about the
"necessary conditions" for and the "precipitating factors"
of events. A church newsletter states: "Due to the pastor's illness, the
evening service was canceled."
Fact: the pastor was ill. Fact: the evening service was cancelled. But
is it then a fact that the pastor's illness caused the cancellation of the service?
No, it is not. Perhaps the unavailability of a substitute, or the pastor's
unwillingness to trust lay leadership with the service, or many other quite out
of sight factors, combined to cause the cancellation. The pastor's illness was
merely the immediate and precipitating factor. Barzun
and Graff conclude that "what history reveals to mankind about its past
does not uncover the cause (one or
more indispensable antecedents) of any event, large or small, but only the conditions (some of the pre-requisites)
attending its emergence" (1985:187). To argue, then, that "sexual
immorality caused the fall of Rome" is not only reductionistic, it is also
"monocausalism." The mono-causal fallacy is assigning a
single cause to an event—something too frequently done and often by people who
should know better. Generally, the causes of events are analogous to a tangled
ball of string. No event
is an isolate. It has a time depth greater than itself. The researcher must be
wary of writers who seem not to be aware of this and who are so sure they know
the "causes" of events. Nor should the researcher fall into the same
trap in interpreting the data he or she has collected. Events doubtless have
causes, but causes are complex indeed and cannot always be obvious and
understood. The researcher must not, therefore, seek to give the impression
that this is not the case. Inference David Pitt
notes that inference or "extended interpretation" is a method
historians use to "get around some of the problems raised by gaps and
deficiencies in the record" (1972:58). If a and b are true, then c must
also be true. Or, stated differently, if we know that all Abaluyia eat obusuma
and do so about noon, we can reasonably infer that any individual in the
society probably generally does so. To infer
is to derive or accept as a consequence, conclusion, or probability. If someone
were to say, "By alertness and hard work, any American can earn a good
living," what could we reasonably conclude concerning those who live in
poverty? They must surely be lazy or stupid or both. This is an inference drawn
from the statement.
Occasionally students say something like, "The person who makes
such a statement is implying that the poor are lazy or stupid." We do not know what the speaker was
implying, since we do not know his or her intentions. But the logical inference
of such a statement is indeed that poor people are lazy or stupid. Thus, their
poverty is their own fault; we have no responsibility for them. Pitt
admits that inferential conclusions are problematic at a number of points
(1972:58). But they are nonetheless very often useful in moving us to new
hypotheses. For example, D.S. Warner's views on sanctification differ
significantly from those of John Wesley and Wesleyans. We must remember that
Warner attended Oberlin College in 1865 and 1866, when Charles G. Finney was
president and professor of theology. Further, that Finney's views on
sanctification strongly influenced Oberlin students—and even after Finney
"retired," continued to do so through The Oberlin Messenger. By
inferring from these facts, we now tentatively conclude that D.S. Warner's
views on sanctification were quite possibly drawn more from Finney than from
Wesley. This hypothesis may, in the end, prove to be quite wrong, or at least
must be modified. But without the use of inference, one probably would not have
come to such a hypothesis in the first place.
Inference can mislead us, since we cannot infallibly know authors'
intentions (see Pitt 1972:58). The whole intentionalist argument—or fallacy,
according to many scholars—is a particularly vexing argument. Unless an author
specifically states his or her intention or aim in writing, it is best to avoid
language such as "The author's intention (or aim) is . . . ."
Nonetheless, inference can be useful, so long as we work in terms of
possibility-to-probability. Beyond this we dare not go without falling into the
trap of over-inference. Fallacious Reasoning Slippery Slope Arguments -A causes B, B causes C,
and so on to X. -X is undesirable (or
desirable). -Therefore A is
undesirable (or desirable). Pro Hominem Arguments -X believes y -X is knowledgeable, trustworthy, free of bias (an
authority). -Therefore y should be
accepted. Ad Hominem Arguments -X says y -X is unreliable-Therefore
we should not accept y. Appeals to Ignorance -We can find no evidence for the truth (or falsity) of x. -Therefore x is false (or
true). Adapted from Good Reasoning Matters: A Constructive
Approach to Critical Thinking, Little, Groark, and Tindale. |