  |
Chapter 2Chapter 2. Gathering the
Data
Research
is an adventure. It continually draws us onward to new dimensions of discovery
and analysis. It leads us into exciting new fields of inquiry and learning,
adding both breadth and depth to our understanding. But, on
the other hand, research is also hard work. It is patient, persistent,
painstaking "detective work," with numerous frustrations and false
leads. Good research has more to do with the shine on the seat of one's pants
than the shine in one's eyes. The good researcher is the one who perseveres,
who is distrustful of easy answers and quick solutions, and who knows that
things are not always what they seem—or what they are popularly believed to be.
The good
researcher is concerned to get at the pertinent data and let those data,
insofar as that is possible, speak for themselves. Here is where the adventure
really begins: one cannot know in advance where the data will lead. They may or
may not support the researcher's hypotheses or answer the researcher's
questions. They may, indeed, point to quite different and unexpected ones, ones
with which the researcher may be somewhat uncomfortable. But, as we have noted
elsewhere, good researchers seek, above all else, to be rigorously
intellectually honest. They do not ignore or cover over unwanted data and
disturbing conclusions. But
neither do researchers delight in unearthing "skeletons" and
publishing "facts," or interpretations of data, that may needlessly
damage the reputations of individuals and institutions. Research is never
morally or ethically neutral. Intellectual honesty is morally necessary, but it
is not the whole of morality. And so
the researcher works critically and honestly at finding the relevant data,
recognizing that even raw data do not exist independently of contexts.
Therefore, one carefully notes the full context in which the data occur. Data
are not detached propositions, floating about in some universal ether, complete
and final in themselves. All of
this impinges upon the gathering of data, but is not, in itself, the substance
of this chapter. Our concern here is with research methodology. That is, how
one goes about the gathering of data. In the following pages, we shall briefly
discuss basic research techniques. This is not to suggest that good techniques
alone make one a good researcher. Research is more an art than a science, so
intuition is also important. But good techniques help. Poor techniques can
certainly cancel out good intuitions. Books
and Articles In an
academic institution, the most obvious place to begin collecting data pertinent
to one's field of inquiry is in books and articles. Even though we live in the
computer age, print libraries are still the richest source of information
readily available to us. So, become familiar, even friendly, with the libraries
available to you, browse along the periodicals shelves, in the stacks, and in
the reference areas. Experiment with online catalogues, microforms and
microform readers, and other electronic aids. (Note Appendix A,
"Theological Library Resources at Anderson University.)" But
above all, become acquainted with the professional library staff and their
assistants. Not only are they paid to assist you in your research—among many
other duties—but are quite happy to do so. Their suggestions can frequently
save you hours of work. They are often our most under-utilized resource in the
gathering of research data. Professional
journals are one of the richest sources of information for the
researcher. A most important source is Religion
Index One: Periodicals published by the American Theological Library
Association (ATLA). This volume contains author, title, and subject indices,
abstracts, and books reviewed in periodical literature for a given year.
However, if time is more important to the student than is money, then the
student can have a reference librarian assist her or him in the computer database search of the Religion
Index (see Appendix B, "Literature Searching by Computer"). Several
other databases are available as well and new ones continue to be added.
Database searching can be quite costly, but if adequately planned can save the
student days of manual searching. Many
times, books and articles that look most promising are not to be found in
Anderson University Library. However, reference librarians can usually quite
quickly determine where these materials are to be found and request them
through interlibrary loan (see Appendix
A, "Theological Library Resources at Anderson University"). Documents Archives
and personal documents are of particular value to those engaged in certain
kinds of historical research. These are not always readily available to the
researcher and often restrictions are placed on their use, such as selected
access only under supervision. Archives
Particularly is this true of archives. Unlike regular library materials,
archival material does not circulate. And generally it must be used on site
under the supervision of an archivist or archival assistant. Further, some
archival material is not available to the ordinary researcher, primarily
because it is "official and confidential." This can prove to be quite frustrating to the researcher, who "knows"
that to which he or she is being denied access is exactly what is needed to
further the research and cannot be obtained elsewhere. Little can be done about
it, however. But,
occasionally, even when the archival material is legally public material, rather than private,
the researcher may still be denied ready access to it. In discussing this
problem, David C. Pitt notes that the researcher not infrequently has to deal
with officials "who have a deep-seated suspicion that the academic is an
iconoclast whose main function is to discredit the establishment"
(1972:36). Anderson
University Library houses the official archives of the Church of God. These are
public archives and generally
available to those doing research. Access is not unlimited or unrestricted,
however. Archives hours are posted at the entrance to the Archives. These are
limited to a few hours each day (Monday through Friday) during academic
sessions. Further, much of the material is available only on specific request,
to be used under archival staff supervision, and none of it can be removed from
the premises. Photocopy machines are available in the Archives. Even
with these restrictions, archives are the best sources—and often the only
sources—of certain kinds of historical data. Before undertaking a research
project requiring this kind of historical data, students should acquaint
themselves with the archives and what is available there. Personal Documents It may
be, however, that even when the archives does not have the primary documents
needed for the kind of data the researcher wishes to collect, they are
available elsewhere. On occasion, the researcher will be able to uncover
primary sources of data quite unknown publicly up to that point. The most
likely sources of such "treasures" are individuals who have retained,
or who have knowledge of, private
letters, journals, or diaries of family members or friends. These
are generally referred to as "personal documents." John Madge defines personal documents as documents
in which the authors "describe events in which they participated, or
[which] indicate their personal beliefs and attitudes . . . . In its narrow
sense the personal document is a spontaneous first-person description by an
individual of his own actions, experiences, and beliefs" (1965:76-77).
Behaviorists formerly considered the use of such documents to be
unacceptable, primarily because they are not "objective." Their essential "subjectivity," so
it was said, made them scientifically suspect. Social scientists and
historians, however, now generally accept personal documents as historically
valid (see Madge 1965:78ff). This is
not to say that such documents are to be taken at face value as literal
descriptions of what happened. But they do indicate personal attitudes,
beliefs, and feelings about what the writer believes to have occurred. This is
historically important information. And it may, indeed, lead the researcher to
question official or published versions of the same events. In such
cases, however, the researcher is obligated to seek independent verification of
the writer's information. If this cannot be done, the researcher must be very
cautious about deciding which version is true. Among the
various materials known as personal documents, personal and family letters are generally the most useful. They are
usually without the high degree of personal image consciousness and pose which
tend to characterize public and official letters. Particularly are private
letters helpful in determining the actions, attitudes, and opinions of their
writers—which are not normally revealed in official or public letters. Such
"official" letters are often propagandistic, promotional, designed to
"sell" an idea or a program. They cannot, therefore, be implicitly
trusted. Journals and diaries are valuable
sources of data, but must be used with some caution. They both select events
and interpret them—often on the basis of "memory," or
after-the-fact—occasionally days or even weeks after-the-fact. So the
researcher must seek to determine how frequently journal or diary entries were
made. Daily? Twice-weekly? Weekly? Periodic? Under such circumstances, gaps are
bound to have occurred. Many of them may be quite critical. Further,
such material is often intended for "publication," if only within a
family. Thus, the material is scarcely free from pose. Particularly is this
true of memoirs or other kinds of autobiographical writing. Here, the memory
factor plays a crucial role. But in spite of these limitations, such personal
documents can provide a great deal of useful data, some of which cannot be
obtained elsewhere. Public Documents A third
frequently used documentary source of data is public documents. These consist of official records, minutes,
reports, accounts, newspaper reports, copies of speeches, pamphlets,
statistics, official histories, and case history records. It must be remembered,
however, that these are secondary sources, not primary
sources as we have been discussing above.
Documents such as annual reports and official histories should be used
with a great deal of caution. Writers of them are usually concerned to put the
best official or institutional foot forward. A healthy dose of skepticism—not
to be confused with cynicism—is the researcher's safeguard against gullibility
and, in the end, embarrassment over having published "facts" that
turn out to be misleading, if not untrue. In the case of these public
documents, a double dose of skepticism is wise. We are
not saying that such reports are deliberate concoctions of truths, half-truths,
and outright lies. But the pressures of public and institutional life seem to
generate a greater degree of self-delusion than is generally characteristic of
the rest of us. Most of us want our work and our institution to appear in the
best light possible, particularly when funding may depend on it. This—often
unwittingly—leads to omission and over-statement. In spite
of all of these handicaps, however, public documents are still potentially good
sources of research data. The wary researcher will use these data with caution,
however, unless they can be corroborated by other, less self-interested
sources. It is always wise when using such information not to take it too
literally, but to get "behind the scenes," as it were. In other
words, check and double-check. Interviews In the
social sciences, one of the most frequently used methods for gathering data is
the interview method. Sociology and
psychology in particular have used this method, structured in the form of case
studies (see Appendix C, "The Case Study Approach") or life
histories. This approach, as with most others, has its limitations—as Daniel
Yankelovich, a noted user of the method, admits (1981:55). In many
areas of research, however, the interview method is most useful, particularly
when combined with other methods. Frequently, when using questionnaires
or other field research instruments, follow-up interviews which selected
respondents can be fruitful indeed. And often they are quite necessary in
clearing up ambiguous responses. Good
interviewing is, of course, linked with good listening, something most of us do
not do well. Good listening is, in this case, intensive listening. Intensive listening is that in which we listen
not only to words, but at the same time observe carefully the emotional
reactions of the informant to what she or he is saying. What emotions and
attitudes are in evidence? How does the speaker feel about what is being said? Any
intensifying of emotion, any negative body language, evidence of uneasiness,
embarrassment, or resistance is significant. Avoidance techniques, such as
evasive or non-committal answers or changing the subject, are important clues
and should be noted carefully by the interviewer. These all form part of the
context within which the verbal data must be interpreted, if it is to be interpreted
at all fairly.
Intensive listening, then, involves careful observation, a method of data gathering to be discussed later. Such
observation can indirectly provide the researcher with a great deal of
information that perhaps could not be gotten at any other way. It can suggest
to the alert researcher other questions that need to be asked, other avenues
that ought to be explored.
Interviews should be planned well in advance and informants carefully
selected. Not everyone is equally well informed about the subject of your
research. Objectives should be well-defined. Know exactly what it is you want
to find out and do not be sidetracked by issues not related to your research.
And clearly differentiate between questions which call for information and
those which call for opinion. If this careful advance planning is not done, the
interview is not likely to be as useful as it could have been. It is
wise to plan well in advance for yet another reason. In interviewing, the
researcher is often dealing with problems of doubt, suspicion, and
defensiveness on the part of the prospective informant. If the inter-viewer
catches the informant at a bad time, or is late for the appointment, the
interviewer may experience abruptness, impatience, or resentment. What is
already a tricky situation is thus made worse. The informant is then not likely
to be as communicative or helpful as he or she normally would. If the
interview method is to be your primary method of data gathering, it would be
wise to spend time learning about it from those who specialize in its use. An
excellent source is The Dynamics of
Interviewing, by Robert L. Kahn and Charles F. Cannell. The authors provide
a most useful discussion of, among other things, motivational, psychological
and linguistic barriers in interviewing. James Engel also provides some useful
"rules" for interviewing (1977:121ff). As we
have noted, interviewing is a difficult, even "tricky," process. It
is doubtful that one can ever know too much about it or practice it perfectly.
But good preparation well ahead of time can enable the researcher to avoid
serious problems. It is wiser to spend time in attempting to anticipate problems than in having to remedy them
later. Questionnaires One of
the most frequently used—and abused--methods of gathering primary data is the questionnaire. The questionnaire is now
one of the facts of life in our society. Educators, market analysts, sales
firms, social services, civic and federal governments, pollsters, politicians,
and preachers all rely on them. And this in spite of the fact that most of the
questionnaires themselves are vague, muddled, and ambiguous. Thus, the
"facts" they are purported to yield can be no less. The
design, administration, and interpretation of questionnaires that yield
trustworthy data is very difficult, difficult because it is so complex. And
also because it is much more of an art than a science, so James Engel argues
(1977:71). The major difficulty in using the questionnaire, Engels asserts, is
"to minimize that ever-present problem of bias—errors and mistakes made in the communicative process which
cause the findings to deviate from the truth" (1977:71). The
questionnaire must then be designed in such a way that the ever-present problem
of bias is at least minimized. Unfortunately, so Engels laments, no formulae
exist by which this can be done. The questionnaire is simply "structured,
goal-oriented communication." And,
as with all other forms of communication, is an art rather than a science
(1977:72). Engels
then goes on to discuss types of questionnaires, their strengths and
weaknesses, and suggests general guidelines for questionnaire construction. The
researcher who plans to use the questionnaire method of gathering data should
heed Engel well here. Since the
validity of the information gathered from questionnaires depends upon the
respondents' understanding of the questions asked, the researcher must give
very careful and detailed attention to the writing of the questions. This will
take much more time than most students anticipate. Here again a writer's maxim
applies: "No such thing as good writing exists, only good
re-writing." How does
one know when questions need to be re-written? One test is to have your
research director read them. If she or he understands exactly what information
you are requesting, then the questions are probably clear—to academicians. A more
certain method is to pre-test the
questionnaire, using locally available respondents of the same general educational and social level as those who will
ultimately be filling out the questionnaires. If a question can be
misunderstood, it will be in enough instances to alert the researcher to the
existence of ambiguity in the question itself. Questions can then be re-stated
and re-tested to assure that the correction is not itself ambiguous. Ambiguous Questions As the
above discussion suggests, most problems in questionnaires come from
inappropriately worded questions. Ambiguous
questions are one of the chief culprits. A question such as, "Would you
say that the pastor's sermons are helpful?" is highly ambiguous. Helpful
in what way? It is conceivable that someone could answer, "Yes, it is the
only nap I get all week." Or,
"No, he keeps shouting and wakening me." The ambiguity
of such questions is intensified by the fact that they most often, as in this
case, call for a yes-no answer. To ask for a yes-no answer to a question that
is not itself a yes-no question adds to the confusion the respondent is already
experiencing. Yet, many questionnaires do exactly this. A
political opinion survey questionnaire in our possession consists of ten
questions, all of which call for yes-no answers. Yet few of them are genuinely
yes-no questions. In some cases, "Don't know" is a more appropriate
answer. And at least in one other, respondents may want to answer both yes and
no. "Is the President doing a good job?" In some areas, yes; in
others, no. But we have no doubt that this politician used the results of his
survey to score political points in Washington—even though the survey instrument could not possibly yield accurate
results.
Potential ambiguity lurks behind almost every word and sentence in any
communication. Ferreting it out is no easy task. But avoiding it is mandatory
if our questionnaire results are to be trustworthy on specific points. We must
then be careful to avoid vague and imprecise language—often introduced by such
words as "usually,"
"generally,"
"normally," or "often." We should also avoid "jargon" or in-house terms that
may not be familiar to the respondents (see Engel 1977:81). Such language
allows ambiguity to occur. We must remember, if it can be misunderstood, it
will be. Leading Questions A second
major problem in the wording of questions is the leading question. A leading question is one which makes it easier
for a respondent to give one answer than another. For example, "Wouldn't
you say that you are opposed to abortion?" The impulse of many respondents
is to answer "yes," simply because they are suggestible persons who
are sensitive to what the questioner, or interviewer, wants to hear. Indeed, the purpose of leading questions is either to find those who agree with the propositions implicitly contained within the questions or those who can be represented as agreeing. In either
case, the motive of the so-called "researcher" is dubious, if not
downright dishonest. Loaded Questions A third
problem question is the loaded question.
Like the leading question, the loaded question is one which makes it easier to
give one answer than another (see Engel 1977:82). The loaded question, however,
also asks the respondent to accept an assumption or set of assumptions that may
themselves be highly volatile or debatable. For example, "If God forgives divorcees,
is his forgiveness the same, in kind, as his forgiveness of other
sinners?" The issue here is not whether God forgives divorcees, but
whether divorcing is sinning. The
unwary respondent who answers this question with a "yes" or
"no"—as the "questionnaire" calls for—may be accepting an
assumption, namely, that divorcing is sinning, with which he or she really does
not agree. Wishing to affirm that God does forgive divorcees, the hapless
respondent must also affirm that divorce is sin, presumably against God. Loaded
questions are not uncommon in the survey instruments masquerading as research
questionnaires. But, as in the use of leading questions, loaded questions are
essentially dishonest. Their real purpose is to assert a questionable "truth"
and then to collect "reliable data" proving that a majority of
"the people" agree with the assumption. This is nothing short of
self-serving manipulation of persons and "facts." Sampling If
questionnaires are to be manageable in terms of time and cost, then it is
clearly impossible in most cases to send out many of them. This will, of
course, create a serious problem for the research project if the
"population" being studied is large. If the "population" is
very large, then it is physically impossible to include every potential
respondent in the survey. To solve
this problem, the researcher employs the technique generally known as sampling. Engel asserts that "a
sample will provide an accurate picture of the larger body within measurable error limits if the sample is properly chosen"
(1977:50). To be scientifically valid, the sample must be both representative and random, so Engel continues. He then discusses in some detail how
this representativeness and randomness is achieved and how the sample size
itself is determined (1977:51f). Since
sampling is based on probability theory and generally accessible only to
mathematicians, it is advisable that the researcher who is not mathematically
skilled consult with someone who is. It will not do simply to
"guesstimate" as to the composition and size of the sample. If it is
not both genuinely representative and random, the data gathered will not be of
any definitive value. Any results then reported as "facts" will be
largely fictitious. Observation One final
method of data gathering to be discussed briefly here is observation. This is an approach used chiefly by the social sciences,
anthropology in particular. We are not talking about just any kind of
observation, however. Rather, our concern is controlled, directed observation.
We are all aware that casual, everyday observation is quite unreliable, as John
Madge convincingly argues (1965:124ff). But observation as a research
methodology is not casual, everyday observation. Controlled,
directed observation may take one of two general forms: (1) detached
observation; and (2) participant observation. In detached observation, the
researcher is trying to observe what is going on with a minimum of
intervention. The problem is that without interruptions, the researcher cannot
ask questions or benefit from interaction. Most researchers get around this
problem, however, by keeping detailed notes, often in journal form, noting
questions that can be asked in follow-up conversations or interviews. The
advantage of this type of observation is that it leaves the researcher free to
concentrate on what is going on rather than on his or her participation in it. The
participant observer, on the other hand, deliberately seeks to enter the life of
a community and to participate as fully as possible in it. This helps to avoid
the distortion which will inevitably result from self-conscious behavior or
responses. And it often opens up avenues of inquiry and response which would
not be available to an "outsider." The
difficulty with participant observation, of course, is that it takes a great
deal of time—often years—to become an "insider," one with whom other
insiders can be themselves. Most researchers do not have that kind of time,
unless their degree program demands it. But if it can be done, it will yield
information and insights that really cannot be gotten at any other way. To be
maximally effective, however, observation as a general research methodology
must, of necessity, be combined with other methodologies. The interview method
is the one most commonly linked to it. Observations do need to be carefully
verified, however. Even the researcher engaged in control-led or directed
observation can all too easily be misled. Observers must ask questions of
actors in events in order to confirm or correct what they think they have seen
in the events observed. Beyond
data gathering, observation as a research method has yet another benefit. It
can be very useful in developing and testing research hypotheses. Many good
research ideas have been born in the minds of patient and careful observers as
they have engaged in the kind of observation we have been talking about.
Further, many inadequate hypotheses have been abandoned or radically modified as
a result of such observation. Conclusion By way of
summary, we have touched briefly on a variety of ways of gathering research
data: the use of books and articles; the use of documents, including archival,
personal, and public documents; interviewing; the use of questionnaires; and
directed or controlled observation. Generally, in any major research project,
such as a thesis, various combinations of these methods are employed. It is
important that the data we gather be as comprehensive and inclusive as
possible. It is for this reason that research topics must be carefully
delimited, otherwise too much has to be left out. By narrowing the research
topic sufficiently, however, one can limit the amount of data needed to
manageable proportions. In the gathering of data, a central concern
is that of accuracy, fairness, and honesty. For a researcher to eliminate from
consideration data which would alter the hypothesis or thesis with which she or
he has been working is neither fair, accurate, nor honest. Data gathering, in
other words, involves moral obligation
on the part of the researcher.
|