Hints for Designing Effective Questionnaires

Robert B. Frary, 
Virginia Polytechnical Institute

The purpose of this article is to offer tips in designing quality questionnaires and on avoiding common errors. Some of the more prevalent problems in questionnaire development are identified and suggestions of ways to avoid them are offered.

Do keep the questionnaire brief and concise. Some questionnaires give the impression that their authors tried to think of every conceivable question that might be asked with respect to the general topic of concern. The result is a very long questionnaire causing annoyance and frustration on the part of the respondents resulting in non-return of mailed questionnaires and incomplete or inaccurate responses on questionnaires administered directly. To avoid this first potential problem the investigator must define precisely the information desired and endeavor to write as few questions as possible to obtain it. Peripheral questions and ones to find out "something that might just be nice to know" must be avoided. A clear-cut need for every question should be established.

Do get feedback on your initial list of questions. Feedback may be obtained from a small but representative sample of potential responders. A field trial of a tentative form of the questionnaire is also desirable.

Do locate personal or confidential questions at the end of the questionnaire. The early appearance of unsettling questions may result in respondents discontinuing the questionnaire.

Do order categories. When response categories represent a progression between a lower level of response and a higher one, it is usually better to list them from the lower level to the higher in left-to-right order, for example,

1) Never 2) Seldom 3) Occasionally 4) Frequently

Do consider combining categories. In contrast to the options listed just above, consider the following:

1) Seldom or never 2) Occasionally 3) Frequently

Combining "seldom" with "never" might be desirable if responders would be very unlikely to mark "never" and if "seldom" would connote an almost equivalent level of activity, for example, in response to the question, "How often do you tell you wife that you love her?" In contrast, suppose the question were, "How often do you drink alcoholic beverages?" Then the investigator might indeed wish to distinguish those who never drink. When a variety of questions use the same response scale, it is usually undesirable to combine categories.

Do ask responders to rate both positive and negative stimuli. There is sometimes a difficulty when responders are asked to rate items for which the general level of approval is high (the "apple pie" problem). There is a tendency for responders to mark every item at the same end of the scale. By offering positive and negative responses the respondent is required to evaluate each response rather than uniformly agreeing or disagreeing to all of the responses.

Do choose appropriate response category language and logic. The extent to which responders agree with a statement can be assessed adequately in many cases by the options:

1) Agree 2) Disagree

However, when many responders have opinions that are not very strong or well-formed, the following options may serve better:

1) Agree 2) Tend to agree 3) Tend to disagree 4) Disagree

These options have the advantage of allowing the expression of some uncertainty. In contrast, the following options would be undesirable in most cases:

1) Strongly agree 2) Agree 3) Disagree 4) Strongly Disagree

Some would say that "Strongly agree" is redundant or at best a colloquialism. In addition, there is no comfortable resting place for those with some uncertainty.

Avoid open-ended questions. In most cases open-ended questions should be avoided due to variation in willingness and ability to respond in writing.

Avoid the response option "other." Careless responders will overlook the option they should have designated and conveniently mark the option "other" or will be hairsplitters and will reject an option for some trivial reason. An exception to the foregoing advice is any case in which the categories are clear-cut, few in number, and such that some responders might feel uncomfortable in the absence of an applicable response.

Avoid category proliferation. A typical question is the following:

Unless the research in question were deeply concerned with conjugal relationships, the distinctions among all of these categories are not useful. Usually, such a question reflects the need to distinguish between a conventional familial setting and anything else. If so, the question could be:

Marital status: 1) Married and living with spouse
2) Other

Avoid scale point proliferation. In contrast to category proliferation, which seems usually to arise somewhat naturally, scale point proliferation takes some thought and effort. An example is:

1) Never 2) Rarely 3) Occasionally 4) Fairly often 5) Often 6) Very often 7) Almost always 8) Always

Such stimuli run the risk of annoying or confusing the responder with hairsplitting differences between the response levels. Psychometric research has shown that most subjects cannot reliably distinguish more than six or seven levels of response. Offering four to five scale points is usually quite sufficient to stimulate a reasonably reliable indication of response direction.

Avoid responses at the scale mid-point and neutral responses. The use of neutral response positions had a basis in the past when crude computational methods were unable to cope with missing data. In such cases, non-responses were actually replaced with neutral response values to avoid this problem. The need for such a makeshift solution has long been supplanted by improved computational methods. Consider the following questionnaire item:

The instructor grades fairly.

1) Agree 4) Tend to disagree
2) Tend to agree 5) Disagree
3) Undecided

There is no assurance whatsoever that a subject choosing the middle scale position harbors a neutral opinion. A subject's choice of the scale midpoint may result from: Ignorance, Uncooperativeness, Reading difficulty, Reluctance to answer, or Inapplicability.

In all the above cases, the investigator's best hope is that the subject will not respond at all. Unfortunately, the seemingly innocuous middle position counts, and, when a number of subjects choose it for invalid reasons, the average response level is raised or lowered erroneously (unless, of course, the mean of the valid responses is exactly at the scale midpoint).

In the absence of a neutral position, responders sometimes tend to resist making a choice in one direction or the other. Under this circumstance, the following strategies may alleviate the problem:

1.      Encourage omission of a response when a decision cannot be reached.

2.      Word responses so that a firm stand may be avoided, e.g., "tend to disagree."

3.      If possible, help responders with reading or interpretation problems, but take care to do so impartially and carefully document the procedure so that it may be inspected for possible introduction of bias.

4.      Include options explaining inability to respond, such as "not applicable," "no basis for judgment," "prefer not to answer."

The preceding discussion notwithstanding, there are some items that virtually require a neutral position. Examples are:

How much time do you spend on this job now?
1) Less than before 2) About the same 3) More time

The amount of homework for this course was
1) too little. 2) reasonable. 3) too great.

It would be unrealistic to expect a responder to judge a generally comparable or satisfactory situation as being on one side or another of the scale midpoint.

Avoid asking responders to rank responses. Responders cannot be reasonably expected to rank more than about six things at a time, and many of them misinterpret directions or make mistakes in responding. To help alleviate this latter problem, ranking questions may be framed as follows:

Following are three colors for office walls: 1) Beige 2) Ivory 3) Light green

           Which color do you like best?            _____

           Which color do you like second best?     _____

           Which color do you like least?           _____

By carefully evaluating the need of every question used in an instrument and carefully wording the responses, you will collect information which will yield more satisfactory and meaningful results.

Additional Reading

Dillman, D. A. (1978). Mail and telephone surveys: The total design method. New York: John Wiley.

Frary, R.B. (1996) Brief Guide to Questionnaire Development. Washington, DC: ERIC Clearinghouse on Assessment and Evaluation (30 pages)

Grunlund, N.E. (1993) How to make achievement tests and assessments. Needham Heights, MA: Allyn and Bacon.

Hinkle, D. E., Oliver, J. D., & Hinkle, C. A. (1985). How large should the sample be? Part II--the one-sample case. Educational and Psychological Measurement, 45, 271-280.

Descriptors: *Educational Research; *Questionnaires; *Research Design; Research Methodology; Responses; Scaling; *Test Construction; Test Format