Response Bias

How do you ensure your questions—and resulting responses—are on track with your survey goals? By taking a few preventative measures, you can avoid question/response bias in your surveys (see related article).

This tutorial will examine techniques for preventing response bias in your survey efforts, as well as present two critical construction principles of effective survey questions.

Techniques for Preventing Response Bias

There are many ways to prevent response bias in your surveys. Below are several key suggestions to consider when developing your survey questions.

Write questions that are clear, precise, and relatively short

Because every question is measuring something, it is important for each to be clear and precise. Your goal is for each respondent to interpret the meaning of each survey question in exactly the same way. If your respondents are not clear on what is being asked in a question, their responses may result in data that cannot or should not be applied to your survey goals. Keep questions short; long questions can be confusing and stressful for respondents (see related article).

Do not use “loaded” or “leading” questions

A loaded or leading question biases the response given by the participant. A loaded question is one that contains loaded words. For example, politicians often avoid the loaded word “environmentalist” because it creates a negative reaction in some people regardless of the content of the statement.

A leading question is phrased in such a way that suggests to the respondent that the researcher expects a certain answer:

Example

Don’t you agree that social workers should earn more money than they currently earn?
Yes, they should earn more
No, they should not earn more
Don’t know/no opinion

The phrase “Don’t you agree” leads the respondent. A more neutral wording would be:

Do you believe social worker salaries are a little lower than they should be, a little higher than they should be, or about right?
Social worker salaries are a little lower than they should be
Social worker salaries are a little higher than they should be
Social worker salaries are about right
Don’t know/no opinion

Avoid double-barreled questions

A double-barreled question combines two or more issues or attitudinal objects in a single question.

Example

Do you think professors should have more contact with university staff and university administrators?

Clearly, this question asks about two different issues: Do you think professors should have more contact with university staff? AND Do you think that teachers should have more contact with university administrators?

Combining the two questions into one question makes it unclear which attitude is being measured, as each question may elicit a different attitude. Tip: If the word “and” appears in a question, check to verify whether it is a double-barreled question.

Avoid double negatives

When respondents are asked for their agreement with a statement, double negatives can occur.

Example

Do you agree or disagree with the following statement?
Teachers should not be required to supervise their students during recess.

If the respondent disagrees, you are saying you do not think teachers should not supervise students. In other words, you believe that teachers should supervise students. If you do use a negative word like “not”, consider highlighting the word by underlining or bolding it to catch the respondent’s attention.

Construction Principles

Below are two critical construction principles you should apply to prevent survey bias.

Use both mutually exclusive and exhaustive response categories for closed-ended questions

Categories are mutually exclusive when there is no overlap:

Example

What is your current age?
10 or less
10 to 20
20 to 30
30 to 40
40 to 50
50 or greater

These categories are not mutually exclusive because there is overlap present. For example, a person who is 20 years old could be placed into two separate categories (same with those respondents aged 30, 40 and 50).

Categories are exhaustive when there is a category available to all potential responses. Below is an example of a question where categories are not exhaustive:

Example

What is your current age?
1 to 4
5 to 9
10 to 14
The categories are not exhaustive because there is no category available for respondents more than fourteen years old or respondents less than one year old.

Here is an example of response categories that are both mutually exclusive and exhaustive:

What is your current age? (Check one box only.)
Less than 18
18 to 29
30 to 39
40 to 49
50 or older

Reverse the wording in some of the questions to help prevent response sets.

A response set is the tendency for a respondent to answer a series of questions on a certain direction regardless of their content.

One technique used to prevent response sets is to reverse the wording in some of the survey items. Below is an example of this in a rating scale question:

Please rate your manager on each of the following descriptive scales. Place a checkmark on the space between each pair of words that best indicates your opinion:

You can see that items 3 and 5 (with asterisks) are “reversed” when compared to the rest of the items, i.e., most of the left-hand descriptors are associated with positive attributes while the right-hand descriptors are associated with negative attributes.

Important: Avoiding response bias is key to the success of your survey project. Implementing the above strategies will ensure that your survey delivers valid data that you will be able to effectively apply to your survey problem.