Question Design in Research

• After the 2008 Presidential election, a poll asked: "What one issue mattered most to you in deciding how you voted for President?" Two versions of the question were posed. The first gave the respondent five options to choose from, and the second was open-ended.
• From the five options offered, 58% of people chose the economy, 10% the war in Iraq, 8% terrorism, 8% health, and 6% energy policy.
• The open-ended question saw 35% of people choose the economy, 6% terrorism, 5% the war in Iraq, and 4% health care. No one chose energy policy. Other responses recorded in the open-ended question responses were candidate mentions, moral values, taxes, other political mentions, and change.
• The impact of defining the answers for participants in a study should not be under estimated, as the above example illustrates.
•  Harvard University provides a good tip sheet on the wording of questions in research. Pretesting questions on a sample audience is a good way of determining if they create a potential bias. Short questions often work best. The order questions are asked can also impact on results.
• The design of the questions in a survey is determined by the type of data that is required. If the research is looking to draw conclusions from a representative data set then closed questions are likely to be of more assistance.

Summary

• There is a considerable amount of information available in the public domain in respect of this information. There are also a number of examples of how question design can skew data.