How to Ensure Survey Reliability and Avoid Garbage In/Garbage Out

In my consulting work we sometimes work with surveys to gather data for strategic plans or organizational assessments. In our 25 years of consulting, we have come across many Do-It-Yourself surveys. While we completely support DYI work in general, surveys are one place the DIYer needs to be careful. Without some thought, the DYI survey can become a “garbage in, garbage out” experience. In order to ensure that the survey is reliable, meaning it is always measuring the same thing, here are some things to consider.

1.     Avoid using compound questions. Any question with an “and” in it is suspect. For example, a question might read, “Our organization’s leaders are trustworthy and live the core values of the organization.” In your mind, these two clauses may seem to go together naturally. However, some survey takers may think of them as being trustworthy but not living the core values of the organization. And depending on if they are thinking about trust or core values, they may respond differently.  This makes the question unreliable.

2.    Make sure terms and directions are clear. For example, does everyone agree on who “leadership” is? In a public sector agency, it could mean the Division Director, the Department Commissioner, or the Agency Secretary. 

3.    Avoid double (or triple) negatives. Questions like this are confusing.

4.    Use the same scale on all the questions. Shifting scales can be confusing and might make participants rate some.

5.    See if questions that others have developed and are already tested for reliability are available. For example, our Climate survey has numerous questions developed by Gallup, as and others that are already known to be reliable.

6.    Consider having someone with survey experience check your work to ensure reliability.