Skip to content Skip to search

Insight Center Resources

Industry knowledge to help you grow your business

Understanding Statistical Significance and Response Rates

One of the most important issues when conducting survey research is to determine the number of returned survey responses necessary to produce valid and reliable results. To this end, we need to consider statistical significance.

To meet these requirements, the accepted percentage for both the confidence level and confidence interval must be determined. The confidence level describes the uncertainty of a sampling method. The generally accepted confidence level utilized in survey research is .95 or 95 percent. Basically, meeting this confidence level indicates that if the study was to be conducted 100 times, the results would fall within the same margins 95 percent of the time.

The confidence interval, also referred to as the margin of error, denotes the range of acceptable error for the data. The most readily accepted margin of error for survey research is five percent, whereby the percentages of data results are known to fall within this margin of error.

If we were to conduct a survey and receive a response sample large enough to meet both the accepted confidence level and confidence interval, we can assume the following:

  • If we were to conduct this survey 100 more times, we would produce the same results 95 percent of the time with the true percentage falling within a range of -5 to +5 percent of the identified percentage. This is the acceptable confidence level and interval to statistically consider the collected data both valid and reliable

Once the confidence level and interval are determined, then the required sample size can be determined. The formula to calculate sample size requirements for statistical significance takes into account many factors, and the calculation is neither intuitive nor linear. Typically, the lower the population size, the higher the percentage for the required sample size. For example, a population of 100 individuals would require a sample size of 79 responses.  However, at a certain point, the sample size necessary to meet statistical significance in terms of representing the entire population reaches a maximum of 384 (many researcher round the number to 400).

Media Browser

A practical example of the interpretation of confidence level and confidence interval would be if you were to survey a population and receive the appropriate number of responses to meet the 95 percent level and five percent interval requirements then you would be confident that the data was both reliable and valid in understanding the results in the following manner:

A statistically significant number of participants responded the following question:

“How satisfied are you with Product X”?

If 80 percent responded that they were satisfied, then you can be assured that if you were to ask this same question to the required number of individuals in a given population 100 times, the results you would consistently achieve would be that satisfaction would be reported by between 75 percent and 85 percent, for 95 of the administrations.