Improving Conclusion Validity

So you may have a problem assuring that you are reaching credible conclusions about relationships in your data. What can you do about it? Here are some general guidelines you can follow in designing your study that will help improve conclusion validity.

Guidelines for Improving Conclusion Validity

Good Statistical Power

The rule of thumb in social research is that you want statistical power to be greater than 0.8 in value. That is, you want to have at least 80 chances out of 100 of finding a relationship when there is one. As pointed out in the discussion of statistical power, there are several factors that interact to affect power. One thing you can usually do is to collect more information ⁠— use a larger sample size. Of course, you have to weigh the gain in power against the time and expense of having more participants or gathering more data. The second thing you can do is to increase your risk of making a Type I error ⁠— increase the chance that you will find a relationship when it’s not there. In practical terms you can do that statistically by raising the alpha level. For instance, instead of using a 0.05 significance level, you might use 0.10 as your cutoff point. Finally, you can increase the effect size. Since the effect size is a ratio of the signal of the relationship to the noise in the context, there are two broad strategies here. To up the signal, you can increase the salience of the relationship itself. This is especially true in experimental contexts where you are looking at the effects of a program or treatment. If you increase the dosage of the program (e.g., increase the hours spent in training or the number of training sessions), it will be easier to see the effect when the treatment is stronger. The other option is to decrease the noise (or, put another way, increase reliability).

Good Reliability

Reliability is related to the idea of noise or “error” that obscures your ability to see a relationship. In general, you can improve reliability by doing a better job of constructing measurement instruments, by increasing the number of questions on an scale or by reducing situational distractions in the measurement context.

Good Implementation

When you are studying the effects of interventions, treatments or programs, you can improve conclusion validity by assuring good implementation. This can be accomplished by training program operators and standardizing the protocols for administering the program.