Designing an empirical study takes planning and careful consideration of existing theory and research in the area under investigation. When testing for simple causal relationships, it should be relatively easy to predict the specific outcome when producing a change in the causal variable. Most modern scientific investigations, however, are far from simple- they often involve several variables all of which interact in ways that are sometimes difficult, if not impossible, to predict. One positive feature of complex studies is that they can yield many interesting outcomes, though some of these outcomes may end up being irrelevant or even contrary to our expectations. When the latter happens, there may be a temptation to try different statistical analyses and select the one that best fits our hypothesized results (e.g., using a less powerful statistical test, removing outliers).
Another temptation is to simply not report null results or only report those statistically significant results that are consistent with our hypotheses. Other techniques, such as the manipulation of graphs, have been used to subtly change, and therefore distort, the visual presentation of results in a way that make them more consistent with our expected findings. Such practices are almost always fundamentally deceptive and are contrary to the basic scholarly-scientific mission of searching for truth. However, there are instances in which practices, such as the removal of outliers, are acceptable, but only when the author follows established procedures, informs readers of these actions, and provides a cogent rationale for carrying them out.
