The Research Integrity Issue: Is There A Problem Behind the Problem? A Reply to List & McDaniel

Edwin Locke

I have no argument with List and McDaniel’s discussion of personal integrity as a problem in research.  I will begin with a story. Many years I and coauthors submitted an important, elegant and carefully designed paper to a major journal. Naturally it was rejected. But it finally found a home in a strange way. We had predictions for parts of the study because there were established literatures. But one aspect crossed three different literatures, and there was no theory there so we had nothing from which to deduce a specific hypothesis. Nevertheless the result came out as we expected. (There was no HARKing in what we had submitted). The action editor said, in effect: You need to make a theory up for that aspect before the paper will be accepted.

What was that about? The action editor held as an axiom the hypotheticodeductive method of science. As I have written elsewhere (Locke, 2007), this is dangerously narrow because science is above all an inductive process (Harriman, 2010).  Theories need to be the end result rather than the starting point of a research program.  But our journals force people to do the opposite. You need a theory (or to pretend you have a theory) or theories to begin with. Then if it comes out, the implication is that you are done. The theory is proved; end of theory building. If you try any form of replication, you may be told that you have found nothing new. At the same time, everyone is demanding replication studies, but exact replication does not build theories (Locke, 2015).

Lots of bad things can happen under the deductive model. People have to start with theories that are not yet validated. They are pushed to stretch for models. As a long time reviewer I have seen authors come up with two or three or more different (unrelated to each other) theories to justify the same hypothesis; they probably think this raises the odds for them. It probably does. When you use the deductive method, the prediction has to come out or you are doomed. As everyone knows, negative results are not wanted. No wonder people are tempted to push the moral envelope.  When you have a system that is not totally rational, do not expect everyone to act rationally.

What are some journal policies that could help mitigate these problems and move real theory building forward? Here are some suggestions:

1. Allow studies to be submitted just based on questions (e.g., see Audia, Kristoff, Brown & Locke, 1996).
2. Allow hypotheses to be made solely on the basis of past results relating to the same phenomenon rather than requiring a formal theory or theories (see #5).
3. Allow hypotheses to be made based on introspection, a very important method (Locke, 2009).
4. Allow exploratory research aimed at discovery to be published if accompanied by a then-predicted replication of that finding.
5. Encourage replication with variation rather than exact replication (Locke, 2015); this helps theory building by establishing generality across, tasks, subjects, method, time span, design, level, culture, etc. (Locke & Latham, 1990). In the process this encourages the discovery of moderators (interactions, boundary conditions). This last allows negative results to be published, viz. X works only if Y.
6. Encourage programmatic research (implicit in #5).
7. Encourage the discovery of causal mediators; the more one knows about mediators the more confidence one can have in making predictions. (Mediators, of course, may be moderators.)
8. Discourage premature theorizing and hasty generalization. Solid theory building takes a long time.
9. Make introductions short and leave any discussion as how a given result might help move theory building forward to the discussion section.

The goal here is to move the journals from a deductive mode to an inductive mode.  Inductive theories are open ended so a surprising result (if valid) is viewed as a potential benefit rather than as a threat. It could help the theory to grow. New discoveries can correct errors and/or allow refinements. It is amazing what a psychological effect the inductive method has on the theory builder. You can look forward to new discoveries rather than fearing them. (Obviously deduction is appropriate once there is good evidence for a theory.)

All this is not to say that my suggestions will eliminate QRPs. In the end every good researcher must create their own soul.

References

Audia, G., Kristof, A., Brown, K.G. & Locke, E.A. (1996). The relationship of goals and micro-level work processes to performance on a multi-path, manual task. Journal of Applied Psychology, 81, 483-497.

Harriman, D. (2010). The logical leap: Induction in physics. New York, NY: New American Library.

List, S. K. & McDaniel, M. A. (2016). I-O psychology’s lack of research integrity. The Industrial-Organizational Psychologist, 54(2), 1-4.

Locke, E. A. (2007). The case for inductive theory building. Journal of Management, 33, 867-890.

Locke, E. A. (2009). It’s time we brought introspection out of the closet. Perspectives on Psychological Science, 4, 24-25.

Locke, E. A. & Latham, G. P. (1990). A theory of goal setting and task performance. Englewood Cliffs, NJ: Prentice Hall.

P.S. See also “The Virtue of Persistence” by me, Williams, and Masuda, TIP April 2015, 52(4), 104.