ED 304: Ed Psych and Human Development

Author: Camille Jamison

Verified by: (Rachel Hardy, 2/3/23)

Disclosure: ChatGPT3 was used in the creation of this resource.

The replication crisis is a recent phenomenon primarily affecting the field of psychology. For several years now, scientists have been unable to reproduce the results of their peer’s studies. As of 2015, an average of just 36% of studies were replicable. This means that the other 64% of psychological study findings are potential false positives.

 A false positive occurs when a study inaccurately indicates a significant correlation that doesn’t exist. Many of these non-replicable studies have been cited often or received media attention, spreading possibly false information. While the exact cause of the replication crisis isn't known, the following are potential causes: 

QPRs, or questionable research habits are one suspect to blame for the replication crisis. Questionable research habits happen when scientists manipulate their data or analysis methods until they can find patterns that seem important. This is called data dredging or data falsification. According to (source 1 [a]) Psychologists have even admitted to data falsification, raising concern and doubt in the field of psychology.

Sample insufficiency is another potential cause of the ongoing replicability crisis. Lack of sample group size or diversity could contribute to false positives - results that are mere coincidence. Sample insufficiency in original or reworked studies can undermine finding credibility. With sufficient numbers and diversity, however, faulty hypotheses can be tested more easily. 

The faulty of some hypotheses is a plausible cause for the replication crisis. When a study claims something novel, exciting, or strange, it has a greater chance of being published in a journal or periodical. This phenomenon is called publication bias and it plays a role in what scientists choose to study in the first place.  Because studies with wild, new, or impossible-sounding claims are more likely to be published, there is more motivation to do new studies than to prove that previous claims are sound. Universities also receive recognition for new findings and motivate their employees in turn with bonuses. 

Additionally, many journals do not require scientists to provide detailed information about their study methods. Many researchers do not share their data or code, making it difficult for other scientists to replicate their findings. This can also make it difficult for readers to understand how a study was conducted.

The replication crisis in psychology is caused by several different factors such as:

• Publication Biases: Many academics are required to publish a minimum number of papers to achieve tenure, and a lack of research also does not bode well for one's career; this phenomenon is often called publish or perish. Many journals, however, will only accept papers with novel research, making replicated research ventures risky for those who want to advance their careers through publications.
• Statistical Issues: Like the publication biases, many journals seek research that offers statistically significant findings. This puts pressure on researchers, even if they are not intentionally altering research, to produce statistically significant data. A researcher who produces statistically significant data on their first try with a particular study may not seek to replicate their experiment before moving towards publication or may unintentionally overlook some questionable aspects of their work's internal validity. (Internal validity refers to how well the study is measuring what it wants to measure.)
• Research Practices: A researcher may stop collecting data when their work shows statistical significance or select only the data that contributes to the desired outcome. Or, they may fail to assess the study's internal validity as objectively as possible, and all scenarios are going to produce problematic results that are unreliable and unlikely to be replicated.
• Hypotheses Testing: There are two issues regarding hypotheses and this crisis. The first issue is that many researchers are too quick to reject the null hypothesis (the version of their hypothesis that states that there is no relationship/difference with the variables tested) in favor of their own. Sometimes this issue is also impacted by things such as the sampling size. The second issue is that some researchers will alter their hypothesis after their results are known so that their hypothesis, not the null hypothesis, is correct. 

The replication crisis has led to calls for more rigorous standards in scientific research. This includes more transparency, greater use of pre-registration, and more emphasis on reproducibility. Additionally, many journals and funding agencies now require that researchers disclose their data and methods and that studies be pre-registered before data collection begins.  False positives sometimes occur due to QRP’s, or questionable research habits. Scientists may manipulate data or analysis methods in their studies until they can find patterns that seem important, this is called data dredging or data falsification. According to (source 1 [a]) Psychologists have even admitted to data falsification, raising concern and doubt in the field of psychology. 

                                                                                 

 Scientists are also often rewarded for original work, especially at universities. There isn’t as much motivation to re-create other original studies. For this reason, replication may not be attempted as often or as thoroughly as it should.

VOCABULARY

• Data falsification:
  

Data falsification involves any manipulation of study results to achieve a desired finding.

• Peer review:

A process of review in which scientists and other professionals review each other's work to ensure quality and accuracy.

• Methodology:

The methods used to research, create, and analyze data.

• False positive:

A study result incorrectly indicates a correlation that isn’t there.

• Publication bias:

The tendency of journals, magazines, or periodicals to publish studies with positive results as opposed to those with negative or inconclusive results.

Media:

This Table, from the open science collaboration project, illustrates the extent of the replication crisis.

Which of the following is NOT considered a contributing factor to the replication crisis in psychology?

Publication bias

Data falsification

Lack of transparency in research methods

High levels of funding for psychological research 

Lack of diversity in research samples

How has the replication crisis affected the credibility of psychological research? 

It has increased the credibility of psychological research 

It has had no effect on the credibility of psychological research

It has decreased the credibility of pscyhological reseach 

It has led to an increase in the number of studies conduct in psychology

It has led to a decrease in the number of studies conduct in psychology

True or False: The replication crisis only affects studies in the field of social psychology

True

False

Select all the contributing factors to the replication crisis in psychology:

Publication bias 

Data drenching

Lack of transparency in research methods

High levels of funding for psychological research 

Lack of diversity in research samples 

Lack of replication studies 

2.8: The Replication Crisis in Psychology - Social Sci LibreTexts

Psychology's Replication Crisis and Clinical Psychological Science | Annual Review of Clinical Psychology (annualreviews.org)

Psychology's Replication Crisis Is Real, Many Labs 2 Says - The Atlantic

Replication Research, Publication Bias, and Applied Behavior Analysis - PMC (nih.gov)