Microplots And Food Security: Encouraging Replication Studies Of Policy Relevant Research

Abstract

Replication research is a valuable, yet often misunderstood, tool for increasing our understanding of promising research findings. In their short paper below the authors discuss their principles for conducting replication research, explain how they picked a candidate study for replication, describe the robustness checks they would conduct in their replication plan, and give an overview of how they would interpret their replication results. The authors also describe some of their lessons learned after working in replication research for over five years.

Generated Summary

This paper outlines a replication study that aims to encourage replication studies of policy-relevant research, specifically focusing on the impact of microplots on food security. The study employs a four-stage replication process, adapting a methodology developed by the International Initiative for Impact Evaluation (3ie). The stages include: push button replication (PBR), pure replication, measurement and estimation analysis (MEA), and theory of change analysis. The authors chose to replicate a paper by Santos et al. (2014) on the effects of a government-allocated microplot program in West Bengal, India, targeting landless households. The primary goal is to verify the original findings’ robustness, assess the impact of microplots on various outcomes, and explore the implications of the research for policy design. The replication plan includes pre-specifying robustness checks and adhering to a replication plan, which maintains transparency and ensures the validity of additions post-replication plan. The authors aim to move beyond simply stating the need for replication studies by providing a concrete example of how replication can be conducted to strengthen or weaken the main takeaways of the original article.

Key Findings & Statistics

• The FAO estimates that between 2010 and 2012, the prevalence of undernourishment in developing regions was 12.9%.
• The paper by Santos et al. (2014) assesses the impact of the Nijo Griha, Nijo Bhum (NGNB) program.
• The original authors find statistically significant increases in program participants, which tracked to increases in plot sizes.
• In the paper being replicated, the original authors present results in table 3 of the original paper.
• The original authors report a 25% attrition rate between the baseline and follow-up survey.
• In Santos et al. (2014) they note “data collection for our baseline took place in two rounds: May-June of 2010 and January-June of 2011.”
• Santos et al. find a number of their results statistically insignificant at the intent to treat (ITT) level.
• In the middle of the project the Government of West Bengal decided to cap land distribution at 5 deciles per household.
• The original researchers note that the greater the microplot land allocation, the larger the intermediate outcome food security effects.
• Based on the summary statistics, 17 per cent of the treatment households are composed of single, divorced, or widowed women.

Other Important Findings

• The replication study will focus on land tenure security results as the original authors found statistically significant increases in program participants tracked to increases in plot sizes.
• The replication study will assess the reproducibility of the paper by recoding the original results, the pure replication stage.
• The study plans to conduct the robustness exercises described in the replication plan for the measurement and estimation analysis and the theory of change analysis stages.
• The paper will analyze the main results of the paper at the intermediate level, our robustness checks will reexamine the influence of providing a microplot on household: i) perceived land tenure security, ii) likelihood of access credit for agriculture, iii) use of improved inputs, and iv) women’s likelihood to be involved in important food and agriculture decisions.
• The authors also describe some of their lessons learned after working in replication research for over five years.

Limitations Noted in the Document

• The study acknowledges the sensitivity around replication studies and will avoid using terminology like “error” or “mistake.”
• The paper will focus on the land tenure security results, as the original authors find statistically significant increases in program participants that tracked to increases in plot sizes.
• The original authors report a fairly large 25 per cent attrition rate between baseline and follow-up survey.
• The timeline of the intervention, as presented in the paper, is difficult for us to follow.
• If sample limitations prevent us from conducting a full heterogeneous impact analysis, we will look at how closely the outcomes of these women correlate with the general sample.
• If missing values or outliers prove to be a factor in the original analysis, we will test the robustness of the results to inclusion of observations dropped by the original analysis or alternative imputation techniques.

Conclusion

The overarching aim of this replication study is to contribute to the ongoing discussion of how to perform replication research to deepen the understanding of the original research, verify policy-relevant results, and facilitate a more comprehensive evaluation of the original findings. The study’s approach to replication, including the use of a pre-specified replication plan, aims to maintain transparency, address potential biases, and ensure that any modifications made during the replication process are justified. The study emphasizes the importance of robustness checks and the application of various analytical techniques. The authors plan to examine the impact of the microplot program on outcomes. The study also aims to explore the implications of the research for policy design by assessing the impact of microplots on food security and by exploring the intermediate outcomes. The results of the replication study will be interpreted by assessing whether the original findings are robust to the various checks, with particular attention to land tenure security and food security outcomes. If the results are robust, the replication study will strengthen the case for microplots. The authors’ goal is to move beyond the ‘success’ or ‘failure’ paradigm by detailing attempts to verify policy-relevant results. The authors hope that this concrete example will encourage discussion about what development programs work well and what evidence looks promising for scaling, promoting a greater conversation in the scientific community.