lynx   »   [go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/ldr/wpaper/127.html
   My bibliography  Save this paper

Randomised trials for policy: a review of the external validity of treatment effects

Author

Listed:
  • Muller, Sean

    (SALDRU, School of Economics, University of Cape Town)

Abstract
The paper provides a first survey of the literature on external validity. The starting point for this are debates regarding the use of randomised evaluations to inform policy. Besides synthesising contributions to the programme evaluation literature we consider definitions of external validity from other subdisciplines within economics, such as experimental economics and the time-series forecasting literature, as well as the disciplines of philosophy and medicine. We suggest - following Cook and Campbell (1979) - that the fundamental challenge arises from interactive functional forms. This somewhat neglected point provides a framework in which to understand how and why extrapolation may fail. In particular it suggests that replication cannot resolve the external validity problem unless informed by some prior theoretical understanding of the causal relationship of interest. Finally, we argue that the assumptions required for simple external validity are conceptually equivalent to those required for obtaining unbiased estimates of treatment effects using non-experimental methods, undermining the idea that internal validity needs be rigorously assessed whereas external validity can be ascertained subjectively. Theory may play a role in aiding extrapolation, but the extent to which this will be possible in practice remains an open question.

Suggested Citation

  • Muller, Sean, 2014. "Randomised trials for policy: a review of the external validity of treatment effects," SALDRU Working Papers 127, Southern Africa Labour and Development Research Unit, University of Cape Town.
    • Handle: RePEc:ldr:wpaper:127
    as

    Download full text from publisher

    File URL: https://opensaldru.uct.ac.za/bitstream/handle/11090/691/2014_127_Saldruwp.pdf?sequence=1
    File Function: Full text
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Charles F. Manski, 2013. "Response to the Review of ‘Public Policy in an Uncertain World’," Economic Journal, Royal Economic Society, vol. 0, pages 412-415, August.
    2. Martin Ravallion, 2009. "Evaluation in the Practice of Development," The World Bank Research Observer, World Bank, vol. 24(1), pages 29-53, March.
    3. Rodrik, Dani, 2008. "The New Development Economics: We Shall Experiment, but How Shall We Learn?," Working Paper Series rwp08-055, Harvard University, John F. Kennedy School of Government.
    4. Gary Solon & Steven J. Haider & Jeffrey M. Wooldridge, 2015. "What Are We Weighting For?," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 301-316.
    5. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    6. Steven D. Levitt & John A. List, 2007. "Viewpoint: On the generalizability of lab behaviour to the field," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(2), pages 347-370, May.
    7. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    8. repec:feb:artefa:0110 is not listed on IDEAS
    9. John A. List, 2011. "Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 3-16, Summer.
    10. Meyer, Bruce D, 1995. "Natural and Quasi-experiments in Economics," Journal of Business & Economic Statistics, American Statistical Association, vol. 13(2), pages 151-161, April.
    11. A. D. Roy, 1951. "Some Thoughts On The Distribution Of Earnings," Oxford Economic Papers, Oxford University Press, vol. 3(2), pages 135-146.
    12. Charles F. Manski, 2011. "Policy Analysis with Incredible Certitude," Economic Journal, Royal Economic Society, vol. 121(554), pages 261-289, August.
    13. Rust, John, 2010. "Comments on: "Structural vs. atheoretic approaches to econometrics" by Michael Keane," Journal of Econometrics, Elsevier, vol. 156(1), pages 21-24, May.
    14. Robert J. LaLonde, 1984. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," Working Papers 563, Princeton University, Department of Economics, Industrial Relations Section..
    15. Jens Ludwig & Jeffrey R. Kling & Sendhil Mullainathan, 2011. "Mechanism Experiments and Policy Evaluations," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 17-38, Summer.
    16. J.J. Heckman & E.E. Leamer (ed.), 2007. "Handbook of Econometrics," Handbook of Econometrics, Elsevier, edition 1, volume 6, number 6b.
    17. Manski, Charles F., 2000. "Identification problems and decisions under ambiguity: Empirical analysis of treatment response and normative analysis of treatment choice," Journal of Econometrics, Elsevier, vol. 95(2), pages 415-442, April.
    18. Roth,Alvin E. (ed.), 1988. "Laboratory Experimentation in Economics," Cambridge Books, Cambridge University Press, number 9780521333924, December.
    19. Ravallion Martin, 2009. "Should the Randomistas Rule?," The Economists' Voice, De Gruyter, vol. 6(2), pages 1-5, February.
    20. Michael P. Keane, 2010. "A Structural Perspective on the Experimentalist School," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 47-58, Spring.
    21. Brian E. Roe & David R. Just, 2009. "Internal and External Validity in Economics Research: Tradeoffs between Experiments, Field Experiments, Natural Experiments, and Field Data," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(5), pages 1266-1271.
    22. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    23. J.J. Heckman & E.E. Leamer (ed.), 2007. "Handbook of Econometrics," Handbook of Econometrics, Elsevier, edition 1, volume 6, number 6a.
    24. Arthur Schram, 2005. "Artificiality: The tension between internal and external validity in economic experiments," Journal of Economic Methodology, Taylor & Francis Journals, vol. 12(2), pages 225-237.
    25. Manski, Charles F., 2013. "Public Policy in an Uncertain World: Analysis and Decisions," Economics Books, Harvard University Press, number 9780674066892, march.
    26. repec:feb:artefa:0087 is not listed on IDEAS
    27. Loewenstein, George, 1999. "Experimental Economics from the Vantage-Point of Behavioural Economics," Economic Journal, Royal Economic Society, vol. 109(453), pages 23-34, February.
    28. Matzkin, Rosa L., 2007. "Nonparametric identification," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 73, Elsevier.
    29. Lant Pritchett, Justin Sandefur, 2013. "Context Matters for Size: Why External Validity Claims and Development Practice Don't Mix-Working Paper 336," Working Papers 336, Center for Global Development.
    30. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    31. Roth, Alvin E, 1988. "Laboratory Experimentation in Economics: A Methodological Overview," Economic Journal, Royal Economic Society, vol. 98(393), pages 974-1031, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sean F. Ellis & Olesya M. Savchenko & Kent D. Messer, 2023. "Is a non-representative convenience sample of adults good enough? Insights from an economic experiment," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 9(2), pages 293-307, December.
    2. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    3. Muller, Seán M., 2020. "The implications of a fundamental contradiction in advocating randomized trials for policy," World Development, Elsevier, vol. 127(C).
    4. Ashley L. Buchanan & Michael G. Hudgens & Stephen R. Cole & Katie R. Mollan & Paul E. Sax & Eric S. Daar & Adaora A. Adimora & Joseph J. Eron & Michael J. Mugavero, 2018. "Generalizing evidence from randomized trials using inverse probability of sampling weights," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(4), pages 1193-1209, October.
    5. Stephanie Rosch & Sharon Raszap Skorbiansky & Collin Weigel & Kent D. Messer & Daniel Hellerstein, 2021. "Barriers to Using Economic Experiments in Evidence‐Based Agricultural Policymaking," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(2), pages 531-555, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    2. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    3. Jonathan H.W. Tan & Zhao Zichen & Daniel John Zizzo, 2023. "Scientific Inference from Field and Laboratory Economic Experiments: Empirical Evidence," Discussion Papers Series 663, School of Economics, University of Queensland, Australia.
    4. Boris Salazar-Trujillo & Daniel Otero Robles, 2019. "La revolución empírica en economía," Apuntes del Cenes, Universidad Pedagógica y Tecnológica de Colombia, vol. 38(68), pages 15-48, July.
    5. Fiore, Annamaria, 2009. "Experimental Economics: Some Methodological Notes," MPRA Paper 12498, University Library of Munich, Germany.
    6. Anirban Basu & James J. Heckman & Salvador Navarro‐Lozano & Sergio Urzua, 2007. "Use of instrumental variables in the presence of heterogeneity and self‐selection: an application to treatments of breast cancer patients," Health Economics, John Wiley & Sons, Ltd., vol. 16(11), pages 1133-1157, November.
    7. James J. Heckman, 2008. "Econometric Causality," International Statistical Review, International Statistical Institute, vol. 76(1), pages 1-27, April.
    8. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    9. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    10. Dagsvik, John K., 2018. "Invariance axioms and functional form restrictions in structural models," Mathematical Social Sciences, Elsevier, vol. 91(C), pages 85-95.
    11. Guido W. Imbens, 2015. "Matching Methods in Practice: Three Examples," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 373-419.
    12. Yannick Gabuthy & Nicolas Jacquemet, 2013. "Analyse économique du droit et méthode expérimentale," PSE-Ecole d'économie de Paris (Postprint) halshs-00746617, HAL.
    13. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    14. repec:hal:wpaper:halshs-00746617 is not listed on IDEAS
    15. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    16. Muller, Paul & van der Klaauw, Bas & Heyma, Arjan, 2017. "Comparing Econometric Methods to Empirically Evaluate Job-Search Assistance," IZA Discussion Papers 10531, Institute of Labor Economics (IZA).
    17. Fatoumata Nankoto Cissé, 2022. "How impact evaluation methods influence the outcomes of development projects? Evidence from a meta-analysis on decentralized solar nano projects," Documents de travail du Centre d'Economie de la Sorbonne 22008, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    18. Benno Torgler, 2021. "The Power of Public Choice in Law and Economics," CREMA Working Paper Series 2021-04, Center for Research in Economics, Management and the Arts (CREMA).
    19. Grüner, S. & Hirschauer, N. & Mußhoff, O., 2015. "Potenzial verschiedener experimenteller Designs für die Politikfolgenabschätzung," Proceedings “Schriften der Gesellschaft für Wirtschafts- und Sozialwissenschaften des Landbaues e.V.”, German Association of Agricultural Economists (GEWISOLA), vol. 50, March.
    20. van der Klaauw, Bas, 2014. "From micro data to causality: Forty years of empirical labor economics," Labour Economics, Elsevier, vol. 30(C), pages 88-97.
    21. Régis Deloche, 1995. "Expérimentation, science économique et théorie des jeux : "Nunc est bibendum" ," Revue Économique, Programme National Persée, vol. 46(3), pages 951-960.

    More about this item

    Keywords

    external validity; treatment effects; randomised trials;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ldr:wpaper:127. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Alison Siljeur (email available below). General contact details of provider: https://edirc.repec.org/data/sauctza.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.
    Лучший частный хостинг