lynx   »   [go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0132990.html
   My bibliography  Save this article

UK Doubles Its “World-Leading” Research in Life Sciences and Medicine in Six Years: Testing the Claim?

Author

Listed:
  • Steven Wooding
  • Thed N Van Leeuwen
  • Sarah Parks
  • Shitij Kapur
  • Jonathan Grant
Abstract
Background: The UK, like some other countries, carries out a periodic review of research quality in universities and the most recent Research Excellence Framework (REF) reported a doubling (103% increase) in its “world leading” or so-called “4*” research outputs in the areas of life sciences and medicine between 2008 and 2014. This is a remarkable improvement in six years and if validated internationally could have profound implications for health sciences. Methods: We compared the reported changes in 4* quality to bibliometric measures of quality for the 56,639 articles submitted to the RAE 2008 and the 50,044 articles submitted to the REF 2014 to Panel A, which assesses the life sciences, including medicine. Findings: UK research submitted to the RAE and REF was of better quality than worldwide research on average. While we found evidence for some increase in the quality of top UK research articles, a 10-25% increase in the top 10%ile papers, depending upon the metrics used, we could not find evidence to support a 103% increase in quality. Instead we found that as compared to the RAE, the REF results implied a lower citation %ile threshold for declaring a 4*. Interpretation: There is a wide discrepancy between bibliometric indices and peer-review panel judgements between the RAE 2008 and REF 2014. It is possible that the changes in the funding regime between 2008 and 2014 that significantly increased the financial premium for 4* articles may have influenced research quality evaluation. For the advancement of science and health, evaluation of research quality requires consistency and validity – the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF.

Suggested Citation

  • Steven Wooding & Thed N Van Leeuwen & Sarah Parks & Shitij Kapur & Jonathan Grant, 2015. "UK Doubles Its “World-Leading” Research in Life Sciences and Medicine in Six Years: Testing the Claim?," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-10, July.
    • Handle: RePEc:plo:pone00:0132990
    DOI: 10.1371/journal.pone.0132990
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132990
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0132990&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0132990?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Lutz Bornmann & Rüdiger Mutz & Hans-Dieter Daniel, 2010. "A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants," PLOS ONE, Public Library of Science, vol. 5(12), pages 1-10, December.
    2. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stephan B. Bruns & David I. Stern, 2016. "Research assessment using early citation information," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 917-935, August.
    2. Emma Terämä & Melanie Smallman & Simon J Lock & Charlotte Johnson & Martin Zaltz Austwick, 2016. "Beyond Academia – Interrogating Research Impact in the Research Excellence Framework," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-18, December.
    3. V. A. Traag & L. Waltman, 2019. "Systematic analysis of agreement between metrics and peer review in the UK REF," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    4. Gianni De Fraja & Giovanni Facchini & John Gathergood, 2016. "How Much Is That Star in the Window? Professorial Salaries and Research Performance in UK Universities," Discussion Papers 2016-13, University of Nottingham, GEP.
    5. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gibson, Elizabeth & Daim, Tugrul U. & Dabic, Marina, 2019. "Evaluating university industry collaborative research centers," Technological Forecasting and Social Change, Elsevier, vol. 146(C), pages 181-202.
    2. Frederik T. Verleysen & Tim C.E. Engels, 2013. "A label for peer-reviewed books," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 428-430, February.
    3. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.
    4. Mohamed Boufarss & Mikael Laakso, 2020. "Open Sesame? Open access priorities, incentives, and policies among higher education institutions in the United Arab Emirates," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1553-1577, August.
    5. Sabrina Petersohn & Thomas Heinze, 2018. "Professionalization of bibliometric research assessment. Insights from the history of the Leiden Centre for Science and Technology Studies (CWTS)," Science and Public Policy, Oxford University Press, vol. 45(4), pages 565-578.
    6. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    7. Karacic, Domagoj & Miskulin, Ivan & Serdarusic, Hrvoje, 2016. "State Investment In Science And Scientific Productivity Of Universities," UTMS Journal of Economics, University of Tourism and Management, Skopje, Macedonia, vol. 7(1), pages 37-48.
    8. Giliberto Capano & Benedetto Lepori, 2024. "Designing policies that could work: understanding the interaction between policy design spaces and organizational responses in public sector," Policy Sciences, Springer;Society of Policy Sciences, vol. 57(1), pages 53-82, March.
    9. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    10. repec:oup:rseval:v:32:y:2024:i:2:p:273-285. is not listed on IDEAS
    11. Degl’Innocenti, Marta & Matousek, Roman & Tzeremes, Nickolaos G., 2019. "The interconnections of academic research and universities’ “third mission”: Evidence from the UK," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    12. Augusteijn, Hilde Elisabeth Maria & Wicherts, Jelte M. & Sijtsma, Klaas & van Assen, Marcel A. L. M., 2023. "Quality assessment of scientific manuscripts in peer review and education," OSF Preprints 7dc6a_v1, Center for Open Science.
    13. Feliciani, Thomas & Morreau, Michael & Luo, Junwen & Lucas, Pablo & Shankar, Kalpana, 2022. "Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model," Research Policy, Elsevier, vol. 51(4).
    14. Чернова Е. Г. & Ахобадзе Т. Д. & Малова А. С. & Салтан А. А., 2017. "Модели Финансирования Высшего Образования И Эффективность Деятельности Университетов Эмпирическое Исследование Европейского Опыта И Отечественная Практика," Вопросы образования // Educational Studies Moscow, National Research University Higher School of Economics, issue 3, pages 37-82.
    15. John Gibson, 2021. "The micro‐geography of academic research: How distinctive is economics?," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(4), pages 467-484, September.
    16. John Rigby & Barbara Jones, 2020. "Bringing the doctoral thesis by published papers to the Social Sciences and the Humanities: A quantitative easing? A small study of doctoral thesis submission rules and practice in two disciplines in ," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1387-1409, August.
    17. Cristian Barra & Ornella Wanda Maietta & Roberto Zotti, 2017. "First, Second and Third Tier Universities: Academic Excellence, Local Knowledge Spillovers and Innovation in Europe," CSEF Working Papers 468, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy.
    18. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    19. Civera, Alice & Lehmann, Erik E. & Paleari, Stefano & Stockinger, Sarah A.E., 2020. "Higher education policy: Why hope for quality when rewarding quantity?," Research Policy, Elsevier, vol. 49(8).
    20. Jürgen Janger & Anna Strauss-Kollin & David Campbell, 2013. "Academic Careers: A Cross-country Perspective. WWWforEurope Working Paper No. 37," WIFO Studies, WIFO, number 46923.
    21. Peter Weißhuhn & Katharina Helming & Johanna Ferretti, 2018. "Research impact assessment in agriculture—A review of approaches and impact areas," Research Evaluation, Oxford University Press, vol. 27(1), pages 36-42.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0132990. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.
    Лучший частный хостинг