Skip to content
BY 4.0 license Open Access Published by De Gruyter Open Access July 28, 2022

OER Impact Assessment: A framework for higher education institutions and beyond. Approaches to assess the impact of Open Educational Resources

  • Martin Ebner , Dominic Orr and Sandra Schön EMAIL logo
From the journal Open Education Studies

Abstract

The purpose of this paper is to provide an overview of approaches to and insights for impact assessment on open educational resources (OER) in higher education and building on this to sketch a framework for university focussed OER impact assessment. The authors describe the literature on impact assessment in the OER context and the existing contributions to OER impact assessment in higher education. Findings of the analysis are that there are few contributions on the effects of OER in general or of specific OER initiatives. Four contributions are presented in more detail. From these examples and the literature analysis, derivations, and challenges for OER impact assessment are drawn, such as the large diversity in OER purposes and the invisibility of the re-usage of OER. The contribution sketches a framework model for describing OER-relevant results, outcomes, and impact, and more specifically demonstrates how this can be done for exemplary OER-related objectives. This contribution is thus of relevance to funding bodies and institutions working in the context of higher education that wish to systematically evaluate and monitor statements about the effectiveness of OER activities according to the UNESCO (2019) OER recommendation.

References

Antadze, N. and Westley, F. R. (2012). “Impact Metrics for Social Innovation: Barriers or Bridges to Radical Change?”, in: Journal of Social Entrepreneurship, Vol. 3, No. 2, 133–150, October 2012, URL: https://uwaterloo.ca/waterloo-institute-for-social-innovation-and-resilience/sites/ca.waterloo-institute-for-social-innovation-and-resilience/files/uploads/files/impact_metrics_for_si_barriers_or_bridges.pdfSearch in Google Scholar

Atkins, D. E., Brown, J. S. and Hammond, A. L. (2007). “A review of the Open Educational Resources (OER) movement: Achievements, challenges, and new opportunities”, pp. 1-84. URL: http://www.hewlett.org/uploads/files/ReviewoftheOERMovement.pdfSearch in Google Scholar

Bundesministerium für Wissenschaft, Forschung und Wirtschaft (2017). “Nationale Strategie zur sozialen Dimension in der Hochschulbildung. Für einen integrativen Zugang und eine breitere Teilhabe.“ Wien. URL: https://ph-tirol.ac.at/sites/default/files/download/Nationale%20Strategie%20%28PDF%29.pdfSearch in Google Scholar

Bundesministerium für Bildung, Wissenschaft und Forschung (2020). „Gesamtösterreichische Universitätsentwicklungsplan (GUEP).“, URL: https://www.bmbwf.gv.at/dam/jcr:b7701597-4219-42f3-9499-264dec94506e/GUEP%202022-2027_Aktualisiert_um_Statistik_final_bf.pdfSearch in Google Scholar

Coughlan, T.; Ebrahimi, N.; McAndrew, P. and Pitt, R. (2013). “Assessing OER impact across varied organisations and learners: experiences from the “Bridge to Success” initiative.”, in: OER13: Creating a Virtuous Circle, 26-27 Mar 2013, Nottingham, UK. URL: http://oro.open.ac.uk/37754/1/OER13-Coughlanetal_Assessing_OER_Impact.pdfSearch in Google Scholar

Conrad, D. and McGreal, R. (2012). “Flexible paths to assessment for OER learners: A comparative study”, Journal of Interactive Media in Education, 2012(2), p. Art. 12. DOI: http://doi.org/10.5334/2012-1210.5334/2012-12Search in Google Scholar

de los Arcos, B.; Farrow, R.; Perryman, L.-A.; Pitt, R. and Weller, M. (2014). “OER Evidence Report 2013-2014”, OER Research Hub. URL: http://oro.open.ac.uk/41866/Search in Google Scholar

Ebner, M. (2018). “OER-Certification in Higher Education”, in Proceedings of EdMedia: World Conference on Educational Media and Technology. pp. 1-6 Amsterdam, Netherlands: Association for the Advancement of Computing in Education (AACE). URL: https://www.researchgate.net/publication/326034702_OER-Certification_in_Higher_EducationSearch in Google Scholar

Ebner, M., Stöckler-Penz, C. (2011). “Open Educational Resources als Lifelong- Learning Strategie am Beispiel der TU Graz.”, in The Lifelong Learning University, pp. 53–60.Search in Google Scholar

Ebner, M. and Schön, S. (2011). “Lernressourcen: Frei zugänglich und einsetzbar.“, in: Handbuch E-Learning - Expertenwissen aus Wissenschaft und Praxis - Strategie, Instrumente, Fallstudien, pp. 1–14.Search in Google Scholar

Ebner, M., Köpf, E., Muuß-Merholz, J., Schön, M., Schön, S., Weichert, N. (2015). “Ist-Analyse zu freien Bildungsmaterialien (OER).“, Book on Demand, Norderstedt.Search in Google Scholar

Ebner, M., Kopp, M., Freisleben-Teutscher, C., Gröblinger, O., Rieck, K., Schön, S., Seitz, P, Seissl, M., Ofner, S., Zimmermann, C., Zwiauer, C. (2016). “Recommendations for OER Integration in Austrian Higher Education.“, in: Conference Proceedings: The Online, Open and Flexible Higher Education Conference, EADTU 2016, pp. 34-44.Search in Google Scholar

Ebner, M., Lorenz, A., Lackner, E., Kopp, M., Kumar, S., Schön, S. and Wittke, A. (2016). “How OER enhance MOOCs – A Perspective from German-speaking Europe.”, in: Open Education: from OERs to MOOCs. Jemni, M., Kinshuk, Khribi, M. K. (Eds.). Springer. Lecture Notes in Educational Technology. pp. 205-220.Search in Google Scholar

Ebner, M., Freisleben-Teutscher, C., Gröblinger, O., Kopp, M., Rieck, K., Schön, S., Seitz, P., Seissl, M., Ofner, S. and Zwiauer, C. (2016). „Empfehlungen für die Integration von Open Educational Resources an Hochschulen in Österreich“, Forum Neue Medien in der Lehre Austria. URL: http://www.fnm-austria.at/fileadmin/user_upload/documents/Buecher/2016_fnma-OER-Empfehlungen_final.pdfSearch in Google Scholar

Ebner, M., Kopp, M., Hafner, R., Budroni, P., Buschbeck, V., Enkhbayar, A., Ferus, A., Freisleben-Teutscher, C. F., Gröblinger, O., Matt, I., Ofner, S., Schmitt, F., Schön, S., Seissl, M., Seitz, P., Skokan, E., Vogt, E., Waller, D. and Zwiauer, C. (2017). “Konzept OER-Zertifizierung an österreichischen Hochschulen.“ Forum Neue Medien in der Lehre Austria, URL: http://www.fnm-austria.at/fileadmin/user_upload/documents/Buecher/OER_Labeling_2017.pdfSearch in Google Scholar

Ebner, M., Schön, S. and Kumar, S. (2016). “Guidelines for leveraging university didactics centers to support OER uptake in German-speaking Europe”, in Education Policy Analysis Archives, 24 (39), URL: http://dx.doi.org/10.14507/epaa.24.185610.14507/epaa.24.1856Search in Google Scholar

Ebner, M.; Schön, S.; Atenas, J.; Havemann, L.; Nascimbeni, F. and Neumann, J.L. (2020). “Ressourcen und Werkzeuge für die Entwicklung einer OER-Policy an Hochschulen. Gemeinsamer Report von TU Graz, Open Education Austria Advanced, OER World Map und Open Education Policy Hub.” Graz: TU Graz. DOI: 10.13140/RG.2.2.13705.47207/1Search in Google Scholar

Ehlers, U.-D. (2011). “Extending the territory: From Open Educational Resources to Open Educational Practices.”, in: Journal of Open, Flexible and Distance Learning, 15(2), 1–10. URL: http://www.jofdl.nz/index.php/JOFDL/article/view/6Search in Google Scholar

Farrow, R.; Pitt, R.; De los Arcos, B.; Perryman, L.-A.; Weller, M. and McAndrew, Patrick (2015). “Impact of OER use on teaching and learning: Data from OER Research Hub (2013–2014).”, in: British Journal of Educational Technology, Volume 46, Issue 5, Special Issue: Open Data in Learning Technology, September 2015, pp. 972-976, URL: https://bera-journals.onlinelibrary.wiley.com/doi/full/10.1111/bjet.1231010.1111/bjet.12310Search in Google Scholar

Fritz, J. (2020). “How to Talk About Nonprofit Impact from Inputs to Outcomes. Inputs, Outputs, Outcomes, Impact - How are they Different?”, in: small businesses, updated May 2020, URL: https://www.thebalancesmb.com/inputs-outputs-outcomes-impact-what-s-the-difference-2502227Search in Google Scholar

Geser, G. (2007). “Open Educational Practices and Resources. OLCOS Roadmap 2012.” Salzburg: Salzburg Research. URL: http://www.olcos.org/cms/upload/docs/olcos_roadmap.pdfSearch in Google Scholar

Grimaldi, P. J., Basu Mallick, D., Waters, A. E. and Baraniuk, R. G. (2019). “Do open educational resources improve student learning? Implications of the access hypothesis”, PloS One, 14(3), e0212508. DOI: https://doi.org/10.1371/journal.pone.0212508Search in Google Scholar

Hodgkinson-Williams, C. and Arinto P. B. (2017). “Adoption and impact of OER in the Global South” DOI: https://doi.org/10.5281/zenodo.600398Search in Google Scholar

Inamorato dos Santos, A. I., Nascimbeni, F., Bacsich, P., Atenas, J., Aceto, S., Burgos, D. and Punie, Y. (2017). “Policy Approaches to Open Education–Case Studies from 28 EU Member States (OpenEdu Policies) (No. JRC107713)”, Joint Research Centre (Seville site). DOI: https://doi.org/10.2760/283135Search in Google Scholar

Jenkins, J. J.; Sánchez, L. A.; Schraedley, M. A. K.; Hannans, J.; Navick, N. and Young, J. (2020). “Textbook Broke: Textbook Affordability as a Social Justice Issue.”, in: Journal of Interactive Media in Education, v2020 n1 Article 3 2020, URL: https://files.eric.ed.gov/fulltext/EJ1253996.pdf10.5334/jime.549Search in Google Scholar

Karunanayaka, S. P. and Naidu, S. (2017). “Impact of integrating OER in teacher education at the Open University of Sri Lanka”, in C. Hodgkinson-Williams and P. B. Arinto (Eds.), Adoption and impact of OER in the Global South, pp. 459–498. DOI: https://doi.org/10.5281/zenodo.600398Search in Google Scholar

Ladurner, C.; Ortner, C.; Lach, K.; Ebner, M., Haas, M.; Ebner, M.; Ganguly; R. and Schön, S. (2020). “The Development and Implementation of Missing Tools and Procedures at the Interface of a University’s Learning Management System, its OER Repository and the Austrian OER Referatory”, in: International Journal of Open Educational Resources (IJOER), Volume 3, No. 2 Fall 2020 Winter 2021, URL: https://www.ijoer.org/the-development-and-implementation-of-missing-tools-and-procedures-at-the-interface-of-a-universitys-learning-management-system-its-oer-repository-and-the-austrian-oerreferatory/10.18278/ijoer.3.2.3Search in Google Scholar

Lerman, S. R., Miyagawa, S., Margulies, A. H. (2008). “Open Course Ware: Building a Culture of Sharing”, in: Opening Up Education, Iiyoshi, T. and Kumar, M.S.V (Eds.), 2008, MIT Press, pp. 213-227.Search in Google Scholar

Marín, V. I., Bond, M., Zawacki-Richter, O., Aydin, C. H., Bedenlier, S., Bozkurt, A., Conrad, D., Jung, I., Kondakci, Y., Prinsloo, P., Qayyum, A., Roberts, J., Sangrà, A., Slagter van Tryon, P. J., Veletsianos, G. and Xiao, J. (2020). “A comparative study of national infrastructures for digital (open) educational resources in higher education”, in: Open Praxis, 12(2), 241–256. DOI: http://dx.doi.org/10.5944/openpraxis.12.2.107110.5944/openpraxis.12.2.1071Search in Google Scholar

Masterman, L., Wild, J., White, D. and Manton, M. (2011). “JISC Open Educational Resources Programme: OER Impact Study”, Research Report. URL: https://process.arts.ac.uk/sites/default/files/jiscoerimpactstudyresearchreportv1-0.pdfSearch in Google Scholar

Mathew, S. and Kashyap, U. (2019). “Impact of OER Materials on Students’ Academic Performance in an Undergraduate Astronomy Course.”, in: Journal of STEM Education: Innovations and Research, v20 n1 p46-49 Apr-Sep 2019, URL: https://www.jstem.org/jstem/index.php/JSTEM/article/view/2369/2123Search in Google Scholar

McAuley, A.; Stewart, B.; Siemens, G. and Cormier, D. (2010). “Massive Open Online Courses Digital ways of knowing and learning, The MOOC model For Digital Practice”, URL: http://www.elearnspace.org/Articles/MOOC_Final.pdfSearch in Google Scholar

McGreal, Rory (2019). “A Survey of OER Implementations in 13 Higher Education Institutions”, in: International Review of Research in Open and Distributed Learning, 20, 5 pp. 141-145; URL: https://files.eric.ed.gov/fulltext/EJ1237334.pdfSearch in Google Scholar

Miao, F., Mishra, S., Orr, D. and Janssen, B. (2019). “Guidelines on the development of open educational resources policies.” UNESCO. URL: unesdoc.unesco.org/in/rest/annotationSVC/DownloadWatermarkedAttachment/attach_import_75b312a4-e696-423c-b30d-f8573e3c819c?_=371129eng. pdf&to=107&from=1Search in Google Scholar

MIT OpenCourseWare (2006). “2005 Program Evaluations Finding Report.” URL: http://ocw.mit.edu/ans7870/global/05_Prog_Eval_Report_Final.pdfSearch in Google Scholar

Naidu, S. and Karunanayaka, S. P. (2017). “Development of the Open Educational Practices Impact Evaluation Index.” Research paper presented at OE Global 2017 Conference, 8–10 March 2017. Cape Town, South Africa. URL: https://idl-bnc-idrc.dspacedirect.org/bitstream/handle/10625/57016/57028.pdf?sequence=1&isAllowed=ySearch in Google Scholar

OECD (2015). “What is impact assessment?” OECD. URL: http://www.oecd.org/sti/inno/What-is-impact-assessment-OECDImpact.pdfSearch in Google Scholar

OECD (2020). “Tax Challenges Arising from Digitalisation – Economic Impact Assessment.” OECD. URL: https://www.oecd.org/tax/beps/tax-challenges-arising-from-digitalisation-economic-impact-assessment-0e3cc2d4-en.htm10.1787/0e3cc2d4-enSearch in Google Scholar

Okada, A. and Sherbone, T. (2018). “Equipping the Next Generation for Responsible Research and Innovation with Open Educational Resources, Open Courses, Open Communities and Open Schooling: An Impact Case Study in Brazil”, in: Journal of interactive media in education, URL: https://jime.open.ac.uk/articles/10.5334/jime.482/Search in Google Scholar

Open Education Group and Lumen Learning (2021). “OER Adoption Impact Explorer”, version 0.8. URL: https://opencontent.shinyapps.io/OER-Impact-Calculator/Search in Google Scholar

Orr, D.; Rimini, M. and van Damme, D. (2015). “Open Educational Resources: A Catalyst for Innovation, Educational Research and Innovation”, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/9789264247543-en10.1787/9789264247543-enSearch in Google Scholar

Pitt, R.; Ebrahimi, N.; McAndrew, P. and Coughlan, T. (2013). “Assessing OER Impact across Organisations and Learners: Experiences from the Bridge to Success Project.”, in: Journal of Interactive Media in Education, Spec Iss 2013, URL: https://files.eric.ed.gov/fulltext/EJ1034706.pdf10.5334/2013-17Search in Google Scholar

Rahman, M.; Panigrahi, M.R.; Panda, S. (2019). “Impact Assessment of OER Repository of Bangladesh Open University.”, in: Pan-Commonwealth Forum, 9-12 September 2019, Edinburgh, Scotland. URL: http://oasis.col.org/handle/11599/3341Search in Google Scholar

Rodgers, E. (2011). “Measuring Our Impact: the Open.Michigan Initiative.InProceedings of OpenCourseWare Consortium Global 2011: Celebrating 10 Years of OpenCourseWare.” URL: http://hdl.handle.net/2027.42/91308Search in Google Scholar

Schaffert, S. (2010). “Strategic Integration of Open Educational Resources in Higher Education. Objectives, Case Studies, and the Impact of Web 2.0 on Universities.”, in: Ulf-Daniel Ehlers and Dirk Schneckenberg (eds.), Changing Cultures in Higher Education – Moving Ahead to Future Learning, New York: Springer, pp. 119-131.10.1007/978-3-642-03582-1_11Search in Google Scholar

Schön S. and Ebner M. (2020). “Open Educational Resources in Austria”, in: Huang R., Liu D., Tlili A., Gao Y., Koper R. (eds.). Current State of Open Educational Resources in the “Belt and Road” Countries. Lecture Notes in Educational Technology, Springer, Singapore, pp. 17-33.Search in Google Scholar

Schön, S., Ebner, M., Hornung-Prähauser, V. (2017). “Digital social innovation within education: Five insights on the role of digital tools in the field of Open Educational Resources (OER) projects.” in: Nata, R. V. (ed). Progress in Education. Vol. 49. Nova publisher, pp. 167-188.Search in Google Scholar

Schön, S.; Kreissl, K.; Dobusch, L. and Ebner, M. (2017). “Mögliche Wege zum Schulbuch als Open Educational Resources (OER). Eine Machbarkeitsstudie zu OER-Schulbüchern in Österreich.“ Band 7 der Reihe „InnovationLab Arbeitsberichte“, herausgegeben vom Forschungsbereich InnovationLab der Salzburg Research Forschungsgesellschaft mbH, gleichzeitig erschienen als Band 15 der Reihe „Beiträge zu offenen Bildungsressourcen“ (http://o3r.eu) Salzburg: Salzburg Research. URL: http://l3t.eu/oer/images/band15.pdfSearch in Google Scholar

Stannard-Stockton, S. (2010). “Getting Results: Outputs, Outcomes and Impact. Measuring Social Impact.”, in: Stanford Social Innovation Review. URL: https://ssir.org/articles/entry/getting_results_outputs_outcomes_impact#Search in Google Scholar

Stern, E.; Stame, N.; Mayne, J.; Forss, K.; Davies, R. and Befani, B. (2012). “Broadening the range of designs and methods for impact evaluations.”, in: DFID Working Paper, volume 38, URL: https://www.ids.ac.uk/publications/broadening-the-range-of-designs-and-methods-for-impact-evaluations/10.22163/fteval.2012.100Search in Google Scholar

Thomas, A., Campbell, L. M., Barker, P. and Hawksey, M. (2012). “Technology for Open Educational Resources - into the wild.” University of Bolton, jISC. URL: http://publications.cetis.org.uk/wp-content/uploads/2012/12/into_the_wild_screen.pdfSearch in Google Scholar

TU Graz and BMBWF - Technische Universität Graz und Bundesministerium für Bildung, Wissenschaft und Forschung (2018). “Leistungsvereinbarung 2019-2021.“ Graz: TU Graz. URL: https://www.tugraz.at/fileadmin/user_upload/tugrazInternal/TU_Graz/Universitaet/TU_Graz_kompakt/Leistungsvereinbarung_2019-2021.pdfSearch in Google Scholar

TU Graz - Technische Universität Graz (2017). “Entwicklungsplan 2018plus zur Beschlussfassung am 14. Dezember 2017“. Graz: TU Graz. URL: https://mibla-archiv.tugraz.at/17_18/Stk_6/Entwicklungsplan2018plus_Beschluss_UR_20171214.pdfSearch in Google Scholar

Technische Universität Graz (2020). “Richtlinie zu offenen Bildungsressourcen an der Technischen Universität Graz (OER-Policy),“ November 2020 URL: https://www.tugraz.at/fileadmin/user_upload/tugrazExternal/02bfe6da-df31-4c20-9e9f-819251ecfd4b/2020_2021/Stk_5/RL_OER_Policy_24112020.pdfSearch in Google Scholar

UNESCO (2019). “Recommendation on Open Educational Resources (OER)”, 25 November 2019. URL: http://portal.unesco.org/en/ev.php-URL_ID=49556&URL_DO=DO_TOPIC&URL_SECTION=201.htmlSearch in Google Scholar

Universität Graz (2020). “Open Educational Resources Policy der Universität Graz, Rektoratsbeschluss vom 4. März 2020“, URL: https://static.uni-graz.at/fileadmin/digitales-lehren-undlernen/Dokumente/OER_Policy.pdf (2020-12-10)Search in Google Scholar

University of Edinburgh (2016). “Open Educational Resources Policy.” URL: https://www.ed.ac.uk/files/atoms/files/openeducationalresourcespolicy.pdfSearch in Google Scholar

Weller, M.; de los Arcos, B.; Farrow, R.; Pitt, B.; McAndrew, P. (2015). “The Impact of OER on Teaching and Learning Practice.”, in: Open Praxis, v7 n4 p351-361 Oct-Dec 2015. URL: https://files.eric.ed.gov/fulltext/EJ1083261.pdf10.5944/openpraxis.7.4.227Search in Google Scholar

Westermann Juárez, W. and Venegas Muggli, J. I. (2017). “Effectiveness of OER use in first-year higher education students’ mathematical course performance: A case study.”, in C. Hodgkinson-Williams and P. B. Arinto (Eds.), Adoption and impact of OER in the Global South (pp. 187–229). DOI: https://doi.org/10.5281/zenodo.601203Search in Google Scholar

White, D. and Manton, M., (2011). “Open Educational Resources: The value of reuse in higher education”, University of Oxford, URL: http://www.jisc.ac.uk/media/documents/programmes/elearning/oer/OERTheValueOfReuseInHigherEducation.pdfSearch in Google Scholar

White Baker, E. and Sibona, C. J. (2020). “Digital OER Impact on Learning Outcomes for Social Inclusion, Journal of Computer Information Systems”, DOI: 10.1080/08874417.2020.1802789Search in Google Scholar

Wiley, D. (2009). “Dark Matter, Dark Reuse, and the Irrational Zeal of a Believer.” David Wiley Weblog, URL: http://opencontent.org/blog/archives/905Search in Google Scholar

Received: 2022-01-31
Accepted: 2022-05-31
Published Online: 2022-07-28

© 2022 Martin Ebner et al., published by De Gruyter

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 6.2.2023 from https://www.degruyter.com/document/doi/10.1515/edu-2022-0018/html
Scroll Up Arrow