Concepts for the design and evaluation of change-oriented research

Research funding bodies, and society in general, have high expectations for research to contribute solutions that have positive societal and environmental impact. This desire to increase societal benefits has prompted an evolution in how research is conducted, and a corresponding need for appropriate research evaluation tools and approaches. The Sustainability Research Effectiveness Program, led by Professor Brian Belcher at Royal Roads University, has responded to this need by developing a Transdisciplinary Research Quality Assessment Framework to evaluate research design and implementation and a theory-based evaluation approach to assess research contributions to outcomes and impacts

There is increasing pressure on researchers and research organisations to demonstrate that their work contributes to positive social and environmental impact and addresses complex societal challenges. In response, research is crossing boundaries between disciplines as it engages more with stakeholders in complex systems. The pursuit of real-world solutions has resulted in an increase in interdisciplinary research (IDR), which integrates research methods and approaches from two or more disciplines, and transdisciplinary research (TDR), which involves stakeholders, research users, and other societal actors in the research process to embrace multiple forms of knowledge and facilitate social learning. This has transformed research to be more engaged, change oriented, applied, effective, and impactful.

This evolution in modes of research brings with it the need for appropriate research evaluation to guide and assess the quality and effectiveness of the research produced. Responding to this need, the Sustainability Research Effectiveness Program, led by Professor Brian Belcher, the Ashoka Chair in Research Effectiveness at the College of Interdisciplinary Studies at Royal Roads University, and Senior Associate Scientist with the Centre for International Forestry Research (CIFOR), has developed a conceptual framework, tools, and methods to assess the quality and effectiveness of change-oriented research and to evaluate research contributions to outcomes and impacts.

Evolving research and assessing quality

Professor Belcher explains how this evolution “is a dynamic process; researchers are experimenting with new ways to design and implement research that is more engaged, pluralistic, and democratic in order to be more effective.” There are many new TDR projects that provide opportunities to learn from experience. With these advances, however, comes the need for tools to assess the design, implementation, and results of these innovative research projects. We need a systematic way to assess these kinds of projects if we are to determine what works, where, and how.

“Researchers are experimenting with
new ways to design and implement research that is more engaged, pluralistic, and democratic.”

The evaluation of problem-oriented research that employs interdisciplinary and transdisciplinary approaches is particularly challenging. There are no widely accepted standards to guide and assess research that crosses disciplinary bounds. Traditional discipline-specific concepts and measures of research quality used in peer review and typical bibliometric indicators of research impact (e.g. publications; citation counts; journal impact factors) are insufficient as they overlook key contributions of disciplinary research and TDR alike. Moreover, additional criteria are required to measure the quality of innovative approaches, together with the diversity of actors, outputs, outcomes, and long-term social impacts of TDR.

Developing the TDR Quality Assessment Framework

Researchers in the Sustainability Research Effectiveness Program have developed a Transdisciplinary Research Quality Assessment Framework (Belcher et al., 2016) based on current TDR theory. They carried out a systematic review of articles that discussed evaluation and assessment in terms of quality and excellence of research conducted in a TDR context. Relevant articles were obtained from searches of databases, targeted journals, and references cited in articles already selected by the study.

Four key principles of TDR quality emerged: relevance, credibility, legitimacy, and effectiveness. Criteria for the assessment of either actual or potential contributions to problem-solving and social change were taken from each article and organised into themes that corresponded to the four principles. These principles and criteria, together with the researchers’ own experience, informed the evaluation rubric that is presented as the TDR Quality Assessment Framework. The research team carried out three rounds of testing on a set of Master’s theses, doctoral dissertations, and research-for-development projects that used transdisciplinary approaches in order to revise, refine, and improve the framework.

Professor Belcher describes how the TDR Quality Assessment Framework can be used to both guide and inform the design of research projects and proposals, as well as assess the research at any stage, from proposal through to the evaluation of a completed project.

The Outcome Evaluation approach

In response to the need for more comprehensive ways to evaluate TDR and research contributions to change processes, the Sustainability Research Effectiveness Program developed a theory-based Outcome Evaluation approach (Belcher et al., 2020) for research evaluation. This approach uses a detailed, actor-specific theory of change and a series of hypotheses about how and why a research project or program is expected to contribute to a process of change. It then tests each step using empirical evidence. Professor Belcher explains that theory-based evaluation is valuable for evaluating research that is operating in complex systems because it can accommodate the diversity of contexts, non-linear causal processes, and lack of replication required for experimental impact assessment.

“The Outcome Evaluation approach provides a conceptual and analytical framework for more precise assessments of actor-specific outcomes and
impact pathways.”

The Outcome Evaluation approach is specifically directed at change-oriented research projects such as TDR, sustainability research, and research-for-development. A systems perspective is adopted in recognition that these projects operate in conjunction with other actors and processes. The approach adopts the concept of spheres of control, influence, and interest, first developed by Montague et al. (2000) and popularized in Outcome Mapping (Earl et al., 2001), a planning and assessment methodology designed for development programs. The key idea is that any program, including a research program, can only control up to its outputs. Beyond that, the program can only influence other actors and processes (sphere of influence) to change, and thereby contribute to higher-level changes in the sphere of interest.

The Sustainability Research Effectiveness Program provides step-by-step guidance on how to apply the Outcome Evaluation approach (Belcher et al., 2020). This includes advice on documenting a theory of change; determining data needs and sources; collecting, managing, and analysing data; and presenting findings. In addition to providing a clear conceptual and analytical framework, the Outcome Evaluation approach provides a conceptual and analytical framework for more precise assessments of actor-specific outcomes and impact pathways.

Conceptualising research impact

The need for appropriate approaches to assess research impact for learning and accountability has already been established. Assessing impact is complicated by the many different concepts and definitions of ‘impact’ in use. These definitions lack clarity and consistency, and some are fundamentally different in their meanings. Professor Belcher and his colleagues examined how various development organizations use and define the terms “outcome” and “impact”, seeking conceptual clarity (Belcher & Palenberg, 2018). Based on this review of common usage, they identified different causal perspectives and defining elements underlying the various definitions and proposed guidelines for better definitional practice, clarifying causal perspectives in use and taking care in the use of qualifiers.

Demonstrating accountability and impact

Professor Belcher emphasizes the need to know if, and how, research actually contributes to positive change so that we can learn from experience and be able to demonstrate both accountability and impact. To complement their theory-based Outcome Evaluation methodology, the researchers applied and tested the TDR Quality Assessment Framework in a series of case studies of research-for-development projects (Belcher et al., 2019; Ramirez & Belcher, 2020). The analyses revealed that projects which employed more TDR principles in their design and implementation were able to make diverse scientific and social contributions, giving them a greater breadth of influence.

Program theory of change

The Sustainability Research Effectiveness Program aims to improve the contributions that research makes to social change processes, as well as increase the effectiveness of research in complex socio-ecological systems. The program’s activities involve the development of tools and methods to characterize effective research design and implementation, and model how research is expected or intended to make a difference; exploration of the underlying theories of social change that explain how and why research contributes to outcomes; case studies and comparative analyses to identify connections between TDR design and implementation and outcomes; and collaborations with researchers to support their learning through the assessment of completed research projects. Professor Belcher remarks that “we can learn what works and what does not work in specific contexts and generate lessons for researchers, research managers, research funders, and society more broadly.”

The researchers intend that their framework and methodology will enable the wider research community to plan and assess their research more effectively. Consequently, other researchers will be afforded the opportunity to build on these ideas and progress the research effectiveness agenda. The program’s goal is to empower more effective research through improved design, implementation, and adaptive management so that research contributions to positive social, environmental, and economic change are maximised.

References

• Belcher, B.M., Davel, R. and Claus, R. (2020). A refined method for theory-based evaluation of the societal impacts of research. MethodsX, 7, 100788. Available at: https://doi.org/10.1016/j.mex.2020.100788
• Belcher, B.M. and Hughes, K. (2020). Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations. Research Evaluation, rvaa024. Available at: https://doi.org/10.1093/reseval/rvaa024
• Ramirez, L.F. and Belcher, B.M., 2020. Crossing the science-policy interface: Lessons from a research project on Brazil nut management in Peru. Forest Policy and Economics, 114, 101789. Available at: https://doi.org/10.1016/j.forpol.2018.07.018
• Belcher, B.M., Claus, R., Davel, R. and Ramirez, L.F. (2019). Linking transdisciplinary research characteristics and quality to effectiveness: a comparative analysis of five research-for-development projects. Environmental Science & Policy, 101, 192-203. Available at: https://doi.org/10.1016/j.envsci.2019.08.013
• Belcher, B. and Palenberg, M. (2018). Outcomes and impacts of development interventions: toward conceptual clarity. American Journal of Evaluation, 39(4), 478-495. Available at: https://doi.org/10.1177/1098214018765698
• Belcher, B.M., Rasmussen, K.E., Kemshaw, M.R. and Zornes, D.A. (2016). Defining and assessing research quality in a transdisciplinary context. Research Evaluation, 25(1), 1-17. Available at: https://doi.org/10.1093/reseval/rvv025
• Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection into development programs. IDRC, Ottawa, ON, CA.
• Montague, S., Young, G., & Montague, C. (2000). Using circles to tell the performance story. Retrieved from: http://www.pmn.net/wp-content/uploads/Using-Circles-to-Tell-the-Performance-Story.pdf

Research Objectives

The Transdisciplinary Research Quality Assessment Framework and the Outcome Evaluation approach have been developed and tested in a series of case studies to learn whether and how research contributes to change in different contexts, and to refine the tools.

Funding

• Canada Research Chairs Program
• The Social Sciences and Humanities Research Council of Canada
• Ashoka Canada
• Royal Roads University

Collaborators

• The Centre for International Forestry Research (CIFOR)
• The CGIAR Research Program on Forests, Trees and Agroforestry (FTA)

Bio

Brian Belcher has a BSc in Ecology, a Master’s degree in Natural Resources Management, and a PHD in Forest Economics. He is the Ashoka Chair in Research Effectiveness and Professor in the College of Interdisciplinary Studies at Royal Roads University. He is also a Senior Associate Scientist with the Centre for International Forestry Research (CIFOR).

Contact
Brian Belcher
Royal Roads University
2005 Sooke Road, Victoria, BC
Canada V9B 5Y2

E: [email protected]
T: +1 250 507 8162 T: +1 250 391 2600 ext. 4407
W: https://researcheffectiveness.ca/

Concepts for the design and evaluation of change-oriented research