Toolbox_Visual-Header_Impact-Evaluation
Impact & Evaluation
Article
7 Minuten Lesedauer

Monitoring, evaluation and learning for think tanks: How to match strategy with objectives for different areas of work

Republished content from On Think Tanks (OTT), edited by Barbara Pongratz

A clear monitoring, evaluation and learning (MEL) strategy is one of the best ways to help think tanks achieve the greatest impact in the most cost-effective way. Yet measuring the impact and results of a think tank’s work is challenging, as these are often intangible (e.g., building relationships with policymakers, playing a key role in debates or networks, etc.). The ultimate goal is policy change, but this takes a long time and often cannot be attributed to a specific action or organization. Policy change is the result of many factors and actors (Lomofsky, 2016). 

Incorporating MEL into the daily life of any organization is well worth the effort – as long as the data collected is analyzed and the lessons learned are put to use.  A smart and proportionate use of MEL tools, and especially a well-thought-out MEL plan, can help think tanks to: 

  • Reflect on and enhance the influence of their research in public policy. 
  • Satisfy their (and their donors) interest in evidencing the uptake of research in policy. 
  • Build their reputation and visibility and attract more support for their work. 
  • Generate valuable knowledge for all members of the organization. 
  • Re-organize existing processes for data collection so that they can be useful for real MEL purposes, and discard processes and data that are not useful. 

Where to start

A good starting place is to define success. For this you must reflect on where you locate yourself in the policy process with any particular issue (Lomofsky, 2016). Looking at the various policy process “parts” individually makes it easier to identify the possible strategies that would have an effect on each part. This consequently helps identify what to look for when evaluating impact down the road. 

A think tank should not only focus on “driving direct policy change but also try to affect what happens before, throughout and after a new policy is designed. For example, to illuminate the way some policy problems are perceived” (Lomofsky, 2016). 

It is best to aim for a balance between a think tank’s accountability requirements and honesty about its achievements. This can be done by: 

  • Clarifying your policy influence objectives (they need to be realistic). These should be drawn from your mission, projects and the problems you are trying to address. 
  • Selecting your policy influence strategies and how this can be measured (e.g., advocacy, communications, capacity building, etc.).
  • Ensuring you have the necessary resources to achieve results. 

Which areas of work to monitor

When think tanks devise good monitoring and evaluation plans, and when they carry out changes and improvements based on the lessons from the MEL exercise, they will be more effective in achieving their objectives (Lomofsky, 2018). OTT researchers have defined specific areas in which a think tank might conduct a MEL exercise.

MEL of research

One way to assess research quality is using a systems approach developed by the International Development Research Centre (IDRC) called Research Quality Plus (RQ+). This framework can be valuable for think tanks to improve their research quality and is relevant for impact-oriented think-tanks – not only for use in external reviews and evaluations but also because it enables critical reflection and adaptive management towards high quality, impactful research. This framework, which is still evolving and being updated by IDRC, can be used for the management of research quality in programs, program portfolios and grant funding portfolios. It follows a specific rationale, proposes a few instruments, and can be applied in external reviews. 

MEL of communications

Doing great research is only a part of a think tank’s mission. To be effective, think tanks must also communicate their findings to a range of stakeholders. The effectiveness of these communication efforts can be monitored and evaluated, and organizations can learn how to improve their communications to achieve greater impact. The MEL of research communications should enable you to better package and disseminate your research findings so they can feed into the policy debate. 

One way is to view MEL as integral to every part of your research and communication, not just something that is tacked on at the end. There is very practical advice (see below) on indicators for the MEL of communications that provides good insights into the successes and challenges of your communications activities such as database management, media engagement, event engagement and website traffic. You need to ask yourself the right questions about your communications efforts and use the data you have, together with your own reflections, to learn and improve.

MEL of governance and management

Without sound governance and management, think tanks cannot deliver their missions. OTT experts provide practical advice and an overview of issues related to governance and management. Particularly the Board of Directors (governance) should pay attention to their think tank’s MEL systems and evaluate methods for approaching this. Think tanks may be at different stages of maturity. As a result, the role of governance will change over time. The MEL of governance should help track the maturity of the think tank governance structures, roles and responsibilities.

OTT experts focus on several key areas that indicate good governance and are applicable to all kinds of think tanks. There are also processes for reviewing governance such as conducting regular SWOT analyses, using an Organizational Capacity Assessment process or developing an MEL plan for governance. 

MEL for projects

If your organization is only at the start of your MEL journey, you may want to start by doing the MEL for just one project, later expanding to other projects and eventually to the whole organization as your confidence in MEL grows. 

One case study by OTT researchers describes the process and lessons from designing an MEL system for a well-respected South African think tank (SAIIA). The case study provides an overview of project-level MEL. The process included identifying and consolidating what was already in place through a situation analysis of the MEL, developing a theory of change, creating an MEL guideline document and plan, developing indicators and tools for data collection and reporting formats that include lessons learned and suggestions for improvement. Organizations need to ensure they have systems for data management, reporting and learning to support the project team.

There are many kinds of MEL strategies, but it is important to understand the reasons for the MEL effort to best adapt the strategies and methodologies to the type of knowledge desired. 

For MEL it is crucial to be clear about what you want to achieve (have clear policy influence objectives). In addition, think tanks need to develop systems for MEL based on what they already have in place. They need to be realistic about what they can achieve and build on strengths. Lastly, think tanks need to not only see monitoring and evaluation as a tedious accountability requirement for donors, but as an opportunity for learning, innovation and adaptive management. 

Sources for this article:

Further Resources:

Barbara Pongratz

Analyst at MERICS

Barbara Pongratz

 

Barbara Pongratz’s research focuses on Sino-German relations, China’s environmental policy and its digital governance. Prior to joining MERICS, she worked in development cooperation in two environmental projects at the German Corporation for International Cooperation (GIZ) and at a German management consultancy in Hong Kong. Barbara studied Sinology as well as European and International Law at Freie Universität Berlin and Ludwig Maximilian University Munich undertaking exchange semesters at Shanghai Tongji University and Peking University.
 

Back to homepage