By Anne Adrian under CC at flickr.com
[Editor’s note: This post was written by Kristie Evenson, a research practitioner on development of policy research organizational capacities, based in Croatia and working in the Europe and Eurasia region. It emerges from reflection from P&I´s current course on Monitoring, evaluating and learning on policy influence, supported by the Think Tank Fund and the Think Tank Initiative.]
One of the more challenging parts of assisting to facilitate a P&I Monitoring Evaluation & Learning (MEL) online learning course for policy research organizations is to try to better understand the influences that shape the participants’ views coming into the course. Without the benefit of face-to-face communication and interaction as might happen in a normal classroom, we have to catch at snippets of statements and feedback that participants give us. From the past several weeks, one underlying influence – that of donors – which we all mention, yet don’t give full attention strikes me as needing more scrutiny.
One on hand, donor influence for MEL is very explicit. Many course participants describe the need to do MEL because donors want to see policy influence. The course early on tries to unpack what policy influence is possible and how to educate organizations to be realistic about policy influence and in turn educate donors to be realistic about what types of particular policy research inputs can result in which potential areas of policy influence.
Getting this out of the way early would seem to then allow more freethinking about MEL and design and implementation. Yet more implicit influences are also at work that we do not necessarily recognize and therefore perhaps do not give enough attention.
Namely much of how we all think about envisioning, capturing, and measuring policy influence is based on frameworks already in our heads and all around us. For example, many of us describe ourselves as causual, linear thinkers or ‘frameworkers’ when anticipating policy influence rather than ‘circlers’ who tend to be responsive to each situation and think in terms of relationships not causal linkages (find here an earlier P&I post by Vanesa Weyrauch on’ frameworkers’ and ‘circlers’). This is not necessarily because this is our ‘natural’ choice but because we have been ‘trained’ by countless years of doing donor based log frames and reporting. While some donors are now less focused on log frame type reporting for this type of work, we continue to carry the legacy. Participants of the course acknowledge that their actual MEL is often more circular, but it becomes linear by the time it makes its way to reporting to donors. This may or may not distort what actually has happened and what can be learned from the process for future work.
This also affects practical aspects of MEL. Selection of indicators, selection of near and medium term anticipated outcomes or impacts, and selection of theories of changes often follow what is provided in donor guidance documents. They are also often ‘safe’ indicators and may or may not capture more precisely the aspects of the project or program that should be emphasized. At least participants from last year’s MEL course were challenged with identifying a set of indicators that was right for them partly because it is tricky. But it also was partly due to what we eventually came to understand as having the freedom to do so without a specific donor expectation.
A side effect of this donor influence seems to be organizations’ tendency to be focused on their attributes as an organization writ large (as if presenting this to a donor) rather than a more specific analysis of their attributes in relation to putting in place and carrying out a MEL system in their organization. Both in last year and this year’s first SWOT exercise in the course, many participants misunderstood the task and did a SWOT of their organization (something that donors regularly request) rather than a SWOT of putting in place a MEL system.
Why? Some likely just read the directions too quickly. Others maybe found it more familiar terrain to do a SWOT on something that they have had practice presenting to external audiences. An organizational SWOT shows the strengths and weaknesses, but a MEL SWOT opens organizations up to showing or at least thinking about more of their ‘warts’ and needs for implementing something than the basic attributes of their organization.
This might also be connected to the fact that while many donors require some type of tracking system for their projects, they are less focused on encouraging organizational-wide MEL systems and financing these. Being serious about MEL, accordingly, means that organizations need to be somewhat serious about their organizational development management and strategy and delineate resources accordingly.
So how can we harness the seen and unseen influences of donors in our approach to MEL? First we can assume that much of what we do and have been trained to do is likely due to how we have reported projects in the past. This is one way of learning and is valuable, but it does not need to be the only way of doing MEL. Rather it is a point from which to explore and design up a manageable, meaningful, and yes donor friendly MEL system that is more than the sum of its very influences.