How we use MEL at Kiron

How can an NGO ensure that it remains committed to its mission and accountable for its decisions? At Kiron we use a framework called MEL. MEL stands for Monitoring (or Measurement), Evaluation and Learning, and represents a set of steps, activities, methodologies and a mindset. We use MEL to continuously understand and learn in order to improve the impact we achieve with and for the Kiron students.

Since the beginning, Kiron’s research and learning analytics teams have built tools and processes to understand platform performance and helped us adapt our programs. We are now dedicating the roles of a MEL officer and a data analyst to build and run a MEL framework that reflects Kiron’s new collective impact model and continues to support Kiron and its partners in creating impact for our learners.

MEL at Kiron

We are building our MEL system on the basis of a Theory of Change (TOC), which contains the high level goals we want to achieve with and for Kiron’s students and the intermediate outcomes which are needed as preconditions to reach those goals. Theories of Change typically come in the form of diagrams and are widely used in impact-oriented fields such as development cooperation, social change and humanitarian work. They are strategic tools, can become quite complex (just like the project they are depicting) and on the other hand have a strong applicability to the operational level of activities and outputs.

‍

One of Kiron’s high level goals is to contribute to SDG 8 so that refugees and underserved communities engage in meaningful and decent work or self-employment. The outcomes we think can lead to these goals are (among others) that “Students gain the skills & recognition needed to succeed in education and employment” as well as “Kiron students are able to organize their entry to the job market”. The above mentioned outcomes are within Kiron’s sphere of influence. Systems level and political changes certainly have a profound impact on achieving SDG 8. We watch those closely but they lie outside our direct sphere of influence.

So how do we go about setting up and implementing our MEL framework at Kiron?

Measurement
We collect relevant data (mostly quantitative such as course/session attendance) and we monitor it closely and continuously. We know what’s relevant from the pathways that were established in the Theory of Change: the outcomes and activities that are crucial for reaching the higher level goals. Not all aspects within the Theory of Change need to be constantly measured though as the system can quickly become very complex. Measurement data already provides insights into what works and what doesn’t, for example, if attendance rates are low, action can be taken to improve it.

Evaluation
Different measures (i.e. data sources and collection techniques) can be used for one single outcome, which is typical for more high level outcomes. E.g. for our aforementioned “Kiron students finish courses in preparation for local or remote job markets: we can employ analytics data such as course completion, qualitative student feedback as well as learning assessments to get a more complete picture. At the Evaluation stage we evaluate if outcomes were achieved and assess the profoundness of change. Evaluation goes beyond measurement.

Learning
How are we doing with our higher level goals? What worked and what didn’t? What should we change on the strategic level regarding general direction? Should we change or discontinue certain programmatic aspects? The Theory of Change can be adapted to include those learnings. While we certainly learn all the time, changes to the program and MEL setup should only be made in longer cycles of 6-12 months. This way we can properly base our decisions on big enough data sets.