By Thomas Brown, INOVASI PDIA Consultant
Traditional M&E approaches for development programs typically consist of monitoring (focussed on inputs, activities and outputs) and evaluation (focussed on outcomes – like whether this intervention increase student literacy). For INOVASI, our evaluation will take place for the pilot phase, and so we need an additional tool for the Design, Try Ideas (or Pre-Pilot) and Review and Iterate phases, one that enables the capture of rich information about the implementation process in fast feedback loops, such that iterations for improvement can be made in real time.
To do this, we introduce the concept of structured experiential learning: learning during the period of project implementation. We all do experiential learning informally, and usually share these learnings in discussions with our colleagues at the water cooler or over a meal. However, the lessons learnt are seldom formally documented, and are rarely fed back into project design during the course of the project cycle.
Adding structured experiential learning into the M&E mix introduces little ‘e’ – leaving us with a so-called MeE tool (Monitoring Experiential Evaluation). The tool is designed for the Try Ideas phase, to help teachers, with the support of local facilitators (FasDa), to try ideas in their classrooms and reflect on them to see how they can make improvements before trying them again and scaling to a full pilot. Many ideas of these ideas would likely be ‘in the heads’ of FasDa and teachers, and discussed amongst themselves, there would be no documentation and much of this information would not reach the rest of the INOVASI team. In the MeE tool, we want to make the implicit explicit, provide a structured approach to thinking about iteration for improvement, and get detailed and rich documentation of the Try Ideas phase.
How do we introduce it?
The MeE tool consists of 3 Components designed to support the Try Ideas phase. Firstly, by creating a ‘Try Ideas’ Action-plan for teachers – completed by teachers with help from FasDa. Secondly, teachers create an activity log as they try ideas in the classroom, and thirdly, reflection on learning and improvement is completed by FasDa with teachers after they have tried ideas in a ‘reflection session’, to help them review and iterate their ideas.
These will be explored in more in turn detail below. But first, we need to talk about how to introduce the tool to FasDa and teachers. Because it doesn’t matter how good your tool is if no-one knows how to use it!
The following is a summary of how the MeE tool was introduced to FasDa for an INOVASI pilot in Bima, which focuses on the transition from mother tongue in early grades. In preparation for the pilot design workshop with teachers, INOVASI staff took 4-5 hours to work through the tool with FasDa, with a focus on simulation and trying it out so that they understood it well. It went well, taking FasDa from zero understanding to being able to complete and explain the tool in detail.
Warm-up Reflection Exercise based on Homework
In a previous workshop we had asked pilot FasDa to complete two sets of Homework: 1) Try out PDIA tools – Fishbone, 5 Why’s and Round Robin 2) Try out Teaching Strategies for Indonesian as a Second Language.
To get feedback on the first set of homework, and to get facilitators used to MeE style of thinking and documentation, we ran a reflection exercise.
Splitting the FasDa into 2 groups, we asked them to write their feedback on post-its, drawing a smiley face for aspects each of the activities that were successful, and a frown face for the aspects that were not so successful. We then grouped and discussed the ideas with participants.
This exercise foreshadows the approach of MeE Task 3, and also demonstrates the power of of documentation. After they complete the reflection, we emphasised that rather than just discussing the issue, we now have a visual documentation of feedback on each of the activities which we can use to improve our approach.
Brief Overview of the Tool
After the reflection exercise, we gave an overview presentation to give FasDa a sense of the different components of the MeE tool, their purposes, who will complete them, and when.
Steps 1, 2 and 3 of the MeE tool are to be used before, during and after the Try Ideas or pre-pilot phase respectively.
In reference to the 9-step PDIA cycle, Tugas 1 occurs at the end of the Design phase/start of the Try Ideas phase, Tugas 2 occurs during the Try Ideas phase, and Tugas 3 occurs and the end of the Try Ideas phase/beginning of the Review and Iterate phase.
Explaining & simulating Task 1 – Creating a ‘Try Ideas’ or Pre-Pilot Plan for teachers
Task 1 of the MeE tool is designed to assist teachers to make concrete plans of how they will try each idea in their classrooms, and to set goals and think of possible indicators of success. Teachers complete the forms in the last session of the design workshop teacher training, being paired with a facilitator who will provide assistance and probing.
We began by explaining the aim of Task 1, the different aspects, and then walking through an example using role play.
We then asked the FasDa to simulate the activity, with one playing the role of a teacher with ideas to try in class, and one to play the role of the FasDa. Since the participants had already tried out teaching strategies to help with Bahasa Indonesia as part of their homework between workshops, we asked them to think back to before they had implemented these ideas in order to complete this exercise.
Explaining & simulating Task 2 – Documenting the Implementation of Ideas
Task 2 of the MeE tool is an implementation log for teachers to complete as they try activities in the classroom. Although teachers will complete this form when trying ideas in school, the FasDa need to understand it well in order to explain the form to teachers during the design workshop, and to support them if they have difficulties in completing it.
In order to complete this, we asked participants to recall the teaching strategies they tried in their classrooms as part of their homework, and create a comprehensive log of the ideas they implemented.
Explaining & simulating Task 3 – Reflection for Learning and Improvement
Task 3 is completed by FasDa when they facilitate a reflection session with teachers after they have tried ideas in their classrooms. The aim of the exercise is to help teachers to review their ideas and iterate for improvement as they enter phase 5 of the PDIA cycle, but also to record this rich data on how what strategies are and are not working for teachers.
It is the most involved component of the MeE tool, with 4 steps. As such, it is crucial that the FasDa have a strong working understanding of the tool, and we took our time to enable extensive simulation, sharing and reflection. Again we asked FasDa to draw from the homework in which they tried ideas in their classes for inspiration for this activity.
For each step, we would present the process, demonstrate with an example using role play, and field any questions or concerns before getting FasDa to simulate and complete the form. There would then be opportunity for sharing ideas and providing feedback and input. Step 2 and 3 of Task 3 are similar to the reflection exercise we completed at the beginning of the session, so FasDa should have a headstart on understanding the approach.
Task 3, Step 1 – Plan vs Realisation
The aim of this is to explore what went off script – what happened that was not planned and what didn’t happen that was in the plan, and why that was. Often we make changes to our plans on the fly as we realise that something is or isn’t working, but often wouldn’t think to record this process. This part tries to capture this information, which can be very useful feedback for program. Anticipating that some teachers may be reluctant to share instances where they went off script, we emphasised the need for facilitators to reassure teachers that this isn’t an evaluation of whether they stuck to the plan, but to explore about what worked and what didn’t, and why, as a learning exercise.
Task 3, Step 2 – Exploring what didn’t go well
The aim of Step 2 to is to explore the less successful aspects of the ideas tried in the classroom, to explore these issues more deeply to identify the root problem, and then come up with ideas to improve the approach based on this insight. This approach, and the approach in Step 3, are an extended version of the reflection exercise we introduced to FasDa at the beginning of the session.
Task 3, Step 3 – Exploring what did go well
The aim of Step 3 to is to explore the successful aspects of the ideas tried in the classroom, to explore these issues more deeply to identify the core drivers of success, and then come up with ideas to build on this success to and further improve the approach based on this insight.
Task 3, Step 4 – Summary and planning for improvement
This section is essentially summarising/wrapping up the findings from Steps 2 and 3. It has the benefit of distilling the ideas with the teachers, and making a plan of action for implementation of an iterated version of the idea, ie to take the teachers into Phase 5 of the PDIA cycle.
Putting it all back together – Creating a MeE Map
After spending so much time working on each each component of the MeE tool in-depth, a summary is needed to ensure it is clear how the components interact and work with one another.
To do this, we asked participants to use the forms they had completed to create a ‘MeE Map’ to build a holistic understanding of the MeE process, how the components fit together, and the linkages between the components.
We created an example template on the wall for the participants to replicate, and led a participatory summary of each of the components, being sure to ask many questions.
We then asked FasDa to work in groups to create their own MeE maps using the MeE forms they had completed during the session.
Once complete, we asked participants to summarise the MeE process by explaining their MeE maps. This process allowed us to gauge how well they understood the process, and gave opportunities to clarify points of confusion.
Overall, this process of building an MeE or structured experiential learning process, as part of pilot design and iteration, is an important one that carries its challenges and opportunities. As INOVASI continues to design locally driven pilots, this process will continue to strengthen and grow.