Qualitative Impact Monitoring - QIM, Malawi

Since 1994, poverty alleviation has been the primary aim of the Malawi Government. In order to track and assess changes in the poverty situation, the Poverty Monitoring System (PMS) was created within the National Economic Council (NEC) in 1996. Poverty monitoring has been recognised as essential to maintaining an effective course in poverty reduction. One aspect of this is to observe the impact of policies and programmes on poverty. Qualitative Impact Monitoring of Poverty Alleviation Policies and Programmes (QIM) is the only instrument of the PMS that systematically generates qualitative information on the impact of policies and programmes on poor people’s lives.

QIM is an approach to monitor the impact of policies and programmes on poverty in such a way that impoverished men’s and women’s experiences and views become a vital input into the ongoing fine-tuning of Government policy and action. Main characteristics of QIM are that it:

• is institutionalised – QIM actively involves Government stakeholders and decision-makers in the research process.
• fills existing gaps in information that are uncovered at the policymaking and administrative levels.
• emphasises the feedback, discussion and actual use of the information generated.
• contributes to creating stronger linkages between the three main levels involved in development: national, decentral and community.
• relies on and incorporates information from other monitoring efforts, especially quantitative data.

The QIM exercise is designed to be carried out in a series of cycles so as to maximise learning and to continuously improve the approach. These cycles take place both within a given round of QIM as well as from one round to the next. The first round of QIM in Malawi took place in 1997/98. The process of that round was analysed and used as one key input for the conception of the second round of QIM. QIM 2 was carried out in 2000, incorporating the lessons learned from QIM 1.

The Guiding Objectives of QIM 2

The development of guiding objectives was performed as one of the first tasks after the decision was taken to proceed with a second round of QIM. The overall vision for QIM 2 was formulated as:

In line with the decentralisation policy, QIM is to support policy-makers and decision-takers at the national, decentral and community levels in developing effective poverty-fighting policies, programmes and strategies. It is to accomplish this through providing the information required to help these actors respond to the needs expressed by the poor.

QIM is to be carried out in four phases: the Preparation and Policy Analysis Phase, the Field Phase, the Data Analysis Phase, and the Presentation and Dissemination Phase.

• The Preparation and Policy Analysis Phase: is designed to accomplish four main tasks. The first is to ensure that the QIM investigation is demand-driven by national policy-makers and decentral administrations. The second task is to produce a policy analysis paper upon which the field research is then based, and against which the field phase findings can be compared. Third, to set the parameters for the study (selection of sites, etc.). Finally, the national-level teams who are to carry out the exercise need to be selected and trained.
• The Field Phase: is to be carried out in two cycles of three weeks, each. During the first week of each field phase cycle, members of the national teams train the decentral teams on QIM, the use of PRA tools, and data analysis. The field research in the communities is conducted during the second week. Data is collected and analysed each day. Before leaving the community at the end of the week, the research teams hold a feedback and information-sharing session with the community. Site and decentral-level analyses are performed during the third week.
• The Data Analysis Phase: A decentral report is prepared for each district. The site reports, decentral reports, plus all the raw data and preliminary analyses are assessed at the national level. An interim report is compiled that represents the first product of a workshop. The analysis has to be continued by the Report Writing Team (a small group from the main Task Force) who are expected to complete a final report of the survey findings.
• The Presentation and Dissemination Phase: is an on-going effort to actively carry the findings of QIM into the various dialogues on poverty taking place in the country (including the PRSP process). Presentations have to be made for a variety of stakeholders (e.g., sector ministries, the Parliament, decentral administrations , NGOs, international development organisations, the media, the QIM survey communities, and the public at large).

Preparatory and Policy Analysis Phase

Figure 1: The organigram for QIM
(click inside to enlarge)

Lessons-learnt:

Concept development, identification of research areas / districts

• Use of the concept document or the previous process analysis during the workshops is an additional way of saving time since the participants merely need to revise the document rather than having to develop it. Nevertheless, it is necessary that the new objectives have to be set by the participants
• The number of policy areas that need to be investigated should be kept to a minimum. The proposal is that apart from the poverty section (which is a permanent part of QIM), only two national policy sectors should be investigated. The districts could still add an additional policy/programme area - as long as the total volume of impact hypotheses to be checked is kept manageable.
• The selection of districts is very important and it is necessary 1) to become clear on why districts and sites would be selected, and 2) to document the reasons and criteria. Stakeholder involvement in the selection process is important.

Figure 2: The policy analysis process

• Step 7: National level policy analysis: Consulting stakeholders and reviewing documents:
  The policy analysis identifies the level of poverty content and poverty focus in the selected policies. It is performed so as to emphasise stakeholder involvement – including at the district level. The analysis is carried out in a number of iterations for continuous fine-tuning and to ensure the incorporation of the views and needs of the broadening circle of stakeholders (demand-driven policy research). The policy analysis provides the basis for the formulation of the impact hypotheses, which set the focus of the study.
• Step 8: Selection of the National Team for the field phase:
  The National Team trains the district teams and assists them with the research in the communities. It also performs the tasks of the data analysis phase. Therefore, the members of the team must be selected with care.
• Step 9: Design of data collection sheets, analysis guidelines and report formats:
  Written guidelines for qualitative data analysis and standard formats for the site and district reports help to ensure a degree of consistency.
• Step 10: Hold workshops for the District Executive Committees at the district selected:
  The DECs of the survey districts need to be informed about the QIM approach and the current version of the Policy Analysis. This workshop accomplishes these objectives and provides the opportunity for DEC members to give feedback on the Policy Analysis, including raising their own policy research needs. The DECs also require instruction on how to select the survey communities as well as the members for the District Teams. QIM are discussed and suggestions on the way forward for QIM are made.
• Step 11: Carry out additional policy analysis to support district research areas (if needed):
  The policy analysis is performed to take up topics identified by the DECs (if any). The Policy Analysis Report is revised accordingly.

Lessons-learnt:

Involvement of stakeholders, districts and communities

• Once the sectors/policy areas that will be subject of the investigation have been approved, then representatives of the relevant ministries and NGOs should be asked to join the Task Force.
• Smaller policy analysis teams proved a more effective way of conducting extensive consultations on the policy areas with the stakeholders. The approach is recommended as long as: 1) sufficient in-depth consultation with stakeholders is ensured in order to guarantee that their information needs are met, and 2) the workload is not too heavy for the individuals involved. Options for maximising stakeholder involvement in the policy analysis should be pursued.
• In the event that the proposal above is not possible, then more time must be allotted to the policy analysis to allow for greater direct involvement of ministry and NGO officers.
• More transparency is needed on the roles to be played by different members of the National Team, how the selection is made to play the roles, and that they are associated with different amounts of pay. This needs to be done early, and definitely prior to the field phase.
• It is necessary to ensure that the site selection in the districts should be confirmed as soon as possible. This can be undertaken by having the District Leaders visit each community and verify that it meets the criteria. They also should meet with the traditional leaders and explain the QIM exercise to them (including allaying false expectations).
• An effective and implementable strategy is needed to ensure gender balance on the research teams. It may be possible that women local consultants can be enlisted if too few women can be found amongst the ranks of the stakeholders. Another possibility would be to enlist women officers from other districts.
• Also a greater proportion of women must be recruited for the National Team.
• Meetings with the district officials (DEC) are an extremely important step for initiating effective involvement of the districts and for generating their interest in, ownership of, and acceptance of the QIM process.

• Step 12: Training the National Team; developing hypotheses and checklist:
  This step sets out to accomplish a number of important tasks for the QIM exercise. The tasks include team-building; training in the concepts, methods and approaches required to carry out the QIM field phase; the formulation of the hypotheses to be tested by the research; and the creation of the checklist that guides the data collection. In addition, a Pre-Test is carried out by the National Team.
• Step 13: QIM 2 publicity meeting:
  A half-day meeting is held to present and publicise the objectives of the QIM round amongst the secondary stakeholders. These include organisations that are not directly involved in the current round (other ministries, NGOs, donors, civil society groups, the media). The intention is to prepare their acceptance of the findings later on, despite their non-involvement in the current QIM process.

Lessons-learnt:

Coordination, pre-testing

• Sufficient time has to be devoted to pre-tests both for the national and district teams and they should be held at the end of the workshop so that all of the research skills (including qualitative data analysis) can be practised. Extending the workshop to six days is one way of achieving this.
• Involvement of key decision makers, including the DEC, during the finalisation of the data collection sheets has to be assured. It maybe necessary to translate the material into the local languages.
• Informing traditional and local leaders early on is recommended so that organisational arrangements including lodging arrangements for the teams are clarified. More important is the need to discourage false expectations with regard to the whole QIM exercise.

Figure 3: Organigram for the field phase in one field phase cycle
(click inside to enlarge)

The Field Phase

• Step 14: Training workshops in the districts:
  The District Teams receive training in order to develop capacity for QIM, to learn about the research objectives of the current round, and to foster team spirit.
• Step 15: Inform traditional leaders in the sites to be visited:
  During the first week of the field phase cycle, someone is sent to inform the traditional leaders at each site that the QIM survey is to be held in their community. The objectives and timetable of the study are explained. The traditional leaders are asked to announce the upcoming exercise to their community.
• Step 16: Introduction of the research teams to the communities:
  The community members are called together. When possible, the traditional leader introduces the Research Team to the community. The leader of the Research Team explains the objectives of the QIM round, and discusses the timetable for the research sessions. The community members are asked about their expectations of the exercise. False expectations are allayed.
• Step 17: Data collection:
  This step aims to access valid and adequate information on the QIM research areas. Critical to the implementation of this step are:
  
  • adopting RRA/PRA philosophies and attitudes
  • appropriately choosing and applying RRA/PRA tools
  • recording the responses completely and accurately
  • triangulation and cross-checking information.
• Step 18: Daily analysis of the data:
  An initial analysis of the data is to take place on the same day it is collected. Both the responses and the data collection methods should be looked at in order to inform the data collection for the remaining days.
• Step 19: Feedback session/data verification/problem-solving workshop in the community:
  The intention of this step is not only to verify the research findings with the community, but also to encourage two-way dialogue. This step represents the main opportunity for “participation”: the researchers are supposed to share information, including any observations unrelated to the research topics that they feel are important, and to open the floor up to the community’s questions – whether related to the findings or not.
• Step 20: Preparation of site reports:
  Each Research Team prepares a report on the QIM findings and research process for the community it visited. These site reports are to benefit the district and the national-level analysis.
• Step 21: Feedback session/data verification at the district offices:
  The district findings are presented to the DEC for feedback and verification. The DEC discusses the findings in the context of their day-to-day work and is facilitated in developing their own, districtlevel recommendations. In addition, the expectations the DEC had for QIM are discussed and suggestions on the way forward for QIM are made.
• Step 22: Preparation of district reports:
  The district reports present an overall analysis of the findings from the three QIM communities from the district perspective. That is, the purpose of this report is to discuss the relevance and implications of the findings for district operations.

Lessons-learnt:

Training, data collection, field work

• There is a need to ensure that training on analysis is emphasised. The training should include practical exercises.
• Sufficient time has to be allocated for work on the data collection sheet so that the participants go through the entire sheet together at least once. This provides an opportunity for clarifying questions that may still be too vague. If the data collection sheet has been translated into local languages, quality control of the translation has to be performed.
• During the training workshop, the issue of community expectations should be brought up explicitly. The Research Team members should be briefed on how to respond to the community expectations.
• Clear guidelines on how to determine the well-being classification of the respondents need to be included in the training and the handouts. Research Teams must be sure to document this type of identification of the respondents on the data collection sheets.
• The training module on gender should include issues of inter-team dynamics. Rules on team behaviour with respect to gender should be drafted by the teams and taken along into the field.
• In depth analysis of the findings and recommendations at the district level is a demanding task. It is necessary that the district leaders are assisted in the process (ideally by members of the National Team). One way to do this would be to bring all the district report teams together for one week between the field phase and the analysis phase, so they could prepare their reports with technical assistance.

The Data Analysis Phase

• Step 23: Analysis of community and district findings at the national level:
  The National Team performs national-level analysis on the findings from the communities and districts in the context of the Policy Analysis.
• Step 24: Preparation of the interim QIM 2 report:
  In order to provide quick information to national-level policy and decision-makers, an interim report is produced that summarises the survey findings.
• Step 25: In-depth analysis of the findings for the national-level and consultations with national-level stakeholders:
  Further analysis of the QIM findings is needed in order to prepare the final report and to fully take advantage of the information in developing recommendations for the national level. During this process, national-level stakeholders are consulted for data verification and for feedback on the usefulness of the recommendations.
• Step 26: The QIM 2 final report:
  The Final Report presents the finalised analysis of the QIM findings along with the recommendations for national-level actors.

The Presentation and Dissemination Phase

• Step 27: Design of the presentation and dissemination phase:
  The presentation and dissemination phase needs to be designed to determine the groups to be reached, and the types of publications and presentations to be produced.
• Step 28: Design of presentations and publications:
  QIM publications and presentations need to be specially tailored to specific audiences.
• Step 29: Making the presentations:
  Presentation events are held for different groups of stakeholders in development and poverty reductions.

Lessons-Learnt:

Reports, presentation

• QIM requires that a ”popular version” brief on the policy areas that are to be investigated is prepared. The briefs should include basic information on the policies/interventions, including the rights and entitlements that they establish. Contact addresses should be provided for further information. The Research Team should present the briefs to the community both verbally and graphically. The Research Teams need specific training on how to carry out this step.
• The number of research areas has to be weighed up against the time required to complete the work. The more areas that will be assessed the greater the time requirements. A careful balance has to be struck between these two variables.
• QIM can also be used as an ”early warning” mechanism for uncovering the effects of new policy intervention if interim reports are developed as quickly as possible

References and Sources Used

Lobb-Rabe, Alison, Renate Kirsch and Nelson Gomonda. 1998. “Qualitative Impact Monitoring of Poverty Alleviation Policies and Programmes in Malawi – Volume 2: Analysis of the Process.”

Lobb-Rabe, Alison. 2002. “The second round of Qualitatitve Impact Monitoring of Poverty Alleviation policies and programme in Malawi - QIM2: Final Report, Volume 2: Analysis of the process.”

Lobb-Rabe, Alison. August 2000. “Analysing the Data of the Field Phase & Writing the Site and District Reports: Guidelines for Carrying Out QUIM 2.”

Lobb-Rabe, Alison and Nelson Gomonda. 1999. “Restructuring QUIM: Decentralising to the Districts.”

National Economic Council. 2001. “Qualitative Impact Monitoring of the Poverty Alleviation Policies and Programmes in Malawi – Volume 1: Survey Findings.”

National Economic Council. November 2000. “Qualitative Impact Monitoring of the Poverty Alleviation Policies and Programmes in Malawi: Interim Report.”

National Economic Council. August 2000. “Qualitative Impact Monitoring (QUIM) Exercise: Report on visit to Northern and Central Region.”

National Economic Council. July 2000. “Report on the QUIM District Briefing, Southern Region.”

National Economic Council. May 2000. “QUIM 2 Concept Document (Draft)”.