Consultative Impact Monitoring of Policy - CoIMPact

CoIMPact and....

.... policy

CoIMPact starts with the premise that the results will be useful for carrying the voices of ordinary people into the policy-making arena. The CoIMPact approach recognizes that to do this, simply producing reports is not sufficient. For this reason a comprehensive results dissemination exercise has to be incorporated as one of the “phases” of CoIMPact to open the space for discussion about policy reforms and changes. This is enhanced by the regular involvement in the exercise of those who are in a position to make policy changes, from the key central and line ministries as well as those delivering services to the poor at the decentral or provincial level.

Further, the CoIMPact exercises are not one-off events, but are rather multi-stage and recurring exercises, integrated in the existing Poverty Monitoring Systems of the country.

In the development of a Poverty Reduction Strategy Programme (PRSP), participation and consultation are key concerns. To date, most countries adopting a PRSP have held widespread consultations during the formulation of the strategies. Often this process has been of an ad hoc nature involving civil society and private sector actors, while neglecting elected officials in parliament and elsewhere on the understanding that these tend to be weak institutions. The participation of these organisations in the policy formulation process has generated a demand for the continued involvement of the ordinary people in the monitoring of policy. The experience of CoIMPact exercises carried out to date suggests the method is able to enhance local level participation.

Figure 2: CoIMPact communication and information flows

Participatory monitoring and evaluation, which involves direct consultation with the beneficiaries, particularly the poor, plays an important role in expanding the information base for poverty diagnostics (identifying who the poor are), monitoring policy implementation (what impact this is having on people) and in the retrospective evaluation of the policy.

In addition to the participation of the direct beneficiaries, the collaborating partners in the exercise can include staff of responsible ministries, relevant local Government and decentral Offices, representatives of NGOs / CSOs and research institutions. This supports improved coordination between different agencies in carrying out anti-poverty activities. Additionally, rather than being a strain on scarce resources, it can act as a means of strengthening the capacity of those involved in performing their official duties, by helping them to understand better the needs and priorities of the people they are to serve.

... participatory poverty assessments - PPA

While CoIMPact and Participatory Poverty Assessments (PPA) essentially belong to the same family and have the same overall intention – to continue to use the Voices of the Poor to reinforce policy messages for politicians, technocrats and the media – they differ in a number of ways. The most striking difference is CoIMPact’s reliance on in depth policy analysis before collecting any data. This has a number of benefits. During data collection, it ensures that those going to the field understand fully what they need to investigate and the areas they need to probe. In the report writing stage it means that the various reports can answer specific questions connected to the impact selected policies are having on poor people.

Other differences are the attention given to the institutionalisation of the exercise. Originally, PPAs were designed and supervised by the staff of donor organisations, and managed and implemented by NGOs or research institutions (something that continues to be the case in a number of PRSPs). CoIMPact has rather taken up the issues of institutionalisation, seeking to be embedded in the national policy-making structure. This is done with the express intention of improving the relationship to the policy process.

The third major difference between a traditional PPA and CoIMPact is the system of feedback loops, incorporating all levels of society from the individual through to decentral administration and national level policy makers. These are ensured by involving decision-makers throughout the exercise, including: the choice of the policy to be investigated, the specific design of each round and consultations during the policy analysis, as well as the field research and report writing. The CoIMPact approach considers that this arrangement:

• Assures the information needs of data users and policy makers are met (demand-driven).
• Increases the likelihood that the findings will be accepted.
• Creates the role for the stakeholders as “champions” of the messages contained in the reports.

...gender

Unlike many other data collection tools, such as household surveys, CoIMPact exercises pay a great deal of attention to issues of Gender at various levels of the exercise. Consideration of this issue needs to be made at virtually every step:

• During the policy analysis.
• In the development of hypotheses related to the dimensions of policy interventions.
• In the training of data collection personnel to ensure they are sensitised to the issue.
• In the composition of data collection teams.
• By interviewing men and women separately to allow women in traditional society settings to express opinions which may not otherwise be allowed.
• By using women to interview women’s groups so they do not feel intimidated.
• By collecting sex-disaggregated data and maintaining this disaggregation throughout the documentation and analysis process.
• Through identifying differences in male and female perception and present these in the report.

.... other monitoring tools

CoIMPact exercises are complimentary to other monitoring efforts. Most obviously, the use of other sources of data helps in the triangulation of results and informing survey design. However, the relationship between the various exercises can take on a number of forms:

• The identification of areas for investigation through the CoIMPact approach can be undertaken based on the findings of quantitative exercises.
• The purposeful sampling of specific areas through CoIMPact can be based on the results of nationwide household surveys.
• The investigation of similar policy areas using CoIMPact can ultimately lead to the production of a more comprehensive set of policy recommendations.
• The results of CoIMPact can provide a good basis for the design of quantitative questionnaires.

Institutional set-up for carrying out CoIMPact

Agreeing on the institutional arrangements is one of the key prerequisites for the exercise. CoIMPact requires a strong institutional base for it to have any meaningful impact on policy. Experience to date has entailed focusing on government, while at the same time generating additional links to other institutions, such as NGOs and development partners. Involving a broad spread of actors helps to ensure that the results of the exercise are used in the reform of policy, and that the most relevant policies are investigated.

Within government, there is a need to focus on a strong central organisation, such as the Ministry of Planning, or the Ministry of Economic Planning and Development. The officials and staff of these institutions are to be involved at all stages, from conceptualisation to the final release of the report. However, the exercises have to involve others as well – including the national statistical offices, decentral government, key line ministries and non-governmental organisations. The key is to ensure that the exercise finds its home in an organisation that is committed to introducing, supporting and improving pro-poor policies (or programmes) in a cross-sectoral way and is in a position to push this agenda forward.

Using people from government institutions in the planning, implementation and analysis helps to improve the access to the policy process and generates additional support for the results at hand.

Providing Timely Information

One of the principles behind the CoIMPact approach is to ensure that information on the implementation of policies and indications of outcomes and impact can be provided expeditiously to allow mitigating interventions to be undertaken when necessary. Experience shows that it is possible to carry out a full round of a CoIMPact exercise (from conceptualisation to production of the report) within six months. However, it must be remembered that every country is different, and the first round of an exercise in a new environment may take more time – this can be justified if all the components are put in place, and the key stakeholders are brought “on board”, which eventually will lead to the use of the results of the exercise. Other issues that will have an effect on the length of time are the number of people involved in the data collection and the number of sites visited.

One of the principles behind the CoIMPact approach is to ensure that information on the implementation of policies and indications of outcomes and impact can be provided in a timely and quick manner, to allow interventions to be undertaken. Timing and sequencing should fit with the policy formulation cycle in order to ensure that the findings have an influence.

When to use CoIMPact

CoIMPact is designed to provide feedback to policy and decision makers on people’s perceptions about the outcomes and impact of policy intervention. For it to be successfully implemented anyone of the following conditions needs to be fulfilled:

• The country’s Poverty Monitoring System or national poverty strategy recognises the need for the collection and use of “Qualitative” or “Contextual Data”.
• The Government is committed to integrating the opinions of ordinary people into the policy process through a system of participation or consultation.
• Specific poverty focused policy interventions have been undertaken that need to be reviewed, or if the Government is interested in examining the poverty impact of existing sectoral policies.
• A unit exists within the government and it takes up poverty issues in a cross-sectoral way and the unit is able to develop channels to influence policy-using information generated by a participatory exercise.

Steps to carry out the CoIMPact process

Five broad phases are necessary in a CoIMPact exercise. The precise number of steps implemented in each phase will vary according to the needs of the country and implementing agency. Some key steps must be undertaken. However, they do not form a blueprint for carrying out the exercise. Each country is different and the exercise needs to reflect these differences. The watchwords then are flexibility not just in the planning stage but also in the implementation stage to allow for adaptation to any particular situation, and capacity building so that those taking part in the exercise are able to carry out future rounds.

1. The Preparatory Phase

The purpose of having a preparatory phase is to establish a smooth conceptual basis for the implementation of a round of the exercise, including reaching consensus on the objective, purpose and results so that they are appropriate to the prevailing national situation. At this stage, it is also important to clarify the roles and responsibilities of the key stakeholders and deal with issues of establishing the institutional framework for the exercise. Further, there are some practical issues that need to be addressed, including deciding on the coverage of the exercise (number of sites), the areas to be investigated (following widespread consultations), and an assessment of the financial costs, including a more general assessment of the resources necessary and available (including personnel). The most important steps include:

• Hold widespread consultations with potential stakeholders to help in the development of a conceptual framework and selection of areas for investigation. Taking this approach at this stage will help to ensure support for the results once they come later. The conceptual framework should outline the objectives of the exercise, identify the institutional arrangements and develop a possible budget and source of funding. While a detailed plan (or concept paper) may result from this step, this should also be produced as a briefing sheet to explain in simpler terms what the exercise entails.
• Review previous research and consultation documents (which should be in place following the development of PRSPs in many countries) to identify information needs and gaps that can be investigated, and to prevent duplication of efforts.
• Select administrative areas and sites for investigation and initiate contact with those in local administration who will be involved in the exercise.
• Select the teams who will carry out the policy analysis and the data collection.

In the first iteration of the exercise, the preparation will require time. Occasionally this may seem disproportionate to the time spent on the other phases, however, in the long run the contribution it makes towards ownership and ensuring that the results find their way into the policy process justify the effort. Subsequent rounds of data collection might not need to spend as much time on the preparation phase of the exercise.

Figure 3: The CoIMPact process
(Click inside the image to enlarge)

2. The Policy Analysis Phase

The policy analysis phase examines the extent to which the priorities of the poor have found their way into policy documents, such as the Poverty Reduction Strategy Paper (PRSP) and whether (sector) policies are genuinely poverty-focused. At the same time, it identifies possible indicators from the policy documents against which success or impact can be assessed in a qualitative way. It results in the development of hypotheses for testing during the field phase. The policy analysis also helps to ensure that the findings of the field phase can be linked to policy interventions in the production of the final report. The policy analysis ties in with the preparation phase when key policy and decision-makers identify the information gaps they would like filled and select the areas for investigation. It is important to keep refining the policy analysis after each step, including:

• Conduct a detailed literature review of existing policy documents.
• Consult widely with those involved in the policy area (donors, NGOs and government agencies) to foster ownership of the process and to ascertain the type and kind of information needed.
• Prepare a report on the policy analysis for discussion with stakeholders.
• Prepare and test the interview guidelines.

The analysis requires a number of iterations to allow continuous finetuning and to ensure the incorporation of the views and needs of the broadening circle of stakeholders. Capacity building to carry out further analysis exercises needs to be one of the key objectives of this stage.

The output from this phase of the exercise is the development of a number of hypotheses to be tested. Questions to achieve this are included in a checklist for field-use. The checklist ensures that the key policy areas are addressed by the teams in the field in sufficient detail to make their analysis meaningful and to facilitate the comparison of reports from different sites, enabling the easier production of a final report at national level. Checklists/questionnaires will require testing prior field implementation; this requires time.

3. The Data Collection Phase

Prior to data collection the selection of the multi-disciplinary team has to be undertaken. A balance has to be found between use of government employees, freelancers, and civil society. The former can provide a link to policy, but may have time constraints. The latter may often want to participate, but also have time constraints, and may not have a direct link to policy making. Freelancers will have more time, but may not be able to provide the link to policy.

The data collection phase should collect reliable and relevant information about poverty, the priorities of the poor and the perception of these people about the impact that policies, designed for their benefit, actually are having on them. As well as the data collection, there are three key elements at this stage - training, testing checklists and methodologies, and documentation, even though paying adequate attention to administrative and management issues at this phase will help to contribute to a successful exercise. The steps of the data collection phase include:

• Training the teams to be involved in data collection must be the first step of the field phase – the training needs to deal with the research objective, the rationale for selection of the policy areas, the tools to be used in participatory data collection and building team spirit. Experienced trainers are needed for the training.
• Testing checklists and methodologies to ensure they are appropriate for the field phase and understood by the team. These include group based semi-structured interviews, key informant interviews, case studies as well as the many visualisation tools (i.e. PRA).
• Consultation with potential providers of important information at local administration level allows for the generation of background information, and the triangulation of community responses.
• Community consultations are the key part of the exercise. At the very start of the visit, it is important to explain the goals and objectives of the exercise, to try to combat the problem of raising expectations amongst the community – something that will probably have to be done more than once. A work plan, which suits the time schedule of the community, needs to be agreed and every attempt to involve the marginalized in the community must be made. Before leaving the community, the researchers should hold a feedback session to tell the participants about their findings and give them the opportunity to correct any errors and to discuss further possible solutions to some of the problems raised. Attention should be paid to possible community “power relations” at this stage, which may marginalize weaker groups.
• Documentation – The distinction in information given by different groups (based upon gender, age, etc.) are maintained throughout the documentation and subsequent analysis phase. This involves the filling of notebooks and pre-structured documentation sheets and recording any visualisation, taking note of the respondents age, gender and other relevant characteristics. The documentation (and by extension, the first steps towards the analysis) needs to be done at many stages - after each interview session, at the end of each day; at the end of each site visit; at the level of local administration (for instance decentral level); and at national level. The recording of information at these specific levels helps to ensure that the data is kept differentiated by respondent groups.

The CoIMPact exercise is founded on the principal of “community immersion” that is, the team spends their time with the participating community. This allows the development of a greater sense of empathy between the two, and allows the team, through observation, to triangulate some of the findings, by seeing elements of the daily lives of the community. To facilitate this, a stay of between five days and a week with each community is recommended. Staying for such a length of time also eases some of the strains placed on the participants’ time – appointments, interviews and discussions can be carried out at their convenience.

4. The Data Analysis Phase

The ultimate goal of the analysis phase is to produce a series of reports that reflect the opinions of the communities visited in a manner that will lead to their voices being heard in the policy-making process at various levels. The analysis and report writing consists of processing the results of the field work so that it is meaningful in the context of the research topics and presenting it in a way that is useful to policy makers. The data analysis phase compares the Policy Analysis with the data collected in the communities in order to examine the impact of the policies and interventions being investigated. The biggest difficulty at this stage relates to drawing macro conclusions from micro-analysis at site report level. It is important in the final report to ensure that minor pieces of information are not used to exaggerate findings beyond their real significance. The acceptability of such data will very much depend on the depth of triangulation of findings in the research process, and re-examining controversial information. The major steps are:

Analysis of the results from community level by members of the data collection teams against the key research areas chosen and the production of a site report.
• Analysis of the results at the decentral level (or other appropriate level of local administration) and the production of a decentral report.
• Production of an early draft of the national report for discussion with key stakeholders and provide a quick feedback of main findings.
• Finalisation of the national report by the report writing team.

The result of this phase will be the production of a comprehensive national level report, which includes policy related recommendations. The main findings of this report then need to be produced in a number of formats to help facilitate the dissemination phase.

5. The Dissemination Phase

The Dissemination Phase is designed to ensure that the findings and results of the exercise are shared with those who may be in a position to take some action on them. This is essential bearing in mind that the overall objective of the exercise is to ensure the voices of the poor are heard in the policy-making process and to keep the policy makers informed about progress in implementing and improving their policies. The list of stakeholders to be included at this stage can be a long one – including the sector ministries, the Parliament, local administrations, NGOs, international development organisations, the media, and the public at large. In this regard, CoIMPact exercises have focused on the production of an array of reports and briefings (products) – these can include:

• Overall technical report.
• Popular version of the findings.
• Individual briefing sheets on specific policy issues.
• Technical reports on the process.
• Press briefings.
• Reports for use at decentralised levels of administration.

Providing such an array of information helps to keep various groups of people informed about the impact that policies have on them, and helps government understand unexpected side effects that have emerged. This contributes to the debate on the effectiveness of policy and supports national dialogue on poverty.