Why undertake a Citizen Report Card Exercise?
The results of the Citizen report cards provides an insight into the citizens perspectives on pro-poor services that are not always captured in traditional information gathering exercises undertaken by institutions or organizations. The views of the citizens are captured with regard to issues such as availability and affordability of services from the different service providers. The method also differentiates between different groups of citizens; or even by geographic region, rural and urban residence, and level of household expenditure. This differentiation also has to be respected later on when the service providers design and implement improvements to their services. What is important is that the appraisals conducted using the Citizen Report Cards are augmented with expert assessments and service provider reports to obtain a comprehensive picture of the strengths and weaknesses of service provision to various groups. To this end, secondary data proves useful to complement and crosscheck the data, if it is available.
Seven steps of undertaking a Citizen Report Card Survey
Step 1: Defining the scope of the Citizen Report Card analysis
- Clarity has to be achieved on the main issues and problems that one wishes to examine. The more focused the problems and issues are the better the information gathered will be targeted. Information is not only of value for those undertaking the survey. It is also necessary to determine what the citizens have to say about the problems and issues being raised. The information augments actions and activities being undertaken by others in the same field.
- Focus of the survey also has to be defined. The focus of the survey encompasses either only a single service provider, several providers or it can be even be used to compare different services providers with each other.
- Purpose and use of the survey has to be agreed upon in order to ensure that the survey can be focused properly. Within the purpose it is also important to determine how the results will be used. If the information is to be use for improving the services of the providers then it is necessary that specific information is gathered needed for this. Very different information would have to be gathered if the results are to be used to improve development cooperation projects.
Figure 2: Seven steps for undertaking a Citizen Report Card Survey
- Data requirements and collection requires different methods to be used. Of importance here is to determine the exact specific population and sample size for the survey. The different possible types of surveys to be undertaken need to be defined, whether random samples will be undertaken or focus groups. This will also determine the skills that the staff and field workers will require while undertaking the survey.
- Resource requirements for the whole exercise have to be thought through. The length and the extent of the survey will be dependent upon and determined by the financial and personnel resources available. Setting the parameters that will be used in the sampling, including sample size and method of interviewing will also dictate the resource requirements. It is important to also reserve some resources for contingencies that may arise.
Step 2: Defining the methodology
- The methodology used should reflect the information that one wishes to gather. Consultations with key stakeholder groups in government, private sector, civil society, and selected academic institutions are undertaken in order to clarify both the purpose and use of the survey and therefore the methodology to be used.
- Inputs of different stakeholders should be integrated into in the design of the survey questionnaire. A balance has to be sought between quantitative and qualitative information. The citizen report card methodology is based upon quantitative methods that are enhanced through qualitative information derived from interviews and observations.
- By mixing the focus group techniques (qualitative approach) with questionnaires (quantitative approach) the citizen report card not only reduces bias, it also reveals errors in measurement. Furthermore, the two approaches ensure data cross checks and verifications can be undertaken and it helps to increase response rates.
Getting the “right mix” between quantitative and qualitative information gathering is important.
- Qualitative methods generally provide information about citizens’ feelings or insights and the main techniques used include observation and unstructured interviews. Qualitative approaches examine the “feelings” and perceptions people have about service providers. The approach often helps to clarify issues in advance of undertaking a quantitative survey and can also be used to generate hypotheses.
- Quantitative methods usually are based more on numbers that can possibly be statistically assessed. Correct sampling procedures, unambiguous questionnaires and ability to process and assess the data using computer programmes are just some of the features of quantitative methods. Quantitative approaches provide hard numerical facts on how much was done where, when and by whom. The method is best used when seeking to rank order answers to establish overall levels of satisfaction with the services provided and to rank these and if the answers are to be weighted in order to assist in decision making on changes and reforms to be undertaken.
Step 3: Selecting the sample
- During the design stage of the citizen report card method the sample also has to be determined. Scientific sampling allows the researcher to make accurate inferences about a larger population, estimations and testing of hypotheses. A decision has to be taken about the kind, method or sample as well as about the size of the population. Information needs, desired level of confidence and the available resources are aspects that need to be considered while deciding upon the sample size.
Sampling follows the following main steps:
- Defining the population
- Census of sample
- Sample design
- Sample size
- Opting for fixed or sequential sampling
Step 4: The questionnaire
- Questions should be focused and valid. Questions can be open-ended or closed-ended (partially close-ended, closed-ended with ordered choices and closed-ended with un-ordered choices).
- The instructions for completing the questionnaire have to be precise, bias in the instructions; the wording of the questions or the choice of responses should be avoided.
- The questionnaires should be pre-tested in order to assess the validity of the questions and whether unexpected responses are possible.
- Ranking the questions in general order of priority (from most important to least important).
- Questions should be clustered, which in turn can also be ranked.
- Clustered questions can follow a basic order: why, who, how and what.
- Lastly, by placing the most interesting or important questions at the start of the questionnaire the highest level of interest of the respondent can be assured since it may gradually decline.
Step 5: Coding, pre-testing and analysing the results
- Coding quantitative questions in advance is practical and time saving process that assists those undertaking the questioning from rapidly filling out the questionnaires. Open-ended responses have to be analysed and then a number attached to the response. To avoid confusion it is advisable that one person undertakes the coding exercise.
- Testing whether the questionnaire actually works is essential as part of the overall quality control.
- Training the enumerators undertaking the survey is equally important, short clear manuals and getting the enumerators to undertake the testing all help to ensure that the process of actual data collection is well done.
- Several techniques exist to analyse the data, including: averages, data ranges, frequency and midpoint. Tables and graphs should be used to present the data visually. Where spatial data exists (i.e. maps) data should also be presented spatially on maps.
- Interpretation of the data is crucial it is at this stage the information gathered becomes tangible. Bringing all of the information together and presenting it in an understandable manner ultimately determines the value of the information gathered. Different perspectives have to be bought together and the issues described in an easily understandable manner for the citizens.
Step 6: Making the results known
- Making the results known to a wide audience requires a well-documented final report. “Popular versions” of the report containing the key findings and recommendations for the masses should be publicised in order to ensure wide circulation. Effective use of the media is also necessary, including print, radio and television media coverage. Media coverage helps in facilitating a broad public discourse about what citizens think and what administrators should do.
- Typical contents for the final report includes: executive summary, objectives, methodology, major findings, implications and recommendations. Popular versions need to summarise the main findings and recommendations. Use of simple graphics and tables in the popular versions improves the readability.
Step 7: Perpetuating the exercise: Institutionalisation
In order to avoid the citizen report card being a single exercise a locally appropriate approach for perpetuating the process has to be considered from the very start. This is not only necessary in order to be able to repeat the exercise but is also of importance in gaining a feedback and checking whether the recommendations from the initial survey are being followed through. Gaining citizen feedback on the quality and availability of services can be of concern for: the citizens and civil society, the service providers themselves or a regulatory government institution. While civil society may establish an own independent organisation to continue or follow-up on the exercise, the service providers may undertake the work through their own public relations office. On the other hand, government may establish some form of “watchdog” institution.