Lessons learnt

Summary

The experience of conducting each IHP+Results review cycle contributes to ongoing improvement of the tools and processes we offer partners to monitor and report on their performance. The following challenges have been identified that should be addressed in the 2011 Review:

Establishment of target dates – Currently there are no agreed target dates. Therefore firm statements cannot be made about the sufficiency and significance of the reported progress.
Weakness in specific indicators
Tighten criteria for rating – Need to clarify if ratings are based on absolute or trend data. We need to decide whether we are rating progress, over time/between two points OR performance at a given point in time.
Risk that measuring skews priorities – There is a concern that the agreed SPMs “assume importance in their own right, becoming a barrier to rigorous thinking and innovative practice that aims to meet the broader objective of aid effectiveness”.
Data verification – Documentation to support certain claims, e.g. civil society support, should be provided.
Development Partner reporting – The decision to whether DPs where “active” in supporting health sector development in a country was left for the agencies to decide – with advise, where requested. Data was only requested for countries where the government had volunteered to participate (i.e. 10 in 2010).
Questions ‘not applicable’ or ‘data not available’ – Improved guidance on when to use either.
Evaluation – Evaluation would be an important addition to the monitoring process in 2011. Potential questions: Have the results been obtained, and why? What factors or actions may have contributed to any changes, or lack of changes, in aid effectiveness?

For country-specific explanations, please click here.

Click here for a detailed description of the methodology used in IHP+Results 2010 monitoring.

Detailed Explanations

Establishment of target dates

No target dates were formally agreed by the Working Group, which makes firm statements about the sufficiency or significance of the progress reported difficult. For 2011 the IHP+ should make a decision if it wants to establish target dates for the indicators. On one hand targets can provide useful focus and galvanise action, on the other hand aid governance is dynamic, subject to huge variables and contextual shifts, and a target reached in 2010 may not be met in 2012 if DPs, governments, bureaucrats, country or global priorities change.

Weakness in specific indicators

In particular DP indicator definition for supporting civil society engagement (8DP) is weak: What constitutes development partner “support”, and how could a broader definition (which includes advocacy by DPs for civil society engagement) be tracked. Further work is also needed on civil society qualitative survey.

Other SPMs that would benefit from a closer look include:

2DPb: Many partners report 0 or 100% for this indicator, suggesting that the metric and assumptions behind it are too crude to accurately measure.
1DP/1G: Definitions of “equivalent agreement” and “documented support for a compact” need to be tightened and better communicated.
6DP/6G: Need to promote greater understanding that single national performance assessment frameworks should be used to assess DP programmes.
7DP/7G: a clearer understanding of mutual assessments.
The value of collecting information on specific health indicators should also be reconsidered where WHS or other data already exists.

Tighten criteria for rating

It should be agreed whether ratings are based on absolute or trend data. In most cases, for quantitative SPMs, an absolute target has been established (either through the Working Group or using the Paris Framework as a precedent). But the rating criteria include a measure of progress, which requires baseline data – which has not always been available, making ratings impossible in some cases. It has to be decided if we are rating progress over time/between two points or performance at a given point in time.

Risk that measuring skews priorities

As with the OECD/DAC Paris survey, there is a concern that the SPMs agreed by the SuRG to monitor IHP+ progress “assume importance in their own right, becoming a barrier to rigorous thinking and innovative practice that aims to meet the broader objective of aid effectiveness”.

Data verification

IHP+Results 2011 survey should seek additional information from respondents to enable verification of the self-reported answers provided. For example, when development partners report that they support civil society engagement in the health policy process, they should provide documented evidence.

Development Partner reporting

In 2010, we left the decision to DPs on where they are “active” in supporting health sector development. Where we were asked to, we advised that data should be collected in only those countries where the agency is actively involved in health sector coordination mechanisms (either directly or indirectly through a silent partnership). It was also agreed, in order to limit transaction costs for development partners, that data should only be submitted for those countries where the government had volunteered to participate (ie 10 in 2010). The SuRG may wish to reconsider this decision.

Questions ‘not applicable’ or ‘data not available’

The IHP+Results 2011 survey should include tighter guidance on when a question may be considered not applicable to a development partner. This needs to be sensitive to their various business models but also to relate accurately to the transaction costs experienced by countries in managing the health aid they receive.

Evaluation

The reported data brings up further questions about why progress has or has not happened. But the agreed 2010 reporting framework did not allow space for in-depth evaluation. Evaluation would be an important addition to the monitoring process in 2011. The critical evaluation questions will be to ask whether the results seen in this survey really have been obtained, and to ask why? What factors or actions may have contributed to any changes, or lack of changes, in aid effectiveness?