It’s been a week since the Busan Outcome Document was made public and the overall consensus seems to be pretty positive. As always, the proof of the pudding will be in the eating and we will not know the actual contents of that dish until after mid-2012 – by then there should be an agreement on the working arrangements for what has been dubbed a ‘Global Partnership’.
Until then the language used gives reason for cautious optimism. It covers a wide range of important issues such as South-South cooperation, increased private sector and civil society engagement, innovative financing, fragile states, stronger country ownership and specific tailored support, transparency and accountability – all with an underlying emphasis on shared and equal partnership between actors. In doing this it acknowledges that this is new era for the world economy, not just development aid. Given that this has been possible while bringing new actors to the table is an important step and creates opportunity going forward. As Brian Atwood summarises, “It does not turn night into day, as some might have hoped, but it does place all parties on equal footing—and on the same path.” So what does this mean for IHP+Results and our work going forward in 2012?
- The New Deal for engagement in fragile states and accompanying language around fragile states is both interesting and timely, given that Sudan and Mauritania have recently opted into the monitoring round for 2012 and several other IHP+ countries currently are, or recently have been, classified as fragile states.
- The language around learning from existing efforts to monitor aid effectiveness is also welcome and echoes what we have been trying to do in our work so far through engaging with the OECD/DAC as closely as possible.
- The increased emphasis on ensuring this really works for countries (and is owned and lead by them) as well as creating space for civil society engagement is very much in line with our aims for the coming year, where amongst other things we are piloting approaches to institutionalise monitoring through existing country processes. We will be exploring additional opportunities through our membership of the IHP+ M&E Working Group.
If you have any comments or would like any further information then please do get in touch by using the form at the bottom of this page. See here for information on IHP+Results’ mini-debate at Busan.
Presentation by James Fairfax on ‘Scorecards as Tool to Strengthen Accountability – An Example from the Health Sector’ as presented at a mini-debate on Busan on 30th November 2011.
I was pleased to be invited to join the International Health Partnership (IHP+) Civil Society Consultative Group (CSCG) meeting on 12 November. It was a great opportunity for me to learn about what is and isn’t working for Civil Society Organisations (CSOs) as they try to engage with the IHP+.
Falling enthusiasm?
I was particularly struck by signs that CSO enthusiasm for engaging with IHP+ processes could be waning, in spite of some evidence (albeit mixed and sometimes contradictory) that CS involvement has improved over the 4 years since the IHP+ was born.
A number of recurring themes emerged as possible limiting factors:
- Finding the time to engage.
- Getting access to relevant and timely information so that it was clear how and on what issues to engage.
- A lack of clarity about whether it was worth the effort.
Key take-home messages
I think there are some important messages to take home from this.
- More work is needed to ensure adequate resources for CSOs in policy processes – and so I was pleased to hear that the Civil Society Health Policy Action Fund has been extended for a further 2 years.
- Communication needs to be improved – more regular, more useful, more accessible and transparent – this could include making better use of the CSO pages on the IHP+ website.
- The structures for formal CS engagement – at global- and country-level – need to be established where they don’t currently exist.
- We need to get better at measuring the quality of civil society engagement – and the CSCG had some good discussions about how to use IHP+Results to take steps in that direction.
Time for cautious optimism
But I also think that right now there is more willingness by IHP+ stakeholders to address these issues than at any stage in the IHP+’s history. And there is reason for cautious optimism about a clear and fundamental role for civil society. Both IHP+Results 2010 Performance Report and the stock-take of IHP+ progress have stressed the importance of CS engagement to enable stronger mutual accountability. Since the IHP+Results 2010 performance report was published, there is more data available on the implementation of IHP+ commitments, and IHP+Results Scorecards provide the tools that can enable a dialogue around mutual accountability. But there is also clear recognition of the potential limitations of the self-reported data on which IHP+Results’ report is based. So civil society’s role during IHP+Results 3rd round of monitoring (to be held in 2012) – in triangulating data, using the findings of the report to promote real dialogue and mutual accountability – will be critical. I’d like to see civil society capitalise on this opportunity through renewed, organised action, and I look forward to IHP+Results providing any support we can to make it happen.
Tim Shorten
IHP+Results Project Manager
We have received a variety of comment and feedback on IHP+Results 2010 performance report since it was launched on 6 April. The suggestion that the scorecards can be used for ‘naming and shaming’ provides us with the opportunity to emphasise how we think these tools can be used.
Scorecard presents an aggregated picture – and aggregated data is different for each agency
Whilst the scorecards provide a simple overview of the performance of each participating signatory, it is important to keep in mind that they are of limited use to make comparisons between agencies. We have set out a number of assumptions and limitations that we have had to make in applying the agreed reporting framework and respectively for the results as shown in the scorecards.
A key message is that each agency scorecard presents an aggregated picture of the performance of that agency using a weighted aggregation. The data that is aggregated for each agency is different – not always covering the same countries, not always available for all of the standard performance measures. There are factors that affect the performance of each agency in each country which are difficult to reflect at an aggregated level.
We have been transparent about these considerations in our narrative report. To better understand the data we have supplemented the scorecards with disaggregated data (available on our website) to further promote transparency and to enable the performance of agencies to be reviewed at the appropriate level.
The Scorecards: A tool to inform ongoing conversations
Even thought there are pre-assumptions and limitations we had to apply, we feel that the scorecards and the results shown, provide a good starting point to further conversations about health sector aid effectiveness, as they help to identify shortcomings but also successes. Using the scorecards as tools for a definitive judgement on performance of an agency would be misleading and moreover be counter-productive in creating mutual accountability. We hope that signatories will continue to use the scorecards as an input for discussions and to further their understanding of the areas covered in the survey to help define their policies, goals and focus.
Reflections on the scorecard rating
Another point made is the concern that the ratings in the scorecards don’t match up with people’s views and experiences. We wanted to briefly take the opportunity to reflect on these concerns, and to set out some thoughts on how to use the scorecards.
Limited opportunity to triangulate self-reported data
IHP+Results had limited opportunity to extensively triangulate the data provided by any agencies during the process of data collection and analysis. This was on one hand due to time and resource constraints, and on the other due to the imperative to minimise transaction costs.
It is important to recognize that the scorecards (as applied in the 2010 monitoring) provide a mechanism to present the data that we have been given by participating signatories in a simplified way, which first and foremost aims to help visualise the data collected, as well as to give a better understanding of progress that’s has been made towards achieving health-related MDGs. The ratings shown in the scorecards are results of applying transparent criteria. They should be seen as an agency’s report (based on their self-reported data) against the standard performance measures.
The following methodological issues need to be taken into account when looking at any scorecard:
- How we perform aggregation – a weighted rather than an unweighted approach (http://ihpresults.net/how/limitations/#quantitative)
- (Low) Level of ambition of the targets that were agreed
- Bias of using a sub-sample of countries to rate the agency
To get a fuller sense of the performance of any participating signatory, we’d suggest looking at the disaggregated data.
Scorecards as a tool to inform ongoing conversation
We’d also encourage anyone who has a different view to use the mechanism that IHP+Results offers (including comments options on our website) to help sharpen the assessment.
IHP+Results believes that the scorecards should be used as an input to a conversation rather than a finite judgment of performance, a view that was reinforced at a WHO seminar on 8 April. We hope that differences in opinion will not serve to undermine the credibility of the results, but can be used to strengthen this process towards the 2011/12 monitoring. This would be consistent with the spirit of ongoing learning that has been visible since the start of IHP+Results in 2009, and will help continue progress towards mutual accountability.
As stated in our 2010 Performance Report, and as with any other monitoring & evaluation process, limitations exist that need to be taken into account when trying to understand the findings.
The IHP+Results team has received feedback on some of these limitations, which we have responded to, and summarised below:
The data that we report is self-reported from participating IHP+ signatories. It was difficult to undertaken significant triangulation during the 2010 process due to the need to minimize transaction costs for participating signatories. Feedback has emphasized that establishing a mechanism to triangulate self-reported data is necessary to further increase the credibility of the data and findings. The need to minimize transaction costs remains, and there is also a clear recognition that this is consequently not an easy question to answer. For now, it is clear that there are likely to be differences of opinion between what we have reported and the views of reporting signatories’ stakeholders. We welcome these opinions as they are an essential part of further debate and taking the next steps towards real mutual accountability.
The Standard Performance Measures that we used to track civil society engagement were acknowledged as weak from the outset. We have been transparent about this, but we should recall that they were developed and agreed by the IHP+ Mutual Accountability Working Group, which included civil society representatives. This suggests a lack of a simple, transaction-light, and strong mechanism to measure civil society engagement. We are consulting with civil society representatives to develop a stronger measure to track this important issue more effectively. We are asking for feedback and suggestions from other individuals and organizations to input into this ongoing discussion.
We supplemented the measures of civil society engagement with a qualitative survey, which was conducted with a limited number of organizations that were already engaged with IHP+ processes. The results from these inputs do point towards an increase in civil society engagement. Using the agreed measures, we can say that 7/10 countries reported that there was more space for CSOs in health sector coordination mechanisms. It is not at all contradictory to state this, but also to hold the view that CSO engagement in country policy processes has not improved enough or is not yet good enough.
In view of the above, IHP+Results 2010 report made clear recommendations that:
- the findings should be discussed by IHP+ signatories at global and country level.
- relevant adaptations should be made to the methodology – in particular on the measures used for civil society engagement.
This is a learning process, and steps will be taken to improve the process and reporting framework. In May, the IHP+ Executive Team discussed recommendations on changes for IHP+Results third round of monitoring, and agreed that the IHP+ Working Group on Mutual Accountability should meet once more (in early July) to discuss a limited number of outstanding issues. We’ll provide an update on these discussions in due course.
Since the launch of the 2010 Performance Report in early April, we’ve received some valuable feedback on how participating signatories use or intend to use it within their organisations:
- The WHO: organised a seminar on 8 April, where IHP+Results gave a presentation of findings and recommendations. Senior WHO staff gave positive feedback on the report and scorecards, with interest in further to explore how this approach could be applied to other areas of work. The WHO have also discussed the report with senior representatives of development agencies.
- The Global Fund: welcomed the availability of data that applies the Paris Declaration indicators to the health sector. They are interested in providing the online scorecard on their website to encourage further comment and feedback to help identify ways to improve their performance.
- UNAIDS and UNICEF both intend to use the report to further discussion with their country office colleagues.
- SIDA will use the information as an input to preparations for the 4th High Level Forum in Busan.
- The EC presented the headline findings from the report at a meeting of the EU Member State health experts, and encouraged all member states to participate.
- The Government of Nigeria has requested a briefing on IHP+Results at a senate hearing on aid effectiveness on 17 May 2011.
IHP+Results has been busy disseminating the report to a range of stakeholders. Each of the 10 participating countries received copies of the report and the scorecards for further use to stimulate discussion on health sector aid effectiveness – as already happened in Burundi. The findings were also presented at the Action for Global Health conference on 3 May and will be discussed at a technical side event at the World Health Assembly on 18 May.
Keep an eye out here for updates on how our report is contributing to the debate.
Lessons to share from monitoring the IHP+
DFID has recently published a tender for a programme to strengthen accountability in the health sector in Uganda. The Health Sector Accountability Project (HSAP), which is part of a wider DFID Uganda Accountability Programme (UAP), will focus on efforts to understand and plan better health service delivery, including the implementation of the Uganda IHP+ Compact (likely to be signed later this year).
Over the past two years IHP+Results has developed and implemented a cutting edge mechanism – using scorecards – for monitoring accountability within the health sector – specifically for an annual independent assessment of the performance of the IHP+. We have just published our second annual performance report.
Our experience to date has provided questions and lessons that should be relevant for the implementation of the DFID HSAP, such as:
- How to ensure that there is an agreed monitoring framework that minimizes transaction costs and promotes comparability amongst partners?
- How to ensure that all partners have a consistent understanding of the key terms related to the monitoring?
- How to ensure that there is sufficient civil society capacity to engage with the monitoring process, particularly around the more technical, bureaucratic aspects of aid effectiveness?
- How to ensure sustainability, including through using agreed indicators in national performance frameworks and monitoring forums?
IHP+Results report provides the first health sector specific picture on the implementation of aid effectiveness commitments, drawing heavily on the Paris Declaration monitoring framework. We are keen to bring the invaluable experience that we have gained through monitoring the IHP+ to the HSAP, and would be keen to work as part of the team that implements the programme.
Dr Shaun Conway, IHP+Results’ Programme Director, presented the findings from our 2010 Performance Report at a high-level seminar at the World Health Organisation (WHO) on 8 April.
Reflections and questions
Shaun’s presentation (What’s the Score on the IHP+?) was warmly received by two panel respondents – Dr Jason Lane, Senior Health Policy Adviser at the European Commission, and Dr Andrew Cassels, Director of Strategy, Office of the WHO Director General – who made the following observations:
- IHP+Results is contributing to transparency and accountability within the IHP+
- The process has helped to link up EC country offices and headquarters
- The findings will be important in the run-up to the High Level Forum (HLF4) in Busan – especially through the OECD Task Team on Health as a Tracer Sector (TT HATS)
Interesting questions were also raised, by the respondents and in the following discussion:
- How can this be institutionalised at the country-level to make it more sustainable?
- Is the IHP+ tackling the biggest problems?
- Is the speed of change sufficient?
- How important is the brand of the IHP+ – how can the added value of the IHP+ which has been articulated by countries be built into a more standardised approach?
- How can agencies use these findings to promote change?
IHP+Results: a tool to support positive change
The meeting expressed strong support for the “systematic and well-presented” approach that IHP+Results had used, and recognised that IHP+Results Scorecards and disaggregated data provide useful tools to promote dialogue at country-level.
This was a central theme of the discussion – a country level focus is a necessary and appropriate means to ensure the sustainability of health sector aid effectiveness monitoring (through Joint Annual Reviews), and to continue the IHP+’s support to countries in making the best use of scarce resources. IHP+Results should focus on promoting awareness and understanding of our work at the country level and within agencies, and ensuring that these findings are used to support positive change. This should also help accelerate the speed of change, particularly if our communication efforts include civil society at country-level.
The IHP+Results team welcomed this feedback, which largely affirmed our plans and goals for the next 3-4 months. Our immediate focus will be on disseminating and stimulating discussion around our findings, and on gathering feedback so that we can seek to strengthen our approach for our next and final year of monitoring in 2011/12. To keep up to date with our progress, follow us on twitter @IHPResults or sign up to our newsletter.
This week sees the launch of our IHP+Results annual performance report for 2010, together with scorecards with ratings of the performance of 15 Development Partners and 10 Country Governments.
Our mandate and call to action for this comes from the IHP+ Global Compact, signed by 49 heads of governments and development agencies, since it was constituted in 2007. This commits signatories to an “…independent evidence-based assessment of results at country level and of the performance of each of us individually, as well as collectively”.
We began the journey of IHP+Results a little over 2 years ago, in 2009, just as the IHP+ was struggling to find ground in a few early adopter countries. The Framework for Monitoring Performance and Evaluation of the initiative, developed by IHP+ signatories provided an initial conceptual map of the territory. This shows causal pathways that suggest improved funding for coherent and prioritised health plans, delivered in more coordinated ways, could result in greater health impacts within countries.
Framework for Monitoring Performance and Evaluation of the IHP+
Of course, the real world is far more messy than this! The IHP+ is trying to influence entrenched international aid for health that is delivered by dozens of agencies through multiple initiatives. It is also trying to address highly variable dynamics that how countries use external assistance to deliver health services. This is a complex, somewhat unpredictable system. Relationships between aid delivery and results are often not really straightforward. Take the phenomenon of fungibility, for instance, where external aid actually decreases government spending on health service delivery.
When we started speaking to people on the ground to make sense of how the system is responding to the IHP+, we found a lot of hard to answer questions. “What value does this new initiative add to the work we are already doing on aid effectiveness?” was one we heard repeatedly within countries.
When the IHP+ signatories convened in Geneva during the first part of 2008 for a high-level Ministerial review of the IHP+, there was a renewed consensus that the IHP+ is really about Mutual Accountability for delivering on past and future commitments. Dr Margaret Chan, Director-General of the World Health Organisation (WHO) and a strong advocate for the IHP+ from the start, directed a plea to me for IHP+Results to “Keep it simple! we just need to know whether it is working.”
We took a view within the IHP+Results consortium that it would not be particularly helpful to use a conventional monitoring and evaluation approach and the over-simplistic theoretical Results Framework if we were serious about facilitating mutual accountability for results. So we set out on a different ‘sense-making’ approach, to probe this complex system in a variety of ways and see what we could learn. We introduced an innovative set of performance monitoring tools, such as scorecards for development partners and countries. These are designed to provide for ‘external’ monitoring as well as an intervention within the system that could facilitate change through enabling accountability.
After a difficult first round of monitoring in 2009 that served as a critical learning experience we recommended that the IHP+ Scaling-up Reference Group set up a Mutual Accountability Working Group to build consensus on the methodology and and evolved set of tools for this process. Most important, this gave representatives from Country Governments, Civil Society, Academia and Development Agencies an opportunity to collaboratively design the set of performance measures and agreement of ways of working that would build ownership and get buy-in to the process.
The second round of data collection for the annual IHP+Results performance appraisal took place in the third quarter of 2010. This yielded almost 5,000 data points for the 15 Development Partners and 10 Country Governments that volunteered to participate in the process. Whilst this raised a number of practical challenges to manage, we overall experienced a genuine demonstration of commitment to the process by the responsible individuals within these institutions, to make this work.
The Annual Performance Report for 2010 now available for download is the result of this collaborative effort. This provides a summary of key findings, an emergent set of observations and conclusions about where the IHP+ is headed and how this is influencing change at both the country and global levels. Individual scorecards are also available for each agency and country government that we are convinced provide new information and tools in a versatile format that can be used to promote individual agency accountability, mutual accountability between the IHP+ signatories (especially between development partners and country governments), as well as greater public accountability.
Our relaunched IHP+Results website provides transparent access to the data and interesting ways of interacting with this from either a specific development partner or country perspective, to gain a better understanding of what this information reveals about the IHP+. We will be publishing a series of articles on this site over the coming weeks to share further insights into the findings.
We are also keen to explore what lessons and insights the IHP+ and our IHP+Results process can contribute to the deliberations of the High-level Panel on Aid Effectiveness when the aid community meets in Busan at the end of this year.
There are interesting and crucial conversations that need to be had about what this new information means for development and how to use this to promote mutual accountability for the health MDGs. We hope that you will contribute to these through the online forum and by posting comments on this site.
Dr Shaun Conway, Director of IHP+Results
Over the coming days and weeks we will use this space to capture the points of interest that are arising from IHP+Results 2010 performance report . These may be things that the IHP+Results team feel are particularly important to highlight, or could equally be drawn from feedback from IHP+ stakeholders. Please send any feedback or points of discussion to nicole@human-scale.net, and please keep coming back to the site to see what’s going on, or sign-up to our newsletter to get monthly updates.
