I have been pleased to participate in two workshops on accountability in recent weeks. The first, hosted by WHO and the Government of Tanzania, focused on assessing accountability mechanisms in 10 countries using the Country Accountability Framework (CAF) developed by WHO under the auspices of the IHP+ and the Commission on Information and Accountability. The second, hosted by Chestrad and the Governments of Ghana and Nigeria (with major support from UNFPA and GAVI) had a much broader agenda focussing on Access, Effectiveness, Results and Accountability.
The participants at each meeting were different – in Tanzania each country group included representatives largely from the Ministry of Health and UN agencies (with the occasional parliamentarian); whereas the meeting in Ghana was naturally targeted at Civil Society Organisations. But there were some similar goals: promote awareness of accountability mechanisms, provide a framework for analysis; identify actions that need to be taken to strengthen accountability mechanisms.
Both meetings presented a framework on accountability with high levels of convergence in the key areas that enable effective accountability, which in my view are neatly presented in the Global Accountability Project framework of the One World Trust:
It is interesting to reflect that, through using the CAF tool, the Tanzania meeting was able to make progress on identifying concrete actions – and even begin to discuss the resource and implementation implications of these in an emergent ‘roadmap’. However the civil society meeting struggled to get past discussions about concepts and language on accountability, drawing on experience across a diverse range of contexts.
Was Tanzania more successful because a plan to strengthen accountability started to emerge? On one hand yes – because creating these opportunities is not easy, and so we need to make as much progress as we can each time an opportunity arises; but the Tanzania meeting was more narrowly and strongly focused on one of the clearer areas of strengthening accountability – generating evidence – so perhaps success was more achievable. On the other hand, Ghana naturally focused on participation and review, and the mechanisms for enabling and strengthening these aspects of accountability are less clear and more political – and so perhaps require a different definition of success. Whereas Tanzania gave a group of relevant technical experts (largely Ministry of Health officials) the opportunity to focus on plans to strengthen elements of the accountability system that were largely within their control[1], Tanzania gave an equally relevant group (of civil society advocates) an opportunity to reflect on what accountability means and to identify issues that need further discussion with their constituents.
For me what became clearer is that these kind of analytical but action-oriented opportunities need to be located in a longer-term process in order to have any chance of success. Tanzania is linked into a broader process, with expectations of and the potential for resources to be allocated to follow-up actions. The follow-up for Ghana is less clear – although the recently completed window for health policy advocacy funding (through the Health Policy Action Fund) could help, as could additional strategic discussion and support through the International Health Partnership (IHP+).
I was struck by the importance of getting the right people round the table, and by the challenge of doing this in a meaningful way for civil society organisations (who often have representation and internal accountability issues to manage).
And I also increasingly feel that there are growing opportunities to make progress on accountability (you can see my closing remarks from the Ghana meeting here): there are a substantial number of mechanisms designed to improve transparency (particularly at the international level[2]); the potential of technology to broaden participation and enable access to evidence for accountability by not just ‘advocacy officers’ in NGOs but also of beneficiaries of services is developing fast; and there are increasingly inspiring examples of how people can hold those in power to account. But significant work is needed to enable country-level actors to develop clearer ideas about what they can do to strengthen accountability in their specific contexts, and more appropriate materials are needed to support this process. This is why Re-Action is embarking on a survey of CSOs to gather views about what could enable their uptake of IHP+Results scorecards for aid effectiveness accountability in the health sector.
If you’d like to share your views please comment below, or to participate in the survey please contact nicole@human-scale.net
[1]Civil Registration and Vital Statistics (CRVS), Results, Maternal Death Surveillance and Response (MDSR), Resources, eHealth, Accountability processes, Advocacy and outreach
[2] Amongst others, the following initiatives have a focus on some aspect of accountability in the health sector: COIA, Countdown2015, PMNCH, IATI, IHP+Results, E4A.
The 3rd round of IHP+ monitoring (IHP+Results) started on 20 February 2012
Thirty-six IHP+ signatories have opted to report on the effectiveness of the health sector aid (up from 25 in 2010) using IHP+Results agreed reporting framework. The process is scheduled to run during the first half of 2012, with a report due to be published in September. More information is available at www.ihpresults.net/2012monitoring.
IHP+Results has articulated a number of objectives for this monitoring round:
- Delivering a high-quality process which enables comparisons with IHP+Results 2010 monitoring;
- Strengthen the methodology and minimise transaction costs;
- Strengthen linkages with other initiatives – in particular with the IHP+Monitoring & Evaluation Working Group, and the Commission on Information & Accountability for Women’s and Children’s health;
- Provide illustrative examples of how the IHP+Results reporting framework can be institutionalised at country-level;
- Improve the use of IHP+Results Scorecards as tools for advocacy and accountability, through focused work with Civil Society.
We’re encouraged that the same 25 signatories that participated in 2010 will be repeating the exercise in 2012. This provides scope for further analysis of performance at country- and global-level. It’s also exciting that 9 new countries and 2 new Development Partners see potential value in this reporting framework.
We are aware that the IHP+Results Scorecards, as tools for advocacy and strengthening mutual accountability, has not been maximised yet.Whilst they have been used in some contexts (eg in Mozambique results were presented at the bi-annual health sector review meeting in July 2011; and the Scorecard was discussed by the Senate Committee on Appropriations in Nigeria in May 2011) there is a lot of work that needs to be done to improve utilization of the tool. We look forward to working more in this area throughout 2012.
We’ll provide updates here as the process unfolds…
It’s been a week since the Busan Outcome Document was made public and the overall consensus seems to be pretty positive. As always, the proof of the pudding will be in the eating and we will not know the actual contents of that dish until after mid-2012 – by then there should be an agreement on the working arrangements for what has been dubbed a ‘Global Partnership’.
Until then the language used gives reason for cautious optimism. It covers a wide range of important issues such as South-South cooperation, increased private sector and civil society engagement, innovative financing, fragile states, stronger country ownership and specific tailored support, transparency and accountability – all with an underlying emphasis on shared and equal partnership between actors. In doing this it acknowledges that this is new era for the world economy, not just development aid. Given that this has been possible while bringing new actors to the table is an important step and creates opportunity going forward. As Brian Atwood summarises, “It does not turn night into day, as some might have hoped, but it does place all parties on equal footing—and on the same path.” So what does this mean for IHP+Results and our work going forward in 2012?
- The New Deal for engagement in fragile states and accompanying language around fragile states is both interesting and timely, given that Sudan and Mauritania have recently opted into the monitoring round for 2012 and several other IHP+ countries currently are, or recently have been, classified as fragile states.
- The language around learning from existing efforts to monitor aid effectiveness is also welcome and echoes what we have been trying to do in our work so far through engaging with the OECD/DAC as closely as possible.
- The increased emphasis on ensuring this really works for countries (and is owned and lead by them) as well as creating space for civil society engagement is very much in line with our aims for the coming year, where amongst other things we are piloting approaches to institutionalise monitoring through existing country processes. We will be exploring additional opportunities through our membership of the IHP+ M&E Working Group.
If you have any comments or would like any further information then please do get in touch by using the form at the bottom of this page. See here for information on IHP+Results’ mini-debate at Busan.
Presentation by James Fairfax on ‘Scorecards as Tool to Strengthen Accountability – An Example from the Health Sector’ as presented at a mini-debate on Busan on 30th November 2011.
I was pleased to be invited to join the International Health Partnership (IHP+) Civil Society Consultative Group (CSCG) meeting on 12 November. It was a great opportunity for me to learn about what is and isn’t working for Civil Society Organisations (CSOs) as they try to engage with the IHP+.
Falling enthusiasm?
I was particularly struck by signs that CSO enthusiasm for engaging with IHP+ processes could be waning, in spite of some evidence (albeit mixed and sometimes contradictory) that CS involvement has improved over the 4 years since the IHP+ was born.
A number of recurring themes emerged as possible limiting factors:
- Finding the time to engage.
- Getting access to relevant and timely information so that it was clear how and on what issues to engage.
- A lack of clarity about whether it was worth the effort.
Key take-home messages
I think there are some important messages to take home from this.
- More work is needed to ensure adequate resources for CSOs in policy processes – and so I was pleased to hear that the Civil Society Health Policy Action Fund has been extended for a further 2 years.
- Communication needs to be improved – more regular, more useful, more accessible and transparent – this could include making better use of the CSO pages on the IHP+ website.
- The structures for formal CS engagement – at global- and country-level – need to be established where they don’t currently exist.
- We need to get better at measuring the quality of civil society engagement – and the CSCG had some good discussions about how to use IHP+Results to take steps in that direction.
Time for cautious optimism
But I also think that right now there is more willingness by IHP+ stakeholders to address these issues than at any stage in the IHP+’s history. And there is reason for cautious optimism about a clear and fundamental role for civil society. Both IHP+Results 2010 Performance Report and the stock-take of IHP+ progress have stressed the importance of CS engagement to enable stronger mutual accountability. Since the IHP+Results 2010 performance report was published, there is more data available on the implementation of IHP+ commitments, and IHP+Results Scorecards provide the tools that can enable a dialogue around mutual accountability. But there is also clear recognition of the potential limitations of the self-reported data on which IHP+Results’ report is based. So civil society’s role during IHP+Results 3rd round of monitoring (to be held in 2012) – in triangulating data, using the findings of the report to promote real dialogue and mutual accountability – will be critical. I’d like to see civil society capitalise on this opportunity through renewed, organised action, and I look forward to IHP+Results providing any support we can to make it happen.
Tim Shorten
IHP+Results Project Manager
We have received a variety of comment and feedback on IHP+Results 2010 performance report since it was launched on 6 April. The suggestion that the scorecards can be used for ‘naming and shaming’ provides us with the opportunity to emphasise how we think these tools can be used.
Scorecard presents an aggregated picture – and aggregated data is different for each agency
Whilst the scorecards provide a simple overview of the performance of each participating signatory, it is important to keep in mind that they are of limited use to make comparisons between agencies. We have set out a number of assumptions and limitations that we have had to make in applying the agreed reporting framework and respectively for the results as shown in the scorecards.
A key message is that each agency scorecard presents an aggregated picture of the performance of that agency using a weighted aggregation. The data that is aggregated for each agency is different – not always covering the same countries, not always available for all of the standard performance measures. There are factors that affect the performance of each agency in each country which are difficult to reflect at an aggregated level.
We have been transparent about these considerations in our narrative report. To better understand the data we have supplemented the scorecards with disaggregated data (available on our website) to further promote transparency and to enable the performance of agencies to be reviewed at the appropriate level.
The Scorecards: A tool to inform ongoing conversations
Even thought there are pre-assumptions and limitations we had to apply, we feel that the scorecards and the results shown, provide a good starting point to further conversations about health sector aid effectiveness, as they help to identify shortcomings but also successes. Using the scorecards as tools for a definitive judgement on performance of an agency would be misleading and moreover be counter-productive in creating mutual accountability. We hope that signatories will continue to use the scorecards as an input for discussions and to further their understanding of the areas covered in the survey to help define their policies, goals and focus.
Reflections on the scorecard rating
Another point made is the concern that the ratings in the scorecards don’t match up with people’s views and experiences. We wanted to briefly take the opportunity to reflect on these concerns, and to set out some thoughts on how to use the scorecards.
Limited opportunity to triangulate self-reported data
IHP+Results had limited opportunity to extensively triangulate the data provided by any agencies during the process of data collection and analysis. This was on one hand due to time and resource constraints, and on the other due to the imperative to minimise transaction costs.
It is important to recognize that the scorecards (as applied in the 2010 monitoring) provide a mechanism to present the data that we have been given by participating signatories in a simplified way, which first and foremost aims to help visualise the data collected, as well as to give a better understanding of progress that’s has been made towards achieving health-related MDGs. The ratings shown in the scorecards are results of applying transparent criteria. They should be seen as an agency’s report (based on their self-reported data) against the standard performance measures.
The following methodological issues need to be taken into account when looking at any scorecard:
- How we perform aggregation – a weighted rather than an unweighted approach (http://ihpresults.net/how/limitations/#quantitative)
- (Low) Level of ambition of the targets that were agreed
- Bias of using a sub-sample of countries to rate the agency
To get a fuller sense of the performance of any participating signatory, we’d suggest looking at the disaggregated data.
Scorecards as a tool to inform ongoing conversation
We’d also encourage anyone who has a different view to use the mechanism that IHP+Results offers (including comments options on our website) to help sharpen the assessment.
IHP+Results believes that the scorecards should be used as an input to a conversation rather than a finite judgment of performance, a view that was reinforced at a WHO seminar on 8 April. We hope that differences in opinion will not serve to undermine the credibility of the results, but can be used to strengthen this process towards the 2011/12 monitoring. This would be consistent with the spirit of ongoing learning that has been visible since the start of IHP+Results in 2009, and will help continue progress towards mutual accountability.
As stated in our 2010 Performance Report, and as with any other monitoring & evaluation process, limitations exist that need to be taken into account when trying to understand the findings.
The IHP+Results team has received feedback on some of these limitations, which we have responded to, and summarised below:
The data that we report is self-reported from participating IHP+ signatories. It was difficult to undertaken significant triangulation during the 2010 process due to the need to minimize transaction costs for participating signatories. Feedback has emphasized that establishing a mechanism to triangulate self-reported data is necessary to further increase the credibility of the data and findings. The need to minimize transaction costs remains, and there is also a clear recognition that this is consequently not an easy question to answer. For now, it is clear that there are likely to be differences of opinion between what we have reported and the views of reporting signatories’ stakeholders. We welcome these opinions as they are an essential part of further debate and taking the next steps towards real mutual accountability.
The Standard Performance Measures that we used to track civil society engagement were acknowledged as weak from the outset. We have been transparent about this, but we should recall that they were developed and agreed by the IHP+ Mutual Accountability Working Group, which included civil society representatives. This suggests a lack of a simple, transaction-light, and strong mechanism to measure civil society engagement. We are consulting with civil society representatives to develop a stronger measure to track this important issue more effectively. We are asking for feedback and suggestions from other individuals and organizations to input into this ongoing discussion.
We supplemented the measures of civil society engagement with a qualitative survey, which was conducted with a limited number of organizations that were already engaged with IHP+ processes. The results from these inputs do point towards an increase in civil society engagement. Using the agreed measures, we can say that 7/10 countries reported that there was more space for CSOs in health sector coordination mechanisms. It is not at all contradictory to state this, but also to hold the view that CSO engagement in country policy processes has not improved enough or is not yet good enough.
In view of the above, IHP+Results 2010 report made clear recommendations that:
- the findings should be discussed by IHP+ signatories at global and country level.
- relevant adaptations should be made to the methodology – in particular on the measures used for civil society engagement.
This is a learning process, and steps will be taken to improve the process and reporting framework. In May, the IHP+ Executive Team discussed recommendations on changes for IHP+Results third round of monitoring, and agreed that the IHP+ Working Group on Mutual Accountability should meet once more (in early July) to discuss a limited number of outstanding issues. We’ll provide an update on these discussions in due course.
Since the launch of the 2010 Performance Report in early April, we’ve received some valuable feedback on how participating signatories use or intend to use it within their organisations:
- The WHO: organised a seminar on 8 April, where IHP+Results gave a presentation of findings and recommendations. Senior WHO staff gave positive feedback on the report and scorecards, with interest in further to explore how this approach could be applied to other areas of work. The WHO have also discussed the report with senior representatives of development agencies.
- The Global Fund: welcomed the availability of data that applies the Paris Declaration indicators to the health sector. They are interested in providing the online scorecard on their website to encourage further comment and feedback to help identify ways to improve their performance.
- UNAIDS and UNICEF both intend to use the report to further discussion with their country office colleagues.
- SIDA will use the information as an input to preparations for the 4th High Level Forum in Busan.
- The EC presented the headline findings from the report at a meeting of the EU Member State health experts, and encouraged all member states to participate.
- The Government of Nigeria has requested a briefing on IHP+Results at a senate hearing on aid effectiveness on 17 May 2011.
IHP+Results has been busy disseminating the report to a range of stakeholders. Each of the 10 participating countries received copies of the report and the scorecards for further use to stimulate discussion on health sector aid effectiveness – as already happened in Burundi. The findings were also presented at the Action for Global Health conference on 3 May and will be discussed at a technical side event at the World Health Assembly on 18 May.
Keep an eye out here for updates on how our report is contributing to the debate.
Lessons to share from monitoring the IHP+
DFID has recently published a tender for a programme to strengthen accountability in the health sector in Uganda. The Health Sector Accountability Project (HSAP), which is part of a wider DFID Uganda Accountability Programme (UAP), will focus on efforts to understand and plan better health service delivery, including the implementation of the Uganda IHP+ Compact (likely to be signed later this year).
Over the past two years IHP+Results has developed and implemented a cutting edge mechanism – using scorecards – for monitoring accountability within the health sector – specifically for an annual independent assessment of the performance of the IHP+. We have just published our second annual performance report.
Our experience to date has provided questions and lessons that should be relevant for the implementation of the DFID HSAP, such as:
- How to ensure that there is an agreed monitoring framework that minimizes transaction costs and promotes comparability amongst partners?
- How to ensure that all partners have a consistent understanding of the key terms related to the monitoring?
- How to ensure that there is sufficient civil society capacity to engage with the monitoring process, particularly around the more technical, bureaucratic aspects of aid effectiveness?
- How to ensure sustainability, including through using agreed indicators in national performance frameworks and monitoring forums?
IHP+Results report provides the first health sector specific picture on the implementation of aid effectiveness commitments, drawing heavily on the Paris Declaration monitoring framework. We are keen to bring the invaluable experience that we have gained through monitoring the IHP+ to the HSAP, and would be keen to work as part of the team that implements the programme.
Dr Shaun Conway, IHP+Results’ Programme Director, presented the findings from our 2010 Performance Report at a high-level seminar at the World Health Organisation (WHO) on 8 April.
Reflections and questions
Shaun’s presentation (What’s the Score on the IHP+?) was warmly received by two panel respondents – Dr Jason Lane, Senior Health Policy Adviser at the European Commission, and Dr Andrew Cassels, Director of Strategy, Office of the WHO Director General – who made the following observations:
- IHP+Results is contributing to transparency and accountability within the IHP+
- The process has helped to link up EC country offices and headquarters
- The findings will be important in the run-up to the High Level Forum (HLF4) in Busan – especially through the OECD Task Team on Health as a Tracer Sector (TT HATS)
Interesting questions were also raised, by the respondents and in the following discussion:
- How can this be institutionalised at the country-level to make it more sustainable?
- Is the IHP+ tackling the biggest problems?
- Is the speed of change sufficient?
- How important is the brand of the IHP+ – how can the added value of the IHP+ which has been articulated by countries be built into a more standardised approach?
- How can agencies use these findings to promote change?
IHP+Results: a tool to support positive change
The meeting expressed strong support for the “systematic and well-presented” approach that IHP+Results had used, and recognised that IHP+Results Scorecards and disaggregated data provide useful tools to promote dialogue at country-level.
This was a central theme of the discussion – a country level focus is a necessary and appropriate means to ensure the sustainability of health sector aid effectiveness monitoring (through Joint Annual Reviews), and to continue the IHP+’s support to countries in making the best use of scarce resources. IHP+Results should focus on promoting awareness and understanding of our work at the country level and within agencies, and ensuring that these findings are used to support positive change. This should also help accelerate the speed of change, particularly if our communication efforts include civil society at country-level.
The IHP+Results team welcomed this feedback, which largely affirmed our plans and goals for the next 3-4 months. Our immediate focus will be on disseminating and stimulating discussion around our findings, and on gathering feedback so that we can seek to strengthen our approach for our next and final year of monitoring in 2011/12. To keep up to date with our progress, follow us on twitter @IHPResults or sign up to our newsletter.
