• Home
• Publications
• Who We Are
• Contact Us

University of Toronto ·  Trinity College

Research Report [pdf]

John Kirton, Caroline Bracht, Julia Kulik and Madeline Koch
Global Health Diplomacy Program, University of Toronto
September 24, 2015

We gratefully acknowledge the valuable research assistance provided in the production of this research report by Alissa Wang, Marissa Young, Rowena Symss, Brittaney Warren, Kayla Jacobs, Aislin Flynn, Sarah Scott and Becky Carpenter.

Introduction

In September 2007, the heads of the governments in the Caribbean Community (CARICOM) met in Trinidad to discuss the prevention and control of non-communicable diseases (NCDs). In the first year following this historic Port of Spain Summit (POSS), how much did the CARICOM members comply with the 27 commitments they had made? The answer is important for several reasons. Compliance with such leaders' commitments is critical to solving this major health problem in the Caribbean and to lowering the substantial economic cost to its countries, territories, companies, communities and citizens alike. The collective commitments made at summits by members' top leaders add a comprehensiveness, coherence and authority to the control effort that no other actor can offer. Commitments made by leaders indicates a high priority. The effective control of NCDs requires top-level, whole-of-government commitment. Compliance in the first year following the summit should be strong, for that is when leaders and their citizens believe that that they are able and willing to comply with the commitments, remember the commitment and expect effective action to take place, before conditions change and other problems and priorities arise to shift attention elsewhere.

Assessing compliance with the leaders' commitments is the most authentic approach for evaluating the effectiveness of the POSS. This approach starts with the exact text of the commitments that the political leaders themselves made, rather than using subsequently selected indicators chosen by and familiar to health professionals, even if the implementation of the latter can be more readily traced (Samuels, Kirton and Guebert 2014). It fills a vital empirical gap, for the existing work on implementing key indicators from the POSS has no data for the first year after the summit, from 2007 to 2008, and thus there is no baseline for assessing advances made since. Using the leaders' actual commitments allows for the application of the proven methodology for assessing compliance with summit commitments, developed since 1991 by George von Furstenberg and Joseph Daniels (1992), Ella Kokotsis (1999) and the G8 Research Group (2007). It thus permits the comparison of the results with those from two decades of analysis of the compliance performance of other plurilateral summit institutions, notably the G7, G20 and BRICS. It also does so embryonically with the POSS progeny — the UN's High Level Meeting (HLM) on the Prevention and Control of NCDs in September 2011 and its review HLM in 2014.

To chart first-year compliance with POSS commitments, the Global Health Diplomacy Program (GHDP) at the University of Toronto conducted top-down, internet-based, retroactive compliance assessments of all of the 27 commitments it had identified for the period beginning with the summit until September 2008 (see Appendix A). In doing so it followed the same method used for its regular annual and special compliance assessments of priority commitments made by the G7/8, G20 and BRICS. As of September 23, 2015, 23 of the 27 commitments had been assessed for compliance and are used in the analysis in this report. They cover all the subject categories where commitments had been made. These findings are sufficient to judge the feasibility and value of applying the global summit methodology to a regional summit institution and to assess any empirical patterns that arise.

Methodologically, the study showed that it was feasible to conduct such research, even eight years after the commitments had been made. Data were obtained for almost all 15 full members of CARICOM and some of the five associate members for most of the 27 commitments. The validity and reliability were similar, and the comprehensiveness was superior, to the two existing sets of POSS implementation indicator data — the indicator grid with 20 components used since 2008 and the more recent indicator implementation data set from the Caribbean Cooperation in Health Phase III strategy (CCH-3), which is more extensive but still incomplete. The grid and CCH-3, for example, omit the cross-cutting POSS commitment on gender. The GHDP compliance research showed the value of moving beyond self-reported data to develop definitions of each commitment or indicator, interpretive guidelines and a scoring metric for what constitutes compliance or implementation, as well as produce detailed empirical reports on each member's behaviour — all publicly reported and thus open to stakeholder feedback, empirical enhancement, and public and scientific use.

Substantively, the results showed that CARICOM members got off to a slow start in complying with the leaders' POSS commitments (see Appendix B). At the end of the first year, overall compliance with the 23 assessed commitments averaged only −0.21 (40%), far lower than almost all G7/8 summits and all G20 and BRICS ones. Across the 23 assessed commitments, no single commitment had full or no compliance. Only five were positive, two neutral and 15 negative. Among the 20 members, none had full or no compliance. Only three members had positive compliance: Trinidad and Tobago, Jamaica and Grenada. One was neutral and 16 were negative.

Methodology

To chart first-year compliance with POSS commitments, during the summer of 2015 nine analysts working under the direction of John Kirton and Caroline Bracht conducted compliance assessments of all 27 commitments, using data available online on implementing actions carried out by governments between the POSS in September 2007 until September 2008. The analysts followed the same methods used for the regular annual and special compliance assessments conducted by the G7 Research Group, the G20 Research Group and the BRICS Research Group (G8 Research Group 2007). In a few cases, contact was made with the Toronto consulates of CARICOM members to aid in the research.

This research contributes to the larger field of assessing compliance with summit and international institutions' commitments in several ways.

First, it was the first systematic application of the standard compliance assessment methodology, developed for the G7/8 and similar global plurilateral summit institutions, to a regional summit institution, that of CARICOM, which, like the G7/8 holds annual summits and sometimes inter-sessional ones. Previously only very select, diagnostic work had been conducted in relation to regional institutions such as the trilateral Security and Prosperity Partnership Summit among the leaders of the United States, Canada and Mexico.

Second, it was the first application of the methodology to a special, ad hoc, subject-specific summit, in this case on NCDs.

Third, it was the first application of the methodology to an institutionalized group consisting of members small in population, geography and, for most, gross domestic product. All were recent or current territories of current G7 countries. The five associate members of CARICOM that United Kingdom Overseas Territories (UKOTS) posed the methodological challenge of assessing whether they or their imperial governments were legally and operationally responsible for the compliance that they had made.

Fourth, it was the first application of the methodology to an international institution with a dedicated secretariat, and in this case with a relatively large one.

Fifth, this application was the first deeply retroactive compliance assessment, requiring searching for eight-year-old compliance data about small governments with limited statistical and reporting capacity. It succeeded in securing sufficient data to provide scores for all the commitments for almost all 15 CARICOM members and many of the five UKOTS.

Sixth, this was the first application a complete set of a summit's commitments — all 27 that the POSS made. Previous work had assessed only a small subset of selected priority commitments. A special compliance study had been conducted of all 24 development commitments made by the G20 Seoul Summit in November 2010 (Kirton, Bracht and Rasmussen 2012). Yet while this offered complete coverage of an issue area, it was only a minority of all the commitments produced by the multi-subject G20 summit.
Seventh, this study allowed for a focused look at commitments containing international institutional "mandates" and a comparison with those that had none (Larionova, Rakhmangulov and Shelepov 2015). Of the 27 POSS commitments, six had explicit institutional organizational mandates embedded in the commitment, while the remaining 21 did not.

Compliance Overall

Across the 23 commitments assessed after the first year, compliance averaged −0.21 (40%). Among the 20 members, the average was −0.24 (38%). POSS compliance thus got off to a very slow start. This could suggest that members were two fifths of the way there after the first year, but only based on the unlikely assumption that all were starting from a zero baseline, having done nothing before the POSS was held.

These −0.21 and −0.24 first-year averages are far lower than the +0.50 (75%) average of the major democratic powers in the G7/8 since its summitry started in 1975. Only in three years, based on very partial compliance data, did the G7 summit have a negative score: 1983 at −0.11 (44%), 1988 at −0.48 (26%) and 1990 at −0.14 (43%). Since 1991, the G7/8 summit has always had a compliance performance in the positive range. Both the G20 and the BRICS summits have had positive compliance since their start in 2008 and 2009 respectively. This suggests that size counts. Bigger countries have a greater ability to comply, assuming that their willingness to comply, the ambition of the commitments and the difficulty of compliance are constant across all summit bodies.

This first-year compliance can be usefully compared with the second-year implementation data for some of the 20 indicators previously used to assess the effectiveness of the POSS (Samuels, Kirton and Guebert 2014).

Compliance by Commitment

Compliance by commitment varied widely. No commitment had either complete compliance or none at all. Five commitments had positive scores, two were neutral and 15 had negative scores (see Appendix C). The commitment with the highest compliance at +0.88 (94%) was on mandating the reintroduction of physical education in schools. The second highest at +0.35 (68%) was a commitment to declare the second Saturday in September "Caribbean Wellness Day." The third highest score at +0.25 (62%) was a commitment to establish the programs necessary for research and surveillance of the risk factors for NCDs with the support of universities and the Caribbean Epidemiology Centre and the Pan American Health Organization. The commitment with the lowest compliance at −0.90 (5%) was on accounting for the gender dimension in all programs to prevent and control of NCDs. The second lowest at −0.71 (14%) was a commitment to ban advertising tobacco products to children. The third lowest at −0.65 (18%) was on increasing public facilities such as parks and other recreational spaces to encourage physical activity.

The cluster of commitments categorized under education and promotion achieved the highest compliance scores, with three commitments above the median score of −0.20 (40%). The cluster on tobacco follows, also with three commitments above the median, but with slightly lower scores. Commitments on nutrition follow, with three compliance scores falling closer to the median score. The issue area of tobacco has the highest number of scores below the median; however, it also has the highest number of overall commitments identified and assessed for compliance.

Compliance by Member

Compliance by member also varied widely (see Appendix D). None of the 20 members had a complete score and all complied to at least some degree. The score ranged from +0.30 (65%) for Trinidad and Tobago to −0.67 (16%) for Haiti, with an overall median score of −0.26 (37%) and an average of −0.24 (38%). Only three members had positive compliance: Trinidad and Tobago, Jamaica and Grenada. Guyana had neutral compliance and all other members had negative compliance.

Support from International Institutions

Support from international institutions through explicit mandates contained in the commitment appear to increase compliance (see Appendix E). Such institutions include the Pan American Health Organization, Caribbean Agricultural Research and Development Institute and the Caribbean Food and Nutrition Institute, among others. Of the 23 commitments assessed for compliance, five contained mandates to international institutions. These were the commitments on research and surveillance, monitoring and evaluation, fair trade, food security and trans fats. Only one commitment had a positive score and the overall average was −0.14 (43%), above the overall average of −0.21 (40%).

In contrast, the 18 commitments without such mandates included four with positive scores. The average score for those 18 commitments was −0.24 (38%). Thus, there is a +0.10 difference between those with mandates and those without, as those containing mandates have a higher level of compliance. However, both averages were in the negative range and a +0.10 difference is not very significant on a 200-point scale. Furthermore, the commitments without mandates had a much broader range of compliance, including a much higher score for the top-ranked commitment and a much lower score for the bottom-ranked one. There is thus little evidence for concluding that the presence of an institutional mandate in a commitment increases compliance by a meaningful amount in the first year of compliance.

References

G8 Research Group. 2007. "Background on Compliance Reports: Methodology." http://www.g8.utoronto.ca/evaluations/methodology/g7c2.htm.

Kirton, John J., Caroline Bracht, and Leanne Rasmussen. 2012. "Implementing the G20 Seoul Development and Employment Commitments: An Assessment." Paper produced with the financial support of the International Labour Office through the ILO–Korea Partnership Program, G20 Research Group. http://www.g20.utoronto.ca/biblio/SDC_Assessment.pdf.

Kokotsis, Eleanore. 1999. Keeping International Commitments: Compliance, Credibility, and the G7, 1988–1995. New York: Garland.

Larionova, Marina V., Mark Rakhmangulov, and Andrey Shelepov. 2015. "Assessing G7/8 and G20 Effectiveness in Global Governance." In The G8-G20 Relationship in Global Governance, Marina Larionova and John J. Kirton, eds., pp. 77–108. Farnham: Ashgate.

Samuels, T. Alafia, John J. Kirton, and Jenilee Guebert. 2013. "Monitoring Compliance with High-Level Commitments in Health: The Case of the CARICOM Summit on Chronic Non-Communicable Diseases." World Health Bulletin 2014(92): 270–76B. doi: dx.doi.org/10.2471/BLT.13.126128.

Von Furstenberg, George M., and Joseph P. Daniels. 1992. "Can You Trust G7 Promises?" International Economic Insights 3(5): 24–27. http://www.g8.utoronto.ca/scholar/furstenberg1992/document.html.

[back to top]

Appendix A: 2007 Port of Spain Summit Commitments

[back to top]

Appendix B: Compliance Scores

[back to top]

Appendix C: Compliance by Commitment

[back to top]

Appendix D: Compliance by Member

[back to top]

Appendix E: Commitments with Mandates