Page - (000122) - in Evaluating Climate Change Action for Sustainable Development

The lowest unit of analysis was the individual project. This was the most appropriate level to unveil factors affecting performance, but as the resources for the evaluation were limited only a sample of projects could be looked at in sufficient depth. The evaluation team prepared rapid reviews of 19 projects – about one third of the entire portfolio. Projectswere selected on the basis of four criteria: thematic area (adaptation, mitigation or REDD), project size (based on estimated cost), project scope (global, regional or national) andmaturity. The evaluationmade use of aTheory of Change (ToC) approach to address severalevaluationquestions.AToCdepicts thelogicalsequenceofdesiredchanges (also called “causal pathways” or “results chains”) to which an intervention, programme, strategy etc. is expected to contribute. It shows the causal linkages between changes at different results levels (outputs, outcomes, intermediate states and impact), and theactors and factors influencing thosechanges. Initially inspired by guidance provided by the Global Environment Fund6 the UNEP Evaluation OfficehasbeensystematicallyusingaToCapproachinprojectandsub-programme evaluations since2009. TheToC for each component of theCCSP, and then for theCCSPas awhole, was reconstructed based on a review of strategic documents and UNEP staff interviews, andusingbest practice indeterminingcorrect results levels. Figure6.3 presents theoverall reconstructedToCfortheCCSP.ThereconstructedToChelped identify the expected outcomes of UNEP’s work on Climate Change and the intermediary changes between outcomes and desired impact. Thus, it allowed to cluster outputs and define summary direct outcome statements cutting across components,whichwouldproveveryuseful toframedatacollectionandsynthesize findings on sub-programmeeffectiveness. The reconstructionof theToCalsohelped todetermine thekeyexternal factors affecting theachievementofoutcomes, intermediarystatesand impact, namely the drivers thatUNEPcouldinfluence throughawareness raising,partnershipsetc., and theassumptions thatwereoutsideUNEP’scontrol.As thesewerekeydeterminants of the likelihood of impact, upscaling and sustainability of the sub-programme, it was important to identify themearlyonso thatadequate informationon their status could be collected in the course of the data collectionphase. The reconstructedToCwas alsoused to assess the internal logic andcoherence of the formal results framework of the sub-programme. Therefore, the formal results framework comprised of the Sub-programme objective, ExpectedAccom- plishments andProgrammeofWorkOutputswascomparedwith the reconstructed ToC,anddifferencesbetween the twowerepointedout.For instance, in the formal results framework the results levels at which Expected Accomplishments and Programme ofWorkOutputs had been set were inconsistent between andwithin 6GEFEvaluationOffice2009,Fourthoverallperformancestudyof theGEF:TheROtIHandbook: TowardsEnhancing the Impacts ofEnvironmental Projects,Methodological Paper #2.Web link: https://www.thegef.org/gef/sites/thegef.org/files/documents/M2_ROtI%20Handbook.pdf 104 M.Carbon