Credible accounts of causation in complex rural contexts

Credible accounts of causation in complex rural contexts

Credible accounts of causation in complex rural contexts James Copestake, July 2013 The ART project 1 Argument There is scope for more and better qualitative impact assessment in development. It is useful to distinguish confirmatory and exploratory qualitative approaches to this. In more complex contexts exploratory approaches are more useful.* But development agencies may have a cognitive bias against them

* the most pervasive fallacy of philosophical thinking is neglect of context (John Dewey, quoted in McGilchrist, 2010:141) 2 Outline Context Definitions Examples of qualitative IA The ART project QUIP Interim conclusions

3 Context (1): historical How to promote food security, diversified agricultural livelihoods and rural poverty reduction across Africa? Increasing complexity of the task globalisation, climate change, rapid ruralurban transformations, new actors. Rising role of private sector and mixed publicprivate modalities e.g. challenge funds. 4 Context (2): politics of evidence Political pressure on the development aid industry for better evidence of what works best, where and why (upward accountability). Political and ethical case for stronger feedback loop

from (and to) intended beneficiaries of development interventions (downward accountability) . Fear that increased accountability may constrain what development intermediaries can do i.e. erosion of professional room for manoeuvre (e.g. see Eyben, 2013). 5 Context (3): methodological 1. Perennial debate over credibility of qualitative, quantitative and mixed methods (e.g. see Shaffer, 2013). 2. Recognition of scope for strengthening the range and quality of qualitative and mixed method approaches (e.g. Stern et al, 2012; Philips and White, 2012). 3. ART project action research into self-reported attribution linked to quantitative monitoring of

change as an alternative to statistically inferred attribution that dispenses with the need for a comparison group. 6 Definitions (1): credibility Sufficient evidence and argument to enable A to convince B that X contributed to Y in a particular context. With qualitative research in a complex context strict replication is impossible. This means it is harder to sustain a sharp distinction between validity (broad research design) and reliability (how implemented). Credibility is closer to reasonable than rational (McGilchrist, 2010) 7

Definitions (2): complex context A complex context can be defined as a setting in which the influence of X on Y is confounded by factors Z that are impossible fully to identify, hard to measure accurately, impossible fully to control. Additional complexity arises if the nature and values of X and/or Y is also uncertain. 8 Definitions (3): causation X credibly causes Y in a particular context if: There is strong evidence that X and Y happened; Several stakeholders independently say X was a necessary cause of Y;

their explanations of how X caused Y are plausible and consistent; There are no equally plausible counter-explanations. Bonus criteria the magnitude of changes in Y they attribute to X is plausible; Y varies according to exposure to X in a way that is consistent with the causal mechanisms identified. 9 Two approaches to Qualitative IA/R Confirmatory Exploratory Closer to impact

evaluation. Closer to impact research. Emphasis on ex ante specification of theories of change and hypothesis testing. Emphasis on more

open-ended enquiry, to minimise proproject biases. A more deductive approach. A more inductive approach. Focused attention (McGilchrist, 2010)

Vigilance, sustained attention & alertness 10 Confirmatory IA: work forward from a given intervention and theory of change Exploratory IA: work backwards and blind from a given outcome to possible causes

X Y Examples of qualitative impact assessment and research Confirmatory Theory based approaches Process tracing General elimination methodology Contribution analysis Outcome mapping Philips and White (2012) Group 1

Exploratory WIDE3 Ethiopia PADev The ART QUIP? Philips and White (2012) - Group 2?? (but they emphasise participation) 12 WIDE Ethiopia (Mokoro)

Long-term complexity informed research into implementation and impact of all development activities in twenty rural sites. 1994-2015 (three stages of field work). Community, household and individual levels. Discrete module based research packages, integrated into a single NVivo database. Synchronic and diachronic analysis focusing on: policy journeys, cultural disconnects, and social interactions at the development interface. See Bevan (2013) 13 PADev (Participatory Assessment of Development) Intense community event to construct a joint history of development activities for an area with up to 50,000

inhabitants. Sequence of PRA exercises focusing on: events, changes, project inventory, project ranking, linking projects and changes, assessing impacts by wealth group, assessment of specific agencies. Pro-forma Excel sheets used to facilitate data collection, analysis and reporting. Eleven workshops conducted (so far) in Northern Ghana and Burkina Faso, supported by Dutch NGOs and researchers. See Dietz et al. (2013). 14 The ART QUIP (1): outline Collective action research to design and test a qualitative impact protocol to supplement internal NGO project monitoring. Piloting with two projects in Ethiopia and two in

Malawi (2012-2015). Strand 1: monitoring of food security using the Individual Household Method (IHM) as part of project implementation (baseline and two repeat visits). Strand 2: two rounds of QUIP studies conducted by independent researchers. Strand 3: evaluation of the QUIP. 15 The ART QUIP (2): design Adaptation of the Imp-Act QUIP. Semi-structured interviews with intended project beneficiaries incorporating open-ended questions about major household level changes and reasons for them. Reliance on self-reported attribution. No control group. To reduce pro-project bias neither researcher nor respondent need be aware of the activities being

evaluated. Triangulation with IHM data to enhance reliability and permit some quantification of the magnitude of changes. 16 The ART QUIP (3): relationships Commissioner (and other end users) Implementing agencies Monitoring (using IHM) Lead evaluator Lead impact

researcher and team Impact assessment (using QUIP) Intended beneficiaries 17 The ART QUIP (4): design issues Briefing of lead researchers, gate-keepers and respondents - managing expectations. Sample selection and size use of IHM data. Timing, recall & respondent fatigue time lines. Intra-household impacts focus groups. Sequencing of open and closed questions. Note taking, recording and reporting quality assurance vs cost.

Generalising from narrative reports coding and scoring options. 18 Interim conclusions (1) Exploratory methods of IA are relatively underdeveloped and tested. This perhaps reflects stronger congruence of confirmatory IA methods with the interests and mind-set of development agencies themselves (e.g. for methods that dovetail with results based management). This is potentially dangerous. To offset this there is scope for more work on the design, piloting and comparative assessment of exploratory IA methods. 19 Interim conclusions (2)

A core problem with IA is how to choose between different methods, taking into account the potential biases associated with each. Empirical evidence on the magnitude of bias is rarely available. Statistical sampling and selection bias are exceptions, and this perhaps helps to explain why they get more attention: a cognitive bias towards the known over the unknown. In the absence of more comprehensive evidence, choice of IA method reflects individual and disciplinary preferences about the likely importance of different biases. 20 Interim conclusions (3) How to assess the relative importance of possible design biases (scoping, sampling, attribution system) versus implementation biases (arising from lack of

knowledge, incentives and conflicts of interest). The table suggests ex ante preferences are important. Source of bias Approach 1. Mostly quantitative (Statistically inferred attribution) 2. Mostly qualitative (Selfreported attribution) DESIGN (validity) First order problems Second order problems IMPLEMENTATION

(reliability) Second order problems First order problems 21 Interim conclusions (4) Where do these preferences come from? McGilchrist (2010) suggests one answer: Type 1 methods fit more comfortably with left brain dominance: rational, abstract, precise, generalising, certainty seeking, depersonalised. Type 2 methods fit more comfortably with right brain dominance reasonable, concrete, less certain, contextual, person tather than

idea oriented, emphasising difference rather than sameness. 22 References (1) Bevan, P. (2013). Researching social change and continuity: a complexity-informed study of twenty rural community cases in Ethiopia, 1994-2015. Mokoro Ltd. Oxford. Copestake, J. (2013). Draft guidelines for the pilot qualitative impact protocol (QUIP). ART Project. University of Bath. Bath. (available on request) Dietz, T, Bymolt, R, Belemvire, A, van der Geest, K, de Groot, Dieneke, Millar, David, . . . Zaal, F. (2013). PADev Guidebook. Amsterdam: African Studies Centre, University of Leiden, and Royal Tropical Institute, Amsterdam. Eyben, R. (2013). Uncovering the politics of 'evidence' and 'results'. A framing paper for development practictioners. .

Institute of Development Studies. Brighton. Retrieved from www.bigpushforward.net 23 References (2) McGilchrist, I .(2010) The master and his emissary: the divided brain and the making of the Western World. Shaffer, P. (2013) Ten years of Q-Squared: implications for understanding and explaining poverty. World Development, 45:269-85. Stern, E, Stame, N, Mayne, J, Forss, K, Davies, R, & Befani, B. (2012). Broadening the range of designs and methods for impact evaluations DFID Working Paper. White, H, & Phillips, D. (2012). Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework. 3ie Working Paper 13. London, Washington DC and New Delhi: International Initiative for

Impact Evaluation. 24

Recently Viewed Presentations

  • The Basis Agreement - learningcircle.ubc.ca

    The Basis Agreement - learningcircle.ubc.ca

    First Nations Health Governance UBC Learning Circles September 29, 2010 Presented by: Grand Chief Doug Kelly Chair, First Nations Health Council Health Governance and Indigenous People - Global Context The Tripartite First Nations Health Plan sets out an ambitious agenda...
  • Reactions to the Peace Treaties - Aurora Public Schools

    Reactions to the Peace Treaties - Aurora Public Schools

    Reactions to the Peace Treaties. Historiography: changing interpretations of events. ... Trianon, St Germain, and Neuilly caused little concern. But Sevres made Britain large and Turkey small, carved up the MidEast in drastic ways, questioned Britain's motives ... the treaty...
  • The Best Investor Pitch Deck Outline 0) Cover

    The Best Investor Pitch Deck Outline 0) Cover

    Lots of articles about us: TechCrunch / Forbes / CNN / FOX. We're growing fast! We've got lots of clients/users, some are brand name clients, look at our monthly growth, and we have a growing pipeline that will generate lots...
  • Board Report | December 16, 2014  ANYONE WISHING

    Board Report | December 16, 2014 ANYONE WISHING

    Board awarded contract for A&E Services in August 2012 to Royal Engineering. Notice to Proceed issued on September 5, 2012. Kick-off meeting was held on October 3, 2012
  • Презентация PowerPoint

    Презентация PowerPoint

    Current trend: from "market" and "power" corruption to "parochial" corruption . Businessmen pay cash bribes and a fixed fee for protection to policemen relatively more rare than in the 90-th. The cases of violent business extortions are less widespread than...
  • Background - QUT

    Background - QUT

    The nursing curriculum at QUT changed in 2013/4 with the introduction of bridging unit in bioscience at CB, and the pharmacology unit being moved forward to 4th semester. Consequently, we were no longer sure that the diploma students were still...
  • The creation that is you!

    The creation that is you!

    What is resilience? "Resilience is the process of adapting well in the face of adversity, trauma, tragedy, threats or significant sources of stress - such as family and relationship problems, serious health problems or workplace and financial stressors" (American Psychological...
  • Engaging with Police and Crime Commissioners (PCCs) Compact

    Engaging with Police and Crime Commissioners (PCCs) Compact

    Compact Voice has also produced a briefing on the Compact and Police and Crime Commissioners which makes the case for why PCCs should engage with their local Compact.It includes information about what local Compacts are and how they can help...