Evaluating Complex System Interventions
1 / 62

Evaluating Complex System Interventions Evaluation 2009 Professional Development Workshop Beverly Parsons and Meg Hargr - PowerPoint PPT Presentation

  • Uploaded on

Evaluating Complex System Interventions Evaluation 2009 Professional Development Workshop Beverly Parsons and Meg Hargreaves. What is a System?. A group of interacting, interrelated, and interdependent elements forming a complex whole

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Evaluating Complex System Interventions Evaluation 2009 Professional Development Workshop Beverly Parsons and Meg Hargr' - conlan

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Evaluating Complex System InterventionsEvaluation 2009 Professional Development WorkshopBeverly Parsons and Meg Hargreaves

What is a system l.jpg
What is a System?

  • A group of interacting, interrelated, and interdependent elements forming a complex whole

  • A configuration of parts connected and joined together by a web of relationships

  • The whole is different from, and greater than, the sum of its parts

Systems thinking l.jpg
Systems Thinking

  • A way of understanding reality that emphasizes the relationships among a system’s parts, rather than the parts themselves.

  • Concerned about interrelationships among parts and their relationship to a functioning whole

  • Sees underlying patterns and structures

Foundations of systems theory l.jpg
Foundations of Systems Theory

  • Cybernetics: system feedback, information; differences (that make a difference); human – machine analogy; inclusion of the observer and the observed in the system

  • General systems theory: open systems; system integrity; nested system hierarchy, boundaries, webs, emergence (sum greater than parts)

Systems theories l.jpg
Systems Theories

  • Soft and critical systems: human systems - multiple perspectives, power issues, intractable problems without simple solutions

  • Systems dynamics: systems have reinforcing and balancing feedback loops, circularity, system archetypes, mental models, unintended consequences

More systems theories l.jpg
More Systems Theories

  • Complexity theory: complex adaptive systems; semi-independent, interacting agents; self-organization; emergence; nonlinearity; co-evolution; past is irreversible; future is unpredictable

  • Learning systems: the way that people learn and the systems in which they learn

System boundaries l.jpg
System Boundaries

  • Shows what is inside and outside of the system

    • Geographical (location)

    • Organization (department, unit or function)

    • Physical (money, material, information)

    • Conceptual (goals, mission, purpose, rules)

    • Intangibles (perceptions, awareness, models)

    • Natural or man-made

System relationships interconnections l.jpg
System Relationships (Interconnections)

  • Connections and exchanges among system parts, parts and the whole, and the whole and its environment

    • Flows of information

    • Flows of funding

    • Client referrals

    • Collaborative partnerships

    • Family, community, and social networks

System perspectives l.jpg
System Perspectives

  • Stakeholders’ worldviews and purposes

  • System agents who have different perspectives may pursue different purposes within a given situation

  • Patterns of (mis)alignment of purposes and processes within and across system levels

System change l.jpg
System Change

  • System differences generate creative tension or energy within a system

  • Positive or negative, energy provides potential for system change

  • System change: shifts in patterns (similarities and differences) of system relationships, boundaries, focus, timing, events and behaviors over time and space

System dynamics l.jpg
System Dynamics

  • Random (unorganized)

  • Organized (simple or complicated)

  • Adaptive (organic, self-organizing)

  • All three system dynamics can be present in a complex situation

Random system attributes l.jpg
Random System Attributes

  • Random activity – no pattern

  • Unconnected collection of parts

  • No cause-effect relationships

  • Turbulence – no stability or equilibrium

  • Answers are unknowable

  • No purpose or direction – people react blindly in a war zone or natural disaster

Organized simple system attributes l.jpg
Organized (Simple) System Attributes

  • Stable, static pattern

  • Parts tightly connected machines

  • Predictable cause-effect relationships

  • System can be reduced to parts and processes and replicated

  • Directive leadership, designed change

  • Answers are knowable, with recipes or prescriptions for action

Single organized system ring around the rosie l.jpg
Single Organized System: Ring-Around the Rosie

Simple organized system riding a bicycle l.jpg
Simple Organized System:Riding a Bicycle

Organized complicated system attributes l.jpg
Organized (Complicated) System Attributes

  • Dynamic patterns of feedback loops with many interrelated parts within and across subsystem levels

  • Recursive, non-linear cause-effect relationships; reinforcing and balancing feedback loops maintain equilibrium

  • Expert analysis can identify causal loops, deep structural causes to actions

Adaptive complex system attributes l.jpg
Adaptive (Complex) System Attributes

  • Dynamical patterns – parts adapting, co-evolving with each other and environment

  • Parts are massively entangled and interdependent; nested webs, networks

  • Parts self-organize, learn, and change

  • Equilibrium in flux, sensitive to initial conditions; system change emerges through interactions among parts

Alignment of context program and evaluation dynamics l.jpg
Alignment of Context, Program, and Evaluation Dynamics

  • Context can be random, organized, adaptive, or combination of dynamics

  • Program design uses random, organized (entity-based), or adaptive (paradigm-based) or a combination of dynamics

  • Evaluation design (content and process) can be entity-focused (organized), paradigm-focused (adaptive) or a combination of both

System dynamics of family nutrition l.jpg
System Dynamics of Family Nutrition

Simple organized dynamics of family nutrition l.jpg
Simple Organized Dynamics of Family Nutrition

  • Context: hungry family

  • Intervention: buy ingredients, bake a cake, serve family at dinner

  • Evaluation: quality of cake, family satisfaction

Complicated organized dynamics of family nutrition l.jpg
Complicated Organized Dynamics of Family Nutrition

  • Context: hungry family with different tastes and preferences

  • Intervention: ask for family preferences, create optional dishes, serve family multiple dishes at dinner

  • Evaluation: quality and variety of dinner options, matching of dishes to tastes

Complex adaptive dynamics of family nutrition l.jpg
Complex, Adaptive Dynamics of Family Nutrition

  • Context: hungry family with different tastes, schedules, and cooking ability

  • Intervention: Buy and store meal options, make dishes for non-cooks, agree on dinner schedule, adapt shopping patterns to use of food and supplies

  • Evaluation: trends, patterns of food use, meals, family nutrition, overall health

Simple organized dynamics of h1n1 flu l.jpg
Simple Organized Dynamics of H1N1 Flu

  • Context – everyone should be protected through vaccination

  • Program design – universal flu shot clinics

  • Evaluation design - How many clinics were conducted, how many people were vaccinated, how many people contracted the H1N1 flu virus

Complicated organized dynamics of h1n1 flu l.jpg
Complicated Organized Dynamics of H1N1 Flu

  • Context – people are at different risk levels for contracting the H1N1 flu

  • Program design – allocate, administer flu shots by risk level, triage patients by level of risk

  • Evaluation design - What proportion of people with high/medium/low risk receive the vaccine? What proportion of people at each risk level contract the H1N1 flu? How many deaths and hospitalizations are avoided as result of shots?

Complex adaptive dynamics of h1n1 flu l.jpg
Complex Adaptive Dynamics of H1N1 Flu

  • Context – Timing of two interacting epidemics (H1N1 and seasonal flu) is ahead of current vaccine production

  • Program design – Multi-level intervention: national media messages, provider triage by risk, populations self-organize multiple responses

  • Evaluation design – What are changing patterns of twin epidemics? How are governments, providers, populations reacting and interacting in response to situation? Population health impacts?

System dynamics discussion l.jpg
System Dynamics Discussion

  • What are the situations’ boundaries, focus, interconnections, perspectives, power, timing, and dynamics?

  • What are the risks of not understanding the system attributes and dynamics?

  • What are the benefits of understanding the system attributes and dynamics?

Context program design of child abuse prevention l.jpg
Context, Program Design of Child Abuse Prevention

  • Context: Many programs exist but child abuse and neglect rates are increasing

  • Program design: AFC funding for 17 grants for the adaptation, implementation, spread, and sustainability of evidence-based home visiting programs through infrastructure development and system change

Evaluation design of evidence based home visiting initiative l.jpg
Evaluation Design of Evidence-based Home Visiting Initiative

  • Program evaluation – tracking of cross-site cost, implementation, fidelity, and child and family outcomes of 17 EBHV programs

  • System evaluation – tracking of cross-site and grantee-specific system infrastructure, theories of action, measures of system change, partner collaboration and network analysis; system unit of analysis

Open space technology system dynamics exercise l.jpg
Open Space Technology: System Dynamics Exercise

  • What are the dynamics (i.e., the nature and balance of types of system dynamics) of the situation as a whole?

  • What are the system dynamics of the intervention?

  • What are the implications for the evaluation design and process?

Three dynamics of a social system and its context l.jpg
Three Dynamics of a Social System and its Context

Match of evaluation designs to dynamics of social systems and their context l.jpg
Match of Evaluation Designs to Dynamics of Social Systems and Their Context

Exploratory Design

Initiative Renewal Design

Organic Design

Predictive Design

Complex adaptive systems and adaptive self organizing dynamics l.jpg
Complex Adaptive Systems and Adaptive (Self-organizing) Dynamics


Sensitivity to initial conditions


Macro pattern

Complex adaptive systems and adaptive self organizing dynamics cont l.jpg
Complex Adaptive Systems and Adaptive (Self-organizing) Dynamics (cont.)



Pattern formation and points of influence

Implications for evaluation and action l.jpg
Implications for Evaluation and Action Dynamics (cont.)

Small differences can create large effects.

The past influences but does not predict the future.

Many points of influence exist.

Boundaries, differences, and relationships are levers of influence toward a purpose.

Implications for evaluation and action43 l.jpg
Implications for Evaluation and Action Dynamics (cont.)

Simple rules underlie patterns.

Pattern-based feedback and actions are iterative.

Tensions are not resolved.

Patterns are outcomes.

Four stages of evaluation l.jpg
Four Stages of Evaluation Dynamics (cont.)

Design Evaluation

Shape Practice

Collect Data

Make Meaning from Data

Example leap l.jpg
Example: LEAP Dynamics (cont.)

Learning through Engineering Design and Practice

Example leap research design l.jpg
Example: LEAP Research Design Dynamics (cont.)

  • Quasi-experimental design embedded in curriculum development process

  • Pre-post assessments of

    • Content knowledge

    • Perceptions of engineers at work

    • Tinkering

    • Self-efficacy

  • Engineering notebooks

  • Career behaviors survey

External evaluation design l.jpg
External Evaluation Design Dynamics (cont.)

The external evaluation focused on:

  • Confirmation of effectiveness

  • Scale-up

  • Sustainability

Conceptual shifts l.jpg
Conceptual Shifts Dynamics (cont.)

The fundamental conceptual shift in this project was from:

  • teacher-directed de-contextualized learning to student-engaged project-based learning

  • fixed skills and knowledge as learning outcomes to the desired outcomes being that students are actively engaged; develop the capacity to explore and figure things out; and act like an engineer.

Confirmation of effectiveness l.jpg
Confirmation of Effectiveness Dynamics (cont.)

  • Knowledge and skills related to project topics and STEM concepts

  • Enjoyment and pride in project work

  • Development of teamwork, collaboration and workplace skills

  • Interest in STEM courses and pursuit of STEM career and educational pathways

Scale up l.jpg
Scale-Up Dynamics (cont.)

Scale-up involved two tracks:

  • Greater use of the curriculum within the school system

  • Transfer of the curriculum to Boys and Girls Clubs

Sustainability l.jpg
Sustainability Dynamics (cont.)

  • Program sustainability

  • Sustainability of learning of participating students

  • Sustainability of collaborations

  • Sustainability of teaching capacity

Program sustainability l.jpg
Program Sustainability Dynamics (cont.)

  • Maintain relationships with the district

  • Professional development for teachers

  • Shape research related to the project

  • Explore ways to continue project at 9th grade

  • Track STEM course selection of project students in high school

  • Collaborate on additional community dissemination and funding

Sustainability of learning of students l.jpg
Sustainability of Learning of Students Dynamics (cont.)

  • Classroom

  • Extracurricular activities

  • Career-related activities

  • Focused attention through high school

  • Continued involvement of university faculty and students

  • Continued contact with science center

Sustainability of collaborations l.jpg
Sustainability of Collaborations Dynamics (cont.)

  • Use current collaborations to spur others over time

Sustainability of teaching capacity l.jpg
Sustainability of Teaching Capacity Dynamics (cont.)

  • Build capacity through formal and informal professional development approaches

    • Training for new teachers

    • Coaching by master teachers

    • Summer professional development activities

    • Technology enhanced training

Example 2 l.jpg
Example 2 Dynamics (cont.)

Communities of Learning, Inquiry, and Practice (CLIPs)

(video at www.insites.org/clip)

Example 3 l.jpg
Example 3 Dynamics (cont.)

Strengthening Families

Strengthening families protective factors l.jpg
Strengthening Families Dynamics (cont.)Protective Factors

Parental resilience

Social connections

Concrete support in time of need

Knowledge of parenting and child development

Social and emotional competence of child

World caf exercise l.jpg
World Café Exercise Dynamics (cont.)

  • What data gathering and/or analysis tools have you found helpful in gaining a deeper understanding of complex systems or interventions?

  • What practices help you develop your capacity to recognize patterns?

  • Do certain practices seem more related to finding surface patterns and others more related to finding deep patterns?

Contact information l.jpg
Contact Information Dynamics (cont.)