Measuring enterprise development impact in complex, resource constrained environments in South Afric...
1 / 27

What SAIE does - PowerPoint PPT Presentation

  • Uploaded on

Measuring enterprise development impact in complex, resource constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development. What SAIE does. Entrepreneurship & Enterprise Development

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' What SAIE does' - teneil

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Measuring enterprise development impact in complex, resource constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development

What saie does
What SAIE does constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development

  • Entrepreneurship & Enterprise Development

  • BEST Game business simulation used in over 60 countries (ILO/GTZ) & a range of adult training programmes

  • Agriplanner and related Ag programmes

  • Wings of Welcome tourism programme

  • Business VENTURES in schools gr 2-12

  • Training for Bankers in assessing loan applications

  • Career guidance programmes

Some of saie s products
Some of SAIE’s products constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development

Land restitution farmers
Land Restitution Farmers constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development

Mixed results with quasi experimental methods 1
Mixed results with quasi-experimental methods, 1 constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development

  • Worked with University of Cape Town Graduate School of Business staff linked to the Global Entrepreneurship Monitor (GEM)

  • Pilot tested methodology in 5 schools with control groups, pre-post testing, random selection & external Q administration

  • Business VENTURES had a significant positive effect on learners in a range of entrepreneurial knowledge, skills and attitude areas

    • Level of understanding of basic business concepts (business concepts)

    • Capability in financial arithmetic (financial arithmetic)

    • Confidence in ability to start a business (entrepreneurial confidence)

Mixed results with quasi experimental methods 2
Mixed results with quasi-experimental methods, 2 constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development

  • Rolled out approach to 41 schools

  • Positive impact results, but also some confusing results (like regressions in financial arithmetic scores)

  • Led us to a critical review of the method

  • Led to a review of entrepreneurship teaching practices in SA schools (new to curriculum)

  • Led us to visit many of the schools to interview the teachers (training dosage, other issues)

  • Looked at disciplines like Math Education & Health Education

Study on instructional improvement sii at university of michigan
Study on Instructional Improvement (SII) at University of Michigan

  • A large scale, multi-method, longitudinal study that examines the effects of instruction on student achievement at 120 schools, including intensive qualitative interviews at 12 of these schools.

  • Future entrepreneurship research studies will need better controls for extraneous factors including (but not limited to):

    • teacher qualifications

    • teacher experience

    • teacher knowledge of entrepreneurship

    • teacher motivation and attitudes

    • classroom implementation of entrepreneurship curriculum

    • how the various materials were used

    • interaction of teachers, students, content and environments.

Tailoires hiv aids criteria for assessing strength of evidence
Tailoires HIV/AIDS criteria for assessing strength of evidence

  • The level of evidence: informed judgment, associations, plausibility evidence, or probability evidence from randomized controlled trials.

  • The quality of the intervention: what are constant and variable components of the intervention, is it feasible, are there identified mechanisms of action, is there an experiential base, and has there been careful pilot testing?

  • The quality of the outcome measures: the robustness and validity of the outcome measures.

  • The process evaluation of the intervention: the reception by target group and practitioners,factors facilitating and inhibiting implementation, and the quality of implementation, in terms of intensity, duration and completeness.

  • The context in which the intervention is delivered: similarities between target groups, the sociocultural setting, the environment and infrastructure, and historical factors such as the phase of the epidemic.

Searching led us to evalulead
Searching led us to EvaluLEAD evidence

  • Team concluded: need to develop richer, more sophisticated ways of measuring impact which also takes into account the rich anecdotal information not captured in quantitative evaluation methodologies

  • Stories from SAIE participants were often encouraging, compellingly powerful & showed developmental impact

  • Complexities of leadership development in EvaluLEAD were appealing because of complexity in entrepreneurship development

How evalulead evolved
How EvaluLEAD evolved evidence

  • In 2001 the Population Leadership Program (PLP) of the Public Health Institute (PHI), a project funded by the United States Agency for International Development (USAID), was wrestling with the question “What form of evaluation would best help us determine if and how leadership development activities make a difference?”

  • At the same time the W.K. Kellogg Foundation was reviewing 55 varied leadership development programmes for good evaluation practices. The two groups collaborated in developing the PLP’s promising emerging framework into what became known as EvaluLEAD.

  • In early 2004, after some further concept development work, 17 organisations running a wide range of leadership development programmes were trained to use the EvaluLEAD methodology and field test it in their organisations throughout 2004, with external expert assistance throughout the year. The EvaluLEAD Guide is the result of the learnings and developments from these groups.

Domains of impact
Domains of Impact evidence

  • Individual Domain

  • Organisational Domain

    • Agencies, departments, programmes, teams, alliances, other

  • Societal/Community Domain

  • SAIE added Facilitators in Organisations as key individuals, so we use two levels of Individuals – Participants and Facilitators

Forms of inquiry
Forms of inquiry evidence

  • Evidential

    • Attempts to capture and represent facts regarding what is happening to people

    • Descriptive, numeric and physical evidence of programme impact – “hard evidence”

  • Evocative

    • Seeks viewpoints and sentiments of those influenced by the programme

    • Stories, views, opinions, discourse

    • Richness and human dimension of what is happening

Result types
Result Types evidence

  • Episodic (Immediate)

    • Cause and effect; time bound; somewhat predictable

  • Developmental

    • Can occur across time; forward progress, stalls, setbacks

    • Open-ended results that are less controllable and predictable

  • Transformative

    • Fundamental shifts; regenerative moments; ‘prize’ to which programmes aspire; crossroads or unanticipated new roads

Implementation process
Implementation process evidence

  • Clarify your vision of change – desired programme impacts

  • Define the context

  • Describe your evaluation context

  • Define desired result types

  • Define domains of impact

  • Create an initial programme results map

  • Prioritize results

  • Develop data collection strategies

  • Refine evaluation plans

Unique challenges for saie
Unique challenges for SAIE evidence

  • Different programmes & desirable to have one evaluation framework

  • Range of target groups - economically poor communities, schools, universities, large NGO youth leadership

  • Dependence on facilitators from other orgs

  • Great materials but mixed facilitator knowledge and skill levels

  • ‘Training dosage’ varies

Saie s process
SAIE’s process evidence

  • Involve a Reference Team

  • Clarify domains of influence – added Facilitators

  • Developed key indicators as main discussion and focus point

  • Discussions led to need to clarify theories of change

  • Shift from measuring attitudes to measuring behaviour – application of what is learned

  • Led to a rich ‘practice to theory to practice’ discussion and helped clarify the organisation’s core business as enterprise development

  • Key theorists and the conceptual areas they represent

  • Key indicators

  • Clarifying most helpful data collection methods

  • One result of this whole process was putting organisational dialogue and learning front and centre as a key strategic issue – (and this needs to be extended now to other constituents)

Theory and theorist summary 1
Theory and Theorist Summary, 1 evidence

  • How to influence behaviour

    • Icek Ajzen, 1987 – Theory of Planned Behavior

    • Bandura, 1977, 1986 – Social Cognitive Theory

    • Resiliency, self-efficacy, self-awareness – Bandura, & others: Linquanti, Grunstein, Garmezy, etc.

  • Influence of culture on risk behaviours

    • Wildavsky and Douglas, 1982 – Cultural theory of risk

Theory and theorist summary 2
Theory and Theorist Summary, 2 evidence

  • Enterprising competencies

    • Kolshorn and Tomecko of CEFE, 1995 – Strengthen enterprising competencies

    • Wood, 2008 - Contribution-based mindset (or value added thinking);

    • Kawasaki – ‘making meaning,’ adding value, helping lots of people, doing what’s right, paying back society;

    • Malloch – strong motivating principles about benefitting society as the reason for existing - that go beyond profit

    • Australian government – Enterprise Education, 2004

Theory and theorist summary 3
Theory and Theorist Summary, 3 evidence

  • Entrepreneurship development

    • Sull, 2004 – Disciplined Entrepreneurship - Cognitive, iterative process of small experiments & improving

    • Lévi-Strauss, 1967 – Entrepreneurial bricolage – making do with what is at hand

    • Networking, connecting & gaining access to critical resources – Mike Morris on critical resources and access to them

    • Necessity; dissonance; pain; survival as key drivers

  • Teaching and learning

  • Career development

Data collection methods
Data collection methods evidence

  • Pre-post questionnaires for adult programmes

  • Key informant interviews and sampling of focus groups for adult programmes

  • Sampling of focus groups for schools programmes

  • Case studies with external assistance

  • Exploring tape recordings with transcripts of interviews and videotaping interviews

Questions and discussion
Questions and discussion evidence

  • What would you like to ask about, discuss, comment on or share?

  • Thanks for your attention and interest!

Collingwood Place, 11 Drake Street, Observatory 7925 evidence

South Africa

• PO Box 13805, Mowbray 7705 •

Phone 27 21 447 2023 • Fax 27 21 447 9911 •

Email: [email protected]

Don Shay

Don Shay & Associates

Small business research and development consultants

[email protected]

+27 (021) 532-2433 tel/fax