第九讲
This presentation is the property of its rightful owner.
Sponsored Links
1 / 100

第九讲 教育政策评估过程的研究 PowerPoint PPT Presentation


  • 45 Views
  • Uploaded on
  • Presentation posted in: General

第九讲 教育政策评估过程的研究. 北京师范大学 教育研究方法讲座系列 (2): 教育政策研究. Conceptualization of evaluation

Download Presentation

第九讲 教育政策评估过程的研究

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


6911724

第九讲教育政策评估过程的研究

北京师范大学

教育研究方法讲座系列 (2):

教育政策研究


Conceptualization of policy evaluation

Conceptualization of evaluation

Fitzpatrick, Sanders and Worthen define “evaluation as the identification, clarification, and application of defensible criteria to determine an evaluation object’s value (worth or merit) in relation to those criteria.” (2004, p.5)

Conceptualization of Policy Evaluation


Conceptualization of policy evaluation1

Conceptualization of policy evaluation:

Accordingly, the constituents of evaluation consist of

The evaluation object: It is of course public policy or more specifically education policy.

The evaluation criteria: Define the criteria upon which the evaluations are based. They may include

Policy objectives: To assess to what extent the identified policy objectives have been fulfilled.

Policy measures and programs: To investigate to what extent the designed policy measures and programs have been implemented

Policy outcomes: To measure to what extent the targeted outcomes have been realized

Cost and benefit of the policy: To assess the cost of the implementation of policy means against the benefits entailed by the policy outcome

Conceptualization of Policy Evaluation


Conceptualization of policy evaluation2

Conceptualization of policy evaluation:

the constituents of policy evaluation consist of …

The judgment: The evaluation can be studies and judge by means of

Analytical-technical study

Interpretive-political study

Discursive-critical study

Conceptualization of Policy Evaluation


Conceptualization of policy evaluation3

Typology of policy evaluation

In regards to the purpose of policy evaluation, policy evaluation can be categorized into

Formative evaluations aim to provide information for personnel implementing the policy or carrying out the program. Hence, their primary purposes are for the development and improvement of the policy and/or program

Summative evaluations aim to pass judgment on the performance of the policy and/or program. They are primarily carried out as accountability mechanism or auditing measures.

Conceptualization of Policy Evaluation


Conceptualization of policy evaluation4

Typology of policy evaluation

In relation to the policy stages, policy evaluation can be categorized into

Need assessments: They the studies conducted at the formative stages of the policy process. They are conducted either to evaluate the urgencies of the policy problems or the needs that are required if the identified desirable states of affairs are to be realized.

Process-monitoring studies: They are studies carried out to monitor to what extent are the designed policy measures .and programs are implemented in real world situations. They also aim to provide information for the improvement of the effectiveness of the policy instruments in use.

Conceptualization of Policy Evaluation


Conceptualization of policy evaluation5

Typology of policy evaluation

In relation to the policy stages, policy evaluation can be categorized into

Outcome studies: They are carried out to assess to what extent have the preconceived policy outcomes been attained. They can also provide information to the revelations of unanticipated consequences (desirable, undesirable or both) of the policy

Conceptualization of Policy Evaluation


Conceptualization of policy evaluation6

Typology of policy evaluation

In terms of the policy context, policy evaluation can be classified into

Endogenous policy evaluation: They are studies carried out to assess the internal effectiveness and coherence among various parts of the policy constituents. The most common types of the endogenous evaluation studies are those to assess whether the policy outcomes are in congruence with the policy objectives.

Exogenous policy evaluation: They are studies assessing whether the policy substance are in congruence with the policy environment. These type of evaluation studies are especially informative in ever changing policy environment.

Conceptualization of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation1

The framework of practical logical of policy evaluation:

Post-positivist methodological approach to policy evaluation: By post-positivism, it refers to "a contemporary school of social science that attempts to combine the discourse of social and political theory with the rigor of modern science. It calls for a marriage of scientific knowledge with interpretive and philosophical knowledge about norms and values." (Fischer, 1995, p. 243) Accordingly, Fischer advocates that policy evaluation research should extend beyond the dominance of empirical-positivism and incorporate practical discourse about rightness of normative claims into policy evaluation research..

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation2

The framework of practical logical of policy evaluation:

Fischer names such a post-positivist approach in policy evaluation the framework of the "Practical Logic of Policy Evaluation".

"It is design to illuminate the basic discursive components of a full or complete evaluation, one which incorporates the full range of both the empirical and normative concerns that can be brought to bear on an evaluation. Structured around four interrelated discourses, the approaches extends from concrete empirical questions pertinent to a particular situation up to the abstract normative issues concerning a way of life." (Fischer, 1995, p. 18)

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation3

The framework of practical logical of policy evaluation:

Accordingly, the framework of practical logic of policy evaluation is made up of

First-order evaluation level:

Technical-analytical discourse

Contextual discourse

Second-order evaluation level

Systems discourse

Ideological discourse

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation4

Technical-analytical discourse: Program verification

By program verification, it refers to concerns "with measurement of the efficiency of program outcome. …The basic questions of verification are

Does the program empirically fulfill its stated objective(s)?

Does the empirical analysis uncover secondary or unanticipated effects that offset the program objectives?

Does the program fulfill the objectives more efficientlythan alternative means available?" (p. 20)

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation5

Technical-analytical discourse: Program verification

"Program verification employs such methodologies as experimental research and cost-benefit analysis. The goal is to produce a quantitative assessment of the degree to which a program fulfills a particular objective (standard or rule) and a determination (in terms of a comparison of input and output) of how efficiently the objective is fulfill (typically measured as a ratio of costs to benefits) compared to other possible means." (p. 20)

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation6

Contextual discourse: Situational validation

Validation focuses on whether or not the particular program objectives are relevant to the situation. …Instead of measuring program objectives per se, validation examines the conceptualizations and assumptions underlying the problem situation which the program is designed to influence. Validation centers around the following questions:

Is the program objective(s) relevant to the problem situation?

Are there circumstances in the situation that require an exception to be made to the objective(s)?

Are two or more criteria equally relevant to the problem situation?" (Pp. 20-21)

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation7

Contextual discourse: Situational validation

"Validation is an interpretive process of reasoning that takes place within the framework of the normative belief systems brought to bear on the problem situation. Validation draw in particular on qualitative methods, such as those developed for sociological anthropological research." (p.21)

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation8

Systems discourse: Societal vindication

"In vindication… the basic task is to show that a policy goal (from which specific program objectives were drawn) addresses a valuable function for the existing societal arrangement. Vindication is organized around the following questions:

Does the policy goal have instrumental or contributive value for the society as a whole?

Does the policy goal result in unanticipated problems with important societal consequences?

Does a commitment to the policy goal lead to consequence (e.g. benefits and costs) that are judged to be equitably distributed?" (p. 21)

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation9

Systems discourse: Societal vindication

As second-order vindication, this type of policy evaluation "steps outside of the situational action context in which program criteria are applied and implemented in order to assess empirically the instrumental consequences of a policy goal in terms of the system as a whole." (p. 21) Hence, the methodological approach adopted by societal vindication is functional and macroscopic-institutional analysis which usually takes the forms of comparative and/or historical-sociological methods.

Frank Fischer’s Conception of Four Levels of Policy Evaluation


Frank fischer s conception of four levels of policy evaluation10

Frank Fischer’s Conception of Four Levels of Policy Evaluation

Ideological discourse: Social choice

"Social choice seeks to establish and examine the selection of a critical basis for marking rationally informed choices about societal systems and their respective ways of life. …Social choice raises the following types of questions:

Does the fundamental ideal (or ideology) that organize the accepted social order provide a basis for a legitimate resolution of conflicting judgments?

If the social order is unable to resolve basic values conflicts, do other social orders equitably accommodate the relevant interests and needs that the conflicts reflect?”

Do normative reflection and empirical support the justification and adoption of an alternative ideology and the social order it prescribes?” (p. 22)


Frank fischer s conception of four levels of policy evaluation11

Ideological discourse: Social choice

"Social choice involves the interpretive tasks of social and political critique, particularly as practiced in political theory and philosophy. Most fundamental are the concepts of a 'rational way of life' and 'good society'. Based on the identification and organization of a specific configuration or values ― such as equality, freedom, or community ― models of the good society serve as a basis for the adoption of higher level evaluative criteria." (p. 22) The typical methodological approach adopted by this type of evaluative studies is critical approach to political, social and philosophical inquiry.

Frank Fischer’s Conception of Four Levels of Policy Evaluation


6911724

Ideological Choice

Because of

Critical Discursive Perspective

Systems

Vindication

Because of

Interpretive Political

Perspective

Situational

Validation

Because of

Warrant

Since

Analytic

Technical

Perspective

Data

Technical Verification

So, (Qualifier), Conclusion


Evaluation study of moi policy for secondary schools in hksar

Evaluation Study of MOI Policy for Secondary Schools in HKSAR


Technical analytical discourse program verification

A policy measure in search of an objective:

Summative or even judgmental evaluation of the efficiency of CMI in comparison with EMI

Informative evaluation of the efficiency of preparing secondary-school leavers to become trilingual and bilateral

Upholding mother-tongue instruction

Summative evaluation of the effectiveness of eliminating mixed-coded teaching in secondary schools in HKSAR

Technical-analytical Discourse: Program verification


The objectives of the longitudinal study 1999 2002 are

“The Objectives of the Longitudinal Study (1999-2002) are

  • to trace the academic & personal development of students in schools adopting either Chinese or English as the MOI;

  • to compare the degree of improvement of students’ academic and personal development in schools adopting either Chinese or English as the MOI;

  • to compare the language ability (in both Chinese & English) of students in schools adopting either Chinese or English as the MOI; and

  • to identify facilitating & hindering factors affecting students learning in school adopting Chinese as the MOI.”

    (p.2 of Tender ref. ED/PR/EMICM/99)


The objectives of the longitudinal study 2002 2004

The Objectives of the Longitudinal Study (2002-2004)

  • Study are to find out

    • The effects of different MOI arrangements at senior levels on students’ learning, e.g. their academic performance, personal development, language ability and high-order thinking skills;

    • the major factors that enhance/hinder students’ learning in Chinese or English MOI at senior secondary levels; and

    • the major factors that enhance/hinder the introduction of English as MOI or in part only at senior secondary levels.

  • based on the findings, to design/recommend measures that support students’ learning under different MOI arrangement for their whole secondary schooling.

    (Tender ref. ED/P&R/EMICMI/02)


Technical analytical discourse program verification1

To compare the academic and personal development outcomes of students in EMI and CMI schools: Summative, outcome and endogenous evaluation research

Instructional paradox in MOI policy outcomes

Socio-psychological paradox in MOI policy outcomes

School-organizational paradox in MOI policy outcomes

Technical-analytical Discourse: Program verification


Subjects

Subjects

  • Cohorts of Students under Study


6911724

  • Subjects: Cohorts of Students under Study

00-01

99-00

01-02

02-03

03-04

04-05

05-06

98 Cohort

HK

CEE

98

AAI

F. 2

Tests

F. 3

Tests

F. 4

Tests

HK

ALE

99 Cohort

F. 1

Tests

F. 2

Tests

F. 4

Tests

HK

CEE

99

AAI

HK

ALE

F. 3

Tests


6911724

Teachers’ Perception of Learning Environment

A. Sense of efficacy

B. Sense of administrative and collegial support

C. Sense of reward & recognition from work

D. Sense of autonomy & empowerment

E. Feeling of stress in work

Students’ Performance &

Perception of Learning

Environment

A. Academic Development

a. Language Subject

- Chinese Language

- English Language

b. Content Subject

- Mathematics

- Science

- Social Subjects

B. Personal Development

a. Self-esteem

b. Citizenship Development

c. Social Efficacy

d. Language & learning attitudes

C. Perception of Learning Environment

a. Learning process & motivation

b. Teacher’s efficacy

c. Classroom climate

d. Quality of school life

EMI

HKCEE

Results

EMI

CHIG

Students’ Academic & Socio-economic Background

Implementation

of the

MOI Guidance

CMI

EMI

CMID

CMI

EMI

CLOW

CMI

School Administration & Learning Environment

A. Principal’s Leadership

B. Organizations of Academic & Disciplinary Orders

C. Organizational Climate & Collaborative Culture.

Conceptual Framework of the Project


Technical analytical discourse program verification2

Instructional paradox in MOI policy outcomes

(Junior Forms)

Technical-analytical Discourse: Program verification


Technical analytical discourse program verification3

Instructional paradox in MOI policy outcomes

(Senior Forms; HKCEE)

Technical-analytical Discourse: Program verification


Technical analytical discourse program verification4

Instructional paradox in MOI policy outcomes

(Access to University Education & HKALE)

Technical-analytical Discourse: Program verification


6911724

Educational Attainment Measure: JUPAS Minimum entry requirement:1. Grade E or above in either 2 AL subjects  or 1 AL subject + 2 AS subjects (other than UE and Chi Lang & Culture)2. Grade E or above in AS UE3. Grade E or above in AS Chi Lang & Culture


6911724

Educational Attainment Measure: JUPAS Maximum entry requirement:1. Grade E or above in either 2 AL subjects  or 1 AL subject + 2 AS subjects (other than UE and Chi Lang & Culture)2. Grade D or above in AS UE3. Grade E or above in AS Chi Lang & Culture


Technical analytical discourse program verification5

To compare the academic and personal development outcomes of students in EMI and CMI schools: Summative, outcome and endogenous evaluation research

Instructional paradox in MOI policy outcomes

Paradox in socio-psychological outcomes

School-organizational paradox in MOI policy outcomes

Technical-analytical Discourse: Program verification


Paradox in socio psychological outcomes

Paradox in Socio-psychological Outcomes

  • Academic Self-concept

  • Attitudes towards English & Motives & Strategies in English Learning

  • Quality of School Life

  • School Identity and Education Aspiration


Paradox in socio psychological outcomes1

Paradox in Socio-psychological Outcomes

  • Academic Self-concept

  • Attitudes towards English & Motives & Strategies in English Learning

  • Quality of School Life

  • School Identity and Education Aspiration


Paradox in socio psychological outcomes2

Paradox in Socio-psychological Outcomes

  • Academic Self-concept

  • Attitudes towards English & Motives & Strategies in English Learning

  • Quality of School Life

    • Linguistic efficacy

    • Learning Opportunities

  • School Identity and Education Aspiration


Paradox in socio psychological outcomes3

Paradox in Socio-psychological Outcomes

  • Academic Self-concept

  • Attitudes towards English & Motives & Strategies in English Learning

  • Quality of School Life

    • Linguistic efficacy

    • Learning Opportunities

  • School Identity and Education Aspiration


6911724

99-cohort 2001/2002 (F.3)


6911724

98-cohort 2001/2002 (F.4)


6911724

Educational Aspiration


6911724

Educational Aspiration


6911724

Educational Aspiration


6911724

Educational Aspiration


Technical analytical discourse program verification6

To compare the academic and personal development outcomes of students in EMI and CMI schools: Summative, outcome and endogenous evaluation research

Instructional paradox in MOI policy outcomes

Paradox in socio-psychological outcomes

School-organizational paradox in MOI policy outcomes

Paradox between instructional effectiveness and educational advancement

Paradox between intake-process-output in different phases of secondary-school education

Technical-analytical Discourse: Program verification


6911724

Form-1 Intakes

Junior-Secondary

Instructional Process

Senior-Secondary

Instructional Process

HKCEE Results

Six-Form

Instructional Process

HKALE Results


Technical analytical discourse program verification7

To prepare of secondary-school leavers to be trilingual & bilateral

Typology of bilingual

Balance bilingual

Chinese dominant bilingual

English dominant bilingual

Dimensions of bilingual proficiency

Cognitive demanding or undemanding

Context reduced or embedded

Levels of bilingual proficiency

Conversational proficiency

Context-reduced and cognitive undemanding proficiency

Context-embedded and cognitive demanding proficiency

Academic language proficiency

Technical-analytical Discourse: Program Verification


6911724

Jim Cummins’ Conceptualization of language proficiency

Cognitively Undemanding

A

C

Context

Embedded

Context

Reduced

B

D

Cognitively Demanding


Technical analytical discourse program verification8

Upholding mother-tongue instruction

Elimination of mixed-coded teaching in secondary schools in HKSAR

at the expenses of

Opportunity to learn

Opportunity for educational advancement

Technical-analytical Discourse: Program Verification


Contextual discourse situational validation of the relevance of the moi policy

To identify the stakeholders (e.g. students, parents, teachers and/or administrators in secondary schools, employers, higher-education institutions, etc.) and their “definition of the situations” and “systems of relevance” in regards to the MOI policy

Contextual Discourse: Situational Validation of the Relevance of the MOI Policy


Contextual discourse situational validation of the relevance of the moi policy1

Situational paradoxes facing stakeholders

Students and Parents: Situational paradox between immediate learning efficacy and long-term prospects in educational or even socioeconomic advancement

School teachers and administrators:

Situational paradox between instructional efficacy and graduates' prospects in educational advancements

Situation paradox between current instructional efficacy and future efficacy of attracting "high-quality" primary-school leavers or even future prospect of survival of the schools.

Employers: The situational paradox is between the prospects of local and global markets.

Higher-education institution: The situation paradox is between short-term instructional efficacy and long-term development of universally applicable scholarship and/or professional competence

Contextual Discourse: Situational Validation of the Relevance of the MOI Policy


Contextual discourse situational validation of the relevance of the moi policy2

To validate whether or to what extent each of the stakeholders’ “systems of relevance” have been fulfilled by the MOI policy measure.

Contextual Discourse: Situational Validation of the Relevance of the MOI Policy


System discourse societal vindication of the moi policy

Systemic vindication of value orientation of the MOI policy

To define the institutional features of the social, political and economic systems of HKSAR: Biglossic or even triglossic structures of a post-colonial society, one-country-two-system polity, and open and small economy in global-informational capitalism.

To evaluate whether or to what extent that MOI policy have instrumental or contributive value for these systemic features of HKSAR

System Discourse: Societal Vindication of the MOI Policy


6911724

Institutional features of the MOI policy environment (socio-cultural, political and economic systems) of HKSAR

Local

political &

economic

systems

Intra-

national

tertiary

education

institution

Global

professional-

intellectual

community

Local

community

of Mother

Tongue

Local

tertiary

education

institution

Global-

informational

economy

Primary

School

System

Secondary

School

System

Global-

political

arena

International

tertiary

education

institution

Intra-

national

political &

economic

systems


6911724

Bilingual

System

with

English

Dominance

Trilingual system

with

English/Putonghua

Dominance

Monolingual

Local

community

of

Cantonese

Trilingual system

with

Cantonese

Dominance

Local

political &

economic

systems

Intra-

national

tertiary

education

institution

Global

professional-

intellectual

community

Local

community

of Mother

Tongue

Local

tertiary

education

institution

Global-

informational

economy

Primary

School

System

Secondary

School

System

Global-

political

arena

International

tertiary

education

institution

Intra-

national

political &

economic

systems


System discourse societal vindication of the moi policy1

To vindicate the intensification effect of MOI policy on the contradictory structure of HKSAR

Biglossic or even triglossic structures of the open and small economy of HK within the global-informational capitalism.

Biglossic structure of the political system of HKSAR within the context of One Country Two System

Biglossic or even troglossic structure of the cultural system of a post-colonial society

To vindicate the unequalizing effect of MOI policy on the structural disparity among the 18 school districts in the SSPA system in terms of distribution of EMI school places

System Discourse: Societal Vindication of the MOI Policy


System discourse societal vindication of the moi policy2

To vindicate the institutionalized effect of MOI policy on the inequality of mobility chances between CMI and EMI students in competing access to higher education.

System Discourse: Societal Vindication of the MOI Policy


Ideological discourse social choice underlying the moi policy

To identify the ideological stance underlying the MOI policy discourse

Orientations to bilingualism

Bilingualism as problem

Linguistic imperialism

Linguistic nationalism

Bilingualism as rights

Bilingualism as resource

Power implications of bilingualism

Bilingualism as coercive power

Bilingualism as collaborative power

Structure-agent stance in bilingualism

Structural imposition stance

Agent resistance stance

Agent appropriation stance

Ideological Discourse: Social Choice underlying the MOI policy


Ideological discourse social choice underlying the moi policy1

To identify the ideological stance underlying the MOI policy discourse

The myth of mother tongue

Ideological Discourse: Social Choice underlying the MOI policy


Ideological discourse social choice underlying the moi policy2

To reflect and choose the ideological stances toward Cantonese, Poutonghau and English, which have been embedded in the MOI policy discourse in post-1997 HK.

Ideological Discourse: Social Choice underlying the MOI policy


Evaluation study of moi policy for secondary schools in hksar1

Evaluation Study of MOI Policy for Secondary Schools in HKSAR

Mother Tongue

or

Other Tongue


6911724

Lecture 9

Policy-Evaluation Process

END


  • Login