Mikko v pohjola nordem oy thl
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Management of assessments and decision making: execution, facilitation, evaluation PowerPoint PPT Presentation


  • 44 Views
  • Uploaded on
  • Presentation posted in: General

Mikko V. Pohjola, Nordem Oy (THL). Management of assessments and decision making: execution, facilitation, evaluation. Basic problem. Lankinen, et al.: State research institutes and research funding: proposal on a comprehensive reform :

Download Presentation

Management of assessments and decision making: execution, facilitation, evaluation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Mikko v pohjola nordem oy thl

Mikko V. Pohjola, Nordem Oy (THL)

Management of assessments and decision making: execution, facilitation, evaluation


Management of assessments and decision making execution facilitation evaluation

Basic problem

  • Lankinen, et al.: State research institutes and research funding: proposal on a comprehensive reform:

    • “Preparation, decision-making and implementation of policy should be based on researched information and evidence.…the use of research-based information in social policy decision-making, must be strengthened”

  • If we want policy to be ”knowledge-based” and ”effective”…

  • …what constitutes effective knowledge-based policy?

  • and effective knowledge support to policy making?


Common perspectives to effectiveness

Policy

Outcomes (vs. objectives)

Behavioral responses + CBA

Monitoring

Ex ante – ex post evaluation

Factors limiting implementation

Context

Modelling/assessment

Enablers

Output

Process

Use

Outcomes

Constraints

Common perspectives to effectiveness

  • Assessment (and modeling)

    • Quality assurance / control

    • Uncertainty analysis

    • Technical assessment of models

    • ”Effectiveness”

    • Other


Management of assessments and decision making execution facilitation evaluation

Concepts

  • Assessment – knowledge support for practical needs

  • Policy – decisions and actions with societal relevance

  • Outcomes – changes in the world resulting from decision and actions

  • Practices – (common) ways of dealing with things

  • Effectiveness – capability (or likelihood) of achieving intended outcomes

  • Evaluation – analysis of the capability (or likelihood) of achieving intended outcomes


Framework for knowledge based policy

Framework for knowledge-based policy

(Knowledge) practices

Policy making

Assessment

Question

Implementation

Outcomes

Answer

Interpretation

  • Process

  • Product

  • Use

  • Interaction

  • Design

  • Execution

  • Follow-up

Evaluation &

management


Framework for knowledge based policy1

Framework for knowledge-based policy

  • Focus on knowledge support to decision making

    • Decision makers' needs?

    • Societal needs?

    • Other needs?

    • Expert knowledge

    • Local knowledge

    • Opinions, views, perspectives, values

    • → collaborative search for questions and answers

    • → good decisions, desired changes in the world

  • Evaluation and management of effectiveness -> proactive guidance of decisions and actions both now and in the future


Management of assessments and decision making

Management of assessments and decision making

Execution

Evaluation

Assessment

Decision

Outcomes

Assessment

Decision

Outcomes

Question

  • Explicit purposes, aims

  • Common practices

  • Properties of good assessment

  • Dimensions of openness

  • Assessment-policy interaction

  • Explicit purposes, aims

  • Monitoring

  • Policy analysis

Answer

Interpretation

Facilitation

  • Open assessment

  • Openness

  • Discussion

  • Collaboration

  • Shared understanding

  • Information structure


Evaluation of decision support

Evaluation of decision support

  • Five ”knowledge-based” approaches to effectiveness (cf. the Puijo-case):

    • Dimensions of openness (DO)(Pohjola and Tuomisto 2011)

    • Knowledge-policy interaction (KPI)(Pohjola manuscript)

    • Properties of good assessment (PoGA)(Sandström et al. manuscript, http://en.opasnet.org/w/Properties_of_good_assessment)

    • Assessment effectiveness (Kauppinen et al. 2006)

    • Relational evaluation approach (Koivisto and Pohjola 2011)

  • Evaluation guides (the management of) execution!

    • Not only afterwards, but also during and even before the making of an assessment


Evaluation

Evaluation

  • Evaluation is possible only in relation to purpose!

  • Taking account of (intended) use is essential

  • Instrumental value of assessment

    • Cf. absolute value (a common science view)

    • Cf. Ad hoc solutions (a common practice view)

    • => Assessment is a tool for policy-support (a means to an end)

      • => A model is a tool for making assessments


Evaluation1

Evaluation

  • Why evaluation of decisions / decision support?

    • Efficient use of resources?

    • Value of work done?

    • Importance/meaning of information?

    • Implications of information?

    • Actual impacts of information?

    • …because funder, customer, user, boss, peer, stakeholder etc. wants/needs to know!


Evaluation2

Evaluation

Assessment is business of creating understanding about reality

Making right questions

Providing good answers

Delivering the answers (and questions) to use

Dual purpose of assessments:

Meeting the needs of use (decision making, policy)

Striving for truth (science)


Kpi how well does assessment address decision needs

KPI: how well does assessment address decision needs?

  • Impacts

  • Causes

  • Problem owner

  • Target

  • Interaction


Do how well can a decision process or assessment accept contributions

DO: how well can a decision process (or assessment) accept contributions?

  • Scope of participation: who are allowed to participate in the process?

  • Access to information: what information regarding the issue at hand is made available to participants

  • Timing of openness: when are participants invited or allowed to participate?

  • Scope of contribution: which aspects of the issue at hand participants are invited or allowed to contribute to?

  • Impact of contribution: to what extent are participant contributions allowed to have influence on the outcomes, i.e. how much weight is given to participant contributions?


Poga how well can an assessment influence decisions and actions

PoGA: how well can an assessment influence decisions and actions?

  • Quality of content

    • Informativeness

    • Calibration

    • Coherence

  • Applicability

    • Relevance

    • Availability

    • Usability

    • Acceptability

  • Efficiency

    • Intra-assessment efficiency

    • Inter-assessment efficiencý


Properties of good assessment

Properties of good assessment


Example what makes a good hammer

Example: what makes a good hammer?


Example what makes a good hammer1

Example: what makes a good hammer?

  • How is the hammer made? By who?

  • What properties does the hammer have?

  • What do you want to do with the hammer?

  • How does the hammer help you do it?


Summary evaluating decision support

Summary: evaluating decision support

Purpose:

Identify and explicate decision maker aims & decision criteria

Identify and explicate other relevant needs

Decision process:

DO: Making use of assessments

Decision support (assessment):

PoGA & KPI: Serving decision needs

DO: Making use of available knowledge


Homework 9

Homework 9

Look into two (other than your own) homework 3 draft assessments

Characterize the assessment drafts according to

Knowledge-policy interaction &

Dimensions of openness

Based on the characterization, evaluate the assessment drafts according to

Properties of good assessment (modified)

Write your development suggestions to the corresponding user pages

Homework 3 deadline is Monday 4.2.!

-> do your homework 9 after that!


  • Login