Mikko v pohjola nordem oy thl
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Management of assessments and decision making: execution, facilitation, evaluation PowerPoint PPT Presentation


  • 47 Views
  • Uploaded on
  • Presentation posted in: General

Mikko V. Pohjola, Nordem Oy (THL). Management of assessments and decision making: execution, facilitation, evaluation. Basic problem. Lankinen, et al.: State research institutes and research funding: proposal on a comprehensive reform :

Download Presentation

Management of assessments and decision making: execution, facilitation, evaluation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Mikko V. Pohjola, Nordem Oy (THL)

Management of assessments and decision making: execution, facilitation, evaluation


Basic problem

  • Lankinen, et al.: State research institutes and research funding: proposal on a comprehensive reform:

    • “Preparation, decision-making and implementation of policy should be based on researched information and evidence.…the use of research-based information in social policy decision-making, must be strengthened”

  • If we want policy to be ”knowledge-based” and ”effective”…

  • …what constitutes effective knowledge-based policy?

  • and effective knowledge support to policy making?


Policy

Outcomes (vs. objectives)

Behavioral responses + CBA

Monitoring

Ex ante – ex post evaluation

Factors limiting implementation

Context

Modelling/assessment

Enablers

Output

Process

Use

Outcomes

Constraints

Common perspectives to effectiveness

  • Assessment (and modeling)

    • Quality assurance / control

    • Uncertainty analysis

    • Technical assessment of models

    • ”Effectiveness”

    • Other


Concepts

  • Assessment – knowledge support for practical needs

  • Policy – decisions and actions with societal relevance

  • Outcomes – changes in the world resulting from decision and actions

  • Practices – (common) ways of dealing with things

  • Effectiveness – capability (or likelihood) of achieving intended outcomes

  • Evaluation – analysis of the capability (or likelihood) of achieving intended outcomes


Framework for knowledge-based policy

(Knowledge) practices

Policy making

Assessment

Question

Implementation

Outcomes

Answer

Interpretation

  • Process

  • Product

  • Use

  • Interaction

  • Design

  • Execution

  • Follow-up

Evaluation &

management


Framework for knowledge-based policy

  • Focus on knowledge support to decision making

    • Decision makers' needs?

    • Societal needs?

    • Other needs?

    • Expert knowledge

    • Local knowledge

    • Opinions, views, perspectives, values

    • → collaborative search for questions and answers

    • → good decisions, desired changes in the world

  • Evaluation and management of effectiveness -> proactive guidance of decisions and actions both now and in the future


Management of assessments and decision making

Execution

Evaluation

Assessment

Decision

Outcomes

Assessment

Decision

Outcomes

Question

  • Explicit purposes, aims

  • Common practices

  • Properties of good assessment

  • Dimensions of openness

  • Assessment-policy interaction

  • Explicit purposes, aims

  • Monitoring

  • Policy analysis

Answer

Interpretation

Facilitation

  • Open assessment

  • Openness

  • Discussion

  • Collaboration

  • Shared understanding

  • Information structure


Evaluation of decision support

  • Five ”knowledge-based” approaches to effectiveness (cf. the Puijo-case):

    • Dimensions of openness (DO)(Pohjola and Tuomisto 2011)

    • Knowledge-policy interaction (KPI)(Pohjola manuscript)

    • Properties of good assessment (PoGA)(Sandström et al. manuscript, http://en.opasnet.org/w/Properties_of_good_assessment)

    • Assessment effectiveness (Kauppinen et al. 2006)

    • Relational evaluation approach (Koivisto and Pohjola 2011)

  • Evaluation guides (the management of) execution!

    • Not only afterwards, but also during and even before the making of an assessment


Evaluation

  • Evaluation is possible only in relation to purpose!

  • Taking account of (intended) use is essential

  • Instrumental value of assessment

    • Cf. absolute value (a common science view)

    • Cf. Ad hoc solutions (a common practice view)

    • => Assessment is a tool for policy-support (a means to an end)

      • => A model is a tool for making assessments


Evaluation

  • Why evaluation of decisions / decision support?

    • Efficient use of resources?

    • Value of work done?

    • Importance/meaning of information?

    • Implications of information?

    • Actual impacts of information?

    • …because funder, customer, user, boss, peer, stakeholder etc. wants/needs to know!


Evaluation

Assessment is business of creating understanding about reality

Making right questions

Providing good answers

Delivering the answers (and questions) to use

Dual purpose of assessments:

Meeting the needs of use (decision making, policy)

Striving for truth (science)


KPI: how well does assessment address decision needs?

  • Impacts

  • Causes

  • Problem owner

  • Target

  • Interaction


DO: how well can a decision process (or assessment) accept contributions?

  • Scope of participation: who are allowed to participate in the process?

  • Access to information: what information regarding the issue at hand is made available to participants

  • Timing of openness: when are participants invited or allowed to participate?

  • Scope of contribution: which aspects of the issue at hand participants are invited or allowed to contribute to?

  • Impact of contribution: to what extent are participant contributions allowed to have influence on the outcomes, i.e. how much weight is given to participant contributions?


PoGA: how well can an assessment influence decisions and actions?

  • Quality of content

    • Informativeness

    • Calibration

    • Coherence

  • Applicability

    • Relevance

    • Availability

    • Usability

    • Acceptability

  • Efficiency

    • Intra-assessment efficiency

    • Inter-assessment efficiencý


Properties of good assessment


Example: what makes a good hammer?


Example: what makes a good hammer?

  • How is the hammer made? By who?

  • What properties does the hammer have?

  • What do you want to do with the hammer?

  • How does the hammer help you do it?


Summary: evaluating decision support

Purpose:

Identify and explicate decision maker aims & decision criteria

Identify and explicate other relevant needs

Decision process:

DO: Making use of assessments

Decision support (assessment):

PoGA & KPI: Serving decision needs

DO: Making use of available knowledge


Homework 9

Look into two (other than your own) homework 3 draft assessments

Characterize the assessment drafts according to

Knowledge-policy interaction &

Dimensions of openness

Based on the characterization, evaluate the assessment drafts according to

Properties of good assessment (modified)

Write your development suggestions to the corresponding user pages

Homework 3 deadline is Monday 4.2.!

-> do your homework 9 after that!


  • Login