s519 evaluation of information systems n.
Skip this Video
Loading SlideShow in 5 Seconds..
S519: Evaluation of Information Systems PowerPoint Presentation
Download Presentation
S519: Evaluation of Information Systems

Loading in 2 Seconds...

play fullscreen
1 / 17

S519: Evaluation of Information Systems - PowerPoint PPT Presentation

  • Uploaded on

S519: Evaluation of Information Systems. Analyzing data: Rank Ch9-p171. Last week. What is synthesis methodology?, why do we need that? What is synthesis for grading? What is the rubric for dimensions What is the rubric for components

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

S519: Evaluation of Information Systems

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
s519 evaluation of information systems

S519: Evaluation of Information Systems

Analyzing data:



last week
Last week
  • What is synthesis methodology?, why do we need that?
  • What is synthesis for grading?
    • What is the rubric for dimensions
    • What is the rubric for components
    • How to merge all the conclusions to get the final grade?
synthesizing for ranking
Synthesizing for „ranking“
  • What are „ranking“ evaluations?
    • Examples?
    • Difference comparing with „grading“ evaluation?
qualitative and quantitative
Qualitative and quantitative
  • Qualitative
    • Qualitative weight and sum (QWS)
  • Quantitative
    • Numerical weight and sum (NWS)
numerical weight and sum nws
Numerical Weight and Sum (NWS)
  • It is a quantitative synthesis method for summing evaluand performance across multiple criteria.
  • It includes
    • Assign numerical importance weight and a numerical performance score to each criteria (dimention)
    • Mutliply weights by performance scores
    • Sum these products
    • The summing result represents the overall merit of the evaluand
numerical weight and sum nws1
Numerical Weight and Sum (NWS)
  • It fits for
    • There are only a small number of criteria (why)
    • There is some other mechanism for taking bars into account (why)
    • There is defensible needs-based strategy for ascribing weights.
training program evaluation
Training program evaluation
  • A comparative evaluation on three different interventions for training managers
    • A mountain retreat featuring interactive sessions with multiple world-class management gurus
    • An in-house training and mentoring program run by human resources,
    • A set of videos and latest book on management from managment guru Peter Drucker
training program evaluation1
Training program evaluation
  • Needs assessment for this evaluation
    • Bear in mind that this is a comparison evaluation
    • How do you want to compare these programs, what are the key features of the programs
  • Identify the dimension of merit (Process, Outcomes and Cost)
  • Decide the importance of the merit (giving weights to merits, based on needs?)
  • See Table 9.8
training program evaluation2
Training program evaluation
  • Next steps
    • Data collection (what are your experiences for your project data collection?)
    • Data analysis
      • Rate their performance based on pre-defined ratings: excellent, very good, good, fair, or poor) (see Table 9.9 for this example)
      • Convert weights into numbers (see Table 9.10)
      • Convert ratings into numbers (see Table 9.10)
    • Synthesis step (how? See Table 9.11)
    • How to interpret Table 9.11
  • Do it by your own hand:
    • Converting Table9.9 to Table9.10 (defining your own numeric value for importance and grading scales) and try to find out which program is the best comparing with others.
  • If suddently, the cost criterias become extremely important, will this change the final result?
    • Work on your own
    • Form the pair and discussion
  • Pros and cons for NWS?
qualitative weight and sum qws
Qualitative Weight and Sum (QWS)
  • It is non-numerical synthesis methodology for summing the performances of an evaluand on multiple criteria to determine overall merit.
  • It is a ranking method for determining the relative merit of two or more evaluands
  • It is not suitable for grading
  • It fits for
    • Personnel selection, products/service/proposal selection
  • Step1: Determine importance in terms of maximum possible value
    • How (see Chapter 7, six strategies)
    • Table 9.12 (compare with Table 9.8)
  • Step2: Set bars
    • Bar is the cut point between acceptable and unacceptable criteria. Such as:
      • Too expensive to afford
      • Too long away from their work
  • Step3: Create value determination rubrics
    • Rubrics are level-based (see Chapter 8)
      • Description on each level, how to deal with bar?
        • Unacceptableno noticeable valuemarginally valuablevaluableextremely valuable
        • Such as what performance would look like at each level
      • Each dimension can have its own rubrics or each group of dimensions can have their own rubrics
      • Each group of questions can have their own rubrics
      • Synthesis step can have its own rubrics
    • Example: Rubric for rating finanical cost of training (see table 9.14)
    • How to set up similar rubric for Table 9.12 on extremely valuable criteria (see Table 9.15)
  • Step4: Check equivalence of value levels across dimensions
    • The validity of the QWS method is highly dependent on ensuring the rough equivalence on the value levels defined for each dimension
      • For example, whether table 9.14 and table 9.15 have the roughly equivalent value levels
    • How to do that? Put them into a matrix.
      • See table 9.16
  • Step5: rate value of actual performance on each dimension
    • Rating table 9.9 according to rubric (table9.16)
    • See Table 9.17
  • Step6: tally the number of ratings at each level and look for a clear winner
    • For each program, how many symbols they got?
    • Throw out programs with unacceptable ratings, see whether there is a clear winner?
  • Step7: refocus
    • Delete the rows with similar score (see table9.18)
    • Count how many symbols each of them got?
    • Can we find the clear winner?
      • Yes or no?
      • Why?
      • How should we go further?
  • Form a group to work on this
  • Library wants to subscribe to one of the three magazines and ask you to conduct an evaluation and propose the best solution:
    • Journal of the American Society for Information Science and Technology
    • Journal of Information Science
    • Scientometrics
    • (or choose some magezines you are familiar with)
  • List the criteria you think are important
  • Following the steps of NWS and QWS and tell me your findings and show me the justification of your findings.