- 95 Views
- Uploaded on

Download Presentation
## S519: Evaluation of Information Systems

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Last week

- What is synthesis methodology?, why do we need that?
- What is synthesis for grading?
- What is the rubric for dimensions
- What is the rubric for components
- How to merge all the conclusions to get the final grade?

Synthesizing for „ranking“

- What are „ranking“ evaluations?
- Examples?
- Difference comparing with „grading“ evaluation?

Qualitative and quantitative

- Qualitative
- Qualitative weight and sum (QWS)
- Quantitative
- Numerical weight and sum (NWS)

Numerical Weight and Sum (NWS)

- It is a quantitative synthesis method for summing evaluand performance across multiple criteria.
- It includes
- Assign numerical importance weight and a numerical performance score to each criteria (dimention)
- Mutliply weights by performance scores
- Sum these products
- The summing result represents the overall merit of the evaluand

Numerical Weight and Sum (NWS)

- It fits for
- There are only a small number of criteria (why)
- There is some other mechanism for taking bars into account (why)
- There is defensible needs-based strategy for ascribing weights.

Training program evaluation

- A comparative evaluation on three different interventions for training managers
- A mountain retreat featuring interactive sessions with multiple world-class management gurus
- An in-house training and mentoring program run by human resources,
- A set of videos and latest book on management from managment guru Peter Drucker

Training program evaluation

- Needs assessment for this evaluation
- Bear in mind that this is a comparison evaluation
- How do you want to compare these programs, what are the key features of the programs
- Identify the dimension of merit (Process, Outcomes and Cost)
- Decide the importance of the merit (giving weights to merits, based on needs?)
- See Table 9.8

Training program evaluation

- Next steps
- Data collection (what are your experiences for your project data collection?)
- Data analysis
- Rate their performance based on pre-defined ratings: excellent, very good, good, fair, or poor) (see Table 9.9 for this example)
- Convert weights into numbers (see Table 9.10)
- Convert ratings into numbers (see Table 9.10)
- Synthesis step (how? See Table 9.11)
- How to interpret Table 9.11

Exercise

- Do it by your own hand:
- Converting Table9.9 to Table9.10 (defining your own numeric value for importance and grading scales) and try to find out which program is the best comparing with others.
- If suddently, the cost criterias become extremely important, will this change the final result?
- Work on your own
- Form the pair and discussion
- Pros and cons for NWS?

Qualitative Weight and Sum (QWS)

- It is non-numerical synthesis methodology for summing the performances of an evaluand on multiple criteria to determine overall merit.
- It is a ranking method for determining the relative merit of two or more evaluands
- It is not suitable for grading
- It fits for
- Personnel selection, products/service/proposal selection

QWS

- Step1: Determine importance in terms of maximum possible value
- How (see Chapter 7, six strategies)
- Table 9.12 (compare with Table 9.8)
- Step2: Set bars
- Bar is the cut point between acceptable and unacceptable criteria. Such as:
- Too expensive to afford
- Too long away from their work

QWS

- Step3: Create value determination rubrics
- Rubrics are level-based (see Chapter 8)
- Description on each level, how to deal with bar?
- Unacceptableno noticeable valuemarginally valuablevaluableextremely valuable
- Such as what performance would look like at each level
- Each dimension can have its own rubrics or each group of dimensions can have their own rubrics
- Each group of questions can have their own rubrics
- Synthesis step can have its own rubrics
- Example: Rubric for rating finanical cost of training (see table 9.14)
- How to set up similar rubric for Table 9.12 on extremely valuable criteria (see Table 9.15)

QWS

- Step4: Check equivalence of value levels across dimensions
- The validity of the QWS method is highly dependent on ensuring the rough equivalence on the value levels defined for each dimension
- For example, whether table 9.14 and table 9.15 have the roughly equivalent value levels
- How to do that? Put them into a matrix.
- See table 9.16

QWS

- Step5: rate value of actual performance on each dimension
- Rating table 9.9 according to rubric (table9.16)
- See Table 9.17
- Step6: tally the number of ratings at each level and look for a clear winner
- For each program, how many symbols they got?
- Throw out programs with unacceptable ratings, see whether there is a clear winner?

QWS

- Step7: refocus
- Delete the rows with similar score (see table9.18)
- Count how many symbols each of them got?
- Can we find the clear winner?
- Yes or no?
- Why?
- How should we go further?

Exercise

- Form a group to work on this
- Library wants to subscribe to one of the three magazines and ask you to conduct an evaluation and propose the best solution:
- Journal of the American Society for Information Science and Technology
- Journal of Information Science
- Scientometrics
- (or choose some magezines you are familiar with)
- List the criteria you think are important
- Following the steps of NWS and QWS and tell me your findings and show me the justification of your findings.

Download Presentation

Connecting to Server..