IFLA’s Satellite Meeting in Montréal
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Leonor Gaspar Pinto Paula Ochôa Maria Helena Vinagre PORTUGAL PowerPoint PPT Presentation


  • 95 Views
  • Uploaded on
  • Presentation posted in: General

IFLA’s Satellite Meeting in Montréal Library Statistics for the 21 st Century World August 18 and 19, 2008 – Concordia University. INTEGRATED APPROACH TO THE EVALUATION OF DIGITAL LIBRARIES AN EMERGING STRATEGY FOR MANAGING RESOURCES, CAPABILITIES AND RESULTS. Leonor Gaspar Pinto

Download Presentation

Leonor Gaspar Pinto Paula Ochôa Maria Helena Vinagre PORTUGAL

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Leonor gaspar pinto paula och a maria helena vinagre portugal

IFLA’s Satellite Meeting in Montréal

Library Statistics for the 21st Century World

August 18 and 19, 2008 – Concordia University

INTEGRATED APPROACH TO THE EVALUATION OF DIGITAL LIBRARIESAN EMERGING STRATEGY FOR MANAGING RESOURCES, CAPABILITIES AND RESULTS

Leonor Gaspar Pinto

Paula Ochôa

Maria Helena Vinagre

PORTUGAL


1 introduction

1. Introduction

  • The growing importance of networked environment and the consequent change in users’ information behaviour challenged the library academia and professionals’ skills to find new and more adequate strategies and methods to assess the performance of traditional, hybrid or digital information services

  • This presentation aims to detail the strategy and methodology used by a research team to evaluate the performance of the Portuguese Digital Library consortium.

Pinto, Ochôa & Vinagre


2 evaluating the network environment

2. Evaluatingthenetworkenvironment

  • For more than four decades, performance evaluation of library and information services has been captivating the attention of professionals and researchers of the Information Science area.

  • Models used to evaluate Information Services (Hernon and Altman, 1996)

    • Extension

    • Efficiency

    • Cost

    • Quality

    • Effectiveness

Pinto, Ochôa & Vinagre


2 evaluating the network environment1

2. Evaluatingthenetworkenvironment

“The Digital Library is:

The collection of services

and the collection of digital objects

that support users in dealing with information objects

and the organization and presentation of those objects

available directly or indirectly

via electronic/digital means”.

(Leiner, 1998)

Pinto, Ochôa & Vinagre

4


3 the digital libraries evaluation context

3. THE DIGITAL LIBRARIES EVALUATION CONTEXT

“As yet, digital libraries are not evaluated on more than one level. This isolation of levels of evaluation could be considered a further and greater challenge for all digital libraries evaluations. In addition, as a rule, many systems are used in ways that their designers never intended”(Saracevic, 2000).

“Digital libraries are complex social, institutional and technical systems. No evaluation can possible address all these aspects together” (Saracevic, 2004).

”In the context of measurement and evaluation means that a more thorough knowledge and understanding of a system can be gained from combining different measures than can be derived than taking those measures separately” (Nicholson, 2004).

Pinto, Ochôa & Vinagre

5


4 digital library integrated evaluation model case study

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

Pinto, Ochôa & Vinagre

6


4 digital library integrated evaluation model case study1

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

This model is formed by five key-components:

Diagnosis;

Strategical groups and performance information needs

Perspectives on performance evaluation

Evaluation criteria and methods

Evaluation points of view.

A holistic user/stakeholder-centred approach

Pinto, Ochôa & Vinagre

7


4 digital library integrated evaluation model case study2

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

Two types of stakeholders were identified :

Primary stakeholders, directly interested or affected by the evaluation: DL consortium top managers, DL operational teams, libraries teams and end-users.

Secondary stakeholders, evolving all persons/institutions interested on the evaluation and its results: suppliers, associations, visitors (national and international), scientific research community, digital libraries professionals, Information and Documentation Sector, Information Management market and citizens in general.

Based on the performance information needs of these two groups of stakeholders and also on an extensive literature review, a diagnosis of the DL environment was carried out.

Diagnosis

Pinto, Ochôa & Vinagre

8


4 digital library integrated evaluation model case study3

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

Strategical groups and performance information needs

After concluding the diagnosis, the next step was the identification of the DL’s strategical groups and segments. The research team concluded that they equal the primary stakeholders that previously had been pinpointed.

The analysis of the DL strategical groups’ performance information needs led to the identification of five Strategical Perspectives.

Information needs on performance

Pinto, Ochôa & Vinagre

9


4 digital library integrated evaluation model case study4

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

The DL BSC is composed of five strategical Perspectives

Resources and

partnerships

Internal

Processes

Clients

VISION

&

STRATEGY

Learning

and

development

Results

Balanced Scorecard

Measurements according to Perspectives and Strategical Areas

Pinto, Ochôa & Vinagre

10

(adap. from Kaplan e Norton, 1996)


4 digital library integrated evaluation model case study5

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

Balanced Scorecard

Measurements according to Perspectives and Strategical Areas

DL’s Matrix of Perspectives and Strategical Measurement Areas

Pinto, Ochôa & Vinagre

11


4 digital library integrated evaluation model case study6

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

Some of performance indicators used in the DL collection evaluation

Balanced Scorecard

Measurements according to Perspectives and Strategical Areas

Pinto, Ochôa & Vinagre

12


4 digital library integrated evaluation model case study7

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

The information needs on performance determined the identification of six criteria in the Integrated Model:

Effectiveness

Extension

Efficiency

Cost

Quality

Impact

Evaluation criteria and methods

Pinto, Ochôa & Vinagre

13


4 digital library integrated evaluation model case study8

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

Despite the great profusion of studies on service quality only one limited academic articles dealt with e-service quality or web services quality evaluation.

Because of its specificity, these services evaluation required the adoption of convenient instruments and scales adequate to its characteristics and users’ relevant evaluating aspects (Parasuraman, Zeithaml and Malhotra, 2005)

Scales – exs.:

e-SERVQUAL - Zeithaml, Parasuraman e Malhota (2000, 2002): 7 dimensions

WEBQUAL - Loiacono, Watson and Goodhue (2002): 12 dimensions

Digital Library Service Quality Model - based on the conceptual model suggested by Zeithaml, Parasuraman and Malhota (2002), developed a scale to evaluate the quality of service delivered by the Portuguese DL: 3 dimensions

Evaluation criteria and methods

The Digital Library Service Quality Model

Pinto, Ochôa & Vinagre

14


4 digital library integrated evaluation model case study9

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

Definition of strategies

Consortium top managers

Implementation of strategies / services provided

Operational Teams

Interaction

Operational institutions

Communication Gap

Knowledge Gap

PERCIEVED EXPECTATIONS

Libraries Teams

Libraries managers

EXPECTATIONS

Users

SQ Gap

PERCEPTIONS

Users

PERCEPTIONS

Libraries Teams

Libraries managers

Perception Gap

Evaluation criteria and methods

The Digital Library Service Quality Model

Pinto, Ochôa & Vinagre

15

(adapted from Zeithaml, Parasuraman & Malhotra, 2002)


4 digital library integrated evaluation model case study10

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

The model identifies some linkages between the identified strategical groups in this digital library evaluation process. These linkages have some critical points that can be identified as gaps:

Fulfilment gap (service quality)

Knowledge gap

Perception gap

Communication gap

This model also assumes that a digital library is a multidimensional construct that includes three dimensions: Efficiency, Competitive advantage , Adequacy of information.

Evaluation criteria and methods

The Digital Library Service Quality Model

Pinto, Ochôa & Vinagre

16


4 digital library integrated evaluation model case study11

4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study

The (re)introduction of the stakeholders points of view on the DL evaluation corresponds to the final step in the construction of the Integrated Evaluation Model and guarantees the multidimensional, integrated and holistic structure of this model.

“The same evaluation criteria will be judged in different ways by different participants in the process. In order to gain a holistic understanding of the evaluation, the viewpoints from different groups must be taken into perspective. (…) Therefore, it is important to be aware of the viewpoint of the group doing the evaluation and ensure different groups who might be affected through decisions made from the evaluation can participate in the process”

(Nicholson, 2004, quoting Brophy and Couling, 1996)

Evaluation points of view

Pinto, Ochôa & Vinagre

17


5 potential interactions between evaluation cultures environments and learning professional skills

5. Potential interactions between evaluation cultures/environments and learning professional skills

A number of Portuguese researchers and practitioners have argued that LIS competences management, more than ever before, experience new notions of professional culture in changing evaluation environments and professional work . Professional communities are supported by change agendas and opportunities, being one of them the potential interactions between evaluation culture and professional skills.

The components of professional competence in an evaluation environment link professional learning skills to librarian’s clients highlighting the importance of practice based on knowledge management and quality dimensions of service. Despite disciplinary variations, it seems evident that evaluation cultures emphasize the importance of expert knowledge in order to become a professional actor in information society dilemmas.

Pinto, Ochôa & Vinagre

18


5 potential interactions between evaluation cultures environments and learning professional skills1

5. Potential interactions between evaluation cultures/environments and learning professional skills

This case provides a basis for discussions about the competence dimension in evaluation cultures, articulating the aim of skills management as part of a professional responsibility.

Two major concepts influence this case: Management Skills Charter and Skills Map, a combination of theories of knowledge management concepts and Quality Management, assumed by all the staff and publicised by various media. It implies that digital libraries are perceived as objective representatives for both professional skills evidences and quality services levels. In this marketing process what is associated with collective competences activities is often perceived as the highest valued.

Pinto, Ochôa & Vinagre

19


5 potential interactions between evaluation cultures environments and learning professional skills2

5. Potential interactions between evaluation cultures/environments and learning professional skills

This approach allows a better evaluation of the library strategical process, identifying the principal attributes of a digital library, namely:

The use of common procedures to evaluate;

The engagement of all participants in results, discussing strong and weak points and designing improvement actions;

The use of an integrated system of evaluation where levels of key-skills are particular important for the effectiveness.;

Skills models must be aligned with strategic maps accordingly with changes within information sector.

The key-factor to success in this experience is the individual professional role identification with a new professionalism model.

Pinto, Ochôa & Vinagre

20


Leonor gaspar pinto paula och a maria helena vinagre portugal

Concern hasbeen expressed over the years about the tendency of library performance measurement to become over-elaborate, to concentrate on ‘measuring the measurable’ and as a result to miss the big picture. It can all too easily become a matter of counting the trees rather than monitoring the growth of the forest.

(Brophy and Wynne, 1997)

Pinto, Ochôa & Vinagre

CONTACTS

Leonor Gaspar Pinto

[email protected]

Paula Ochôa

[email protected]

Maria Helena Vinagre

[email protected]

21


  • Login