The Framework of Qualitative Evaluation in Online HRD of In-service Teachers

1. The Framework of Qualitative Evaluation in Online HRD of In-service Teachers Yoshida, Masami, Ph.D. Professor Faculty of Education, Graduate School of Humanities and Science Chiba University, Japan

2. Study Outline • In-service teachers’ training • Computer room • E-Learning (TCU-LMS) • Online assignment • Qualitative evaluation • Forth Generation • Criteria for Validity: Guba’s criteria • QSR NVivo • Interpretive Approach • Scoring Policy: TRIZ • Action Chart

3. Teachers’ License • Personalization of education • Small sized school • Small sized classroom • Team teaching • Increasing special education • Need expand adaptabilities of each teacher in terms of expertise • From pre-service (number) to in-service training of enhancement • Continuing education and HRD for teachers • Summer vacation • Credit • License

4. Incoherence • Entire distance method was not permitted by stipulations • Objective-oriented evaluation was not fairly suitable to evaluate teacher-students learning under constructivist paradigm of education This study argues effects of new way of absolute scale evaluation

5. The Course Details • “Educational Methodology and Skills” : 1 credit • Use two days (9:20-16:40) • E-learning was used as prime method of learning and all lesson contents were accessed through online (LMS: TCU-LMS, the Japanese-language version) • The 38 teacher-students who had at least 12 years teaching experience • Content: 15 sections, include 182 Web links (knowledge resource: 22, others are vital communication pages), 94 pages in printed volume • Motivate teacher-students to activate potential resources & expansive learning activity through net surfing

6. Historical Changes of Evaluation Target

7. First Generation: Period of Educational Measurement • 1900- • To determine potential skills of people • Indicators of social skills • Used to enforce conscription, arrangement of study groups in a school • Intelligence quotient, aptitude tests, personality tests were developed • Influence of Thorndike’s theory of learning: law of readiness (Hergenhahn & Olson, 2000)

8. Second Generation: Period of Educational Evaluation • 1930- • Expanded secondary and tertiary education • Educational Indicators • To know changes of students’ • Needs to value educational effectiveness • Continuous monitoring: time series based study • Statistical methods to judge significance • Achievement test, standardized test • Influence of theory of John Dewey

9. Third Generation: Objective Based Evaluation • 1960- • Accountability of educational institutes • Indicators from curriculum • Objectives based evaluation • Used various educational materials and media (Stake, 1967) • Formative evaluation and masterly learning • Standard score (T-score) for education

10. Forth Generation: Responsive Constructivist Evaluation • 1990- • Scientific evaluation frequently led managerialism (Guba & Lincoln, 1989) • Indicators from specific classroom • Constructivist paradigm • Qualitative evaluation; Naturalistic inquiries • Evaluation criteria were anti-preordinate, determined through observation and discussion of stakeholders • Pay more attention on validity than reliability

11. Inductive Procedure Positivism Approach (Generation 1st-3rd): Evaluation = Rule (situation, action) This aims to explain action and predict another situation to define a rule that is available in other cases (Belk, Sherry Jr., & Wallendorf, 1988) Use deductive data analysis. Interpretive Approach (Generation 4th): Evaluation = Interpretation (situation, action) This aims to interpret action and recognize specific situation. Use inductive data analysis

12. Methods

13. Purposive Sampling • Final assignment: designed each student to post proposals in a Web board as homework “Describe your proposal that improves learning of your students to be more self-directed. Especially, explain the way how you intend to apply educational methodology and knowledge of technologies.” • Purposive Sampling • All descriptions of proposals were introduced into QSR Nvivo7, and processed qualitative data analysis • Valued by stakeholders: an instructor and a coordinator • Use naturalistic inquiries, that is similar to moderation (Gipps, 1994) • 7 x 4 treatment by seven action criteria (action chart) and four adequacy criteria (validity).

14. Action Criteria All proposals were arranged into the table framework of the action chart that was postulated by Mills (2000), where the following elements are positioned. • Research question (findings) • Recommended action that targeted a given finding. • Who responsible for specific actions. (responsibility) • Who needs to be consulted or informed about the findings of the lesson and the associated actions? • Who will monitor or collect the effects of actions. • Dates when the actions and monitoring will occur. • Any resources and media that will be needed to carry out the action.

15. Adequacy Criteria Each element of an action chart was reviewed based on Guba’s four criteria of the qualitative paradigm, and valued trustworthiness. • Credibility (CR): roughly parallel to internal validity. It poses the questions of whether the explanation fits the description and whether the description is credible. It refers teacher’s ability to take into account all of the complexities. • Transferability (T): roughly parallel to external validity, generalizability of explanation. This concern only case-to-case domain specific data transfer in a naturalistic qualitative study view. It is teachers’ beliefs that everything they propose is context bound. • Dependability (D): roughly parallel to reliability. Proposal is responsible for ensuring that the process of lesson is logical, traceable and clearly documented. • Confirmability (CO): roughly parallel to objectivity. Data and interpretations of the proposals are not figments of the teacher-student’s imagination, but are clearly derived from the learned contents.

16. Scoring Policy • Developed through the discussion of stakeholders • TRIZ and invention cards were used to determine the warrantable scoring policy (Altshuller & Altov, 1994). • No.2: taking out • No.5: merging • No.40 composite materials • Decided a way to count positive points in valid measures that were applied from lesson contents as long as negative points did not exceed positive points • If a negative point exceeds over a positive point, we valued the action chart one level below.

17. Findings Possible adequacy criteria were marked on elements. Then, positive and negative adequacy criteria were assigned.

18. Action Chart

19. Results • For reference, scores of achievement test were compared (Table above). • The mean of achievement test on the evaluation levels were not significantly different at the alpha =.05 level with one-way ANOVA procedures. • The calculation results of mean positive score by criterion (Table below). • Credibility and Confirmability had higher score than Transferability and Dependability.

20. Discussion

21. Appeared Expansive Learning • Vigorous learned things were appeared as their applied methods • An expansive learning process in which lesson contents and information of other communities are used together for generating new idea. • It was prominent that using e-learning and Web surfing enabled them to invoke potential information that could not be possible to overview in the traditional F2F lecture with limited knotworking. • The author sometimes sensed insufficient descriptions • Insufficient context • Lack: details explanation, situational suitability, cares about limitation of methodology, and involvement of other staff in the process

22. Appeared Changes • Qualitative evaluation method was vigor for interpretation and easy to lead diagnostic opinions. • The proposals were made in the situation of their peculiar classroom, but adequacy criteria have enabled to value their varied proposals • Figure shows summed number of how many plural criteria were marked together in one element of an action chart • This data showed that their plans were valid to show combined marking

23. YOSHIDA Masami Professor of Chiba University yoshida-m@faculty.chiba-u.jp