Formally assessing an instructional tool: A controlled experiment in software engineering

1. Formally assessing an instructional tool: A controlled experiment in software engineering IEEEE Transactions on education, 48(1), 2005 報告人：楊易霖

2. 摘要 • 本研究描述一控制實驗(controlled experiment)在教學過程中，有關於學習輔助工具(learning aid)的運用。 • 研究的核心議題為學習輔助工具可藉由個人化教學所提供的知識(offered knowledge)，以增加學習者認知的移轉。 • 針對這目的，一控制實驗以79位大學生進行。 • 教學範圍有二個課程有關於人機互動界面：一為可用性工程；二為界面評估方法。 • 檢定部份進行學生知識領域增加的評估。 • 描述性統計與推論統計來檢定研究假設。 • 本研究結果呈現有關於簡單資訊的移轉部份，課程單(Lesson sheet)這學習輔助工具不能提供顯著的好處；複雜資訊部份，在運用此工具上有顯著的改進表現。 • 最後，呈現此工具應用的事項與在此領域的未來研究。

3. Introduction • Various “learning aids” have been proposed, such as study frameworks, time scheduling, and summaries, because learning issues are, after all, complex and highly case sensitive. • Thus, there is a broad consensus that instructional tools aim to support the student in his or her work in order to confront eventual learning difficulties and to facilitate him or her in personalizing the offered knowledge. • This study was motivated by the need to investigate a tool that could support students during the lesson, as well as after the lesson during their study at home, providing them with the ability to personalize the offered knowledge. • This study investigates a “lesson sheet” for use during the live lessons, which aids the student in capturing the offered knowledge and provides assistance after the lectures so that the student can understand and assimilate the material presented.

4. The Lesson Sheet • A sample reference sheet is shown in Fig. 1. • The left-hand column contains the outline of the lesson in paragraph form. • While the righthand column provides a variety of context-sensitive information that could be a summary of the subject under consideration. • such as charts, graphs, pictures, or other material related to the issues under discussion. • There is adequate free space for the student to keep his or her own notes, thus personalizing the sheet by completing it with information that is relevant to his or her work at home.

5. The Lesson Sheet

6. The Lesson Sheet - basic construction guidelines 1.The lesson sheet utilizes a table to frame and structure the offered information. 2.The left-hand side contains the information presented in terms of “titles” and “paragraphs.” 3.There must be adequate recall of situated knowledge, such as an already known definition, image, graph, etc. 4.On the right-hand side, students can note whatever they want, thus personalizing the lesson by the use of their own comments. 5.Images, figures, charts, and any other appropriate means are included to visualize the information. 6.Notes that have nothing to do with personalization, such as Internet links, tables, additional or external information, and resources, must be provided on the sheet to accelerate the procedure and diminish the distraction that occurs during the students’ noting time.

7. The Lesson Sheet - basic construction guidelines 7.The left-hand side includes the necessary self-assessment exercises and the activities the student has to perform during the lesson, while the corresponding space on the right-hand side is usually reserved for the answer. Exercises of gradual difficulty must also be included. 8.The lesson sheet must provide immediate feedback on self-assessment exercises in order for the students to correct their mistakes. 9.It must provide text and theory of no more than two to three paragraphs for every “chunk” of information. An alternative option must be provided, such as an exercise, an external resource, or a student activity. 10.It must be sound in its size. The use of the sheet in practice has shown that four to five pages are sufficient for a three-hour lesson. In addition, good image quality and high-quality printing is a must.

8. The Experiment • The formal definition of the experiment is as follows: “Analyze the application of the lesson sheets for the purpose of assessment with respect to their effectiveness during the instructional procedure from the point of view of the instructors in the context of university-level lessons on usability engineering (UE) and for simple or complex kinds of information.” • Variables • The independent variable (factor) • the use of the lesson sheet. It is of nominal type, with two possible values (yes or no). • subdivide into two factors: • the use of the lesson sheet concerning simple kinds of information • the use of the lesson sheet for complex kinds of information

9. The Experiment • The dependent variable • the students’ acquisition of knowledge on the taught domain, measured in terms of a grade in a test. • measure on a ratio scale, from 1 to 150, and is operationalized in terms of the 15 questions depicted in the final test questionnaires. • Hypothesis statement • Ha: There is no difference in the students’ knowledge on the taught domain because of the use of the lesson sheets for simple kinds of information. • Hb: There is no difference in the students’ knowledge on the taught domain because of the use of the lesson sheets for complex kinds of information.

10. The Experiment • Selection of Subjects • Two independent groups were formed from 79 participating students attending the fifth semester of the Department of Informatics at Aristotle University of Thessaloniki, Thessaloniki, Greece. • They were divided randomly into a control group of 40 students and a treatment group of 39 students. • Experimental Design • The students were taught two three-hour lectures (one per week) concerning two selected aspects of UE: introduction/definitions of UE and evaluation. • The control group students received a “blank” sheet (to keep notes during the lessons), while the treatment group students received the real fully developed lesson sheets.

11. The Experiment • Instrumentation • The two lectures concerned UE basics and evaluation methodologies. These were chosen because they provide many new ideas, and they include simple and complex aspects. • the lesson sheets included “simple” and “complex” information in random order • “Simple” information is defined here to be information, such as definitions (requiring only memorization), list items (requiring common sense), or simple ideas(maybe already known). • “Complex” information is defined here to be information, such as new ideas (often requiring further explanations), more than ten-item correct-sequence tasks (the student has to remember the tasks and the correct order), matrix filling (the student performs cognitive combinations), or information depicted in a figure (which often needs further explanation). • Questionnaires • In the final test, the questionnaire consisted of 15 questions—eight simple and seven complex.

12. Data analysis and interpretation

13. Data analysis and interpretation

14. Discussion and conclusion • Results Interpretation • one can argue that no significant improvement of the students’ performance occurs while working with “simple” kinds of information. • Although the statistical analysis did not provide any statistically significant difference between the groups, the control group performed slightly better. • This result can be interpreted only as an indication that this group could better personalize the offered information by keeping notes on their own and not being framed by the depicted information on the tool. • A statistically significant performance of the treatment group was observed. • The students seemed to have difficulty denoting this kind of information during the lectures under time pressure, but the tool seemed to provide fair support in this direction by providing them with all the information that was too complex to be recorded manually and in quickly.

15. Discussion and conclusion • Conclusion • The lesson sheets appeared to provide a valuable instructional aid in cases where complex information must be learned and assimilated, yet they can hinder the personalization of the offered knowledge in the case of simple information where students should be allowed to adapt and transform their own learning style and idiosyncrasy. • This conclusion should be kept in mind when designing the sheets. • Although a generalization can hardly be concluded, indications show a common acceptance of the tool by all classes, especially according to qualitative approaches such as interviews. Therefore, more study on multiple domains is needed to reach valid results for a broad application of the proposed tool.