The effectiveness of learning undergraduate economics with animated videos: A quasi-experiment

1. The effectiveness of learning undergraduate economics with animated videos: A quasi-experiment Dr. Alice Shiu School of Accounting and Finance, Hong Kong Polytechnic University Ms Judith Watson School of Economics, UNSW Sydney Business School Dr. Joseph Chow Educational Development Centre, Hong Kong Polytechnic University Presentation for eLearning Forum Asia 2018, May 23-25, 2018, Taipei, Taiwan

2. Background • Designed and constructed animated videos for - a self-financed econometrics subject at School of Accounting and Finance (AF) in 2015 (supported by T&L grant at AF) - a University Grants Committee (UGC)-funded economics subject at AF during 2016-2017 (supported by Dean’s Reserve at Faculty of Business (FB)) • Teamed up with the Resource Centre for Ubiquitous Learning and Integrated Pedagogy (ULIP), now Apps Resource Centre at Baptist University to develop a mobile app including the animated videos for the UGC-funded subject during 2016-2018 • A mobile app (PolyU Econ) “Principles of Economics – An Interactive Glossary” including Topics 1 to 4 of AF3625 Engineering Economics available in Google app store since January 22, 2018 • Conducted a laboratory experiment in Business Experimental Research laboratory (Bizlab) at the University of New South Wales (UNSW) Business School – testing effectiveness of four mobile app videos compared to written text

3. The Mobile App

4. Laboratory Experiment at UNSW • Purposes: • Evaluate the general effectiveness of animated videos compared to written text in learning microeconomic concepts • Measure different aspects of students’ perceptions of the two types of learning materials • Obtain students’ feedback for the two types of learning materials • Research Design (Qualtrics surveys): • A pre-test • A post-test • One type of learning materials (either four animated videos or four sets of written text) • Open-ended questions about learning perception • Open-ended questions for feedback about learning materials

5. Recruitment of Participants and Pilot Run • Recruitment process for participants: • Started in September, 2017 • Recruited 10 students (pilot run) and 80 students (actual experiment) at UNSW • Time of pilot run and actual experiment: • Mid-November and late November, 2017 • Location of pilot run and actual experiment: • Bizlab at UNSW Business School

6. Participants Economics experience Gender • 79 participants: 39 students (video group) and 40 students (text group)

7. Participants First Language Nationality

8. Procedures of Experiment • First section • Participants read the online participant information statement, signed the consent form and read the instructions. Then, they started to fill in their demographics information in the Qualtrics survey installed in the computers • Second section • Participants attempted a pre-test with eight multiple choice questions in the Qualtrics survey to check their prior economics knowledge • Third section • Participants watched four animated videos or read four set of written text across microeconomics topics (following the order of (1) Opportunity cost, (2) Market demand and supply, (3) Price ceiling and (4) Elasticity of demand). After watching each video or reading each set of written text, they answered two survey questions on the usefulness of the learning materials and provided open-ended feedback of further improvement and effectiveness of the learning materials. Then, they answered six survey questions about their learning experience and knowledge of the learning materials • Fourth section • Participants completed the experiment by doing the post-test which comprised of the same eight multiple choice questions (but randomized) as in the pre-test for evaluating learning outcome of watching videos or reading written text. The correct answers of the post-test questions were shown on the computer screen

9. Incentives, Rules and Administration of Experiment • Incentives • A show-up fee of 5AUD and a participation fee of 30AUD • 1AUD for each correctly answered multiple-choice question in the post-test • Participants needed to fill in the receipt form which stated the number of correctly answered questions in the post-test and payments were given before they left the laboratory • Rules of experiment • Not allowed to communicate with each other or to use the Internet or their mobile phones • Not allowed to return to the previous page once started to fill in the Qualtrics survey • Not allowed to watch the videos or written text or any written notes while they did the pre- and post-tests, despite no time and quantity limit of watching each video or each set of written text during the experiment • Administration of experiment • Pens and paper were provided for making notes • Team member at UNSW monitored the experiment and a student helper provided assistance for technical issues and making appropriate payment

10. Animated Videos and Written Text • Production Process of Animated Videos • Design of storyboards for animated videos (topics including opportunity cost, market demand and supply, price ceiling and elasticity of demand) • Icon searching • Video construction in PowToon and Camtasia • Voice recording through UNSW • Production Process of Written Text • Selection of relevant content in existing self-study notes of teaching subject • Adjustments in content and length of the notes

11. MC Questions in Pre- and Post-tests

12. Survey and Open-ended Questions

13. Summary of MC Score Change and Time Spent by Video/Text Group MC test score change is defined as (Post-test score – pre-test score)

14. Summary of Negative and Positive MC Score Change for Video and Text Group

15. Percentage of Participants Choosing Option E “I’m unsure” in MC Questions

16. Summary of Participants’ Perception by Video/Text Group Likert scale for Questions 1 to 7: 0= strongly disagree, 1 = Somewhat disagree, 2 = Neither agree nor disagree, 3 = Somewhat agree, 4 = strongly agree Likert scale for Question 8: 0 = very poor, 1 = poor, 2 = average, 3 = good, 4 = excellent

17. Summary of Results • Both groups, on average, showed improvement in the post-test and growing confidence of students in answering individual questions • The two groups were also assessed by their perception of learning experience. Our findings supported a positive student perception of videos • First, the videos showed similar rating with the text material in students’ perception of whether the medium has provided valuable knowledge of micro-economics • Second, in comparison with the text material, it showed a comparable level of perceived learning experience, including helpfulness for understanding, motivational power, easiness of instruction, and ability to raise understanding and usefulness for lecture preparation • These findings suggest that animated videos can be an effective learning tools in microeconomics higher education which has comparable educational impact to similar text-based material

18. Keyword Occurrences in Qualtrics • Searching for keyword occurrences in Qualtrics

19. Quotes from Participants (Written text)

20. Quotes from Participants (Video)

21. Suggested Improvements • There is scope to make improvements in both the videos and text materials by looking carefully at feedback from students • Participants see daily examples used as illustration in the videos as very important to their understanding so we need to fine-tune the existing ones and ensure appropriateness of these examples • It is also important to provide sufficiently detailed definitions, so that concepts are more accessible to those who haven’t studied economics before

22. Further Work • Possible Improvement of Materials • Consider feedback from students and fine-tune the existing examples used in the learning materials • Ensure visual devices such as diagrams are displayed clearly • Provide more detailed definitions of economics concepts • Further Work • Conduct detailed analysis on qualitative comments collected for the animated videos which will help improving the quality of existing videos • Construct a framework which optimizes the use of both types of learning materials for a better learning experience for the students

23. - Thank You -