The Weather in a Tank Project

1. The Efficacy of Using an Experimental Approach in Teaching College-Level Courses in the Atmospheric SciencesPresenterKathleen J. Mackin, Ph.D., Weather in a Tank Project Evaluator

2. The Weather in a Tank Project • A three-year project (2006-2009) funded by the National Science Foundation • Directed by MIT faculty, Department of Earth, Atmospheric, and Planetary Sciences, Program in Atmospheres, Oceans, and Climates (PAOC) • Purpose: To provide laboratory experiments, rotating tank and equipment, and web-based curricular materials to science professors and students in universities nationally to enhance teaching and learning in the field of Atmospheric Sciences. • More information can be found on the project website: http://paoc.mit.edu/labguide

3. Weather in a Tank University Collaborators Years One and Two • Massachusetts Institute of Technology (MIT), • University of Massachusetts, Dartmouth, • Pennsylvania State University, • Johns Hopkins University, • Millersville University, and • University of Wisconsin, Madison

4. Courses at Collaborating Universities Years One and Two Students in twenty-two courses were exposed to the experiments in years 1 and 2 of the project. These courses ranged from large introductory courses to small lab-based sessions. Examples: • Physical Oceanography (Pennsylvania State University) • Meso and Storm Scale Meteorology (Millersville University) • Climate and Weather Laboratory (MIT) • Atmospheric and Oceanic Circulation (MIT) • Introduction to Physical Oceanography (University of Wisconsin-Madison) • Fluid Earth (Johns Hopkins University) • Introduction to Weather (U. Massachusetts-Dartmouth)

5. Students Enrolled in Courses at Participating Universities Using Weather in a Tank Experiments • Approximately 500 students have been engaged in the experimental classes in the first two years of the project. • Gender: • 55% (256) of the students were males. • 37% (172) were females. • 8% (39) did not provide a code for gender. The following charts display students by Class Level and Major.

6. Weather in a Tank Experiments

7. The Purpose of the Weather in a Tank Evaluation The evaluation investigates the extent to which the Weather in a Tank project results in the following: • Active engagement of collaborating faculty. To what extent were faculty and students at participating universities actively engaged in integrating atmospheric data and laboratory fluid experiments in teaching and learning the basic principles of rotating fluid dynamics? • Enhanced learning outcomes for students. Is there a significant difference in posttest scores between the Treatment and Comparison groups, providing evidence that the model is working? Are there subgroups of students who respond to the Weather in a Tank model better than others, as evidenced by the gains on pre/posttest results? • Increased appreciation of the value and usefulness of this experimental approach. What were the benefits and challenges for collaborators in using the project equipment, experiments, and pedagogy? • Systematic efforts to sustain the experimental approach. What efforts are in place to systematically embed this kind of teaching in the curriculum and sustain the use of experiments beyond the project funding cycle?

8. Data Collection Instruments and Protocols • Web-Based Weekly Instructor Logs: Evidence from collaborators of number and type of demonstrations used, instructional learning benefits/challenges, etc. • Student Assessments: • Pre and Posttests: Evidence of student learning gains on 27-item multiple choice test implemented in Treatment and Comparison groups. • Performance-Based Assessment: Evaluation of student oral and written reports using a rubric to determine such factors as ability to understand and use scientific terms and concepts; design and implement experiments, analyze data, and communicate findings. • Collaborator Survey: Information from collaborators about their experiences with the project, including success with equipment , web-based materials, and level of project support. • Site Visit Reports: MIT project directors visit collaborating sites and provide on-site technical assistance and collect feedback on implementation of project. • Document Review: Review of project website, communications with collaborators via email and project listserv, collaborator meetings, etc. Note: Quantitative analysis procedures were used to analyze the pre/posttest results; qualitative methods such as content analysis were used to determine frequency of various activities and patterns of response from the Instructor Logs, Collaborator Survey, Site Visit Reports, and Document Reviews.

9. Preliminary Evaluation Results from Years 1 and 2

10. Summary of Demonstrations Used in Years 1 and 2 Questions: Why were some experiments used more than others (e.g. relevancy to course, effectiveness, ease of use)? How can instructors be encouraged to expand their repertoire of experiments?

11. Description of Pre/Posttest and Summary of Student Outcomes • Pre/Posttest: To determine the effect of the Weather in a Tank experiments on student learning, a 27-item multiple-choice test covering content related to climatology and meteorology was developed by project directors and the evaluator and administered to students in all Treatment and Comparison group classes during the first and last week of each term. • Treatment Groups: Students in Atmospheric Science classes who were exposed to at least four experiments during their course. • Comparison Groups: Students in Atmospheric Science classes at some of the same colleges who were not exposed to the experiments. • Analysis: The student pre and posttest scores in the first three iterations of the project (spring and fall of 2007, and spring of 2008) were analyzed using SPSS version 16.0. A t-test was conducted initially on Treatment and Comparison groups to statistically equate the two groups for the purposes of comparison. A series of ANCOVAs were conducted to control for the initial differences between the two groups at the pretest to determine differences at the posttest for the two groups overall and subgroups (e.g. students by major, gender, etc.)

12. a. b. c. d. warm cold warm cold cold warm cold warm pole equator pole equator pole equator pole equator Sample Pre/Posttest Questions • Which answer best explains why it is hotter in the summer than in the winter? (a) earth is closer to the sun in summer than winter (b) the sun burns more brightly in summer than in winter (c) the earth’s spin axis is tilted toward the sun in summer (d) the hemisphere experiencing summer is closer to the sun than in the winter • When cold air from the pole meets warm air from the tropics, the boundary between the two air masses looks most like:

13. Student Outcomes on Pre/Posttest Measure • Findings: • There was a highly significant difference between the posttest scores for the two groups in the Spring, 2007 and Fall, 2007 (p<.001) with the Treatment group scoring higher than the Comparison group during each of these testing phases. Preliminary Analysis of Spring 2008 scores indicate a similar trend. • These results suggest that exposure to the MIT Weather in a Tank experiments and curriculum, such as that received by the Treatment group, contributed to student learning outcomes in these classes.

14. Climatology-Related Majors Compared with Other Science Majors n Pretest Mean Score Level of Significance n Posttest Mean Score Level of Significance Treatment Group Climatology-Related Majors 54 18.4 P<.05 49 19.8 N.S. Other Science Majors 19 16.1 16 19.2 Comparison Group Climatology-Related Majors 28 17.0 P<.05 30 17.9 P<.05 Other Science Majors 15 13.0 13 14.1 The Significance of Effect of Major on Pre and Posttest Scores (Spring, 2007) • Treatment group students majoring in Climatology-related fields scored significantly higher on the pretest (18.4) than the “other science majors,”(16.1), but this statistical difference was erased at the posttest (19.8 and 19.2 respectively). These same results were found in Fall, 2007 data. Analysis of Spring, 2008 data is not yet complete. • The “Other Science Majors” in the Comparison group did not perform as well as their counterparts in the Treatment group and scored significantly lower than the Treatment Group at the pre and posttest. • These results suggest that the experiments can be especially beneficial in helping students, especially science majors in non-Climatology related fields, understand and use content that was initially unfamiliar to them.

15. Collaborators’ Perspectives on the Benefits and Challenges of Using the MIT Experiments in Their Instruction Benefits of Using the Experiments: • Better enabled professor to illustrate a point, • Prompted student engagement in questioning and interpreting data, • Assisted students in looking beyond the facts and making predictions, • Encouraged students to conduct further inquiry into a phenomenon, • Contributed to a livelier, more engaged classroom experience, and • Enhanced professors’ instruction by allowing them to develop more empirical explanations of phenomena. Challenges in Using the Experiments: • Equipment difficult to move around-best to have a stationary location, • Difficult for large audiences to view, • Lighting for projector is not adequate in all settings, and • Need for training prior to using experiments. Source: Instructor Logs

16. Sample Comments from Collaborators • Students proved more motivated to develop mathematical theory to explain the observations. • Weather in a Tank was a great way to connect with students, especially in larger classes. • Students were motivated to develop experiments on their own during the term and over the summer. • Students love getting their hands wet after so many theory classes. • The students were impressed that the problem of a ball rolling around on a rotating parabola led to the equations of simple harmonic motion that they had studied in Physics classes. Thus, the experiment seemed to help bridge between fields. • It may be wishful thinking, but I believe that this first experiment (Dye Stirring) helped some of the students appreciate the theory. The demonstrations greatly helped students visualize how fronts adjust to a cone shape under the effect of rotation. • Source: Instructor Logs and Collaborator Survey

17. Summary • Instructors used the project equipment, experiments, and web-based materials extensively. • The pre and posttest measure provided evidence of student gains in content knowledge as a result of participating in classes where Weather in a Tank experiments were used. • The model appears to be efficacious for all levels of students, but particularly for some subgroups, such as those with a background in “other” sciences, but who are new to the field of Atmospheric Sciences. • Instructors reported that the experiments, project website, and curricula were very effective in enhancing their instruction. • Instructor feedback provided evidence that student motivation, engagement , and level of scientific inquiry increased as a result of exposure to the experiments and project curricula.

18. Remaining Questions • Which subgroups of students benefit most highly from this kind of instruction? • Is this kind of experimental approach equally useful for large and small classes as well as labs? • What other kinds of learning outcomes do students experience as a result of being exposed to the experiments (e.g. decisions to major in science or increased interest in science, decrease in science-phobia, increased enthusiasm for experimental inquiry and research, etc.)? • What kind of instructor training and implementation strategies are necessary to obtain optimal results from the experiments? • To what extent will instructors continue to incorporate these kinds of experiments in subsequent classes? • How can these efforts be sustained at the institution level beyond the funding cycle of the project?