Learning Gains on Newtonian Conceptual Reasoning in an Iterative, Project-Based Design

1. Learning Gains on Newtonian Conceptual Reasoning in an Iterative, Project-Based Design Paul J. Camp Spelman College Why U-shaped Development Severely Limits the Validity of Pre/Post Testing for Research Abstract I compare the normalized gain on conceptual Newtonian reasoning in a course based on iterative design projects to a linearly organized non-project based course in middle school. In addition to pre/post gains, I also measure incremental changes over time and detect a degradation of performance at intermediate times. This is interpreted as an example of U-shaped development, which in turn implies a fairly stringent criticism of standalone pre/post testing as an assessment or research strategy. A greatly expanded version of this paper is online at http://arxiv.org, document reference arXiv:1007.1647v1 U-shaped Development Cartoon Pre/post testing uses performance as a proxy for understanding. U-shaped development shows that it is not. Performance and conceptual understanding are not tightly coupled. Without another line of evidence, you cannot tell where in the developmental cycle the posttest occurred. This is especially problematic for short time scale measures as are often used in tutorial validation. Where is your posttest? Performance Design for a Study of Cognitive Development The Evidence for Non-monotonic (U-Shaped) Developmental Processes Here is your pretest Because of its iterative nature, LBD (see below) easily affords a study of the development of Newtonian Concepts and scientific reasoning over time, which we undertook in 2002-2004. This study deployed a multiple measures approach to determine the developmental trajectory, including: time As can be seen, the LBD class significantly outperformed the comparison (p << 0.01, r=0.83, d=3.0). What happens between pre and post in the LBD class is considerably more interesting and forms the core of this paper. Quiz items were divided into clusters by concept. Student performance at each quiz was placed into one of three categories – N (Newtonian, defined as at most one error), NN (Non-Newtonian, defined as at least one correct answer), and T (transitional, defined as anything else). Due to the nature of the data it was not possible to make every classification at every point on every concept. Nevertheless, the pattern is quite interesting. Standalone pre/post testing gives a measure of a state of understanding at one moment in time. It does not tell us if that change is meaningful, or stable, or how it came about, or if there was no change does that imply no understanding. It may be useful as a rough measure during curriculum development but it is not a reliable method for documenting the effectiveness of an intervention. Conceptual quizzes pre, post and after each activity cycle. Questions drawn from FMCE, FCI and some custom based on the developmental stages identified by Thornton. Structured Interviews of a subset of students along with each quiz, based on a simple experiment and intended to determine the level of prompting necessary to elicit Newtonian responses. Ethnographic observations in class between two and three times per week throughout the implementation Performance assessments pre and post based on a single day group design and experiment activity with discussions videoed. Why the Pre/Post Results for LBD Are a Valid Measure The short answer is that they are not standalone. The presence of intermediate quizzes, ethnographic class observations and structured interviews over the same period of time allows us to document where in the developmental cycle the post test occurred. This paper is primarily concerned with the quiz data but will draw on other sources for support and interpretation. Generalization of Results U-shaped cycles have been observed in language development ,artistic creativity, facial perception , social cognition, ratio reasoning, musical understanding, intuitive thinking, scientific reasoning, phoneme discrimination, auditory localization, motor development , development of written representations, object permanence, vocabulary acquisition, gesture-referent relations, memory and false memory storage and retrieval, category formation, and transitions between levels of expertise in various knowledge domains among adults. This vast array of totally unrelated skills showing the same developmental process suggests that it is a general feature of human learning and cognition. I have now documented it in physical reasoning. It seems, given the nature of the explanation, that it will be a pervasive feature of physics pedagogy that has hitherto gone unnoticed, perhaps due to overreliance on pre/post testing. In any case, results from pre/post testing that do not involve independent means of determining where in the developmental cycle the post test occurred are intrinsically impossible to interpret. Pre/Post Between LBD and Comparison Class At a suburban middle school in Atlanta, GA, we compared an honors LBD class with a standard honors class in two successive years. Teachers were carefully matched with regard to background and experience. The non-LBD classroom was not entirely lecture based, but activities tended to be much smaller scale, more directive in approach and of significantly shorter duration. Observation Most of the data (acceleration excepted) shows the same general pattern of performance change: There is an advance to a high state of performance, followed by regression to a prior state. With continued experience, good performance eventually returns. Why Does Non-monotonic (U-Shaped) Development Happen? Student A Siegler2 describes three general categories of developmental explanations for this phenomenon. I: What laws do you recall? A: I think the first one is any object in motion stays in motion unless it is acted upon by another object. Third law [long pause] I don’t remember third law. It’s got something to do with acceleration and velocity because I remember the second and third law quiz was finding velocity from, finding acceleration from the starting velocity and the final velocity over time. I: This semester you guys learned about forces in pairs. What do you know about forces in pairs? A: They’re the same size. I: What does it mean? What does forces in pairs mean? A: That two forces acting upon each other, like going against each other I guess. I: OK, give me an example. A: Like I know we did a lot of forces in pairs with the balloon cars, where the air was pushing out on the balloon and the balloon was pushing back in on the air which forced the air out. . . I: In this experiment, with the car going down the track, are there any places along here where forces in pairs are demonstrated, where you see forces in pairs? A: I don’t think so but I’m probably wrong. I: When the car is in the launcher, do you think there’s forces in pairs in the launcher? A: [quickly] Yeah, yeah, yeah. Because the car pushes back on the launcher, and the red block [the launcher] pushes on the car. I: Can you talk to me about those two forces in terms of their size and direction? A: Um, well, the car has the same force on the red block as the red block has on the car. [Indicates opposite directions by pointing fingers when mentioning each force]. Newly acquired reasoning processes may be over generalized, and applied in situations for which the new approach is inherently flawed. A newly acquired processing approach places greater demands on the processing system than does one that is of long experience. Therefore, the greater cognitive load may result in temporary decreases in efficiency. The skill in question is actually composed of multiple subskills which develop (monotonically) at different rates. One monotonically developing subskill may therefore temporarily outstrip another, producing multiple peaks in performance as different subskills come on line. Since it is impossible to over generalize Newton’s laws in this context, and since the prior reasoning is phenomenological so the second category would not account for the first performance peak, the most relevant for us appears to be the third type of explanation. There are two obvious broad categories of subskill involved in the production of physically correct Newtonian performance – conceptual understanding, and appropriate access to that understanding. In the case-based reasoning model of cognition [7], access to understanding crucially depends on links (or indices) between that memory and contextual cues. We would say that when performance is variable, the relevant memories are poorly and/or sparsely indexed and therefore mostly inaccessible except in very specific situations. The correct pedagogical approach, predicted by this model, would therefore be to shift the same content to a series of different contexts, which require drawing on the same memories in order to be successful, and thereby deliberately triggering a series of representation failuresand therefore dips in performance. Learning by Design™ Course Structure Learning by Design (LBD)1 uses engineering design projects as a context within which to learn qualitative Newtonian concepts along with general scientific reasoning and design skills. It is fundamentally a cognitive apprenticeship approach to learning with a learning cycle and activities based on Case-Based Reasoning, a cognitive model of reasoning by analogy that contains within it a model of how memories are stored that will afford transfer to new situations. The class is organized around a nested learning cycle in which design-and-test activities motivate and contextualize scientific investigations which in turn provide information relevant to redesign. Student B Prediction of the Model I: Earlier we were talking about laws, Newton’s laws, and stuff, and you said “I don’t know the numbers.” Uh, and you said something about objects pushing on each other. C: Yeah, equal but opposite forces. I: Can you explain what that is, or just give me an example or something? C: Ms. [Teacher] did a really good one, it was when, like a huge truck and a little car and they hit each other they would each experience a force and it would be equal but it would be opposite . . . I: Are there any instances with this launcher. . .where we’re seeing these equal and opposite forces being demonstrated to us? C: Uh, [long pause] well, the only thing I can think of that would really affect it is if it hit something, like another car on the ramp or something. Um, because I’m not so sure about the force of friction, um, if that would be an equal, I don’t think that’s an equal and opposite force. I: What about the launcher and the car? C: [quickly] Yeah, the car’s pushing back on the launcher and the launcher is pushing forward on the car, and the forward force of the launcher If this explanation is correct, then an immediate prediction is that during dips in performance we should see students exhibit understanding of Newtonian explanations but having difficulty accessing them in appropriate situations. This would imply that understanding is continuing to advance while performance degrades. We can use the structured interviews to check this prediction. Structured Interviews In the main physical science unit, students make at least four passes through this cycle in the course of designing and testing powered vehicles which must traverse a specified test track. Each time, they vary the engine, requiring progressively more extensive Newtonian reasoning to understand its functioning. In contrast to many curriculum designs, in LBD students are wrestling with the full range of relevant force and motion concepts from the beginning. Learning comes from successive refinement of that range of concepts through multiple passes in varying contexts. Those passes are labeled according to the engine: Coaster Car (CC – a ramp), Balloon Car (BC – balloons), Rubber Band Car (RB – a rubber band wrapped around the axle) and Hybrid Car (HC – best functioning design drawing on all the engines, including a falling weight that is introduced by demonstration after RB). The structured interview task involved a toy car launched down a track with a spring loaded launcher. A rough surface could be added to or taken away from the track, and the spring launcher could be substituted with a hill that caused the car to travel the same distance. The following excerpts come from the third law portion of the interviews, for which the quizzes show the clearest case of U-shaped development. Four students were interviewed from the LBD class. Two of them had spontaneously used third law logic at an earlier point in the interview and two had not. I present one example of each. Each of these excerpts shows a student with an understanding of the third law but an inability to access it as needed. That is in fact the story for all of the interview participants at this time. 2R. S. Siegler, U-Shaped Interest in U-Shaped Development – and What it Means, J. Cog. Dev.5, 1, p1 (2004) 1J. Kolodner, P. Camp, D. Crismond, B. Fasse, J. Gray, J. Holbrooke, S. Puntambekar and M. Ryan, Problem-based learning meets Case-based reasoning in the middle-school science classroom: Putting Learning by Design into practice, J. Learn. Sci., 12, 4, p495 (2003)