Assessing engineering skills ( in a way that also supports students in developing them )

1. Assessing engineering skills(in a way that also supports students in developing them) Kristina Edström & Jakob Kuttenkeuler Lyngby, 21 December 2009

2. Large project groups (5-20 students) Entire final year 1/4 time speed Multi-disciplinary Very open-ended A final year experience- based, design-build, project course

3. Open-ended tasks Task 2002: Design, build and fly a solar powered aircraft that can take 2 kg payload.

4. Ahh, the course where you teach the students to build cool stuff!?

5. What is the purpose of project work in education? Learning objectives Project goals Project

6. Powerful principle 1:the purpose is student learning

7. A Helsinki teacher said: ”It is a pain to have students build anything. It takes them 3 months to do what the technicians can do in 3 days.” No. It takes them 3 months to learn. They don’t build for the purpose of building, the purpose is to learn (from building). The purpose is learning Project goals Learning objectives

8. New projects every year...

9. After the course the participant is expected to be able to: analyse technical problems in a systems view handle technical problems which are incompletely stated and subject to multiple constraints develop strategies for systematic choice and use of available engineering methods and tools make estimationsand appreciate their value and limitations pursueown ideasand realisethem practically make decisions based on acquired knowledge assess quality of own work and work by others work in a true project setting that effectively utilises available resources explain mechanisms behind progress and difficulties in such a setting communicate engineering – orally, in writing and graphically ... always the same learning objectives

10. Teachers advice & coach but avoid to impose solutions Students create new things

11. CDIO conceive design implement operate Students create new things

12. New year - New group - new task Neither the students nor the teachers know the answers Students create new things

13. Applied use of theoretical skills Whatever is designed has to be realised Students create new things

14. infernus 3 min

15. Individual grades A-F Assessing individual performance in a group setting Students work on many different tasks Teachers see only fragments of the actual performance Assessment challenges

16. Staying focused … Project Product ”according to plan” Performance Course Learning objectives Constant battle between product focus and maximizing learning

17. Powerful principle 2:feedback and reflection on experience • The process is rich with peer feedback and self-reflection activities • because experience results in learning only if reflected upon

18. Powerful principle 3: reversing the ’burden of proof’ • Each individual student is responsible for collecting and presenting evidence of how well they fulfil the learning outcomes (portfolio) • this enhances reflection (learning) • this makes the course sustainable in terms of teaching resources

19. Assessment – the Introduction Faculty • communicate course goals • instruct students to collect evidence in “portfolios” Students • express personal individual goals • plan own activities SEPTEMBER JUNE

20. Mid Course Poll Faculty • repeat course goals • lecture on giving/receiving feedback Students • write summary • read summaries, write feedback, suggest peer grades • read feedback & reflect • revisit/revise personal goals • follow-up on the process SEPTEMBER JUNE

21. Summary template Summarize your work mapped on the learning objectives on max 1 A4 • Learning objective 1. • Learning objective 2. • … • Learning objective 10. • References: Xxxx (reports, presentations, protocols, pictures, sponsring…)

22. Sample Summary • L10. Present technical work (orally, in writing and using other tools). Status: Satisfied. Ref: [2] [4] [5] [6] [7], - I am author of 7 reports of which [2][5][6] as main author. - Prepared and given the presentation on the preliminary design [3] together with Jocke. References: 1. Meeting minutes from … 2. Presentation, Preliminary design at design review #1 3. Experiment 4, Planning, execution and results 4. Report A 12, Hydrostatic stability - analysis 5. Report A107, Engine, design and mounting …

23. Sample Summary • L7. Effectively choose and use available engineering methods. Status: Approaching. Ref: [4][5][6] , - I am trying but find it hard to find the balance between rough estimates and sophisticated computerized methods. Further, the word “effectively” does not apply on me. References: 1. Meeting minutes from … 2. Presentation, Preliminary design at design review #1 3. Experiment 4, Planning, execution and results 4. Report A 12, Hydrostatic stability - analysis 5. Report A107, Engine, design and mounting …

24. Final Assessment Faculty • recap course goals Students • write summary • read summaries, write feedback & suggest grades • follow up discussions on the contribution from each student • reflective exercises • write reflective documents SEPTEMBER JUNE

25. What do you think, should the success of the product influence the grades?

26. A Sydney teacher said: ”We have them build a rubber band-driven car and compete. If the car runs 15 meters they get the highest grade, 12 meters the next grade and so on...” No. They should be graded on how well they reach the learning outcomes, not the product performance. The purpose is learning Project goals Learning objectives

27. Grades The grades are set in relation to the intended learning outcomes based on a holistic assessmentof portfolios(reports, protocols, presentations, sketches, hardware, …) given feedback received feedback + recommended grades from peers participation and to be honest, a little logged time continuous observations by two teachers, independently

28. Grading criteria • For grade A you should also • Distinguish yourself in several of the above task areas and learning outcomes • Show special personal engagement, responsibility and initiative for the project and group work • For grade B you should also • Work actively with analysis, practical implementation, administration and communication • Clearly show that you reached the learning outcomes • For grade C you should also • Work in most of the fields analysis, practical implementation, administration and communication • Clearly show personal initiative and engagement in the course • For grade D you should also • Work with several types of tasks in the project • To some extent take on responsibilities in the course • Clearly show that you approach most of the intended learning outcomes • For grade E you should • Actively participate in the course seminars and project meetings • Actively participate in the course activities, read and answered emails from course leaders and delivered the course assignments • Spend time on task corresponding to 20 credits • Show that you approach the intended learning outcomes to a significant extent

29. Powerful principle 4:’for the good of the project’ • The projectand thegroup drives the specifications, the needs, the deadlines... not the teachers! • makes everything students do in the course meaningful, reporting comes natural for the first time • makes the course format sustainable and inexpensive

30. Let’s hear some student voices Interviews with students in the 2004 & 2005 cohorts(not the students in the picture...)

31. Interviewer: What did you learn about working in teams? You knew theories before, empty phrases. But now I have seen them in reality. These things are so easy to say. Like [...]. I mean, you don’t have to be a rocket scientist to realise that, everyone knows it. But it’s one thing to know and another thing to apply, and we really got first-hand experience from applying it. It is so obvious, you can stop anyone on the street and they would say ‘of course, everyone knows that’. But it is a completely different thing to experience it in reality.

32. Experience-based learning is necessary since If taught: the content/goals are perceived as obvious and trivial If taught: when a real situation occurs one might fail to recognise it but remember Experience only results in learning if reflected upon

33. Interviewer: So you chose not to switch project leader? “[Changing the project leader] wouldn’t have furthered the project. It could only have suffered. But if you completely drop [considerations for] the product - and maybe you should, actually – it might have furthered the course. It's hard to tell...you simply tend to put your focus on the product you are making.” Tension between project and learning...

34. Interviewer: How do you think this course could be improved? In the beginning I think there should have been some technical seminars to give a faster start of the project. Technical specialists who could have given a few lectures. To help you see possible designs for instance? Yes, technical solutions. And whom we could have contacted later with questions. Hmm. I wonder if you may risk the main idea of the course? Yes... that is a risk... If they say ‘this is what you should do’... Yes, you are right. I can see that it’s been painful though. Yes, but maybe that’s what is good for us. But you think it would have been better with a more efficient start. Yes, but that is perhaps because it had led to a better end result, I mean the boat. But maybe the learning wouldn't... Tension between project and learning... Conceptions of teacher’s and student’s roles are challenged...

35. Interviewer: What was it like to apply knowledge from previous courses? I think some [of my fellow students] feel that we haven’t used much of our previous knowledge. Because it’s not like exam tasks, it’s not extremely difficult, but you have to think more widely. We are building this thing in parts that have to work together and as I see it that is pretty advanced [engineering]. And it is [application of knowledge from several other courses]. Here we use mostly the fundamentals, but it is advanced to apply them, the situations are difficult because it’s real problems. In the previous courses I had to learn a lot of advanced stuff, but never how to use them. Previous course experiences create a narrow view on knowledge...

36. They should have been more like teachers. We had to do all the hard work ourselves and we don’t feel that we got as much help from the teachers as we could have had. [...] When we went and asked them ‘does this look alright’, they tried to answer as vaguely as they could. Just because they tried to make us solve things ourselves I think. Interviewer: How do you think this course could be improved? Student’s views on knowledge are challenged... Conceptions of teacher’s and student’s roles are challenged...

37. Quote from a mid-course evaluation Not that these were the only calculations needed, but the only ones that could be made. All the calculations assuming kinematic equilibrium seem to give various degrees of unreasonable results. This is not just a pity and shame, but it is also terribly bad pedagogy now towards the end of an education. I would really have liked to see that the theory we have learnt was possible to use. We cannot even calculate the strength since everything is so tiny. Students with a dualistic view on knowledge are seriously challenged...

38. We notice that students’ conceptions of learning or attitude towards knowledge is challenged in students’ eyes, learning is often overshadowedby the project per se The teacher or course will often be blamed, as students think the teacher should have saved themfrom the inconvenience. But relevant challenges are not ”flaws” that should be eliminated. They are key learning opportunitiesand we have no intention to protect the students from them. It is then not appropriate to behave in conformity with student expectations. But knowing they exist was valuable... Student views must always be interpreted

39. Lessons learned • Peer assessment is delicate to compose – not enough to use the right instruments, one has to play them carefully too • We keep students informed and invite them to discuss aspects of the course – they need to be able to trust the process

40. Powerful principle 1:the purpose is student learning • As opposed to reaching project goals(but note that the project still drives learning & creates a motivational context) • As opposed to teacher popularity, or giving students an ’easy ride’ (but note that students must still have trust in the process)

41. The beautiful sound of students growing into engineers... The greatest thing I have learned from this course is humility. I'll approach similar tasks more humbly in the future. We thought we were better than we were. No, not better, but we have taken courses with well-defined problems, where there is an answer, the key. And that went well. But now you realised that as soon as you are confronted with reality, it’s quite another story.

42. ...and more of the same... “At the beginning of the course I was somewhat worried about finishing the education and starting to work as an engineer. Those worries are gone now. My confidence in approaching technical problems and solving them has grown a lot.” “Feedback was exchanged on everything between napkin scribbles at lunch to things you had built. This was valuable since it both gave me, and trained me to give, critique. It also helped me to see how other people are thinking and how they solve problems.” “One of the best things during the project was that written documentation was called for and that we in much lived up to those demands. It allows you to cross check things and check the work of yourself and others, and things are always available.”

43. Powerful principles: • The purpose is student learning • Feedback and reflection on experience • Reversing the ’burden of proof’ • ’For the good of the project’

44. INTEGRATED LEARNING • The key idea of CDIO is that engineering skills and disciplinary knowledge are interdependent and should be learned together. • The teamwork and communication skills should be so strongly interwoven with the students’ application of technical knowledge, that the two are inseparable.

45. WHAT WE ASSESS IS WHAT WE GET • It is the assessment system that constitutes the real learning objectives to the students (rather than the espoused learning objectives). • Assessment is the most powerful tool we have to guide and support student learning. • Engineering skills must be given proper attention in the assessment system, as legitimate and required learning outcomes of the course.

46. LET’S ADD UP + WHAT WE ASSESS IS WHAT WE GET INTEGRATED LEARNING • THEREFORE • We will not get integrated learning if we separate disciplinary knowledge and engineering skills in assessment. • In order to fully communicate the intention to the students they must be assessed together. • CONCLUSION • Integrated learning calls for integrated assessment.

47. REFERENCES - Edström, El Gaidi, Hallström and Kuttenkeuler (2005).Integrated assessment of disciplinary, personal and interpersonal skills - student perceptions of a novel learning experience, Proceedings of the 13th Improving Student Learning, OCSLD, Oxford, UK. - Hallström, Kuttenkeuler and Edström (2007). The route towards a sustainable design-implement course, Proceedings of the 3rd CDIO Conference, Cambridge, MA.

48. Formulating objectives Designingassessment Designing activities CONSTRUCTIVE ALIGNMENT What should the student be able to do as a result of the course? What work should the student do, to reach the objectives? What should the student do, to demonstrate that they reached the objectives?

49. How important is assessment?”If we want to discover the truth about an educational system, we must first look to its assessment procedures.”[Rowntree 1987]”It is not the curriculum that shapes assessment, but assessment which shapes the curriculum.”[Brown & Knight, 1994]”Students can escape bad teaching; they can’t avoid bad assessment.”[Boud 1994]