Project Evaluation Student Feedback

1. FDTL Phase 3 Project Based Learning in Engineering Project Evaluation& Student Feedback Katharine Carter PBLE Internal Evaluator University of Nottingham Teaching Enhancement Officer, Centre for Teaching Enhancement My role has been to work collaboratively with the project team and consortium Academics in formulating evaluation questions, raising awareness of developments in teaching and learning and ensuring smooth communication between the different sections of the projectMy role has been to work collaboratively with the project team and consortium Academics in formulating evaluation questions, raising awareness of developments in teaching and learning and ensuring smooth communication between the different sections of the project

2. FDTL Phase 3 Project Based Learning in Engineering Presentation Outline Expectations & Strategy formulation: the need for student centred evaluation Outline of method and methodology Results & interpreting qualitative data Possible future project action This session may have a slightly different angle to those previous. I start not with a particular aspect of running a PBL module, or even a good example of success from within our consortium , but with the PBLE project as a whole and some expectations placed on the project team by the FDTL National Co-ordination team in planning and implementing a focussed, balanced and structured evaluation strategy. We perceived a need for a student centred evaluation stream within our overall strategy. Implementation and collation of this work has formed a major part of my work over this academic year for the project. Like curriculum and skills development activity at modular level, the results of this work should be viewed in conjunction with other available evaluation and management information (for example?). So I start briefly with strategy formulation, and then move on logically through to outline methodology, and will then stay a little longer on results, and their implications.This session may have a slightly different angle to those previous. I start not with a particular aspect of running a PBL module, or even a good example of success from within our consortium , but with the PBLE project as a whole and some expectations placed on the project team by the FDTL National Co-ordination team in planning and implementing a focussed, balanced and structured evaluation strategy. We perceived a need for a student centred evaluation stream within our overall strategy. Implementation and collation of this work has formed a major part of my work over this academic year for the project. Like curriculum and skills development activity at modular level, the results of this work should be viewed in conjunction with other available evaluation and management information (for example?). So I start briefly with strategy formulation, and then move on logically through to outline methodology, and will then stay a little longer on results, and their implications.

3. FDTL Phase 3 Project Based Learning in Engineering Purpose Obtain base line information about the context and perceptions of PBL This has been one strand of the overall project evaluation strategy, which has a distinct focus on formative evaluation and qualitative data gathering in year one. The idea behind this has been to provide Academics working within the consortium with short-cycle developmental support that will feed directly into project activitiesThis has been one strand of the overall project evaluation strategy, which has a distinct focus on formative evaluation and qualitative data gathering in year one. The idea behind this has been to provide Academics working within the consortium with short-cycle developmental support that will feed directly into project activities

4. FDTL Phase 3 Project Based Learning in Engineering The PBLE bid: �Initially the project evaluation will be formative and will focus on the developmental processes involved�� Working with students often focuses on gathering and using qualitative, rather than quantitative information, which has dovetailed neatly with the formative nature of this early stream of evaluation activity. It is well known that project work is an important aspect of engineering courses1. Modules used in the student centred evaluation stream ranged across the full undergraduate intake (1st to final year). The results discussed later were obtained from a student learning diary used like a questionnaire and returned from groups of students over a time period of April to June 2001 (ie Semester two) when students had adjusted to the requirements of their learning year. One of the main reasons why this work is needed is that we expect the results of this project to include improved performance and awareness from graduating engineers who enter industry and are judged by their employers. We recognise the need to include students within our stakeholder group and have sought to include them as such right from year one. Our focus here was on perceptions and early involvement of students representing the broad range of Project Based Learning experience across the consortium. Other evaluation streams were concerned with project compliance and quantitative data. Working with students often focuses on gathering and using qualitative, rather than quantitative information, which has dovetailed neatly with the formative nature of this early stream of evaluation activity. It is well known that project work is an important aspect of engineering courses1. Modules used in the student centred evaluation stream ranged across the full undergraduate intake (1st to final year). The results discussed later were obtained from a student learning diary used like a questionnaire and returned from groups of students over a time period of April to June 2001 (ie Semester two) when students had adjusted to the requirements of their learning year. One of the main reasons why this work is needed is that we expect the results of this project to include improved performance and awareness from graduating engineers who enter industry and are judged by their employers. We recognise the need to include students within our stakeholder group and have sought to include them as such right from year one. Our focus here was on perceptions and early involvement of students representing the broad range of Project Based Learning experience across the consortium. Other evaluation streams were concerned with project compliance and quantitative data.

5. FDTL Phase 3 Project Based Learning in Engineering Outline of Method: Student learning dairies 1. Identification and Analysis of key areas 2. Formulation of reflective questions 3. Structure of the learning diary 1. QAA code of practice, NCT guidelines, developments in L&T (skills +lg outcomes) 2. See list 3. Using CDELL template; principles reflective questioning 4. Each module has its own pattern & timetable, students have different needs & expectations.1. QAA code of practice, NCT guidelines, developments in L&T (skills +lg outcomes) 2. See list 3. Using CDELL template; principles reflective questioning 4. Each module has its own pattern & timetable, students have different needs & expectations.

6. FDTL Phase 3 Project Based Learning in Engineering Key Areas Internal & external (QAA) guidance on: 1. Assessment of students 2. Programme approval, monitoring & review 3. Supporting students with disabilities Other projects in learning & teaching: key skills Needs of employers of engineering graduates

7. FDTL Phase 3 Project Based Learning in Engineering Modules Mechanical Engineering Year one: Individual project Year Two: Group project, supported by third year mentors Final years: Industrially based group project Electrical Engineering Year two: individual project Civil Engineering Final years:group project

8. FDTL Phase 3 Project Based Learning in Engineering The Learning Diary The student learning diary template consists of ten sections, each relating to a broad topic area. For each individual module these topic areas remained almost the same, but these are each broken down into a number of reflective questions and these were amended to specifically relate to that week�s activity within the taught material of each module.. For example, each diary started in week one with �expectations�, and ended with a summary. Most also included (at about week three) a section on project title or team choice but this was not relevant within the first year or individual second year Project Based Learning modules worked with. We chose to employ a paper diary but an on-line template would be relatively easy to establish within some modules. Four of the five engineering module where diaries were introduced contained �design� in their title � since students are likely only to be motivated if they see value in what they are doing, the incorporation of �design� in these modules could be a motivator for students. Why a learning diary? Modules used in the student centred evaluation stream ranged across the full undergraduate intake (1st to final year). The results discussed in the next three slides were obtained from a student learning diary used like a questionnaire and returned from groups of students over a time period of April to June 2001 (ie Semester two) when students had adjusted to the requirements of their learning year. Other sections in the diary included: Support needed Choosing a project Transferable skills development Team work FeedbackThe student learning diary template consists of ten sections, each relating to a broad topic area. For each individual module these topic areas remained almost the same, but these are each broken down into a number of reflective questions and these were amended to specifically relate to that week�s activity within the taught material of each module.. For example, each diary started in week one with �expectations�, and ended with a summary. Most also included (at about week three) a section on project title or team choice but this was not relevant within the first year or individual second year Project Based Learning modules worked with. We chose to employ a paper diary but an on-line template would be relatively easy to establish within some modules. Four of the five engineering module where diaries were introduced contained �design� in their title � since students are likely only to be motivated if they see value in what they are doing, the incorporation of �design� in these modules could be a motivator for students. Why a learning diary? Modules used in the student centred evaluation stream ranged across the full undergraduate intake (1st to final year). The results discussed in the next three slides were obtained from a student learning diary used like a questionnaire and returned from groups of students over a time period of April to June 2001 (ie Semester two) when students had adjusted to the requirements of their learning year. Other sections in the diary included: Support needed Choosing a project Transferable skills development Team work Feedback

9. FDTL Phase 3 Project Based Learning in Engineering Questions Supporting Students: motivations, needs and perceptions of employability Developing a project: aims, learning outcomes & skills development Assessment: motivation, feedback & assessment

10. FDTL Phase 3 Project Based Learning in Engineering ResultsExample of Year One diary collation This is an example of some of the more valuable issues raised by students in collated learning diary results. This particular example is from a first year group project module and draws out some interesting points. Find that students and staff, despite coming from different perspectives, are often raising very similar issues.This is an example of some of the more valuable issues raised by students in collated learning diary results. This particular example is from a first year group project module and draws out some interesting points. Find that students and staff, despite coming from different perspectives, are often raising very similar issues.

11. FDTL Phase 3 Project Based Learning in Engineering Results (2)Year One, Group project, Department of Mechanical Engineering�Support Needed� diary section

12. FDTL Phase 3 Project Based Learning in Engineering Results (3)Year Three, industrially based group project Department of Mechanical Engineering�Support needed� section

13. FDTL Phase 3 Project Based Learning in Engineering Lack of FeedbackFinal year group project, Civil Engineering Introduction of the concept: a short focussed session with opportunity for discussion Timing: Early introduction, ideally within assessment credit weightings Academic culture: Students may resent introduction of new ways of working Length of document: Short, focussed, integrated We had one module where we experienced a lack of feedback from the diaries. This was within a final year group project module. We perceive that the reasons for this lack of feedback and "lessons" to be learnt are as follows: .Introduction of the concept needs to be smooth: Our diary was introduced during a project planning workshop with 'lecture' style presentation. Short, focussed session with opportunity for discussion to be employed next time. Timing: the above session was a number of weeks into semester. Final year students had already prioritised their work effort. Reflective diaries need to have a credit rating to guarantee use, especially in final year. .Academic culture: working in a reflective way is a new concept to many engineering students. Because it was additional to their workload some students commented that they resented being asked to work in such a different way. .Length of diary: the diary concept was introduced during a workshop session. Students were asked to take it away and fill it in progressively but many wanted to complete it in one session and were daunted by its length. Ideas for next session include: weekly question sheets; sections restricted in length; Amalgamation of some sections to make a shorter diary. We had one module where we experienced a lack of feedback from the diaries. This was within a final year group project module. We perceive that the reasons for this lack of feedback and "lessons" to be learnt are as follows: .Introduction of the concept needs to be smooth: Our diary was introduced during a project planning workshop with 'lecture' style presentation. Short, focussed session with opportunity for discussion to be employed next time. Timing: the above session was a number of weeks into semester. Final year students had already prioritised their work effort. Reflective diaries need to have a credit rating to guarantee use, especially in final year. .Academic culture: working in a reflective way is a new concept to many engineering students. Because it was additional to their workload some students commented that they resented being asked to work in such a different way. .Length of diary: the diary concept was introduced during a workshop session. Students were asked to take it away and fill it in progressively but many wanted to complete it in one session and were daunted by its length. Ideas for next session include: weekly question sheets; sections restricted in length; Amalgamation of some sections to make a shorter diary.

14. FDTL Phase 3 Project Based Learning in Engineering Conclusions Student�s motivations alter as they progress towards graduation Reflection needs reward in terms of marks For good project modules to fully meet student expectations there needs to be specific skills development support

15. FDTL Phase 3 Project Based Learning in Engineering Possible Future Project Action Concentrate on skills development, support & documentation across the consortium Revise and amend learning diary template Develop use of learning diary with Module convenors for specific purposes Final point: developing with Peter Wilmott, lg diary as part of assessment of 3rd year mentors for 2nd year module (teaching contract)Final point: developing with Peter Wilmott, lg diary as part of assessment of 3rd year mentors for 2nd year module (teaching contract)

16. FDTL Phase 3 Project Based Learning in Engineering Articulation of activities Working with students to record their perceptions of benefits and pitfalls of PBL, is additional to these broad areas covered by our overall evaluation strategy . National Co-ordination Team guidance to Project leaders draws on the principle that evaluation should ideally be carried out at points which are naturally occurring within the project timetable2. For example I used a confidential pro forma for staff, and discussed progress with academics at the end of Framework and Executive Committee meetings � no extra travel, meetings or correspondence generated. Working with students to record their perceptions of benefits and pitfalls of PBL, is additional to these broad areas covered by our overall evaluation strategy . National Co-ordination Team guidance to Project leaders draws on the principle that evaluation should ideally be carried out at points which are naturally occurring within the project timetable2. For example I used a confidential pro forma for staff, and discussed progress with academics at the end of Framework and Executive Committee meetings � no extra travel, meetings or correspondence generated.