Balancing Performance Measures in an Academic Setting

Balancing Performance Measures in an Academic Setting PowerPoint PPT Presentation


  • 188 Views
  • Uploaded on
  • Presentation posted in: General

Purpose. To provide a data-based description of the processes and outcomes associated with the behavioral system analysis of the Behavior Analysis Program at UNR. Discuss certain process related elements that are unique to an academic. Plans for ongoing enhancement.. INPUT(Students). PROCESS(Graduate Training).

Download Presentation

Balancing Performance Measures in an Academic Setting

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


1. Balancing Performance Measures in an Academic Setting Ramona Houmanfar, Jared Chase, Charna Mintz-Resudek, Thomas Boyce, Patrick M. Ghezzi, Linda J. Hayes, Michele D. Wallace & Larry Williams University of Nevada, Reno Good morning. The title of this presentation is “Balancing Performance Measures in an Academic Setting. I would like to take the time to thank my co-authors, Charna Mintz-Resudek, Thomas Boyce, Patrick M. Ghezzi, Linda J. Hayes, Michele D. Wallace & Larry Williams , and my translator, Fernando Guerrero who is a doctoral student in our graduate program at UNR. Good morning. The title of this presentation is “Balancing Performance Measures in an Academic Setting. I would like to take the time to thank my co-authors, Charna Mintz-Resudek, Thomas Boyce, Patrick M. Ghezzi, Linda J. Hayes, Michele D. Wallace & Larry Williams , and my translator, Fernando Guerrero who is a doctoral student in our graduate program at UNR.

2. Purpose To provide a data-based description of the processes and outcomes associated with the behavioral system analysis of the Behavior Analysis Program at UNR. Discuss certain process related elements that are unique to an academic. Plans for ongoing enhancement. Many academic institutions promote a balance between quality and quantity for their academic products. However, subjectivity in measuring and comparing the quality of academic and professional products is particularly challenging. Further, performance criteria and standards vary between and within academic and professional activities.   In order to maximize its entrepreneurial success while enhancing its academic performance system, the Behavior Analysis Program at the University of Nevada, Reno has implemented Abernathy’s Total Performance System. This paper provides a data-based description of the processes and outcomes associated with the TPS in this Program. A number of problems and solutions unique to an academic environment are discussed along with future plans.   Many academic institutions promote a balance between quality and quantity for their academic products. However, subjectivity in measuring and comparing the quality of academic and professional products is particularly challenging. Further, performance criteria and standards vary between and within academic and professional activities.   In order to maximize its entrepreneurial success while enhancing its academic performance system, the Behavior Analysis Program at the University of Nevada, Reno has implemented Abernathy’s Total Performance System. This paper provides a data-based description of the processes and outcomes associated with the TPS in this Program. A number of problems and solutions unique to an academic environment are discussed along with future plans.  

3. By taking a behavioral system perspective, we believe that in order for us to survive and grow, we need to consider the potential trainees (i.e., students), training resources (e.g., faculty, curriculum, funding), the effectiveness of our training practices, and receivers of our services (organizational community) within the context of our academic and professional communities and competitors. And, we need to meet the community demand in an effective manner. Effective means that we need to focus on the speed, quality, and quantity of training which is the number of graduates trained per unit of time (e.g., annually).   By taking a behavioral system perspective, we believe that in order for us to survive and grow, we need to consider the potential trainees (i.e., students), training resources (e.g., faculty, curriculum, funding), the effectiveness of our training practices, and receivers of our services (organizational community) within the context of our academic and professional communities and competitors. And, we need to meet the community demand in an effective manner. Effective means that we need to focus on the speed, quality, and quantity of training which is the number of graduates trained per unit of time (e.g., annually).  

4. Background Information Unique Graduate Program Self-capitalized model of training Important need to arrange contingencies to balance entrepreneurial and academic activities Collaborative effort with Abernathy and Associates TPS balances entrepreneurial and academic activities Our program is unique in that it is not funded through the University. Rather, faculty must create businesses to support themselves. This provided the opportunity for students to gain practical experience as employees of the program. However, it is important to arrange contingencies in a manner that balances this practical experience with academic activities. We are currently collaborating with Abernathy and Associates in an effort to develop these contingencies through the use of a total performance systemOur program is unique in that it is not funded through the University. Rather, faculty must create businesses to support themselves. This provided the opportunity for students to gain practical experience as employees of the program. However, it is important to arrange contingencies in a manner that balances this practical experience with academic activities. We are currently collaborating with Abernathy and Associates in an effort to develop these contingencies through the use of a total performance system

5. Mission Statement To produce skilled behavior analysts who meet the needs of the community and contribute to the vitality of Behavior Analysis as a scientific discipline in a socially valid and cost effective manner. The mission statement of the behavior analysis at the University of Nevada Reno is to strengthen behavior analysis as a scientific discipline and as a professional occupationThe mission statement of the behavior analysis at the University of Nevada Reno is to strengthen behavior analysis as a scientific discipline and as a professional occupation

6. Inputs: students admitted Let me orient you to the graph. Along the x-axis is the school year and along the y-axis is the number of students admitted each year. The blue bars represent the master’s students admitted and the red bars represent the number of PhD students. This graph demonstrates that over time, we have increased the overall number of students admitted to the program. Let me orient you to the graph. Along the x-axis is the school year and along the y-axis is the number of students admitted each year. The blue bars represent the master’s students admitted and the red bars represent the number of PhD students. This graph demonstrates that over time, we have increased the overall number of students admitted to the program.

7. Inputs: Total Students and Faculty Along the x axis is the school year and along the y axis is the number of students attending UNR in a given year. The red bars represent doctoral students, the blue bars represent masters students and the white bars represent full time faculty. The data indicate that the total number of doctoral students attending the program has increased each year while the number of masters students has increased until 97 and then decreased. In addition, as the number of students attending the program has increased substantially, the total number of full time faculty has only increased slightly.Along the x axis is the school year and along the y axis is the number of students attending UNR in a given year. The red bars represent doctoral students, the blue bars represent masters students and the white bars represent full time faculty. The data indicate that the total number of doctoral students attending the program has increased each year while the number of masters students has increased until 97 and then decreased. In addition, as the number of students attending the program has increased substantially, the total number of full time faculty has only increased slightly.

8. Total Performance System (TPS) TPS is an organization wide incentive-pay system based on targeted, objective, personal performance measures (Abernathy, 1999) It uses balanced performance scorecards that allow for the objective measurement of performance across an organization (Abernathy, 1996) Cascading scorecards tie the performance of the individual employee to each level of the organization as well as to the overall objectives of the organization The total performance system (as developed by Abernathy) is an organization wide incentive pay system. It relies on scorecards that evaluate objective performance measures across an organization In addition, the scorecards cascade in an effort to tie the performance of all employees to the overall objectives of an organization.The total performance system (as developed by Abernathy) is an organization wide incentive pay system. It relies on scorecards that evaluate objective performance measures across an organization In addition, the scorecards cascade in an effort to tie the performance of all employees to the overall objectives of an organization.

9. Organizational Critical Measures BA Program At UNR Strategic Scorecard Measure Weight Revenue 20% Expense/Revenue 20% Cash Flow 10% Academic Progress 20% Professional Performance 20% Consumer Satisfaction 10% The strategic scorecard at UNR is made up of 6 measures: revenue, expense over revenue, cash flow, academic progress, professional performance, and consumer satisfaction. These measures are weighted according to overall value to the organizationThe strategic scorecard at UNR is made up of 6 measures: revenue, expense over revenue, cash flow, academic progress, professional performance, and consumer satisfaction. These measures are weighted according to overall value to the organization

10. Academic Scorecard Measures Critical Measures Grade Point Average Progress Toward Completion Program Participation Professional Development Research Productivity Our academic score card is divided into 5 critical measures. Each of these critical measures is again divided into driver measures. Go through each: Fernando: Please translate my explanation for each category one at a time. I will use the same words that are indicated on the slide. Our academic score card is divided into 5 critical measures. Each of these critical measures is again divided into driver measures. Go through each: Fernando: Please translate my explanation for each category one at a time. I will use the same words that are indicated on the slide.

11. Academic Scorecard Measures Critical Measures Grade Point Average Progress Toward Completion Program Participation Professional Development Research Productivity

12. Academic Scorecard Measures Critical Measures Grade Point Average Progress Toward Completion Program Participation Professional Development Research Productivity

13. Academic Scorecard Measures Critical Measures Grade Point Average Progress Toward Completion Program Participation Professional Development Research Productivity

14. Academic Scorecard Measures Critical Measures Grade Point Average Progress Toward Completion Program Participation Professional Development Research Productivity

15. Data collection Student and advisor meet at the beginning of the semester to set goals Student and advisor meet in the middle and at the end of the semester to review progress on previously set goals Data is collected by each faculty member via semester based meetings with their students. Students meet at the beginning of the semester to set goals and at the end of the semester to review these goals. Data is then turned in to the TPS team for data entry and analysisData is collected by each faculty member via semester based meetings with their students. Students meet at the beginning of the semester to set goals and at the end of the semester to review these goals. Data is then turned in to the TPS team for data entry and analysis

16. Data collection Academic progress scores are calculated for each student (score = goals met / goals set) Advisement scores are calculated for each advisor (score = avg. score of students advised) For each critical measure a score is generate by dividing the number of goals met by the number of goals set The scores from each critical measure are added together to calculate an overall academic score These scores are then averaged across each faculty’s students For each critical measure a score is generate by dividing the number of goals met by the number of goals set The scores from each critical measure are added together to calculate an overall academic score These scores are then averaged across each faculty’s students

17. Driver Measures: Progress Toward Completion This example depicts the critical measure: Progress toward completion In this example, two goals were set, a thesis proposal and a comprehensive exam However, only the thesis proposal goal was achieved Therefore 1 out of 2 goals were met that resulted in a score of .50 or 50%This example depicts the critical measure: Progress toward completion In this example, two goals were set, a thesis proposal and a comprehensive exam However, only the thesis proposal goal was achieved Therefore 1 out of 2 goals were met that resulted in a score of .50 or 50%

18. Key Academic Measures Once a score is calculated for each critical measure, they are multiplied by their assigned weights and added together In this example, the progress toward completion score of .50 is multiplied by the weight of . 25 to come up with an adjusted score of .13.Once a score is calculated for each critical measure, they are multiplied by their assigned weights and added together In this example, the progress toward completion score of .50 is multiplied by the weight of . 25 to come up with an adjusted score of .13.

19. Cascading System The students scores under each advisor are averaged to generate an academic score for that advisorThe students scores under each advisor are averaged to generate an academic score for that advisor

20. Academic Progress Current Data Along the x-axis is the semester and along the y-axis is the academic progress score for the BA program. Currently these data are on a decreasing trend.Along the x-axis is the semester and along the y-axis is the academic progress score for the BA program. Currently these data are on a decreasing trend.

21. Professional Performance Measure A majority of students work for the BA program Faculty supervisors evaluate their performance on a semester basis Students are evaluated in two areas Productivity Quality In addition to an academic scorecard, students are also rated on professional performance. As mentioned earlier, a majority of students are funded by one of the current BA businesses. At the end of the semester students are evaluated on a liker scale in both productivity and quality These scores are then averaged for each faculty memberIn addition to an academic scorecard, students are also rated on professional performance. As mentioned earlier, a majority of students are funded by one of the current BA businesses. At the end of the semester students are evaluated on a liker scale in both productivity and quality These scores are then averaged for each faculty member

22. Critical Measure: Productivity Students are expected to earn 100% in all productivity measures These driver measures include Task completion Timeliness of task completion Timeliness to meetings Meetings attended Students earn a percentage score at the end of the semester Specifically, students are evaluated on 4 measures in the area of productivity. These include task completion, timeliness of task completion, timeliness to meetings and meeting attended. Students are scored in each of these areas out of 100% These 4 measures are then added together and divided by 400 to generate a productivity scoreSpecifically, students are evaluated on 4 measures in the area of productivity. These include task completion, timeliness of task completion, timeliness to meetings and meeting attended. Students are scored in each of these areas out of 100% These 4 measures are then added together and divided by 400 to generate a productivity score

23. Critical Measure: Quality Students are rated on a scale from 1-4 It is expected that a student earn 4s in all measures These driver measures include Satisfaction of task completion Interactions with clients Interactions with co-workers Interactions with the supervisor Level of independent task completion Student are evaluated on on a scale from 1-4 on 5 measures in the area of quality. These include satisfaction of task completion, interactions with clients, interactions with co-workers, interactions with supervisors, and level of independence These scores are then added together and divided by 20 to get a quality score. Student are evaluated on on a scale from 1-4 on 5 measures in the area of quality. These include satisfaction of task completion, interactions with clients, interactions with co-workers, interactions with supervisors, and level of independence These scores are then added together and divided by 20 to get a quality score.

24. Weight of Measures The productivity score and the quality score are equally weighted. Each score is multiplied by .50 and then they are added together to generate a professional performance scoreThe productivity score and the quality score are equally weighted. Each score is multiplied by .50 and then they are added together to generate a professional performance score

25. Professional Performance Current Data Along the x axis is the semester and along the y axis is the possible score. Currently these data are on an upward trend.Along the x axis is the semester and along the y axis is the possible score. Currently these data are on an upward trend.

26. Consumer Satisfaction Measure This measure evaluates student satisfaction with advisement and supervision Students rate advisors and supervisors on a 4 point scale in a variety of areas Ratings are averaged together resulting in a consumer satisfaction score A third measure on faculty score cards in that of consumer satisfaction Students rate advisors and supervisors on a 4 point scale in a variety of areas. Ratings are averaged together in the categories of advisement and supervision and then these categories are averaged together to generate a consumer satisfaction scoreA third measure on faculty score cards in that of consumer satisfaction Students rate advisors and supervisors on a 4 point scale in a variety of areas. Ratings are averaged together in the categories of advisement and supervision and then these categories are averaged together to generate a consumer satisfaction score

27. Critical Measures Faculty are evaluated in the following areas Adherence to BA program protocols Timeliness Accessibility Advisement Supervision Professional Interactions Some of the areas evaluate include adherence to BA program protocols, timeliness, accessibility, advisement, supervision, and professional interactionsSome of the areas evaluate include adherence to BA program protocols, timeliness, accessibility, advisement, supervision, and professional interactions

28. Consumer Satisfaction Current Data Along the x-axis is the semester and alone the y-axis is the score This graph represents the overall program consumer satisfaction data or the average of all faculty scores These scores have remained relatively highAlong the x-axis is the semester and alone the y-axis is the score This graph represents the overall program consumer satisfaction data or the average of all faculty scores These scores have remained relatively high

29. Outputs: Graduate Alumni 17/20 M.A. Students (1990-2000) received degree. 15/38 Ph.D. Students (1990-1997) received degree. 85% of students admitted to the masters program earned their degree between 1990 and 2000 39% of all doctoral students earned their PhDs between 1990 and 199785% of students admitted to the masters program earned their degree between 1990 and 2000 39% of all doctoral students earned their PhDs between 1990 and 1997

30. Outputs: Revenue Along the x-axis is the school year and along the y-axis is dollars. The red bars represent gross income and the blue bars represent net income Gross income has consistently increased over the years while net income has been variable. The estimated revenue for the program for Fiscal Year 2003 is $1,124,014.70. Along the x-axis is the school year and along the y-axis is dollars. The red bars represent gross income and the blue bars represent net income Gross income has consistently increased over the years while net income has been variable. The estimated revenue for the program for Fiscal Year 2003 is $1,124,014.70.

31. Alumni Feedback 7 of the 15 doctoral alumni responded to an on-line questionnaire Current placements (of 7 respondents) 4 academic, 1 clinical consultant, 1 internal consultant, 1 clinical research 7 of 7 respondents currently in desired work environment. Initial placements 6 of 7 in desired work environment A survey of doctoral alumni was conducted 7 of 15 participants responded All participants are currently employed in desired environments and 6 of the 7 were initial employed in desired placements 4 participants were in academic placement, 2 were employed as consultants and 1 was employed as a clinical researcherA survey of doctoral alumni was conducted 7 of 15 participants responded All participants are currently employed in desired environments and 6 of the 7 were initial employed in desired placements 4 participants were in academic placement, 2 were employed as consultants and 1 was employed as a clinical researcher

32. Processes: Alumni Feedback – Most Important Training Experiences (7 respondents) This graph represents those areas that the respondents felt were the most important in terms of their training experience Data collection and analysis were selected by all 7 participants as an important training area 5 of the 7 participants stated that project management and research were also important areas of training Internships, grant writing, teaching and professional development were only deemed important by 1 participantsThis graph represents those areas that the respondents felt were the most important in terms of their training experience Data collection and analysis were selected by all 7 participants as an important training area 5 of the 7 participants stated that project management and research were also important areas of training Internships, grant writing, teaching and professional development were only deemed important by 1 participants

33. Processes: Alumni Feedback – Weakest areas of training (7 respondents) This graph depicts the areas determined to be the weakest in terms of training. 3 of 7 participants felt that their training in grant writing was weakThis graph depicts the areas determined to be the weakest in terms of training. 3 of 7 participants felt that their training in grant writing was weak

34. Competitors Academic BA programs, IO Programs, Business Management, Education, Rehabilitation Business Human service providers and consultation firms in Nevada & Northern California The BA program’s current competition can be divided into two categories: academic and business. Academic competition includes other BA, IO, business management, education and rehabilitation programs Business competition includes human service providers and consultation firms in Nevada and CaliforniaThe BA program’s current competition can be divided into two categories: academic and business. Academic competition includes other BA, IO, business management, education and rehabilitation programs Business competition includes human service providers and consultation firms in Nevada and California

35. The Receiving End of the System Distribution of a survey among established placement sites (in progress) We are currently conducting a survey of placement sites for students who have graduated from the BA program at UNR The goal is to use the feedback for potential curriculum enhancement  The survey will gather the following information:             1. Employer’s feedback regarding with skill acquisition a. What they have looked for? b. Have we provided the product (alumni who are effective in providing what they have looked for)?             2. Employers’ updated information regarding available internship opportunities that includes specific criterion, funding, etc.              3. Employers’ updated information regarding employment opportunities . We are currently conducting a survey of placement sites for students who have graduated from the BA program at UNR The goal is to use the feedback for potential curriculum enhancement  The survey will gather the following information:             1. Employer’s feedback regarding with skill acquisition a. What they have looked for? b. Have we provided the product (alumni who are effective in providing what they have looked for)?             2. Employers’ updated information regarding available internship opportunities that includes specific criterion, funding, etc.              3. Employers’ updated information regarding employment opportunities .

36. Discussion The process has been long but successful Further development of the project Fine tuning of TPS measures to bring the current system at UNR closer to the ideal model Software installment for TPS Graduate programs can benefit from the implementation of the behavioral system analysis technology including the TPS model Although the process of evaluating our program from a systems perspective has been tedious, it has been successful We are currently fine tuning TPS measures to bring the UNR model more in line with Abernathy’s system This is being accomplished through a software system which will manage the database and generate TPS reports We believe that graduate programs can benefit from the implementation of a behavioral systems analysis, such as TPS.Although the process of evaluating our program from a systems perspective has been tedious, it has been successful We are currently fine tuning TPS measures to bring the UNR model more in line with Abernathy’s system This is being accomplished through a software system which will manage the database and generate TPS reports We believe that graduate programs can benefit from the implementation of a behavioral systems analysis, such as TPS.

  • Login