PENNDOT’S LTAP EVALUATION ACTIVITIES

1. PENNDOT’S LTAP EVALUATION ACTIVITIES Kim M. Ferroni, PennDOT LTAP Project Manager Barbara T. Harder, B. T. Harder, Inc. LTAP Evaluations

2. PennDOT LTAP • PennDOT owner and project management • Contracted operations -- Pennsylvania State Association of Township Supervisors; since December 2005 • Sponsored during 2007 • Training workshops, 227 • Technical assistance sessions, 89 on-site (field), off-site 251 TRB Technology Transfer Committee Meeting

3. PennDOT LTAP Evaluation Activities • Training Workshops • “Happy Sheet” information – evaluation of the experience (facility, time, materials, instructor) • Self assessment of learning • Initiating pre- and post-testing • Follow-up survey TRB Technology Transfer Committee Meeting

4. PennDOT LTAP Evaluation Activities • Technical Assistance • Self assessment of learning • 6 month follow-up evaluation • General benefits assessment TRB Technology Transfer Committee Meeting

5. Training Workshops • Report some basic statistics • 3 workshops for illustration • Asphalt Roads – Common Maintenance Problems • Bridge Maintenance • Work Zone Traffic Control • Most frequent workshop offerings during 2 month period of 2007 (representative sample – about 60% of total) TRB Technology Transfer Committee Meeting

6. What Evaluation Sheets Tell Us • Based on rating of three specific learning goals for each workshop • 100% rating means “doubled” the knowledge TRB Technology Transfer Committee Meeting

7. Self Evaluation of Learning • Self Evaluation of Learning • Highest assessed learning reported in Work Zone Traffic Control Workshops • Understand requirements or WZTC in accordance with state and national guidelines (83%) • Able to develop traffic control plans using PennDOT publication (100%) • Understand recent changes in work zone rules (115%) 100% rating means “doubled” the knowledge TRB Technology Transfer Committee Meeting

8. Use of Training Information • 95% of workshop attendees reported that what they learned would improve job performance • 76% will use the information they learned within 6 months of the workshop • 9% use within 1 year, 13% unsure TRB Technology Transfer Committee Meeting

9. “Happy” Ratings TRB Technology Transfer Committee Meeting

10. Follow-up Training Workshop Surveys • Initiating use of Survey Monkey to determine follow-up actions (distributed 6 months after training) • Performance • Implementation • Learning • Do not yet have sufficient data to analyze TRB Technology Transfer Committee Meeting

11. Maintenance and Safety Technical Assistance • Technical assistance • Off-site – telephone, email • On-site • Maintenance and safety topics • Traffic signs • Engineering and traffic studies • Work zone traffic control • Guiderails, posting and bonding • Evaluation requested for every tech assist session – 40% response rate TRB Technology Transfer Committee Meeting

12. Maintenance & Safety Tech Assists Types of Evaluation Respondent Organizations June 06-Nov 07 County, 0, 0% TRB Technology Transfer Committee Meeting

13. Maintenance & Safety Tech Assists Frequency of Benefits Evaluations ReceivedJune 06-Nov 07 TRB Technology Transfer Committee Meeting

14. Maintenance & Safety Tech AssistsUsefulness of Information Evaluations Received June 06-Nov 07 • 80% of respondents anticipate using information provided in Tech Assists within 6 months and 91% within one year AND • 94% of Tech Assist information adequately addressed the issue of concern TRB Technology Transfer Committee Meeting

15. Use as exception flag for problems Provides opportunity for suggestions and other feedback Maintenance & Safety Tech Assists “Happy Ratings” June 06-Nov 07 * Maximum rating = 4 TRB Technology Transfer Committee Meeting

16. Maintenance & Safety Tech AssistsSummarized Comments Evaluations Received June 06-Nov 07 • The LTAP staff was (multiple respondents) …quick to respond, answered all my questions, help{ed to} educate me throughout several processes, very helpful, polite, and professional, excellent to work with, easy to talk with, an encyclopedia of traffic engineering and detailed issues… • The information provided was (multiple respondents) … understandable, {enabled us} to use the correct material for our signs, gave us confidence in knowing the correct way, delivered quickly… • We learned a lot. Written info will come in handy in the future also. Thank you for providing this type of assistance. TRB Technology Transfer Committee Meeting

17. Technical Assistance Follow-up • Follow-up evaluation for on-site visits, 22.6% response rate • “Increase Safety” and “Do Job Right Way” are leader benefits after 6 months TRB Technology Transfer Committee Meeting

18. Types of Benefits • Quantitative Benefits • 10 of 34 Benefits Evaluations provided some quantitative information (29%) • Candidates for more in-depth assessment • Qualitative Benefits • 32 of 34 Benefits Evaluations provided some type of qualitative information (94%) TRB Technology Transfer Committee Meeting

19. Example Quantitative Benefits • Engineering & Traffic Studies • Saved $10,200 – did not have to do a traffic study • Traffic Signs & Guiderails • 30 hours saved/month; $11.50 rate/hour • Saved the cost of putting guiderails at the wrong places and the time to put up the correct signs TRB Technology Transfer Committee Meeting

20. Example Qualitative Benefits • Risk reduced for walking and motoring public. Liability risk reduced for Borough and school district. Safety increased significantly. • The project was beneficial to establishing better working relationship with the township. • Better or cleaner stone for chip roads. Trying to make our tar & chip projects last longer. • Avoid possible law suits; made drivers more aware of upcoming intersection. TRB Technology Transfer Committee Meeting

21. Questions and Issues for Discussion • Training workshop pre- and post-testing – what works, what doesn’t? • Literacy issues • Voluntary or mandatory • When used • Learning assessment from supervisors • What experiences have others had? • Response rates • Effort required • Benefits measurements – what programs are doing this and how? • Who is providing the information? TRB Technology Transfer Committee Meeting

22. For More Information • Kim M. Ferroni – PennDOT LTAP Project Manager, kferroni@state.pa.us • Carol Kilko – Contract Manager, PSATS, ckilko@psats.org • Barbara T. Harder – B. T. Harder, Inc. Contract Staff, Evaluations, btharder@verizon.net TRB Technology Transfer Committee Meeting