RIS Research Consultation Outcome Survey Report to Public Services Council on the Pilot January 19, 2006

1. RIS Research Consultation Outcome Survey Report to Public Services Council on the Pilot January 19, 2006 This report, the web questionnaire and related documents are posted on Data Farm | Shared File Depository [http://metrics.library.upenn.edu/prototype/sfd]

2. Purposes of this pilot Attempt to gauge the impact of research service on student research skills Measure student perception of service effectiveness Design/test a questionnaire and an automated procedure for delivering it Procedures Commence survey process after mid-term, as students begin to hone in on research projects. Request student’s permission to do a follow up survey after the consultation. With consent, request email address Include subjects’ emails in Data Farm reports of sessions Two weeks (later one week) after sessions, start Data Farm process to pole eligible subjects via email and web questionnaire Dump survey inputs into a table linked to Research/Instruction database (NB: The poling is a convenient way of collecting student feedback, not a scientific sampling of the student body.)

3. Each day, DF scans RI db for Van Pelt entries that are 14 days old Van Pelt staff record session, with email into DF using consultation form Students complete survey on the web Extract email address, patron name, staff name and form_id from RI db Data Farm RI db Data Farm Survey db Insert names and survey url into email text and send PK form_id Survey Process

4. Invitation Dear Jane, In an effort to learn if the Penn Library is doing its best to help students with their research, I am writing to ask if you would be willing to complete a brief (4 Question) survey about your recent appointment with Joe Holub, on October 19,2005. We are interested to know if you found the session useful and if the information you learned will be helpful for future projects. Please click on this link to reach the questionnaire: http://datafarm.library.upenn.edu/surveyForm.html?formid=$form_id Any additional comments you add to the survey will be appreciated. If you have any questions, or if I can be of any further assistance, please feel free to be in touch. With thanks on behalf of the Penn Library, Marjorie Hassen Assistant Director,Research and Instructional Services Van Pelt Library hassen@pobox.upenn.edu (215) 898-2817

5. Web Survey

6. What We Asked • 1. We would like to know if the information covered in the session (for example, research tools, search strategies or specific resources) helped you with your research project? Do you feel the session was... • 1) very helpful 2) helpful 3) not too helpful 4) of no value • If the session was of little or no help, please tell us why, in a few words. 2. We would like to know if the session provided you knowledge or abilities that could be useful in the future. As a result of the session do you feel better prepared to… Identify resources relevant to a field of study Construct a search to find reseach material on a topic Evaluate information I use in my resarch Develop and produce a bibliography for a project Carry out future research effectively on my own Scheduling was convenient? If none of the preceding cases apply, tell us about skills you might have acquired. 3. Was the scheduling of the session convenient? 4. How can our research services better meet student needs?

7. Student Response (from Nov.2 – Dec. 20) 127 Persons Surveyed 57 Responses [45%]

8. What the Students Said Evaluation of Service Ugradn=39Gradn=18Alln=57 Q. How helpful was the session? Avr. Score [Very helpful (1)…of no value (4)] 1.31.1 1.2 Q. Was the scheduling convenient?” pct.who said yes95%100%97% Perceived Outcome of the Service All n=57 Q. Do you feel better prepared to… pct. who checked a. Identify resources relevant to a field of study 81% b. Construct a search to find research material on a topic 86% c. Evaluate information I use in my research 23% d. Develop and produce a bibliography for a project 28% e. Carry out future research effectively on my own 72%

9. Identify resources relevant to a field of study Outcome: “I feel better prepared to…” Undergrad (39) Grad (18) Carry out future research effectively on my own Construct a search to find research material on a topic Pct. who checked Develop and produce a bibliography for a project Evaluate information I use in my research

10. Gauging Impact

11. Summary Conclusions • Response was higher than expected (45% rate, during the busiest weeks of the term) • No “dissatisfied” customers answered the survey. Service was highly rated as helpful • Everyone who answered perceived at least one positive outcome from the session • About 90% of the respondents identified more than one outcome from the session • “I’m better prepared to construct searches” was the most frequently identified outcome • “I’m better prepared to evaluate information used in research” was the least frequent outcome • Tests validated the reliability of the automated survey procedure, notwithstanding potential problems with email (e.g. spam-filtering) • Twenty-two surveys (39%) contained comments—mostly praise and some suggestions for improvements

12. They [research services] are wonderful as is. Very prompt response to original email. Helpful, personalized service. I think that you do a good job, thanks! I still feel a overwhelmed by accessing many of the resources and knowing what the library provides,but Patty helped me figure out where to start with my searches and some helpful resources to use for my area of research. A lot of the sources Mr.Holub and I found were not available at Penn, so we had to place orders on inter-library loan. I ended up receiving only ONE of those orders before my paper was due. Give a sheet of codes for searching the databases I would just try and publicize the service more, I had no idea it existed. I would like to see more of your research services advertised to the student body. If possible, electronic appointment scheduling might be helpful. Longer sessions as needed. Let student do the work while reference expert watches. Work through the research process more slowly with the student Cookies? Comments

13. Next Steps