From an art to a science…

How to Select Students Likely to Pursue PhD ProgramsRick McGee, PhDAssociate DeanFaculty Recruitment and Professional DevelopmentNorthwestern University, Feinberg School of Medicine Director, IMSD/CLIMB Program From an art to a science…

How DO we predict the future – i.e. select students who will start and complete a PhD? • Evidence of appropriate skills and abilities • Evidence of appropriate motivation • Nature of the motivational goals? • Internal vs. external motivations? • Do we reward past experiences and accomplishment or really attempt to predict the future? Are they the same? • Differences between medical and graduate school… • How do we assess the past and predict the future differently for students from atypical life experiences? Can we? Are criteria similar or different? • To what extent do we validate our selections – remember our logic…treat each selection as an experiment…collect data to see if the selection criteria worked?

Gathering data to predict and select • Objective data – GPA, test scores, count publications, etc. • Even though objective, still boils down to subjective interpretation – not definitive or necessarily predictive • Subjective/qualitative data • Answers to questions on forms • Interviews • Is the goal to just see what someone will say or look for specific answers? BIG DIFFERENCES! • Again, to what degree do we treat our selections as experiments and collect data to validate them? • Emphasis of Northwestern IMSD and my previous jobs – educate faculty on validity(?) of typical selection data • especially for students from atypical backgrounds – unlike many faculty (underrepresented, disadvantaged, others)

For research programs, how do you identify and choose the right students? • Grades, especially in science courses • What other information do you ask for and use? • What are your favorite questions to ask on paper/web applications to your programs, or when interviewing? • Why do you want answers to these questions? • What data do you expect the questions to give you? • What will you do with the data? How will it contribute to your selection? • What evidence do you have that the data the questions provide predicts persistence to the PhD and/or a career as a scientist, or anything? • Think of it all as oral data – even if written

Asking questions, interpreting answers • Applies to both written (paper or web) and oral • Most commonly we ask questions to just hear what the student says – often not consciously listening for a particular response • How we interpret the data is all over the map • Our ‘evidence’ for the importance or predictive validity of the answers is largely visceral, almost never grounded in objective evidence or outcomes • Even if listening for something, often unwilling to make definitive decisions from what we hear because we have not validated the data • Question design REALLY MATTERS!!!! • Especially on a form or when you can’t ask for elaboration/clarification

How did we get started on this topic? • One of the first IMSD programs awarded in 1997 – Mayo Clinic • New requirements for robust evaluation plan/process • Renewed emphasis on PhD and MD/PhD as outcomes for undergraduate programs – e.g. summer research • Core question – Does high yield reflect quality of program or quality of selection, or both? • Experience with summer undergraduate researchers fed hypothesis that selection critical, but poor program or research experience might deflect • Also, we had HUGE problem with MD/PhD students dropping the PhD in first two years! • Random life event – met Dr. Jill Keller, education qualitative researcher from U of AZ, at reception in Washington, DC

Basic question… • Is it possible to predict with high reliability those college students likely to go into research, often vs. clinical medicine? If so, how, when, and using what criteria and methods?

Delicate issue… • NO desire to come up with formulaic approach that could limit opportunities and student development • Many URM students enter college with no clue about research as a career and what a PhD is all about – programs essential for raising awareness and interest – can’t constrict too much • But resources are limited and NIGMS had to show it was accomplishing program goals • MOST importantly, our job is to help students find their niches….not good to push square pegs into round holes or push them into unsatisfying lives

More delicacies… • Basic scientists often want to avoid taking ‘pre-meds’ into their labs – Why? • Mentor-based research training is predicated on interactions with those with like interests • Pre-med students seldom get ‘turned on’ to the lab and research – frustrating when they can’t be converted • Feels like taking opportunity away from someone who really needs experience for PhD or MD/PhD • But learning about research is valuable to future MDs • Many medical schools put high premium on UG research – some REQUIRE it – so need is high • Many pre-meds have no idea about research

How do we do this kind of research? • We ask students questions – called interviews – but interested in the answers with no intent to judge them or use the data for selection • Basic questions start the conversations but then we follow it wherever it goes • Conversations are recorded and transcribed • We look for patterns in what people tell us • Sounds pretty simple….

Qualitative Research - A long history in social sciences • NOT an oxymoron • Also includes surveys, focus groups, observation • Used by anthropologists, sociologist, some psychologists, education researchers, etc. • Powerful way to study behavior, beliefs, experiences • Can find common traits that ‘define’ a group – basis for Cultural Anthropology, Sociology • Can test hypotheses or generate them

Data Analysis Using Grounded Theory • Grounded in Reality – something that is concrete and an important part of the life of almost everyone in the group • If it is an important trait people will express it naturally, without thinking, and in high frequency • Should be discoverable with as few as 8-10 individuals

Our study subjects • Undergraduates doing summer research at the Mayo Clinic – from all over the US, including Puerto Rico – rising JR and SR students • Semi-structured interviews at the start, the end and 9-12 months after summer research • Vast majority go on to PhD, MD or MD/PhD but outcome unknown at the start • No trouble getting them to talk about themselves and their experiences! • Interviews done by trained interviewer • Transcripts analyzed by R. McGee and Jill Keller

Example entrance interview questions… • How did you “get here” - i.e. past experiences? • What roles have family, friends, teachers and mentors played in your experiences? • What are your career plans and goals at this point? • What do you think you will get form this career or the careers you are choosing between? • Etc. The interviewer listens for what is important to the student and asks questions to extend and clarify

Example exit interview questions… • Describe the research you did this summer. • What do you think will stay with you from the summer? • Can you identify people who have acted as mentors to you this summer? What did they provide? • What is your current view of and interest in research? Do you see research as part or all of your career? Why or why not? • Answers have no impact on students – very different from application/interview questions • Etc.

Results – Two themes consistent across almost all students… • High self-confidence • Action-orientation • Equally present in those who went on to research and clinical training • Qualities of successful applicants who rose to the top of the SURF applicant pool

5 Themes predicted the 18 (of 28) who went toward PhD, MD/PhD, or MD to do research… • 1. Curiosity to discover the unknown…. especially something that nobody had ever seen before • expressed by 15 of the 18 students who started toward research • 1 of the 3 who did not show curiosity changed to DVM after 1 year – we predicted it was not for her • only seen in 1 of 8 other students who went on to clinical MD or other direction

Curiosity to Discover the Unknown ...it’s like what do you call it? Like frontier land, sort of. You know, it’s like people have general ideas about things, I mean it’s based on a lot of factors, some of it’s theory, but it seems like it’s always changing and you can always make new discoveries. I guess I like the fact that it seems like you can always find something or be on the brink of discovery.

2. Enjoyment of Problem Solving • Similar but slightly different from discovery of the unknown • 10 of 12 students who went on to PhD or MD/PhD • Did not come out as strongly or emotionally as curiosity

Enjoyment of Problem Solving ….when I first started working in the lab, it just killed me to see nothing for three months. But I still learned from it and I learned what I was doing wrong and different ways to fix it and you get... real creative ... you might spend a whole day thinking what can I do to prevent this gel from leaking out the bottom…. So finally you look at everything at the lab and you find something sticky and you say, "I'll try this". So you try Vaseline or something so it makes you very creative. So when you're trying to rig something up - it just makes you creative when things aren't going right.

3. Independence • All of the students who expressed the intent to do research in the future talked of wanting to being able to think or act on their own with limited consultation or guidance from others. • None of the students who went on to other advanced training expressed this as a strong trait or something they sought

Independence I: So what makes you feel that you want a career as a scientist? S: I’ve always loved science. I haven’t been that good in math, and I wanted to be a scientist…I went to a medical magnet to experience different things and through there I got involved in different researches. It was like, “That’s what I want to do. I want to work in a lab.” I like finding something using my analytical skills. It’s like you’re trying to search something and I really enjoyed that. It’s very independent, like you go in there and they teach, and you’re on your own and no one’s looking over your shoulder, “you need to do this-- you need to do that.” I like that independence -- learning new things by yourself that they just give me a protocol and I can do it. Maybe not perfect the first time. But I am my own woman and you learn little tricks so it’s very independent and I like that. It’s an individual job.

4. Helping others indirectly through research • 24 of 26 expressed a desire to help others • Not unexpectedly, those who went on to medicine expressed this strongly and directly – seeing the direct impact of their caregiving • Virtually all of those who headed toward research did too • expressed as a desire to work on research that might help others, realistically knowing the chances were not high they would see it

Helping others indirectly through research – went on to PhD I: But when you think of ‘making a difference’, how do you define making a difference? S: Many ways. One of the ways that comes more clear to me would be changing someone’s life from worse to better… -- let’s say I’m working with asthma or something and I develop something that will prevent them from suffocating or needing a ventilator every time they get wheezy. I have made a difference in their life making them more comfortable.

5. Approaches to the Future…. • Very surprising theme that took a long time to emerge • Clearly differentiated two patterns among students

Approaches to the Future Pattern #1 • Clear and often well thought out goals for the future and plans of how to attain them • Often included both professional and personal goals and strong sense they were attainable • Planning often included strategic decision-making to achieve sequential steps along the way to goals

Approaches to the Future Pattern #2 • Minimally structured view of the future • Long-term goals more vaguely defined, often expressed as several possible outcomes • Minimal concern about which outcome achieved • Sometimes exasperated that everyone kept asking them about what they planned to be/do • Enjoyed seeking out or creating options rather than focus on moving directly toward a single goal • Often part of both professional and personal lives

Minimally Structured Approach to the Future I: So you're definitely planning to have research as some part of your career. S: Right now. Who knows....? I just like to leave the door open because just talking to people - the professors or something - most people don't end up in what they think they're going to end up in college so I'd like to say for sure but right now that seems like where I'd like to be.

Additional Tidbits • Able to replicate results with online survey at the end of the summer – wording VERY critical… • “If you are currently planning to pursue a research career as a PhD or MD/PhD, what do you find attractive about doing research? Why have you chosen it as a career path?” • High fraction came up with curiosity/discovery as primary reason • Similar pattern with MDs who expressed strong intent to continue research after completing Masters in Clinical Research

Words say it all…. • “I feel it is rewarding to discover new things, especially those that can help people. It is also just really interesting.” • “I like the idea of discovering things that no one has known before and that the research I do will contribute to the greater scientific knowledge about a topic. I also hope that my work will someday be used help to treat human disease.”

Some other themes emerging in current research… • Decisions commonly driven by interests • For some these interests can flip quickly • Others hold on to them tenaciously – immunologists, neuroscientists • Don’t expect research-focused students to be interested in everything…. we aren’t • Mismatch between labs and interests can be misinterpreted – interest alignment important – don’t take it lightly • Motivation seems to be more internally driven, but it is difficult to separate internal and external motivation • Acquiring an identity as a scientist emerging in work of others and our current research as pivotal step in commitment to PhD

Take-away messages… • It may be easier to predict future scientists than we thought, but what you look for is different • At least by late college, tendency toward research is largely set – explains great difficulty of converting pre-medical students to research • We do see it in some students consciously choosing to do MD as a route to get there w/o MD/PhD, and later in other MDs, but quite limited • Can use to cautiously help students figure out if research is a good fit for them • Whole different approach to gathering and using selection data required

Can you apply this? • Mayo MD/PhD program story • Mayo SURF story • Rescue of a failing MARC program story

Caveats on the characteristics… • Only beginning to study how soon these tendencies are set, how to find them prior to doing research • Have not tried to systematically detect their expression outside of the research context but do have ideas • May take months to years for student to know what to do when they discover a ‘fit’ with research if it conflicts with earlier goals/plans • Only beginning to study these characteristics at college application • Also only beginning to study the relationships between these characteristics and progression through the PhD and beyond…

So how do you detect/find these qualities? • Stage of development and life experiences matter a lot • In high school • Entry into/early college – RISE, IMSD, Bridges, REU • Big developmental step ~ junior year – RISE, MARC, IMSD, Bridges, REU • Late bloomers/slow committers – PREP, Bridges • PhD – Bridges, IMSD • Postdoc

So how do you detect/find these qualities? Start with the easy ones… • Easiest in verbal exchanges but can be done with carefully crafted written questions • Best to use open ended, tangential questions that reveal what comes out naturally • Easiest case – student has just returned from well-designed summer research experience in good lab • “Tell me about your time at U of X this summer.” • Reveals what was most important/meaningful • Hopefully research will come out very early • Asking follow-up questions to pull out reasons for what they talk about – reveals what is important

Questions continued… • “What did you like best about your time in the lab?” • Listening for curiosity/discovery/problem solving • If not, tease out if the absence bothered the student • “Sounds like you had a pretty established project to work on, not a lot new to discover or problems to solve. Was that ok with you?” • If it didn’t bother them, or other most important things come out, probe to see what they were looking for in the experience to reveal insights into them

Questions continued… • “Why DO you like doing research?” • If there is a key question, this is it, but becomes revealing only AFTER student has experienced REAL research • Research experience need not be highly successful, get good data or publication – absence of success actually probably more revealing • But should be positive experience from social or developmental context • Listening for curiosity, discovery, solving problems or puzzles – ideally the first thing that comes to mind and pops out • Often hear elements of making a difference, other things, but listening for these being secondary

Questions continued… • If someone has been in more than one lab…… “What has been your most rewarding experience in any of your research to date? When was your biggest ‘high’?” • Again looking for discovery and problem solving – best when content with small pieces they see add up • Sometimes will hear the thrill of seeing something for the first time nobody has ever seen before • Getting a paper published, seeing your name in ‘print’ also comes up – likely this should not be the primary motivation given it is so infrequent – also suggests external motivation more than internal

Questions continued… • “Have you had periods of time when nothing was working in the lab? How do you deal with it? How does it affect you?” • Listening for high tolerance of failure • More mature students will talk about how you learn new things even from failure – small problems solved • If answer is no, “How do you think you will handle it?”

Harder situation – no prior research • “What got you interested in science?” • Listening for curiosity-driven experiences and pure interest • “What have been your favorite sciences classes? What made them your favorites?” • Listening for classes that encouraged development of student’s interests, problem solving, experiments with unknown outcomes – topic aligned with or triggered student interests • Allowed independence • Be caution with things like ‘teacher made it really easy, teacher was very clear on what we needed to know’, etc. • “Since you have never done research before, what makes you think you will like it?” • Ideally listening for accurate idea of what research is about, solving problems, finding new information • Tricky handling this kind of situation

Contrast with the classic question • “Why did you apply for this program?” • Student very likely trying to figure out what you want to hear • Much less likely to get true, unfiltered answer • Still can be listening for what seems like honesty… • Friend who is doing research and it sounds like “fun” • I need to make money and it will be a lot more valuable than working at McDonalds – this is ok, don’t loose sight of students’ realities – would treat this as neutral information but honest • Pre-med is unlikely to tell you to improve their resume • Fine to ask, harder to know what to do with the information – like a poorly designed experiment

View of the future • “Where do you see yourself in 15-20 years?” • Purposely pushing out to long vision • Listening for open frame or branching options • Concern when there is a very clear and narrow outcome • Fine line if totally clueless or too uninformed • “What will you feel like if that is NOT where you end up?” • Listening for flexibility, comfort with alternative outcomes • “How will you feel if near the end of your career you have been very successful, gotten grants, published papers, but nothing you have discovered has made any measurable impact on health or well being of others?” • Listening for no problem with that – often will easily respond ‘that will be ok’…things like others may have used their work, or trained others have had an impact – should be clearly no problem with it • Be cautious with the long pause or negative reaction

Independence • Harder to explicitly draw out – we struggle in our own research • Comes out spontaneously and in multiple contexts • “When you are working in the lab or on school projects, do you prefer to have your own project, knowing it might go slower or problems may be harder to solve, or work on a group project?” • Have not used much but might reveal independence • Despite rise in research collaborations, independence likely a preference driver at early stages • Often comes out as wanting to have own project • “When you run into a tough problem in the lab or school, how do you approach solving it?” • Listening for drive to solve it themselves but also able to get guidance from others if need be

What questions would you like to try? • Throw out a question have used or have considered using or might use now after what you have heard… • Are there specific issues you would like help figuring out how to get specific data?

Acknowledgements - Our Team… • “Scientific Careers Research and Development Group” • “Career Decision-Making of Future Minority Biomedical Faculty” – NIGMS-funded • Jill Keller, PhD (Arizona) – Co-Investigator • Patricia Campbell, PhD, Campbell-Kibler Assoc. – Co-Invest. • Robin Remich, MAT, MEd – Research Associate • Bryan Breau – Research Study Coordinator • “Pivotal Career Decisions Guiding Potential Women Science Faculty” – NINR-funded • Patricia Campbell, PhD – Co-Investigator • Jill Keller, PhD – Co-Investigator • Sandy LaBlance, PhD – Research Associate • Bryan Breau – Research Study Coordinator

Our Team… • Integrating multiple social science theories to interpret career decisions and guide intervention design – NIGMS • Lynn Gazley, PhD, MPH – Research Associate • “Mentoring for Success: Developing Fundamental Skills for Biomedical Research” – NIGMS • Steve Lee, PhD – Assistant Director • Karl Keller, MA – Communications Professional • “Translating Theory to Practice to Diversify the Biomedical Research Community” – NIH Pathfinder – “The Academy for Future Science Faculty” the whole team plus… • Michelle Naffziger, PhD – Research Associate • Jennifer Richardson, MA, PhD – Research Associate • Simon Williams, PhD – Research Associate • Beth Morrissey, MA – Research Project Coordinator

My contact information • Rick McGee, PhD Associate Dean for Faculty Recruitment and Professional Development Northwestern University, Feinberg School of Medicine Chicago, IL 312-503-1737 r-mcgee@northwestern.edu

References • McGee, R. and Keller, J.L.: (2007) Identifying Future Scientists: Predicting persistence into research training. CBE-Life Sciences Education 6(4), 316-331. http://www.lifescied.org/cgi/content/full/6/4/316 • McGee, R and DeLong, M.J. (2007) Collaborative co-mentored dissertations spanning institutions: Influence on student Development. CBE Life Science Education 6(2), 119-131. http://www.lifescied.org/cgi/content/full/6/2/119 • McGee, R., Almquist, J, Keller, J.L. and Jacobsen, S. J.: Teaching and Learning Responsible Research Conduct: Influences of Prior Experiences and Conflicting Messages. (2008) Accountability in Research 15, 30-62.

From an art to a science…