1 / 14

Behind the Scenes in AEA: The Process of Conducting the AEA Internal Scan

Learn about the process of conducting the AEA internal scan, including the origin, contracting & collaborating, and steps taken to minimize nonresponse. Discover the challenges, lessons learned, and strengths of this comprehensive member survey.

waynes
Download Presentation

Behind the Scenes in AEA: The Process of Conducting the AEA Internal Scan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Behind the Scenes in AEA: The Process of Conducting the AEA Internal Scan Colleen Manning, Goodman Research Group, Inc. Internal Scan Project Lead Leslie Goodyear, EDC Internal Scan Task Force Chair

  2. Origin of the Scan • A strategic planning tool • More accurate and comprehensive data on the nature of the membership: • Composition of the membership • How members practice evaluation • Value of AEA membership

  3. Contracting & Collaborating • Internal scan RFP process • Dissemination of RFP • Selection of contractor • AEA envisioned a collaborative relationship with the contractor. The Task Force would: • Co-develop instruments with contractor • Give input on any sampling strategies • Help interpret results

  4. Meet GRG’s Scan Team Colleen Manning Elizabeth Bachrach Margaret Tiedemann Marianne McPherson Irene Goodman 20 year-old evaluation firm in MA and CA Diverse sectors, content areas, methods

  5. Can you imagine what our Board discussions are like?! Meet the Scan Task Force Leslie Goodyear Tom Schwandt Mary Stutzman Tom Chapel Susan Kistler Let’s make this language a little friendlier! What’s in it for members? I’ll get this out to my committee members for review!

  6. How did we scan the membership? • Web-based survey of full membership • N=5,460 • Web-based survey of a sample of non-respondents • N=200 • In-person and telephone interviews with a sample of members • N=56 • Online Q&A groups with a sample of members • N=32

  7. Steps to Minimize Nonresponse • Reminders to nonrespondents • Personalized email invitation • Important information in invitation • Pilot test of survey • An aesthetically pleasing, professional looking survey • Moderate navigational controls • Only one “required” question • Branching • An incentive

  8. Do evaluators make the best survey respondents? 24 hr reminder  2nd reminder  1st reminder  Encouragement from AEA President

  9. Exploring Nonresponse Bias • Three steps: • Nonrespondent bias survey • Investigation of differences between earlier and later responders • Comparison of respondents to known data for the population • Findings: • Earlier responders more likely to be White, longer-term members • Respondents may be skewed slightly toward intermediate term members • Respondents proportionally representative in terms of race, gender, US/international status • Stronger affiliation with a professional association other than AEA may have been a factor in nonresponse

  10. Scan Software • Web-based survey - Remark & AY • Online Q&A - vBulletin • Relational database - Microsoft Access • Quantitative analysis - SPSS • Qualitative analysis - NVivo • Communication - Microsoft Outlook

  11. Internal Scan Reporting • Interim progress report to the Board • Final report and presentation to the Board • Reports and analysis available online: • Report to the Membership • Index of Quantitative Analysis • Index of Qualitative Analysis of the Survey • Index of Qualitative Analysis of Interviews • Index of Qualitative Analysis of Online Q&A Groups • Presentation of Findings to the Board

  12. Challenges/Lessons Learned • Making one survey relevant to all members • Our strategy  Branching • Potential future strategy  More than one survey • Welcoming members into the survey • Our strategy  An open-ended question • We’d do it over again! Creative questions support creative reporting. • Creating meaningful profiles/clusters of members • Our strategy  Primary professional identity & Employment • Potential future strategies  Too many to list!

  13. Strengths • Methodologically: • Every member could provide input • Generalizeable snapshot and exploratory/explanatory stories • Leveraged resources - technology, annual meeting • Recognized group differences • Allowed for individual experiences and opinions • Working collaboratively with AEA • Breaking new ground - in AEA and among professional associations - in terms of the size and scope of a member survey

  14. For more information • For more information about the member scan, go to: http://www.eval.org/Scan/aea08.scan.htm Thank you!

More Related