1 / 33

Where’s the Data? Conducting a First-Year Data Audit

Where’s the Data? Conducting a First-Year Data Audit. Karen Paulson Asheville, North Carolina – July 2002. The Data Audit Toolkit. Developed Through a Partnership The Policy Center on the First Year of College John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright NCHEMS

dschwartz
Download Presentation

Where’s the Data? Conducting a First-Year Data Audit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Where’s the Data?Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002

  2. The Data Audit Toolkit • Developed Through a Partnership • The Policy Center on the First Year of College • John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright • NCHEMS • Karen Paulson and Peter Ewell • Generously Supported by: • The Pew Charitable Trusts • The Atlantic Philanthropies

  3. Pilot Institutions • Augustana College (IL) • Blue Ridge Community College (VA) • Lynchburg College (VA) • Northeast State Technical Community College (TN) • Ohio University • Santa Fe Community College (FL) • The University of Texas at El Paso • University of Cincinnati • University of Minnesota Duluth • Washington State University

  4. Data Audit “The process of identifying data resources and uses wherever they may be within an institution and gathering them into a useable information system.” NCHEMS Administrative Rationale manual, page 15

  5. Basic Premise I • Most of the Data Needed for Meaningful Analysis Already Reside in Institutional Data Systems • Admissions • Registration/Transcripts • Student Surveys • Assessment Databases • Individual “Service Office” Records

  6. Basic Premise II • The Major Task Is: • To Organize These Data in Useful Ways • To Make Access and Analysis as Flexible and Straightforward as Possible

  7. The Data Audit Process • Is NOT Itself an End • Is a Means for Understanding and Improving the Use of Data and Information on Campus  Key to Improving the First Year of College

  8. “…we tend to assume that all first-year programs were implemented as planned and that the experiences of all students were uniform.”

  9. In Reality There Are Three First-Year Experiences* • The “Official” One (Designed and Published Plans) • The “Delivered” One (Actual Institutional Actions) • The “Experienced” One (Reality Including Student Choice) * Adapted from Joan Stark

  10. The Main Question…. • How can your institution analytically disentangle the many elements of the first college year and provide evidence about the effectiveness, or lack thereof, of related first-year programs, policies, and procedures?

  11. Some Preliminary Questions • What Is Unique About the First Year of College at Your Institution? • What Is it Like to Walk in a Student’s Shoes at Your Institution? • How Many Surveys, Tests, or Assessments Does a Student Really Experience During the Year? • Does Any Information from These Data Collections Get Communicated Back to the Students? How? • How Are These First-Year Data Used by Faculty and Administrators?

  12. A Quick Self-Assessment • What Are the Primary Sources of Data About Students, Curricula, and Programs—Specifically Related to the First Year of College—Currently in Place at Your Institution? • Are Any of These Underutilized? Why? • What’s Missing About Which You Would Like Information?

  13. Why a Data Audit? • Identify and Inventory Data Sources • Identify and Inventory Data Needs • Support Assessment of... • What Happened • What Mattered • Foster a Culture of Data Use

  14. Primary Data Audit Activities The Supply Side Conducting a Campus-Wide Examination of Existing Data Sources The Demand Side Determining Which Data Are Most Needed for Evaluation, Assessment, and Decisionmaking

  15. Key Components of a Data Audit • Identifying Sources of Data • Inventorying These Data Sources • Compiling Information About Data Sources • Identifying Gaps in Data Sources • Assessing What Data Users Need and Want to Know • Determining Which Analyses (Existing or Suggested) Have Utility for Stakeholders of Your Institution or Unit

  16. Organizing a Data Audit The Data Audit Leadership Team • Institution-Wide Committee • IR, Student Affairs, Academic Affairs, Advisors, Orientation Staff, Faculty, Registrar Staff, Students, etc. • Team Members • People Directly Involved with the Institution • People Directly Involved with Data • First-Year Professionals not Normally “Data Connected” • Leader(s) with Broad Campus Support and Respect

  17. The Right Attitude • Fresh Perspective • Open Attitude • Collaborative Approach Not a “Gotcha” Mentality

  18. Visiting Potential Data Sites • Reasons for Physical Visits • Honors Unit Personnel on Their Turf • Builds Relationships • Allows You to Read Reactions • Allow You to Do Immediate Follow-Up and Collect Artifacts • Discovers Hidden Databases

  19. Types of Data • Official Data • Registrar Systems • Admissions • Institutional Research Offices • Assessment Offices • “Unofficial” (Guerrilla) Databases • Unit Records Kept in Local Computers • Viewed as “Single” Purpose/Disposable Records

  20. Supply Side Questions • Types of Data? • Who Collects Data and Why? • How Complete Are Data? • Where Do Data Go? • “Walking the Process”

  21. Supply Side Questions to Ask • What Kinds of Records or Data Do You Keep on First Year Students? • How Complete Are Data Collected? • How Are Data Entered? • What Schedules Govern Your Data Collection? • How Are Data Updated?

  22. Finding Existing Data • Follow Student “Footprints” • Time-Based Investigation • Who Collects Data? • Structures-Based Investigation • Why Are Data Collected? • Functions-Based Investigation

  23. Supply Side: Items to Collect • Actual Forms and Questionnaires • Data Element Dictionaries • Data Element Definitions • Database Structures and File Formats

  24. Demand Side Questions • Who Are the Key Constituencies? • What Are the Existing Reports and Requirements? • To Whom Are Data and Information Reported? • What, and When, Are the Decision Cycles? • How Current and Accurate Does the Information Need to Be? • What Are the Gaps in Existing Data?

  25. Demand Side Questions to Ask • To Whom Do You Report Data and Information? • What Information Do You Need or Wish You Had? • What Are the Gaps in Data? • How Are Data Used By Others?

  26. Demand Side: Items to Collect • Representative Reports to Constituents • Copies of External Data Reporting Requirements

  27. Looking for... • Completeness of Data Gathered • Availability of Data • Integrity of Data • Consistency of Data Definitions • Who Coordinates Data Collection? • Who Coordinates Use of Data? • Who Controls Data and Data Processes?

  28. Data Audit Output • Synthesized and Coherent Picture of Existing Data • Data Element Lists • Data Definitions • Data Locations and Locus of Responsibility • Data Collection Timetables • Plan for Common Data Collection/Sharing • Identification of Unmet Data Needs

  29. Design of Recommended Data Structure • Define Core Data Elements • Responsibilities and Data Flows • Core Indicators and Calculational Routines

  30. Data Leverage Points • Problem Identification • Persistence Rate • Violations of Prerequisite Sequencing • Student/Advisor Ratios • Context Setting • Achievement by Gender or Race • Patterns of Student Performance • Entering Student Characteristics • To Inform Discussion • Overall Patterns Rather than Anecdotal or Single • Selling Decisions • Creating Buy-In for Action

  31. Using Evidence to Stimulate and Manage Change • Start with Obvious Discrepancies • Between: • Perceptions and Reality • Designs and Delivery • Expectations and Results • Among Different Constituencies and Stakeholders • Avoid “Perfect Data Fallacy” • Recognize There Will Always Be “Errors” • Use Triangulation and Multiple Indicators • Know How Good Is Good Enough • Use Information in “Layers” • Avoid Excessive Complexity • Package Data Around Problems • Disaggregate as Needed in Response to Questions and Opportunities • Use Information to Start Discussions, not “Give Answers” • Develop Multiple Interpretations • Establish “Data Dialogue”

  32. The Data Audit Toolkit(coming fall 2002) • Administrative Rationale • A non-technical overview and rationale for a data audit. • Technical Manual • A detailed guide, listing data elements and data, as well as step-by-step plans for a data audit.

  33. Thank You! • Karen Paulson, 303.497.0354, Karen@nchems.org • Mike Siegel, 828.877.6009, siegelmj@brevard.edu If you have questions or want to discuss the data audit and analysis project further, please contact:

More Related