Methodological Challenges in Conducting University Rankings and Institutional Classifications: How to secure reliable d

1. Methodological Challenges in Conducting University Rankings and Institutional Classifications: How to secure reliable data Robert J. Morse, Director of Data Research, U.S. News, Wash DC rmorse@usnews.com Presented at: IREG Forum on Ranking Methodologies Warsaw, Poland May 16, 2013

2. America’s Best Colleges: A Brief History • America’s Best Colleges now nearly 30 years old • 1983, 1985 and 1987 rankings were reputation only • Published annually starting in 1987 in U.S. News magazine and in a separate guidebook • Starting in 1988 the ranking methodology has been a mix of reputation and statistical data. • Other rankings: Best Values, Best Publics, Debt load, Diversity, Undergraduate business/engineering • Since 1997 America’s Best Colleges rankings online www.usnews.com. More rankings/data online. • Best Graduate Schools: annually starting 1990 • America’s Best High Schools: started in 2007 • World’s Best Colleges: Started late 2008

3. Best Colleges: The Editorial Philosophy Behind the Rankings • To help consumers: prospective students and their parents make informed choices about: • expensive investment in tuition, room & board, etc. cost is around $200,000 plus in some cases in U.S. at private colleges • one-time career decision • Provide the public and prospective students with an understanding of latest trends in higher ed • Practical advice on many aspects of attending, financing and applying to college. One part of our ongoing reporting on educational issues. • Transparency in how we do the rankings. • The rankings should just be used as one tool in college search, application and choice process, not sole factor

4. USNEWS Statistical Data Collection • USNEWS collects statistical data yearly from each school using our own online data collection system. • Statistical data is self-reported by U.S. colleges/univ. • Data collection has many built in edit checks. • Data collection has both an analytical assessment and final data verification phase . • Survey instruments use standard definitions of variables from: Common Data Set developed with higher education experts, U.S. Dept of Education-Integrated Postsecondary Education Data System (IPEDS), American Association of University Professors (AAUP) and Council for Aid to Education (CAE) • USNEWS cross-check large proportion, but not all reported data with IPEDS, AAUP, CAE and NCAA

5. USNEWS Data Collection Realities • Does this system produce 100% accurate data = No • Do some schools make mistakes on their data submissions that hurt them in the rankings= Yes • Do some schools purposely misreport data to U.S. News that result in a higher rankings= Yes • Is it possible for U.S. News to audit the data submitted by U.S. colleges to U.S. News = No • Has there been highly publicized cases over the years of data misreporting = Yes • Is it embarrassing and costly to schools and the persons involved when this happens =Yes • Do schools ever say the misreporting was done to rise in U.S. News rankings = No

6. Data Integrity Issues in 2012/13 – All were explained in detail in Morse Code blog • Morse Code – Inside the college rankings link http://www.usnews.com/education/blogs/college-rankings-blog • Iona College, NY -- 9 years ending 2011, falsification many data points: Unranked starting in 4/12 for rest of Best Colleges 2012 edition • Claremont McKenna – 8 years falsified 2004 thru 2011-SATs/ACTs/HS standing/application counts (not unranked) in 2012 Edition • Emory University--2000-2011 falsified SATs/ACTs and HS standing (not unranked) in 2012 Edition • Significant misreporting Best Grad Schools March 2012: University of Houston (Ed); St. Thomas Law, MN- both unranked for 2013 Edition

7. Data Integrity Issues in 2012/13 – All were explained in detail in Morse Code blog • Tulane Business School's 1/24/13 -GMAT scores and the number of applications for the full-time MBA program for the classes that entered from fall 2007 through 2011 misreported. (unranked for Best Grad 2013 edition) • George Washington University erroneously over-reported its high school class standing % in top 10 for more than a decade Unranked 11/14/2012 for rest of the 2013 Best Colleges edition • Bucknell, PA 2/4/2013 misreported its freshman classes entering fall 2006 through 2011—some students were excluded from the school's originally reported SAT and ACT scores. (Not unranked in 2013 Best Colleges) • University of Mary-Hardin Baylor, TX and York College of PA . UMHB incorrect acceptance rate 27.4%; correct 89.1%. York College of PA excluded 20% students from SATs, boosting SATs. • On 5/14/2013 Unranked for remainder of the 2013 Best Colleges edition

8. Data Integrity Issues in 2012/13 – All were explained in detail in Morse Code blog • U.S. News believe these are isolated, but highly visible cases. • The fact that schools go through painful process of public disclosure of misreporting shows data becoming more important to schools • Each case reflects an acknowledgement by schools that reporting accurate data to a variety of sources is essential to the schools' integrity and long-term best interests.

9. Data Integrity Issues in 2012/13 – All were explained in detail in Morse Code blog • In most, but not all cases, the misreported data were also given to the U.S. Department of Education through IPEDS, bond rating agencies and other major college data collectors and publishers like College Board and Petersons. • The point is that the same misreported data went to many places and cross-checking wouldn’t have caught it.

10. Data Integrity Issues in 2012/13 • Each school had specific reasons for the cause of their data misreporting. • Each school has handled the release of the data misreporting differently. • It has never been said that the reason was to improve in the U.S. News ranking

11. Data Integrity Issues in 2012/13 • However, in some of the cases regarding admissions data it was stated that one person –sometimes head of admissions –had sole control of the data for external reporting purposes and had the ability/power to report incorrect data compared to what the accurate data would have produced. • In these cases apparently there were no other persons at schools who had access to cross check the results using the actual data.

12. Data Integrity Issues in 2012/13 – All were explained in detail in Morse Code blog • What are the consequences for schools that misreport data? • In the cases where the misreported data improved a school's numerical ranking, we have dropped that school from our rankings tables for the current year of the rankings where the misreported data were originally included.

13. Data Integrity Issues in 2012/13 – All were explained in detail in Morse Code blog • What are the consequences for schools that misreport data? • The school is dropped from the rankings until at least the time that the next edition of the rankings is published. We restore the school to the next year's ranking when it has provided assurances that the data it is providing are accurate. • US News also posts the corrected data on usnews.com

14. What is next in data integrity? • There will continue to be data misreporting issues in the U.S. • Schools are aware that they are under growing scrutiny to be honest and establish better procedures to ensure accurate external data reporting. • It’s unclear whether the U.S. Department of Education will take any action against the schools who have misreported