Beyond Commercial Rankings of Colleges and Universities:
An Alternative Movement by the Annapolis Group

A Personal Point of View

William G. Durden
September 5, 2007

Last month, the Annapolis Group, an association of approximately 125 liberal arts colleges, announced the formation of a subcommittee of member presidents and institutional research officers to advance a common format for presenting information about liberal arts colleges to aid students and their families in the college search process. I have been appointed to chair the Annapolis Group Common Information Template subcommittee as we seek to develop an “alternative” to the commercial rankings.  The result would consist of an array of quantitative and qualitative data points as well as guidance in how to use this information to find the right match between a student and a college.  The work of the subcommittee has just begun; therefore, the following represents my views only, although many of my Annapolis Group colleagues, I believe,  share them.

At the outset, let me state clearly that I fully and completely support accountability and transparency. What I oppose are numerical rankings. There is a tremendous difference among these approaches; in fact, there is absolutely no correlation among accountability, transparency and rankings. The Spellings Commission on the Future of Higher Education, for example,  is trying to address accountability and transparency. Rankings, on the other hand, are a money-making invention masquerading as science. They sell the falsehood that one can determine the overall academic and educational impact on a student from one college over another college, through a simple serial list (1,2,3... 1 being better than 10...), and that in the absence of this contrived system, there is no other way by which to compare one college against another.

U.S.News has marketed its rankings so well that some segments of the public aren't sure how to conduct their own college search. But as U.S.News also points out, college is expensive.  So isn't it worth some extra effort to find the college or university that truly fits a student's educational needs? In this age of iPods, Facebook, YouTube and other innovative ways to demonstrate one's identity and individuality, isn't choosing an institution of higher learning worth more effort than leafing through a magazine that is manipulating publicly available data to build a numerical list for the purpose of selling more magazines. It is amazing that this "trick" works, since the list of the "top 50" does not change significantly from year to year.

In short, it is the simplistic ranking (and the peer assessment) that is objectionable.  This information is being applied well beyond its area of usefulness and contribution.

Background

Colleges and universities have voiced concerns with U.S.News & World Report rankings for years. Many op-ed pieces, letters to the editor, and a great many informal conversations among college representatives have taken place since the inception of the rankings decades ago. High school college counselors, by the hundreds, also have spoken out about the negative effect of the rankings on a thoughtful and deliberate college search by students and their families. We also hear from alumni, parents and students who reject U.S.News rankings as a reputable tool for researching colleges and universities.

In fact, in September 1998 my predecessor, A. Lee Fritschler, and his co-author Richard H. Hersh, wrote an op-ed piece for the Harrisburg Patriot-News indicating that the placement of schools on the list could be directly (and solely) linked to the size of an institution's endowment. They wrote, "On a study we recently completed, we found that the U.S.News rankings of private colleges and universities closely correlate to institutional wealth as measured by endowment. Indeed, readers of U.S.News could save themselves a few dollars if they simply looked at a list of college endowments, which predict virtually all of the rankings."

That dynamic still exists. Wealth secured high placement in U.S.News rankings then, and still does.  Of course, U.S. News doesn't ask how the wealth is invested in programs, but simply elevates a college for having money.

A more recent collective voice in opposition to U.S.News grew out of an Education Conservancy meeting, led by Lloyd Thacker, held in New York City two summers ago. Ten liberal-arts college presidents were invited to advise Mr. Thacker as to how his organization might work to reduce the admissions frenzy associated with the application process. Some of the discussion turned to rankings, their effects on students, families and the behavior of colleges and universities themselves. 

Earlier this year, Mr. Thacker introduced a pledge to liberal-arts colleges. The pledge was targeted specifically at U.S. News, the most prominent entity to serial rank institutions of higher education and claim scientific justification for its methodology.  By signing the pledge letter, an institution agreed to cease two practices:  filling out the peer assessment and using rankings for marketing purposes. This was no issue for Dickinson College since we were already doing those two things.

The Annapolis Group's annual meeting in June 2007 provided a forum to discuss our membership's disposition towards rankings. As an organization we agreed to move forward with a new instrument, one that puts students' needs and wants at the center of the admissions process.

The Challenge

Higher education itself has refrained too long from offering an organized and simple way for the public (the prospective student) to evaluate whether a certain college or university is a match for application and enrollment.

While many colleges and universities (including Dickinson) objected strongly to some of the observations and recommendations contained in the report of the Commission on Higher Education sponsored by Secretary Margaret Spellings, she did inject the topic of accountability and transparency into the national conversation and caused many of us to look at our own transparency. She and the Commission are to be praised for that.

In fact, we have been accountable. For years, we have offered data about our colleges to the public. All the statistics and figures that U.S.News uses in its ranking formula (except the opinion-based peer assessment) are already available to the public and U.S. News. This information was consolidated in the mid-90s in the Common Data Set Form, so that colleges could complete one survey about their performance rather than the myriad of surveys coming to them from all sorts of organizations. Information also is included in the Integrated Postsecondary Education Data System (IPEDS), a comprehensive set of data that all colleges and universities are required to fill out each year and submit to the federal government. Numerous college websites also post this data directly to the public.

With all of this verified information readily available, U.S.News still insists that colleges and universities fill out its special survey, refusing to use its own staff to gather the data from publicly available sources. In essence, U.S.News wants nonprofit organizations (colleges and universities) to do its work—the work of a for-profit magazine. We are actually "working" for U.S.News and taking valuable time, energy and money away from other mission-focused projects.

When U.S.News says to the media that those of us seeking alternatives to it are shirking accountability, it is totally incorrect and disingenuous. This information is available to the public, for free—the same information for which commercial ranking publications charge the public!

The following list of information, on which U.S.News reports, is already accessible and free to the public: finances (endowment, etc.), development (alumni giving, etc.); admissions statistics (SAT scores, etc) and classroom characteristics (faculty-student ratio, etc.)

Many of these "objective" data points are, in fact, subject to interpretation.  U.S. News has a clear bias that the higher the wealth of an institution, the better the education. U.S. News misses the fact that having a lot of money does not necessarily mean that a college uses it well to further a student's education, just as we know that a person with great wealth is not necessarily more intelligent, happy, or personally fulfilled than a person with fewer financial resources.

U.S. News also asserts that the percentage of alumni giving is directly related to quality. The higher the percentage of giving, the higher an individual college's score. Well, there is another way that a college might approach this category and still come out with a very positive result. For instance, is a 60 percent participation rate that raises $10 million dollars a better reflection of academic quality than a 20 percent participation rate that yields $100 million dollars?

There is a final category that U.S. News relies heavily upon—the peer assessment survey—that is both random and subjective.  Let's look more closely at this variable.

At 25 percent, the peer assessment measure is the single most heavily weighted category of the numerical rankings and one of the most objectionable. Some in the general public think that peer assessment is equated with "academic reputation" and that it is derived from hard data—number of faculty publications, graduating GRE scores, etc.  In fact, university presidents, provosts and deans of admissions are simply asked by U.S.News to judge some 200 other institutions in one's category (for me, liberal-arts colleges) as excellent, good, fair, poor or don't know. U.S.News assumes that colleges should know everything about all other colleges, which is ridiculous. There is no way that I, for example, could be well enough versed about 200 other colleges—their academic programs, student selectivity, faculty resources, admissions, financial resources, development and alumni giving etc.—to make such a serious judgment about the overall quality of education for all its students. 

In other words, the information used to determine this data point is essentially based on the subjective opinions of college presidents about competing institutions.  I suppose some might argue, and some have, that this subjective material is legitimate information.  But is it reliable information and the basis upon which students should make one of the most important decisions of their lives?  Again, this unscientific, subjective section is worth 25 percent of the ranking—the most of any category.

In short, U.S.News has taken the responsibility of determining for the public which variables should be more highly valued than others—it assigns differing weights to each data point.  But there are justifiable alternative ways of looking at this distribution that would greatly influence the rankings. For example, student-faculty ratio is extremely important for learning (pre-collegiate literature on this is extensive) so why not count this data point as 25 percent of the ranking? The ranking results would automatically be quite different.

In the final analysis, U.S. News rankings advance bad information and place otherwise potentially interesting information in a faulty context. For example, U.S.News suggests that it can determine with scientific authority a discrete ranking of American colleges and universities as "best." "Best" would imply that the quality of number 1 is better than number 5, and that number 10 is far better in every way than number 20, and so on. Brian Kelly at U.S.News said in his magazine's editorial on Aug. 19, 2007, "The difference between No. 10 and No. 23 is not that great." I say, "No, it is meaningless."  In fact, absent the heavily weighted subjective peer assessment measure, it is likely that the statistical differences among many schools would be judged insignificant.

The Alternative

I believe that the authority for making a match between a student's educational needs and wants and an appropriate college or university should reside with the student. I contend that prospective students want to make informed, objective decisions about colleges based on particular points that are important to him, not those randomly determined by a commercial enterprise focused on making a profit. The power of inquiry (and the college application process) should be restored to the student and parent. I believe colleges are now ready to step forward and facilitate that connection.

The Annapolis Group does not want to provide an alternative in the form of another "numerical ranking" system. Rather we want to offer a simple, but efficient student-controlled data system—a dashboard if you will—but with far more indicators than are available through commercial numerical rankings.  Such indicators might include student learning results; student life-residential life satisfaction levels; off-campus study data; or alumni accomplishments such as percentages going to graduate and professional schools or finding employment after graduation. A student could quickly enter the databank and select those essential points.  Then the program would provide, unranked, a list of colleges and universities matching that individual's specifications.

Students could link to the respective Web sites that would further explain an individual institution's disposition towards a certain area and assist the student to make sense of the mere data. For instance, study abroad. Colleges would be able to provide additional annotated information revealing details such as the lengths of study (summer session, semester, year-long), language requirements, locations, housing options. Our goal is to provide a context to the data that would allow individual schools to distinguish themselves based on criteria that is judged to be of importance to the student.

This system must be simple and efficient and predicated on what is important to students and parents. In contrast to U.S.News, that now dictates to students what is important by pre-selecting data points, we must restore authority to those for whom this matters most.

Conclusion

U.S. News has long maintained that its college rankings are popular because nobody else offers organized information about colleges and universities to help prospective students.  The magazine contends that it is filling a void.  The time has now come not only to fill this void, but also, to evaluate if this void is being filled with information that is truly useful and meaningful for the primary “customer”—the prospective student.  The fact that a number of associations—the Annapolis Group, the National Association of Independent Colleges and Universities, the National Association of State University and Land Grant Colleges and the Education Conservancy—are all seeking alternatives minus rankings suggests that the void will soon be filled with potent, legitimate alternatives.

The college selection process is a highly complex one and a successful outcome requires a thorough and thoughtful exploration by the prospective student of the institutions in which she is interested.  The simplistic numerical ranking offered by U.S. News does not provide the kind of nuanced and contextually valid information that will necessarily ensure a good fit between the prospective student and the college or university. 

The Annapolis Group intends to develop an easy-to-use vehicle that will provide students directly with a rich variety of data points about various institutions, as well as the means to dig deeper to acquire more substantive information in any particular area.  We intend to develop a system that will invite inter-institutional comparisons by providing meaningful and contextual information rather than a simplistic numerical ranking. We intend, as is our mission as non-profit institutions of higher education, to approach a prospective student and her family as a “learner” rather than a “consumer.” Part of the learning process inherent in any respectable liberal arts education is for that learner to distinguish fact from opinion. The opinion in this case is U.S. News’s representation that its interpretation of a college’s relative merit as numerically expressed in its rankings is a legitimate source of valuable information for college selection.

When recruiting prospective students, the goal of any college or university should be to attract those whose own interests, abilities and talents mesh nicely with those of the institution.  Colleges and universities are not all alike; we have different academic strengths, different extracurricular offerings, and different residential experiences.  These variables, among others, are those prospective students should be thoughtfully and directly evaluating themselves rather than relying on information that has been massaged into rank order.

The challenge of creating an alternative to the commercial rankings that gets at these fundamental characteristics is a worthy one.  My Annapolis Group colleagues and I look forward to working together and with others over the next several months to give the American public an authentic, substantive, and useful choice to use in the college selection process.  The high cost and high stakes of a college education in America warrant such ambition.

Back to Rankings