Measuring excellence: The methodology behind Maclean’s universities rankings

Maclean’s 25th rankings evaluate universities across the country

September 30, 2015 - BURNABY:  SFU basketball player Vanessa Gee (left) works out as fellow player Elisa Homer watches at Fraser University in Burnaby.  (Photograph by Della Rollins)

SFU basketball player Vanessa Gee (left) works out as fellow player Elisa Homer watches at Fraser University in Burnaby. (Photograph by Della Rollins)

Maclean’s places universities in one of three categories, recognizing the differences in types of institutions, levels of research funding, diversity of offerings, and breadth and depth of graduate and professional programs. Primarily Undergraduate universities tend to be smaller in size and have fewer graduate programs and graduate students. Those in the Comprehensive category have a significant degree of research activity and a wide range of programs at the undergraduate and graduate levels, including professional degrees. Medical Doctoral universities offer a broad range of Ph.D. programs and research; all universities in this category have medical schools.

In each category, Maclean’s ranks the institutions in five broad areas based on 14 performance indicators, allocating a weight to each indicator. Figures include data from all federated and affiliated institutions. The magazine does not rank schools with fewer than 1,000 full-time students, those that are restrictive due to a religious or specialized mission, newly designated universities or those that are not members of the national association Universities Canada.

The rankings use the most recent and publicly available data. Statistics Canada provides student numbers, as well as data for total research income and five financial indicators: operating budget; spending on student services; scholarships and bursaries; library expenses; and acquisitions. Financial figures are for fiscal year 2013-14; student numbers are for 2012. Since Statistics Canada no longer collects faculty numbers, the 2013 figures were obtained directly from the universities. Data for the social sciences and humanities research grants indicator and the medical-science research grants indicator are for fiscal year 2014-15 and were obtained directly from the three major federal granting agencies: the Social Sciences and Humanities Research Council (SSHRC), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canadian Institutes of Health Research (CIHR). In addition, Maclean’s collects information on dozens of student and faculty awards, and conducts a reputational survey, canvassing the opinions of university faculty and senior administrators, high school guidance counsellors and a variety of businesspeople. Two new indicators were added this year: a major survey of students asking them about their university experience; and a measure of the amount and impact of the faculty’s published research.

Maclean’s weights the rankings as follows:

STUDENTS (28 per cent of final score) Maclean’s collects data on the success of students at winning national academic awards (weighted 10 per cent) over the previous five years. The count includes such prestigious awards as the Rhodes scholarships and the Fulbright awards, as well as scholarships from professional associations and the three federal granting agencies. Each university’s total of student awards is divided by its number of full-time students, yielding a per-student count.

To gauge students’ access to professors, Maclean’s measures the number of full-time-equivalent students per full-time faculty member (eight per cent). This student-faculty ratio includes graduate, as well as undergraduate students.

This year, a new indicator was added to provide an important element to the overall assessment: student satisfaction (10 per cent). Maclean’s surveyed thousands of students, asking them about their experience on campus. In the end, 10,856 students responded. Responses to eight questions served to create the new measure. We asked students to rate their course instructors (professors and lecturers), student-life staff and administrative staff, as well as how much red tape they encounter. Opportunities for extracurricular activities and residence life were evaluated, as well as two increasingly important areas: mental health services and the university’s actions to prevent sexual assault on campus. The question on course instructors contributed the most weight to this indicator, at 25 per cent; extracurricular contributed 15 per cent; the other six questions contributed 10 per cent each.

FACULTY (24 per cent)
In assessing the calibre of faculty, Maclean’s calculates the number who have won major awards over the past five years, including the distinguished Killam Prizes, the Royal Society of Canada awards, the 3M Teaching Fellowships and nearly 40 other award programs (seven per cent). To scale for institution size, the award count for each university is divided by each school’s number of full-time faculty.

In addition, the magazine measures the success of faculty in securing research grants from SSHRC, NSERC and CIHR. Maclean’s takes into account both the number and the dollar value received in the previous year, and divides the totals by each institution’s full-time faculty count. Research grants are reported by how many are awarded to the primary investigator on a project. Social sciences and humanities grants (six per cent) and medical-science grants (six per cent) are tallied as separate indicators.

A new bibliometric indicator was added this year to assess faculty members’ scholarly output and its impact (five per cent). Two measures contribute equally to the assessment. A field-weighted citation impact measure shows how often a work is cited, adjusted to reflect the global expectation in a given discipline.

The second measure, the total number of publications produced, uses a fractional count that takes into account multiple authors on a paper and lowers the count accordingly. The publication number is not adjusted for the size of faculty. While the number will undoubtedly be higher at larger institutions, it serves to indicate the scale of research conducted. The data were supplied by Elsevier from their Scopus database of peer-reviewed literature.

RESOURCES (20 per cent)
This section examines the amount of money available for current expenses per full-time-equivalent student (five per cent). Students are weighted according to their level of study—bachelor, master’s or doctorate—and their program of study.

To broaden the scope of the research picture, Maclean’s also measures total research dollars (six per cent). This figure, calculated relative to the size of each institution’s full-time faculty, includes income from sponsored research, such as grants and contracts, federal, provincial and foreign-government funding, and funding from non-governmental organizations.

Libraries are an important resource for students. Maclean’s measures the percentage of a university’s operating budget allocated to library services (five per cent) and the percentage of the library budget spent on acquisitions, including electronic resources (four per cent).

STUDENT SUPPORT (13 per cent)
To evaluate the assistance available to students, Maclean’s examines the percentage of the budget spent on student services (6.5 per cent), as well as scholarships and bursaries (6.5 per cent). Expenditures are measured as they are reported to the Canadian Association of University Business Officers.

REPUTATION (15 per cent)
For the reputational survey, Maclean’s solicits the views of those whose professions put them in a position to form opinions about how well universities are meeting the needs of students, and how ready their graduates are to embark on successful careers. For the first time, the survey was conducted online, and the number of university academics canvassed was greatly expanded. Senior university administrators and faculty members at each ranked institution, as well as high school guidance counsellors and a variety of businesspeople across the country, were asked to rate Canada’s universities in three categories: Highest Quality, Most Innovative, and Leaders of Tomorrow. Best Overall represents the sum of the scores for all three categories. The reputational survey has a regional, as well as a national, component that divides the country into four key areas: the Western provinces, Ontario, Quebec and the Atlantic provinces. All respondents completed a national survey; university officials and guidance counsellors also completed regional ones, allowing them the opportunity to focus on the region they know best. The surveys are combined to produce the final results.

Additional data are obtained directly from universities or from university websites. These figures are not used in the rankings calculations, as not all universities consistently make this information available.

As a measure of student quality, Maclean’s presents incoming students’ average high school grades. The figures are for full-time students attending university in their home province. No conversion formula is applied to incoming grade averages to adjust for provincial differences or varying admission policies, although CÉGEP grades are converted to a percentage grade. To provide a more detailed picture of grade averages, Maclean’s displays grades divided into six grade ranges, extending from less than 70 per cent to 95 per cent and higher.

Maclean’s counts the proportion of out-of-province students and international students in the first-year undergraduate class, as well as the percentage of international graduate students for schools in the Medical Doctoral and Comprehensive categories.

In taking stock of retention rates, Maclean’s asks for the percentage of full-time, first-year students who return in second year. While many factors can affect a student’s choice not to return—personal considerations or a decision to transfer to a program unavailable at his or her home university—student retention, on the whole, reflects a university’s success in keeping its students on course. Maclean’s also measures graduation rates by tracking an incoming cohort of full-time, first-year undergraduate students to determine if they received a degree within seven years.

In assessing faculty, Maclean’s counts the percentage of full-time instructional faculty members who have a Ph.D., a first professional degree or a terminal degree (other highest-level degree) in their field.

Finally, in looking at the classroom experience, Maclean’s presents figures on average undergraduate class sizes at the first- and second-year level, as well as at the third- and fourth-year level.

Looking for more?

Get the Best of Maclean's sent straight to your inbox. Sign up for news, commentary and analysis.