During the 2013 academic year, librarians undertook a project to assess the information literacy skills of Dickinson students across the curriculum. The methodology used was borrowed from the Writing Program assessment project completed the same year.
Results of this assessment showed:
- Students in all categories performed less well than expected
- Seniors generally outperformed students in first-year seminars and writing intensive courses in all categories
- A lack of progression between the first-year seminar and a writing-intensive course
- Regression in certain categories between the first-year seminar and a writing-intensive course
Methodology & Rubric
Following Chris Bombaro’s participation in the Writing Program Assessment Workshop (Summer 2012), librarians used the methodology taught by consultant William Condon of Washington State University to develop a rubric and used it to measure student information literacy competencies. Using sample essays collected by the Writing Center staff, librarians identified elements that indicated good practice and due diligence in information literacy. Those elements were used to build a new information literacy rubric. Following the creation of a first draft, the rubric was tested against several additional papers in order to refine it. The rubric is available here.
After the final draft of the rubric was established, 8 liaison librarians set out to assess 111 samples. Librarians were not provided with the prompts for the assignments and the samples were stripped of information that identified the course level. The writing samples were the last or penultimate assignment for the course. Of the 111 samples, 45 were from first-year seminar classes, 32 were from writing intensive courses, and 34 were from senior seminars. These samples were selected from the same batch used to conduct the writing assessment in summer 2012 when randomly selected faculty members were asked to submit 6-8 writing samples from a course (e.g. student #2, 4, 6, 8, 10, 12 on the roster).
Because the original batch included more than 200 essays, the sample size was reduced in order to make this project manageable and to ensure that there was a close sample size from each level. Any papers not written in English were removed from consideration, leaving 36 senior-level papers in the group. A balance of science, social science, and humanities papers were selected from the writing intensive courses by choosing random papers within each course. Three papers were randomly selected from each of the 15 first-year seminars represented.
The rubric uses a six-point scale to measure information literacy skill across six developmental levels. As it is an absolute rubric, first-year students were expected to score in the 2-3 range, WR students in the 3-4 range, and seniors in capstone courses to score in the 5-6 range.
The overall results of the assessment are represented below, with the vertical black lines representing standard deviation.
Results by Course Level
The results demonstrate relative improvement in information literacy skills from the first-year to the senior level. First-year students performed as expected in all categories, with scores falling between 2 and 4 in all categories; in 2 categories (Source Selection and Acknowledgement), first-year students performed somewhat better than expected. In addition, seniors significantly outperformed first-year and writing intensive students in all categories.
A closer analysis, however, reveals additional results that are somewhat troubling if not entirely surprising. Results demonstrate that students in first-year seminars and writing intensive courses performed similarly in all categories. In the Source Selection, Integration, Citation Style and Acknowledgement categories, students in writing intensive courses demonstrated some regression in skill.
The data further indicate that students in writing intensive course and seniors performed lower than expected. Writing intensive students scored consistently in the 2-3 range in all categories, though were expected to be in the 3-4 range. Seniors scored consistently in the high 4 range, but did not score as high as might have been desired, reaching above 5 in only the Source Selection category and slightly below 5 in the Acknowledgement category.
The possibility that the lack of improvement between first-year and writing intensive classes was due to first year seminar students enrolling concurrently in writing intensive courses was considered. However, the Registrar reports only 6% of first-years took a WR course in fall 2012. Most of the first-year students who took a WR course did so for an upper-level language requirement. Papers written in languages other than English were not considered for this assessment.
Results by Category
Students in all three categories scored best in Source Selection and Acknowledgement categories. Source selection represents the ability to choose sources directly related to the investigation, and acknowledgement indicates that students are using research material ethically and avoiding plagiarism.
Students at all levels performed the least well in the Scope category. This category represents the author’s use of a suitable breadth of material, including, where appropriate, primary and secondary sources and scholarly and non-scholarly sources. The relative lack of range in type of research material may contribute to students’ difficulty in analyzing and contextualizing sources (Analysis and Integration categories).