A PIRLS approach to literary understanding in secondary education

. This study explores the reading behaviour of secondary school students at about 16 years of age. Their actual performance was assessed with a specially compiled comprehension test covering four levels of text understanding. The test consisted of a set multiple choice items about a one-page passage from a literary novel. This procedure was derived from the PIRLS testing methodology for students of about 12 years old. We managed to successfully apply it to the older age group. The scores displayed the assumed decline in comprehension with increasing item difﬁculty. The performances showed diverging relations with personal characteristics. There were no differences in regard to ethnicity or gender. There were large differences in the scores between the educational levels investigated, students from vocational education showing considerably lower cores than grammar school students.


Introduction 1.1 Literary skills in education
Teachers often claim that students' proficiency in understanding texts is not sufficient for the successful completion of their courses.Students' comparatively low reading competence is responsible for all kinds of literacy problems in the classroom, and it ultimately affects school success [1].
To substantiate these claims of alleged shortcomings, we surveyed young adults' literary competences (ages 15-16).We assessed their actual performances on a specially compiled comprehension test with respect to four levels of understanding, from recognition of literal elements to inferences of causal relations and integration with world knowledge.These objective measurements were compared with participants' subjective self-assessments.
We applied the framework provided by the PIRLS-project [2].This is a large international survey of literacy held among elementary school students (aged 9-10).We adapted the PIRLS methodology for use with secondary school children (aged 15-16).In addition, we wanted to find out whether secondary school pupils displayed systematic differences in reading skills in relation to educational level, gender and ethnicity.

Objectives of the PIRLS project
The PIRLS project (Progress in Reading Literacy Study) focuses on the reading level (achievement) and reading experiences of children in 4 th grade (aged 9-10).Every 5 years since 2001, information has been collected on reading behavior, reading attitude and reading proficiency.This is done for a large number of countries, cultures, and languages (52 countries in 2011).PIRLS is designed to help countries improve the teaching and learning of reading.The goal is to provide policy makers and teachers with the information they need to help all students become better readers [2].And indeed, the impact of the international bench-marking of the educational systems in the countries involved has been substantial and has raised a lot of public attention, especially with respect to the effectiveness of local educational systems.A simple Google search for "PIRLS 2011" yields the following quotes, which are quite illustrative for the impact of the project: "States [in the US] with the biggest gaps are ..", "Dutch children perform well (rank 10), but hardly excel", "Best education system Singapore", "Including exam training in curriculum results in higher PIRLS scores".
The PIRLS framework distinguishes two purposes for reading, often associated with a text type: -Reading for literary experience (reading narrative fiction), -Reading to acquire and use information (reading informative articles and instructional texts).
In constructing meaning, the reader focuses on specific ideas, makes inferences, interprets and integrates information and ideas, and examines and evaluates text content.The PIRLS assessment of reading distinguishes four levels of understanding: Level 1 (Low): Recognizing information stated explicitly in the text.Level 2 (Intermediate): Inferring information that follows directly from the text.Level 3 (High): Interpreting and integrating information found in different parts of the text.Level 4 (Advanced): Evaluating textual information in relation with extra-textual sources (world knowledge).

Research questions
Three questions are central to our study.The first two concern our adaptation of the PIRLS approach [2,3].The third is related to the empirical embedding of the instrument [4,5].The first question is whether it is possible to test literary understanding using a short passage from a (larger) novel.The second is whether it is possible for each of the four levels of understanding to cast all items in a multiple choice format.The subsequent and third question is whether secondary school pupils display systematic differences on the test in relation to their educational level, gender and ethnicity.

Sample characteristics
In the spring of 2012, a survey was carried out in which a total of 949 secondary school pupils participated.They all lived in the southern part of the Netherlands and were aged 15 to 18 (mean 16.1 years of age).Participants were recruited in such a way that they were evenly spread over school type (vocational training, grammar school), gender (boy, girl) and ethnicity (native Dutch, immigrant non-Dutch).

Questionnaire
A passage was selected from a novel by A.F.Th.van der Heijden [6], currently one of the foremost literary authors in the Netherlands.It is a one-page passage in which an eye-witness account is given of events taking place in Amsterdam on April 30, 1980 during the coronation of Queen Beatrix.That day there were heavy protests against the shortage of affordable housing, instigated by squatters and resulting in fierce fights with the police.The main character in the novel reports his experiences on passing the Blauwbrug (Blue Bridge).We chose this particular passage because it combines the two text types central within the PIRLS framework, i.e., narrative fiction and informative/instructional text.The questionnaire consisted of four sets of items: one set with the actual test items, one set with text evaluations as a control measure, and two sets with self-assessment questions to provide background information on the test results.These sets are specified in (1) to ( 4).
(1) Testing text understanding For each level of understanding, four (multiple choice) items were formulated, drawn up in accordance with the instructions given in the PIRLS 2011 Item Writing Guidelines [3].Examples are: Level 1: Recognizing information stated explicitly in the text E.g.What color were the uniforms of the police?Black -blue -green -gray.

E.g. What did the main character think of the fighting parties?
The police were fairly rough -the protesters were going to losethe police were going to lose -the protesters were fairly rough.
(2) Evaluations of story and protagonist Students evaluated the text from two perspectives -the story and its protagonist -on 7-points disagree/agree-scales.With regard to the story, the questions focused on three aspects: readability (6 items: I think this story is difficult to read, hard to follow, difficult to understand, clear, wellstructured, transparent), appropriateness (3 items: I think this story is strange, weird, odd), and attractiveness (3 items:I think this story is exciting, thrilling, compelling).For the protagonist, the questions concentrated on the degree of empathy (2 items: I sympathize with the feelings of the main character, I like the main character).For each aspects homogeneity of the scale was sufficient (Cronbach's alpha .70 or higher).
(3) Self-assessment of performance on the test Students were asked to estimate the percentage of items they thought they had answered correctly, and to indicate on a 7-points disagree/agree-scale how easy they found the test to complete.

Level 2 :
Inferring information that follows directly from the textE.g.What made the police look like robots?Their masks -their shields -their uniforms -their marching Level 3: Interpreting and integrating information found in different parts of the text E.g.What showed that the police felt threatened?They withdrew back to back -they took a straddle-legged positionthey put on gas masks -they speeded up their assaults.Level 4: Evaluating information in relation with extra-textual sources (world knowledge)