The impact of PISA in teaching practices in Portugal : The case of Portuguese L 1

Portugal has participated in PISA since 2000 and the national results in reading literacy have been very poor. To improve these results, Portuguese authorities implemented an impressive exam model that has strongly affected teaching practices in Portuguese L1 classrooms. To understand some positive and negative effects of this model we will compare two actual classroom tests, used in 1992 and 2011, and will summarize the results of a report about the relation between exams results and student performance at university. The effects of the Portuguese exam model fostered by PISA seem to call for a study about the incentives PISA has been directly or indirectly encouraging in participant countries. Introduction The primary reason for the implementation of PISA “is to provide empirically grounded information which will inform policy decisions” of participant countries in order to prepare “their students to become lifelong learners and to play constructive roles as citizens in society”. National results are supposed “to provide direction for schools’ instructional efforts and for students’ learning as well as insights into curriculum strengths and weaknesses. Coupled with appropriate incentives, they can motivate students to learn better, teachers to teach better and schools to be more effective” [3: 7]. Since 2000, PISA has been assessing cross-curricular general skills, disciplinary skills and curriculum content in reading literacy, mathematical literacy and scientific literacy. Reading literacy, the one that more closely matters for the case of Portuguese L1, is “the capacity to develop interpretations of written material and to reflect on the content and qualities of texts” [3: 9]. Unlike science and mathematics, reading literacy “does not have any obvious “content” of its own” [3: 12] but its disciplinary skills are mostly cross-curricular general skills. Portugal has participated in PISA surveys since its first edition in 2000. The national results in reading literacy have been deeply disappointing. In spite of a small improvement between 2000 (470) and 2009 (489) ratings, the results are still below PISA average and they recessed one position in the OECD international ranking, from 26th to 27th. These overall results have been having a paramount impact in assessment policies and therefore in teaching practices. During this decade, Portuguese authorities have implemented a huge examination apparatus that has caused slight improvements in results but seems not to promote lifelong learning. a e-mail: paulo.feytor@ese.ips.pt This is an Open Access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Article available at http://www.shs-conferences.org or http://dx.doi.org/10.1051/shsconf/20151601003 SHS Web of Conferences To better understand those changes, this brief study unfolds in three steps: an outline of the assessment framework one decade before PISA and one decade after it (1-2); a comparison of two actual classroom Portuguese L1 tests answered by 7-graders in 1992 and 2011 (3); and the results of a research lead by the University of Porto about the relation between national matriculation exams’ results and students performance at university (4). 1. Assessment before and after PISA Ten years before the publication of the national results in PISA’s first edition, in 2002, there were no national or local assessment standards whatsoever. These depended on each teacher’s interpretation of the curriculum and its relation to the marks range available. Between 1989 and 1993, there was a general matriculation exam assessing L1 proficiency and cultural knowledge. It was the only national exam in the whole school system and it was compulsory to all candidates to university. After three years without exams, they were resumed in 1996. Since then all matriculates have three disciplinary exams at the end of secondary school: one in Portuguese L1, compulsory for all, and two exams in the specific subjects of their choice. Also in 1996 were published the results of the first National Literacy Survey, inspired by OEDC’s International Adult Literacy Survey (IALS) that Portugal would integrate in 1998 [2: 3]. Ten years after PISA assessment of Portuguese L1 has strikingly changed. Exams have been gradually introduced in grades 4, 6 and 9, only for Portuguese L1 and Mathematics. First the exams results were not considered in the pupils’ final mark and only a national sample was tested. Currently, the exams season goes from mid-May to mid-July, including two exams for each subject and grade (1st call and 2nd call), besides the matriculation exams in 21 subjects. In the whole school system, with a population of 1.360.000 pupils, around 340.000 of them answer to 33 different exams, summing up to circa 750.000 centrally controlled tests, every year. The results of the exams are published in the media where schools are ranked according to their pupils’ results and these results are compared with continuous assessment results emphasizing that good schools should have the same result in continuous assessment and exams. Besides, school funding is conditioned by exams’ results. This overwhelming examination apparatus with wide public support and just some opponents has obviously had an impact in teaching practices. Changes also occurred in continuous assessment. Now, every local group of schools – they comprehend all grades, from kindergarten to grade 12 – has to set its assessment standards for every syllabus subject. In Portuguese L1 they are usually divided into five topics: attitudes and behaviour; oracy; reading; writing and grammar or language awareness. Due to national rankings pressure local school standards have gradually become similar to exam standards for writing and similar to both PISA and exam standards for reading. 2. Current reading and writing assessment standards Reading and writing literacy relates to the understanding and the production of written material. According to PISA [3] the features of this material, hence the strategies to read and write it, depend on three factors: (i) the situation or context of use of the text – private, public, occupational or educational –, (ii) the type of text – continuous or non-continuous – and (iii) the instruction given by questions. PISA reading assessment measures five aspects associated with the full understanding of a text: broad understanding, retrieve information, develop an interpretation, reflect on content and reflect on form. Figure 1 organizes these aspects and shows the relationships between them. Since PISA does not assess writing literacy, the standards used in national exams have become the general reference for Portuguese L1 testing. The Ministry of Education (MOE) set seven criteria to assess texts written by the students during exams: subject and genre; coherence and relevance of information; structure and cohesion; morphology and syntax; vocabulary; spelling; and size [1].


Introduction
The primary reason for the implementation of PISA "is to provide empirically grounded information which will inform policy decisions" of participant countries in order to prepare "their students to become lifelong learners and to play constructive roles as citizens in society".National results are supposed "to provide direction for schools' instructional efforts and for students' learning as well as insights into curriculum strengths and weaknesses.Coupled with appropriate incentives, they can motivate students to learn better, teachers to teach better and schools to be more effective" [3: 7].
Since 2000, PISA has been assessing cross-curricular general skills, disciplinary skills and curriculum content in reading literacy, mathematical literacy and scientific literacy.Reading literacy, the one that more closely matters for the case of Portuguese L1, is "the capacity to develop interpretations of written material and to reflect on the content and qualities of texts" [3: 9].Unlike science and mathematics, reading literacy "does not have any obvious "content" of its own" [3: 12] but its disciplinary skills are mostly cross-curricular general skills.
Portugal has participated in PISA surveys since its first edition in 2000.The national results in reading literacy have been deeply disappointing.In spite of a small improvement between 2000 (470) and 2009 (489) ratings, the results are still below PISA average and they recessed one position in the OECD international ranking, from 26 th to 27 th .These overall results have been having a paramount impact in assessment policies and therefore in teaching practices.During this decade, Portuguese authorities have implemented a huge examination apparatus that has caused slight improvements in results but seems not to promote lifelong learning.

SHS Web of Conferences
To better understand those changes, this brief study unfolds in three steps: an outline of the assessment framework one decade before PISA and one decade after it (1-2); a comparison of two actual classroom Portuguese L1 tests answered by 7-graders in 1992 and 2011 (3); and the results of a research lead by the University of Porto about the relation between national matriculation exams' results and students performance at university (4).

Assessment before and after PISA
Ten years before the publication of the national results in PISA's first edition, in 2002, there were no national or local assessment standards whatsoever.These depended on each teacher's interpretation of the curriculum and its relation to the marks range available.Between 1989 and 1993, there was a general matriculation exam assessing L1 proficiency and cultural knowledge.It was the only national exam in the whole school system and it was compulsory to all candidates to university.After three years without exams, they were resumed in 1996.Since then all matriculates have three disciplinary exams at the end of secondary school: one in Portuguese L1, compulsory for all, and two exams in the specific subjects of their choice.Also in 1996 were published the results of the first National Literacy Survey, inspired by OEDC's International Adult Literacy Survey (IALS) that Portugal would integrate in 1998 [2: 3].
Ten years after PISA assessment of Portuguese L1 has strikingly changed.Exams have been gradually introduced in grades 4, 6 and 9, only for Portuguese L1 and Mathematics.First the exams results were not considered in the pupils' final mark and only a national sample was tested.Currently, the exams season goes from mid-May to mid-July, including two exams for each subject and grade (1 st call and 2 nd call), besides the matriculation exams in 21 subjects.In the whole school system, with a population of 1.360.000pupils, around 340.000 of them answer to 33 different exams, summing up to circa 750.000 centrally controlled tests, every year.The results of the exams are published in the media where schools are ranked according to their pupils' results and these results are compared with continuous assessment results emphasizing that good schools should have the same result in continuous assessment and exams.Besides, school funding is conditioned by exams' results.This overwhelming examination apparatus with wide public support and just some opponents has obviously had an impact in teaching practices.
Changes also occurred in continuous assessment.Now, every local group of schools -they comprehend all grades, from kindergarten to grade 12 -has to set its assessment standards for every syllabus subject.In Portuguese L1 they are usually divided into five topics: attitudes and behaviour; oracy; reading; writing and grammar or language awareness.Due to national rankings pressure local school standards have gradually become similar to exam standards for writing and similar to both PISA and exam standards for reading.

Current reading and writing assessment standards
Reading and writing literacy relates to the understanding and the production of written material.According to PISA [3] the features of this material, hence the strategies to read and write it, depend on three factors: (i) the situation or context of use of the text -private, public, occupational or educational -, (ii) the type of text -continuous or non-continuous -and (iii) the instruction given by questions.
PISA reading assessment measures five aspects associated with the full understanding of a text: broad understanding, retrieve information, develop an interpretation, reflect on content and reflect on form.Figure 1 organizes these aspects and shows the relationships between them.
Since PISA does not assess writing literacy, the standards used in national exams have become the general reference for Portuguese L1 testing.The Ministry of Education (MOE) set seven criteria to assess texts written by the students during exams: subject and genre; coherence and relevance of information; structure and cohesion; morphology and syntax; vocabulary; spelling; and size [1].

Portuguese L1 classroom testing
The impact of the exam apparatus, gradually implemented since PISA first came to place, is apparent in classroom tests.Appendix 1 is a test created in 1992 by a group of teachers to assess reading, grammar and writing by 7-graders in the beginning of the school year.It deals with two texts, one to read and one to write.Both texts are private and continuous.The first is an excerpt of a renowned children's narrative (157 words), the second a description of a fictional situation (8-10 lines).Four questions assess four aspects of reading literacy: retrieve information, develop an interpretation, reflect on content and reflect on form.These reading aspects though are assessed through written answers of the students.The writing task is assessed according to four criteria: subject and genre; coherence and relevance of information; morphology and syntax; and size.There are four items to assess language awareness.The marks for each question or criterion are not shown.
Two decades later, classroom tests seem to have improved in several different aspects.Appendix 2 is a test created in 2011 by a single teacher to assess reading, grammar and writing by 7-graders at the end of the first term.It deals with three different types of texts: one public and non-continuous to read, one public and continuous to read and one occupational and continuous to write.The first one is a newspaper cover page, the second is a newspaper news (141 words) and the third is a report about a specific event the students attended in the school library (140-180 words).Reading literacy is assessed both through writing and only transcription.Three transcription questions assess retrieving information.There are another three questions to assess this aspect through written answers.The same kind of written task is asked to measure broad understanding, interpretation and reflect on form.To assess writing skills, all seven criteria set by the MOE for exams are used.There are six items about grammar.The marks for each question or criterion are clearly shown.
Between the 1990's and the 2010's, Portuguese L1 classroom tests tended to become richer and more accurate.They now deal with a wider variety and quantity of texts, reading assessment is separate from writing, there are more questions about reading and grammar, and writing literacy is measured in longer texts by seven explicit criteria.All marks are now shown.

Exams' results and students performance
Since school rankings based on exams results have first been published, in the early 2000's, the media and the public has enhanced the actual difference between public and private schools.Public school students' results have always been lower than the results of their peers from private schools.These facts have promoted a lively discussion about education quality in Portugal.In this context, the University of Porto (UP) made a report comparing its students' performance in 2011 with the matriculation results they had in 2008.The study aggregates all 4280 UP matriculates according to the secondary school of origin where each student has answered three exams in 2008, including Portuguese L1.Around one fifth of these students come from private schools.Figure 2 shows the school ranking by exam results.Among the 12 best schools only two are public and the first of these two is the 7 th in the ranking.It confirms the national pattern of exam excellence of private schools.
However, after three years of higher education, considering those students who got more than 75% of ECTS expected for that period, the ranking by school of origin is completely different.Figure 3 shows the ranking by university performance between 2008 and 2011.The best performing UP students mostly come from public schools.Only three private schools are among the 12 best.Furthermore, within each group, schools are ranked in the opposite order when compared to Fig. 2: Garcia Orta is better than the absent Aurelia Sousa; Rosario is worse than Paulo VI and Lamas, the best private in UP performance is not among the 12 best in exam success.The second ranking actually looks like an inverted picture of the first one showing a very weak relation between matriculation exam success and good university performance.In other words, students that perform well in exams are not the best students at university.These results reveal a very negative impact of the exam apparatus in teaching practices namely in Portuguese L1.Because of the strong official and public demand for good exam results teachers tend to centre classroom activity both in exam-like tasks and in the part of the curriculum assessed in exams.Oracy, research or collaborative work are hence excluded and a lot of time is spent in answering exams of previous years.This tendency is more acute in private schools since only these can be chosen by parents -public schools are (supposed to be) attended by youngsters of their circumscription.For both public and private schools, the MOE prepares every year intermediate optional exams either in intermediate grades without exams or to be answered some months before the actual exams.This policy has made 01003-p.4some schools become "exam training camps", as opponents put it.The ultimate example is a school in the Algarve, considered the best school of the region in 2010 because of its excellent results in exams.The only extra-curricular activity they have is "exam training".Every now and then, the whole school stops classes for pupils to simulate an exam: formal call, one student in each table, official answer sheets, two supervising teachers of a different discipline and, of course, an exam-like test to answer.

Final remarks
Since the beginning of the 21 st century, Portuguese authorities, under conservative and socialist governments, have implemented a huge exam apparatus.This apparatus seems to be directly connected with the urgent need to improve national results in PISA reading literacy ratings.For the public, exams are an effort to improve literacy quality that should reflect on PISA.However, national results have only slightly improved so far and exams are not able to accurately promote lifelong learners.
During the decade, Portuguese L1 teaching practices have been affected by this impressive assessment model.They gradually became PISA-oriented.Some positive new practices have been introduced.Classroom tests now have a wider variety of texts, reading literacy is assessed without writing and there are local assessment standards based on a national framework.But negative new practices seem to put education quality at risk.A much narrower curriculum, both of knowledge and skills, caused by focus on exam-like tasks and by continuous assessment based in exam-like tests.
These effects question whether Portugal and/or PISA are actually promoting the appropriate incentives to provide the best direction for students, teachers and schools.In PISA's words, is this assessment model motivating "students to learn better, teachers to teach better and schools to be more effective"?After five editions or cycles of PISA assessment now there seems to emerge a need for some kind of OECD assessment of the processes it has been directly or indirectly causing in each different participant country.The primary reason for the implementation of PISA is to provide information that should affect policy decisions.So far data provision has been delivered.We now lack information about policies influenced by that information.

Figure 3 .
Figure 3. Best schools of origin of students: university success [4].