A FAHP approach to select students ’ performance assessment criteria in task-based English language teaching

Assessing students’ performance is a main means to improve students’ learning and can play an important role in strengthening the evaluation process in task-based language teaching. This paper adopts a novel approach to select students’ performance assessment criteria based on fuzzy AHP method. Specifically, after determining dimensions and sub-dimensions of learning, the students’ performance assessment instrument can be established. In the instrument, the dimensions and sub-dimensions can be selected by experts’ opinion. Employing the fuzzy AHP method in group decision-making can facilitate a consensus of decision-makers to reduce uncertainty. The application of this method can make the assessment results more scientific, accurate and objective. It is expected that this work may serve as an assistance tool for members of higher education institutions in improving the educational quality level.


Introduction
Assessing students' performance is an important aspect of learning and teaching, which includes several components employed by instructors to draw conclusions about their achievements (Ma & Zhou, 2000).In task-based approach, this component is more complex than standard language assessment as it requires observation in real-world context of students' language and communicative ability (Bachman 2002;Weigle 2002).The aims of assessing performance is to improve students' learning processes, motivation, self-esteem and thinking (EL- Koumy, 2004) that, preferably lead to productive citizens.In student-centred learning environment, language instructors prefer using criterion referenced techniques to assess the performance of students.In addition, these criteria and assigning weightage to these criteria is determined by language instructors themselves (Ma & Zhou, 2000).
Many frameworks have been suggested over the years on assessing performance of students.A framework provided by Ellis (2003) focuses on task evaluation based on the following three criteria: 1) student-based, which only measures the extent to which the task was useful and enjoyable; 2) response based, which compares actual task outcomes with predicted outcomes; 3) and learning-based evaluation, which measures the level of actual learning when the task is completed.However, these evaluations are limited to measuring teaching effectiveness which only determines whether a task was meaningful, useful and enjoyable.American Council for Teaching Foreign Languages scale (ACTFL) early proficiency scales were limited to describing learner behaviour (Bachman, 2002).Therefore, the current assessment models which are based on vague and ambiguous criteria are likely to limit language instructors understanding how the characteristics of task affect students' performance (Bachman, 2002).Apart from aforementioned studies, other related researchers focused on measuring abstract and psychological qualities related to the performance such as aptitude, motivation and language proficiency (Nunan, 2004).In other words, there is a need to address those aspects of students' learning performance which are important but missing and may play an important role to make judgments about their performance.Hence, to determine the quality of students' performance, using a scientific method plays an important role to draw true picture of their performance.Therefore, the main purpose of this paper is to explain the process and steps involved to select and prioritize the assessment criteria according to their relative importance.
Assigning weight to the factor is a multi-criteria decision-making problem and giving judgments in linguistic variables is more convenient for decision-makers rather than giving in the form of numerical values (as cited in Lee, Chen & Chang 2008).

_______________________________
In recent years, several researchers have focused on assessing students' performance.To produce 'fair' mark distribution, (Echauz & Vachtsevanos, 1995) presented a fuzzy grading method that utilizes students' and instructors performance measure.Kahraman (as cited in Lee et al., 2008) proposed a methodology based on FAHP to improve the quality of prioritization of human capital measurement indicators under fuzziness.The studies discussed above revealed the high applicability of fuzzy set theory.Therefore, to deal with imprecise and uncertain data, Zadeh introduced fuzzy set theory in 1965 (as cited in Lee et al., 2008) and is used in this paper to determine the dimension and sub-dimensions weights.It is an effective method to represent the uncertainty and fuzzy terms in the assessment environments (Ma & Zhou, 2000).
The reminder of the paper is organised as follows: Section 2 presents Zadeh fuzzy set theory and its application in the development of the instrument.Section 3 discusses the essential steps and sub-steps includes in the establishment of the performance assessment instrument.Section 4 presents an example of the instrument; and finally, the conclusions are given in section 5.

Fuzzy Set Theory
Fuzzy set theory was introduced to solve problems when the criteria is not clearly defined (Zadeh, 1965).To avoid misleading results, it is very important to consider fuzziness in human decision making.And this theory has been used in different fields to solve such kind of problems for last four decades.The two distinct directions of this theory are: treating fuzzy sets as precisely defined mathematical objects subject to the rules of classical logic, and the linguistic approach.The logic behind linguistic approach is that the truth values are fuzzy sets and rules inferences are approximate rather than exact.
A triangular fuzzy number is a special case of trapezoidal fuzzy number, is very popular in fuzzy applications.As shown is Figure 1, the triangular fuzzy number M is represented by (a,b,c) and memberships function is defined as With − ∝ <  ≤  ≤  < ∝ The strongest grade of membership is parameter b, that is   () = 1 , while a and c are the lower and upper bounds.An important step of fuzzy sets is the α-cut.For a fuzzy number M and any number ∝ ∈ [0,1], is the crisp set (Khr & Yan, 1995): (2) The αcut of a fuzzy number m is the crisp set M that contains all the elements of the universal set U whose membership grades in m are greater than or equal to the specified value of a, as shown in fig. 2. By defining the interval confidence at level, a, the triangular the triangular fuzzy number can be characterized as (Cheng,1999;Cheng 1996;Cheng and Mon, 1996); (3) The distance between the 2-triangular fuzzy number can be defined by the vertex method (Chen, 2000).Let  ̃1 = ( 1 ,  1 ,  1 )and  ̃2 = ( 2 ,  2 ,  2 ) be two triangular fuzzy numbers, the distance between them is (4) Different methods have been proposed to rank fuzzy numbers and each has its own advantages and disadvantages (Klir & Yan, 1995).Since the conventional method only used numerical equivalence method, that is why linguistic approach is used to consider any ambiguity when decision makers rate using linguistic values whose membership functions are usually characterized by triangular fuzzy numbers (as cited in Lee et al., 2008).

Steps involved in instrument development
In this paper, six perspectives of performance assessment dimensions and sub-dimensions are extracted from extensive review of literature.The next step is to get experts opinion to select final list of dimension and sub-dimensions.They can add or remove any perspective in the final list.According to Adler and Ziglio (1996) the selection of experts should be based on four 'expertise' requirements: I) knowledge and experience with the issues under investigation; ii) capacity and willingness to participate; iii) sufficient time to participate in the study; iv) effective communication skills.In the construction of framework, Expert Choice, 2006 software will be applied.Different software packages have been applied to solve AHP problems however, expert choice will be used in this research.

Data Collection
A conventional AHP method will be used to design a questionnaire based on nine-point Likert scale to select and then to finalize the selection by making pairwise comparisons.The questionnaire will then be distributed among forty experts to get their opinions.There are two phases of the data collection: In the first phase, experts will select the dimensions and sub-dimensions and in second phase, they will make pairwise comparisons.After the selection of dimensions and subdimensions and pairwise comparisons, the last step is to obtain final ranking of each dimension and sub-dimension based on their importance.The fig. 3. below shows the dimensions and sub-dimensions extracted from the review of literature to develop performance assessment tool.These dimensions and sub-dimensions are selected from the studies by (Bachman, 2002), (Nunan, 2004), Ellis (2003) and Norris (2001) to develop performance assessment tool.However, the final selection for developing the assessment tool will be based on expert's opinion.