Multiple-choice questions and essays in assessing economics

. The article is devoted to assessment tools in teaching economics. The authors distinguish and define four levels of understanding economics: elementary, intermediate, systemic and creative. They describe multiple choice questions and essay questions as two possible assessment tools in teaching economics. Multiple-choice questions are represented as the most popular testing format. The advantages of multiple-choice questions include low grading costs, perceived objectivity and availability of comparative analysis. The authors have developed multiple-choice tests, which measure students’ knowledge at three first levels of understanding economics. They enable instructors to see where exactly the students’ understanding has stopped and provide guidance. The authors conclude that multiple-choice questions can be used to measure the basic levels of students’ understanding economics. In measuring higher levels the essay as an assessment tool has a great potential. The authors highlight the advantages and pitfalls of essay testing in economics.


Introduction
In today's world, there are more and more requirements for the content of education, forms of its implementation, pedagogical technologies and teaching methods, as well as for modern means of evaluating learning outcomes. All classifications of teaching methods involve testing students' knowledge, evaluating their activities and correcting them if necessary.
Assessment is one of the most relevant issues, both in pedagogical theory and in pedagogical practice. The teaching of economics requires of the instructor choosing the assessment tools that could measure the level of students' understanding. The results of the assessment provide the teacher with feedback, they also help to plan, adjust and improve the teaching and achieve the learning outcomes.
The purpose of the study is to distinguish and define the levels of economic training and to develop assessment tools taking into account the proposed levels.

Literature review
The need for standardised procedures for objective measurement of students' educational achievements is stressed in the studies of V. Avanesov [1], V. Zvonnikov, M. Chelyshkova [2], O. Katurzhevskaya [3] et al. Special aspects of the assessment materials development, as well as methods and tools for evaluating students' learning outcomes are described in the works of Ye. Alegushina, M. Prokhorova [4], O. Abramova, M. Gladkova, O. Vaganova [5] et al.
Assessment in teaching economics is based on the achievements of pedagogical science and takes into account the features of the subject. In research of W. Walstad and W. Becker [6], W. Walstad [7; 8], N. Chan and P. Kennedy [9], P. Kennedy and W. Walstad [10] the advantages and disadvantages of various assessment tools used in economics are considered.
Researchers apply various taxonomies to assess students' understanding. For example, Santos and Lavin have used a two-level system to categorize economic understanding: deep learning and surface learning. They define deep learning as not only a command of the subject matter but also an ability to think independently; to determine necessary information; to provide a persuasive substantiation of claims; to organize thoughts clearly; and to use precise, correct, and effective language [11]. Some researchers, for example, S. Buckles and J. Sigfrid use B. Bloom's cognitive taxonomy, which identifies six cognitive levels of thinking. Knowledge questions test whether a student can select the best definition of a given economic term or concept. Comprehension questions ask students to explain the reasoning behind an economic concept or principle. Application questions ask students to identify economic concepts needed to solve a particular problem, to distinguish between correct and incorrect application of economic concepts. Analysis questions are aimed at students' linking essential economic concepts and principles and identifying causal steps underlying economic events. Synthesis is an ability to produce something original based on one's experience and understanding. Evaluation is making a recommendation or justification based on a specified set of criteria [12]. The above-mentioned researchers also consider that in-depth understanding includes three types of skills: an ability to identify economic concepts; an ability to apply concepts and an ability to link logically.

Assessment in teaching economics
We think that assessment can be carried out on different levels. We distinguish four levels of understanding economics: elementary, intermediate, systemic and creative.
The elementary level is determined by the student's knowledge of economic concepts, definitions, facts, laws and classifications. This amount of economic knowledge is manifested in the student's ability to reproduce economic concepts, definitions, facts, laws and classifications.
The intermediate level of economic proficiency requires knowledge of formulas, rules and functional dependencies. It is reflected in the student's ability to apply formulas in practice, build graphs, tables, schemes and diagrams.
The systemic level indicates the student's understanding of cause-and-effect relations between economic phenomena and processes, knowledge of alternative points of view on solving economic problems and hands-on examples of real economic processes. This body of economic knowledge is expressed in students' ability to use their analytical skills in building cause-and-effect relations between economic phenomena and processes, to draw conclusions in various socio-economic situations, as well as to formulate their answers correctly. The creative level is accessible mainly to students who are fluent in educational material and know additional information about the subject under study. This level of economic knowledge implies the students' ability to form their own opinion on the most relevant and controversial economic issues. The student's amount of economic knowledge allows him or her to learn how to find original ways out of non-standard economic problems, to offer solutions to problematic situations and to forecast the development of economic phenomena and processes.
Most classes in economics use both multiple-choice questions and essays as assessment tools. Multiple-choice questions are used frequently in large introductory economics courses. W. Becker and M. Watts reported that, on average, about 45 percent of the grade in a typical introductory course is based on multiple-choice questions [13].
Multiple-choice questions are the most popular testing format in economics. They are useful as a diagnostic tool since students' wrong answers can indicate misunderstanding. They allow a large class and a lot of material to be tested in a short time and they are easy and quick to grade. Multiple-choice questions have several disadvantages. It is difficult and time-consuming to construct effective distracters. Sometimes they encourage students to find the answer by eliminating or guessing. Multiple-choice questions can measure some levels of students' understanding but they cannot assess higher levels of cognition. W. Walstad underlines that multiple-choice items do a reasonably good job of testing for students' understanding at the knowledge and comprehension levels because they provide a reliable and efficient way to measure understanding. But they can also be used to measure simple achievement related to application and analysis. S. Buckles and J. Siegfried argue that multiple-choice questions can be used to test students' achievement up through Bloom's level four, analysis.
To assess students' knowledge of different topics we have developed multiple-choice tests, each consisting of 45 items and measuring knowledge at three levels (elementary, intermediate and systemic).
The following set of questions illustrates how a series of questions on different topics can be used to assess a student's level of understanding. Example 1. Topic "Elasticity and its application". 1. The price elasticity of demand can be defined as: a) the percentage change in quantity demanded in response to one percent change in price; b) the percentage change in price in response to one percent change in quantity demanded; c) the change in quantity demanded in response to change in price; d) the change in price in response to change in quantity demanded. The first question asks students merely to define elasticity. It makes students recognize a definition and measures the elementary level.
2. The price elasticity of demand is higher if: a) the product has more substitutes; b) the product takes a smaller share in the consumer's expenses; c) the product is new; d) the product is more important for the consumer.
Giving an answer to this question students demonstrate that they know the factors that affect price elasticity of demand and understand the dependence of elasticity on a specific factor.
3. Due to the drought, the production and supply of coffee decreased, but the total revenue of coffee producers increased. This can be explained by: a) the demand for coffee is elastic; b) the demand for coffee is inelastic; This question requires not only knowledge of the factors influencing the supply, but also an understanding of the relationship between elasticity and total revenue. When answering, it is necessary to understand that with a decrease in supply, the price will increase, while the total revenue of producers will also increase. This situation is possible if the demand for the product is inelastic. The question requires a deeper understanding of the information studied. Because this question requires a number of logical steps, it may test systemic level.
Example 2. Topic "The costs of production".

If output increases in the short-run, average fixed cost: a) stays constant; b) increases; c) decreases; d) increases or decreases.
The first question asks students merely to define the average fixed cost. It makes students recognize a graph illustration.

If marginal cost of production increases, average variable cost: a) increases; b) decreases; c) stays constant; d) increases or decreases.
Giving an answer to this question students demonstrate that they understand the relationship between curves. When marginal cost (MC) increases, average variable cost (AVC) can be either increase or decrease. Marginal cost begins increasing, but average variable cost decreases, it happens because MC is below AVC and pulls AVC down. Then MC cuts through the minimum point of AVC, and AVC begins to rise because MC is above it.

The firm is in the short-run, the labor is the only variable input to the production process and it is purchased in a competitive market at a constant market price. With a small increase in output marginal costs increases. In this case: a) average product of labor increases; b) marginal product of labor decreases; c) average variable cost decreases; d) average total cost increases.
Answering this question, students need not only to have an idea of the graph illustration of average and marginal cost, average and marginal product. It is also important to understand the relationship between marginal cost and average and marginal product of a variable input. At a fixed wage rate, there is a strictly inverse relationship between marginal cost and marginal product of labor. The maximum of marginal labor productivity coincides with the minimum of marginal costs. Consequently, as marginal cost increases, the marginal product of labor begins to decrease.
A series of questions assessing first the elementary, then the intermediate and finally the systemic levels may enable instructors to see where exactly the students' understanding has stopped and provide guidance as to what to emphasize in review. The results obtained allow the instructor to adjust and improve the teaching and achieve the learning outcomes.
To decrease the process of elimination or guessing the correct answer, students may be asked to choose the correct option and then to explain why each incorrect answer is wrong. One additional use of multiple-choice questions is to distinguish distracters that are correct all the time and that are correct under some circumstances from those that are never correct. Multiple-choice questions can be constructed with distracters aimed at students who cannot apply an appropriate concept or who miss some logical steps. The results obtained can be used to identify the most common misunderstandings. An example is the following: If nominal GDP grew by 6% and real GDP grew by 2% over the year, the rate of inflation is: If students do not apply the concept of GDP deflator properly or miss particular steps, they often choose the first or the last option, which are not correct.
Multiple-choice questions give the teacher information about correct and incorrect answers but they do not say anything about students' confidence. We are not sure whether a student with high score was confident or had low confidence in chosen options. J. Bruno [14] used multiple-choice questions where students were asked to point the degree of confidence in choosing one or more answers. The results of this assessment allow teachers to see whether the students are correct, incorrect in their understanding or have no knowledge.
Multiple-choice questions scoring can be different. Students can be given full credit for the correct answer only and zero credit for an incorrect choice. It is also possible to give students partial credit for an appropriate but incomplete option. Partial credit applying can differentiate students who have the ability to use the concepts from those who are able to give a definition or recognize the necessary concept. But it is time-consuming for the instructor to develop these questions. Partial credit grading should include questions with answers which are correct all the time or correct under some circumstances. For example: To maximize profit the firm should have the level of output where: a) price equals marginal cost; b) total cost is minimized; c) marginal revenue equals marginal cost; d) price equals average total cost. Any firm wanting to maximize profit should have the output where marginal revenue is equal to marginal cost. The third option will give a student full credit as the correct answer for all market structures. Students may also identify perfect competition as a special case where marginal revenue equals the market price because firms are price takers. So, the first option is correct under perfect competition and might be granted partial credit for an appropriate but incomplete option. The second and the forth answers are incorrect options.
So, multiple-choice questions can measure some levels of understanding. However, W. Walstad underlines that Bloom's highest levels of understanding, synthesis and evaluation, cannot be accurately measured with multiple-choice questions. S. Buckles and J. Siegfried argue that synthesis and evaluation (Bloom's fifth and sixth levels of understanding) are important additional elements of in-depth understanding and these elements may be more efficiently assessed with essays. An essay question is better for measuring complex achievement related to the application of concepts, analysis and synthesis of problems, or evaluation of decisions. We apply essay questions to test the creative level.
Many economics instructors consider the essay to be one of the easiest methods for testing the depth of economics content understanding. The chief advantage of the essay test is the freedom of response. It means that students have an opportunity to choose, prepare, and present ideas in their own words in response to essay questions. The instructor has an opportunity to see the student making a response and not just selecting the best answer from the options suggested. An essay asks students to select, organize and integrate economics SHS Web of Conferences 99, 01032 (2021) DIHELT 2021 https://doi.org/10.1051/shsconf/20219901032 material to construct a response. . It encourages students to study in a more integrated manner, they are given more opportunities to show creativity and originality. But there are some potential pitfalls of using the essay. One serious problem of the essay is that it may not be a reliable measure of students' achievement. Ease of test construction, which is often considered to be as an advantage, can contribute to the unreliability of essay scores if the questions are stated incorrectly, so answers to such questions may not be accurately graded. For example, "Describe the ways of state budget deficit financing and the consequences of their application" is a vague question because this task does not give the student a clear idea of what methods should be considered, what consequences should be characterized, whether it is necessary to conduct a comparative analysis of various options. This example of an essay task leads to responses that have too wide a range of content, or do not contain answers that, in the opinion of the instructor, should have been presented. So, the essay should be constructed correctly to measure higher cognitive levels. W. Walstad underlines, that questions using the following stems are more likely to produce such learning outcomes: − comparing and contrasting; − explaining or anticipating the possible effect; − summarizing from a stimulus (for example, a quote, a table, a graph, or a model); applying an economic concept or principle; − analyzing an economic problem; − evaluating an economic decision using established criteria or theories. For example, the above essay task can be written as follows: "Let's assume that the state budget deficit is financed partly by loans on the domestic market, partly by issuing money and partly by foreign loans. Think about how this might affect interest rate, investment, exchange rate and net export compared to the situation of financing the deficit only with domestic loans". Doing this task, students understand what ways of financing budget deficit and what consequences they should take into consideration. They are also asked to compare various options. Carefully defining the test outcomes and framing clear questions to measure those outcomes can improve the reliability of the essay as a measure of student achievement.
The time for scoring essays is a second problem, because grading essays requires reading one or more times and writing comments on the responses. It is also difficult to standardize grading because of variability of answers. This work is time-consuming. N. Chan and P. Kennedy underline that essays are more expensive to grade than multiplechoice questions, so highly structured free-response questions may generate incremental costs without commensurate benefits. S. Buckles and J. Siegfried argue that essays give opportunities to observe a student's ability to create a chain of reasoning, a wellconstructed multiple-choice question can measure much of the same skill.
The third disadvantage with essays is limited sampling of the domain content. A typical multiple-choice test can include items covering a wide range of content. By contrast, so few questions which are included in an essay may have a limited content coverage.
So essay items require more instructors' time and work, but they are used as an effective and reliable measurement tool for the depth of understanding in economics.

Conclusion
Assessment is an integral part of the educational process, allowing both the instructor and the students to track progress in achieving the learning goals. The article considers multiple choice questions and essays as assessment tools. The advantages of good multiple-choice questions include low grading costs, perceived objectivity, and availability of comparative analysis, but sometimes they encourage students to find the correct answer by process of elimination or guessing. Higher levels of students' understanding cannot be accurately SHS Web of Conferences 99, 01032 (2021) DIHELT 2021 https://doi.org/10.1051/shsconf/20219901032 measured using multiple choice questions. Essay questions may be able to assess levels of understanding that are beyond the reach of multiple-choice questions. Essay questions take less effort to create, yet are more costly to grade. However, when choosing an assessment tool, various factors should be taken into account: the expected learning outcomes, the opportunity to cover a wide range of content, the probability of guessing the answer option, the ease of developing tasks and grading results and the ability to assess high levels of understanding.