In multiple-choice questions, only one of the choices is correct, while in multiple-selection questions the participant can select several of the available choices. The selection, then, can be graded as one complex answer or as individual answers to the same question. SAGA supports only multiple-choice questions.

In a closed-type multiple-choice question, all available choices are predefined by the quiz-creator. On the contrary, in an open-ended question, the participant is free to add his/her own text as an answer. SAGA supports only closed-type multiple-choice questions. However, if you decide to use justifications in your quizzes, then the students will be able to justify their answers by writing text. In other words, the quiz questions will be closed-type (e.g., “What year was the University of Twente established? A: 1960; B: 1961; C: 1963; D: 1964”), but the accompanying justification question will be an open-type one.

• The same question is posed to all students. This means that all students will have to reflect on the question and provide an answer. The alternative would be to ask a question orally to the audience (or to a specific participant), in which case not all students will think of an answer.
• Immediate feedback. With an ARS, it is possible to provide immediate and personalized feedback to all students. Personalized feedback means feedback based on the answer or the progress of the student in the quiz.
• Anonymity provides psychological safety and acceptance. It has been noted in several studies that students are more comfortable answering questions privately in a quiz than answering them orally in front of an audience.
• Increased engagement and motivation. Research findings have shown that in courses where ARSs are used, students pay more attention in the class and tend to have higher attendance rates. Having been engaged with the questions of the quiz, the students are also more likely to then ask or answer questions during the lecture. Finally, students usually find classes more enjoyable and satisfying.
• Students are invested in their answers. Another reported finding of the use of ARSs is that students are more prone to defend their answers in group/classroom discussions.
• ARS are routinely credited for developing critical thinking. A multiple-choice question does not have to be about memory recall. It may very well require the application of critical thinking, knowledge transfer, and creativity. The successful use of an ARS is based heavily on the quality of the questions. And, writing challenging and effective questions is not a trivial task!
• Uncovering preconceptions and assumptions. By getting the answers of the classroom in a quiz, the teacher may be able to evaluate the classroom’s level of understanding and modify instruction accordingly.
• Monitoring students’ progress. With ARSs, it is possible to monitor the progress of a student within a quiz or a series of quizzes. Progress, of course, does not necessarily mean form assessment.
• Low technological threshold for the teacher. An ARS is a rather simple technological tool. Most time will be spent on writing questions of high quality.
• Increased retention and knowledge acquisition. Studies have repeatedly reported that by using ARSs, students tend to remember more about the lectures and the course, in general, and they have better overall performance in the formal assessment of the course.

• Some studies in the literature reported low or non-significant impact on learning. On one hand, the good thing is that there are no studies that show the ARSs impeded students’ knowledge acquisition. On the other hand, there is an ongoing discussion about the potential of ARS and whether the educational gains can be attributed to the tool or the pedagogy. In any case, any technology must be meaningfully integrated into a sound pedagogical approach. Using a tool without some thought of the instructional model and the learning goals the tools should serve would probably lead to shallow learning experiences for the students.
• ARSs are not appropriate for elaboration. There are several higher-level learning goals that are strongly linked with the ability of students to demonstrate or elaborate their answers. Inherently, ARSs are based on closed-type questions and offer little room for elaboration. SAGA’s justification question is an attempt to allow student voice within the confined space of multiple-choice questions.
• Writing challenging and effective questions is not a trivial task!

This is a hard question to fully answer, but there are several guidelines to help you write compelling questions.

• Keep it simple! Both the question and the answers must be clear and easily understood. The challenge should NOT be if the students understand the question.
• Avoid negatives. Questions such as “Which of the following is not correct?” can be tricky for the students as they are trained to focus on the correct answer. If you do need to use negatives, then emphasize negation in the question phrasing (e.g., “…is NOT correct?”).
• Have the question choices with similar length and complexity. Longer and more detailed choices are usually a “tell” of a correct answer. You should not provide hints as this counteracts the purpose of the quiz. In addition, a purposefully detailed wrong choice that tries to trick students is also counterproductive. The goal of the quiz is to objectively identify the level of knowledge and understanding of students. Therefore, it is necessary for all students to have a good understanding of the questions and their choices.
• Spread out the position of the correct answer. It has been a running joke that if you do not know the answer, then you should choose “B” as this is where the correct answer is mostly places. Actually, quiz-creators tend to have the correct answer in the last places (e.g., “C” or “D”) – it is easier to think of a few wrong answers before you write the correct one. Review the quiz and check where the correct answer is placed, but make sure that the choices of a question appear in a meaningful order (e.g., descending or ascending, in case of numbers or dates).
• The question should be meaningful even without the choices. Avoid questions that will continue into the choices (also questions such as “Which of the following is correct?”). This type of phrasing is based on students’ inferential ability and may interfere with their ability to demonstrate their actual level of knowledge.
• The question should not contain irrelevant or tricky information. This decreases the reliability and validity of the quiz. As mentioned earlier, the goal is to assess students’ knowledge and understanding and not their ability to understand the quiz.
• Fill-in-the-blank questions should be avoided. They increase the cognitive load of the students. So, they are more challenging, but for the wrong reasons.
• All answers should appear plausible. The alternative answers should be close to the correct one. A knowledgeable student should be able to differentiate between subtle differences, but a less knowledgeable student should not.
• Less knowledgeable students will go for the most informative answer. This is a very common gaming strategy for students that do not know the right answer. Their assumption is that it would be easier for the teacher to be more precise in the correct answer than in the made-up alternatives. So, make sure that you do not have this “tell” in your questions.
• The question choice “All of the above” should be avoided. The students will need to know that at least two answers are correct to answer the question.
• The question choice “None of the above” should be avoided. If “None of the above” is the correct answer it does not provide information that the students actually know the correct answer to the question. In addition, if “None of the above” is selected, but is a wrong answer, it does not provide accurate measurement of students’ knowledge. The students may actually know that some of the choices are wrong or they have no knowledge at all.
• The number of answers may vary across questions – but it is best to keep it the same. Plausible alternatives act as distractors for less knowledgeable students, so researchers on assessment reported that there is no significant difference in quiz reliability in relation to the number of alternatives. But, it is better to keep the number of choices the same across your quiz because it will be easier for you to provide a homogenous experience to the students.
• Do not provide information in one question that can be used to answer another question. This includes the information given in the choices of a question.
• Use mutlilogical thinking questions to assess higher-order thinking. Multilogical thinking is thinking that requires knowledge of more than one fact to logically and systematically apply concepts to a problem.
• Require knowledge transfer and application to assess critical thinking. Students may need to apply several principles and theories to reach the correct answer. Having a short question does not mean that the students will find the answer easily or quickly. Critical thinking should still lead to only one correct answer.

Depending on the type and purpose of the quiz, you can use it at different times in your course’s timeline to address different learning goals.
In the beginning of a lecture.

• Identify misconceptions.
• Provide feedback and time to the teacher to modify instruction accordingly.
• If combined with a flipped-classroom approach, then this can:
  • Provide an extra motive to the student to engage with the material.
  • Provide the opportunity for students to review material before the lecture.

During a lecture.

• Identify misconceptions.
• Provide feedback to the teacher to modify instruction accordingly.
• Provide the opportunity to students for reflection on what has been presented and discussed so far.

At the end of a lecture.

• Provide feedback to the teacher on students’ understanding of the lecture.
• Provide the opportunity to students for reflection on the whole course.
  • This further enhances retention.

The use of audience response systems in the classroom became a popular technique of instruction because, in large part, of the work of Eric Mazur, a Professor of Physics and Applied Physics at Harvard University. His underlying pedagogy in using ARSs was called Peer Instruction and provided the basis for using ARS in the classroom. While variations of the approach exist now, many tools (including SAGA) are still based on the basic principles of Mazur’s approach.
The Peer Instruction approach of using ARSs in the classroom includes the following steps:

1. The teacher poses a question.
2. The students respond individually.
3. Aggregated feedback on the responses is provided to the students.
4. The students discuss their answers in small groups.
5. The students respond again either individually or in groups.
6. The correct answer is presented by the teacher.
7. The teacher may organize class-wide discussions or provide further explanations.

There are three variations of the above model in SAGA:

1. The teacher poses a question.
2. The students respond individually.
3. Rich aggregated feedback that includes several objective and subjective metrics based on the responses is provided to the students.
4. Group discussion is not happening to adhere to lecture time constraints.
5. The students see their previous answers and decide whether to revise or not individually.
6. The correct answer is presented by the teacher.
7. The teacher may organize class-wide discussions or provide further explanations.

You can develop your own approach, but the basic principle is to let the student know at some point what the classroom thinks on the same question and if you cannot allocate the time for classroom discussion, then you will need to provide additional information on the class to the students – this is what the additional feedback metrics in SAGA are trying to do.

Several feedback metrics have been used in past studies with different versions of SAGA. These metrics aimed at providing a better picture of the audience to the student:
Objective metrics

• Percentage: How many students selected each choice?
• Past performance: How did they perform in previous quizzes?

Subjective (self-reported)

• Preparation: How prepared do they feel to take the quiz?
• Confidence: How confident do they feel that they got the right answer?
• Self-assessment: How well do they think they did in the whole quiz?
• Justification: Why did they select that particular choice?

Past performance and self-assessment are not available in the current version of SAGA. However, you are also able to create your own accompanying questions on the feedback metrics of your choice.