CAP-ET left bannerCAP-ET bannerCAP-ET right banner

Computer Aided Assessment


Introduction to Educational Technology :: Introduction to CAA :: CAA at Warwick ::
Introduction to Objective Testing and Computer Aided Assessment at Warwick

Introduction to Objective Testing and Computer Aided Assessment at Warwick

Assessment

With higher class sizes than ever before and a wider variety of student capability, the need to undertake more assessment can result in an oppressive workload. Objective testing through Computer Aided Assessment (CAA) may be a partial solution.

Classroom assessment has been defined as "a simple method used to collect feedback, early and often, on how well students are learning what they are being taught. The purpose of classroom assessment is to provide tutors and students with information and insights needed to improve teaching effectiveness and learning quality."

However, even simple methods of assessment involve three processes:

  1. Data collection,
  2. Analysis, and
  3. Utilization of results.

When these techniques are used often, they amount to extra work that lecturers cannot always afford to include in their schedule. As a result, the most common practice is to perform assessment as summative in a capstone exercise and use the results to improve future offerings of the course. CAA can change the situation by greatly facilitating the two initial processes - data collection and analysis, thereby providing teachers with the necessary elements to complete the third process.

The immediate and constant feedback from the students, in conjunction with consequent correction of course direction when necessary, constitutes "interactive assessment". CAA provides an environment where students can complete quizzes and surveys online and immediately receive the results and their interpretation.

At the outset, we should be careful to distinguish between objective testing and computer aided assessment (CAA). CAA is usually simply a convenient way of automating the delivery of objective tests and offer a number of benefits over paper based objective tests both in the range and richness of questions that can be set and in managing the assessment process. Non-objective tests can be delivered with CAA but this usually means manual marking.

Objective testing and computer aided assessment

  • What objective tests are
    • Types of objective test questions
    • Objective test terminology
  • Advantages of objective testing
  • Disadvantages of objective testing
  • Advantages of CAA
  • Disadvantages of CAA
  • CAA provision at Warwick
  • The future of CAA at Warwick
  • Where to start
  • Contacts

What are objective tests?

As the name implies, the mark achieved by the student does not depend on the person who performs the marking. An objective test requires the student to select or input a correct answer from a predetermined set of alternatives. Herein lies both the major strength and weakness of objective testing.

Types of objective test questions

Objective tests can contain a number of question types:

  • MCQ - Multiple Choice Question
  • MRQ - Multiple Response Question
  • True/False
  • Matching
  • Labeling Images - a form of Matching question
  • Rank Ordering
  • Gap Filling
  • Numerical
  • Visual identification

More advanced question types may be constructed

  • Assertion-Reason
  • Multiple True/False

Objective test terminology

A traditional objective test question consists of four discrete elements:

stem the text of the question
options the choices provided after the stem
key the correct answer in the list of options
distracters the incorrect answers

For comprehensive guidance on designing objective tests see the CAA Centre web site (www.caacentre.ac.uk).

Objective tests are generally not well received in HE where the general perception is that they can test only rote or surface learning. It is true that poorly conceived questions are limited in this way but it is possible with more thought to devise questions that test higher levels of learning. In general, the higher the order of learning one is trying to assess, the more difficult it is to construct the question.

Objective tests may be delivered in a paper format and either marked manually or by scanning the answers with an optical mark reader (OMR) system. OMR software can include reporting and analysis features enhancing the overall process.

OMR Provision at Warwick
More about OMR support at Warwick

The next obvious step is to deliver the assessment itself through computer systems - CAA. This is accompanied by advantages and disadvantages over a paper based objective assessment.

It is the design of effective objective tests that is the most time consuming and challenging part of using CAA. CAA and OMR software vary in their complexity but mastering the software itself is usually fairly straightforward.

Objective tests are simply another tool for teaching and the process should start with careful consideration of how objective tests will fit in to the overall assessment and teaching strategy for the unit and to do this we need to be aware of the advantages and disadvantages of this tool.

Frequently asked questions about objective testing and CAA
http://www.caacentre.ac.uk/resources/faqs/index.shtml

Advantages of objective testing

  • They are objective…
  • Speed - even when entirely paper based, the increased speed of marking offered is an advantage for both the lecturer and student. (24 hr turnaround using the Warwick OMR service, CAA systems can be immediate).
  • Diagnostic data is easy to collate and analyse.

Disadvantages of objective testing

Hastily prepared objective tests can have serious shortcomings due to factors such as inability of lecturers to produce items related to the objectives, technical defects in the actual item and finally weaknesses in the final interpretation of results.

Thoughtful design of questions can extend the potential of objective testing beyond the assessment of the recall of facts and but the higher the order of learning being assessed the harder it is to devise good questions.

Designing the questions is therefore the most difficult and time-consuming aspect of CAA. Care needs to be taken to ensure that the questions are accurate, unambiguous and suitably challenging. The distracters (incorrect answers) need to be plausible and should ideally match common errors and misconceptions, while the feedback must be sufficiently detailed to help students learn from their mistakes. The questions must then be entered into the CAA system (for example Blackboard) but this only takes a few minutes per question.

Being objective, the questions can only test for anticipated responses. So what objective tests cannot deal with is creativity - the unanticipated response. But we would not advocate that objective tests be used as the only form of assessment.

For example, CAA is most effective when used to assess recall of facts and their application - and this may be far from trivial, whereas essays allow students to demonstrate their understanding of a topic. It follows that the choice of technique should match the type of learning objective that is being assessed, and that objective tests are simply a useful addition to the techniques available to the tutor.

However once you have designed your questions, entering them into a CAA system usually takes only a few minutes and the benefits are many:

Advantages of CAA

For students

Students are assessed to ensure that the quality of our graduates is assured, but assessment offers other benefits:

  • Motivation - helps to establish priorities
  • Skill development - the opportunity to practise skills already learned
  • Recognition - acknowledges effort spent learning
  • Diagnostic - identifies difficulties and weakness
  • enables rapid delivery of assessment results to students
  • supports of formative assessment to facilitate reflective learning
  • Web based delivery (tests are available on or off campus) ·

For tutors

  • Saves staff time.
  • On-line exams, tests, self assessment
  • Supports distance learning assessment
  • Formative and summative testing
  • Many type of question design available. Graphics and multimedia can be included in a test
  • Reporting software and instant feedback
  • Tests can be scheduled automatically (according to those registered on a module) and down to the second
  • Adaptive testing can be used to match the test to the students' ability
  • Formative assessments can be repeated as frequently as desired to aid student learning
  • Quality can be monitored by looking at the performance of questions
  • Support for question design
  • Support for exam/test administration, data delivery
  • Can be used for applications other than CAA (research questionnaires, module evaluation, registration, application forms etc.)
  • Once entered into a CAA system, questions may be reused in any number of assessments. Questions can be exchanged with your peers and you may draw upon large question banks for your discipline if these are available.

Use of multimedia

One of the key advantages of CAA is the ability to include images as part of the question and/or the answers. Of course the images (photos, diagrams, maps, music scores, equations etc.) must be located or created, then prepared for display on a computer. This adds to the time needed to create the question and can also introduce copyright issues, but is nevertheless an extremely useful technique. In theory it is possible to use other types of multimedia (audio, video, animation) but these will present daunting technical challenges for most tutors. Of course, the delivery software must support this.

Disadvantages of CAA

The disadvantage of CAA as a means of delivering objective test sis that the question authoring system in CAA is always a tradeoff between ease of use of flexibility of question design/formatting which means that simple CAA systems may simply not allow for the construction of more complex question types that test higher orders of learning. Teachers' whose first introduction to objective testing is through CAA rather than peper based, may not be fully aware of the potential of objective tests due to the constraints of the software. On the other hand, software that offers more flexibility is also usually more expensive and difficult to learn.

When used for summative assessment, a whole raft of additional issues need to be raised such as the need for standard procedures and contingency plans, as well as security, accessibility and the potential for cheating offered by networked PCs.

CAA provision at Warwick

ITS has set up the Question Mark Perception which is a Web delivered CAA system

Question Mark is a market leader, supports a variety of question types, and provides support and help facilities. Academics who are interested in using this service should contact

Currently, the only central provision for CAA is Question Mark Perception. A server is run by ITS but authoring licenses must be purchased at £300 each - contact Rachel Parkins for details of Warwick licensing requirements.

Warwick departments that currently make use of QM Perception include:

  • Engineering
  • Biological Sciences
  • Warwick Business School

If you require further information, email Natasha Nakaroakova in ITS elab

Perception is compliant with the emerging IMS standards which will help to ensure the question database is compatible with any other IMS compliant software (a future proofing measure).

The future of CAA at Warwick

Despite this, it may be that Warwick, through the elab will develop its own in-house CAA solution. This is likely to be less complex than Perception and aimed at the casual or first time user of CAA.

Training in objective test design and CAA

The Warwick Teaching Certificate contains a module on assessment which includes objective test design and a brief introduction to CAA

CAP runs occasional workshops on objective test design for all staff and can run a tailored session for a department upon request.

The Warwick Online Course Construction programme contains a half-day session on using a CAA system (currently this is the CAA tool within the Blackboard VLE but this will change to Question Mark Perception in the next run of the programme).

Planning the use of CAA

  1. Learn how to create effective objective tests in a paper based format where the software neither constrains the flexibility of the question design nor imposes a learning curve to getting started.

    Writing effective CAA questions is a skill that improves with experience. Some staff development is needed to learn about question types that enable higher order learning to be assessed. Available resources include:

    CAA Centre Resources
    Centre for Academic Practice
    Warwick Interactions issue (Vol2no3) on CAA:

  2. Decide what you are using the objective tests for:
    a. Students' self diagnosis
    b. Diagnostic testing
    c. Summative assessment

    Your choice of delivery mechanism will depend on this.

    The more summative the assessment, the greater will be the demands on the software and delivery system for authentication, data security and reliability. For formative assessment, a simple self diagnostic quiz of half a dozen questions where the test can be open requires little more than an interactive web page and no authentication and no data storage. At the other end of the scale, formal examinations might require an institution-wide strategy for appropriate work areas, scheduling of groups and randomized test delivery. For this, you may want to access question banks in your subject area - try LTSN Subject Centres.

    Will the CAA system support the question designs you require? If not, you may need to use a more sophisticated system such as one capable of delivering full summative assessment even though you may only be using it for formative assessment.

Contacts and references

Warwick

Centre for Academic Practice
Graham Lewis & Jay Dempster (CAP) - CAA systems in general
Adrian Stokes (CAP WTC) - Objective test design

IT Services
Natasha Nakariakova (eLab) - Question Mark Perception and other CAA provision at Warwick
Paper Assessment
Web Assessment

Wendy Murray - ITS Data Entry

National services and support

CAA Centre Online guidance resources:

LTSN Generic Centre

Resources logo


Topic Guides


More Publications