to Objective Testing and Computer Aided Assessment at Warwick
class sizes than ever before and a wider variety of student capability,
the need to undertake more assessment can result in an oppressive
workload. Objective testing through Computer Aided Assessment
(CAA) may be a partial solution.
assessment has been defined as "a simple method used to collect
feedback, early and often, on how well students are learning what
they are being taught. The purpose of classroom assessment is
to provide tutors and students with information and insights needed
to improve teaching effectiveness and learning quality."
simple methods of assessment involve three processes:
- Data collection,
techniques are used often, they amount to extra work that lecturers
cannot always afford to include in their schedule. As a result,
the most common practice is to perform assessment as summative
in a capstone exercise and use the results to improve future offerings
of the course. CAA can change the situation by greatly facilitating
the two initial processes - data collection and analysis, thereby
providing teachers with the necessary elements to complete the
and constant feedback from the students, in conjunction with consequent
correction of course direction when necessary, constitutes "interactive
assessment". CAA provides an environment where students can
complete quizzes and surveys online and immediately receive the
results and their interpretation.
At the outset,
we should be careful to distinguish between objective testing
and computer aided assessment (CAA). CAA is usually simply a convenient
way of automating the delivery of objective tests and offer a
number of benefits over paper based objective tests both in the
range and richness of questions that can be set and in managing
the assessment process. Non-objective tests can be delivered with
CAA but this usually means manual marking.
and computer aided assessment
- What objective
of objective test questions
of objective testing
of objective testing
- CAA provision
- The future
of CAA at Warwick
- Where to
What are objective
As the name
implies, the mark achieved by the student does not depend on the
person who performs the marking. An objective test requires the
student to select or input a correct answer from a predetermined
set of alternatives. Herein lies both the major strength and weakness
of objective testing.
Types of objective
tests can contain a number of question types:
- MCQ - Multiple
- MRQ - Multiple
Images - a form of Matching question
- Rank Ordering
- Gap Filling
question types may be constructed
Objective test terminology
objective test question consists of four discrete elements:
text of the question
choices provided after the stem
correct answer in the list of options
guidance on designing objective tests see the CAA Centre web site
tests are generally not well received in HE where the general
perception is that they can test only rote or surface learning.
It is true that poorly conceived questions are limited in this
way but it is possible with more thought to devise questions that
test higher levels of learning. In general, the higher the order
of learning one is trying to assess, the more difficult it is
to construct the question.
tests may be delivered in a paper format and either marked manually
or by scanning the answers with an optical mark reader (OMR) system.
OMR software can include reporting and analysis features enhancing
the overall process.
OMR Provision at
More about OMR support at Warwick
obvious step is to deliver the assessment itself through computer
systems - CAA. This is accompanied by advantages and disadvantages
over a paper based objective assessment.
It is the
design of effective objective tests that is the most time consuming
and challenging part of using CAA. CAA and OMR software vary in
their complexity but mastering the software itself is usually
tests are simply another tool for teaching and the process should
start with careful consideration of how objective tests will fit
in to the overall assessment and teaching strategy for the unit
and to do this we need to be aware of the advantages and disadvantages
of this tool.
questions about objective testing and CAA
Advantages of objective
- They are
- Speed -
even when entirely paper based, the increased speed of marking
offered is an advantage for both the lecturer and student. (24
hr turnaround using the Warwick OMR service, CAA systems can
data is easy to collate and analyse.
objective tests can have serious shortcomings due to factors such
as inability of lecturers to produce items related to the objectives,
technical defects in the actual item and finally weaknesses in
the final interpretation of results.
design of questions can extend the potential of objective testing
beyond the assessment of the recall of facts and but the higher
the order of learning being assessed the harder it is to devise
the questions is therefore the most difficult and time-consuming
aspect of CAA. Care needs to be taken to ensure that the questions
are accurate, unambiguous and suitably challenging. The distracters
(incorrect answers) need to be plausible and should ideally match
common errors and misconceptions, while the feedback must be sufficiently
detailed to help students learn from their mistakes. The questions
must then be entered into the CAA system (for example Blackboard)
but this only takes a few minutes per question.
the questions can only test for anticipated responses. So what
objective tests cannot deal with is creativity - the unanticipated
response. But we would not advocate that objective tests be used
as the only form of assessment.
CAA is most effective when used to assess recall of facts and
their application - and this may be far from trivial, whereas
essays allow students to demonstrate their understanding of a
topic. It follows that the choice of technique should match the
type of learning objective that is being assessed, and that objective
tests are simply a useful addition to the techniques available
to the tutor.
you have designed your questions, entering them into a CAA system
usually takes only a few minutes and the benefits are many:
Advantages of CAA
assessed to ensure that the quality of our graduates is assured,
but assessment offers other benefits:
- helps to establish priorities
development - the opportunity to practise skills already
- acknowledges effort spent learning
- identifies difficulties and weakness
rapid delivery of assessment results to students
of formative assessment to facilitate reflective learning
- Web based
delivery (tests are available on or off campus) ·
- Saves staff
exams, tests, self assessment
distance learning assessment
and summative testing
- Many type
of question design available. Graphics and multimedia can be
included in a test
software and instant feedback
- Tests can
be scheduled automatically (according to those registered on
a module) and down to the second
testing can be used to match the test to the students' ability
assessments can be repeated as frequently as desired to aid
can be monitored by looking at the performance of questions
for question design
for exam/test administration, data delivery
- Can be
used for applications other than CAA (research questionnaires,
module evaluation, registration, application forms etc.)
- Once entered
into a CAA system, questions may be reused in any number of
assessments. Questions can be exchanged with your peers and
you may draw upon large question banks for your discipline if
these are available.
Use of multimedia
One of the
key advantages of CAA is the ability to include images as part
of the question and/or the answers. Of course the images (photos,
diagrams, maps, music scores, equations etc.) must be located
or created, then prepared for display on a computer. This adds
to the time needed to create the question and can also introduce
copyright issues, but is nevertheless an extremely useful technique.
In theory it is possible to use other types of multimedia (audio,
video, animation) but these will present daunting technical challenges
for most tutors. Of course, the delivery software must support
of CAA as a means of delivering objective test sis that the question
authoring system in CAA is always a tradeoff between ease of use
of flexibility of question design/formatting which means that
simple CAA systems may simply not allow for the construction of
more complex question types that test higher orders of learning.
Teachers' whose first introduction to objective testing is through
CAA rather than peper based, may not be fully aware of the potential
of objective tests due to the constraints of the software. On
the other hand, software that offers more flexibility is also
usually more expensive and difficult to learn.
for summative assessment, a whole raft of additional issues need
to be raised such as the need for standard procedures and contingency
plans, as well as security, accessibility and the potential for
cheating offered by networked PCs.
CAA provision at
ITS has set
up the Question Mark Perception which is a Web delivered CAA system
is a market leader, supports a variety of question types, and
provides support and help facilities. Academics who are interested
in using this service should contact
the only central provision for CAA is Question Mark Perception.
A server is run by ITS but authoring licenses must be purchased
at £300 each - contact Rachel
Parkins for details of Warwick licensing requirements.
that currently make use of QM Perception include:
If you require
further information, email Natasha
Nakaroakova in ITS elab
is compliant with the emerging IMS standards which will help to
ensure the question database is compatible with any other IMS
compliant software (a future proofing measure).
The future of CAA
it may be that Warwick, through the elab will develop its own
in-house CAA solution. This is likely to be less complex than
Perception and aimed at the casual or first time user of CAA.
Training in objective
test design and CAA
Teaching Certificate contains a module on assessment which includes
objective test design and a brief introduction to CAA
CAP runs occasional
workshops on objective test design for all staff and can run a
tailored session for a department upon request.
Online Course Construction programme contains a half-day session
on using a CAA system (currently this is the CAA tool within the
Blackboard VLE but this will change to Question Mark Perception
in the next run of the programme).
Planning the use
- Learn how
to create effective objective tests in a paper based format
where the software neither constrains the flexibility of the
question design nor imposes a learning curve to getting started.
effective CAA questions is a skill that improves with experience.
Some staff development is needed to learn about question types
that enable higher order learning to be assessed. Available
Centre for Academic Practice
Interactions issue (Vol2no3)
what you are using the objective tests for:
a. Students' self diagnosis
b. Diagnostic testing
c. Summative assessment
of delivery mechanism will depend on this.
summative the assessment, the greater will be the demands
on the software and delivery system for authentication, data
security and reliability. For formative assessment, a simple
self diagnostic quiz of half a dozen questions where the test
can be open requires little more than an interactive web page
and no authentication and no data storage. At the other end
of the scale, formal examinations might require an institution-wide
strategy for appropriate work areas, scheduling of groups
and randomized test delivery. For this, you may want to access
question banks in your subject area - try LTSN
CAA system support the question designs you require? If not,
you may need to use a more sophisticated system such as one
capable of delivering full summative assessment even though
you may only be using it for formative assessment.
Contacts and references
for Academic Practice
& Jay Dempster
(CAP) - CAA systems in general
Adrian Stokes (CAP WTC) - Objective test
Nakariakova (eLab) - Question Mark Perception and other CAA provision
ITS Data Entry
services and support