COMPUTER ADAPTIVE TESTING

Computer Adaptive Testing (CAT) can be also known as “tailor testing”. Computer Adaptive Testing is a computer-based assessment that adapts to the test taker’s ability This type of computerized test can alter the next group of questions based on the test taker’s answers to the questions prior. For example, if the person gives an incorrect answer, the computer will select an easier follow-up question. However, if the test taker answers correctly, the follow-up question would be more difficult. “CAT” is designed for each individual allowing for more of an accurate measurement on that individual’s ability.

"In Georgia the Ministry of Education and Science decided in September 2010 to use computer adaptive testing (CAT) as the delivery mode for the re-introduced external school graduation exams and to conduct the first administration in May 2011. International experts were quite sceptical about the feasibility of a nation-wide rollout of such a logistically and technologically complex measurement instrument as a large scale CAT at such short notice. In May 2011, 44,000 students sat eight computer adaptive subject tests in one of the 1500 test centres established in Georgian schools." (“The Introduction of Large Scale Computer AdaptiveTesting In Georgia” by Michael Trucano)

Some Advantages of CAT

Shorter tests: Research (e.g., Weiss & Kingsbury, 1984) has shown that CAT can reduce testing time by 50% or more. This can obviously translate into huge financial benefits. Suppose an organization tests 50,000 candidates per year, paying $50 per hour for seat time at professional © Copyright 2014 Assessment Systems Worldwide, LLC All Rights Reserved. centers. If the test is decreased from 2 hours to 1 hour, the annual costs are reduced by $250,000.

Equiprecision: CATs can be designed so that examinees are all measured with the same level of precision, even though they all potentially see different items. This makes the test extremely fair from a psychometric perspective.

More precision: Because CATs are more efficient, the organization has the option to design the CAT to actually be more precise than a conventional test while still using fewer items.

Examinee experience: As mentioned previously, a CAT will provide an appropriate challenge for each examinee. Low examinees are not discouraged or intimidated. High examinees enjoy receiving difficult items.

Increased motivation: Because of the better experience, there is likely an increase in examinee motivation. Low examinees feel better, and high examinees feel challenged. Both will try harder than with a conventional test.

Security: Because the CAT algorithm is very flexible and can adapt with potentially millions of permutations, there is much greater security than if everyone was administered the same set of 200 items.

Frequent retesting: The high number of permutations also enables more frequent retesting. If a student takes a CAT, then receives a few weeks of instruction, by the time they take a CAT again their ability has increased somewhat, and they will receive a completely different test. CBT advantages: CAT also benefits from the same advantages of non-adaptive computer-based testing (CBT). For example, tests delivered by computer (whether adaptive or not) can easily utilize multimedia such as audio and video files. This topic is also discussed in a separate white paper.

Disadvantages of CAT

No review: CATs rarely allow for examinees to return to items already administered, as the CAT has since adapted and it cannot unadapt.

Item exposure: CATs are designed to select the best items in the bank, and these items will often become overexposed if a control algorithm is not implemented.

Recovery of poor starts: CATs are susceptible to issues with examinee test anxiety, as the elimination of item review prevents someone from going back to the first few items. If they answered all those items incorrectly due to severe test anxiety, the test cannot correct itself. Public relations: Because of the complexity and the departure from the familiarity of the traditional exam paradigm, an organization must put forth more effort into public relations, explaining CAT and the reasons for using it.

Requires calibration: Nonstandard item formats are often not easily scored in real time, or are not able to be calibrated with IRT. Both are required for CAT. Requirements: CAT requires large sample sizes and extensive expertise. Testing programs need at least several hundred examinees, and an experienced Ph.D. psychometrician. Another white paper discusses the requirements to develop a CAT examination program, but some information is presented next.(“Advantages of Computerized Adaptive Testing (CAT)” by Nathan A. Thompson, PhD.)

Trucano, Michael (December 2015). “The Introduction of Large Scale Computer Adaptive Testing In Georgia”. Published by EduTech

Thompson, Nathan A. PhD (August 2011). “Advantages of Computerized Adaptive Testing (CAT)”.

The Glossary of Education Reform for Journalists, Parents and Community Members (August 2013). “Computer Adaptive Test”.