Background

Education is an ever-changing field that attempts to adapt to the needs of the various stakeholdes involved. Recently, there have been two ideas in education that have led to many members of education to yearn for changes and improvements. The first is that the needs of individual students should be met. No longer should educators aim to teach the average and hope that students on the edges of the continuum are also successful. The learning experiences, content, processes, and even assessments should be designed and implemented with the specific individual students in mind. The second idea is that there needs to be some kind of accountability for students and teachers. We have entered the age where concrete data is needed to prove that learning has taken place. Measuring what students know, what students need extra support on, and where students need to be challenged provides the teacher with a better idea of how to carry out the idea mentioned previously. The tricky situation of how to hold teachers accountable has been a hot topic in the last few years. As with many other jobs, educators need some kind of performance evaluation to make sure that students are recieving a high quality learning experience. While student performance should not be the sole indicator of a quality teacher, there are many that feel that assessment scores should carry weight in the evaluation of a teacher.

There have been many, many criticisms regarding the assessments designed to measure student growth, especially after states have tied the results to teacher evaluations and student interventions. With the adoption of the Common Core State Standards, new standardized tests have been developed and will be used for the first time in the 2014-2015 school year. There are two main coalitions that have received Federal funding for this development: the Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (SBAC). Both have designed computer-based testing programs that feature text, graphics, and interactive questions to maintain engagment and also accurately measure what students know. (Davis 2012)

One major difference between the two approaches to this new computer-based standardized assessment is that PARCC uses a fixed-form test. This means that each student will generally receive the same assessment that is designed at their appropriate grade level. SBAC has designed theirs to be a Computerized Adaptive Test (CAT) in which questions increase or decrease in difficulty based on the responses of the student. Adaptive testing has actually been around a long time, but has only come into the limelight recently. Alfred Binet used adaptive testing to administer intelligence tests.

Since his concern was with the diagnosis of the individual candidate, rather than the group, there was no issue of fairness requiring everyone to take the same test. He realized he could tailor the test to the individual by a simple stratagem - rank ordering the items in terms of difficulty. He would then start testing the candidate at what he deemed to be a subset of items targeted at his guess at the level of the candidate's ability. If the candidate succeeded, Binet proceeded to give successively harder item subsets until the candidate failed frequently. If the candidate failed the initial item subset, then Binet would administer successively easier item subsets until the candidate succeeded frequently. From this information, Binet could estimate the candidate's ability level. Binet's procedure is easy to implement with a computer. (Chae, Kang, Jeon, and Linacre, 2000)

In the following sections of this work, the SBAC standardized test will be analyzed futher to describe how it works, what ideas it is based upon, what it looks like in practice, and finally, what benefits it provides and costs it incurs.

How It Works

Computerized Adaptive Testing (CAT) is unique in that the assessment is constantly updating as the student is completing the test items. CAT assesments follow and repeat three basic steps once the test begins:

1) The collection of available questions is searched for an item that is at the appropriate level for the studnet completing the test. (If the test is just beginning, the question will most likely be at grade level).

2) The question is presented to the student, who answers it correctly or incorrectly.

3) The computer program notes how the student anwered the question and uses this information to update the estimated ability level of the student. Each time the student answers another question, the data is added to the prior answers and used to recalculate the ability level. The process then begins again with step 1.

This graphic shows how CAT programs update based on the responses that are given. (Picture taken from https://www.trytalentq.com/how-to-complete/elements/)

The visual above shows two assessment experiences that could occur on the same test. Students will start with a question of intermediate difficulty. If the student answer the question wrong, denoted by the red 'X,' the next question he or she sees would be at a slightly lower level. If the student answers the question correctly, the next question he or she is given is at a slightly higher level. Once the student has answered the desired number of questions, the test will terminate and a score is reported based on the level they ended at.

Connected Theories and Ideas

The big idea behind Computerized Adaptive Testing is that the assessment adjusts to the ability of the student taking it, minimizing frustration and getting a more accurate picture what the students true ability is. A well-constructed fixed-form test will do a decent job of providing accurate information about students that are at or near the ability level that the test is designed at, but decreases in accuracy the further away a students ability is above or below the level of the test. Having an assessment adjust to the student and provide a version that is tailored to their level will give teachers better insight into what the student actually knows.

One learning theorist who could be connected with CAT is Vygotsky. Jeremy Bohonos (2013) summarizes one of Vygotsky's ideas, the zone of proximal development:

Vygotsky (1978) believed that each student operates within a range of ability and that educators would best facilitate learning by presenting students with work that challenges without overwhelming them. If work is too easy the student will be bored, while if the work is too difficult the student will not have the intellectual tools necessary to learn anything from attempting the work.

Vygotsky supposed that students could work a little bit above their ability level with the support of a more capable peer. This could take the form of a fellow student, a teacher, or other supportive adult. Just as there are limits to what a student can do on their own, there are also limits to what a student can successfully accomplish with the support of a more capable peer within the zone of proximal development. By using an adaptive assessment, the Zone of Proximal development can be found more easily. Being able to give students questions that exceed or fall below their level gives a more accurate measure, especially students who are at the edges of the continuum.

As a student gets answers correct, the test keeps increasing the difficulty until a student begins to answer questions incorrectly. On a traditional test, it is the goal for the student to correctly complete the test independently. On an adaptive test, it is measuring the limits of what the student is able to accomplish independently that is the primary focus.

Zone of Proximal Development (Image taken from http://computerassistedcooperativelearning.wordpress.com/2012/05/02/theoretical-perspectives/)

A CAT program adjusts to the student so that they can be appropriately challenged or supported. The test will change so that it assesses what the student can do on their own. This in turn, allows the instruction that follows to be in a more accurate zone of proximal development. Providing a student test items that are below his or her level does not provide accurate information of how far to challenge them. Likewise, test items that are beyond a student's abilities only creates frustration and does not provide useful insight as to where support should be put into place.

Seeing It in Action

SBAC and PARCC are the two standardized tests that have been created to assess the Common Core State Standards. Not all states have adopted the Common Core State Standards and therefore have not adopted either SBAC or PARCC. Below is a map showing which states plan to use the adaptive standardized test developed by SBAC.

Map showing which states will use the standardized test developed by the Smarter Balanced Assessment Consortium (taken from www.smarterbalanced.org)

SBAC has developed three different tools to be used in assessing students: Interim Assessments, Performance Tasks, and End-of-Year Assessments. Grades 3-8 and High School all have these tools developed by SBAC for both English Language Arts and Mathematics. The interim assessments are optional and can be given a couple times each school year. The Performance tasks and End-of-Year Assessments are required by the states that have adopted SBAC.

Summary of SBAC Testing Tools (taken from http://www.k12center.org/rsc/pdf/Coming_Together_April_2012_Final.PDF)

The Smarter Balanced Assessment Consortium has also estimated the time it would take for students to complete the CAT and the Performance task for different grade ranges. According to SBAC, the time ranges from 7 to 8.5 hours of total testing time, depending on grade level.

On the SBAC website, they provide practice items and performance tasks for teachers and/or students to attempt and become familiar with the type of activities they will see. For example, in the 5th grade English Language Arts performance task, the student is required to read through three different sources on a related topic. The student is allowed to take notes while reading these sources. Using the information, the student is asked to answer three questions: one selected response and two written response. Below is a screenshot of what this performance task looks like:

The source materials appear on the left and can be scrolled through. The questions are on the right where students will enter in their responses.

Illinois has chosen to use PARCC instead of SBAC. Just how do these two assessments compare to one another? In the publication Coming Together to Raise Achievement (2012), the Center for K-12 Assessment and Performance Management at ETS developed a table to illustrate the similarities and differences between the two assessments.

Similarities and Differences between PARCC and SBAC

The main difference between the two standardized tests is that Smarter Balanced Assessment Consortium has designed theirs to be adaptive, while PARCC developed a fixed-form assessment.

It also important to note that CAT is not just used in education, but in other fields as well. The U.S. Department of Defense uses the Armed Services Vocational Aptitude Battery (ASVAB), "a multidimensional instrument, assessing ten vocational abilities (such as Mathematices Knowledge, Electronics Information, and Mechanical Comprehension) that help to determine potential placement for examinees in military occupations." (Fetzer, Dainis, Lambert & Meade, 2011) Another example is the Graduate Management Admission Test, or GMAT-CAT, which is used for students pursuing graduate degrees, typically MBAs. Microsoft also has a series of adaptive exams to provide certification to informational technology professionals.

Strengths and Weaknesses

As with any change, especially that which includes technology, there are strengths and weaknesses to computer adaptive standardized testing. Setting aside the debate of standardized testing itself, let us assume that some form of standardized test is required. There are many benefits to using a CAT for this task:

The data will be more reliable as students are tested to the limits of their abilities
Security is increased as each student will receive a different test
The cost is lower than that of a fixed-form paper test
Scores and feedback will return to the teacher/student quicker
The questions can include media, graphics, and other forms that are more engaging

There are also weaknesses to this form of testing. One critical question is to what extent should students be assessed below or above their grade level? Some critics worry that CAT will lead to more lenient testing and eventually "dumbed-down instruction" for struggling students. (Herold, 2013) However, only giving grade-level items severely limits the adaptability of the assessment. One other criticism is that while the CAT form of standardized testing may be cheaper per student the initial financial investment of schools to upgrade their infrastructure to handle the test could be pretty large. While Smarter Balance claims their test can be run on minimum processors and memory, many schools will need to procure bandwidth, devices, and other technology to have an adequate testing environment.

Testing windows have also become a concern. If students are to take PARCC or SBAC, schools are committing to more testing windows than the ISAT. While this can give more data points, it takes away from instructional time and knocks students out of their routine. A daunting thought is the amount of time students are being tested. Some schools will have students endure F&P, AIMSweb and MAP testing on top of PARCC or SBAC and this collection does not include any progress monitoring that may also have to be completed. While data should be used to drive instruction, how much time is appropriate for data collection?

Conclusions and Recommendations

Teachers are working in the age of accountability. Students are learning in the age of assessment. While it appears that standardized testing is here to stay, at least for the time being, a "lesser evil" may have emerged with CAT. The data it gathers can be more informative and reliable, more cost-effective if the school does not have to do any major upgrades, and can provide more timely feedback for teachers and students. CAT is also becoming more prevalent in many career paths. If educators are going to be required to subject their students to a point in time measurement to determine growth of student learning in addition to teacher effectiveness, then a well-designed, adaptive test should be used over a traditional fixed-form test. This at least gives the students and teachers a fighting chance at getting some useful information out of the experience. While it still just gives a snapshot of what a student can achieve, at least it is a little bit more in focus.