You unlock this door with the key of imagination. Beyond it is another dimension.

A dimension of sound. A dimension of sight. A dimension of mind.

You're moving into a land of both shadow and substance. Of things and ideas.

You've just crossed over into...

The Twilight Zone.

As a middle school student in the 1970s, I enjoyed watching re-runs of the The Twilight Zone where my young mind would enter into a world of imagination – sometimes dark and spooky - always intriguing. After watching each 30-minute episode, my mind returned to the safety of the den in my suburban home, but snippets of several episodes - "The Eye of the Beholder", "Nighmare at 20,000 Feet" - remain with me to this day.

Four decades later, computer simulatiions and games allow middle school students to open the door to new worlds with the key of imagination - in less dark and spooky ways. Computer simulations and games can immerse students into the sights and sounds of new worlds in math, science and social studies. In these worlds, students can learn concepts and be assessed while being entertained.

This paper will focus on computer simulations as a method of assessment in middle school. In doing so, it will focus on the following questions.

• What is a computer simulation versus a computer game?
• Why would a teacher want to use computer simulations to assess student learning?
• How is assessment design changing with the introduction of Common Core Standards and advancements in technology?
• How do computer-based assessements differ from traditional assessments?
• How does current technology support or limit the development of simulation-based assessment?
• Whose learning theories or philosophies support simulation-based assessment?
• What is an example of a computer-based simulation that is appropriate for middle school learning and assessment?

After exploring each question, the benefits and challenges of simulation-based assessement will be explored. Lastly, recommendations for future developments and a vision of the future of simulation-based assessment will be provided. 

What is a computer simulation versus a computer game?

According to Wikipedia, a computer simulation "uses a computer model, or a computational model to simulate (a) system. Computer simulations have become a useful part of mathematical modeling of many natural systems in physics, (computational physics), astrophysics, chemistry and biology, human systems in economics, psychology, social science and engineering" ("Computer Simulation", 2014).But how does a computer simulation differ from computer games? According to Dr. Simon Usherwood, Associate Dean of the University of Surrey, in conjunction with the Higher Education Academy has developed a variety of resources about on why and how to do simulations. Dr. Usherwood delineates simulations versus games as follows.

In simple terms, a simulation is a recreation of a real-world situation, designed to explore key elements of that situation. It is a simplification and essentialisation of some object or process that allows participants to experience that object or process. Put differently, we take out some element of the real world and create a simple space in which to consider and interact with it. Games tend to fall at the simpler end of the spectrum – e.g. in creating very stylised environments – but also shade into the related worlds of video-gaming and serious games. (Usherwood)

Common to each of these definitions is that computer simulations recreate and explore elements of the real world - often math, science, social sciences, technology, or languages. While simulations are simpler than the original phenomenon, they are more complex than games. Simulations capture elements of the phenomenon that is not easily recreated without technology. Figure 1 shows the quality of the graphics of Alelo a simulation-based assessment for learning a second language ("Simulation Spreads from Professional License Exams to K-12 Assessments", 2012).

Figure 1 - Screenshot from Alelo, a Simulation-Based Assessment

Why would a teacher want to use computer simulations to assess student learning?

According to K-12 Center at ETS,

simulation-based assessment is exciting to educators because it opens a whole new realm of opportunities to better understand what students know and can do. Simulations are engaging to young people, who interact every day with a technology-enhanced, multimedia world. Simulations present students with an authentic scenario that, ideally, requires them to behave as they would in the real world, testing their theories and applying their knowledge to complete complex tasks. Data gathered in simulation-based assessments can measure not just the correctness of one final answer, but multiple aspects of the student's ability to apply skills to solve problems ("Simulation Spreads from Professional License Exams to K-12 Assessments", 2012).

Simulation-based assessment can provide data not available in other paper or computer assessment models. Simulation-based assessments could "provide the teacher with evidence that suggests, for example, whether the student's strategy was sound, whether that student chose the right tools, where the strategy went wrong, and why." The information available - especially for formative assessements - will be richer and more efficiently attained. With this information, either the computer and teacher can provide constructive feedback to the student ("Simulation Spreads from Professional License Exams to K-12 Assessments", 2012).

How is assessment design changing with the introduction of Common Core Standards and advancements in technology?

In the past, most summative standardized tests were completed with paper and pencil. With the introduction of Common Core Standards, assessment designers have been tasked with redesigning summative assessments to measure higher-level thinking skills. Technological advancements allow designers to create Technology Enhanced Assessments (TEA) that can measure the multiple choice answers of the past and also measure problem solving strategies, decision making, and thought processes that were previously unavailable.

Technology Enhanced Assessments - either summative or formative - provide the opportunity to collect a richer set of data. These interactive assessments allow students to explore alternative thought processes while the assessment device adapts to each student's level of mastery. Once the summative or formative data is collected, teachers can review the data to understand a student's overall thought processes along each phase of the assessment.

To design TEAs, assessment designers use Evidence Centered Design (ECD). Figure 2 illustrates the elements of ECD and the iterative process among the elements. ECD begins with the claim space stage where designers and educators consider students' knowledge - what they should know and how they should know it. In the evidence stage, designers and educators identify acceptable evidence of knowledge and how they will analyze and interpret the evidence. For the task/situation stage, designers and educators determine how the students will communicate their knowledge. As needed, the team moves forward and backward to refine the design to obtain accurate measurable evidence ("Navigating Change", 2012).

Figure 2 - Evidence Centered Design Stages

How do computer-based assessements differ from traditional assessments?

To understand ECD more fully, it is helpful to contrast ECD with the standard assessment paradigm.

Assessment in the standard assessment paradigm is thought of just in terms of the highly scripted circumstances in which students solve constrained tasks, usually answering verbal questions, and results that simply accumulate independent item scores. The ECD framework neither requires nor implies this limited view of assessment. It is flexible enough to describe a wide range of activities and goals associated with assessment as conceived more broadly, including familiar tests but accommodating the informal assessment activities of instructors interacting with students in the classroom, students working through open-ended simulation tasks [Frezzo et al. 2009; Mislevy 2011; Williamson et al. 2004], multi-student interactions in role playing or simulated situations [Shute 2011], and game-based assessments [Behrens et al. 2007; Mislevy et al. in press] (Mislevy et al. 2012)

How does current technology support or limit the development of simulation-based assessment?

While using ECD to develop a summative TEA to measure the Common Core Standards is a necessary but difficult process, developing a simulation-based assessment is even more challenging. According to Dr. Roy Levy, of Arizona State University, " 'in some cases, the rush to adopt the physical machinery (of engaging simulations) has outstripped the adoption of the assessment machinery' required to draw valid and definable inferences from the data." According to Dr. John Behrens, VP of Pearson Center For Digital Data, Analytics & Adaptive Learning, the goal of simulation-based assessment should be to create a "naturalistic assessment" not the "tightly constrained workspace of a multiple-choice test" ("Simulation Spreads from Professional License Exams to K-12 Assessments", 2012).

Using ECD in simulation-based assessment often provides "an incredibly large and possibly infinite number of behaviors and thus has the potential to capture the richness of student performance in terms of processes and nuances.” Within simulation-based assessment, the methods to convert student responses to student proficiency measures are "in their infancy" compared to traditional standardized summative tests. To achieve more accurate data, designers would need to constrain the work space which could hinder "the very things that are attractive about simulation-based assessments" (Simulation Spreads from Professional License Exams to K-12 Assessments", 2012).

From the standpoint of assessment data collection, games and simulations have similar issues. How can the game or simulation collect accurate data that can measure and predict student's learning? Within the game or simulation are numerous "clicks" where the student made choices. Which of these data clicks should be mined as relevant measures of student knowledge? As stated in the August 2013 EdWeek "Video Games as 'Petri Dishes for Assessment Data" article, researchers are optimistic and point to recent success in mining telemtry data in the game Progenitor X where the player must assume the role of a scientist and grow new organs to save him/herself from zombies. Hidden within the game is stem cell science. Researchers compared students pre- and post-test knowledge of stem cells and found that the game increased students' knowledge even if they did not win the game. As James Paul Gee states, "With standardized testing, you're sampling your population two or three times a year, because you don't have a better way to get data. Now, we can collect a lot more data, in the kid's real environment. We can mine it. And we can represent it in ways that help the kid as well as grade the kid" (Herold, 2013).

Whose learning theories or philosophies support simulation-based assessment?

Simulation-based assessment can be considered a computer-based version of experiential learning. Experiential learning was developed by David Kolb and others and evolved from Jean Piaget, John Dewey, and Kurt Lewin. The Experiential Learning Center at the University of Colorado Denver answers the question, "What is Experiential Learning?" as follows:

Experiential learning is a process through which students develop knowledge, skills, and values from direct experiences outside a traditional academic setting. Experiential learning encompasses a variety of activities including internships, service learning, undergraduate research, study abroad, and other creative and professional work experiences. Well-planned, supervised and assessed experiential learning programs can stimulate academic inquiry by promoting interdisciplinary learning, civic engagement, career development, cultural awareness, leadership, and other professional and intellectual skills ("What is Experiential Learning", 2014).

They go on to state that "learning that is considered 'experiential' contain all the following elements:

1. Reflection, critical analysis and synthesis
2. Opportunities for students to take initiative, make decisions, and be accountable for the results
3. Opportunities for students to engage intellectually, creatively, emotionally, socially, or physically
4. A designed learning experience that includes the possibility to learn from natural consequences, mistakes, and successes ("What is Experiential Learning", 2014).

In K-8 grades, projects or project-based learning, field-trips, and museum visits are ways to learn through experience. In high school and especially college, experiential learning opportunities could expand to include an internship, laboratory research, or a study abroad experience. For any age student, participating in experiential learning may be difficult due to time, geographic or financial constraints. Simulation-based assessments allow students to participate in and learn from a richer set of experiences than they could otherwise.

As stated above, constructivism was a precursor to experiential learning. Two important contributors to the theory of constructivism were Jean Piaget and John Dewey. In Constructivism, Technology and the Future of Classroom Learning, Strommen and Lincoln propose that constructivism is how we must "educate the 'new child,' raised in a world of instant information, where interactive technologies have led them to believe they can act on the world with a press of a button." Strommen and Lincoln state that constructivism "emphasizes the careful study by which children create and develop their ideas. Its educational applications lie in creating curricula that match (but also challenge) children's understanding, fostering further growth and development of the mind." They further state that "two specific features of constructivist philosophy hold particular promise. The first is the notion...that play and experimentation are valuable forms of learning...It is a form of mental exploration...Both play and exploration are self-structured and self-motivated processes of learning. Both also encourage children to reflect on their ideas in ways not generally promoted by current school curricula (Strommen and Lincoln, 1992).

While Strommen and Lincoln's discussion of constructivism was focused on technology in general, their comments ring true for simulation-based assessment as well. Simulation-based assessment allows students to explore and play within the computer-based simulation. Students' thought processes during the exploration can be captured, analyzed and provided to the teacher as formative or summative assessment data.

What is an example of a computer-based simulation that is appropriate for middle school learning and assessment?

Currently, there are a number of e-learning and e-assessments - games and simulations - available for middle school learners and more are being developed. One of the more sophisticated simulation-based learning and assessment tools is SimScientists. This tool was developed by WestEd's STEM program as part of a series of projects that focused on how simulations can enrich science learning and assessment. SimScientists explores the middle school science topics of life science, physical science and earth science.

SimScientist an Educational and Formative Assessment Tool

This SimScientists video explains how this computer-based simulation provides the opportunity for the student to learn and for the teacher to gather formative data about the student's learning. SimScientists eschews traditional science lectures and texts and allows students to explore science interactively. Students can explore scientific models, investigate problem-based investigations, make inferences, test and analyze predictions, and discuss experiments. Students can break down complex ideas into more manageable ideas. SimScientists' embedded formative assessments allow students to explore evidence, analyze events and questions, and get feedback or coaching. Within the SimScientists tool, there are opportunities for discourse, reflection, and self-assessment.

Throughout the SimScientists simulations, assessment data is being gathered on each student and the class as a whole. The students can view their ongoing progress and the teacher can view progress for the student or the entire class. This report by SimScientists provides additional information about the technology and the assessment data. Figure 3 shows an example of the formative data provided to a student. Figures 4 and 5 show examples of the summary and detailed formative data available to the teacher (Quellmalz).

Figure 3 - SimScientists Example of Student Formative Data

Figure 4 - SimScientists Example of Teacher's Summary Formative Data

Figure 5 - SimScientists Example of Teacher's Detailed Formative Data

Simulation-based assessments have a dual purpose as a simulation and an assessment tool. As a computer-based simulation, the software provides content knowledge in a real-world like environment. The simulation software is designed to increase students' understanding of topics that are often difficult, costly or impossible to recreate in the classroom. As a computer-based assessment, the software collects and analyzes data on students' content knowledge and higher level thinking skills.

The benefits of computer-based simulations have been documented in a number of studies. Dr. Lisa Huelskamp studied problem-based learning with computer simulation in middle school science programs and found positive benefits for the teachers and students (Huelskamp, 2009). Kurt Michael cited the benefits of computer simulation in several studies (Michael, 2001). These studies had shown that computer simulation can:

1. Be equally as effective as real life, hands-on laboratory experiences in teaching students scientific concepts (Choi and Gennaro, 1987).
2. Enhance the learning achievement levels of students (Betz, 1996).
3. Enhance the problem solving skills of students (Gokhale, 1996).
4. Foster peer interaction (Bilan, 1992).

Several of the challenges of simulation-based assessment and Evidence Centered Design have been mentioned previously. To begin with, writing software to create an engaging and accurate simulation is time consuming and challenging. Much more difficult is the task of writing simulation software that can identify, collect, tabulate and report the data points that demonstrate student understanding and level of proficiency. While the technology is in its infancy, technologists believe that improvements will come over time. To date, the majority of efforts have been focused upon computer-based summative assessments. Over time, experiences with designing and deploying those assessments will be used to improve simulation-based formative assessments.

Technology-enhanced assessments are being developed in response to the Common Core Standards and will be summative in nature. These TEAs will be operational by the spring of 2015 and will include some simulation-based questions (Tools for Real-Time Personalized Learning, 2012). The TEAs which will measure Common Core Standards are being built using the concept of learning progressions, but researchers agree that more work needs to be done to fine tune the measurements for both formative and summative TEAs including simulation-based assessments. Figure 6 is a learning progressions model for proportional reasoning ("Navigating Change", 2012).

Figure 6 - Learning Progressions Model for Proportional Reasoning

According to K-12 Center at ETS, "complex skills and abilities, such as those within the Common Core State Standards, must be assessed within the summative assessments – as best they can at this point in time." They go on to say "the summative assessments to be delivered in 2014-2015 will be limited in the advances that they can incorporate, they must be designed to support a continued evolution in the assessments over time" ("Navigating Change", 2012).

“This isn’t a problem to be solved in one year or two years or probably even five years,” said Carl Wieman, Associate Director for Science for the White House Office of Science and Technology Policy. “Moving forward, we need to have the best assessments we can as the Common Core come online. And we have to do that in such a way that this is very clearly the first step, not the end point, so we don’t get locked in to where we are but [instead] are automatically upgrading routinely as we improve our understand of better ways to assess and better technology to do it.” ("Navigating Change", 2012).

A simulation is a recreation of a real-world situation and is more complex than a computer game. A computer-based simulation allows the student to explore a world that he or she would not be able to without the computer. Simulation-based assessment provides the student an opportunity to explore the simulated world, learn and be assessed without leaving his or her classroom or home.

Simulation-based assessments allow students to explore a breadth of experiences they would not receive otherwise. For the teachers, simulation-based assessemnts provide formative and summative assessment data about the students' choices and learning processes. The information can include whether a student is right or wrong, but more importantly students' understandings and misunderstandings can be identified so that assistance can be provided.

Simulations are in keeping with the learning theories of Constructivism and Experiential Learning which support students using play, experience, and the real-world to create knowledge. Computer-based simulations and simulation-based assessment such as SimScientists add technology to the mix so that students can explore virtual worlds that would not otherwise be available to them due to geography, time or financial restrictions.

Simulation-based assessment, which can be formative or summative, is in its infancy but will continue to develop as other summative Technology Enhanced Assessments such as those for the Common Core are developed. The TEAs for the Common Core are being designed to measure higher level thinking skills and will include some simulation. Over time, the knowledge gained from designing and refining summative TEAs such as the Common Core will inform the development of simulation-based formative assessments.

Recommendations for the future design of simulation-based assessments are provided by Roy Levy of Arizona State University in Psychometric Advances, Opportunities, and Challenges for Simulation-Based Assessment (Levy, 2012).

Recommendation 1 - Simulation-based tasks may be most easily deployed or integrated into assessments in contexts where they are used in concert with traditional or existing evidentiary arguments. Simulation-based tasks may be used in support of evidentiary arguments that are richer than those of traditional assessments. Doing so will likely involve innovative measurement models.

Recommendation 2 - The recent developments in statistical modeling allow for a broad set of choices for such innovative measurement models. Developers of simulation-based assessment will be well served to adopt a perspective that views that a measurement model can be built or customized to their specific needs. This will be best accomplished by including psychometric considerations from the outset and throughout the development of the assessment.

Recommendation 3 - Basic psychometric research is needed on these innovative measurement models. Such research should address the use of measurement models in service of assessment needs and could include methodological or applied research on areas such as model calibration and parameter estimation, reliability, validity, test form creation, linking and equating, adaptive administration, data-model fit, and evaluating assumptions.

Recommendation 4 - The key challenges to simulation-based assessment are likely to be most successfully addressed by integration of subject matter expertise and data analysis at all phases. Accordingly, psychometrics should play an integral role in the design, development, revision, and use of simulation-based assessments.

Finally, Levy provides an encouraging prediction of what simulation-based assessment could bring to not just assessment, but more importantly to instruction and learning.

Looking further ahead, simulations offer the potential to enact a number of changes to the typical way assessment is conducted. Simulations offer enormous potential for integrating assessment with learning and instruction; that is, we can replace the current “Teach. Stop. Test.” mode in which the assessment is clearly marked as different from instruction and learning...If we continue with this vision of simulations as a way for assessment to become intertwined with instruction and learning, assessment then becomes less of a static, cross-sectional snapshot of proficiency and more of a longitudinal tracking of student proficiency as it changes over time (Levy, 2012).