Produced with Scholar

Evaluation of an Educational Intervention

Project Overview

Project Description

An analysis of the outcomes of an educational intervention.

Icon for Evaluation: Close Reading through Educreations

Evaluation: Close Reading through Educreations

I presently teach a team-taught junior English course, serving nineteen students. The class is unusually small to accommodate the needs of students, some of whom have Individualized Education Plans and are in their first mainstreamed English class. Because this class is a mix of ability levels and proficiencies, differentiating instruction has been a goal - and a struggle - all semester. Also, acclimating all students to the expectations of a junior year English course has been an effort; encouraging students to read on their own, identify areas in which they are confused (both in literature and in the course), and seek assistance when needed have been the jobs of my co-teacher and I throughout the year, and we are always excited to see moments of improvement and achievement for our students.

Before beginning this intervention period, we had recently completed our first unit, in which we read Jonathan Safran Foer's Extremely Loud and Incredibly Close; as we studied the novel, we focused on the skills needed for the unit's final assessment, which was considered the semester's first District Assessment. Each semester, there are two District Assessments, which are assessments given to all students in a grade level, across the school district; the results of this assessment allows us to compare progress across teachers, courses, and high schools. This first District Assessment included a cold read (a new, unfamiliar text in a timed setting) with selected response questions and a paragraph response.

During the novel study, we learned and practiced close reading, which asks students to analyze text on a more granular level, looking for rhetorical devices and determining these devices' impacts on the piece.Specifically, we ask that students use their understanding of the rhetorical structures to explain or support the author's purpose in a passage or larger piece; during this unit, students studied how parallelism, syntax, descriptive language, and figurative language revealed the author's purpose, especially with regard to a specific mood, tone, or theme.

As we read the novel, students practiced close reading by marking their novels and marking provided excerpts of the novels. I would model close reading for a paragraph and then ask students to continue for the rest of the page, and then students would share the next day by re-creating their annotations on the white board (with the text projected) or by discussing their annotations with partners or small groups. During this time, students became familiar - too familiar, many of them would likely say - with the concept of close reading, and their skills improved from bafflement at my assignments to hesitant participation, which evolved into the beginnings of thoughtful, analytical reflection on the text.

Happily, my students did very well on the selected response of the unit assessment; less happily, my students struggled through the paragraphs, which we had prepared for with formative paragraphs based on our novel. Like the novel paragraphs, the unit assessment paragraphs tended to discuss simple conclusions and shallow analysis; students were identifying devices correctly - they understood when an author used a metaphor or when parallelism was employed - but they could not originally or thoroughly discuss what these devices did for the piece.

After grading the assessments, it became clear that my students could slow down enough in their reading to select correct multiple choice responses but not enough to make significant connections on their own. More practice is necessary, but the tactics I used previously were clearly not enough to help students make insightful connections within the text. To assist in this, I have selected a lively text that students often feel strongly about, The Things They Carried, by Tim O'Brien. I decided to take close reading, a skill students are ambivalent about, and set it on a digital stage, a place my students feel unequivocably comfortable, with the hope that students will be able to see close reading, demonstrate close reading, and discuss close reading to a depth at which they can reveal something significant about the text and about themselves as readers.

Objectives

Our objectives in this endeavor were to improve students' ability to close read, or to read texts for their rhetorical structure in order to make determinations about the author's purpose and to make connections with other tones, themes, or trends within the larger text.

Because close reading is an intense, slow process, students will be close reading short portions of the text (a paragraph, passage, or page) and will not be asked to read extended portions of the text "closely." The purpose of developing the skill of close reading is to use it when it is necessary or useful, not to use it all the time (Brown & Kappes, 2012). This skill is in direct preparation for Common Core Anchor Standard 1 (CCSS.ELA-Literacy.CCRA.R.1), in addition to other, specific standards addressed below.

In preparation for the end of this unit, when students were to be assessed on the skill of close reading through a full-length essay for the semester's second District Assessment, we focused on a portion of the essay rubric for this exercise. This portion of the rubric directly addresses two of the Common Core State Standards, both of which assess reading literature:

RL.11-12.3 Analyze the impact of the author’s choices regarding how to develop and relate elements of a story or drama.

RL.11-12.5 Analyze how an author’s choices concerning how to structure specific parts of a text contribute to its overall structure and meaning as well as its aesthetic impact.

In the essay for the course, the rubric regarding these two standards measures both standards for students at once, on a continuum, from "Does not meet [the standards]" to "Exceeds [expectations for the standards]". The rubric appears below:

Exceeds Meets Somewhat meets Does not Meet
Insightfully and thoughtfully interprets the effectiveness of the explicit and implicit argument based on textual elements and devices. Analyzes the text to support reasonable conclusions. Evaluates how the author uses text structures to express meaning. Thoughtfully interprets the effectiveness of the explicit and implicit argument based on textual elements and devices. Analyzes the text to support reasonable conclusions. Evaluates how the author uses text structures to express meaning. Interprets the effectiveness of the explicit and implicit argument based on textual elements and devices. Analyzes the text to support reasonable conclusions. Recognizes the author’s use of text structures to express meaning. Accepts or formulates a misguided interpretation of theme/message that may misinterpret textual elements and devices.

For this intervention, we hope to find that students all increase their proficiency by one category, meaning that if a student presently falls into the "somewhat meets," he will improve into the "meets" category.

The graph below shows where each student performed in the first District Assessment, which was a formal written paragraph on a piece of nonfiction. The information below includes only those students for whom all information is available; some students have not yet finished the intervention or final assessment. In the table, students' performance has been translated from rubric descriptors into numerical representations of their proficiency. The number "1" indicates a score of "Does Not Meet" on the rubric, and the number "4" indicates that the student "Exceeds" the standard. The numbers that indicate proficiency or above ("Meets" or "Exceeds" standards on the rubric) are colored green for easy reference, while the scores that fall below proficiency are in red.

These are the test results from the first District Assessment (DA). Each number corresponds with a rubric score; the number "1" indicates "Does Not Meet" standards, with the number "4" indicating "Exceeds" standards.

My goal for students in this intervention was to see them improve one column on the rubric, by improving from "does not meet" to "somewhat meets," for example. This was measured once through the intervention grade, and then again during the final unit assessment, which is an essay and is also considered "District Assessment II".

Another goal that I had, which was more informal and less measurable, was to engage students with close reading through technology and, optimistically, teach them something about both. After the intervention, I gave students an opportunity to provide feedback on this experience, comparing it to our traditional method of close reading, on paper, to determine if this goal was met.

Methods

Students created an Educreation, in which they demonstrated their close reading after a series of lessons revolving around close reading.

Educreations is an app that serves as a whiteboard for users, while also recording their voice. The app appears to be primarily created for educators to create "lessons" for students to view, but I repurposed it for our close reading assignment, having students use the white board and record their own voices.

This is what a blank "whiteboard" looks like on Educreations. Students will upload their text in order to be able write their annotations on it, while discussing their choices using the "record" feature.
An image of the text is uploaded to be the background on the whiteboard, and students can then record their annotations on the text as they speak.

The lessons included direct instruction on what close reading is (this was review for students), guided practice with text projected on our classroom white board, modeled practice on the classroom whiteboard, independent practice with photocopied excerpts from the text, and group practice in which student discuss their close reading and the conclusions they reach through their reading. Then, students created an Educreation as a final intervention prior to the final written essay of this unit.

Students were given passages, differentiated by ability according the results chart above, and close read those passages in their Educreation, marking the text and explaining their thinking, both of which are captured by the app. They also informally answered the question "What is the author's purpose in this passage and how does that relate to his purpose throughout the novel?" on their Educreation, which is a simplified version of the essay question they are asked at the end of the unit. After creating their recording, students shared these links on our Canvas site, the LMS in our district.

An example of a student-created Educreation can be accessed here.

The results of all three assessments - the first District Assessment, the Educreation intervention, and the second District Assessment - appear below. The fourth column below records the change that took place between the first District Asssessment and the second; improvement in this area was the ultimate goal fo the intervention. The assessment results are:

This table shows results for all students, through their three assessments: The first District Assessment ("First DA"), the Educreation intervention ("Edu"), and the second District Assessment ("Second DA"). Changes are also recorded above, with the loss, gain, or same score noted between the first District Assessment and the second.

Students who demonstrated a "Gain": Of the sixteen students, eight students demonstrated a gain overall; six students gained one level of proficiency, while two gained two levels. I believe these gains can be attributed to several factors. The Educreation may have certainly helped students hone their close reading skills, but it is also important to acknowledge that the students have, at this point, had an entire semester of close reading instruction. Though I wouldn't dismiss the impact of the Educreation, I also would not attribute our gains directly to the app. Additionally, I would be remiss if I didn't acknowledge the point in semester at which the District Assessment was completed; this was the last major grade of the semester, and these essays likely received the most focused, careful attention of the students in order to maintain or improve their course grade.

Students who stayed the "Same": Six students remained the same overall: two of these students were already in the "Exceeding" category and therefore could not improve; two students gained proficiency in the intervention but did not maintain this increase in the written District Assessment; and two students did not improve in either the intervention or the second District Assessment.

Students who demonstrated a "Loss":Two students lost proficiency from the first to the second District Assessment. I strongly believe that these losses can be attributed to difficulties these students have in written expression. Had these students had the opportunity to verbally approach the final District Assessment, I believe the results would have been different, and this is something I will consider in the future with these assignments.

Implementation Processes

Before beginning the intervention:

To set up the assignment for students, I showed them a model Educreation and answered questions about the handout, which outlined what students would do and what they requirements of the activity would be (Nichols, Wood, & Rickelman, 2001).

This is the document students received to introduce this project.

After going over the handout, students received their passages; each passage was comprised of two pages from the text, and students scores on their previous district assessment determined which passage they received. There was a more accessible passage, full of very clear references and accessible literary devices, and a more difficult passage, which was still very rich but required students to be able to contextualize the action of the page within the novel in order to fully understand it. Students who scored in the "meets standards" and "exceeds standards" ranges on the previous assessment received the more difficult passage.

As the handout outlines for students, the assignment asks them to:

1. Select a passage within the larger passage that they find interesting or important

2. Determine the author's purpose in this passage that can been connected to the rest of the book (there are multiple right answers)

3. Find three devices that support the author's purpose and explain them.

4. Read their passage, discuss the purpose and the devices, and conclude in 3:00-5:00 minutes.

For example, in the model that I showed students, I determine that the author's purpose is to characterize the war as shocking for the soldiers. The three devices I see as supporting this purpose are the juxtaposition of images (sun and shade; laughing and dying), the dichotomy of diction (childishly "shinny[ing] up" a tree and once there, throwing down pieces of a comrade who has been violently killed), and the allusion of the song "Lemon Tree," which refers to the way that a lemon tree looks "pretty," but is actually filled with sour fruit that is "impossible to eat." I explain that I see all of these choices that the author has made as representative of the shock of the war, especially the way that the soldiers are fighting a violent, bloody war instead of enjoying the glory-filled victory lap that was implied at the start of their tenure in the Vietnam.

During the intervention:

Students spent one class period planning their Educreation by adhering to the parameters above, and then they proceeded to recording their work.

I asked students to volunteer when they felt ready to record, and, somewhat suprisingly, this task generated a lot of anxiety for many of my students; many balked at recording after only one day of preparation and asked to "go last." A few of my more confident and generally proficient students eventually began the process of recording. As students left the classroom to record, the rest of the class stayed with my team teacher to engage in continued work on their Educreations (for the first couple of days) and then in class discussion or activities based on the novel.

The recording process began each day as I led three volunteering students to the hallway and briefed them on how to navigate the app, showing them how to record, write, and re-record their work. Then, I led students to three different rooms, where they could work in privacy. As students worked, I circulated in the hallway to see if students needed help, and as they finished, students would come out to meet me in the hallway to learn how to save and then share their work.

Because of the public nature of the final product, students expressed some desire for perfection and recorded and re-recorded their work many times. Some students took a fifty minute class period to record their four minute work. In general, this attention to detail is appreciated; because we had a limited numer of iPads (three for the first few days of recording, and four for the last two days), students taking extended time to record became a small obstacle to overcome. The resultant need for an extra day of recording was a minor adjustment to our schedule, and it was well worth it to see students taking the task so seriously.

After the intervention:

In class, students were given the opportunity to reflect on their work in the Educreation. First, students viewed their own video and gave themselves a grade and then provided feedback on the process of the activtiy. Then, students were given a partner, whose video they viewed and commented on. Students were partnered with a peer who wrote about the same passage of the novel so that they could see how others interpreted the same text.

Sharing these creations allowed students to see and comment on the work of others and take note of strengths and weaknesses in their own close reading and that of others. After students shared, commented, and discussed their work, they were given another, new passage to close read, to analyze in an essay. Here, we hope to see improvement in their close reading skills and their understanding of the text. This assessment will be where I will determine the improvement that students have made on the rubric above.

Our timeline was:

11/4-11/14: Review concepts and methods of close reading; practice annotations and close reading

11/14: Demonstrate Educreations App and model assignment; distribute passages for students

11/15-11/21: Students create their Educreations during classtime (rotating through on the available devices; there will likely be three devices that students will have to share); while students rotate on the device, other students will be completing reading assignments and class discussions.

11/22: Educreation discussion and reflection in partners; class discussion on close-reading definition.

11/25-12/6: Close read passages for essay; write paper (due 12/6)

Learner Outcomes

After completing the intervention, students were asked to complete a review of the activity, in which they reviewed their work, the work of a peer, and the experience of creating the Educreation.

Specifically, students were asked to rate their understanding of "close reading" after this activity on a scale of one to ten, and they were also asked to give themselves a grade on the Educreation, using the rubric that I used to grade them. They were asked to complete this activity before they received any feedback from me on their work to encourage them to consider their performance and also the way that they thought about their own processing in their Educreation, as metacognition is an important element of the learning process (Kalantzis & Cope, 2012, p. 274). The results appear below; each student's perceived proficiency and self-given grade appear, with the grade that they actually received included in the right-most column.

Students provided feedback on their perceived proficiency in close reading and their own opinion about the grade their Educreations should earn. Their teacher-assigned grade appears in the right-hand column.

Thirteen of sixteen students either correctly identified their level of proficiency or underestimated themselves, showing that students seemed to be adept at understanding the rubric and identifying the rubric's language in their own work. Only three students overestimated their level of proficiency, but they still came close to their actual score, deviating one or less than one on the rubric's scale.

The final activity also asked students the following question:

"Did creating your Educration and viewing another person's help you understand close reading better? Did you enjoy this activity? (Should I do this with another class?)"

In response, thirteen of the sixteen students responded positively to the experience, many of whom specifically encouraged me to use the app with another class. Some of their reflections were:

  • "This did help me think about how to go aobut [close reading]. It was also semi-fun."
  • "I thought this helped me better understand the text."
  • "It was fun and was able to get me more focused on the important parts."

Negative responses included:

  • "I would rather do a Socratic Seminar."
  • "No, [creating an Educreation] was very awkward."

Overall, students tended to respond positively to the experience, with some concerns about being uncomfortable (recording their voices) and some nostalgia for other types of assessments students have experienced (the Socratic Seminar, for example). The collaborative aspect, while stressful for some students, seemed to be an overall positive experience whose effects we may parlay in future activities (Nichols, Wood, & Rickelman, 2001).

Teacher Outcomes

This experience was fascinating for me; so often, this kind of analysis only occurs either informally in classroom discussion or formally in writing, but both of those venues have limitations. Speaking in class is affected by the time available, the topics presented, and the feedback given from peers and teachers. Writing is influenced by the writing ability of the student, including the student's comfort level with the act of writing, familiarity with English, and understanding of argumentative and analytical writing structures.

Hearing students' interpretation, planned by not particularly polished, gave me surprising insight into how the students think about literature and stylistic choices. It also provided me with insight into areas I didn't expect in creating this intervention.

Specifically, this activity offered me insight in the following areas:

Analysis: Many students who performed in the "somewhat meets" range on the preassessment (which translates to a D or C letter grade) demonstrated a relatively high level of proficiency on this activity. This serves as a reminder to me that students who struggle with writing sometimes cannot demonstrate their proficiency in reading skills adequately through writing. More, varied, and possibly less extended opportunities should be presented to assess students in order to have a more true, holistic understanding of the students' proficiencies.

Perceived Proficiency: Asking students to reflect on their understanding of their proficiency in close reading and then, specifically, in the Educreation they created gave me insight into students' understanding of their understanding. Many students seemed to have an idea of how well they knew the material, but for those who didn't accurately predict their score, important questions are raised. This feedback, when juxtaposed against the actual scores of both the intervention and the second District Assessment, highlighted students who are possibly lacking confidence (those who underestimated their ability) and students who are possibly lacking a clear understanding of the rubric (those who overestimated their ability). Both groups of students will require further attention, but they had not previously been apparent.

Automaticity: Hearing students read aloud in a relatively pressure-free environment (alone, with multiple attempts) gave me an interesting glimpse into the reading fluency of each student. I'm not qualified to make too much meaning of hearing these readings, but as a reader myself, hearing others' read - and thereby hearing their phrasing, pacing, and misreadings - I perceived strengths and weaknesses that I hadn't before considered. Certain students who I expected to be fluent readers are not, and the converse is also true; though comprehension is a skill distinct from automaticity, the two are incontrovertibly related, and this was information that was previously unavailable to me.

Vocabulary: Students have specifically studied certain stylistic elements this year, but for this activity, they were given a list of vocabulary terms that they have learned in this and in previous years, so that they could vary their analysis. In using the vocabulary words, students revealed some misunderstandings that I can now use to plan my future instruction. For instance, the words "satire" and "sarcasm" were both frequently misidentified; these are obviously terms that students may need more instruction in order to understand.

As helpful as this activity was in transforming my understanding of my students, the limitations are real. For instance, because the intervention asked students to speak aloud, the extent to which it could prepare students for the second District Assessment was inherently limited. Though I only referenced the portion of the District Assessment rubric assessing the same reading standards assessed in the intervention (and did not include writing standards for the purposes of this reflection), writing skills undeniably impact the ability of a students to convey ideas clearly and logically. A student who can explain her understanding and analysis of literature proficiently verbally may not be able to do the same in writing; that this writing was also a full-length essay brings with it further issues of stamina, background knowledge, and even home support. Though this intervention provides interesting insight, it does not account for all of the nuance this learning standard truly represents. 

Conclusions and Recommendations

Overall, creating Educreations was a positive experience for my class and for my own practice as their teacher. Some items to consider before utilizing Educreations as a way to reinforce close reading skills are:

Devices available: In my district, devices are presently reserved for teachers completing pilot programs for those devices (Slates, iPads, etc.). There were three machines that my co-teacher and I could borrow (and a fourth lent to us by a co-worker during the last two days of recording), but because of how carefully students were completing their work, this number of machines proved to be too few. For a class of nineteen, I anticipated having at least six and possibly nine students finish each day; in reality, it was closer to four students who finished each day. More devices would have been helpful.

Technological familiarity: Many of my students were familiar with the iPad platform and could navigate comfortably on the Apple interface. For students who did not have this familiarity, though, the technology was sometimes an impediment to their work; they could not easily proceed with their assignment if something unexpected occurred (like if they accidentally closed the app, for instance). Teaching students how to use the device prior to this intervention would have been helpful and may have calmed some of the students' nervousness about completing this work.

Multiple Exposures: Because creating Educreations the first time was such an investment of our class's time and effort, using the app repeatedly would follow logically. Asking students to complete Educreations, for close reading or other endeavors, would be a wise extension of this activity. In schools with one-to-one device-to-student ratios, having these activities completed at home or during a single class period would save valuable instructional time.

End Assessment: Educreations allowed my students to practice close reading, but it did not allow them to practice writing about their close reading, which was what the unit's end assessment required. An extension activity, linking this experience more closely with the final assessment, would likely support students in maintaining the gains from the intervention in the ending assessment. Having students write a paragraph about their close reading (using the author's purpose as their topic sentence's argument and their stylistic choices as the quoted support material) may have been a practical extension for students.

The ultimate gain for students was significant enough in my experience to make this activity worthwhile for a classroom teacher who seeks to create unique opportunities to practice and share foundational skills, while taking into consideration the above concerns and observations.

  • Elizabeth Skopec