Produced with Scholar

Evaluation of an Educational Intervention

Project Overview

Project Description

An analysis of the outcomes of an educational intervention.

Icon for Modern Technology, Meet Ancient Egypt

Modern Technology, Meet Ancient Egypt

Utilizing Google Documents/Presentations and Creative Book Builder to Revitalize a Sixth Grade Research Project

Background/Context

For the past four years,the entire length of my teaching career, I've taught ancient world history at the middle school level. Each fall, my sixth grade students complete a project related to our ancient Egypt unit wherein they research a topic of their choosing and then create a picture book for younger students(5-8 years old) that shares the information they've uncovered in a unique way. The theoretical underpinnings for the overall project have their roots in the Project-Based Learning(or PBL) model. In PBL:

A project typically begins with an entry event to prompt student interest in and engagement with the issue, the development of a guiding question, collaborative in-depth research about the problem and the consideration of possible responses, the choice of a solution and the development and presentation to an authentic audience of a final product of some kind to further that solution. The entire process involves a significant level of student "choice and voice." (Petzen)

In this particular situation, the "guiding question" students develop relates to what they believe a third-grader would need to know about any given topic of their choosing from ancient Egyptian history. They work with their partner(s) to research their topic. The "problem" they are trying to solve, of course, is how can they communicate the complex information they've discovered in a way that will appeal to and have an impact on children younger then themselves. On the day the project is due, they present their "final product" (the children's book) to their "authentic audience" (their classmates).

Support for research projects is growing, as a quick glance at this Common Core Standard for middle grade English Language Arts shows. Project-based learning opportunities, such as this one, may be used not only to meet these new rigorous national guidelines, but also as " engaging learning experience(s)" that "create a context for a powerful learning community" ("Does PBL Work?"). The life-long skills of first seeking and then applying information are also well exercised in this type of project.

Impetus for Intervention

Because the children's book format forces my students to develop a plot, setting, and characters while simultaneously incorporating factual information on their topic, they are synthesizing what they've learned from the research phase. The design of this project has typically prevented them from just spitting out paraphrased information...troubles that can occur with other more straightforward research outputs, such as essays, posters, and PowerPoint presentations.

For all its strengths, however, this output of hand-illustrated children's books has its shortcomings. Often times, because the students work in pairs on the assignment, one partner carries the burden of the majority of the work. Because there is only one final product, it can be hard for them to coordinate and equitably divide the work. This problem bleeds into the research process as well, because, for example, one student will spend hours on hand-written notes, only to have a meltdown when they become lost and the team is then left with the indecipherable notes written by the second partner(who, of course, had been banking on the fact that the first partner was pulling their collective weight). As far as the project output is concerned, I've seen the concerns about creating attractive artwork for the story eclipse the attention that should be given to crafting quality written content.

When designing this intervention, it occured to me that the incoporation of technological elements might help combat a few of these issues. The hypothesis was that giving the pupils some "real-world tools they are excited and empowered to use" could only improve an already valuable educational experience(Petzen). With this year's group of students, I used the same basic project structure but employed Google Docs and Google Presentations (apps associated with Google Drive) to facilitate the research component, and the Creative Book Builder app as the medium through which the final work is presented.

Objectives

The essential question driving this intervention was: In what ways and to what extent can the incorporation of technology-based components strengthen the research project process for middle school students in terms of motivation, organization, and final output? Specific objectives that stemmed from this question are listed below:

  • Students will report overall positive feelings about this form of research project, in comparison to research projects they've completed in the past. There will be a correlation between the positive feelings and the technological components being used.
  • Participants will express that Google Docs and Presentations assisted them in dividing their workload,and easily managing and keeping track of the source information they've collected.
  • The quality of the children's stories eBooks written by the pupils will be of a higher caliber than those produced previously in terms of plot, characters, setting, and theme.This will be due to the fact that time-consuming element of hand-drawing illustrations will be eliminated by the eBook format and more time can be spent tweaking storylines.

Google Docs is the technological component examined most closely in this project. It is a relatively new application,made available to the public only in 2006, and there is not much research to support its efficacy, particularly at the middle school level. A 2009 report by a teacher-librarian at a Canadian high school did, however, extol Google Apps for "providing a common collaborative system that virtually supplies all the applications and communication tools needed, under one platform, and at no extra cost" and noted as well that because it "eliminates or limits the need for printing and photocopying it is...easier on the environment"(Nevin 38). In addition, a study of college students who were introduced to Google Docs for collaborative work found that " 93% of students considered Google Docs a useful tool for group work" and half reported a desire to work with the techonology again (Zhou, Simpson, Domizi 359).

The Creative Book Builder app is even younger that Google Docs, first appearing on the scene in July of 2011. There does not seem to be any formal research regarding it in particular, but much has been said of eBooks and digital storytelling tools in general. Bill Cope and Mary Kalantzis mention the "multimodality" of these types of projects in Literacies. They argue that it is important to "expand the idea of literacy beyond reading and writing" and "include the literacies of new, digital media"(198). The eBooks the students will be making in Creative Book Builder will allow them to incorporate so much more than just mere words. If they wish to incorporate short animation clips (such as GIFs),give voices to their characters, add silly sound effects, or include an image turned up by an Internet search, they may.

A study conducted in Egypt in 2008 gives credence to the idea that unique opportunities for telling stories electronically can have positive impacts for learning and student motivation. This research explored a different platform(called Microsoft Photo Story) than I will be using, but found evidence that "teachers perceived students to be motivated and excited" and "students enjoyed the idea of piecing together their thoughts and connecting them any which way they wanted to by titles, audio,narrations, motions, transitions and other...effects" (Sadik 502).

The data provided by this intervention should help explore whether Google Documents technology is well-suited to research projects at a lower educational level.In addition, it will examine the power of eBooks to motivate students to create high quality mulitmodal work.

Methods

This project consisted of three major phases, each with a different technological component.

Phase One: Google Documents for Research

At the beginning of the project, students selected a topic and developed a research question. They then located appropriate sources, both print and digital, with information on their topic, and recorded, using a Google Doc developed by the school technology coach, librarian and myself, the information they found in their own words. Below is a screen shot of a segment of one group's completed Google Document. Student names have been removed to protected their information. This image shows three pages of a total of nine that these students created, but the main elements of the document are shown.

Student Work for Phase One of the Intervention

A blank copy of this form was shared by me electronically with every partnership, and all students were provided this structure and scaffolding to help them through the research process. Only the students and I were be able to view what was typed as they worked.Changes to the document were recorded in real time(no saving necessary)and multiple people were able to type on the same document at once,which allowed students to examine and record information from multiple research sources at once.The boxes provided in the form grew as they were typed into so there were none of the spacing issues of traditional paper worksheets. It was possible to search through the revision history in each partnership's document and restore versions if need be, which allowed me to assist students when they accidentally deleted a section, as well as to monitor who was writing how much and when.

Phase Two: Google Presentations for Collaborative Story Writing

After they Google Document was created and approved, one student in each group initiated the creation of a Google Presentation that they shared with their partner(s) as well as with me. Over the course of several days, students had some class time but worked primarily at home to write their creative stories incorporating the facts that they recorded in phase one. Because Google Presentations works in real time, they did not even necessarily need to be in the same location in order to collaborate.In most regards, Google Presentations is extremely similar to Google Documents. The only reason I elected to use it for this portion of the project is that it was useful to the students to visually break up their story by having each slide serve as a separate page for the eBook. Here is a glimpse at the user interface my students encountered while working with Google Presentations.The work shown is from the same students whose research was showcased in the previous section:

Student Work in Google Presentation

Phase Three: Creative Book Builder for Multimodality

Following the completion of phase two, stories were exported to the Creative Book Builder app by means of a spreadsheet from Google Docs. The school technologist walked us through this process.Once the writing was transferred the students had free reign to work on incorporating images(either those they found through legal online searches or those they created by drawing within the app), videos, and audio(again, they found these online or recorded their own voices) to complement their story. Here's a screenshot showing a work in progress. Once again, this product was created by the same students whose work is displayed above:

Creative Book Builder page

The students had several days to locate digital enhancers for their stories and put the finishing touches on their eBooks before they were published. The publishing process involved exporting the books in ePub format to the iBooks app, so that they could be easily viewed and shared with others. On the due date (still forthcoming as of the publication date for this piece)students will read their stories to one another and enjoy a relatively carefree day of collaboration.

Evaluation Methods

To best address the way I evaluated the effectiveness of this technology-mediated intervention, let's revisit the objectives I outlined in a previous section. Here they are again, in simplified form, along with the primary way they were analyzed:

Positive feelings about this research project(in comparison to past projects) Students were given initial survey with questions regarding past research experiences before the project began and then a survey regarding their thoughts on this particular research project as it neared completion.
Division of workload among partners/information organization The pre-project survey also included questions related to how students have felt when working with partners in the past. The second survey had them reflect on how their partner working was progressing These questions on the survey alluded to how work(research notes, story writing) done in Google Docs is different from the same type of work completed on paper with pencil.
Quality of writing in eBooks I accessed my gradebook from last year, in which I had stored the comments I wrote to former students who did the same project, without the technology-based interventions. I investigated the number of comments I wrote last year that had suggestions about improving any element of the story the students wrote. I compared that number with the current number of student projects about which I had concerns related to the written content.

The two surveys I created were administered as a Google Form. These surveys were logistically quite easy to create and for the students to complete, given the fact that both they and I have a relatively high degree of familiarity with Google Apps for Education(many of its features were introduced in our district at the beginning of last school year). Students were given class time to completed the surveys, which were pretty limited(fewer than ten questions), so as to not be overwhelming. The surveys provided an opportunity to get quantitative data, as respondents selected a numerical value to represent their feelings on several topics, but also qualitative data, because students needed to explain their ratings in writing. The two surveys I distributed to students are visible in the appendix section of this work.

It was made clear to the participants that how they responded on the surveys would not impact their grade in any way, shape, and form, and that their only purpose is to help me understand how I could best shape research projects to help them be beneficial and enjoyable.

Implementation Processes

The project took place in the space of three full weeks of school. It contained of a small amount of teacher-led instruction, but primarily consisted of student partnerships working independently on their own projects.The majority of the direct instruction was provided by me, however, on certain days the school's librarians provided support. In addition, when we began using Creative Book Builder, the school's technology coach "pushed in" to each class section to describe and demonstrate the more complicated features of the app. All 109 of my students(grouped into four class sections) participated in the project. A summary of what occurred during each week of the project is provided below:

  • November 20th-22nd and December 2nd-5th.Students worked on phase one. They first selected topics and then completed the research using the Google Doc template shown previously in this work. By December 5th, most had begun writing their story using Google Presentations(phase two) at this time as well.
  • December 6th-11th. During these class periods, content related to a different unit was presented but the students continued to write their stories in Google Presentations outside of class time(after school hours and during lunch). These stories were completed by the morning of December 11th.
  • December 11th-13th and 16th. These class periods were dedicated to learning how to use Creative Book Builder and then searching for audio and visual components to incorporate into the eBooks.
  • December 17th-19th(post publication).Students who require additional time to tweak their stories and find suitable multimedia elements for their eBooks will be able to use iPads before school, after school, and at lunch prior to the December 19th due date. Material not connected to the project will be covered in class.

As far as devices are concerned, our school is not at 1:1, but the necessary equipment was procured with the help of the school's technology coach. During weeks one and two a classroom set of MacBook laptops on a mobile cart were utilized, and then an iPad cart was used from December 11th-19th.

Overall, the time frame was sufficient for the task at hand. The fact that a week in the middle of the intervention was lost to Thanksgiving Break was simultaneously positive and negative. The students got their feet wet but did not become overwhelmed early on because after two days of research they were able to step away from things for a nice long rest, but by the same token, a fair amount of momentum might've been lost. Certain students did occasionally seem to lose sight of the end product they would be creating, and forget that the eBook was the ultimate goal. It was also a bit difficult to begin new content (a unit on the ancient Israelites) in the midst of the project, but timing restrictions dictated that this was the way it needed to be. On the other hand, too many days of complete immersion in the project also might've resulted in drudgery and the students did not seem overly burdened about the fact that some of the story writing occurred outside of class. The timing I allocated was the same as in years past, but I was anxious that the introduction of the new technology might've slowed things down. On the contrary, the students adapted quite readily and worked on (or perhaps even ahead of) pace from previous years.

Learner Outcomes

This section primarily consists of an analysis of the data collected from the surveys distributed prior to and towards the end of the intervention. There were a few items included on the initial survey that are not discussed here, because as the project became more refined these questions were not relevant to my objectives. The number of respondents to each survey varies slightly because while all of my students were present on November 15th, when the intial survey was taken, around twenty of them were absent on the day of second survey(December 13th) due to a club field trip. The absent students were advised to take the survey at home and some complied but many others did not. Thus, the number of respondents for the initial survey is 109, and for the second survey only 91. Although this might weaken the data slightly, because the majority of students took both surveys the results are worth sharing.

I have paired up results related to four questions that stem from my objectives. The raw data is provided first in the form of pie charts with corresponding percentages and then interpreted below with samplings of student responses to various questions.

Initial Survey Data Point #1
Follow-Up Survey Data Point #1

As the first pie chart(blue) indicates, only 38% of students indicated clearly positive opinions about research projects overall, with 20% declaring outright negative feelings. In response to this particular project, however, almost 60% of students indicated that they were enjoying the creation of the final product, and only 8% expressed purely unhappy feelings about the project.This seems to indicate that the creation of an eBook fares better than average as a research project from the students' perspectives. Some comments students made about Creative Book Builder app included:

  • It is easy to work with and you can add a lot of media.
  • Yeah, I'm not a tech genius so I'm having some trouble but it's good either way. It's cool how we are using eBook
  • I like how you can add images and sound effects and that it will be different from everyone else's.

_____________________________________________________________________________________

Initial Survey Data Point #2
Follow-Up Survey Data Point #2

From the first survey we see that the majority of students(57%) are positively inclined towards computer technology, a percentage that is higher than those with identify predisposition towards research projects in general. An even higher percentage (89%) indicated in the second survey that they enjoyed doing this project in a technology-based fashion, particularly through Google Docs and Presentations, and only one student indicated that they disliked those applications all together. Some interesting comments that students made elaborating on their response to the question on the second survey include:

  • Because I think that it is kind of boring when we have to hand write everything. It is better in my opinion to do the story on the I pad[sic] and the laptop
  • I like google drive/docs because it is very easy and it saves your work automatically and you don't have to worry about losing your work.
  • It doesn't hurt my hands and we can get the correct spelling.
  • I like hands on, but computer tech-y stuff is cool too! After all, in the future we will need to do computer stuff more and more!!!!!
  • I like to use google drive because you can continue it from wherever you are
  • It can be very scattered when you use paper and it's good to have it on one thing.

Some students did express concern about technical difficulties souring their feelings towards computers ("Sometimes technical stuff can be annoying because it glitches sometimes and you can easily delete things that are really important."), but the majority seemed to see the potential for the technology to enhance their work and make their tasks easier.

_____________________________________________________________________________________

Initial Survey Data Point #3
Follow-Up Survey Data Point #3

The top chart here shows that initially only one-third of the students communicated no trepidation whatsoever over working with others on projects. On this particular project, 82% of students seemed satisfied with their partners overall. What is interesting to note, however, is that no comments directly mentioned Google Docs/Presentations are something that helped make things more equitable in groups. So all this project appears to have shown in improvement in feelings towards partner work, it seems that the technology component may not be behind this. In fact, the technology seems to have added, in some cases, a new level of tension. Several different respondents expressed in writing that they were feeling agitated over the fact that they had only one iPad on which to create the final product(i.e. "I feel like I hog the iPad" or, on the opposite end of the spectrum, "I would like to use the IPad more though."). This was an unanticipated side effect of the intervention.

_____________________________________________________________________________________

Initial Survey Data Point #4
Follow-Up Survey Data Point #4

Finally, 42% of respondents indicated a definite level of stress associated with long-term projects. On the second survey,however, an impressive 44% said that they were not feeling in the slightest concerned about this particular project one week out from the due date, and barely over one-tenth of students declared outright anxiousness. Once again, however, none of the students directly connected technology to their relatively low stress levels (one student, in fact, pointed out technology in a negative way: "I'm just kinda stressed about the technology").Perhaps the positive results of this survey might be more connected to the overall design and timeline of the project than the technological elements I introduced this year. This data point might have been more compelling, perhaps, if I had comparable data about how students completing the project last year felt a week prior to the due date.

The learner data overall seems to point towards the technology incorporation as a motivating factor to students, and also as tool to help them work through multiple steps in an organized fashion. It is also suggested by the data, however, that technology cannot be "cure all" for all difficulties associated with a collaborative project experience.

Teacher Outcomes

The other data I evaluated for this project is related to my third objective. Last year,I wrote critical comments regarding the quality of the children's stories on 36%(11 out of 30) of the projects. A sampling of the comments I made to last year's students are:

  • developing your main character's personality more would’ve made the story a little more enjoyable for the reader.
  • I thought the ending of the story was a bit sudden.
  • I do think you might’ve done a bit more to keep things exciting.
  • Overall, though, your story was a bit bumpy…some parts didn’t connect very well to the next part.
  • At times, because of all the characters and events, things got a bit confusing.

As I scrolled through this year's presentations on Google Presentations, I noted concerns regarding plot, setting and characters in 29%(12 out of 41) of projects. To me it seems that the overall caliber of the stories is about the same, so even though the statistics may show some improvement, from my perspective, it does not seem that the technology afforded much improvement in this regard. What should also be noted, however, is that there are no groups I worry about not submitting anything at all on the project due date. The technology was beneficial in this regard. In years past, because the majority of the work might've been done either by hand or using a word processor, kids could claim to be further along than they were in reality and I would have no way of truly double checking them.

This intervention was a success in general. The implementation went smoothly and the technology presented only minor hiccups. Observationally, the students seemed more gung-ho about the research simply because they were able to use the computers. One advantage I had not really anticipated was the ease with which I was able to provide immediate feedback to the students as they worked with both research and story writing. Because I had access to all of their work in one folder, I could simply click through and offer a quick piece of advice verbally or through written comments before moving on to the next group. This would've been much more difficult had their research notes, for example, been hand-written. I could access their work from any computer at any time...and even on my mobile phone!

In terms of the classroom architecture, I was able to remain seated in one central location with my laptop for the most part, where I could keep an eye on almost every group simultaneously, as opposed to trying to flit around as much as I typically would.The Google Documents and Presentations allowed me to keep tabs on everything easily because even if I did not touch base with a group much during class time, I could follow up with their progress later that day or evening. The research notes were undeniably more organized for the students than in years past. I printed off their research Google Document for them prior to the part of the project where they composed their stories and the students were quite responsible about using these print-offs as a checklist as they collaborated to write.

Although it was my hope that the Google Documents and Presentations would help ease tensions between partnerships by holding everybody more accountable, the fact remains that there is no magical solution that can ensure that work is divided equally among partners. Though the students were continuously reminded that I could see who was doing what when via the Google apps, there were roughly the same amount of  interpersonal problems that I'd encountered in the past.

There was a great deal of extra work on my end to go about the project with this many technological components. The creation and distribution of the research template, for example, took about an hour of time that would not have been necessary had I gone about the project in the previous way. It was a bit of headache to ensure all of the devices I needed were available as well. The extra feedback I was able to give to the students while they were in the midst of the project, however, made this extra trouble worth the effort in the end. I am certain that I will execute this project next year again with the elements explored in this intervention.

Conclusions and Recommendations

A shortcoming of this study is that aside from its impact on motivation,the validity of Creative Book Builder as a tool must be explored in greater detail.It was unfortunate that I was unable to fully investigate its components on a deeper level at this time.

This project seems to support the notion that Google Documents and Presentations are useful tools for research at the middle school level, in that they allow students to organize their work and collaborate in real time as they write notes and synthesize information. It does require quite a bit of know-how on the part of the educator to help students navigate this(or any) computer-mediated application with ease. Indeed,experimenting with and investigating the impact of technology on a real-life classroom is an exhaustive, labor-intensive process, but one that is well worth the effort. If students and teachers alike operate with open-minded patience and remember to have realistic expectations about how technology can affect the learning process, then, slowly and steadily, progress can be made.

Works Cited

"Does PBL Work?" Buck Institute for Education. N.p., 2009. Web. 10 Nov. 2013. <http://www.bie.org/research/study/does_pbl_work>.

Kalantzis, Mary, and Bill Cope. Literacies. Port Melbourne, Vic.: Cambridge UP, 2012. Print.

Nevin, Roger. "Supporting 21St Century Learning Through Google Apps." Teacher Librarian 37.2 (2009): 35-38. ERIC. Web. 10 Nov. 2013.

Petzen, Barbara."Project Based Learning in a Computer-Mediated Environment".Scholar, 27 October 2013. Web. 9 November 2013. <https://cgscholar.com/community/cg_community/profiles/barbara-petzen/publications/21843>

Sadik, Alaa. "Digital storytelling: a meaningful technology-integrated approach for engaged student learning." Educational technology research and development 56.4 (2008): 487-506

Zhou, Wenyi, Elizabeth Simpson, and Denise Pinette Domizi. "Google Docs In An Out-Of-Class Collaborative Writing Activity." International Journal Of Teaching And Learning In Higher Education 24.3 (2012): 359-375. ERIC. Web. 10 Nov. 2013.

Appendix

Initial Survey Administered to Students: 

Part 1
Part 2
Part 3

Follow Up Survey Administered to Students:

Part 1
Part 2
  • Madeline Kiem