Latest News

We are pleased to announce that Merola Research LLC in partnership with the Y in Central Maryland and Riverview Elementary School in Baltimore and Baltimore Information Technology Academy were awarded 21st Century Community Learning Centers grants. Merola Research LLC will be the evaluator on these projects. We are excited to be working with such a great team!

Discovery Education evaluation highlighted in eSchool News

eSchool news, a website devoted to news, information, and resources on how today’s educators are using technology to advance learning, recently featured the results of the Discovery Education evaluation of its Science Techbook, performed by Merola Research LLC on its site.

See the full article.

Study shows effect of Science Techbook on student achievement

Merola Research LLC is proud to have conducted a recent study of the effectiveness of Discovery Education’s Science Techbook in helping to improve student achievement in science. Discovery recently released the results of the evaluation.

During the 2011-2012 school year, Discovery Education’s Science Techbook was implemented in Collier County Public Schools. Professional development was provided to all teachers during the summer of 2011 to train them in the use of Science Techbook, and teachers subsequently chose whether to use Science Techbook in their classrooms or not. Data related to the results on the Florida Comprehensive Assessment Test (FCAT) in science for fifth and eighth-grade students were analyzed to assess whether there were differences in student science achievement when teachers used Science Techbook versus when they did not.

The findings indicate that the relationship between Science Techbook use and science achievement is mediated by teacher experience. Science Techbook is effective in improving student achievement in science when it is used by teachers that already have a pedagogical foundation gained through prior years of teaching. Students whose teachers had greater amounts of teaching experience and used Science Techbook scored higher on the FCAT than students whose teachers had similar levels of experience but had not used Science Techbook. This result was not evident for students in classes with beginning teachers, indicating that beginning teachers may need to be provided with alternate training when implementing Science Techbook.

The full report is available at: http://www.discoveryeducation.com//feeds/www/media/pdf/Case-Study/Collier-County-Florida-Science-Techbook-Report-11-20-2013-long-v4.pdf

An abridged version is available at: http://www.discoveryeducation.com//feeds/www/media/pdf/Case-Study/Collier-County-Florida-Science-Techbook-Report-Abridged-v5.pdf

MdBioLab Increases Interest in STEM

Increasing the numbers of U.S. students who go on to choose STEM careers has been a national priority for decades, and stake holders in both the education and business sectors have struggled to develop ways of encouraging students to pursue these careers. Communities have a vested interest in promoting STEM careers as they strive to develop local technology centers.

One example of an initiative in Maryland providing marketing for STEM careers is the MdBioLab. The MdBioLab is a mobile science lab in a tractor-trailer that travels to high schools across the state during the school year, spending about a week at each high school. It was created by the MdBio Foundation as a way of increasing interest in science careers among middle and high school students in Maryland by giving students hands-on experience in a lab. Since its 2003 launch, the MdBio Lab, which recently received at $20,000 donation from AT&T, has visited all 24 school districts in Maryland, with roughly 10,000 students participating each year, or about 100,000 since the inception of the program.

In an evaluation context, a sample of 10,000 participants would be a large study. Evaluating the long term impacts of short programs like the MdBio Lab is often a challenge, as is determining whether any specific program inspired career choices that might occur years after the program; however, short term impacts could be assessed by surveying students about their attitudes towards science and STEM careers prior to their participation in the program, and comparing their responses to a matched sample of similar students that did normal coursework for the same week. To obtain preliminary data about the long-term impacts of the program, former participants could be sent a short survey about their careers and how they feel the program impacted their lives. Given the large number of former participants, this could be a very informative study about the effectiveness of these types of programs that would be useful to other localities interested implementing similar programs.

Chow Tests

While in graduate school I knew a woman working on her dissertation in Economics. Her committee chair advised her to use a Chow Test as part of her dissertation analyses. Since I was increasingly becoming interested in all things related to statistics and research methodology, I looked up what her advisor had recommended. A Chow Test allows one to compare the coefficients in two linear regressions to determine whether the independent variables have differing impacts on two groups (Chow, 1960). It was originally conceived of as a way of detecting structural breaks in time series data. At the time it was not really applicable to what I was doing, but I filed it away as a technique I might get to use one day.

I was recently asked to do an evaluation of a one-year educational program. The way the study was set up I had to use regression techniques to take into account student and teacher characteristics when assessing whether the program was related to an increase in student achievement or not. We found that the program enhanced the effectiveness of more experienced teachers. In response, we got the question, “If you give anything to a more experienced teacher, won’t students automatically do better?” Basically there was doubt that the program had anything to do with it, which was reasonable given that the study was not a randomized controlled trial.

In response, I was asked if it would be possible to use data from another year with the same model and compare whether there was a change after the program was implemented. I used assessment data from the prior year for the schools to see if there were differences in the behavior of the variables between the two years. The students who took the assessments across the years were different, but teachers and schools were the same, thus creating a small time series. I realized I was finally able to use this technique I had learned about many years before. I was able to compare the models from both years using a Chow Test, and found a significant change in the behavior of the teacher experience variable. Though not conclusive, in the absence of other evidence, this seemed to suggest that use of the program was related to a change in the relationship of teacher experience to achievement in the schools. As evaluators we often presented with imperfect data, and one never knows when a random bit of information, possibly learned years before, will come in handy.

Chow, Gregory C. (1960). “Tests of Equality Between Sets of Coefficients in Two Linear Regressions”. Econometrica 28 (3): 591