While in graduate school I knew a woman working on her dissertation in Economics. Her committee chair advised her to use a Chow Test as part of her dissertation analyses. Since I was increasingly becoming interested in all things related to statistics and research methodology, I looked up what her advisor had recommended. A Chow Test allows one to compare the coefficients in two linear regressions to determine whether the independent variables have differing impacts on two groups (Chow, 1960). It was originally conceived of as a way of detecting structural breaks in time series data. At the time it was not really applicable to what I was doing, but I filed it away as a technique I might get to use one day.
I was recently asked to do an evaluation of a one-year educational program. The way the study was set up I had to use regression techniques to take into account student and teacher characteristics when assessing whether the program was related to an increase in student achievement or not. We found that the program enhanced the effectiveness of more experienced teachers. In response, we got the question, “If you give anything to a more experienced teacher, won’t students automatically do better?” Basically there was doubt that the program had anything to do with it, which was reasonable given that the study was not a randomized controlled trial.
In response, I was asked if it would be possible to use data from another year with the same model and compare whether there was a change after the program was implemented. I used assessment data from the prior year for the schools to see if there were differences in the behavior of the variables between the two years. The students who took the assessments across the years were different, but teachers and schools were the same, thus creating a small time series. I realized I was finally able to use this technique I had learned about many years before. I was able to compare the models from both years using a Chow Test, and found a significant change in the behavior of the teacher experience variable. Though not conclusive, in the absence of other evidence, this seemed to suggest that use of the program was related to a change in the relationship of teacher experience to achievement in the schools. As evaluators we often presented with imperfect data, and one never knows when a random bit of information, possibly learned years before, will come in handy.
Chow, Gregory C. (1960). “Tests of Equality Between Sets of Coefficients in Two Linear Regressions”. Econometrica 28 (3): 591