Measurement Challenge

Posted: July 8th, 2013 | Author: Michael Goldstein | | 3 Comments »

MCAS, I’ve taken you for granted.

Until now.  Forgive me. 

MCAS is a test that measures the progress of all Massachusetts students in Grades 3 to 8, plus Grade 10, in English and math.  All (or most?) American states have such tests.

Such test data isn’t the be all and end all.  Schools measure other things, or should.  At Match, that included: Parent satisfaction, college persistence among our alumni, literacy tests for young children called STEP, and all sorts of teacher-created quizzes and tests and other assessments (commonly summarized on report cards).

But student achievement data — how much the kids are actually learning — is a critical foundation for all decision-making about schools.

Kenya has an 8th grade exit exam, called KCPE.  Do well, you can go to high school.  Don’t do well, you can’t.

But there aren’t annual nationwide tests designed to measure student growth.

There are “zonal exams,” but they vary quite a bit, and they’re designed to create rankings within a region based on absolute scores, rather than student growth.

Nor is it easy to compare Kenyan schools to those in the rest of the world.

Kenya doesn’t participate in TIMMS. (Though the director of TIMMS told me he is trying to get Kenya to sign up).

There is an African assessment called SACMED, which seems to be only each 7 years.

So now what?

Absolute data is most valuable when you can compare to schools with similar demographics.

Even more important, growth data matters.  It can help you find what works.

For example, at a certain point, it became clear that one Massachusetts high school, with many kids from poor families, did unusually well in raising English scores.  It ended up on the front page of the New York Times.  Without growth data, there’s no way to discern something like that.  The absolute scores at Brockton High were not impressive.  Only the growth scores.

I asked Esther Duflo about this.  She agreed it’s a challenge.

I’m paraphrasing from memory, but she said:

“From our point of view, as economists, we mostly conduct randomized control trials.  So any reasonable assessment will work for our particular work, so long as they’re given to control and treatment groups.  Essentially, we create our own Value-Added numbers.  But as for large existing data, equivalent to annual MCAS, where you can look up VAM instead of needing to calculate it from scratch, to my knowledge it doesn’t exist yet.”

To deliver quality education to any kids, you have to be able to measure growth, so you can find out what works.  You can find standout teachers, standout schools; you can try new ideas with a small number of schools and measure them carefully, rather than what most USA school districts do (never test an idea — just roll it out to all of the schools).

A friend in Kenya challenges me:

Your writing implies that the KCPE and KCSE are a way that teachers can track strengths and weakness and refine their teaching, which you say happens among some teachers in USA.  In reality, the two are pretty unrelated here.  Teachers seem to teach in whatever way, and the KCPE and KCSE outcomes appear unrelated.  While teachers follow the syllabus, I’ve not heard of a teacher changing their teaching strategy in order to pass these exams.

Perhaps.  But measurement might affect teacher behavior there too.  The normal thing, without VAM, is to think “Well our pupil test scores are mostly a function of how smart the students were before they even showed up in my class.”  If you have VAM, though, you can say “How much progress did my students make this year, compared to the progress of other teachers?”

Which gets us back to:

How should we measure kids’ annual growth before Grade 8, before the KCPE?

More on this puzzle to come.

---