The Problem of Using SDs to Compare Effect Sizes

Posted: January 14th, 2015 | Author: | | 2 Comments »

Ryan suggested yesterday that I check out this blog by Abhijeet Singh. It’s over at World Bank blog.

Title? “How standard is a standard deviation? A cautionary note on using SDs to compare across impact evaluations in education.”

C’mon. If that doesn’t get you excited, then you’re not geeky enough.

Singh writes:

In this post, I explore some of the reasons why such comparisons might be flawed and what we might do to move towards less fragile metrics.

Dispersion is not constant across populations

The most fundamental issue with such comparisons is that standard deviations are merely a measure of dispersion – and this is not constant across samples. So an intervention delivering the same absolute increment in learning would look less effective in a context with high variance in test scores than in another with low variance. These differences can be important. In the 2012 PISA assessment in math, the SD of test scores was 75% lower in Costa Rica than in Taiwan. On the same test and at the same age. So, like David’s hypothetical example in his post, the impact of an identically-effective (in absolute terms) intervention in Costa Rica will look 75% larger than in Taiwan.

More worryingly, it is possible that…

Read the whole thing here.

Also an idea from Justin Sandefur in the comments:

A modest proposal:

All we need to do is gather up all the disparate and uncoordinated exams being administered by researchers around the world. Get a bunch of kids, ideally from diverse backgrounds, and stick ’em in a room. Make them take ALL the tests. Then link them up. That way we can compare PISA with SAT with SACMEQ with anything. Simples.

Only half joking.

2 Comments on “The Problem of Using SDs to Compare Effect Sizes”

  1. 1: Veda said at 4:55 am on September 11th, 2015:

    What I really like about this diva is that she stays away from the
    limelight. This is a football simulation game in which you can select teams
    from real NCAA conferences, create and use the plays, and manage recruiting and development of players.
    Football greats like Ronaldo of Brazil are remembered
    by their numbered jerseys.

  2. 2: Web design said at 8:35 pm on December 19th, 2015:

    Creating an ontology for the provided information is the main step in the logical data
    design phase. Like everything in business is important, getting a website is important too.
    Choosing among these vast numbers of provider is not
    so easy.

Leave a Reply