A Blog Covering D.C. Education [ABCDE]

Closing the Teach For America Blogging Gap
Dec 12 2010

Less Circumstantial, and More Direct, Evidence in Education, Please!

weighing evidence

One of a grand juror’s primary responsibilities is to evaluate the credibility of evidence presented by a prosecutor. Can one trust what the witness is saying? Were there any inconsistencies in the statement? Additionally, a grand juror must factor in the type of evidence involved: is it “direct” (e.g. eyewitness account or offender’s confession) or “circumstantial” (e.g. police reports, shell casings, blood patterns) evidence? The judgment exercise that is known as “weighing” the evidence is as important, if not more so, than the actual “content” relayed by such evidence.  Over the last 4 weeks, I’ve become as critical of the source of information as I have of the information itself.

We need the same sort of critical eye in education, since, in my view, there is often too large of a disconnect between classrooms and “research.” All too often, reports are published–by think tanks, councils, departments of ed, statistical houses–and their conclusions are trumpeted as “evidence” that such and such change is needed in education. Yet, many of these conclusions are drawn from circumstantial–”secondhand”–evidence. Test researchers might infer that X policy change is needed to remedy Y problem based on looking at Z dataset.  All this without said researchers ever setting foot inside a classroom during test-taking season and truly understanding the attitudes students and staff have towards tests and the unique dynamic of test “culture” (which, believe it or not, affects overall school culture in substantial and less noticeable ways).

This is not to say that I expect every research conclusion to be “validated” or supplemented in some way by classroom observations or interviews of students and teachers. That would be entirely unrealistic–costly, logistically-frustrating, and, in some situations, unreliable.

What I think education needs, however, is a shift in mindset, where every attempt is made to include such “direct” evidence whenever appropriate. This, I would say, is entirely possible.

I was inspired to think that education could make this shift after reading a Forbes profile on Gerrick Johnson, a toy industry analyst who is taking the idea of “firsthand” research to a new level. Johnson makes plain-clothes visits to up to six stores per week and takes note of the nuanced indicators of toy industry trends.  Which toys have the most shelf space?  Which toys are on sale?  What items have the acronym “ncf” (“not carried forward”) on them?  When shoppers enter the store, to what section do they gravitate? All of these observations go into his reporter’s notebook:

Everything I see means something for my earnings model. It’s all about incremental information.

What caused Johnson to lift his behind off of his office chair in the BMO Capital Markets office and instead spend his hours making store visits? One of his clients tipped him to the idea–so basic, yet often ignored–that the reports that analysts generate rely almost wholly on secondhand information. Instead of using information from the ground-level, analysts typically cited industry reports and market surveys to conduct their analyses.  That needed to change:

I decided I wanted to be the guy who’s on the street kicking the tires.


The “Johnson approach” can and must be replicated in education if we want to eliminate the disconnect between practitioners and policymakers. A public school system, after all, is one large bureaucracy, with a hierarchy managed by people who are stationed at varying degrees of separation from students–those the system is ultimately intended to benefit.  Sadly, the orders that are handed down from top-level administrators often show little sensitivity to the nuances of the classroom itself. This is why ground-level educators rebuke so many of these directives (see this disheartening story as one example of the disconnect I speak of).

Can we nurture more Gerrick Johnsons in the world of education research and reform? Definitely yes.  Will it be easy? Definitely not.  One big barrier is our current notion of what “research” should look like. Research today is about using R or SPSS to analyze datasets, to conduct certain studies and pontificate on whether the results have a low p-value.  Johnson doesn’t reject such “secondhand” evidence.  But he certainly takes an out-of-the-box approach by attempting to nurture his entrepreneurial spirit in pursuit of his passion for toys.

I think the best example of education’s Gerrick Johnson is Doug Lemov, whom I’ve already written about here and here. He analogously spent thousands of hours in the classroom recording the nuances of great teachers and their techniques.  He used such direct evidence to support his deep understanding of learning theory and pedagogy to write what has become an Amazon Top-100 book (unthinkable for an education book).  He didn’t abandon “traditional” notions of research; he augmented it through atypical means.

Education needs sharp analysts like Johnson and Lemov, who will get their hands dirty by jumping right into the trenches and drawing conclusions that may not be statistically sound, but have a nuanced and hopefully “truer” take on the issues at hand.  Sadly, there aren’t many Lemovs in this world.

Edit: One thing I neglected to mention was that the Gates Foundation-funded “Measures of Effective Teaching” (MET) project recently released what appears to be a large-scall Lemovian analysis of teaching. NYT news article here.  Policy brief here.


But I think I also have a broader, related point to make, which is that TFA teachers who “abandon” classroom teaching in order to pursue other careers still retain the direct evidence of their teaching experience. And such evidence is invaluable. Particularly within education itself, no amount of experience at an education think tank can replace the in-classroom experience (even if only for two years). Outside education, such evidence can greatly influence the approach that the aspiring doctor, lawyer, policymaker or businessperson takes.  As I have learned, some times life’s biggest lessons can come from a single day or even a single moment in the classroom.

Ironically, my time on the grand jury has given me some “distance”–physical and emotional–from the class, which has, in turn, allowed me to reflect as I have just done. A change of pace can work wonders on the brain’s ability to come up with new ideas.

Perhaps the ultimate point, then, is that we always need to see things from new angles and not allow what we have grown accustomed to doing to narrow our window on the world, lest we miss the surprise that lies just around the corner (wow that was filled with cliches–apologies).

profile counter

No Responses Yet

    Post a comment

    About this Blog

    Really, "A Blog Covering Dilemmas in Education": A (former) English teacher's reflections…

    D.C. Region
    High School

    Subscribe to this blog (feed)